How to deal with Big Data challenges for financial application performance

Przemysław Szyszka - Senior Software Developer

Applications that handle large amounts of data are naturally more prone to performance issues. The result can be slower operations, which in turn may have a huge negative impact on user experience. In our experience at iRonin, it’s best to invest in upgrading your app and architecture sooner rather than later, to avoid hurting your brand and to optimize your costs. Let’s dive into why we recommend this approach.

Big Data performance issues

The simple truth is that, despite the advances of modern technology, operations carried out on large amounts of data take time. And while speeding up data processing is a complex problem (which tends to result in micro-optimization), the root of it isn’t always just the fact that a lot of data is needed. Sometimes, the trouble begins much earlier - with the creation of unscalable software.

To combat this, it’s good to stay up to date regarding major software updates that your project may need. We recently covered the topic of limiting the risks that come with upgrading applications. Now, let’s take a look at what we can do for Big Data projects, especially in a highly regulated sector such as fintech.

Maintaining scalable applications

At iRonin, we use a number of strategies to make sure our clients’ application will scale well and be prepared to handle large volumes of data.

Separating migrations from code changes
Doing so leaves us the option to easily revert code changes at any time without worrying about locks. Migrations in Ruby on Rails, when they are being applied to a database table that already exists, lock the reads (in most cases) and sometimes even the writes (updates) to the given table. When no database migration is run, then there is no lock on the database.

Strong migration gems
By using the strong migrations gem, we can make sure we will write migrations effectively. The strong migrations gem catches unsafe migration code at the time of development, which lowers the risk of any kind of issue when the code is deployed to production.

Writing migrations effectively means doing so in a way that not only avoids problems, but also doesn’t cause reads or writes blocks. This is important when working with a lot of data, because database migrations tend to take more time.

Scheduled background processes
We recommend moving certain actions do the background and scheduling them using the push_bulk feature from sidekiq.

Using the database for calculations
Better performance can be achieved by completing amount calculations on the database.

Reducing the amount of data
This may sound simplistic, but often applications don’t use all the data available to them, or at least not for every operation. In such cases, we can apply a scope for database queries. It means that, when when the app performs payment calculation, for example, the scope is narrowed only to a given customer.

Code reviews
Reviewing code and making sure that the correct indexes are in place should be obligatory. Also, from time to time, it’s good to check your database to make sure that the indexes which were used previously are still being used. Sometimes, a certain functionality is modified or removed, but the indexes might be overlooked and left in the database. Invalid database indexes can cause a lot of unnecessary operations, which can lower the database’s performance.

Conclusions

As you can see, it’s often worth it to optimize your app’s code while you don’t have to handle that much data. Your database will grow, and if you don’t prepare for this, the app will stop working smoothly, resulting in poor customer experience. Much like with your physical health, the health of your app can be all about prevention.

Ready to consider upgrading your app to improve it’s performance? We can help.
iRonin’s experts have supported many clients through software updates and we continue to maintain projects with large user bases.


Let’s get in touch
Author's Bio
Przemysław Szyszka

Senior Software Developer

Full-Stack software developer focused mostly on Ruby, but in love with Elixir. Enjoys learning new things (patterns, programming languages). Besides programming spends free time on swimming, playing board games or ballroom dancing. Doesn't drink coffee ;)

Similar articles
Comments

Bulletproof your development with remote team augmentation

Read how
This page is best viewed in portrait mode
Our websites and web services use cookies. We use cookies and collected data to enhance your experience, provide additional communication channels, improve marketing materials and enhance our offer. IRONIN SP. Z O.O. SP. K. is committed to protecting all the data that we collect or process in any way, especially data of personal nature. By accepting these terms you agree to our usage of cookies and processing your data, according to our Privacy Policy, and you declare that your browser settings reflect your preferences. Read more You have the right to revoke this agreement at any time, based on the terms of our Privacy Policy. You can change cookies settings in your browser. If you do not agree with us using cookies and processing your data, please change your cookies settings in your web browser and reject these terms. You can find more information about cookies, your data privacy This site uses cookies. By continuing to browse the site, you are agreeing to our use of cookies. data processing, and your rights in our Privacy Policy.