As financial institutions gravitate towards systems that can deliver instantaneous analytics, real-time decision making gives companies a competitive advantage. But organisations need advanced technologies and robust infrastructure to process this data. Integrating real-time data processing with existing systems and legacy systems can be a difficult task as data inaccuracies can lead to significant business impact.
YourStory, in association with Snowflake, recently sat down with leaders from fintech companies to discuss how to harness the power of data and AI to build unparalleled data products and experiences for customers. The session titled ‘Real-time data processing: Is it a bane or a boon?’ explored how companies can undertake this complex endeavour, along with the benefits and considerations of real-time data processing and decision making for the fintech sector.
Panellists included Anirban Roy, Vice President of Engineering and Architecture at Financial Software and Systems; Swapnil Dixit, Vice President of Products at Avanti Finance; Venkatesh Vootla, Head of Analytics at Scapia; Sameer Goyal, Senior Director and Head of Engineering at Acuity Knowledge Partners; Amit Sharma, Co-founder and Chief Technology Officer at twid; Harshvardhan Kathotia, Vice President of Product at SignDesk; Gaurav Mehrotra, Chief Technology Officer of Northern Arc Capital; and Vikram Singh, Head of Product at Freo.
Addressing gaps in data collection
One of the key challenges that was highlighted during the discussion is in creating a governance structure and managing the infrastructure around a product. When there are multiple systems, the structure of data across those systems is not unified. Roy, of Financial Software and Systems, said, “While building the system, we have tried to use best practices of domain modelling and created a target operating model across the system, as well as an interfacing layer where the data structure is different.” The lag time becomes very important while processing data in real time.
While there is a lot of data that fintech companies can capture, the focus should be on optimally utilising it and discarding it if it’s not of any use months later. The discussion also explored the importance of capturing data in a way that makes it easily accessible when needed, even after several years. Not every data point needs to be processed real time; therefore data processing must be done very judiciously.
“Unless I’m not clear about what we are processing, why are we processing it, and where do I see the needle move, and if it’s not metric driven, I wouldn’t want to pursue that path because we all know that real-time configuration becomes very expensive. So we don’t want to inherit huge monoliths of data, not knowing how much sense to make out of it,” said Sharma, of twid.
Echoing similar sentiments, Kathotia, of SignDesk, remarked, “Having tons of data will only increase the cost, and that’s where the profit and loss gets hit. A lot of storage cost is also incurred because of data. So a lot of irrelevant data can be either deleted or be moved to a whole storage part. This is where intelligence comes into the picture, where we can use AI to process the data and turn it into action items that we can use. Only data will not really help any of our use cases.”
For Goyal, of Acuity Knowledge Partners, data is definitely a friend and not a foe “Volumes of data help you detect patterns, which then help you to train your models to be better. There has to be a cost benefit consideration as well. In my experiences, data has mostly been around transactional monitoring, where flip seconds can actually matter a lot. Another concern with data is data residency. So since we work with North American or European customers, there are strict regulations over which data can pass on across the border, and ones that have to necessarily stay within the borders,” he added.
Data integration and governance
With companies having separate entities like insurance, investment and loans, it’s also important to understand how data is shared across the board. When it comes to data and AI foundation, companies fight a continuous battle on whether it is really a cost or an investment they are making.
While on one hand we talk technology which is all about uniformity, the other side is all about life, according to Dixit, of Avanti Finance. He said, “The fundamentals of our platform is hardcore traceability, right down to location, and not at the document or topic level, but at the attribute level. Who made? What changed? Which platform?”
While organisations focus a lot on latency requirements, they also need to look for concurrency. They need to be mindful of all of these elements when it comes to upgrading a model. One of the best practices, according to Snowflake, is to see the declarative measures of transforming data. While technology is one part of the solution, it’s a best practice that needs to be integrated with people, because as much as it is a technological concept, it’s also a social construct.
The speakers also discussed another key challenge: data sharing. Data sharing or collaboration is essential, and just enabling that would solve some fundamental existing problems.
Fintech companies are not working in isolation; suppliers, partners, vendors, and other agencies are involved. It is all about ecosystem play. Speakers discussed ways to go from a data ownership mindset to a data collaboration mindset, and find a very secure, reliable government for sharing data with its entities without having to expose IPs and PII data to these entities. It’s important to be very judicious with resources. The price to performance is something that companies must always consider during real-time data processing.
The intersection of data and technology
Whether it’s managing risks, detecting fraudulent activities, or personalising customer interactions, real-time data processing is proving to be not just a luxury, but a necessity. Video KYC is one of the norms that banks need to adhere to for dispersing any kind of loan or considering change of address. Despite the implementation of 4G and 5G, Tier II and Tier III cities still struggle with connectivity issues where video stability connection is poor. For some fintech companies, whose volume starts picking up during the second half of the month, processing and queuing up a whole lot of transactions together is challenging.
“While I’m happy that the National Payments Corporation of India got banks together to do real-time transactions, I’m hoping that a similar model will evolve where companies, corporates, and banks will all be sharing data almost in real time, even if it’s at a high cost, because in the long term, if that happens, everybody wins,” said Singh, of Freo.
With the government and fintech players setting up a proper infrastructure in place, hyperscalers like Microsoft and AWS have been spending a lot on offering services and solutions that make it easy for these players to achieve scale and ensure economy of operations.
Mehrotra, of Northern Arc Capital, said, “There are few things we can try through architecture and technology, which will shape up in business, but mostly what I have experienced is businesses are what will drive the architecture. So we pursue that path and try to be as relevant as possible, so that business thrives.”
Vootla of Scapia, a travel fintech organisation, feels the current generation is very impatient to get credit-related products, especially when it comes to travel as a need.
“Think of any product from a travel point of view such as flights, stays, visas, tours, cabs that we would want to provide the customer. The spends they do on credit cards will come to them as rewards. So we are asking them to redeem those rewards via travel as a product,” he explained.
One of the key takeaways from the discussion was the ease of use of technology – it should be simple for everyone to use, scalable and instantly available when needed. And, it must be governed, secure, compliant, and cheap.
Looking into the future, the speakers concurred that real-time analytics will enable deeper insights into consumers’ financial behaviours, integrating data across various platforms such as banks, fintechs, and non-financial sources and consolidating it onto a single platform. While fintechs have started investing heavily in implementing robust measures to protect sensitive information, a lot remains to be done to ensure data security on an on-going basis.
