Blog

In today’s data-driven world, organisations are increasingly reliant on their ability to manage and analyse information effectively. Extracting value from vast amounts of data is crucial for making informed decisions, optimising processes, and gaining a competitive edge. This is where intelligent data platforms come into play. These powerful tools provide a centralised hub for data storage, processing, and analysis, empowering businesses to unlock the true potential of their information.

In a thought-provoking roundtable discussion titled ‘Architecting Tomorrow: Pioneering the next-gen intelligent data platform’, co-hosted by Databricks and YourStory on the sidelines of 2024 India Tech Leaders’ Conclave in Bengaluru, some of India’s leading minds in data management and analytics gathered to explore the future of this critical field. Moderated by Ipsita Basu, Director – Creative Content at YourStory, the session offered valuable insights into the challenges and opportunities of data management across various industries.

The esteemed panel guiding the discussion comprised some of India’s brightest minds in data management and analytics. Paulami Das, Head of Data Science and Engineering at PayU, brought her expertise in financial data. Navendu Agarwal, CIO of Ola Electric, offered insights into data management in the electric mobility sector. Mathangi Sri Ramachandran, Chief Data Officer at Yubi, shared her knowledge on data governance. Saurav Ghosh, Head of Data Science at BharatPe, provided his perspective on fintech data. Srivatsa Srinath, Principal Architect-ML at Amagi, shed light on data management in the media and entertainment industry. Finally, Rahul Shukla, CTO of MFine, offered his experience with data platforms in the healthcare sector.

The discussion highlighted the critical role of centralised data platforms with high-quality data for organisations of all sizes. It highlighted the importance of accessibility, quality, and affordability, especially in the healthcare sector. Experts from healthcare technology companies emphasised the role of platforms that connect patients with doctors and streamline payments, ensuring a seamless and efficient experience.

The conversation delved into the key challenges hindering data platforms, including data inconsistency, expensive infrastructure, and navigating the complexities of data compliance. Participants also explored the evolving landscape of data science tools, with a shift towards open-source solutions offering greater flexibility and cost-effectiveness.

Decentralised management with centralised oversight

An intriguing proposal was presented – a decentralised data management system with centralised governance and accountability. This approach aims to address the challenges associated with managing diverse data sources from various departments while ensuring proper oversight and data integrity.

Panellists also highlighted the technical aspects of building and utilising intelligent data platforms, particularly in the Indian context. Speakers discussed the challenges of managing large-scale data in real time, using AI to identify patterns and process complex data formats, and ensuring compliance with evolving regulations.

Data platform challenges: The way forward

Experts highlighted the challenges of managing vast volumes of data, including real-time data streams from the Internet of Things (IoT). They stressed the importance of AI in identifying patterns within petabyte- and exabyte-sized datasets and the need for customised solutions for different data types like video, audio, and corporate data. Data cataloguing and feature pipelines were identified as crucial elements for ensuring compliance with regulations like GDPR.

The discussion emphasised the importance of data literacy at the C-suite level. Speakers pointed out the limitations of reactive strategies based on unreliable data and called for increased education around data governance best practices. Balancing the cost of data platforms with the need for timely and accurate insights emerged as a key concern. Panellists highlighted the need for transparent cost structures and dedicated resources to manage data platforms effectively.

Data platform opportunities across industries

Specific examples from the financial services and healthcare sectors showcased the value proposition of data platforms. Speakers highlighted the importance of explainability, data quality, and innovative solutions. They also explored the use of graph technologies and efficient feature engineering methods to enhance productivity and detect fraud.

Packaging and migrating data platforms present a complex challenge, but speakers also discussed exciting new value offerings, such as using health data analytics to create personalised prevention plans and explainable AI systems for querying specific healthcare aspects.

Data management and AI adoption

The roundtable discussion highlighted how data and AI have demonstrably improved efficiency and reduced costs in various Indian contexts. Decreasing costs of data and AI tools were noted, encouraging organisations to evaluate them for potential benefits. The importance of centralised data platforms for effective data management and utilisation was stressed. Companies like Databricks are fostering a more open and accessible data ecosystem by open-sourcing solutions, empowering organisations to manage their data more effectively.

The panel also addressed data migration challenges and the need for a unified data format. Additionally, the dominance of expensive big-tech licensing models has diminished, with studies revealing minimal cost differences between open-source and cloud-based solutions offered by data platform providers. This cost reduction is making data platforms more accessible to a broader range of organisations.