Artificial intelligence (AI) has dramatically reshaped the landscape of speech recognition and human-machine interaction. Over the past two decades, speech recognition systems have evolved from a mere 50% accuracy rate to over 90%, enabling seamless integration of AI in real-world applications. Jithendra Vepa has worked for many years in the field of speech recognition and as the Co-founder and CTO at Observe.AI, he is transforming contact centers with AI-powered conversation intelligence.
Jithendra’s fascination with AI and speech technology began during his time at the Indian Institute of Science (IISc), Bangalore. With a background in electrical engineering, his initial research focused on signal processing and machine learning (ML). His master’s thesis on detecting defects in high-voltage transformers, using machine learning, not only won accolades but also cemented his passion for AI-driven problem-solving.
His interest in speech processing grew as he explored real-life applications of AI, leading him to pursue a PhD at the University of Edinburgh in 2000. At Edinburgh, he worked on some of the earliest ML models for text-to-speech conversion, laying the groundwork for modern speech technologies. Jithendra’s PhD research pioneered the application of hidden Markov models and early neural networks to speech synthesis, a significant departure from traditional rule-based approaches.
Advancing voice AI
After completing his PhD, Jithendra expanded his expertise through postdoctoral research at IDIAP Research Institute in Switzerland, where he collaborated with global institutions like MIT, Carnegie Mellon University (CMU), and Cambridge University. His research focused on automatic transcription of meetings, speech summarization, and Natural Language Understanding (NLU), all of which are foundational to today’s AI-driven contact centers.
Jithendra then made the shift from academia to the industry, spending over eight years at Samsung Electronics, where he worked on voice assistants, smart home devices, and AI-driven consumer electronics. He contributed to specific projects that called for integrating voice interaction into devices including TVs and smartphones, enabling seamless hands-free control for users.
Joining Observe.AI to transform contact centers with AI
In 2018, Jithendra joined Observe.AI, drawn by the vision of its co-founder & CEO, Swapnil Jain. He was intrigued by the complex voice challenges in contact centers and saw an opportunity to apply his AI expertise in a space that seemed complex to automate. Unlike AI in consumer devices, contact centers deal with high-stakes conversations — customer support, financial queries, compliance monitoring — where accuracy and efficiency are paramount.
At Observe.AI, Jithendra has focused on developing a proprietary Contact Center Large Language Model (LLM) with 40 billion parameters, trained specifically on customer service interactions. This AI system can analyze, summarize, and optimize conversations in real-time, helping businesses improve efficiency, reduce response times, and enhance customer satisfaction.
AWS’ role in Observe.AI’s success
AWS has been an important partner in Observe.AI’s AI journey, providing the infrastructure needed to train and deploy large-scale AI models. Observe.AI leverages Amazon SageMaker for model training and Bedrock for fine-tuning AI models tailored to contact center applications.
With AWS’s capacity blocks, Observe.AI can train AI models in hours instead of weeks, significantly reducing costs and accelerating innovation. On the role of AWS, Jithendra said, “AWS has always been one of the most customer obsessed and customer first companies. We are big now, but even when we were a small company, support was always available. AWS has really helped us in our growth journey.”
AWS helped Observe.AI deploy its solutions in a flexible way. Since contact centers experience fluctuating call volumes, AWS Elastic Inference and EC2’s (Amazon Elastic Compute Cloud) auto-scaling makes sure that Observe.AI’s AI models operate smoothly, handling peak loads without latency issues.
AWS provides cloud support, AI tools and technical knowledge, letting Observe.AI focus on innovation rather than managing cloud operations.
AI’s impact on consumer-brand interactions
AI is also redefining the way brands and consumers interact with each other. According to Jithendra, traditional customer service relied solely on human agents handling calls, leading to long wait times and inconsistent experiences. With AI-powered conversation intelligence, companies can analyze 100% of customer interactions (compared to just 2-5% with manual audits); automate routine queries with AI-driven virtual agents; assist human agents with real-time recommendations and enhance compliance monitoring by detecting anomalies in conversations.
“For customers, this means faster resolutions, more personalized responses, and improved service experiences. AI is also making businesses proactive, predicting customer needs before issues arise and ensuring more seamless interactions,” said Jithendra.
Multimodal systems and speech automation
Observe.AI addresses the language challenge with a two-pronged approach. As a startup, the primary focus is on developing English-language solutions, such as speech-to-text and large language models (LLMs). But, Jithendra said, “we have customers from different regions and we need non-English capabilities as well.” The startup collaborates with AWS to leverage tools like transcription services for European and other languages. “Additionally, we utilize multilingual LLMs to extract intents, entities, and generate summaries from non-English transcriptions,” Jithendra said.
The collaboration extends beyond language support to meeting compliance requirements across various regions. Because of its global reach, AWS aids in rapidly deploying localized environments that adhere to local regulations. “This partnership strengthens our multilingual capabilities and facilitates our expansion into diverse geographies while ensuring compliance,” he added.
Looking ahead, Jithendra envisions AI moving toward multimodal systems, where text, speech, and images seamlessly integrate to enhance machine understanding. Jithendra said, “there will be significant progress in speech-to-speech or voice-to-voice models, allowing machines to converse more naturally with humans without relying on text-based processing.”
He also predicts the rise of industry-specific AI models, where AI will not only assist agents but fully automate simple, and even moderately complex, customer interactions. However, he emphasized that human agents will remain essential, particularly for nuanced interactions requiring empathy and critical thinking.
Balancing work, innovation, and passion for speech systems
Jithendra is an avid badminton player and a technology enthusiast. He dedicates few hours per week in staying updated on AI advancements by attending conferences, reading research papers, and experimenting with new AI models. He frequently collaborates with AWS AI conclaves and industry forums to exchange insights on the future of AI. Recently, Jithendra was speaking at one of the AWS conclaves in Bangalore, sharing Observe.AI’s journey. His passion for continuous learning ensures that he remains at the forefront of AI innovation.
Jithendra’s journey from an AI researcher to an industry leader at Observe.AI reflects the transformative potential of AI in real-world applications. His work at the startup is shaping the future of AI-driven customer engagement, ensuring that businesses can provide faster, smarter, and more personalized interactions.
In fact, Jithendra’s career spanning academia to AI innovation showcases how cutting-edge research can drive real-world impact. Interestingly, in this journey, Jithendra noted, “AWS has been a key partner, whether it is for cloud solutions, managing scale, trying out different models or comprehending language nuances.”
With AWS as a technology enabler, Observe.AI is poised to drive the next wave of AI-powered contact centers, bridging the gap between human intelligence and machine efficiency. As AI continues to evolve, Jithendra’s contributions will undoubtedly play a pivotal role in defining the future of voice AI and conversation intelligence.
