In today’s AI-driven job market, data science plays a crucial role in connecting talent with opportunity. In this interview, Vijaya Chaitanya Palanki, Sr Manager of Data Science at Glassdoor, shares insights on emerging trends in machine learning, the challenges of building AI-driven job recommendations, and the balance between innovation and scalability. Vijaya also discusses fostering a data-driven culture, essential skills for data scientists, and ensuring fairness in AI recommendations. Read on to explore how AI is shaping the future of hiring and career growth.
Explore more interviews like this here: Jarrod Teo, Data Science Leader — Building AI Products, Data Acquisition Strategies, AI and LLMs in Commerce, Business Integration of Data Science, Skills for Modern Data Scientists, Evolving ML Strategies Amid Economic Uncertainties
As a published researcher in AI, what are some emerging trends in machine learning that excite you the most?
As a Sr. Manager of Data Science at Glassdoor, I’m particularly excited about several emerging trends in machine learning:
First, the evolution of agentic AI systems that can autonomously perform complex tasks with minimal human oversight. These systems are moving beyond basic automation to handle nuanced decision-making, which is transformative for how we connect job seekers with opportunities and employers with talent.
Second, I’m seeing remarkable progress in multimodal models that integrate different types of data – text, images, numerical data – to provide more comprehensive insights. This is particularly valuable for analyzing job descriptions, user interactions, and employer reviews to create more meaningful matches between candidates and companies.
Third, the democratization of machine learning through no-code and low-code platforms is opening up AI capabilities to domain experts without requiring advanced programming skills. This has been valuable at Glassdoor for enabling more of our teams to leverage data in their decision-making.
Finally, I’m fascinated by the potential of AI systems that can reason about causality rather than just finding correlations. In my work building prediction models for business lead scoring and consumer journey analysis at Glassdoor, the ability to understand causal relationships significantly enhances the strategic value of these tools.
These developments are creating opportunities to solve complex business problems that were previously intractable, particularly in the job marketplace space where I’m currently focused at Glassdoor.
What leadership principles do you follow when scaling and managing high-performing data science teams?
When scaling and managing high-performing data science teams, I follow several core leadership principles that have consistently proven effective. I create a balance between autonomy and alignment by establishing clear business objectives while giving team members freedom to determine implementation approaches. I prioritize continuous learning through structured knowledge sharing and ensure diverse perspectives are represented on every team. Data-driven decision making applies to team management as much as to our work product, allowing me to make objective resource allocation decisions based on team velocity and project outcomes.
Technical excellence and business impact must coexist, which is why I encourage teams to pursue innovative approaches while maintaining focus on measurable outcomes. Every project must demonstrate value through clearly defined metrics, not just technical sophistication. I believe in transparent communication about priorities and constraints, as this helps teams make better decisions and feel more invested in outcomes. These principles have helped me build agile, innovative teams that consistently deliver significant business impact through data science initiatives.
How do you balance innovation with scalability when developing machine learning models for large-scale applications?
Balancing innovation with scalability in machine learning for large-scale applications is something I navigate daily through a multi-faceted approach. I compartmentalize innovation efforts through a proven framework, starting with rapid prototyping on small data samples to validate concepts before scaling. Infrastructure planning is critical – we design with scale in mind from the beginning, selecting tools and frameworks with proven reliability. I’ve found that modular architecture is essential, breaking complex models into reusable components that can be individually optimized and scaled. Performance benchmarking at each development stage helps identify bottlenecks early, ensuring models function effectively in production environments.
Maintaining this balance requires organizational alignment where stakeholders understand the tradeoffs between cutting-edge techniques and production reliability. Sometimes this means implementing proven approaches first while developing more innovative solutions in parallel. I ensure modular design principles are followed, allowing teams to update specific parts of systems without rebuilding entire solutions. This balanced approach has allowed my teams to successfully deploy sophisticated machine learning solutions that combine innovative methodologies with robust scalability to support millions of users, delivering both technical excellence and business impact simultaneously.
How do you foster a culture of experimentation and data-driven decision-making in a cross-functional organization?
Fostering a culture of experimentation and data-driven decision-making begins with creating both infrastructure and mindset shifts across the organization. I establish clear frameworks for experimentation, including standardized metrics, documentation processes, and evaluation criteria that make running tests accessible to teams regardless of their technical expertise. Implementing tools like Amplitude’s A/B testing platform has been transformative in this process – it democratizes experimentation by embedding statistical rigor, proper test design, and analysis frameworks directly into the tool’s interface. This allows marketing teams, product managers, and other stakeholders to confidently run sophisticated tests without requiring advanced statistical knowledge, while maintaining scientific validity in their approach.
The second essential element is aligning incentives with data-driven approaches. I ensure performance evaluations acknowledge evidence-based decision-making, not just outcomes. By celebrating instances where data from Amplitude experiments contradicted our assumptions and changed our direction, we reinforce that the goal isn’t being right but making better decisions. The visual reporting and intuitive significance indicators in Amplitude make it easier for everyone to understand and communicate test results, breaking down traditional barriers between technical and non-technical teams. This comprehensive approach has consistently transformed organizational cultures to embrace experimentation as a core competency with tools like Amplitude serving as the operational backbone of our testing infrastructure.
What are the most critical skills data scientists need to develop to stay relevant in an AI-driven future?
To stay relevant in an increasingly AI-driven future, data scientists must develop a unique blend of technical depth and business acumen that goes beyond traditional programming skills. The ability to effectively translate business problems into data science solutions has become paramount – understanding stakeholder needs, framing problems appropriately, and communicating insights in business language rather than technical jargon. With foundation models becoming widely accessible, the value increasingly lies in identifying which problems need solving rather than simply knowing how to implement algorithms.
Causal inference and experimental design skills are becoming essential as organizations move beyond predictive analytics to understand intervention effects. Strong product intuition allows data scientists to build solutions that provide genuine user value rather than just technical elegance. Additionally, ethical AI considerations – including bias mitigation, transparency, and responsible deployment – are no longer optional but core competencies. As model development becomes increasingly automated, the data scientists who will thrive are those who can navigate this complex landscape of business needs, technical possibilities, and ethical considerations while developing systems that create measurable impact.
What are the biggest challenges in building AI-driven job recommendations, and how do you ensure they remain relevant and unbiased?
Building effective AI-driven job recommendations presents several significant challenges that require thoughtful solutions. The first major challenge is balancing personalization with exploration – creating systems that provide relevant matches based on a candidate’s background while still exposing them to new opportunities they might not have considered. This requires sophisticated approaches to cold-start problems for new users with limited profiles and preventing recommendation loops that reinforce existing career paths. Another critical challenge is handling the inherent complexity of job data, including unstructured job descriptions, varying terminology across industries, and the need to understand both hard skills and cultural fit factors.
Ensuring relevance and mitigating bias demands a multi-layered approach. I implement rigorous bias testing across different demographic groups, examining recommendation distributions to identify and address disparities. Regular A/B testing with clearly defined success metrics helps validate that recommendations truly benefit users, not just optimize for engagement. I also incorporate explicit diversity goals into model development and maintain human oversight for edge cases. Beyond technical solutions, I find that transparent recommendation explanations are essential – when users understand why certain jobs are recommended, they can provide better feedback, which improves system quality while building trust in the platform. This comprehensive approach creates job recommendation systems that are both powerful and fair.
