Lead Machine Learning Engineer (Evaluations)

Join ASAPP, a company dedicated to delivering the best AI-powered customer experience. As a Lead Machine Learning Engineer, you will be responsible for owning and growing the evaluation platform that measures quality, safety, and performance across ASAPP's agentic AI systems. You will work closely with research, product, and platform teams to productize experiments into robust AI solutions and represent the evaluation platform to stakeholders outside the immediate team.

Responsibilities

• Concevoir et posséder des cadres d'évaluation qui garantissent la qualité, la sécurité et la performance des systèmes d'IA agentique complexes.
• Développer la feuille de route technique et l'architecture de la plateforme d'évaluation, de l'évaluation hors ligne à la surveillance en ligne/production.
• Construire l'infrastructure de données dont l'évaluation dépend : pipelines d'annotation et d'étiquetage, versionnage des ensembles de données, contrôles de qualité des données.
• Experience designing data pipelines for ML evaluation- labeling/annotation workflows, dataset versioning and quality control, and reproducible benchmarking
• Demonstrated experience leading the technical direction of a project or small team: setting architecture, driving design reviews, and being accountable for a system's long-term health (not just shipping features)
• Desire to learn, teach, and collaborate closely with cross-functional peers

Requirements

• A Bachelor’s Degree in CS or other related fields
• Demonstrated technical mentorship of junior and mid-level engineers, driving adoption of best practices and architectural alignment for scalability and extensibility
• Deep, hands-on experience building and operating evaluation systems for modern ML/LLM/agentic systems- not just consuming existing eval tools
• Strong architectural skills, with proven experience designing complex, data-intensive software systems and production experience with Python, AWS, Kubernetes, and/or Docker
• Experience with voice/audio quality evaluations

Experience building and evaluating agentic systems at scale

• Production experience with LLM-centric services (e.g., inference, orchestration, evaluation, monitoring)
• Experience with conversational/customer-support AI domains (e.g., containment rate, conversation quality, goal completion)
• Knowledge of techniques for optimizing model architectures for faster inference
• Experience with AWS, CI/CD, Kafka, Athena

Preferred

• Familiarity with large-scale ML experimentation, benchmarking, or simulation frameworks

Pay

USD 170,000 - 190,000