AI Engineer (Data and Evaluations)

Join Glean, a company focused on AI products. As an AI Engineer, you will be responsible for data and evaluation science, building evalsets, benchmarks, and the platform that runs evaluations at scale. You will turn production usage insights into evaluations that guide improvements across the entire stack and partner with R&D, product, and GTM to enhance products and customer outcomes.

Responsibilities

• Concevoir et construire des ensembles d'évaluation et des benchmarks à partir d'exemples d'experts, de données de production, de données synthétiques et d'échecs passés.
• Développer et mettre à l'échelle la plateforme d'évaluation et l'observabilité des agents : pipelines d'expérimentation, capture et replay des traces, et tableaux de bord.
• Exécuter des évaluations à chaque étape, des suites CI et pré-release au trafic de production échantillonné, et rapporter les résultats avec des intervalles de confiance.
• Experience building eval or experiment infrastructure that runs reliably at scale, including CI/CD integrations and observability tooling like OpenTelemetry
• Strong analytical skills: you dig into traces and failure cases, write SQL, and use data and metrics to explain why quality moved

Requirements

• 5+ years of software engineering experience with strong Go and/or Python and backend fundamentals: APIs, data modeling, testing, and distributed pipelines. TypeScript is a plus
• Experience building datasets and benchmarks for LLM use-cases, RAG, or agents, including evalset curation and synthetic data generation
• Familiarity with eval methodologies and their tradeoffs: deterministic metrics, LLM-as-a-judge, pointwise vs pairwise grading, human labeling, and judge calibration
• Nice to have: experience with simulation environments and/or model post-training techniques

Pay

USD 200,000 - 280,000