jobapplier

Lead Product Data Scientist, Agents

Klaviyo

Skills

Required

  • tool-call outcomes
  • LLM-based classification

Join Klaviyo, a leading marketing automation platform, as a Lead Product Data Scientist. In this role, you will shape the measurement and data science strategy for our AI-powered products, Composer and Customer Agent. You will work closely with cross-functional teams to develop comprehensive measurement frameworks, evaluate agent performance, and connect AI-driven experiences to meaningful business outcomes. This is a highly visible, hands-on technical leadership role that combines deep technical expertise with strong product and business judgment.

Responsibilities

  • Own agent measurement end to end across Composer and Customer Agent, developing comprehensive measurement frameworks that connect adoption, engagement, agent performance, customer value, and revenue impact.
  • Define what successful agent performance looks like by partnering with Product and Engineering to establish meaningful measures of agent effectiveness, task completion, customer outcomes, failure rates, and the trade-offs between quality, cost, and business value.
  • Build scalable agent evaluation frameworks by combining conversation data, agent traces, tool-call outcomes, human evaluation, and LLM-based classification to assess quality, identify failure patterns, and understand how changes to models and agent behavior influence product performance.