jobapplier

Member of Technical Staff (Evals Platform)

Fireworks AI

Skills

Required

  • software engineering / data science
  • LLM evaluations
  • system design
  • sandbox infrastructure
  • how to design
  • GenAI Lifecycle
  • prompting a base model
  • curating a dataset
  • fine-tuning, and productionizing agents

Preferred

  • backend
  • design new benchmarks
  • internal evaluation
  • designing systems
  • eliminate bottlenecks
  • tracking state-of-the-art benchmarks

Join our team as a Member of Technical Staff, Evals & Post-Training Product. In this role, you will be responsible for building the infrastructure and workflows that connect evaluation and post-training.

Responsibilities

  • You will work across backend systems, sandbox infrastructure, and user-facing surfaces to improve programmatic access and scale.
  • You will also track state-of-the-art benchmarks, design new benchmarks, and build user-facing workflows for post-training.
  • Take ownership of the internal evaluation setup and evolve it for the future, designing systems to eliminate bottlenecks and improve programmatic access.
  • Act as a "metrics obsessive," tracking state-of-the-art benchmarks, rigorously reproducing published results, and designing new benchmarks for complex use cases.
  • Build and improve user-facing workflows for post-training, including fine-tuning experiences across various model-improvement capabilities.

Requirements

  • The ideal candidate will have 1-7 years of software engineering or data science experience, a deep understanding of LLM evaluations, and strong system design skills.
  • Experience: 1–7 years of software engineering or data science experience (We are hiring at multiple levels for this role)
  • Analytical & Data Science Mindset: You possess a deep understanding of LLM evaluations, how to design them, and how to use the results to guide model improvement. You are meticulous about data and metrics
  • Understanding of the GenAI Lifecycle: You understand the end-to-end workflow—from prompting a base model to curating a dataset, fine-tuning, and productionizing agents
  • Sandbox Infrastructure: Hands-on experience building or working with sandbox environments and sandbox infrastructure for secure code execution and testing
  • Strong System Design Skills: You know how to architect scalable, programmatic systems and transition legacy setups into robust infrastructure
  • Frameworks & Orchestration: Experience working with the Harbor framework or similar container registry and orchestration tools
  • Experience: 3+ years of software engineering or applied data science experience
  • Public Writing & Analysis: A strong interest in discovering where different models excel and fall short, with a desire to write up and publish these insights publicly (e.g., technical blogs, whitepapers)
  • Inference & Hardware Knowledge: Interest in the hardware side of AI—understanding GPU constraints, inference optimization techniques, and how they relate to model performance
  • Startup DNA: Experience in fast-paced environments where you own features end-to-end

Pay

USD 210,000 - 320,000