Member of Technical Staff (Evals Platform)
Fireworks AI
- United States
- Lead/Staff
- 1+ years
- Posted
Skills
Required
- software engineering / data science
- LLM evaluations
- system design
- sandbox infrastructure
- how to design
- GenAI Lifecycle
- prompting a base model
- curating a dataset
- fine-tuning, and productionizing agents
Preferred
- backend
- design new benchmarks
- internal evaluation
- designing systems
- eliminate bottlenecks
- tracking state-of-the-art benchmarks
Join our team as a Member of Technical Staff, Evals & Post-Training Product. In this role, you will be responsible for building the infrastructure and workflows that connect evaluation and post-training.
Responsibilities
- You will work across backend systems, sandbox infrastructure, and user-facing surfaces to improve programmatic access and scale.
- You will also track state-of-the-art benchmarks, design new benchmarks, and build user-facing workflows for post-training.
- Take ownership of the internal evaluation setup and evolve it for the future, designing systems to eliminate bottlenecks and improve programmatic access.
- Act as a "metrics obsessive," tracking state-of-the-art benchmarks, rigorously reproducing published results, and designing new benchmarks for complex use cases.
- Build and improve user-facing workflows for post-training, including fine-tuning experiences across various model-improvement capabilities.
Requirements
- The ideal candidate will have 1-7 years of software engineering or data science experience, a deep understanding of LLM evaluations, and strong system design skills.
- Experience: 1–7 years of software engineering or data science experience (We are hiring at multiple levels for this role)
- Analytical & Data Science Mindset: You possess a deep understanding of LLM evaluations, how to design them, and how to use the results to guide model improvement. You are meticulous about data and metrics
- Understanding of the GenAI Lifecycle: You understand the end-to-end workflow—from prompting a base model to curating a dataset, fine-tuning, and productionizing agents
- Sandbox Infrastructure: Hands-on experience building or working with sandbox environments and sandbox infrastructure for secure code execution and testing
- Strong System Design Skills: You know how to architect scalable, programmatic systems and transition legacy setups into robust infrastructure
- Frameworks & Orchestration: Experience working with the Harbor framework or similar container registry and orchestration tools
- Experience: 3+ years of software engineering or applied data science experience
- Public Writing & Analysis: A strong interest in discovering where different models excel and fall short, with a desire to write up and publish these insights publicly (e.g., technical blogs, whitepapers)
- Inference & Hardware Knowledge: Interest in the hardware side of AI—understanding GPU constraints, inference optimization techniques, and how they relate to model performance
- Startup DNA: Experience in fast-paced environments where you own features end-to-end
Pay
USD 210,000 - 320,000
