Senior Backend Engineer (Inference Platform)

Join our team as a Senior Backend Engineer, where you will own and scale our inference gateway, the critical layer that routes every model call from our products and customers. You will build an intelligence layer to autonomously select the right model for each task, make routing decisions that account for various factors, and develop cost attribution and metering across providers. This is a remote-first position with competitive benefits and the opportunity to collaborate with a team at the forefront of AI research.

A strong backend generalist (Python, Kubernetes). We expect you to pick up whatever you have not touched yet

Responsibilities

• Posséder et faire évoluer la passerelle d'inférence qui gère chaque appel de modèle, y compris la sélection du fournisseur et du modèle, le repli et les limites.
• Construire une couche d'intelligence pour choisir de manière autonome le bon modèle pour la bonne tâche, en tenant compte du comportement du cache.
• Construire l'attribution des coûts et le mesurage à travers les fournisseurs, y compris la comptabilité des jetons mis en cache, suffisamment précis pour facturer et faire respecter.
• Comfort with streaming responses and the parts that get awkward: partial state, mid-stream failures, and work that can only happen once the stream closes
• Usage metering, billing, or cost optimization at scale

Requirements

• Experience building or operating a proxy, gateway, or data plane on the critical path of production traffic, with sound instincts for failing open
• Hands-on production experience with model serving on stacks such as vLLM, SGLang (or similar platforms), with a working understanding of continuous batching, KV and prefix caching, and the tradeoff between time to first token and throughput
• Fluency across provider APIs: tokenization, prompt-caching semantics, and the differences in how usage and cached tokens are reported and priced
• You have tuned a serving stack against live traffic and know what moves p99 in practice, not only in theory
• We weigh trajectory and recent growth more heavily than years of experience
• Experience at an inference provider or GPU cloud, or contributions to an open-source serving stack

Pay

• Fine-tuning and evaluating models in production
• USD 200,000 - 250,000