jobapplierSoftware Engineer (Inference)
- Sunnyvale, United States
- Mid Level
- Posted
Skills
Required
- Slurm / Ray
- SLI / SLOs / postmortem improvements
- Kubernetes
- CI/CD
- metrics / alerting
- Prometheus / Grafana / OpenTelemetry
- structured logging
- GPU inference
- KV
- caching
- streaming tokens
- Python / Go
- gRPC/HTTP APIs
Preferred
- move key SLIs
- automate runbook steps
Join our Inference team as a Junior Software Engineer. You will own a production service end-to-end, collaborating with peers across various teams to improve operational excellence. Your responsibilities will include designing, implementing, testing, deploying, and maintaining a model-serving service, as well as improving latency and throughput. You will also participate in on-call duties and contribute to team reviews.
Responsibilities
- Collaborate with peers across runtime, scheduling, and networking to move key SLIs.
- Own one model-serving service end-to-end: design, implementation, tests, deployment, and ops.
- Add and tune observability (dashboards, alerts, traces); automate runbook steps to reduce toil.
Requirements
- Working knowledge of Kubernetes (or Slurm/Ray) and CI/CD pipelines
- Experience with metrics/alerting (Prometheus/Grafana/OpenTelemetry) and structured logging
- Basic understanding of GPU inference concepts (KV caching, mixed precision, streaming tokens)
- Proficiency in Python or Go; familiarity with gRPC/HTTP APIs and containerized deployments
- 0–3 years of software engineering experience (or equivalent) building cloud/distributed services in production
- Experience with inference frameworks (Triton, vLLM, TensorRT-LLM, Ray Serve, TorchServe)
- Hands-on performance profiling/benchmarking (e.g., wrk, vegeta, perf, Nsight)
- Contributions to reliability practices: SLI/SLOs or postmortem improvements
Pay
USD 120,000 - 160,000