jobapplier

Research Engineer / Performance Engineer (RL Distributed Systems)

Anthropic

Skills

Required

  • storage
  • networking
  • scheduling
  • reinforcement learning / large language model training workloads
  • Python
  • orchestration
  • software engineering
  • Rust / C++ / Go
  • design documents
  • incident writeups
  • ML training / inference infrastructure
  • container orchestration
  • Kubernetes

Preferred

  • distributed systems
  • RL
  • async
  • Trio / asyncio
  • data movement
  • coordination
  • debugging complex failures

Join Anthropic, a leading AI safety and research company, as a Research Engineer on the Distributed Systems team within RL Engineering. In this role, you will design, build, and operate the distributed systems that run reinforcement learning (RL) at scale. You will work on various aspects of the system, including scheduling, data movement, storage, networking, and fault tolerance. Strong candidates should have experience building and running large distributed systems and a deep understanding of distributed systems fundamentals.

Responsibilities

  • Design, build, and operate the distributed systems that run reinforcement learning (RL) at scale, focusing on training, sampling, and environment execution.
  • Identify and remove limitations in the system, addressing areas such as scheduling, data movement, storage, networking, or coordination.
  • Implement fault tolerance, resource management, and autoscaling to ensure the system can adapt to changing demands and continue making progress.
  • Experience debugging complex failures across many hosts and services, including failures you can't reproduce locally
  • Deep understanding of distributed systems fundamentals, including consistency, coordination, consensus, failure modes, and recovery
  • Ability to reason quantitatively about throughput, latency, and resource costs across compute, memory, storage, and network

Requirements

  • Experience designing, building, and operating large-scale distributed systems in production
  • Strong software engineering skills in Python and at least one systems language such as Rust, C++, or Go
  • Strong written communication, including design documents and incident writeups
  • Minimum years of experience: Years of experience required will correlate with the internal job level requirements for the position
  • Required field of study: A field relevant to the role as demonstrated through coursework, training, or professional experience
  • Minimum education: Bachelor’s degree or an equivalent combination of education, training, and/or experience
  • We encourage you to apply even if you do not believe you meet every single qualification
  • Experience running ML training or inference infrastructure at scale
  • Experience across several layers of the stack, such as scheduling, storage, networking, and orchestration
  • Experience building schedulers, autoscalers, or resource management systems
  • Experience with container orchestration such as Kubernetes, and with sandboxed or virtualized code execution at scale
  • Experience with high-performance networking, RDMA, or collective communication libraries
  • Familiarity with reinforcement learning or large language model training workloads

Preferred

  • Experience building observability or automated remediation for large fleets
  • Experience with async Python frameworks such as Trio or asyncio

Pay

USD 500,000 - 850,000

Every job at Anthropic