Principal Software Engineer (Orchestration)

Roblox as a Principal Software Engineer in the Data Platform Orchestration team. You will be the technical leader for the system that schedules and runs the company's data workloads, impacting over 200 million daily active users. Your responsibilities will include defining the orchestration strategy, scaling and hardening Airflow, owning orchestration on Kubernetes, building platform tooling for internal engineers, driving observability and reliability, leading across teams, and raising the engineering bar.

Responsibilities

• Define the Orchestration Strategy: Own the multi-year architecture for Airflow and the surrounding platform, including scheduler and executor design, multi-tenancy, upgrades and migrations, and where the platform should evolve next.
• Scale and Harden Airflow: Find and remove scaling limits across the scheduler, metadata database, executors, and DAG parsing. Make sure thousands of daily DAGs run with predictable latency and strong SLAs.
• Own Orchestration on Kubernetes: Lead the design and operation of our EKS clusters for orchestration and Spark workloads, including autoscaling, resource isolation, cost efficiency, and deployment automation.

Requirements

• You should have 10+ years of experience in designing, building, and operating production-grade, large-scale distributed systems, with a strong proficiency in Python and experience managing production Kubernetes clusters on AWS.
• Data Workload Familiarity: Working knowledge of how Spark jobs run on Kubernetes (Spark on K8s, spark-operator, or EMR on EKS) and how orchestration interacts with them

B.S. equivalent in CS or sufficient experience

• 10+ years of experience designing, building, and operating production-grade, large-scale distributed systems
• Workflow Orchestration Depth: Deep, hands-on experience designing, running, and scaling a production data-related workflow orchestrator or job scheduler (thousands of workflows or DAGs a day). Apache Airflow is ideal, but experience with an equivalent system (Dagster, Prefect, Temporal, or a large in-house scheduler) is also welcome. You understand scheduler and executor design, metadata store performance, task isolation, retries and idempotency, and safe upgrades and migrations
• Software Engineering Foundations: Strong proficiency in Python, plus Go or Java, with a record of writing clean, well-tested platform and tooling code. You are a software engineer who builds and improves systems, not someone who mainly handles tickets and operations
• Kubernetes Expertise: Proven experience managing and scaling production Kubernetes clusters on AWS (EKS), including Helm, autoscaling (Karpenter or Cluster Autoscaler), and GitOps-style deployment
• Extreme Ownership: You define the problem when requirements are unclear, unblock dependencies, and see complex work through to stable production
• Cross-Organizational Technical Leadership: A track record of influencing direction across a large engineering organization and driving multi-quarter projects from concept to production
• Platform Mindset: Experience building platforms used by other engineers, with strong instincts for APIs, observability, reliability, and developer experience
• Experience with other orchestration systems (Dagster, Temporal, Argo Workflows, Flyte) and the judgment to know when each fits

Preferred

• Contributions to Apache Airflow or related open-source projects (Airflow providers, Helm chart, spark-operator, Karpenter, Argo CD)
• Experience with Terraform or other infrastructure-as-code, Prometheus/OpenTelemetry, and cost optimization or chargeback for shared platforms
• Experience operating infrastructure at consumer-internet scale

USD 295,250 - 345,040