Senior Software Engineer (Distributed Systems)

Join Workday's Data Platform and Observability team as a Senior Software Engineer. You'll be responsible for designing, building, and improving critical observability services, owning distributed tracing end-to-end, and exploring AI-driven observability techniques. This role offers a hybrid work schedule and the opportunity to work on large-scale distributed systems and AI-driven products.

Responsibilities

• Design, build, and improve critical observability services, including monitoring, logging, alerting, and tracing.
• Own distributed tracing end to end, instrumenting services, propagating context across service boundaries, and using trace data to understand system behavior.
• Explore and apply AI-driven observability techniques, such as anomaly detection, intelligent alerting, and automated root-cause analysis, to reduce noise and speed up incident response.
• Ability to design, maintain, and optimize Time-Series DBs, document stores, and log collection/management systems

Requirements

• 7 years of coding in Python, Go, or Java
• Experience with Linux
• 8 years of software engineering/design experience
• BS in Computer Science or a related technical field, or equivalent experience
• MS Degree a plus
• Hands-on experience with LLM orchestration frameworks (e.g., LangChain, LlamaIndex, Semantic Kernel) and a working knowledge of agentic AI build patterns — how agents plan, call tools, hold state, and hand off work across multi-step chains

Excellent interpersonal, technical, and communication skills

• Solid understanding of distributed tracing concepts (context propagation, spans, sampling) and hands-on experience with tracing tooling
• Experience with service mesh, Prometheus, and cloud-native technologies
• Ability to prioritize multiple tasks in a fast-paced environment
• A knack for spotting where AI systems quietly go wrong in production: a model call that's suddenly slower than usual, a token bill creeping up for no clear reason, or answers that subtly drift in quality over time. You know how to build the dashboards, alerts, and instrumentation that catch these early — turning "the AI feels off" into a measurable, debuggable signal
• Public cloud experience (AWS/GCP), including native observability tooling such as AWS CloudWatch and GCP Cloud Operations (Stackdriver)

Preferred

• Experience with containerization and infrastructure automation (Docker, Kubernetes, Ansible, Chef, Terraform)

Pay

USD 168,500 - 285,100