Principal DevOps Engineer

Responsibilities

• Platform Architecture & Vision: Architect a Kubernetes-native platform that models broadcast infrastructure as custom resources.
• Lead the technical strategy leveraging Crossplane compositions and custom Go functions to automate provisioning across multi-account AWS environments and on-prem control rooms.
• Design, build, and maintain production-grade Kubernetes operators, controllers, and internal platform APIs in Go.
• You will actively develop custom Crossplane providers to deeply integrate external enterprise platforms (such as NRCS, Venafi, and Infoblox) into our control plane, managing resource lifecycles and approval workflows.
• Lead the design of cloud networking, DNS strategies, and cross-account connectivity across hybrid environments, automating VPC topology and dynamic network routing.
• Partner closely with broadcast systems engineers, system integrators, and external vendors to bridge the gap between broadcast hardware and automated infrastructure.
• You will lead efforts to "Puppet-ize" bare-metal compute configurations and integrate proprietary vendor solutions into our configuration-as-code ecosystem.
• Own the platform's authorization model, designing hierarchical RBAC systems, resource identifier schemes, and identity integrations that enforce fine-grained access control.
• Drive GitOps-based continuous delivery (Flux, Kustomize, Helm) and manage configuration-as-code for compute fleets using Puppet.
• Ensure deep operational visibility by designing comprehensive observability and alerting stacks.
• Oversee the integration of remote desktop/VDI connectivity solutions, focusing on session authentication, credential management, and gateway routing.
• Familiarity with advanced testing frameworks (k6, KUTTL, etc), SOPS for encrypted GitOps configurations, and local development workflows (Air, kind/colima).

Requirements

• 10+ years of experience designing, building, and operating production infrastructure and cloud-native platforms at enterprise scale.
• Strong proficiency in Go (systems-level programming, API servers) and deep experience building Kubernetes controllers/operators using patterns like controller-runtime and kubebuilder.
• Hands-on experience with Puppet, including module development, PuppetDB, Hiera, and r10k.
• Working knowledge of PostgreSQL, SQLite or similar relational databases, encompassing schema design, migrations, and query optimization.
• Excellent problem-solving skills with a proven ability to present architectural decisions to executives, engage with vendors, and write clear technical documentation.

Preferred

• Familiarity with broadcast/media production workflows and the strict operational constraints of live production environments.
• Experience with the Crossplane function SDK for building custom composition functions in Go, and operating in Kubernetes disaster recovery situations (Velero cluster restoration, backups).
• Familiarity with VDI Solutions (NICE DCV, Leostream, PCoIP, etc), machine identity workflows, and PKI certificate management (Venafi or similar).

Technical Leadership

Serve as a technical authority for the team. Write RFCs, drive architectural decisions, mentor engineers, and establish high-confidence CI/CD pipelines, testing strategies, and GitHub Actions automation.

Kubernetes Internals

Expert-level knowledge of the Kubernetes ecosystem, including CRD/XRD generation, operators, informers, admission webhooks, and RBAC.

Crossplane & Control Planes

Deep production experience with Crossplane, including composite resources, composition functions, and specifically developing custom Crossplane providers in Go to integrate external enterprise platforms.

AWS & Networking

Extensive production experience with AWS multi-account architectures, cross-account networking patterns, and identity federation. Requires depth across EKS, EC2, VPC, IAM, STS, SSM, Secrets Manager, Route 53, and S3.

GitOps Delivery

Production experience with GitOps tooling, specifically Flux (HelmRelease, Kustomization) or ArgoCD for continuous delivery on Kubernetes.

APIs & Security

Experience designing REST APIs with middleware patterns and modern authentication (OAuth/JWT). Keen eye for information security, including cross-account IAM trust chains, least-privilege policies, JWT token lifecycles, and secrets abstraction.

Observability

Strong background in designing telemetry platforms using Grafana, Prometheus/Mimir, Loki, OpenTelemetry, and metrics collection agents (Alloy, Prometheus Node Exporter).

Networking & Gateway

Experience with hybrid DNS architectures (Infoblox), software-defined networking (VPC peering, Transit Gateway, Direct Connect, CloudWan), and Envoy Gateway / Gateway API.

Scripting

Ability to script routine tasks in Bash and PowerShell.

Open Source

Active contributions to open-source projects, particularly within the CNCF / Kubernetes ecosystem.

Fully Remote

This position has been designated as fully remote, meaning that the position is expected to contribute from a non-NBCUniversal worksite, most commonly an employee’s residence. This position is eligible for company sponsored benefits, including medical, dental and vision insurance, 401(k), paid leave, tuition reimbursement, and a variety of other discounts and perks. Learn more about the benefits offered by NBCUniversal by visiting the Benefits page of the Careers website.

Salary range

$180,000 - $230,000 (bonus eligible)

As part of our selection process, external candidates may be required to attend an in-person interview with an NBCUniversal employee at one of our locations prior to a hiring decision. NBCUniversal's policy is to provide equal employment opportunities to all applicants and employees without regard to race, color, religion, creed, gender, gender identity or expression, age, national origin or ancestry, citizenship, disability, sexual orientation, marital status, pregnancy, veteran status, membership in the uniformed services, genetic information, or any other basis protected by applicable law. If you are a qualified individual with a disability or a disabled veteran, you have the right to request a reasonable accommodation if you are unable or limited in your ability to use or access nbcunicareers.com as a result of your disability. You can request reasonable accommodations by emailing AccessibilitySupport@nbcuni.com. For LA County and City Residents Only: NBCUniversal will consider for employment qualified applicants with criminal histories, or arrest or conviction records, in a manner consistent with relevant legal requirements, including the City of Los Angeles' Fair Chance Initiative For Hiring Ordinance, the Los Angeles County Fair Chance Ordinance for Employers, and the California Fair Chance Act, where applicable.

NBCUniversal is one of the world's leading media and entertainment companies. We create world-class content, which we distribute across our portfolio of film, television, and streaming, and bring to life through our global theme park destinations, consumer products, and experiences. We own and operate leading entertainment and news brands, including NBC, NBC News, NBC Sports, Telemundo, NBC Local Stations, Bravo, and Peacock, our premium ad-supported streaming service. We produce and distribute premier filmed entertainment and programming through our powerhouse film and television studios, including Universal Pictures, DreamWorks Animation, and Focus Features, and the four global television studios under the Universal Studio Group banner, and operate industry-leading theme parks and experiences around the world through Universal Destinations & Experiences, including Universal Orlando Resort, home to Universal Epic Universe, and Universal Studios Hollywood. NBCUniversal is a subsidiary of Comcast Corporation. Visit www.nbcuniversal.com for more information. Our impact is rooted in improving the communities where our employees, customers, and audiences live and work. We have a rich tradition of giving back and ensuring our employees have the opportunity to serve their communities. We champion an inclusive culture and strive to attract and develop a talented workforce to create and deliver a wide range of content reflecting our world.