Test Automation Staff Engineer

Join the Hivemind Software Engineering Integration and Test team as a Staff Automated Test Engineer. In this hands-on role, you will provide technical leadership for automated verification across our next-generation autonomy platform. You will define and evolve test architecture, validation infrastructure, MLOps quality systems, and CI/CD pipelines.

The ideal candidate has deep Python automation experience, distributed-systems testing experience, and technical leadership experience.

Define and evolve test architecture, validation infrastructure, MLOps quality systems, and CI/CD pipelines for automated verification across the autonomy platform.

Responsibilities

• You will work across various systems, including production flight code, machine learning models, simulation environments, and operator-facing applications.
• Lead complex, cross-functional initiatives, establish automation and verification standards, identify systemic quality risks, and develop scalable test infrastructure.
• Design and maintain scalable test frameworks for autonomy software, backend services, APIs, operator-facing applications, and distributed hardware environments.
• Experience qualifying integrated hardware and software systems, including automated validation of compute, GPU, storage, networking, drivers, firmware, and deployed software configurations
• Strong understanding of asynchronous and concurrent Python programming for scalable automation and parallel test execution
• Experience with performance benchmarking, profiling, and observability for GPU workloads, including identifying compute, memory, storage, networking, and data-loading bottlenecks
• Experience with automated observability, log collection, analytics, reporting, and root-cause analysis in complex software, data, and infrastructure systems
• Demonstrated success designing test, CI/CD, or MLOps infrastructure used across multiple engineering teams
• Experience validating machine learning systems across the data, training, evaluation, packaging, deployment, and monitoring lifecycle
• Experience developing integration and regression strategies for internally developed, third-party, open-source, and partner software, including dependency management, compatibility testing, and upgrades
• Experience defining model-performance baselines, automated evaluation suites, release thresholds, and candidate-to-production comparison workflows
• Experience with package and dependency management, reproducible environments, and build systems such as Conan, pip, setuptools, Poetry, Nix, or similar
• Strong system-design skills and experience testing distributed systems, backend services, APIs, and integrated hardware and software environments
• Experience with model registries, experiment tracking, dataset or feature versioning, model serving, and automated artifact promotion using MLflow, Kubeflow, Weights & Biases, SageMaker, Vertex AI, or similar platforms
• Experience with GPU scheduling and orchestration platforms such as Run:ai, NVIDIA GPU Operator, KAI Scheduler, Kueue, Volcano, or similar technologies
• Experience with NVIDIA GPU infrastructure, including CUDA, drivers, container runtimes, Multi-Instance GPU, GPU fractionalization, and hardware/software compatibility testing
• Experience validating perception, planning, decision-making, reinforcement learning, or other autonomy models in simulation and on deployed systems
• Experience testing models on embedded or edge-compute platforms, including latency, memory, power, accelerator compatibility, quantization, and hardware-specific behavior

Requirements

• 5+ years of experience building scalable automation frameworks or developer tooling in Python
• Experience testing GPU-accelerated infrastructure and workloads, including GPU scheduling, allocation, utilization, and resource contention in Kubernetes environments
• Typically 8+ years of relevant experience in software engineering, test infrastructure, developer tooling, MLOps, systems integration, or systems verification, or an equivalent combination of experience and demonstrated impact
• Experience validating multi-tenant Kubernetes environments, including RBAC, resource quotas, workload isolation, and scheduling behavior

Experience working in Linux-based development environments

• Understanding of ML quality risks such as data leakage, training-serving skew, nondeterminism, distribution shift, drift, model regression, and statistical acceptance criteria
• Experience qualifying production servers or appliances, including hardware validation, burn-in, provisioning, firmware, networking, storage, and software-stack validation before deployment
• Experience with reliability, fault-injection, and recovery testing across distributed compute, storage, networking, and GPU infrastructure
• Experience validating reproducible installation, operation, upgrades, and rollback in cloud, on-premises, disconnected, or air-gapped environments
• Experience with containers, Kubernetes, cloud infrastructure, infrastructure as code, and reproducible test environments
• Proficiency with Go or TypeScript for automation tooling or UI test development
• Experience integrating Python with native C or C++ applications through bindings, wrappers, subprocess interfaces, or similar interoperability tooling
• Familiarity with software-in-the-loop, hardware-in-the-loop, requirements-based verification, configuration management, artifact traceability, or standards such as DO-178C and MIL-STD-882

Preferred

• Experience with GPU profiling and performance-analysis tools such as NVIDIA Nsight, PyTorch Profiler, or similar technologies
• Aerospace, robotics, autonomy, embedded systems, or safety-critical software experience

Pay

USD 150,000 - 230,000