Principal Infrastructure Engineer, AI Cluster Performance & Validation

Role overview

Set the technical bar for production readiness and performance across very large AI and HPC clusters. This principal-level role combines infrastructure architecture, hands-on workload testing, deep failure analysis, and software tooling to improve reliability and delivered compute capacity.

Responsibilities - Define cluster health criteria, performance targets, and acceptance processes for clusters containing thousands of GPUs, including before and after customer handover. - Run distributed training and inference workloads at scale, use their results to identify fleet-wide issues, and turn findings into engineering improvements. - Investigate slow or stalled jobs and cluster regressions across accelerators, firmware, host topology, high-speed networking, storage, schedulers, and communication libraries. - Build automated qualification and burn-in pipelines for nodes, racks, and clusters, covering benchmarks, collective communication, thermal and power behavior, and link stability. - Tune networking, communication libraries, storage paths, and host settings to improve usable workload throughput. - Develop Python tools for triage, telemetry correlation, and regression detection; establish validation and observability practices; and mentor teams through runbooks and incident reviews. - Work with infrastructure, platform, reliability, and customer-facing teams to incorporate operational needs into provisioning and remediation workflows.

Requirements - Bachelor’s degree or equivalent practical experience in a relevant technical field. - At least 10 years building, operating, or debugging large-scale compute infrastructure, including substantial staff- or principal-level technical leadership across teams. - Hands-on experience running large-scale AI training or inference, with knowledge of distributed parallelism and frameworks such as PyTorch, Megatron-LM, or DeepSpeed. - Experience qualifying large GPU clusters and defending benchmark methods and results to technical and customer audiences. - Strong skills diagnosing distributed performance problems with tools such as NCCL tracing, Nsight, PyTorch Profiler, perf, or fabric telemetry. - Deep knowledge of InfiniBand or RoCE, RDMA, Linux systems behavior, and production Python; experience with C/C++ or Go and SLURM or Kubernetes at scale.

Nice to have

A graduate degree; experience bringing a new GPU supercluster into production; familiarity with NVIDIA or AMD accelerator stacks, parallel storage, observability platforms, infrastructure-as-code, and bare-metal management systems; or published benchmark and scaling work.