Senior GPU Infrastructure Engineer

Role overview Own the reliability, performance, and automation of infrastructure supporting large scale GPU workloads. The role spans server hardware and firmware, GPU software, high speed networking, storage, and cluster orchestration, with responsibility for resolving complex incidents and preventing repeat failures.

Responsibilities - Investigate GPU, PCIe, memory, and driver faults, tracing intermittent problems to their root causes. - Build automated node health checks and diagnostics using GPU monitoring and collective communication test tools. - Plan and automate firmware, BIOS, and driver rollouts across production clusters. - Diagnose performance and reliability issues in InfiniBand and RoCE networks, and optimize parallel storage and GPU data paths. - Lead incident response, coordinate technical escalations, write post incident reviews, and implement preventive fixes. - Improve hardware observability and automate degraded node detection and recovery.

Requirements - At least five years in SRE, systems, or infrastructure engineering for production Linux environments. - At least three years managing and troubleshooting data center NVIDIA GPUs in distributed environments. - Strong knowledge of CUDA drivers, DCGM, NVLink/NVSwitch, and GPU fault diagnosis. - Advanced Kubernetes skills, including networking and device plugins, plus experience with Slurm. - Practical knowledge of InfiniBand, RoCEv2, and NCCL tuning. - Strong automation skills with Python, Go, or Bash, plus Terraform or Ansible; experience with Prometheus and Grafana and clear incident documentation.

Nice to have - Professional certifications in NVIDIA AI infrastructure or InfiniBand, or Kubernetes administration or security. - Experience with liquid cooled data center racks.

Benefits and work setup Remote work with flexibility, a four day work week except when attending events, and uncapped holiday.