Remote job
HPC Engineer
Job details
About this role
Role overview Operate the bare-metal and virtualized GPU clusters that support AI training and inference. The work spans provisioning, networking, storage, scheduling, hardware troubleshooting, and incident response, with responsibility for keeping production clusters healthy and performant.
Responsibilities - Administer bare-metal GPU and HPC clusters from provisioning through ongoing operations, as well as virtualized clusters and their hypervisor and GPU virtualization layers. - Design and tune InfiniBand fabrics, including topology, subnet management, and performance validation. - Deploy and operate shared or parallel filesystems, balancing throughput, capacity, and reliability for training and inference workloads. - Diagnose issues across fibers, transceivers, network adapters, switches, drivers, firmware, and physical hardware; coordinate repairs and expansions with data center teams. - Operate and tune Slurm or a comparable workload scheduler. - Maintain accurate issue tracking, IP address management, and data center inventory records, and participate in on-call and cluster incident response. - Coordinate cluster integration with platform, network, and storage teams.
Requirements - Strong Linux administration skills; knowledge of memory management, PCIe topologies, or virtualization is useful. - Deep InfiniBand knowledge, including fabric design. - Familiarity with GPU computing and the NVIDIA stack, including tools or technologies such as NCCL, CUDA, and DOCA. - Experience with Slurm or another workload scheduler. - Ability to troubleshoot production hardware and work with remote data center teams on diagnosis and repairs. - Scripting or automation skills for repeatable operations, such as Python, Bash, or Ansible. - Careful operational habits, including keeping tracking and infrastructure records current, and comfort supporting systems where uptime and performance affect customer workloads.
Nice to have - RoCEv2 or Spectrum-X; GPU health diagnostics such as DCGM or field diagnostics. - Bare-metal provisioning tools such as MAAS, Foreman, or PXE/iPXE pipelines. - GPU-aware virtualization or containerization, including Kubernetes device plugins, KVM/QEMU passthrough, or SR-IOV. - Cluster observability with tools such as Prometheus, Grafana, or Loki.
Benefits and work setup - Mid- to senior-level, full-time permanent role, remote within the EU. - Cash and equity compensation, plus fringe benefits.