Senior Site Reliability Engineer

Role overview Operate and improve large, distributed GPU clusters that support demanding AI training and inference workloads. This senior infrastructure role combines systems engineering, customer support, capacity planning, and incident leadership, with performance traced across hardware, network fabric, orchestration, and ML software.

Responsibilities - Design multi-provider, multi-region GPU clusters, making topology, scheduling, networking, and storage choices that affect training throughput and cost. - Onboard customers running large training jobs, serve as a technical point of contact, and resolve performance or reliability problems with them. - Set reliability objectives and error budgets for GPU-specific failures; plan capacity across varied hardware fleets. - Monitor and troubleshoot high-speed interconnects such as InfiniBand, RoCE, and NVLink, including issues that slow distributed communication. - Build GPU-focused observability for utilization, memory pressure, fabric throughput, job performance, and hardware health. - Develop automation for provisioning, health checks, scheduling, self-recovery, and driver or firmware lifecycle management; lead incident response and follow-up improvements.

Requirements - Direct production experience operating large GPU clusters, such as NVIDIA A100, H100, B200, or equivalent, and understanding their hardware failure modes. - Production experience with high-performance networking for distributed training, including diagnosing degraded links, congestion, or slow collective operations. - Working knowledge of distributed training systems such as NCCL, CUDA, PyTorch distributed, DeepSpeed, Megatron, or FSDP. - Strong Kubernetes operations experience with GPU workloads; familiarity with device plugins, topology-aware scheduling, multi-cluster setups, or custom operators is relevant. - Software engineering ability in Python, Go, or Bash, plus infrastructure-as-code experience with tools such as Terraform, Helm, or Ansible. - Experience building GPU-specific monitoring and leading incidents across hardware, firmware, network, drivers, orchestration, and applications.

Nice to have - Parallel storage experience, training performance optimization, or hands-on cluster buildout involving rack, power, cooling, network design, or hardware validation. - Experience mentoring or leading infrastructure engineers.

Benefits and work setup The position is listed as global remote or San Francisco. It offers substantial autonomy to shape infrastructure and technical direction while working with customers and compute providers.