Principal AI Infrastructure Engineer - Highrise AI

Role overview

Take large AI compute clusters from physical installation through testing, customer handoff, and production operation. The focus is compute systems and workload performance: you’ll lead cluster bring-up, provisioning, systems tuning, and acceptance, then provide Tier 3 support for complex incidents.

Responsibilities

- Bring up installed clusters and establish validated baselines for firmware, BIOS/BMC, GPU drivers, operating systems, and cluster software. - Automate bare-metal provisioning, imaging, node configuration, and lifecycle processes at scale. - Configure and troubleshoot Linux and GPU systems, including kernel, driver, accelerator, and host performance settings. - Integrate and validate Kubernetes and Slurm environments with platform, network, and operations teams. - Run hardware qualification, burn-in, stability checks, GPU benchmarks, and distributed performance tests; analyze results across node, rack, and cluster scales. - Lead root cause analysis and remediation for production issues, support upgrades and expansions, and create diagnostics, runbooks, and operational handoff materials.

Requirements

- Strong expertise in Linux, bare-metal provisioning, systems diagnostics, and automation using tools such as Python, Bash, or Ansible. - Hands-on experience with NVIDIA GPU platforms, NCCL, GPU diagnostics, and multi-node performance testing. - Practical experience with Kubernetes and Slurm. - Ability to investigate issues spanning hardware, firmware, operating systems, GPU software, and distributed workloads. - Strong ownership of testing, documentation, root cause analysis, and operational reliability. - Clear communication and effective collaboration with engineering, vendors, site teams, and customer-facing stakeholders.

Nice to have

- Familiarity with high-speed Ethernet, InfiniBand/RDMA, and parallel or distributed storage.

Benefits and work setup

Full-time engineering role supporting infrastructure at multiple sites. Hybrid or remote work depends on location. Data center travel is required for commissioning, acceptance, and complex troubleshooting, and occasional after-hours incident or maintenance support is expected.