ML Platform Engineer, Distributed Training

Role overview Build and improve the software and training workflows used to train large foundation models on a multi-node GPU platform. You’ll work closely with research scientists and infrastructure engineers to tune, benchmark, and troubleshoot training jobs end to end, with ownership of your technical area.

Responsibilities - Improve the training platform’s GPU workloads, scheduling, and capacity management in a Kubernetes environment. - Diagnose platform and training failures at their root cause. - Tune distributed GPU networking, including RDMA or GPUDirect and NCCL, to improve training throughput. - Run and debug multi-node training jobs, then use benchmarks and model evaluations to guide improvements. - Coordinate with scientists and infrastructure colleagues, explaining technical decisions and adapting as priorities change.

Requirements - At least five years of engineering experience, including recent hands-on work with GPU training infrastructure and large-scale model training. - Strong Python skills and experience writing production-quality code, tests, and code reviews; shell scripting ability is also expected. - Experience with Kubernetes on Google Cloud, GPU node pools, job scheduling such as Kueue, and quota or capacity management. - Practical experience running and debugging distributed PyTorch and Ray Train jobs. - Knowledge of RDMA or GPUDirect, NCCL tuning, and multi-node H200 or GB200 GPU topology. - Experience designing benchmarks and model evaluations, and communicating clearly while taking ownership of work.

Nice to have - Experience with healthcare or life-sciences data, including genomics, clinical, or biomedical data, or work under HIPAA requirements. - Familiarity with training platform observability, including metrics, tracing, alerting, or end-to-end pipeline tests.

Benefits and work setup The role includes health insurance, paid vacation and holidays, sick and parental leave, bereavement leave, an employer-matched 401(k), and referral bonuses. The listed annual salary range is $159,300–$219,245; the final range depends on level, experience, and location.