Senior Machine Learning Engineer, LLM Inference Optimization

Role overview

Own performance improvements for large language and vision-language model inference, from model artifacts through production serving. You’ll combine model-level techniques, inference engines, and distributed serving design to improve latency, throughput, memory use, GPU utilization, and cost while preserving output quality and reliability. The work includes reproducible benchmarking, collaboration with kernel and platform engineers, and safe rollout of changes under real workloads.

Responsibilities

- Lead optimization projects for model families, customer endpoints, or serving backends; compare inference engines and recommend workload-specific configurations. - Diagnose performance and model-quality regressions, tracing bottlenecks across model code, kernels, runtimes, schedulers, gateways, and clusters. - Develop compression workflows, including quantization, distillation, and accuracy recovery, and apply methods such as speculative decoding and KV-cache optimization. - Improve serving through caching, batching, prefill and decode coordination, routing, and scheduling; assess when disaggregated prefill and decode designs are useful. - Scale dense and mixture-of-experts inference across GPU nodes, tuning parallelism with attention to hardware topology, communication, and expert load balance. - Build repeatable benchmarks for measures such as time to first token, generation speed, tail latency, memory, reliability, GPU efficiency, and cost; document results and rollout plans.

Requirements

- Strong Python and PyTorch engineering skills. - Hands-on experience deploying or optimizing LLM, VLM, or other high-throughput transformer inference. - Practical experience with a modern serving stack such as vLLM, SGLang, TensorRT-LLM, Triton Inference Server, NVIDIA Dynamo, or an equivalent system. - Understanding of inference bottlenecks involving attention, KV cache, memory bandwidth, batching, parallelism, and long contexts. - Experience making and measuring trade-offs among latency, throughput, output quality, utilization, and cost. - Practical work designing or optimizing distributed inference, such as routing, scheduling, prefill/decode coordination, multi-node serving, or expert parallelism. - Strong collaboration and communication across engineering, research, product, infrastructure, and customer teams.

Nice to have

- Experience with quantization methods or formats such as FP8, INT8, INT4, NVFP4, MXFP4, AWQ, GPTQ, or SmoothQuant; quantization-aware training is also relevant. - Familiarity with distillation, speculative decoding, CUDA, Triton, GPU communication libraries, or interconnects such as NCCL, NVLink, InfiniBand, or RoCE. - Experience with agentic workloads, cache-aware routing, disaggregated serving, or wide expert parallelism. - Contributions to inference, machine learning, or serving open-source projects.

Benefits and work setup

The role offers competitive compensation, learning and career development, flexibility, and ownership. It involves working with an international team on applied AI systems.