Senior ML Engineer | Kimchi (Inference Stack Optimization)

Role overview

Improve the performance and economics of large-scale language-model inference, with ownership spanning the serving stack from model loading and scheduling to caching and distributed execution. The goal is to reduce latency and cold-start time while increasing throughput and controlling cost, using measured results to guide technical decisions. The role combines hands-on systems work with responsibility for setting optimization priorities.

Responsibilities

- Tune serving engines, batching, prefill, decoding, and scheduling to raise throughput across GPU types. - Profile time-to-first-token and token-generation latency, identify real bottlenecks, and optimize the responsible compute, memory, storage, network, or scheduling paths. - Improve KV-cache usage through techniques such as prefix caching, paging, tiering, eviction policies, and quantization. - Shorten model startup by improving weight-loading paths, compilation caches, and shared-filesystem reads. - Evaluate quantization on real workloads, balancing model quality with memory and performance gains. - Design and improve multi-GPU and multi-node inference, including placement and separation of prefill and decode; establish reproducible benchmarks and explain technical recommendations.

Requirements

- At least five years building production machine-learning systems, with substantial experience in inference or training infrastructure. - Strong production Python skills and practical experience running vLLM, SGLang, or TensorRT-LLM. - Understanding of GPU performance and experience measuring quantization effects on quality, not only compression. - Experience with multi-GPU or multi-node systems, parallelism, and collective communication. - Knowledge of memory and storage behavior, including page caches, direct I/O, and large reads over network filesystems. - Ability to instrument systems and distinguish repeatable improvements from benchmark artifacts.

Nice to have

Experience with LLM routing evaluations, cost-quality-latency trade-offs, calibration or abstention methods, or production A/B experiments with drift monitoring.

Benefits and work setup

Remote-first work, equity options, a learning budget, equipment support, extra days off, and time allocated for personal development are listed. The role does not include visa sponsorship.