Tech Lead - AI Inference

Role overview

Lead a small engineering team building high-throughput, low-latency infrastructure for serving large language models. The role combines hands-on systems development with technical direction, delivery ownership, and people management, working across serving frameworks, GPU memory, storage, and networking.

Responsibilities - Own technical decisions and delivery for inference infrastructure, including NVMe-backed token storage, GPU Direct Storage data paths, and model serving components. - Guide three developers through system design, implementation, and production delivery while maintaining standards for reliability and code quality. - Contribute directly to production systems using Python, C++, CUDA, and the surrounding infrastructure stack. - Address challenges such as separating prompt processing from token generation, persistent KV caching beyond GPU memory, RDMA transport, and tiered GPU memory. - Coach engineers through regular one-to-ones, career discussions, and sprint reviews. - Evaluate emerging inference frameworks and tools through benchmarking, helping determine when to adopt, build, or change direction.

Requirements - At least five years of professional software engineering experience, including leading engineers and owning complex production systems. - Hands-on knowledge of LLM serving, including KV cache reuse, continuous batching, disaggregated prefill and decode, and tiered GPU memory. - Strong Python and C++ skills and a solid understanding of CUDA, GPU memory management, and high-performance I/O. - Experience with GPU storage and transport technologies such as GPUDirect Storage, RDMA, and NVMe data paths. - Experience deploying and scaling GPU workloads on Kubernetes, with familiarity in bare-metal GPU clusters and distributed storage. - Demonstrated ability to mentor engineers and balance technical delivery with team development. - Track record of building reliable, high-throughput systems and improving engineering practices.

Nice to have - Experience in AI/ML infrastructure or high-performance computing. - Familiarity with serving stacks such as vLLM, LMCache, NIXL, or comparable frameworks; Rust experience is also useful. - Experience with SGLang or TensorRT-LLM and GPU clusters using H100- or A100-class accelerators.

Benefits and work setup - The source mentions potential medical, dental, vision, and life insurance, retirement benefits, flexible time off, sick leave, and applicable statutory leave. The role involves hands-on work with a three-person engineering squad.