Remote job
Staff Software Engineer, AI Inference
Job details
About this role
Role overview
Lead the design and development of a production inference platform for serving large language models. This hands-on staff engineering role sets the technical direction for model serving and runtime optimization while building systems that make inference fast, reliable, cost-conscious, and adaptable as models and infrastructure evolve.
Responsibilities - Define the architecture and roadmap for inference infrastructure, model serving, and runtime optimization. - Build and operate scalable systems for production language model workloads. - Evaluate and integrate inference frameworks, serving runtimes, and related technologies. - Improve latency, throughput, GPU utilization, memory efficiency, and infrastructure costs. - Develop deployment, traffic routing, autoscaling, scheduling, observability, and operations capabilities. - Work with machine learning engineers to productionize models and inference techniques, and establish benchmarks for models, runtimes, and hardware. - Make architectural decisions about building capabilities internally or adopting external solutions.
Requirements - Staff-level software engineering experience designing and operating production systems. - Strong expertise in distributed systems and infrastructure for serving large language models. - Ability to optimize performance across latency, throughput, accelerator use, memory, and cost. - Experience evaluating technical options and making architecture decisions for evolving systems. - Strong collaboration skills across machine learning, infrastructure, and product engineering.