Remote job
ML Systems Engineer, On-Device Inference
Job details
About this role
Role overview Build and optimize inference systems that let language and voice models run effectively on phones, personal computers, and other devices. You will find where time, memory, and energy are spent, improve execution on real hardware, and collaborate with model researchers so model choices translate into responsive, sustained performance.
Responsibilities - Profile model loading, prompt processing, decoding, and streaming output across mobile and desktop hardware. - Optimize kernels, execution graphs, memory allocation, and cache behavior for target processors. - Develop mixed-precision and quantized execution paths while checking numerical correctness and model quality. - Measure sustained performance with realistic context sizes, concurrent workloads, and thermal conditions. - Build repeatable benchmarks and regression checks for latency, memory, energy use, and reliability. - Work with research teams on model and runtime design, and package improvements as maintainable interfaces.
Requirements - Strong systems programming in C++, Rust, or a comparable language, plus Python for experiments and tooling. - Understanding of transformer inference, memory hierarchies, and compute or bandwidth bottlenecks. - Experience with inference runtimes, hardware acceleration, or deploying machine learning on constrained devices. - Ability to use profiling and measurements to identify bottlenecks and verify improvements. - Careful debugging and testing, including attention to numerical changes and performance regressions.
Nice to have - Experience with Metal, MLX, Core ML, CUDA, or mobile GPU/NPU execution. - Work on low-bit quantization, custom kernels, compiler optimization, or streaming audio systems.
Benefits and work setup Full-time position; location is San Francisco or remote.