Remote job
ML Systems Engineer, On-Device Inference
Job details
About this role
Role overview This full-time systems engineering role builds inference software that helps language and voice models run effectively on phones, personal computers, and other devices. The work spans profiling, execution-path optimization, and collaboration with model researchers to translate architectural choices into responsive and sustained performance on real hardware. The role is listed as based in San Francisco or remote.
Responsibilities - Profile model loading, prompt processing, decoding, and streamed output across mobile devices and desktop hardware. - Optimize kernels, execution graphs, memory allocation, and cache use for target processors. - Develop mixed-precision and quantized execution paths, checking numerical correctness and model quality as well as speed. - Measure sustained performance across realistic context lengths, concurrent workloads, and thermal conditions. - Build repeatable benchmarks and regression checks for latency, memory use, energy consumption, and reliability. - Collaborate on model and runtime design, and package improvements in maintainable interfaces for other engineers.
Requirements - Strong systems programming skills in C++, Rust, or a comparable language, plus Python for experiments and tooling. - Understanding of transformer inference, memory hierarchies, and limits on compute or bandwidth utilization. - Experience with inference runtimes, hardware acceleration, or machine-learning deployment on constrained devices. - Ability to use profiling and measurements to identify bottlenecks and verify improvements. - Careful debugging and testing practices, including attention to numerical changes and performance regressions.
Nice to have Experience with Metal, MLX, Core ML, CUDA, mobile GPU or NPU execution, low-bit quantization, custom kernels, compiler optimization, or streaming audio systems.