Remote job
Performance & Debug Software Engineer – AI Inference
Job details
About this role
Role overview
Improve end-to-end performance for large language model inference on custom accelerator systems. The work spans attention computation, accelerator configuration, distributed KV-cache management, host-to-device data movement, runtime behavior, and serving systems. You will use real workloads and measured evidence to identify bottlenecks, implement improvements, and help shape hardware and software interfaces.
Responsibilities
- Profile and optimize latency, throughput, utilization, and efficiency across accelerator, runtime, host, and on-device execution. - Use performance counters, traces, telemetry, and simulation to isolate regressions and assess fixes. - Develop distributed inference capabilities, including KV-cache management and data movement across heterogeneous hardware. - Build representative benchmarks and workload traces that expose compute, memory, interconnect, networking, scheduling, and model-loading limits. - Turn performance findings into regression tests and release gates in collaboration with test engineers. - Inform hardware and software architecture where measurements reveal structural performance constraints.
Requirements
- Strong C++ and Python skills, with practical experience profiling and optimizing complex Linux systems that use VFIO interfaces. - Deep understanding of memory hierarchies, concurrency, data movement, batching, scheduling, and performance measurement. - Hands-on profiling experience; the available source cuts off before listing the specific tools or any further qualifications.
Benefits and work setup
The role is remote-first, with a company-provided computer and home-office setup. Listed benefits include company-paid medical, dental, and vision coverage, life and disability coverage, supplemental insurance options, unlimited paid time off, 13 paid holidays, equity, and a 401(k) match. Applicants must already be authorized to work in the U.S.; new visa sponsorship is unavailable, though eligible H-1B transfers may be supported.