Remote job
Senior Software Engineer - Dev Platform
Job details
About this role
Role overview
Build and operate an internal AI gateway that gives engineering teams governed access to large language model providers. The platform supports model selection, access controls, usage visibility, and operational safeguards, helping teams adopt LLM capabilities while managing cost, reliability, and security. You’ll own features from design through production and work closely with platform, security, and reliability partners.
Responsibilities - Add model and provider integrations, routing rules, budgets, rate limits, and usage and cost reporting. - Design APIs, SDKs, and self-service tools that make onboarding straightforward for internal engineering teams. - Help migrate existing LLM workloads with minimal disruption. - Implement access control, key management, and audit logging in partnership with security specialists. - Instrument services to expose latency, errors, token usage, and costs for each workload. - Own production features, support internal users with documentation and debugging, and contribute to reviews and design practices.
Requirements - Production experience building and operating backend APIs or distributed services. - Strong ability in a backend language such as Rust, Python, or TypeScript, with a focus on maintainable, tested code. - Working knowledge of LLM APIs, including authentication, streaming, token accounting, rate limits, and provider failures. - Experience with observability, CI/CD, and containers, plus familiarity with Kubernetes and a major cloud provider. - Ability to design clear, documented APIs for internal users and incorporate their feedback. - Strong communication and ownership, carrying work from proposal through deployment.
Nice to have - Experience with API gateways, proxies, or traffic management. - Experience building LLM systems involving routing, caching, evaluation, or cost controls. - Familiarity with OAuth, OIDC, or role-based access control. - Contributions to open-source infrastructure or developer platform projects.