Remote job
Senior Software Engineer, Reliability
Job details
About this role
Role overview
Improve the reliability of a cloud service across engineering, infrastructure, and product teams. Alongside incident response and reliability engineering, this role will build an AI-assisted reliability agent that gathers operational context and helps teams triage routine issues. The work aims to make reliability practices, service health, and operational knowledge easier to access and improve across the organization.
Responsibilities - Drive measurable reliability improvements across services, coordinating with engineering, infrastructure, and product partners. - Build and evolve an agent that uses telemetry, incidents, runbooks, deployment data, and service history to support triage. - Design agent workflows with human oversight, clear escalation paths, auditability, and safeguards against incorrect actions. - Strengthen incident detection, response, mitigation, and communication; lead reviews and track corrective actions through completion. - Define service objectives, alerting thresholds, operational readiness criteria, and escalation practices. - Plan gamedays and chaos exercises, maintain reliability scorecards, and lead load, performance, and capacity testing.
Requirements - Strong computer science fundamentals in distributed systems, concurrency, performance, and fault-tolerant design. - Experience building or operating complex systems under high load and partial failure. - Experience with AI-powered tools, agents, or production automation, including their evaluation, failure modes, and human-in-the-loop design. - A record of improving reliability across multiple services, systems, or teams. - Hands-on incident response and post-incident review experience, including leading incidents and following through on fixes. - Experience with one or more of chaos or load testing, capacity planning, observability programs, gamedays, or reliability scorecards. - Clear communication and sound judgment in ambiguous situations; able to mentor, align teams, share operational knowledge, and join an on-call rotation.
Nice to have - Multi-tenant service operations, internal reliability tooling, LLM observability, agent evaluation, retrieval-augmented generation, workflow orchestration, or relevant open-source contributions.
Benefits and work setup - Remote role based in the United States, with no regular travel expected.