Member of Technical Staff | Observability & Reliability

Role overview Own observability and reliability for a model-serving platform that runs in both cloud and customer-managed environments. You’ll make service health, releases, telemetry, and ephemeral workloads visible across these deployments, while improving incident response and reducing interruptions to customer operations.

Responsibilities - Improve logging, metrics, tracing, and alerting across the platform. - Ensure each deployment reports its active release, health, heartbeat, usage, and telemetry to a central control plane. - Enable telemetry from customer clusters when agents can make outbound connections only. - Detect differences between intended and running deployment state, and monitor deployment and runtime agents. - Add visibility into short-lived workloads, including distributed batch-inference clusters. - Define service-level objectives, lead incident response and postmortems, and reduce recovery time, including for customer-environment incidents. - Reduce telemetry cost by removing redundant data and improving signal quality.

Requirements - Deep experience with OpenTelemetry and observability backends. - Hands-on experience with service-level objectives, error budgets, actionable alerting, and incident management. - Strong Kubernetes and infrastructure-as-code experience, such as Terraform or Helm. - Experience operating software in environments where you have limited control. - Ability to write and review production-quality code and take responsibility for operating it.

Nice to have - Deploying software to customer-managed Kubernetes, including outbound-only connectivity patterns. - Experience with GCP/GKE or AWS/EKS, production ML workloads across multiple nodes or clusters, or regulated environments.

Benefits and work setup Full-time, remote role listed for São Paulo. The scope covers cloud and on-premise customer deployments.