Senior System Test Engineer

Role overview

Own system-level testing that helps ensure a distributed data platform remains correct, reliable, and performant under demanding production workloads. You’ll shape the test strategy, build repeatable frameworks, and work closely with engineers to uncover and resolve failures before they affect users.

Responsibilities

- Set priorities for testing correctness, reliability, scalability, and performance, focusing on the platform’s highest-risk behaviors. - Build automated frameworks and realistic workloads for ingestion, concurrent reads and writes, indexing, queries, and dataset maintenance. - Test consistency and data integrity under concurrency, retries, partial failures, and recovery; introduce faults such as crashes, network disruption, and storage errors. - Run load, stress, and soak tests, tracking regressions in latency, throughput, resource use, and search quality. - Validate upgrades, compatibility, and recovery as software and workloads change. - Investigate failures through logs, metrics, traces, and source code; reduce issues to reproducible cases and drive fixes into regression coverage and CI. - Use AI-assisted development tools to build test harnesses and investigate results, checking generated code and conclusions independently.

Requirements

- 10 or more years in software engineering, system testing, or test infrastructure, including substantial work with distributed databases or large-scale data systems. - Deep understanding of distributed database behavior, including consistency, replication, partitioning, concurrency control, durability, and recovery. - A record of designing tests that uncover invariant violations, data corruption, race conditions, and failures under load or partial outages. - Strong Python skills and proficiency in at least one systems language, such as Rust, C++, Java, or Go; able to build test infrastructure and debug product code. - Practical experience with fault injection, performance testing, workload generation, and automated tests in distributed environments. - Experience with Linux, cloud infrastructure, containers, CI/CD, and observability tools. - Able to take ownership of ambiguous work, prioritize risk, communicate findings clearly, and collaborate through resolution. - Regular use of AI coding tools or agent-based workflows, with sound judgment in validating their output.

Nice to have

- Testing experience with storage or query engines, vector search, or cloud object storage; familiarity with columnar data and query technologies. - Experience with property-based, fuzz, differential, or distributed consistency testing; Kubernetes, infrastructure as code, or large-scale test orchestration. - Open-source contributions in databases, distributed systems, or testing frameworks.