Staff / Senior Software Engineer (Agentic Search) - Index

Role overview

A senior or staff-level software engineer is sought to own the indexing and data processing layer of a search engine purpose-built for AI agents rather than human users. The platform exposes programmatic, low-latency, observable search APIs that other AI systems use to retrieve and reason over real-world information. The role is heavily systems-oriented, focused on offline and nearline pipelines that must move tens of gigabytes per second while keeping data fresh, complete, and efficiently queryable.

Responsibilities

- Design, build, and operate large-scale indexing systems and data pipelines at the core of the search infrastructure. - Develop and tune indexing strategies that balance performance, freshness, and resource efficiency, including storage formats, compaction strategies, and update mechanisms. - Define and implement observability primitives such as structured logs, metrics, and data-quality signals, and monitor throughput, resource usage, and cost over time. - Build well-tested components with clear contracts, enabling safe experimentation through controlled rollouts and clearly defined quality signals. - Collaborate with runtime and machine-learning teams to ensure indexed outputs meet retrieval and ranking requirements. - Drive reliability work, incident response, and ongoing optimisations for pipelines operating under sustained high throughput.

Requirements

- Five or more years of experience building production backend or data infrastructure systems. - Strong proficiency in Go, with working knowledge of C++ or Rust considered a plus. - Hands-on experience with large-scale data processing at multi-gigabyte-per-second throughput and petabyte-scale datasets. - Background building or operating databases, storage systems, data planes, or indexing pipelines. - Deep understanding of distributed systems, including fault tolerance, consistency, and scalability. - Comfort running production systems and handling operational incidents, with a systems-thinking mindset for end-to-end data flows.

Nice to have

- Experience with distributed data processing frameworks such as Spark, Flink, MapReduce, or Beam. - Familiarity with content systems including scraping, proxying, or anti-bot infrastructure, or backgrounds in ad tech, social platforms, or other large-scale content systems. - Knowledge of DBMS internals and cloud infrastructure, open-source contributions, competitive programming or CTF experience, advanced technical programmes, conference talks, or technical publications.

Benefits and work setup

- Competitive compensation with career growth and learning opportunities. - Flexible, high-ownership culture within an international, collaborative engineering environment, focused on impactful AI infrastructure projects.