Senior Software Engineer

Role overview Improve a data infrastructure system used to manage large-scale datasets for AI and research workloads. The work combines distributed systems engineering, high-performance computing, open-source integration, and improvements to usability and operations.

Responsibilities - Design and maintain efficient operations for distributed datasets. - Build indexes that support predicate pushdown and speed up queries in systems such as Spark, Ray, or Trino. - Develop table formats, data encodings, and related format capabilities in Rust. - Lead integrations with data infrastructure projects, including Spark, Hive Metastore, Presto, Trino, and Ray. - Operate and improve internal data processing infrastructure. - Contribute to open-source community engagement and share the format’s capabilities at data conferences.

Requirements - At least ten years building high-performance databases, big data systems, or large-scale data services. - Deep knowledge of internals in open-source data or AI training systems, such as Hadoop, Spark, Flink, Ray, Iceberg, Delta Lake, Hudi, ClickHouse, Trino, Presto, PyTorch, or JAX. - Strong high-performance computing experience in C++, Java, and/or Scala. - Rust experience or willingness to learn the language. - Ability to work independently, move quickly, and collaborate effectively.

Nice to have Open-source contributions to Apache or other large projects; experience with Apache Arrow, DataFusion, Parquet, Iceberg, or Delta Lake; a record of delivering major distributed-systems features or integrations; and active participation in open-source communities.