Senior Systems Software Engineer - HPC and AI Storage

Role overview

Improve storage software and systems for large HPC and AI clusters through engineering, experimentation, and integration. You will trace performance and reliability issues across the I/O stack, turn workload evidence into software or configuration changes, and lead storage qualification and rollout at cluster scale.

Responsibilities

- Integrate and engineer shared storage such as Lustre and NFS with Linux compute nodes, networks, and storage hardware. - Investigate performance and reliability across applications, filesystem clients, kernel paths, network protocols, servers, and devices. - Develop and validate fixes with engineering teams, storage partners, and open-source maintainers, guiding changes through review and deployment. - Characterize training-data reads, checkpoint and restart behavior, metadata-heavy operations, and concurrent workloads; use findings to improve software and system design. - Create repeatable benchmarks, diagnostic tools, and regression tests for latency, metadata operations, scalability, contention, and efficiency. - Lead storage qualification and commissioning, checking functionality, data integrity, recovery, and sustained performance; define placement, caching, and lifecycle policies based on operational needs.

Requirements

- Deep engineering experience with Lustre, NFS, or comparable shared storage, including implementation-level debugging and Linux cluster integration. - Strong C and Linux debugging skills, including experience diagnosing or modifying kernel or user-space storage software. - Understanding of filesystem and distributed-system behavior, including metadata, caching, consistency, locking, concurrency, client/server communication, and recovery. - Experience analyzing application I/O patterns and tuning storage systems for measurable performance gains. - Ability to take complex correctness or performance issues from reproduction through root-cause analysis, implementation, and validation. - Python and shell scripting skills, strong cross-team technical leadership, and a relevant degree or equivalent practical experience with 12+ years of relevant work.

Nice to have

- Contributions to storage or filesystem projects, or expertise in areas such as Linux VFS, page cache, NFS internals, Lustre metadata, distributed locking, and recovery. - Experience with hierarchical storage, data migration and recall, RDMA transports, GPUDirect Storage, or optimizing GPU data loading and checkpointing. - Knowledge of storage security and multi-tenancy, including identity mapping, permissions, authentication, or encryption.