AI - Software Development Engineer in Test (AI - SDET)

Role overview Build quality practices for AI-powered applications, including language model workflows, copilots, retrieval systems, and autonomous agents. You’ll work across engineering, AI, platform, product, and operations teams to assess system behavior and develop automation that supports reliability, safety, and scale.

Responsibilities - Design evaluations for response quality, factual grounding, bias, toxicity, context relevance, consistency, and safety. - Create automated checks, test datasets, and benchmarks for language model outputs, agent workflows, and retrieval-augmented generation systems. - Test agent memory, planning, tool use, multi-agent coordination, and integrations with external tools and protocols. - Build and maintain Python-based UI, API, and service automation; connect quality checks to CI/CD pipelines. - Validate retrieval components, model performance, latency, cost, throughput, and behavior in distributed or cloud environments. - Use observability data to investigate failures and improve quality practices across the development lifecycle.

Requirements - Strong Python programming skills and a solid foundation in software testing and quality engineering. - Practical experience testing AI or language model systems, prompt and context design, evaluation methods, retrieval systems, or AI agents. - Experience with API, UI, microservice, and data validation, including SQL. - Familiarity with CI/CD, containers, Kubernetes, and at least one cloud platform. - Strong analytical, debugging, and troubleshooting skills; experience working in Agile teams.

Nice to have - Experience with Robot Framework, Playwright, vector databases, embeddings, or agent development frameworks. - Knowledge of model lifecycle testing, performance or security testing, AI observability, or AI red teaming. - A portfolio of AI or automation projects, open-source contributions, or technical writing. - A relevant degree is preferred; equivalent practical experience and demonstrable projects are also valued.

Benefits and work setup Work may be remote, hybrid, or office-based. The listing describes learning and development opportunities, paid time off, annual bonus, health coverage, tuition and internet reimbursement, and other benefits; eligibility and details may vary.