AI Platform Engineer

Role overview A contract engagement for two senior engineers to own the evaluation and observability layer of a major AI platform programme in a high-traffic, heavily regulated consumer sector. The work is embedded with the client's own engineering and product teams, and the central question is whether shipped models can actually be trusted to perform in production.

Responsibilities - Design and build evaluation pipelines on a platform such as Braintrust, LangSmith, Arize or Weights & Biases, choosing the right tool for each problem. - Curate golden datasets reflecting real user behaviour, and design LLM-as-judge and human-in-the-loop scoring with a clear sense of where each is reliable. - Build production observability: token-level tracing, quality signal monitoring, hallucination and refusal detection, and per-use-case cost attribution, with alerting that catches degradation early. - Wire eval gates into CI/CD promotion paths so regressions are blocked rather than discovered after release. - Establish baselines, regression suites and drift detection so prompt, model and retrieval changes can be made with statistical honesty. - Present eval findings in a way that drives decisions, push for adoption of eval-first habits, and manage the delivery stream including scope, estimates and risk.

Requirements - Senior-level experience operating AI or machine learning systems in production. - Practical knowledge of evaluation, observability and quality-gating tooling for LLM applications. - Strong statistical intuition, including the ability to size eval sets and distinguish real signals from noise. - Experience working inside an existing client delivery process, with the credibility to challenge assumptions with evidence. - Comfortable with cloud platforms such as AWS or GCP at an associate or equivalent practitioner level.

Nice to have - AWS Certified Machine Learning Engineer (Associate) or AWS Certified AI Practitioner. - An associate-level AWS or GCP certification relevant to the platform work in scope.

Benefits and work setup - Contract engagement with duration, rate and IR35 status confirmed at first contact. - Largely remote working with travel to the client site or to the consultancy's UK hubs (Edinburgh, Leeds, Manchester, London) as the programme requires.