Prompt Engineer & LLM Evaluator

Role overview Evaluate conversational AI agents that handle appointment booking, and improve their behavior through prompt design and testing. This part-time internship focuses on finding failure modes, measuring performance across realistic and adversarial conversations, and refining system instructions.

Responsibilities - Run manual and automated stress tests across more than 50 conversation scenarios. - Create adversarial prompts to probe agent behavior. - Assess appointment-booking accuracy, language, and tone consistency. - Record hallucinations and other failure cases. - Revise system prompts to improve booking precision. - Maintain evaluation logs and take part in weekly model reviews.

Requirements - Strong curiosity about how language models behave and how prompts shape their responses. - Analytical attention to nuance, tone, and detail. - Basic Python or JSON data evaluation skills. - Proactive approach to problem solving.

Nice to have - College students and recent graduates are welcome to apply.

Benefits and work setup - Paid, remote internship with flexible hours suited to college schedules. - Hands-on AI training and mentorship from experienced architects. - Certificate and recommendation letter offered; potential progression to a junior AI engineering role after graduation.