Job Description

We’re looking for an AI Platform Engineer to evolve and extend our internal evaluation framework for assessing the quality of our AI-driven experiences. This engineer will work with enough eval systems to quickly make sense of Khan's internal eval framework and recognize opportunities for improvement. You’ll work closely with ML data engineers and platform developers to help internal teams adopt an eval-driven development process.

Responsibilities

• Be fluent in the range of offline and online evaluation strategies
• Have intuitions about how to specify eval pipelines succinctly using declarative syntax
• Understand the role of stratified datasets and ground truth labeling
• Appreciate the range of eval scoring schemes from human raters to automated LLMs-as-judge

Requirements

• Bachelor’s or Master’s degree in Computer Science, Data Engineering, or a related field
• 5 years of Software Engineering experience with 2+ of those years working on the evaluation of generative AI systems
• Strong programming skills in Go, Python, SQL, and at least one data pipeline framework
• Familiarity with the architecture of large language models and their industry-standard APIs

Benefits

• No benefits