Job Description

Seeking a Lead Machine Learning Engineer / Applied Scientist to improve LLMs and AI agents. This role involves building feedback loops, defining metrics, and enhancing the quality and reliability of intelligent systems. The engineer will collaborate with various teams to integrate insights into Upwork's AI infrastructure, design testbeds, refine prompts, and guide ML-powered feature iteration.

Responsibilities

• Develop and own evaluation pipelines for agentic LLM systems
• Define and iterate on quality metrics for LLMs and agents
• Lead experiments to improve system behaviors
• Collaborate with cross-functional partners to integrate evaluation insights
• Build automated testing and monitoring tools
• Share findings and improvements through documentation and demos

Requirements

• Deep familiarity with evaluation methodologies for LLMs or autonomous agents
• Hands-on experience in Python and ML frameworks such as PyTorch
• Proven ability to work across teams
• Comfortable operating in ambiguous problem spaces
• Passion for continuous improvement, strong documentation habits, and a collaborative working style

Benefits

• No benefits