Lead Engineer, AI Quality - Circle

About Us

Circle is building the world’s leading AI-powered, all-in-one platform for digital businesses, enabling creators, coaches, educators, and businesses to bring together audiences with engaging discussions, live streams, events, chat, courses, and payments under their own brand.

We are a fully remote company of around 270 team members across 30+ countries. We collaborate across time zones, operate with high async workflows, and value thorough documentation. Twice a year, we bring the whole company together in incredible destinations around the world for our company offsites.

About the Role

The AI Quality engineering team at Circle owns the foundation for measuring, diagnosing, and improving the quality of Circle's AI-powered features. We focus on building infrastructure to measure, diagnose, and improve production AI systems rather than ML research or model training.

We're looking for a Lead Engineer to help build evaluation frameworks, observability tooling, and diagnostic infrastructure to ensure our AI Agents work reliably, efficiently, and at scale.

This is a hands-on player-coach role where you will also lead and manage the AI Quality engineering team.

What You'll Be Doing

• Build and own our evaluation infrastructure: Design CI/CD pipelines, scorers, and datasets to measure whether Circle's AI agents (planners, tool-callers, and sub-agents) work effectively from single tool calls to multi-turn conversations.
• Diagnose quality breakdowns: Trace failures across agent pipelines including plan creation, execution, and tool selection across workflows, site builders, and analytics, turning findings into prototypes or priorities for core AI engineering.
• Grow evaluation datasets: Stand up annotation workflows and build AI-generated and simulated conversations to expand product coverage beyond manual labeling.
• Run structured experiments: Evaluate new and open-source models against production baselines, build framework selection tools, and chase cost and latency optimizations.
• Lead and grow the team: Set technical direction and manage day-to-day priorities while remaining hands-on in the code.
• Partner with AI Core engineering: Collaborate with product AI engineers to ensure changes measurably improve system quality.

What You'll Need

• 7+ years of experience building and shipping production software, including LLM-powered agents that take real actions in a product.
• Proficiency in Ruby on Rails / Python or readiness to pick them up quickly (Ruby on Rails is our production system; Python experience is a strong plus).
• Experience building evaluation or observability infrastructure for ML/AI systems (e.g., Braintrust, LangSmith).
• Experience designing datasets and annotation workflows for ML/AI evaluation.
• Experimental mindset: Thrive in dynamic environments, running and learning from rapid experiments.
• Strong alignment with our company values and C2 English proficiency.