Lead Engineer, AI Quality - Circle

About Us

Circle is building the world’s leading AI-powered, all-in-one platform for digital businesses. We make it possible for creators, coaches, educators, and businesses to bring together their audience with engaging discussions, live streams, events, chat, courses, and payments — all in one place, all under their own brand.

We’re proud to be a fully remote company of around 270 team members from 30+ countries around the world. We collaborate across time zones, are highly async, and twice a year, we bring the whole company together for our company offsites in beautiful places around the world.

About the Role

The AI Quality engineering team at Circle owns the foundation for measuring, diagnosing, and improving the quality of Circle's AI-powered features. This team focuses on building the infrastructure to measure, diagnose, and improve production AI systems, rather than ML research or model training.

We're looking for a Lead Engineer to help us build out the evaluation frameworks, observability tooling, and diagnostic infrastructure that tell us whether our AI Agents are working well, where to improve them, and how to make them faster and more cost-efficient.

This is a hands-on player-coach role where you’ll also lead and manage the AI Quality engineering team against an ambitious and growing roadmap.

What You'll Be Doing

• Build and own our evaluation infrastructure: Design the CI/CD pipelines, scorers, and datasets that tell us whether Circle's AI agents are actually working, from a single tool call to a full multi-turn conversation.
• Diagnose quality bottlenecks: Trace failures across the agent pipeline including plan creation vs. execution, tool selection, and tool trajectory in complex areas like workflows, site builder, and analytics.
• Grow evaluation datasets: Stand up annotation workflows and build out AI-generated and simulated conversations to cover more of the product faster than manual labeling alone.
• Run structured experiments: Evaluate new and open-source models against our production baseline, build frameworks for model shifting, and pursue cost and latency optimizations.
• Lead and grow the team: Set technical direction and manage day-to-day priorities while staying hands-on in the code.
• Partner with AI Core engineering: Work with product engineering to ensure AI changes measurably improve quality.

What You'll Need

• 7+ years of experience: Building and shipping production software, ideally including LLM-powered agents taking real actions in a product.
• Stack proficiency: Comfortable in Ruby on Rails / Python or ready to pick them up quickly (Ruby on Rails is the core production system; Python is a strong plus).
• Eval/Observability experience: Background building eval pipelines, scorers, dashboards, or CI/CD for ML/AI systems using tools like Braintrust or LangSmith.
• Dataset design: Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
• Execution: Ability to learn fast, experiment aggressively, handle ambiguity, and thrive in a dynamic, fast-paced environment.
• Communication: Proficient in English (CEFR Level C2 / ILR Level 5).

Compensation & Benefits

• U.S.-benchmarked global compensation
• Company equity with ongoing refresh grants
• 35 days of paid time off each year
• Biannual remote company retreats in incredible global destinations
• Comprehensive health, wellbeing, and professional growth support