Lead Engineer, AI Platform - Circle

About Us

Circle is building the world’s leading AI-powered, all-in-one platform for digital businesses. We make it possible for creators, coaches, educators, and businesses to bring together their audience with engaging discussions, live streams, events, chat, courses, and payments — all in one place, all under their own brand.

We’re proud to be a fully remote company of around 270 (and growing!) team members from 30+ countries around the world. We don't track hours, collaborate across time zones, are highly async, and value strong documentation. Every year, we bring the entire company together in destinations around the world for our team retreats.

About the Role

The AI Quality engineering team at Circle owns the foundation for measuring, diagnosing, and improving the quality of Circle's AI-powered features. This team focuses on building infrastructure to measure, diagnose, and improve production AI systems, rather than ML research or model training.

We're looking for a Lead Engineer to help build evaluation frameworks, observability tooling, and diagnostic infrastructure to ensure our AI Agents operate reliably, quickly, and cost-efficiently. This is a hands-on player-coach role where you will also lead and manage the AI Quality engineering team.

What You'll Be Doing

• Build and own evaluation infrastructure: Design CI/CD pipelines, scorers, and datasets that validate Circle's AI agents (planners, tool-callers, sub-agents) from individual tool calls to multi-turn conversations.
• Diagnose quality bottlenecks: Trace failures across the agent pipeline (plan creation vs. execution, tool selection, trajectory in workflows, site builder, and analytics) and build prototypes to resolve issues.
• Grow evaluation datasets: Establish annotation workflows and build AI-generated and simulated conversations to scale coverage beyond manual labeling.
• Run structured experiments: Benchmark new and open-source models against production baselines, build model transition frameworks, and optimize cost and latency via caching, routing, and model swaps.
• Lead the team: Set technical direction, manage day-to-day priorities, and remain hands-on in the codebase.
• Partner with AI Core engineering: Collaborate closely with feature engineers to provide high confidence in production releases.

What You'll Need to Be Successful

• 7+ years of production software engineering experience, ideally building and shipping LLM-powered agents that execute actions in complex environments (multi-tool usage, planning/orchestration, sub-agents).
• Proficiency in Ruby on Rails and/or Python: Ruby on Rails is the foundation of our production systems and AI agents; Python proficiency is a strong plus for data and evaluation workflows.
• Experience with AI/ML evaluation and observability infrastructure: Practical experience with eval pipelines, scorers, dashboards, and frameworks such as Braintrust, LangSmith, or similar.
• Dataset & annotation design: Experience setting up labeling pipelines, annotation workflows, and synthetic data simulation for evaluation.
• Experimentation mindset: Ability to run aggressive experiments, rely on data-driven results, and adapt quickly in a fast-paced environment.
• English proficiency: CEFR Level C2 / ILR Level 5 proficiency in written and spoken English.

Compensation & Benefits

• U.S.-benchmarked compensation globally
• Equity in the company with ongoing refresh grants
• 35 days of paid time off per year
• Bi-annual company retreats in international locations
• Health, wellbeing, and professional growth support