Honeycomb logo
Honeycomb·

Senior Software Engineer II - LLM Observability

What We’re Building

Honeycomb is defining observability and raising expectations of what developer tools can do. We work with leading companies like HelloFresh, Slack, LaunchDarkly, and Vanguard across a range of industries. We’ve closed Series D funding, scaled past the 200-person mark, and were named to Forbes’ America’s Best Startups.

Who We Are

We come for the impact, and stay for the culture. We’re a talented, opinionated, passionate, fiercely inclusive, and responsible team. We have conviction and strive to live our values every day, doing what we truly love amongst humble, highly talented peers.

How We Work

We are a fully distributed company. We believe it is not where you sit, but how you deliver that matters most. We invest in our people, orient you to our culture, and offer high trust, autonomy, and accountability from Day 1.

About the Team

The team builds observability for AI in Honeycomb. We help customers see what their agents are doing, what those agents cost, and where errors happen, while providing insight into individual conversations to enable debugging. We bring LLM and agent observability together, owning both the data foundation and the product experiences built on top of it. You will work in a small, cross-functional group of engineers with a dedicated product manager and designer, shipping production features.

Responsibilities

  • Own end-to-end development of AI observability capabilities, from ingestion to in-product visualizations.
  • Architect and evolve the data model and infrastructure that lets us observe agents and their ecosystems across Honeycomb.
  • Build systems that analyze and surface AI-specific signals, such as cost, token usage, errors, and anomalies.
  • Collaborate with product and design partners to build intuitive workflows.
  • Teach best practices for exploring and understanding AI system behavior across conversations and workloads.
  • Iterate through fast feedback loops and experimentation, sharing both wins and failures.

Requirements

  • Significant software engineering experience, with a track record of shipping and owning features end to end.
  • Experience shipping customer-facing features at a SaaS company beyond internal tooling.
  • Full-stack range, with backend proficiency (Go and/or Python) and frontend proficiency (TypeScript and React).
  • Track record of close collaboration with product managers and designers.
  • Regular, hands-on use of AI tools in your day-to-day engineering workflows.
  • Experience participating in an on-call rotation (joining our rotation after ~90 days).
  • Comfort navigating ambiguity in a fast-moving field with shifting requirements.

Nice to Have

  • Experience building LLM or agent-powered features in production.
  • Hands-on experience with observability tooling, such as OpenTelemetry.

Compensation & Benefits

  • Base Salary: $250,000 — $280,000 CAD
  • Generous equity with an employee-friendly stock program
  • Transparent pay levels relative to experience
  • Unlimited PTO
  • Distributed-first mindset and culture
  • Home office, co-working, and internet stipends
  • Comprehensive health benefits coverage
  • Up to 16 weeks of paid parental leave
  • Annual professional development allowance

Timezone overlap

UTC-8–-4

Open to

NA

Sign in to track applications and earn points.

More roles at Honeycomb

Similar remote roles