Senior Software Engineer II - Agentic Intelligence

About Honeycomb

Honeycomb is defining observability and raising expectations of what developer tools can do. We work with well-known companies like HelloFresh, Slack, LaunchDarkly, Vanguard, and more across a range of industries. We are a fully distributed company where trust, autonomy, and accountability are core to how we work.

About the Role

AI agents at Honeycomb investigate, reason, and act on real observability data inside Canvas—the agentic workspace where engineers go to understand their systems.

The Agentic Intelligence team has shipped Canvas, the Honeycomb MCP server, and the Canvas Agent and Canvas Skills surfaces. We are looking for someone with deep agent expertise to expand what the team can build: new agents, new surface area in Canvas, memory, spatial awareness, and improved performance on our Bedrock loop.

Honeycomb's fast data store accepts high-cardinality data, enabling agents to reason over signals that conventional observability backends cannot serve. This role is about taking full advantage of that foundation to build production-grade agents that take work from rough prototypes all the way through to scalable production systems.

What You'll Do

• Design and deliver production-grade agents: Build agents that investigate, reason, and act on live observability data inside Canvas. Ensure agents are trustworthy for engineers in high-pressure situations, including mid-incident.
• Own agent systems & product features: Scope, build, ship, and maintain agents, including the eval frameworks that measure performance improvements. This role is agent-focused and includes fullstack development.
• Build unique agents: Leverage a high-cardinality data store to reason over signals across services, drill into single traces, and compare pre- and post-deploy states.
• Extend Canvas capabilities: Ship new capabilities into Canvas, the MCP server, and Canvas Skills (memory, spatial awareness, faster Bedrock loop) and advocate for next steps with working code.
• Set the engineering standard: Define measurable evals, maintainable agent architecture, and realistic boundaries for agent capabilities.

Example Projects

• Collaborative Agents: Multiple agents collaborating on a shared Canvas investigation, each claiming a hypothesis, publishing findings, and narrowing the search space.
• Auto-Investigation: Triggering automated investigations the moment an SLO burn alert fires, forming hypotheses and preparing visualizations before an engineer steps in.
• Domain Expertise Skills: Encoding operational knowledge (e.g., Kubernetes thresholds) into reusable skills for agents and human teammates.

What You'll Bring

• AI & Agent Engineering Experience: Proven track record of shipping production-grade LLM-based systems that users rely on (not just research demos or fine-tuned model experiments).
• End-to-End Ownership: Ability to take concepts from rough prototype to durable, production-ready engineering independently.
• Up-to-Date Technical Judgment: Informed opinions on recent shifts in agent design over the past 6–12 months and how to apply them today.
• Agent Architecture Depth: Deep understanding of how fast, high-cardinality data stores impact agent reasoning and system design.
• Product Judgment: Ability to identify what the agent layer should do next and demonstrate it via working code prototypes.

Nice to Have

• Experience in observability or developer tooling.
• Familiarity with eval frameworks, agent tooling, RAG, and prompt engineering.

Compensation & Benefits

• Salary Range: $183,340—$206,000 USD
• Equity: Generous equity package with employee-friendly stock options
• PTO: Unlimited PTO
• Perks & Stipends: Home office, co-working, and internet stipends
• Healthcare: Full benefits coverage for employees, with dependent coverage available
• Parental Leave: Up to 16 weeks of paid parental leave
• Growth: Annual professional development allowance

Note: Honeycomb cannot sponsor or support visa transfers at this time.