Senior Software Engineer II - LLM Observability

Honeycomb is defining observability and raising expectations for developer tools. We work with well-known companies like HelloFresh, Slack, LaunchDarkly, and Vanguard. We've secured Series D funding, scaled past 200 people, and were named to Forbes’ America’s Best Startups of 2022 and 2023. We are a talented, passionate, fiercely inclusive, and responsible group, committed to our values and fostering a culture of impact and growth.

We are a fully distributed company, emphasizing delivery over location. We invest in our people, offering trust, autonomy, and accountability from day one.

About the LLM Observability Team

The LLM Observability (LLM O11y) team builds the foundational layer for making AI workloads observable in Honeycomb. We define how AI systems are represented in telemetry (spans, traces, token usage, cost, eval signals, etc.) and ensure data from diverse ecosystems (e.g., LangChain, OpenLLMetry, Braintrust) is normalized into a consistent OpenTelemetry-based model. This team owns the infrastructure and data model that all downstream AI observability experiences rely on, powering everything from cost analysis to anomaly detection. If you're excited about defining and building the "source of truth" for how AI systems behave in production, this is the team for you.

What You'll Do

• Design, build, and deliver backend systems and APIs: Architect and implement APIs and backend services in Go, focusing on clarity, maintainability, and backward compatibility. Proactively consider cost, scale, and risk, anticipating technical issues and exploring alternatives.
• Build and maintain full-stack product features: Contribute frontend (React/TypeScript) and backend (Go) code to build robust product features, maintaining proficiency across the stack. Deliver impactful features for customers and the company.
• Provide technical leadership: Define patterns and operational guidelines. Plan, lead, and deliver complex, multi-team projects, including those with external dependencies, ensuring alignment and mentoring teammates. Tech lead responsibilities rotate to foster skill growth.
• Collaborate across disciplines: Work closely within a product triad model (design, product, engineering) from discovery through delivery, iterating quickly based on user feedback. Contribute to technical and quarterly product roadmaps, tying technical considerations to business outcomes.
• Support and own your software in production: Participate in on-call rotations to ensure reliable operations and proactively address user pain points. Balance forward-looking feature work with production needs.
• Communicate clearly and multiply your impact: Utilize regular written and verbal communication, collaboration, and feedback. Mentor engineers, share knowledge, and delegate to help others grow, aiming to make everyone more effective.

What You'll Bring

• Deep experience designing, building, and maintaining backend systems: Substantial expertise building APIs and backend services in Go (or similar languages), with strong opinions on API design, error handling, and developer experience. Ability to evaluate new technical challenges.
• Frontend proficiency with React and TypeScript: Professional experience delivering frontend features and contributing to full-stack software through all development lifecycle stages, with primary expertise in backend systems.
• Experience leading complex technical projects: Planned and delivered projects spanning multiple teams or systems, navigating external dependencies and uncertainty. Ability to break down complex work, ship incrementally, and balance technical investment with business goals.
• Ownership and autonomy: Proactively identify problems, propose solutions, and own projects end-to-end (planning through deployment and maintenance). Tackle tech debt and debug tough issues across the stack.
• A track record of making others better: Mentored engineers, led technical design reviews, and raised the quality bar for a team or domain. Sought out for guidance and helps build leaders.
• A collaborative and flexible mindset: Comfortable working cross-functionally, pivoting priorities to address evolving customer or business needs, and maintaining strong teamwork.
• Clear, open communication: Views clear, thoughtful, and regular written and face-to-face communication (including giving/receiving feedback) as a fundamental engineering skill and strong predictor of impact.
• An interest in observability and production culture: Supported software in production and has opinions on effective practices. Bonus points for wanting to integrate this experience into the Honeycomb product.

Nice to Have

• Experience with AI-assisted development: Proficient with AI coding tools (Copilot, Claude Code, Cursor, etc.) or excited to incorporate them into your workflow.

Compensation & Benefits

• Base Salary: $250,000—$280,000 CAD
• Equity: Generous equity with an employee-friendly stock program.
• Transparent Pay: Pay is based on transparent levels relative to experience.
• Time Off: Unlimited PTO.
• Remote Work Support: Home office, co-working, and internet stipend.
• Parental Leave: Up to 16 weeks of paid parental leave.
• Professional Development: Annual development allowance.

Important Notes

• We cannot currently sponsor or support visa transfers.
• We are committed to building a diverse, inclusive, and equitable workplace. Reasonable accommodations are available during the hiring process.