Staff Backend Engineer - Grafana Second Horizon

The Opportunity

At Grafana Labs, we build observability tools that help users understand, respond to, and improve their systems. We are launching a skunkworks initiative to build an AI-native data intelligence system that provides agents with reliable, governed access to enterprise context. As a Staff Backend Engineer, you will design and ship the core architecture for this system, moving from internal dogfooding to production-grade SaaS.

What You'll Be Doing

• Core Backend Development: Design, implement, and operate services for context ingestion, indexing, retrieval orchestration, and API access.
• Scalable SaaS Foundation: Build multi-tenant architecture, including tenant isolation, usage tracking, quotas, and audit logs.
• Agent-Facing Workflows: Develop APIs and interfaces for AI agents, MCP tools, and CLIs to retrieve context, provenance, and confidence signals.
• System Reliability: Instrument services with metrics, logs, and traces; own the reliability and performance of the systems you build.
• Technical Leadership: Shape the architecture, storage choices, and engineering practices for this new product area.

What Makes You a Great Fit

• Strong experience building production-grade, user-facing software systems.
• Practical experience with AI technologies, LLMs, prompt engineering, and GenAI applications.
• Ability to work through ambiguity and make pragmatic architectural tradeoffs.
• Experience with cloud-native environments (AWS, GCP, or Azure).
• Proficiency in using observability tools to troubleshoot and optimize system behavior.
• A collaborative, solutions-oriented mindset with a track record of delivering software to production.

Bonus Points

• Experience building agent frameworks or multi-agent workflows.
• Background in data engineering or working with data platforms (e.g., Snowflake, Databricks, Looker).

Why Grafana Labs?

• 100% Remote, Global Culture: A distributed team across 40+ countries.
• Innovation-Driven: High autonomy and trust to ship meaningful work.
• Open Source Roots: A culture built on community-driven values.
• Balance: Global annual leave policy of 30 days, including company shutdown days.
• Growth: Defined career pathways and approachable leadership.