Staff Backend Engineer - Grafana Second Horizon

Grafana Labs is the company behind Grafana Cloud, the fully managed observability platform trusted by more than 10,000 organizations to ensure reliability, resolve incidents faster, and optimize telemetry at scale. Built on open source and open standards and designed for interoperability across any stack, Grafana Cloud brings AI to observability and observability to AI, giving teams unified visibility so they can see, understand, and act on all their disparate data.

We are looking for a Staff-level Backend Engineer to join a skunkworks initiative bringing observability to the rest of the business. As part of this team, you will build an AI-native data intelligence system that gives agents reliable, governed access to enterprise context.

What You'll Be Doing

• Build core backend services: Design, implement, test, and operate the first services for context ingestion, context indexing, retrieval orchestration, API access, source configuration, and system administration.
• Create a scalable SaaS foundation: Help define and build the architecture for a multi-tenant service, including tenant isolation, usage tracking, quotas, audit logs, background jobs, and reliable service boundaries.
• Power agent-facing retrieval workflows: Build APIs and service interfaces that allow AI agents, MCP tools, CLIs, and internal applications to retrieve relevant context, provenance, confidence signals, and warnings.
• Work across product and infrastructure: Partner with the team to make practical tradeoffs between fast experimentation and long-term reliability as the project moves from prototype to production.
• Operate what you build: Instrument services with metrics, logs, traces, alerts, and dashboards using observability tools to understand system behavior and improve reliability.
• Contribute to technical direction: Shape the architecture, service boundaries, storage choices, API contracts, deployment patterns, and engineering practices for a new product area.
• Ownership and take charge: Take full ownership of the AI solutions you develop, ensuring they are scalable, maintainable, and aligned with real user workflows.

What Makes You a Great Fit

• Strong engineering skills: Solid experience building production-grade, user-facing software systems with minimal supervision.
• AI experience with a practical mindset: Familiar with AI technologies, frameworks, and GenAI applications (LLMs, prompt engineering) focused on delivering real-world value.
• Quick iteration & experimentation: Comfortable releasing prototypes, collecting feedback, and iterating with a pragmatic mindset.
• Proven initiative: Ability to deal with ambiguity, define scope, and drive impactful projects forward.
• Collaborative attitude: Effective communication skills with peers, open to feedback, and bringing a solutions-oriented mindset.
• Cloud-native experience: Exposure to cloud-native environments (e.g., AWS, GCP, Azure) and production observability tools.

Bonus Points For

• Experience building or working with agent frameworks or multi-agent workflows.
• Experience as a data analyst or working with data platforms (e.g., Looker, Tableau, PowerBI, Snowflake, DataBricks).
• Experience building tools for data engineering.