Staff Backend Engineer - Second Horizon

About the Role

Grafana Labs is seeking a Staff-level Backend Engineer to join our "Second Horizon" skunkworks initiative. We are building an AI-native data intelligence system designed to provide AI agents with reliable, governed access to enterprise context. You will play a pivotal role in designing and shipping the core backend architecture that powers ingestion, context storage, retrieval APIs, and agent-facing integrations.

Key Responsibilities

• Core Backend Development: Design, implement, and operate services for context ingestion, indexing, retrieval orchestration, and API access.
• Scalable SaaS Architecture: Define the architecture for a multi-tenant service, including tenant isolation, usage tracking, quotas, and audit logs.
• Agent-Facing Workflows: Build APIs and interfaces for AI agents, MCP tools, and CLIs to retrieve context, provenance, and confidence signals.
• System Reliability: Instrument services with metrics, logs, and traces. Use observability tools to ensure high availability and performance.
• Technical Leadership: Shape the architecture, storage choices, and engineering practices for this new product area.

What We’re Looking For

• Engineering Excellence: Proven track record of building production-grade, user-facing software systems.
• AI/GenAI Expertise: Experience with LLMs, prompt engineering, and building applications powered by Generative AI.
• Pragmatic Mindset: Ability to balance fast experimentation with long-term reliability and maintainability.
• Cloud-Native Experience: Proficiency working in cloud environments (AWS, GCP, or Azure).
• Ownership: A self-starter who thrives in ambiguous, early-stage environments and can drive projects from prototype to production.

Requirements

• Experience with LLMs, prompt engineering, and GenAI applications.
• Proven track record of delivering production-ready software.
• Experience with cloud-native environments and observability tools.
• Location: Must be based in the U.S. or Canada (excluding Quebec).

Why Join Us?

• 100% Remote: A global, collaborative culture with team members across 40+ countries.
• Innovation-Driven: High autonomy and trust to ship meaningful work.
• Open Source Roots: A company built on community-driven values.
• Balance: 30 days of annual leave, including company-wide shutdown days.