Staff AI Engineer - Grafana AI/ML

The Opportunity

At Grafana, we build observability tools that help users understand, respond to, and improve their systems. The Grafana AI teams play a key role in this mission by helping users make sense of complex observability data through AI-driven features that reduce toil and surface meaningful signals.

We are looking for an AI Software Engineer with a strong software engineering background, a quick iteration mindset, and a passion for experimentation to help us ship and scale impactful features.

What You’ll Be Doing

• Build and deliver AI solutions: Develop high-performance AI features to help users detect, triage, and resolve incidents.
• Rapid experimentation: Prototype, test, and validate with real users, including shipping LLM- or agent-powered workflows for incident lifecycle management.
• Collaborate cross-functionally: Work with data analysts, product managers, and designers to integrate agentic components with internal tools, alerting systems, and runbooks.
• Ownership: Take full ownership of AI solutions, ensuring they are scalable, maintainable, and aligned with real user workflows.

What Makes You a Great Fit

• Strong engineering skills: Solid experience building production software systems (backend and/or full stack).
• Practical AI experience: Familiarity with AI technologies and frameworks with a focus on delivering real-world solutions.
• Quick iteration: Comfortable releasing prototypes, collecting feedback, and iterating with a pragmatic mindset.
• Proven initiative: Ability to deal with ambiguity and define scope in loosely defined projects.
• Collaborative attitude: Effective communication skills and a solutions-oriented mindset.

Requirements

• Experience with LLMs, prompt engineering, and building applications powered by GenAI.
• Proven track record of delivering production-grade software.
• Exposure to cloud-native environments (e.g., AWS, GCP, Azure).
• Experience using observability tools to troubleshoot system behavior.

Bonus Points

• Experience building agent frameworks or multi-agent workflows.
• Experience with infrastructure/DevOps tooling (Kubernetes, Docker, Terraform).
• Familiarity with model fine-tuning techniques.
• Experience building observability tooling.

Why Grafana Labs?

• 100% Remote: A global culture with team members across 40+ countries.
• Innovation-Driven: Autonomy and support to ship great work.
• Developer Productivity: Access to modern AI coding assistants and frontier models (OpenAI, Anthropic, Google) with a company-funded usage budget.
• Benefits: Competitive compensation, equity (RSUs), and a global annual leave policy of 30 days.