Senior AI Engineer - Grafana AI/ML

Grafana Labs is the company behind Grafana Cloud, the fully managed observability platform trusted by more than 10,000 organizations. We are a 100% remote company with team members across 40+ countries.

This is a remote opportunity for applicants located in Canada time zones only at this time.

The Opportunity

At Grafana, we build observability tools that help users understand, respond to, and improve their systems. The Grafana AI teams play a key role in helping users make sense of complex observability data through AI-driven features. We are looking for a Senior AI Software Engineer with a strong software engineering background, a quick iteration mindset, and a passion for experimentation.

What You’ll Be Doing

• Build and deliver AI solutions: Develop high-performance AI features to help users detect, triage, and resolve incidents using observability data and tools.
• Rapid experimentation and iteration: Quickly prototype, test, and validate with real users, including shipping and evolving LLM- or agent-powered workflows.
• Collaborate cross-functionally: Work with data analysts, product managers, and designers to shape AI-driven product features and integrations.
• Take ownership: Ensure solutions are innovative, scalable, maintainable, and aligned with real user workflows.

What Makes You a Great Fit

• Strong engineering skills with solid experience building production software systems (backend and/or full stack).
• Practical AI experience, focusing on delivering real-world solutions.
• Comfort with quick iteration, releasing prototypes, and collecting feedback.
• Strong initiative, ability to deal with ambiguity, and define scope.
• Excellent communication and collaborative attitude.

Requirements

• Experience with LLMs, prompt engineering, and building applications powered by GenAI.
• Proven track record of delivering software into production actively used by users.
• Exposure to working in cloud-native environments (e.g., AWS, GCP, Azure).
• Experience using observability tools to troubleshoot system behavior.

Bonus Points For

• Experience building or working with agent frameworks or multi-agent workflows.
• Experience with infrastructure/DevOps tooling: Kubernetes, Docker, Terraform.
• Familiarity with model fine-tuning techniques.
• Experience building observability tooling.