Staff Backend Software Engineer - Loki

About Grafana Labs

Grafana Labs is the company behind Grafana Cloud, the fully managed observability platform trusted by more than 10,000 organizations to ensure reliability, resolve incidents faster, and optimize telemetry at scale. Built on open source and open standards, Grafana Cloud integrates metrics, logs, traces, and profiles across any stack.

We are a 100% remote company with team members across 40+ countries, backed by leading investors including Lightspeed Venture Partners, Sequoia Capital, GIC, Coatue, and J.P. Morgan.

Note: This role is open to candidates located in the United States or Canada.

The Opportunity

The Databases team owns telemetry databases including Mimir (metrics), Loki (logs), Tempo (traces), and Pyroscope (profiles). Our databases are open-source projects offered both as a Cloud service and on-premise solutions. They are multi-tenant distributed systems implemented in Go and running on Kubernetes across major cloud providers (GCP, Azure, AWS).

Our tech stack consists primarily of services written in Go running on Kubernetes clusters backed by cloud object storage. We invest heavily in developer productivity, offering company-funded budgets for AI coding assistants and access to frontier models from OpenAI, Anthropic, and Google.

What You'll Be Doing

• Design, build, operate, and maintain critical distributed systems, taking ownership of reliability, performance, and availability.
• Influence technical roadmaps and shape team objectives.
• Deliver features collaboratively, iterating based on production results and customer feedback.
• Drive projects from ideation through operational production rollouts.
• Contribute to open-source software and collaborate across adjacent engineering teams.
• Participate in team on-call rotations and assume operational ownership of services.
• Mentor and support fellow engineers, conduct code reviews, and drive design discussions.

Requirements

• Solid experience with at least one programming language (Go preferred; strong experience with Python, C, C++, or Rust translates well).
• Proven track record of independently driving technical projects from requirement gathering to production delivery.
• Hands-on experience developing software running in the Cloud or systems engineering experience.
• Demonstrated ability to write clean, robust, and maintainable software for high-scale environments.

Bonus Points

• Experience writing low-level code for databases, storage layers, or query engines.
• Hands-on operational experience with Kubernetes.
• Prior usage of Grafana, Prometheus, or Loki in production or homelab settings.
• Deep understanding of microservices architecture and distributed systems.
• Experience with infrastructure-as-code and site reliability engineering (SRE) practices.

Compensation & Benefits

• Base Salary (Canada): CAD 186,500 – CAD 223,500 (US salary discussed by market)
• Equity: Company equity granted with the offer
• Annual Leave: 30 days per annum (including 3 company shutdown days)
• Developer Tooling: Dedicated budget for AI coding assistants and modern frontier models