Staff Backend Software Engineer - Loki

About Grafana Labs

Grafana Labs is the company behind Grafana Cloud, the fully managed observability platform trusted by more than 10,000 organizations to ensure reliability, resolve incidents faster, and optimize telemetry at scale. Built on open source and open standards and designed for interoperability across any stack, Grafana Cloud brings AI to observability and observability to AI.

We are a 100% remote company with team members across 40+ countries. This is a remote position open to candidates based in the US and Canada.

About Grafana Cloud & The Team

Grafana Cloud is our composable observability platform that integrates metrics, logs, traces, and profiles with Grafana. It allows customers to leverage open-source software—including Prometheus, Mimir, Loki, Tempo, and Pyroscope—without the overhead of maintaining their own observability stack.

The Databases team owns and operates telemetry databases including Mimir (metrics), Loki (logs), Tempo (traces), and Pyroscope (profiles). These are multi-tenant distributed systems implemented in Go and operating at scale on Kubernetes across major cloud providers (AWS, GCP, Azure).

What You Will Do

• Take an active role in influencing the product roadmap and your engineering career objectives.
• Work with your squad to deliver new features and iterate continuously based on operational feedback.
• Drive projects from initial concept through design, implementation, and deployment into production.
• Contribute to open-source projects and embrace Grafana Labs' open-source culture.
• Design, build, operate, and maintain critical systems with high reliability, performance, and availability.
• Participate in follow-the-sun on-call rotations and take end-to-end ownership of live services.
• Mentor and support other team members through design reviews and PR reviews.
• Deepen your understanding of large-scale distributed systems and customer telemetry requirements.

Qualifications

• Pragmatism: Ability to tackle complex distributed systems challenges and break them down into modular solutions with short feedback loops.
• Communication: Excellent remote collaboration and written/verbal communication skills.
• Programming Experience: Strong proficiency with Go, or deep experience in languages like Python, C, C++, or Rust with the ability to transition.
• End-to-End Delivery: Track record of driving projects from requirements gathering to running in production.
• Cloud & Systems: Solid background in cloud software development or systems engineering.
• DevOps Mindset: Experience being on-call and supporting production systems using modern DevOps practices.
• Observability: Strong familiarity with observability systems (metrics, logs, traces) and how to leverage them for debugging.

Nice to Haves

• Hands-on experience working with Kubernetes in production.
• Experience with distributed queue systems (e.g., Apache Kafka protocol).
• Prior user experience with Grafana and Prometheus in operational or homelab environments.
• Background in microservices architecture and large-scale distributed systems.
• Familiarity with Infrastructure as Code (IaC) tooling.