Staff Software Engineer - OpenTelemetry Python Instrumentation

Grafana Labs is the company behind Grafana Cloud, a fully managed observability platform trusted by over 10,000 organizations. Built on open source and open standards, Grafana Cloud integrates AI into observability, providing unified visibility for teams across disparate data. We are a 100% remote company with team members in 40+ countries.

We are scaling fast, driven by our open-source legacy, global collaborative culture, and passion for meaningful work. Our team thrives in an innovation-driven environment where transparency, autonomy, and trust are core values.

Grafana Cloud is our composable observability platform that integrates metrics, logs, and traces with Grafana, allowing customers to leverage the best open-source observability software (Prometheus, Mimir, Loki, Tempo) without the overhead of self-managing their stack.

The Observability department focuses on enabling developers to understand application and infrastructure health and performance by providing tools for instrumentation, data ingestion into Grafana Cloud, and visualization.

The OpenTelemetry SDKs team ensures users succeed in instrumenting their software with OpenTelemetry. We collaborate closely with OpenTelemetry SIGs to advance SDKs and instrumentation libraries. Team members are experts in at least one programming language with broad knowledge of others.

In this role, you will focus on the Python implementation of OpenTelemetry, with a particular emphasis on OpenTelemetry for Generative AI. You will adopt an upstream-first approach, accelerating the OpenTelemetry project while bringing innovations to Grafana Labs products.

As a remote-first and global company, we embrace diverse experiences and backgrounds. We seek Engineers passionate about communicating with data and providing seamless customer experiences. Engineers at Grafana also have opportunities to contribute to Open Source communities.

At Grafana Labs, Individual Contributors (ICs) have a dedicated career path without needing to become managers to progress. Staff Software Engineers possess extensive experience across multiple areas, capable of estimating, planning, coordinating, and delivering large, multi-system tasks. They actively coach and mentor team members and resolve technology and product process issues.

What You’ll Be Doing

This is a hands-on role for someone with deep Python expertise and an interest in AI. You will apply your software engineering expertise to develop improvements to the OpenTelemetry Python SDK and instrumentation libraries, with a strong focus on AI applications and agent workflows.

• Collaborate in open-source communities: Develop and maintain the OpenTelemetry Python SDK and instrumentation libraries, working with maintainers and other vendors on specifications, APIs, and implementation decisions.
• Build Python instrumentation for AI agents: Create instrumentation that connects agent runs, model calls, and tool execution, including across asynchronous code. Contribute to evolving OpenTelemetry conventions for generative AI, considering sensitive data and metric cardinality.
• Improve Python SDK performance: Use profiling and benchmarks to evaluate implementation approaches with the upstream community. Make sound tradeoffs around runtime overhead, compatibility, and maintainability.
• Be visible as an OpenTelemetry expert: Teach others, promote best practices around OpenTelemetry, semantic conventions, and various technologies and frameworks, both within Grafana Labs and the broader OpenTelemetry community.
• Support technical direction and vision: Contribute to business conversations on the OpenTelemetry strategy at Grafana Labs. Own projects from ambiguous problem definition through design, implementation, testing, documentation, and release.
• Partner in an open and respectful way: Lend your knowledge and skills to our GTM, Product, and Support teams. Join customer conversations, investigate instrumentation issues, and translate learnings into fixes and practical guidance.

As a remote-first engineering organization, we provide guidance and meet regularly via video calls, so high agency and strong communication skills are essential.

What Makes You a Great Fit

Staff Engineers at Grafana are defined by their impact and their ability to enable others. You take ownership of complex problems, set technical direction, and help others deliver improvements that matter to customers and the OpenTelemetry community.

• 8+ years of experience in software development, with strong proficiency in Python and experience debugging production software, language internals, libraries, and asynchronous code.
• An interest in AI applications and agent systems, and curiosity about how to instrument and troubleshoot them. Prior experience building AI systems is helpful, but not required.
• Experience working in open-source communities. You are eager to become an active member of the OpenTelemetry community, collaborate with maintainers and users, and continuously learn across languages, frameworks, and the wider observability ecosystem.
• Familiarity with observability tooling such as Grafana, Prometheus, and OpenTelemetry, alongside experience with cloud-based software or systems engineering. You understand production environments and can assess operational risks.

Bonus Points For

• Experience working on open-source observability projects, ideally in OpenTelemetry or Prometheus.
• Experience in Python instrumentation techniques, profiling, runtime diagnostics, or asynchronous context propagation.
• Experience in a hands-on technical leadership role, setting technical direction and leading project teams.
• Experience instrumenting AI frameworks or agent workflows, or working with OpenTelemetry semantic conventions for generative AI.
• Experience contributing to or maintaining Open Source projects, with evidence of successful pull requests and community collaboration.
• Rust experience is a plus for exploring potential Python SDK performance improvements, including Rust-backed implementations.

Compensation & Rewards

In the UK, the compensation range for this role is GBP 107,000 - GBP 134,000. Actual compensation may vary based on level, experience, and skillset. All roles include Restricted Stock Units (RSUs), providing ownership in Grafana Labs' success.

Why You’ll Thrive at Grafana Labs

• 100% Remote, Global Culture: Talent from around the world, united by collaboration and shared purpose.
• Scaling Organization: Tackle meaningful work in a high-growth, evolving environment.
• Transparent Communication: Open decision-making and regular company-wide updates.
• Innovation-Driven: Autonomy and support to ship great work and try new things.
• Open Source Roots: Built on community-driven values.
• Empowered Teams: High trust, low ego culture that values outcomes.
• Career Growth Pathways: Defined opportunities to grow and develop your career.
• Approachable Leadership: Transparent, involved, visible, and human executives.
• Passionate People: Join a team of smart, supportive individuals who care deeply about their work.
• In-Person Onboarding: Thrive from day one with fellow new 'Grafanistas' to learn about our work and culture.
• Balance is Key: Global annual leave policy of 30 days per annum, including 3 Grafana Shutdown Days for team disconnection.