Sr. Software Engineer - Go/MongoDB

About the Team

The MongoDB Tools Team builds Percona's open source operational tooling for MongoDB. Two projects sit at the center of what we do: Percona ClusterSync for MongoDB (PCSM), which clones and continuously replicates data between clusters, and Percona Backup for MongoDB (PBM), a distributed, low-impact backup and restore solution for replica sets and sharded clusters. Both are written in Go, Apache 2.0 licensed, and built fully in the open.

This role sits primarily on PCSM, a younger project moving fast where you will have real influence over its shape, while also working across PBM. The two tools share complex challenges including cluster topology, the oplog and change streams, consistency across shards, and performance in large production clusters.

The Projects

• PCSM (Primary Focus): Initial data cloning followed by continuous change replication over MongoDB Change Streams for replica sets and sharded clusters. Still pre-1.0 and evolving quickly.
• PBM (Secondary): Consistent backup and restore with point-in-time recovery, using oplog capture across replica sets and sharded clusters with S3-compatible and filesystem storage. Driven by pbm-agent processes on each node and a pbm CLI.

What You Will Work On

Primary: PCSM

• The Core Replication Engine: Initial collection cloning followed by continuous change capture over MongoDB Change Streams, with correct handling of resume tokens, ordering, and resumability after failures.
• Correctness and Fault Tolerance at Scale: Recovering cleanly from network drops, primary elections, and restarts without losing or duplicating changes.
• Sharded Cluster Support: Replicating across shards, handling chunk migrations, balancer activity, and keeping the target consistent.
• Namespace Filtering & Index Management: Handling edge cases with DDL, TTL, and index differences between source and target.
• Performance & Throughput: Parallelizing the clone, applying backpressure, and keeping memory and connection use optimized for large clusters.
• CLI & HTTP API: Building tools to observe sync operations, alongside metrics and logging for operators.

Secondary: PBM

• Consistent Backup & Recovery: Point-in-time recovery across replica sets and sharded clusters.
• Backup Storage: Integrating reliably with cloud object storage (S3, GCS, Azure Blob Storage) and remote filesystems.
• Agent & CLI: Developing pbm-agent and pbm CLI, and control-collection state coordination in MongoDB.

Shared Across Both

• Working in the Open: Pull requests, code review, JIRA, and community engagement.
• Release Quality: Tests, packaging, CI pipelines, and security scanning.

What You Bring

• Strong Go Experience: Production-level fluency in concurrency (goroutines, channels, context cancellation, worker pools, and backpressure) with demonstrated experience debugging complex race conditions.
• Distributed Systems Knowledge: Solid grounding in data consistency, at-least-once vs. exactly-once delivery, idempotency, ordering, and stateful process resumability.
• MongoDB Expertise: Hands-on knowledge of change streams, the oplog, resume tokens, replica sets, and sharding.
• Tooling Experience: Comfort building and operating command-line tools and HTTP APIs with metrics and structured logs.
• Quality Focus: A habit of writing rigorous tests and working across a mixed toolchain.
• Open Source Collaboration: Experience with Git pull request workflows, code reviews, and asynchronous technical communication.

Nice to Have

• Prior work on database internals, CDC pipelines, ETL, or data migration tooling.
• Prometheus-style metrics and observability familiarity.
• Experience with golangci-lint, vulnerability scanning (Trivy), and deb/rpm packaging.
• Background maintaining or contributing to open-source projects.
• Exposure to large-scale MongoDB sharded clusters.