Staff Engineer

Role details

We are seeking a Staff engineer to design, build, and operate the internal and external Observability stack for the MongoDB platform. Tens of thousands of customers depend on our Observability stack to monitor their database clusters and to generate actionable alerts to safeguard critical workloads. The Collections team is a newly formed team within MongoDB's Observability & Adoption Org focused on making telemetry onboarding and collection significantly easier across MongoDB. We own key parts of the observability collection stack, including onboarding experience, telemetry collection agents across the data and control planes, and ingestion services for metrics, logs, and traces that support both internal and customer observability in MongoDB, driving insights, recommendations, and alerting. Our mission is to reduce friction for teams implementing and iterating on Observability while partnering closely with development teams to instrument their services using shared best practices, helping define the conventions our telemetry should follow, and building collection and ingestion systems that are stable, performant, secure, well-documented, and self-service. We also work closely with the Data Pipeline and Storage & Query teams to help ensure MongoDB has a stable and performant observability stack end to end. This is an opportunity to join a team shaping how observability works across MongoDB and to have outsized impact on both the developer experience and the reliability of the platform underneath it. As MongoDB Atlas and its supporting infrastructure continue to experience rapid growth, the demand for high-cardinality observability data for internal and external use cases means we need to continually innovate and scale our systems to the next level. For example, MongoDB Observability systems need to handle 10’s of billions of metrics time series, all whilst processing petabytes of logs, traces, and events. Our stack includes VictoriaMetrics, Grafana, Splunk, Flink, WarpStream/Kafka, Java, Golang, Fluentbit and OpenTelemetry. In addition to owning critical components of our observability infrastructure, as a Staff engineer on the team, you’ll also work closely with other SWE, Product and SRE teams to promote and implement best practices in instrumenting and monitoring their services. This is a highly collaborative role, and you will get to own some of the most relied upon internal infrastructure at Mongo. Our team champions a strong culture of inclusivity, diversity, and collaboration. If you want to be a deeply technical leader on a collaborative team that applies low-level systems expertise to build the foundational infrastructure of a popular database, join us! Let’s build a faster, more reliable, and exceptionally observable database system together. We are looking to speak to candidates who are based in Dublin for our hybrid working model. Candidate Profile Solving complex problems at scale, and to high standards, excites you Minimum 10 years of experience in designing, programming, debugging, and tuning distributed and/or highly concurrent Go/Java/C/C++ mission critical software systems Experience running latency sensitive, high throughput systems Strong systems fundamentals, including multi-threaded programming, performance profiling, and expert-level programming Familiarity with database internals or building core components for data processing systems (Nice to have) Indexing or database performance tuning experience Familiarity with observability ecosystem and best practices Excellent verbal and written technical communication skills and a strong desire to collaborate with colleagues and mentor engineers Excellent time and project management skills including the ability to make realistic assessments of project cost and complexity Has a good understanding of information security management Responsibilities Help build and architect systems for the mission-critical observability platform, leading the architecture and implementation of components that drive performance, scalability, cost-efficiency, and resiliency Design and implement observability improvements that enable MongoDB engineers and customers to quickly and accurately diagnose the root cause of production issues Handle production customer escalations from Technical Support team and coach teammates to do the same Write production-ready database code, improve the existing code, and mentor their team to write higher quality code Own areas of code that the Observability Team maintains, ensuring it achieves a high standard for quality (including security, durability, availability, and performance) and maintainability Diagnose test failures, identify bugs in existing code, fix them, and prevent bugs from being introduced in new code Investigate the performance impact of code changes that may cause software performance regressions Interview candidates for advanced software engineering positions Develop and maintain expertise on cutting edge database, and observability developments from industry and academia Lead development and project management of some of the larger projects across the company Collaborate with stakeholders and engineering teams across the company to jointly work on large initiatives Advise Product Management on technical product direction, engineering complexity and inter-project dependencies Collaborate with Product Management and Engineering leadership to define product roadmaps Champion and improve unit and integration tests to demonstrate correctness Work with customers and support engineers to fix issues and become part of our on-call rotation Success Measures In the first month, you will have understood the high level architecture of MongoDB observability, and fixed a few bugs In three months, you will have contributed to the development of a project slated for the next major release of MongoDB, and diagnosed and fixed a few customer or testing-reported issues In six months, you will have taken on code re