Staff Software Engineer, Data & Scalability Platform

Role details

The application window is expected to close on: 10/31/2026 This is a Hybrid position requiring Tuesdays and Thursdays on-site. In CA, the on-site requirement is Tuesdays in San Francisco and Thursdays in San Jose, CA; in NYC, Tuesdays and Thursdays are on-site in our NYC office. The Splunk Agent Resilience team is defining the future of AI resilience. Our team provides scalable, cost-effective evaluation and guardrails that ensure AI agents behave as intended, improving reliability and reducing risks. This unified approach empowers our customers to confidently deploy and run AI-powered applications with enhanced observability and control. As a Staff Software Engineer on the Data & Scalability Platform team, you will provide technical leadership for the architecture, scalability, and cost of the entire data plane: the stores that hold the data, the streaming and compute components that move and transform it, and tooling proving how all of it behaves under load. You will define the multi-quarter, technical direction for that data plane, establish the standards other teams build against, own the initiatives that keep it fast and correct as it grows by orders of magnitude, and influence the technical direction of the platform with engineering team leadership and product teams. Your Impact Define and drive the multi-quarter technical roadmap for the data platform — storage, streaming, compute, and the tooling that measures them. Lead the architecture and evolution of the data plane, including storage engine selection, sharding and tenancy models, and control/data plane boundaries. Establish platform-wide standards for schema design, migration safety, retention of data, and lead the review process that enforces them. Lead the capacity and performance model, and define the SLOs that set the definition of what correct and fast mean. Lead major scalability and performance initiatives spanning data stores, streaming pipelines, and distributed worker fleets. Drive cost efficiency, the cost per unit of telemetry, cost attribution, and quota and rate-limit design. Set the direction for multi-tenant isolation so that no tenant can degrade another. Design and build the internal frameworks and performance tooling — load testing, profiling, and benchmarking — that other engineers rely on to support design and capacity decisions. Establish observability standards for the platform so signals are correct, complete, and affordable. Lead the most production investigations and drive systemic remediation rather than point fixes. Make and document technology-selection and build-versus-buy decisions, with explicit cost, risk, and migration analysis. Partner with engineering leadership to influence platform direction and investment, and represent the data platform to the product teams building on it. Mentor engineers and raise the engineering bar through design reviews, technical standards, and documentation. Minimum Qualifications Bachelor's degree with 8+ years of related experience, or Master's degree with 6+ years, or PhD with 3+ years, in Computer Science, Software Engineering, or a related field — including at least 6 years building and/or operating data-intensive backend services and/or platforms. Technical leadership experience, including experience setting technical direction, driving multi-quarter initiatives, and/or influencing decisions beyond your immediate scope. Experience with designing, operating, and/or delivering highly scalable, reliable, production-grade data-intensive platforms. Experience designing large-scale distributed systems and solving problems of scalability, availability, performance, reliability, and/or fault tolerance. Backend software engineering experience with either Go, Java, Rust, C++, and/or Python. Experience with cloud-native technologies, containerized environments (Kubernetes), and public cloud platforms (AWS, GCP, or similar) Preferred Qualifications Demonstrated performance engineering ability — profiling, benchmarking, and load testing real systems, and turning the measurements into capacity, design, and cost decisions. Experience operating and tuning both an OLTP database (e.g., PostgreSQL, MySQL) and an analytical, columnar, or time-series store (e.g., ClickHouse, Druid, BigQuery, Snowflake) at production grade at scale — schema design, indexing, query tuning, and migration safety. Deep production experience with streaming or queueing systems (e.g., Kafka, RabbitMQ, Pulsar, Kinesis) — partitioning, consumer-group semantics, delivery guarantees, and backlog and dead-letter handling. Experience scaling data-processing compute — queue consumers, async task workers, or streaming jobs — including concurrency tuning, batching, backpressure, and throughput behavior under sustained load. Deep experience running ClickHouse or a comparable columnar store at scale — sharding and replication topology, materialized views, merge and mutation behavior, retention and TTL mechanics. Experience with a capacity model or a FinOps practice at platform scope: cost per.unit of telemetry, cardinality governance, and quota or rate-limit design. Experience designing multi-tenant isolation and noisy-neighbor mitigation in shared data systems. Experience leading a significant data migration or storage-engine transition in production without customer-visible impact. Experience defining SLOs for data services and driving an organization to meet them. Experience with backup and restore, tested disaster-recovery drills against stated RPO/RTO, and data retention or deletion compliance (e.g., GDPR/DSR hard delete). Experience building observability instrumentation and pipelines (OpenTelemetry, collectors, metrics/tracing backends) rather than only consuming dashboards. Experience building or operating performance tooling as a product for other engineers — the load-test harnesses, continuous profiling and flame-graph workflows (e.g., Pyroscope), or benchmarking suites. Experience with distributed task