Staff Security Data Engineer

Role details

About the role Adobe’s Security Data Platform team builds and operates a petabyte-scale security data lakehouse that turns enterprise telemetry into trusted data for threat detection, investigations, compliance, and security analytics. As a Staff Security Data Engineer, you will own full-system designs and lead critical engineering initiatives across ingestion, storage, governance, orchestration, and data serving. This is a hands-on individual contributor role: you will write and review production code, build infrastructure, define test strategy, fix complex failures, and work with other engineers to deliver the architecture you help shape. You will partner with Security Operations, Threat Intelligence, Compliance, Security Engineering, and infrastructure teams to resolve technical tradeoffs and deliver reliable, cost-effective platform capabilities. Your impact will come through engineering delivery, sound architectural decisions, and helping others become stronger technical leaders. Help us evolve our team from petabyte scale toward exabyte scale! Core responsibilities Platform architecture and engineering Own end-to-end designs that shape the Security Data Platform’s fundamental architecture on Databricks and AWS, using Spark, Delta Lake, and Unity Catalog. Translate ambiguous needs into design proposals, implementation plans, and production capabilities. Build and evolve distributed batch and streaming pipelines for EDR telemetry, network logs, identity events, cloud audit logs, and new security data sources. Design for throughput, fault tolerance, replay, schema evolution, and sub-minute freshness where use cases require it. Write maintainable Python and SQL, develop complex platform features, and refactor unreliable code. Establish reusable engineering patterns, lead code and design reviews, and own test strategy and automation choices across pipelines and infrastructure. Build and maintain tested, reusable Terraform modules for AWS and Databricks infrastructure, including S3, IAM, VPC networking, and Amazon Managed Workflows for Apache Airflow (MWAA). Automate delivery through version control and CI/CD. Data modeling governance and operations Design layered lakehouse data models and reusable normalization patterns, including Open Cybersecurity Schema Framework (OCSF) mappings where appropriate. Preserve source fidelity and make datasets useful for investigations, analytics, and detection workloads. Establish data contracts, quality checks, cataloging, classification, lineage, access controls, and retention policies with data owners. Use Unity Catalog to make data ownership and governance traceable from raw events to analyst-facing datasets. Engineer and operate Airflow and MWAA orchestration, including dependencies, retries, backfills, alerting, and automated recovery. Define service level objectives with consumers and instrument data freshness, completeness, pipeline health, and query performance. Lead diagnosis and resolution of complex production issues across Spark, storage, orchestration, and cloud infrastructure. Improve incident response, runbooks, recovery testing, and operational readiness to prevent recurring failures. Performance and technical leadership Profile workloads and lead measurable cost and performance improvements through query and Spark tuning, storage lifecycle policies, partitioning, and compute right-sizing. Establish baselines and dashboards that show savings alongside reliability, latency, and capacity. Lead cross-team technical discussions, reconcile competing requirements, and build agreement on architecture and delivery priorities without formal management authority. Communicate decisions, landmarks, risks, and business impact proactively to your manager and partners. Partner with detection and security analytics engineers to deliver the data interfaces, governance, and pipeline capabilities needed for SIEM modernization and automated detection deployment. Mentor engineers through pairing, technical reviews, documentation, and knowledge sharing. Help others lead technical work, promote inclusive debate, and contribute to hiring decisions through technical assessment. Evaluate and introduce useful platform innovations, including AI-assisted engineering and operational automation. Validate outputs, protect sensitive security data, measure improvements, and help partner teams adopt proven approaches. Required qualifications Substantial experience building and operating production data platforms, typically 10 or more years in data or software engineering, with demonstrated Staff-level scope and impact. Equivalent depth and outcomes are welcome; a prior Staff or Principal title is not required. A record of personally delivering complex distributed systems and leading full-system design and implementation across team boundaries. Ability to turn general, unstructured assignments into clear technical approaches and working software. Deep production expertise in Spark's traditional and real-time data workloads, Databricks, Delta Lake, and lakehouse architecture; strong Python and advanced SQL skills for data modeling, profiling, and query optimization. Strong AWS experience across storage, compute, networking, and IAM, plus production Terraform experience designing, testing, and maintaining reusable infrastructure modules. Practical understanding of distributed systems, including partitioning, replication, consistency, back pressure, scalability, and fault tolerance, with experience diagnosing failures across system boundaries. Experience with Airflow, MWAA, or equivalent orchestration, and with production observability, data quality, automated testing, CI/CD, and reliable deployment and recovery practices. Demonstrated ability to improve platform cost, performance, and reliability with measurable results; sound judgment in balancing these goals against security and consumer requirements. Effective technical communication and multi-functional influence, inclu