Senior Infrastructure/Site Reliability Engineer On-Call

Role details

ABOUT THE ROLE This is a senior on-call SRE role at an early-stage AI infrastructure company, where you will be the technical expert enterprise clients depend on when critical systems fail. You will own incident response across Kubernetes clusters, Ceph storage, and bare metal servers, keeping high-value AI workloads running at all times. Your calm judgment and deep distributed systems expertise will have a direct and immediate impact on client operations. WHAT YOU'LL DO - Respond to and resolve production incidents across client infrastructure spanning Kubernetes, Ceph, and bare metal environments. - Troubleshoot complex distributed systems problems including pod scheduling failures, CNI networking issues, storage performance degradation, and hardware faults. - Handle escalations requiring deep expertise in etcd clusters, Ceph RGW authentication, Cilium networking, and bare metal load balancers. - Communicate directly with enterprise clients during incidents, providing clear status updates and resolution timelines. - Participate in a follow-the-sun on-call rotation with engineers across multiple time zones. - Document incidents thoroughly and improve runbooks based on recurring patterns. - Collaborate with the infrastructure team on long-term reliability improvements and automation to reduce incident frequency. WHAT WE'RE LOOKING FOR - 5 or more years of production experience with Kubernetes in enterprise environments, including cluster operations, bare metal troubleshooting, admission controllers, and control plane architecture. - Deep production experience with distributed storage systems, particularly Ceph, or equivalent platforms such as Weka or VAST. - Production experience with at least one CNI plugin, preferably Cilium or Calico. - Strong modern Linux systems administration skills and comfort with bare metal infrastructure, IPMI, hardware troubleshooting, and networking. - Demonstrated ability to systematically diagnose and resolve complex distributed systems issues under pressure during live outages. - Production experience with etcd cluster management, including backup and restore procedures. - Experience with GPU infrastructure for AI/ML workloads, including the NVIDIA Kubernetes operator. - Familiarity with infrastructure-as-code tools such as Ansible, Kubespray, or similar orchestration frameworks. - Clear, composed communication with both technical and non-technical audiences during incidents. - Background in AI inference or training infrastructure is a strong plus. LOCATION On-site in San Francisco, CA. This role involves a follow-the-sun on-call rotation and requires timezone flexibility. Visa sponsorship is not available.