About the company
Datadog is the leading observability and security platform for the AI era, providing unified visibility across the technology stack. Trusted globally by Fortune 500 companies and high-growth AI leaders.
Responsibilities
- Build zonal-resilience automation that coordinates safe workload evacuations, switchovers, and recovery.
- Design and build fault-injection systems for production environments, including infrastructure- and application-level testing, incident replay, and controlled resilience experiments.
- Develop safeguards such as blast-radius controls, kill switches, validation mechanisms, and rollback paths.
- Build agents and automation that propose failure scenarios, triage experiment results, and connect reliability findings to tracked remediation.
- Lead gamedays from hypothesis and scenario design through execution, documented findings, remediation tracking, and validation.
- Design and implement reliable distributed systems, including gRPC services, Kubernetes controllers, and shared platform components.
- Contribute to technical design and mentor other engineers.
Requirements
- Strong distributed systems fundamentals: consistency, failure modes, backpressure, idempotency, quorum, retries, failure recovery.
- Understanding of Kubernetes workload lifecycles: pods, controllers, scheduling, draining, eviction.
- Experience designing, building, or operating production systems with safety, availability, and controlled failure handling.
- Clear communication of complex technical decisions through design documents, runbooks, postmortems.
- Comfortable collaborating across engineering teams.
- Beneficial: reliability engineering, chaos engineering, zonal failover, AI-assisted operational workflows, traffic interception, large-scale observability systems.
Conditions
- Hybrid workplace.
- Benefits and growth opportunities may vary based on country of employment.