SRE Interview Questions

I’d start with the failure mode and ask, what is actually limiting us: design, human coordination, or headroom? Then I’d use data, not instinct, to pick the lever.

• More capacity if the issue is saturation, like CPU, memory, connections, queue depth, or traffic spikes pushing us past safe operating margins.
• More engineering if the system is fragile by design, like noisy retries, bad failover, missing backpressure, poor isolation, or no graceful degradation.
• More process if incidents keep happening because handoffs, ownership, change control, or response patterns are inconsistent.
• I look at frequency, blast radius, cost, and time to mitigate. Cheap, fast process fixes can buy time, but repeated incidents usually justify engineering.
• My rule is, if adding capacity only postpones the same failure, or process relies on heroics, fix the system.

I’d answer this with a quick STAR structure: situation, tension, action, result, then call out the tradeoff explicitly.

At a previous team, product wanted a major checkout change before a holiday campaign, but our error budget was already burning fast and the service had a history of latency spikes under peak load. I pushed back on a full release and proposed a staged rollout instead. We cut two non-critical features, added a feature flag, tightened SLO alerts, and ran a smaller load test focused on the riskiest path rather than trying to test everything.

The tradeoff was speed versus blast radius. We shipped the core revenue-impacting path on time, delayed nice-to-have functionality by one sprint, and limited exposure to 5 percent, then 25 percent, then full traffic. Result: no major incident during the campaign, conversion improved, and we paid back the reliability work right after launch.

I’ve mainly used SLIs around availability, latency, error rate, and data freshness, depending on the service. For APIs, a common set was successful request rate, like non-5xx responses, plus p95 or p99 latency. For async pipelines, we tracked end-to-end completion time, backlog age, and freshness of downstream data. The key is choosing indicators that reflect user experience, not just system internals.

SLOs and error budgets drove prioritization. One example, we had a 99.9% monthly availability SLO on a customer-facing API. When we burned too much error budget early in the month, we paused a risky rollout and shifted the team toward reliability work, better canaries, tighter timeouts, and fixing retry storms. If budget was healthy, we moved faster on features. That made the tradeoff between delivery speed and reliability very explicit.

I’d answer this with a tight STAR flow, then make sure each SRE skill shows up: detection, triage, mitigation, communication, and learning.

At my last team, latency spiked on a checkout API during peak traffic. We detected it from an SLO burn-rate alert, then I checked dashboards for error rate, saturation, and deploy history. Triage showed a recent config change had overloaded a Redis-backed dependency. I mitigated by rolling back the config, shedding non-critical traffic, and scaling the cache tier to stabilize the service. While doing that, I opened an incident channel, assigned roles, and posted updates every 15 minutes to engineering and support so customer-facing teams had clear status.

After recovery, I led the postmortem. We added a canary for config changes, better Redis saturation alerts, and a runbook with rollback steps. That cut time to mitigate on similar issues later.

I think in the "golden signals plus context" model: metrics tell me something is wrong, logs explain what happened, traces show where it happened.

• Metrics: latency, traffic, errors, saturation. I also want queue depth, retry rate, timeout rate, GC or heap, CPU, memory, disk, network, and dependency health.
• Business metrics matter too, things like request success by tenant, orders processed, or messages consumed, because systems can look healthy while users are failing.
• Logs should be structured, correlated with request IDs, and include key fields like service, instance, user or tenant, error code, downstream target, and timing.
• Traces are essential for request flow across services. I care about span duration, error tags, retries, fan-out, and where time is spent.
• The key is correlation, every metric spike should be traceable to related logs and spans fast.

I’d treat this like a low-signal, high-cardinality incident: stabilize first, then narrow by correlation.

• Confirm the symptom with SLIs, break errors down by region, AZ, host, endpoint, client version, protocol, and dependency.
• Look for a common thread in the failing slice, like one shard, one LB pool, one deployment ring, one DNS resolver path, or one backend replica.
• Compare good vs bad requests using tracing and structured logs, especially latency hops, response codes, retries, and timeout boundaries.
• Check recent changes everywhere, app deploys, config, feature flags, cert rotation, network policy, autoscaling, and regional failover behavior.
• Validate infrastructure edges, load balancer health checks, partial packet loss, NAT exhaustion, TLS handshake failures, and DNS skew.
• Mitigate fast, drain suspect nodes, disable a flag, pin traffic away from a bad pool, or widen timeouts if safe.
• Afterward, add per-dimension alerting and synthetic probes to catch the pattern earlier.

I start with business impact, not technology. RTO is how long the service can be down, RPO is how much data loss is acceptable. You get both by talking to product, finance, compliance, and ops about revenue loss, customer impact, legal risk, and manual workaround options.

• Tier services by criticality, for example payments vs internal reporting.
• Quantify downtime cost and data loss cost, then set realistic targets.
• Validate with failure scenarios, region loss, database corruption, bad deploy, dependency outage.
• Match architecture to targets. Low RTO often means active-active or hot standby, automation, fast failover.
• Low RPO drives synchronous replication, frequent snapshots, WAL shipping, or multi-region writes.
• Tighter objectives increase cost and complexity, so I push for the cheapest design that still meets the business need.

Example, if checkout needs 15 minute RTO and near-zero RPO, I would not rely on nightly backups and manual restore.

I start with business goals, not architecture. Multi-region only makes sense if you truly need higher availability across regional failures, lower latency for distinct geographies, or data residency compliance. Then I define RTO, RPO, traffic patterns, and whether the system can tolerate active-active complexity or should stay active-passive.

• Split the problem: stateless services are easy, stateful systems are where the pain is.
• Be explicit about data strategy, async replication, conflict handling, read locality, failover mechanics.
• Assume region loss and test DNS, load balancer behavior, dependency failover, and operational runbooks.
• Keep blast radius small, often multi-AZ plus strong backup and restore is enough.
• It is not worth it when the app is internal, latency gains are negligible, regional outage cost is low, or the team cannot safely operate the extra complexity.

I have seen teams build active-active too early, then spend more time debugging replication edge cases than delivering reliability.

I define SRE as applying software engineering to operations so systems stay reliable, scalable, and cost efficient. The core idea is to treat reliability as a measurable product feature, usually through SLIs, SLOs, and error budgets, then automate away repetitive ops work.

• Traditional ops is often ticket driven and reactive, focused on keeping servers up.
• SRE is more engineering heavy, focused on automation, observability, incident response, capacity, and reducing toil.
• DevOps is more of a culture or operating model, breaking silos between dev and ops.
• SRE is a concrete implementation of that mindset, with specific practices, guardrails, and reliability targets.
• In practice, an SRE team partners with developers, sets reliability goals, builds platforms and automation, and uses error budgets to balance feature velocity against stability.

I’ve worked a lot with Kubernetes in production, plus some ECS and Nomad. Most of my hands-on work has been around running stateless APIs, background workers, ingress, autoscaling, secrets, and observability. I’m comfortable with day-2 ops, not just deployments, things like node upgrades, capacity issues, incident response, and tuning probes and resource requests so the platform stays stable.

Common failure modes I’ve seen: - Bad liveness or readiness probes causing restart loops or blackholing traffic. - CPU or memory requests set wrong, leading to noisy neighbors or OOMKills. - DNS issues, especially CoreDNS saturation, causing random app timeouts. - CNI or kube-proxy problems breaking pod-to-pod or service networking. - Control plane degradation, API server latency, etcd pressure, stuck reconciliations. - Autoscaler lag during traffic spikes, pods pending because of quota or fragmentation. - Misconfigured PDBs or rolling updates causing accidental capacity drops during deploys.

I measure toil by looking for work that is manual, repetitive, low judgment, and scales linearly with headcount. I usually track it with a few simple signals:

• Time spent per week on recurring ops tasks, from tickets, calendars, and on-call notes.
• Frequency and variance, like how often the task happens and whether it spikes during incidents.
• Error rate and customer impact, since high-volume manual work often creates avoidable mistakes.
• Cognitive load, context switching, and whether it pulls engineers away from project work.
• Automation ROI, estimated as hours saved, reliability gained, and risk reduced.

I automate the obvious wins first: noisy alerts, repetitive remediation steps, common service requests, flaky deploy tasks, and incident data gathering. My rule is high frequency plus low judgment plus high pain. If a task happens often, burns time, and follows a runbook, it should probably become software.

A healthy on-call culture is sustainable, blameless, and boring most nights. It means alerts are actionable, rotations are fair, people have clear runbooks, and incidents drive learning instead of finger-pointing. I’d answer this with a quick framework: what healthy looks like, what was broken, what I changed, and the outcome.

At one company, on-call was noisy and uneven, with too many low-value pages and the same few people carrying the load. I helped tighten alerting by removing non-actionable pages, adding severity levels, and turning repeated fixes into automation and better runbooks. We also started lightweight postmortems focused on system gaps, not blame. Over a couple of quarters, alert volume dropped a lot, escalations went down, and engineers felt more comfortable taking the rotation because it was predictable and supported.

Cascading failures usually start when one dependency gets slow, not fully down. That drives retries, queue growth, thread pool exhaustion, connection saturation, and then healthy services start failing too. Other common triggers are shared dependencies like databases or caches, bad timeout settings, correlated deployments, and load spikes that push systems past capacity.

To prevent them, I design for isolation and controlled degradation: - Set strict timeouts, bounded retries, and exponential backoff with jitter - Use circuit breakers, bulkheads, and separate pools for critical paths - Make overload visible with queue depth, saturation, and tail latency alerts - Shed load early, rate limit, and return partial or cached responses - Remove single shared choke points, or at least scale and protect them hard - Test failure modes with chaos drills, dependency blackholes, and traffic spikes

The key mindset is, assume partial failure is normal and stop one bad component from consuming everyone else’s resources.

I’d design for no single point of failure, fast failure detection, and safe recovery. Start by defining the target, like 99.99% or 99.999%, because that drives the architecture and cost.

• Run stateless app instances across at least 3 AZs behind health-checked load balancers.
• Use active-active where possible, active-passive only if failover is well tested and fast.
• Keep data highly available with replicated databases, automated failover, backups, and tested restores.
• Add caching, queues, and circuit breakers so partial dependency failures do not take down the service.
• Use autoscaling, capacity headroom, and rate limiting to survive spikes and abuse.
• Build strong observability, SLOs, alerting, synthetic checks, and runbooks for quick detection and response.
• Do regular chaos and disaster recovery drills, including regional failover if strict uptime justifies multi-region.

The key is balancing availability, consistency, complexity, and operational readiness.

I usually answer this with a tight STAR structure, situation, actions, results, then the long-term learning.

At a previous company, I helped write the postmortem for a 90-minute checkout outage caused by a bad config change and weak dependency isolation. My role was incident commander during recovery, then co-author afterward. In the postmortem, I focused hard on systemic causes, not who pushed the change. - We mapped the full timeline, customer impact, detection gaps, and recovery friction. - We found three root issues, unsafe config rollout, no automatic canary for that service, and poor timeout defaults between services. - I drove follow-up actions, staged config deploys, synthetic checkout tests, tighter SLIs, and timeout budgets. - We also changed the template so every postmortem had owners, due dates, and validation steps.

The meaningful part was six months later, config-related incidents dropped a lot, and one similar bad change got caught in canary with no customer impact.

I’d separate capacity from efficiency, then use demand, saturation, and user impact to decide.

• Start with SLOs and traffic patterns. If latency and error rate are fine only because utilization is tiny, that often points to overprovisioning.
• Check resource saturation across CPU, memory, disk, network, connection pools, threads, and queue depth. Underprovisioning shows up as sustained high utilization plus rising latency, retries, drops, or OOMs.
• Compare actual load to reserved capacity. If average usage is 10 to 20 percent with low peak-to-average ratio, you likely have excess headroom.
• Look at efficiency metrics, requests per core, cost per request, memory per session, cache hit rate. Low utilization with poor throughput can mean inefficiency, not just oversizing.
• Run load tests and right-sizing experiments. Reduce replicas or instance size safely, watch SLOs, and validate autoscaling behavior before calling it overprovisioned.

I build alerts around user impact first, then system symptoms, then noisy low-level signals. The goal is that every page should tell the on-call engineer what is broken, how bad it is, and what to check first.

• Start from SLOs, alert on things like error rate, latency, saturation, and availability that reflect customer pain.
• Make alerts actionable, include service, severity, likely cause, runbook, dashboards, and recent deploy info.
• Use multi-window, multi-burn-rate alerts for SLOs so I catch both fast outages and slow burns.
• Reduce noise with proper thresholds, time tolerance, deduplication, grouping, and dependency-aware inhibition.
• Route by severity, page only for urgent issues, send tickets or Slack for noncritical signals and trends.
• Review alert quality regularly, track false positives, stale alerts, and tune or delete anything nobody acts on.

A practical example, I once cut pages by about 40 percent by converting CPU alerts into saturation plus user-latency alerts, adding inhibition for downstream failures, and tightening runbooks.

I’d treat it as a queueing or dependency problem, not a host saturation problem. Normal CPU and memory just mean the bottleneck is probably elsewhere.

• First, confirm where latency increased: client side, load balancer, app, database, or an external dependency.
• Check RED and USE signals: request rate, errors, duration, plus saturation on threads, connection pools, disk, network, and queues.
• Compare p50 vs p95/p99. If tail latency spikes, I’d suspect lock contention, retries, GC pauses, noisy neighbors, or slow downstream calls.
• Look at recent changes, deploys, feature flags, schema changes, traffic mix, and cache hit rate.
• Use tracing and logs to find the slow hop. I’d inspect DB query latency, connection exhaustion, packet loss, DNS, TLS handshakes, and retransmits.
• If needed, capture thread dumps or runtime metrics to catch blocked workers, deadlocks, or event loop stalls.

My approach is: stabilize first, understand second, fix third, learn last. In an incident, I separate mitigation from investigation so the team is not trying to do both under pressure.

• Build a clear timeline from alerts, logs, deploys, config changes, and human actions.
• Ask "what conditions made this possible?" not "who caused it?"
• Use methods like 5 Whys or causal trees, but stop when you find systemic gaps, not a person.
• Look for contributing factors, weak safeguards, unclear runbooks, alert noise, risky defaults, review gaps.
• In the postmortem, write facts, impact, detection, response, root causes, and action items with owners.

To keep it blameless, I set the tone early: people acted on the information and system constraints they had at the time. For example, after a bad deploy caused API errors, we found the real issues were missing canary checks and weak rollback automation, so the actions focused on safer releases, not the engineer who pushed it.

I look for anything that turns a small fault into a customer-facing outage. My mental checklist is usually failure domains, dependency risk, operational readiness, and recoverability.

• Single points of failure, including hidden ones like one database writer, one DNS provider, or one region.
• Tight coupling between services, where one slow dependency can cascade across the stack.
• Capacity and scaling limits, especially queue buildup, connection exhaustion, and noisy-neighbor issues.
• Weak observability, if we cannot detect, triage, and alert on bad states quickly, reliability is mostly luck.
• Unsafe deployments or config changes, missing canaries, rollback paths, or feature flags.
• Poor data durability, unclear backup strategy, restore testing, and undefined RPO or RTO.
• External dependency risk, third-party APIs, auth providers, and network assumptions.
• Ambiguous ownership, weak runbooks, and no incident process, because operability is part of reliability.

I’d treat it as a container lifecycle problem first, not an app logging problem. Clean logs often mean the process is getting killed from the outside, or exiting before it can log anything useful.

• Start with kubectl describe pod, look for Last State, exit codes, OOMKilled, probe failures, and events.
• Check restart reason with kubectl get pod -o wide and kubectl describe, especially CrashLoopBackOff versus node eviction.
• Inspect previous container logs using kubectl logs <pod> --previous, that often shows the real failure before restart.
• Verify liveness, readiness, and startup probes. Bad probe paths, short timeouts, or slow startup are common.
• Check resource limits, CPU throttling, and memory pressure. OOM kills often leave little app logging.
• Look at node-level issues, kubectl describe node, kubelet events, disk pressure, or runtime errors.
• If needed, add an ephemeral debug container or override command to keep the container alive and inspect filesystem, env vars, and dependencies.

I’d compare them across control, operational overhead, isolation, and scaling.

• VMs give the most OS-level control and strong isolation, but you own patching, image management, capacity planning, and slower boot times.
• Containers improve density and portability, start fast, and fit well with CI/CD, but you still manage the cluster, networking, runtime security, and noisy-neighbor risks.
• Serverless removes most infrastructure work and scales automatically, which is great for bursty event-driven workloads, but you give up runtime control and can hit cold starts, timeout limits, and vendor-specific constraints.
• Cost-wise, VMs are steady-state friendly, containers are efficient at scale, serverless is great for spiky low-to-medium usage but can get expensive under constant load.
• In practice, I pick based on workload shape, compliance needs, and how much undifferentiated ops work the team can absorb.

I treat capacity planning as a mix of forecasting, protection, and fast feedback. With unpredictable traffic, the goal is not perfect prediction, it is graceful scaling and controlled failure.

• Start with baselines, traffic by RPS, latency, CPU, memory, queue depth, and growth by endpoint or tenant.
• Model both average and peak behavior, then run load tests to find saturation points and bottlenecks.
• Add headroom and define SLO-driven thresholds, for example scale before latency or error rate degrades.
• Use autoscaling carefully, based on leading signals like queue length or concurrency, not just CPU.
• Prepare for spikes with rate limits, caching, backpressure, circuit breakers, and priority shedding.
• Revisit forecasts often, weekly or monthly, because fast growth makes old assumptions stale.

In practice, I also partner with product teams on launches and seasonality so surprises become smaller and more manageable.

I design for graceful degradation first, then recovery. The goal is to keep core user paths working even when a dependency is slow, flaky, or partly down.

• Set explicit dependency budgets: timeouts, retries with jitter, and capped concurrency so one bad service cannot exhaust threads or connections.
• Use circuit breakers and load shedding. Fail fast on noncritical calls, return cached or default responses, and preserve the main transaction.
• Isolate blast radius with bulkheads, separate pools, queues, and asynchronous workflows where possible.
• Design for idempotency and replay, so transient failures can be retried safely.
• Add health signals and SLO-based alerting, not just up/down checks, to detect brownouts early.
• Test with failure injection and dependency drills, like latency, error spikes, and partial regional loss.

A practical example is checkout: if recommendations fail, hide them; if tax is slow, use a bounded fallback path; if payment degrades, queue and reconcile safely.

I’d build DR around business impact first, then prove it with regular tests. The key is turning vague “we need DR” into clear recovery targets and an executable runbook.

• Start with BIA, define RTO, RPO, critical dependencies, and what “minimum viable service” looks like.
• Pick a pattern based on those targets, backup and restore, pilot light, warm standby, or active-active across regions.
• Replicate not just app data, but configs, secrets, DNS, infrastructure as code, observability, and access paths.
• Automate failover and recovery steps where possible, with runbooks for the manual parts and clear ownership.
• Validate with game days, restore tests, region failover drills, dependency failure injection, and data consistency checks.
• Measure actual recovery time versus RTO, recovery point versus RPO, capture gaps, then update architecture and docs.

A concrete example, for a payment API, I’d run quarterly regional failover tests, monthly backup restore tests, and require every drill to produce evidence, timings, and action items.

I’d treat it as stabilize first, then clarify ownership, then improve the system. The first 90 days should reduce pain fast while building habits that last.

• Days 1 to 30, learn the service map, incident history, paging patterns, top alert sources, and who actually responds today.
• Cut obvious noise quickly, remove duplicate alerts, tighten thresholds, add routing, and define what is actionable versus informational.
• Establish clear ownership, create or fix service owners, on-call rotations, escalation paths, and lightweight runbooks for top recurring incidents.
• Start incident reviews focused on systemic fixes, not blame, and track recurring themes like capacity, deploys, dependencies, or missing observability.
• Days 31 to 90, prioritize a small backlog, improve dashboards and SLOs, measure MTTR, alert volume, and repeat incidents, then show progress weekly.

I like to stack safety mechanisms so rollback is cheap and impact is limited. My rule is, separate deploy from release, reduce blast radius first, then automate the decision points.

• Feature flags for anything user-facing, so code can ship dark and be enabled by cohort, tenant, or percentage.
• Canary rollouts on stateless services, start with 1 to 5 percent, watch SLIs like error rate, latency, saturation, and business metrics.
• Blue-green for bigger platform changes, especially runtime upgrades or risky infra swaps, because instant traffic cutback is easy.
• Progressive rollouts tied to automated analysis, if metrics breach thresholds, promotion stops or rolls back.
• For databases, I use expand-contract migrations, dual reads or writes only when necessary, and backward-compatible app versions.

Example: we rolled out a new auth service behind a flag, canaried internal users first, then 5 percent of traffic, caught a token refresh regression, disabled the flag, fixed it, and resumed.

I’d answer this with a tight STAR story: healthy internal metrics can still hide user pain, so I focus on signals from the edge and the user journey.

At one company, our dashboards were all green, CPU, memory, pod health, and even service level error rates looked fine, but customers were reporting checkout timeouts. I started by comparing synthetic checks and support tickets against our service metrics, then looked at the path outside the app. We found the gap in monitoring was at the CDN and third-party payment dependency layer. Our health checks only validated internal /health endpoints, not a real checkout flow. The CDN had a regional routing issue, and retries masked it internally. We fixed it by adding end-to-end synthetic transactions, per-region external probes, and alerting on user-facing latency, not just service health.

I treat it as a risk and business alignment problem, not an argument about “stability vs speed.” Product teams usually respond when reliability work is tied to customer impact, revenue, and developer velocity.

• Start with data, SLO misses, incident frequency, MTTR, error budget burn, customer pain.
• Translate reliability gaps into business terms, checkout failures, churn risk, on-call load, slower launches.
• Prioritize by risk, focus on high-likelihood, high-impact items first, not generic cleanup.
• Use error budgets to create a shared rule, if reliability is degrading, feature velocity slows temporarily.
• Bundle reliability into delivery, require observability, rollback plans, capacity checks, and runbooks for launches.
• Offer small, high-leverage fixes first, like better alerts or removing a top toil source.

In practice, I’ve used incident reviews to show that one flaky dependency caused repeated launch delays. We funded a two-week fix, reduced pages, and actually helped the team ship faster.

They all reduce blast radius, but at different layers.

• Rate limiting controls how much traffic a client can send, like 100 req/s, so one noisy tenant cannot starve everyone else.
• Load shedding drops low-priority or excess requests when the system is already saturated, which keeps core paths alive instead of letting everything time out.
• Backpressure tells upstream producers to slow down when consumers, queues, or downstream services are full, preventing memory blowups and queue storms.
• Circuit breakers stop calling a dependency that is failing or timing out, so threads, connections, and retries are not wasted on doomed work.
• Together, they preserve latency and availability, fail fast, and give the system room to recover instead of collapsing under cascading failure.

Good incident communication is audience-specific, time-bound, and honest. Same facts, different level of detail. I’d keep one incident commander or comms lead responsible for message consistency, then tailor updates by stakeholder.

• Executives: business impact, customer impact, current risk, ETA confidence, next decision point.
• Engineers: symptoms, scope, timeline, mitigation status, owners, hypotheses, next actions.
• Support teams: plain-language issue summary, affected features, workaround, what to tell customers.
• Customers: acknowledge the issue, explain visible impact, say what you’re doing, give next update time.
• For everyone: use a regular cadence, even if the update is “still investigating,” avoid speculation, timestamp everything.

In practice, I like a shared source of truth, like an incident channel plus a status page. During a payment outage, we sent engineering updates every 15 minutes, exec summaries every 30, and customer status page updates on a fixed cadence, which reduced confusion fast.

I’d answer this with a tight STAR story, focusing on risk, decision criteria, and communication.

At a previous company, we had a major checkout latency spike during a peak sales window. We did not yet know if it was app code, database saturation, or a bad downstream dependency, and error rates were climbing fast. I had to choose between waiting for more data or reducing blast radius immediately. I decided to fail over read traffic, disable a noncritical recommendation service, and temporarily rate limit a noisy partner integration. My call was based on two things, customer impact was rising, and each action was reversible within minutes.

I told leadership what we knew, what we did not know, and the trigger for rollback. That stabilized the platform, then we confirmed the downstream dependency was the primary issue.

I handle it by making the tradeoff explicit, then aligning on business impact instead of arguing from opinion. Reliability debates usually get easier when you translate risk into customer pain, revenue impact, and engineering cost.

• First, I clarify the decision, what risk we’re accepting, for how long, and who owns it.
• I bring data, error budget burn, incident history, MTTR, customer impact, and likely failure modes.
• I frame options, for example, ship now with guardrails, or delay and reduce blast radius.
• If leadership wants to accept risk, I document it clearly, including triggers that force re-evaluation.
• If it’s heated, I stay collaborative, “I’m fine with this path if we agree on rollback, monitoring, and ownership.”

Example, a team wanted to skip load testing before launch. I proposed a limited rollout, tighter alerts, and a rollback plan. They shipped on time, and we reduced the chance of a full outage.

I use a fast isolation approach: start with symptoms, then narrow by dependency boundaries and recent change data.

• Check blast radius first, one service, one AZ, one region, or all users.
• Look at golden signals, latency, errors, saturation, traffic, and compare app metrics vs node, DB, and network metrics.
• Correlate timelines with deploys, config changes, autoscaling, cloud events, and dependency status pages.
• Trace the request path, client, edge, load balancer, app, cache, DB, external APIs, to find the first failing hop.
• Use logs and traces to separate app exceptions from timeouts, packet loss, DNS, TLS, or connection issues.
• Run controlled tests, hit the app locally, bypass layers if possible, test dependency health, and compare from multiple regions.
• If infra and app look healthy but failures align with a vendor or upstream, treat it as external and activate mitigation, retries, failover, or feature degradation.

I balance safety with developer velocity by putting guardrails into the platform, instead of relying on manual reviews for everything.

• Start with least privilege, short lived credentials, and strong IAM boundaries, so defaults are secure.
• Automate security checks in CI/CD, like SAST, dependency scanning, secret detection, and policy as code, so issues are caught early.
• Standardize paved roads, hardened base images, approved Terraform modules, golden Kubernetes templates, so engineers can ship without reinventing controls.
• Use progressive delivery, feature flags, canaries, and fast rollback paths, so changes are low risk.
• Centralize observability and audit trails, then alert on meaningful signals, not noise.
• For high risk changes, require stronger approvals, but keep low risk paths self service.

The key is making the secure path the easiest path.

A solid way to answer this is: name a few real failure modes, then explain how you validate recovery, not just backups.

• I have seen silent backup corruption, incomplete snapshots, expired credentials to object storage, replicas that looked healthy but had unusable WAL/binlogs, and restores that failed because schema versions or IAM rules changed.
• I have also seen the classic issue where backups existed, but RTO and RPO were impossible because restore time was never tested at production scale.
• To make recovery work under pressure, I treat restore as a regularly tested workflow, not a document. We run scheduled restore drills, measure actual RTO/RPO, and verify application integrity, not just database startup.
• I automate runbooks where possible, keep backups immutable and cross-region, and alert on backup freshness plus restore test failures.
• In an incident, clear roles, a practiced checklist, and known-good recovery points reduce panic and bad decisions.

I treat it as three goals with different controls: reliability through determinism, speed through parallelism and smart caching, safety through progressive delivery and guardrails.

• Make builds reproducible, pin versions, use immutable artifacts, and promote the same artifact across environments.
• Keep pipelines fast by running tests in parallel, caching dependencies and layers, and splitting unit, integration, and end-to-end stages by risk.
• Fail early with linting, schema checks, IaC validation, and contract tests before expensive deploy steps.
• Make deploys safe with blue/green or canary, feature flags, automated rollback, and health checks tied to SLOs.
• Reduce human error with trunk-based development, small changes, templates, and policy-as-code for approvals and secrets handling.

I also watch pipeline SLIs like lead time, flaky test rate, rollback rate, and mean time to restore, then fix the biggest bottlenecks first.

I look for SPOFs in two lenses, systems and people/process. The goal is to find anything whose failure stops delivery or recovery, then either remove it or make it survivable.

• Start with critical user journeys and incident paths, then trace dependencies, databases, queues, DNS, cloud regions, CI/CD, secrets, on-call.
• For tech, check redundancy, failover, backups, restore tests, load balancer health checks, multi-AZ or multi-region, and manual runbook steps.
• For teams, look for knowledge silos, one approver, one admin, one person who knows a service, or undocumented tribal workflows.
• Use game days, chaos testing, PTO coverage, and access audits to expose hidden fragility.
• Reduce risk with automation, cross-training, documentation, rotations, shared ownership, and least-privilege access for multiple people.
• Track it in a risk register with severity, owner, mitigation deadline, and test evidence, not just good intentions.

I’d narrow it fast by asking, “What changed for that slice?” Then I’d compare failing vs healthy traffic across logs, metrics, traces, and recent deploys.

• Scope it: isolate by endpoint, customer segment, region, auth type, app version, and dependency path.
• Compare baselines: request rate, latency, payload size, status codes, retries, and saturation for affected vs unaffected traffic.
• Check recent changes: deploys, feature flags, config, schema changes, rate limits, WAF rules, partner integrations.
• Trace a few failed requests end to end, look for a shared hop like DB shard, cache key pattern, or downstream API.
• Validate data-specific issues: malformed payloads, tenant-specific config, expired credentials, bad entitlements, hot partitioning.
• Mitigate first if needed: disable a flag, reroute, relax a limit, or roll back.

If I were answering in an interview, I’d emphasize hypothesis-driven debugging and keeping customer impact low while finding the common denominator.

I treat documentation like production infrastructure, it has owners, standards, and a review cycle. The goal is that a reasonably skilled engineer can handle common tasks and incidents without needing the one person who “just knows.”

• Start with the highest-risk areas, on-call runbooks, recovery steps, architecture diagrams, and service dependencies.
• Keep docs close to the code, in the repo when possible, versioned and reviewed in pull requests.
• Use lightweight templates, purpose, prerequisites, commands, rollback, validation, and escalation paths.
• Test docs during onboarding, game days, and real incidents, if someone gets stuck, the doc is incomplete.
• Assign ownership and expiry, stale docs are often worse than no docs.
• Capture decisions too, short ADRs help explain why systems work the way they do.

I’ve used Terraform heavily, plus CloudFormation in AWS-heavy teams. My approach is to treat infrastructure like application code, with reviews, tests, and safe rollout patterns.

• Keep all infra in Git, require pull requests, code owners, and versioned modules.
• Use remote state with locking, and separate state per environment or service boundary.
• Prevent drift by making the IaC pipeline the only path to change, then run scheduled plan or drift detection jobs.
• Block unsafe changes with policy checks, terraform plan in CI, static analysis like Checkov or tfsec, and approval gates for destructive diffs.
• Reduce blast radius with least-privilege IAM, smaller modules, progressive rollouts, and backup or rollback plans.

In one role, we caught manual security group edits via nightly drift checks and auto-opened tickets before they caused incidents.

I’d answer this with a tight STAR structure: situation, what broke, how I responded, then what changed permanently.

At a previous team, we deployed a config change to our API gateway that tightened timeout settings. It looked safe in staging, but in production it caused retries to pile up between services, latency spiked, and we started returning 502s to customers. I was on call, saw error rates jump in Grafana, and we rolled back within about 10 minutes while I coordinated updates in Slack and our incident channel.

Afterward, I led the follow-up. We added canary deploys for gateway changes, better load tests for timeout and retry behavior, and alerts on retry storms instead of just host metrics. We also added a deployment checklist for high-risk config changes and made rollback steps one command, not a manual runbook.

From the customer’s perspective, reliability is simple: the product works when they need it, performs fast enough, and does not surprise them with errors, data loss, or weird behavior. Customers do not care that CPU is healthy if checkout fails, or that latency looks fine on average if their requests time out during peak traffic.

To make metrics reflect reality, I start with user journeys, not infrastructure: - Define critical paths, like login, search, checkout, API success. - Use SLIs tied to outcomes, availability, latency, correctness, durability. - Measure from the edge, synthetic checks plus real user telemetry. - Segment by customer experience, region, tier, device, dependency. - Set SLOs based on impact, then use error budgets to guide tradeoffs.

Averages can hide pain, so I prefer percentiles, success rates, and journey-level metrics over internal system health alone.

The most confidence comes from layered testing, but if I had to pick one, I’d say production-like integration testing. Unit tests are great for speed and coverage, but most outages happen at boundaries, APIs, databases, queues, config, auth, timeouts, and retries.

• Unit tests catch logic bugs early and make refactoring safe.
• Integration tests give me the best signal, because they validate real dependencies and failure handling.
• Load tests matter when scale or latency is part of the risk, especially for hot paths.
• Chaos testing is valuable for resilience, but I use it more to improve systems than as a release gate.
• The highest confidence comes from canaries, feature flags, and strong observability, because real production traffic finds what pre-prod misses.

So my ideal answer is, integration tests plus safe progressive delivery.

A solid way to answer is: name the issue, how you detected it, what you did short term, then what you changed to prevent it.

• Replication lag, usually on read replicas during traffic spikes or long-running writes. I watched replica delay and query latency, then throttled heavy jobs, tuned indexes, and sometimes split reads by freshness requirements.
• Failover problems, mostly around stale DNS, unready replicas, or apps not retrying cleanly. I added health checks, practiced failovers, shortened TTLs, and made clients use retry with backoff.
• Lock contention, often from large transactions or missing indexes. I used slow query logs and lock wait metrics, then reduced transaction scope, added indexes, and moved batch updates to smaller chunks.
• Connection exhaustion, usually from app pools sized too high. I capped pool sizes, added pooling like PgBouncer, killed leaks, and alerted on active versus idle connections.

I treat third-party dependencies like part of my production system, but with extra failure planning because I do not control them.

• Build explicit observability around them, latency, error rate, saturation, quota usage, and dependency-specific SLOs.
• Use synthetic checks from multiple regions so I can detect provider issues before customers report them.
• Isolate blast radius with timeouts, retries with jitter, circuit breakers, bulkheads, and strong caching where possible.
• Track vendor health signals, status pages, incident feeds, changelogs, and API deprecation notices.
• Keep an inventory of owners, contracts, SLAs, auth methods, rate limits, and runbooks for each dependency.
• Have fallback plans, degraded modes, secondary providers, queued writes, or feature flags to disable noncritical paths.
• Review them regularly in incident postmortems and game days, especially around dependency failure scenarios.

The key is observability plus graceful degradation, not just hoping the vendor stays healthy.

I treat chaos engineering as a safe way to validate assumptions before production does it for me. My approach is, start small in staging, define a steady-state metric, inject one failure, then confirm detection, failover, and recovery all behave as expected.

• I have used pod kills, node drains, DNS blackholes, latency injection, and dependency timeouts with tools like litmus, chaos-mesh, and cloud fault injection.
• One exercise killed a message broker leader; the system stayed up, but consumer lag exploded because retry backoff was too aggressive and alerts only watched total errors.
• Another test added 500ms latency to an internal auth service; it revealed thread pool exhaustion and a missing circuit breaker in upstream APIs.
• The biggest value was finding hidden coupling, weak alerting, and recovery steps that were documented but not actually automated.

I try to shift from being the fixer to building systems, habits, and guardrails that let teams make good reliability decisions on their own.

• Set clear reliability standards, like SLOs, incident severity, paging rules, and production readiness checklists.
• Teach through real work, join design reviews, postmortems, and incident debriefs, then ask questions instead of prescribing every answer.
• Create reusable tools and templates, dashboards, runbooks, alert patterns, and rollout playbooks, so teams do not need 1:1 help every time.
• Use office hours and group sessions for common issues, rather than ad hoc interrupts.
• Delegate ownership, pick reliability champions in each team, coach them deeply, then let them scale the practice.

One example, I inherited a team that escalated every alerting issue to SRE. I introduced alert quality guidelines, monthly incident reviews, and trained two senior engineers as embedded reliability leads. Within a quarter, escalations dropped a lot, and service health improved.

I’d answer this with a quick STAR structure, then focus on prioritization under uncertainty.

At a previous company, I inherited a legacy batch processing system that failed unpredictably and had almost no metrics, logs, or runbooks. First, I improved observability, not architecture, because without visibility every fix was guesswork. I added basic golden signals, structured logs with correlation IDs, and alerts tied to user impact, not just host health. I also documented dependencies and created a simple dashboard so on call engineers could see where jobs were stalling.

Once we had signal, the biggest issue turned out to be retry storms against a downstream database. Then I added backoff, concurrency limits, and clearer failure modes. That reduced incidents quickly, and it gave us the data to plan deeper reliability work instead of firefighting blindly.

I’d evaluate an SRE team on whether they improve reliability without slowing the business down. The key is balancing user impact, operational load, and engineering velocity.

• Reliability outcomes: SLO attainment, error budget burn, incident frequency, duration, and customer-facing impact.
• Detection and response: MTTD, MTTR, paging quality, escalation health, and how often incidents are caught before users report them.
• Toil reduction: time spent on repetitive ops work, automation coverage, and whether engineers are getting interrupted less over time.
• Change safety: deployment frequency, change failure rate, rollback rate, and whether teams can ship safely during normal hours.
• Learning and leverage: strong postmortems, repeated issue reduction, and platform improvements that help multiple teams.

I’d also look at team health, burnout, and whether product teams trust SRE as an enabler, not just a gatekeeper.

I usually answer this with a quick pattern, identify toil, automate the safe 80 percent first, then measure time saved and reliability impact.

At one company, on call was full of noisy disk and service restart alerts. I built a small remediation workflow that checked health signals, cleaned temp space, restarted only if dependency checks passed, and opened a ticket if it failed. That removed a big chunk of after hours pages.

Another one was manual Kubernetes access and deploy housekeeping. I standardized RBAC requests, added self service templates, and automated expired sandbox cleanup with scheduled jobs. The impact was fewer interruptions, faster developer turnaround, and better auditability. Across both, we cut a few hours of weekly toil per engineer, reduced alert fatigue, and let the team spend more time on capacity work and reliability improvements.

I treat game days as a way to validate people, process, and tooling under controlled stress, not just break things for fun.

• Start with clear objectives, like failover time, alert quality, runbook accuracy, or team handoff behavior.
• Pick realistic scenarios from past incidents, top risks, or recent architecture changes. Define blast radius, stop conditions, and who can abort.
• Prepare observability, comms channels, roles, and success metrics ahead of time. Make sure stakeholders know whether it is tabletop or live.
• During the exercise, run it like a real incident: incident commander, timeline, status updates, and disciplined decision logging.
• Capture gaps in detection, escalation, access, dashboards, and runbooks, not just technical failures.
• End with a blameless review, assign concrete actions with owners and due dates, then rerun the scenario later.

Example: I once simulated a regional dependency failure, and we found failover worked, but DNS validation and exec comms were too slow.