Distributed Systems & Resilience Engineering: Complete Developer Learning Guide

Learn how distributed applications communicate, handle failures, recover safely, and remain observable—with practical C# and ASP.NET Core examples for modern .NET developers.

Introduction

Modern applications rarely work in isolation. A business application may contain an ASP.NET Core API, a SQL database, a background worker, a message broker, a cache, and external services. These components communicate over networks and may fail independently.

When an application grows, writing business logic is only one part of the engineering challenge. Developers must also consider network delays, service outages, duplicate requests, message delivery, data consistency, resource limits, and recovery.

Distributed Systems & Resilience Engineering is the practice of designing and operating applications so that they continue to provide correct and useful behavior when components become slow, unavailable, overloaded, or inconsistent.

This guide explains the concepts step by step, starting with fundamentals and progressing to patterns and implementation techniques used in modern .NET applications.

Learning approach: We will use a generic order-processing application as a running example. It may contain an ASP.NET Core API, a relational database, inventory and payment dependencies, a message broker, and background workers. The same concepts apply to many business systems.

Distributed Systems: The Big Picture

A distributed system is a collection of independent computers or services that communicate over a network and work together to provide a capability. A user may experience one application, even though its work is handled by several processes, APIs, databases, queues, and infrastructure components.

For example, an online order may pass through an ASP.NET Core API, an order database, a payment provider, an inventory service, and a notification worker. Each component can fail or become slow independently.

Why Distributed Systems Are Different

Networks are unreliable: requests can be delayed, dropped, or interrupted.
Partial failure is normal: one service can fail while other services remain healthy.
Latency is variable: a remote call is slower and less predictable than an in-process method call.
Data is distributed: different services may own separate databases and update at different times.
Independent deployment: components may run different versions during a rollout.

Key idea: A distributed system must be designed for failure, delay, duplication, and recovery—not just for the successful request path.

Monolith, Modular Monolith and Microservices

A monolith packages an application as one deployable unit. A modular monolith keeps the deployment unified but separates business capabilities into well-defined modules. Microservices split capabilities into independently deployable services that communicate over explicit boundaries.

Approach	Characteristics	When it fits
Monolith	One application and often one deployment unit.	Small teams, simpler domains, early products.
Modular monolith	One deployment with strong internal module boundaries.	Growing systems that need structure without distributed operations.
Microservices	Independent services, ownership and deployment.	Domains and teams that genuinely need independent scaling or release cycles.

Microservices introduce network calls, service discovery, observability, distributed data ownership, deployment coordination, and failure handling. They are not automatically faster, cheaper, or easier to maintain.

Practical advice: Start with a well-structured monolith or modular monolith unless independent deployment, scaling, team ownership, or isolation requirements justify service boundaries.

Communication: Synchronous vs Asynchronous

Services communicate either by waiting for an immediate response or by exchanging messages that can be processed later.

Synchronous communication

An HTTP request from one API to another is synchronous from the caller's perspective: the caller waits for a response. It is useful when the caller needs an immediate result, but it couples availability and latency between the services.

Asynchronous communication

A producer publishes a message to a broker or durable queue. A consumer processes it independently. The producer can often return before the work finishes.

Aspect	Synchronous HTTP	Asynchronous messaging
Response	Immediate response expected	Processing happens later
Coupling	Caller depends on callee availability	Producer and consumer are more decoupled
Good for	Queries and immediate validation	Notifications, report generation, integration events
Trade-off	Latency and cascading failures	Eventual consistency and message operations

Do not turn every HTTP call into a message. Choose based on whether the user or calling process needs the result immediately.

Failure Modes and Fault Boundaries

A fault boundary is a place where one component's failure can affect another. In a distributed application, boundaries commonly exist between an API and a database, between two services, between an application and a broker, and between your system and an external provider.

Common failure modes

Connection refused or DNS resolution failure.
Request timeout or a slow dependency.
HTTP 429 throttling or 5xx server errors.
Database deadlocks, connection pool exhaustion, or failover.
Broker unavailable, message lock expiry, or consumer crash.
Duplicate delivery, out-of-order events, or partial completion.

Classify failures before choosing a response. A validation error (400) usually will not improve when retried. A temporary network reset may be worth retrying. A permission error (401/403) needs a configuration or identity fix, not repeated attempts.

Design habit: For every remote dependency, ask: what if it is slow, unavailable, overloaded, returns an error, or succeeds but the response is lost?

Timeouts and Deadlines

A timeout limits how long an operation may wait. Without timeouts, requests can occupy threads, connections, memory, and other resources while waiting for a dependency that may never respond.

Set timeouts for outbound HTTP calls, database operations where supported, message processing, and user-facing requests. The timeout should fit the overall request budget.

Timeout budget example

Incoming API request budget:       8 seconds
  Authentication / validation:     0.5 seconds
  Inventory dependency:            2 seconds
  Pricing dependency:              2 seconds
  Database work and response:      2.5 seconds
  Safety margin:                   1 second

This is an illustrative budget, not a universal setting. If each downstream call can consume the full incoming timeout, a chain of calls can exceed the caller's deadline.

Distinguish a per-attempt timeout from a total operation timeout. Retries must fit inside the total deadline. Propagate cancellation tokens through application and data-access methods so abandoned requests can stop unnecessary work.

Retry Pattern and Exponential Backoff

The Retry pattern repeats an operation after a failure that is likely to be temporary. Examples include a short network interruption or a dependency returning a retryable status.

Exponential backoff

Instead of retrying immediately at a fixed interval, exponential backoff increases the delay after each failed attempt. Jitter adds randomness so many clients do not retry at exactly the same moment.

Attempt 1  -> wait about 200 ms
Attempt 2  -> wait about 400 ms
Attempt 3  -> wait about 800 ms
Attempt 4  -> stop and report failure

These values are illustrative. Real delays should be bounded and tuned to the dependency and request deadline.

When retry is appropriate

Temporary connection failures.
Selected 5xx responses from a dependency.
429 responses when the service's retry guidance is respected.

When retry is dangerous

Validation and authorization failures.
Non-idempotent operations where the first attempt may have succeeded.
Long-running requests with no remaining time budget.
Retries that multiply load on an already overloaded service.

Important: Retrying a payment or order-creation POST can create duplicates if the server completed the operation but the response was lost. Use an idempotency strategy before retrying such operations.

Circuit Breaker Pattern

A circuit breaker prevents an application from repeatedly calling a dependency that is currently failing. It temporarily stops calls after a configured failure condition is reached, giving the dependency time to recover.

Three states

State	Behavior
Closed	Requests are allowed; failures are measured.
Open	Calls are rejected or short-circuited for a configured period.
Half-open	A limited number of trial calls check whether the dependency has recovered.

A circuit breaker is not a retry mechanism. Retry repeats an operation; a circuit breaker stops calls when continued attempts are unlikely to succeed. They are often combined carefully.

Caller
  |
  v
Circuit Breaker
  | Closed: forward calls
  | Open: fail fast / fallback
  | Half-open: allow trial calls
  v
Remote Dependency

Define what the application should return when the circuit is open: a clear temporary error, cached data, a queued request, or a degraded response. Do not hide a critical failure behind a misleading success response.

Bulkhead and Concurrency Isolation

The Bulkhead pattern limits the resources one dependency or workload can consume, so it cannot exhaust resources needed by the rest of the application. The name comes from ship compartments that limit flooding to one area.

For example, a report-generation integration may be limited to a small number of concurrent calls. If that dependency becomes slow, it should not consume every outbound connection or worker slot used by critical operations.

Ways to apply bulkheads

Separate bounded worker queues for different workloads.
Limit concurrent outbound calls to a dependency.
Use separate connection pools or clients where appropriate.
Set resource limits for containers and background workers.
Prioritize critical work over optional work.

Bulkheads and rate limiters solve related but different problems: concurrency limits cap simultaneous work; rate limits cap how frequently work is admitted.

Rate Limiting and Throttling

Rate limiting controls how many requests a client, tenant, user, or service can make within a defined period. It protects capacity and helps provide fair usage. Throttling is the act of slowing or rejecting requests when a limit or capacity threshold is reached.

ASP.NET Core includes rate-limiting middleware. A simple fixed-window example is shown below; production policies should be based on authenticated identity or a trusted client key, not blindly on user-controlled headers.

using System.Threading.RateLimiting;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddRateLimiter(options =>
{
    options.AddFixedWindowLimiter("api", limiter =>
    {
        limiter.PermitLimit = 100;
        limiter.Window = TimeSpan.FromMinutes(1);
        limiter.QueueLimit = 0;
        limiter.AutoReplenishment = true;
    });
});

var app = builder.Build();

app.UseRateLimiter();

app.MapGet("/api/status", () => Results.Ok("Available"))
   .RequireRateLimiting("api");

app.Run();

This policy is an introductory example. The built-in in-process limiter maintains counters per application instance. If the API runs on multiple instances, each instance may enforce its own limit; use an API gateway or a shared/distributed design when a global limit is required.

Common algorithms include fixed window, sliding window, token bucket, and concurrency limiter. Choose based on burst tolerance, fairness, and the resource being protected. Test the policy under load and return a useful 429 response when appropriate.

Fallback and Graceful Degradation

Graceful degradation means keeping essential application capabilities available when a non-critical dependency is unavailable. A fallback is an alternate response or action used when the preferred operation cannot complete.

Examples include showing recently cached product information when a recommendation service is down, accepting a request for asynchronous processing when a report service is busy, or disabling a non-essential feature while checkout remains available.

Good fallback design

Identify which features are essential and which can be degraded.
Make stale or partial data visible when it matters.
Do not return fake success for an operation that did not happen.
Log and measure fallback usage.
Define how the system returns to normal operation.

Business rule: A fallback must preserve correctness. Returning an old catalogue price may be acceptable in some contexts; pretending a payment succeeded is not.

Idempotency and Duplicate Request Handling

An operation is idempotent when repeating it with the same intent produces the same final effect as performing it once. Idempotency is essential when clients retry requests, networks fail after a server commits, or message brokers redeliver messages.

HTTP GET, PUT, and DELETE are defined with idempotent semantics in HTTP, while POST is not inherently idempotent. Application behavior still needs to follow the intended semantics.

Idempotency key flow

Client creates unique request key
          |
          v
POST /api/payments
Idempotency-Key: 8f2...
          |
          v
API checks stored key
     |             |
  Existing       New key
     |             |
Return saved    Process operation
result          and store outcome
     |             |
     +------ Return response

Store the key with an appropriate scope, request fingerprint, processing status, and result. Enforce uniqueness transactionally. Define how long keys are retained and what happens if the same key is reused with a different payload.

For message consumers, persist a processed-message identifier or use a business-level uniqueness constraint so duplicate delivery does not repeat the business effect.

Message Queues and Event-Driven Communication

A message queue stores work until a consumer is ready to process it. This separates the producer's request intake from the consumer's processing capacity. Event-driven systems publish facts about something that has happened, allowing interested consumers to react.

Queue vs publish-subscribe

Queue / competing consumers	Publish-subscribe
Consumers compete to process messages from a queue.	Each subscription receives its own copy of a published event.
Good for distributing a work backlog.	Good for notifying multiple independent capabilities.
Example: generate one report.	Example: OrderPlaced event consumed by billing, inventory, and notifications.

Design messages as contracts, version them carefully, avoid placing secrets or unnecessary personal data in payloads, and make consumers tolerant of duplicate delivery and delayed events.

Delivery Guarantees, Ordering and Dead-Letter Queues

Message systems commonly provide at-most-once, at-least-once, or service-specific processing guarantees. At-least-once delivery means a message can be delivered more than once. It does not mean the business operation is automatically performed exactly once.

At-most-once: A message is delivered zero or one time; loss may be possible.
At-least-once: A message is retried until acknowledged or otherwise handled; duplicates are possible.
Effectively-once business effect: Achieved through idempotent processing, deduplication, and transactional state changes—not by assuming the broker eliminates every duplicate.

Ordering is usually guaranteed only within a defined scope or under specific broker features. Parallel consumers can change completion order. If order matters, use partitioning, sessions, sequence numbers, or per-aggregate processing rules supported by the broker.

A dead-letter queue stores messages that cannot be processed successfully after configured attempts or conditions. Monitor it, investigate root causes, and define a safe replay process. A dead-letter queue is not a place to ignore failed work.

Transactional Outbox Pattern

The Transactional Outbox pattern addresses a common consistency problem: an application updates its database and publishes an event, but the database commit and message publish are separate operations. One can succeed while the other fails.

With an outbox, the application writes both the business change and an event record to the same database transaction. A background publisher later reads unsent outbox records and publishes them to the broker.

ASP.NET Core command handler
          |
          v
Single database transaction
  +-- Update Order table
  +-- Insert OutboxEvent row
          |
          v
Transaction commits
          |
          v
Outbox background publisher
          |
          v
Message broker
          |
          v
Consumer processes event

The publisher can crash after publishing but before marking the row as sent. Therefore, consumers still need idempotency. Add retry, locking/claiming, monitoring, retention, and cleanup for outbox records.

Saga Pattern and Distributed Workflows

A distributed transaction spanning independent databases is difficult to coordinate safely. The Saga pattern models a long-running business operation as a sequence of local transactions. If a later step fails, the workflow runs compensating actions for completed steps.

Example: Order workflow

Create order as Pending.
Reserve inventory.
Authorize payment.
Confirm order and request shipment.

If payment authorization fails after inventory reservation, a compensating action releases the inventory. Compensation is a new business action; it is not always a perfect rollback (for example, a refund may take time).

Orchestration vs choreography

Orchestration	Choreography
A coordinator tells participants what step to perform.	Services react to events published by other services.
Workflow visibility is centralized.	Participants are more autonomous, but the flow can be harder to trace.
Useful for explicit multi-step processes.	Useful for event-driven interactions with clear event ownership.

Persist workflow state, make each step idempotent, define timeouts and compensation, and provide operational tools to inspect and recover stuck workflows.

Eventual Consistency and Data Ownership

Eventual consistency means replicas or services may temporarily show different values, but under the system's assumptions they converge after updates propagate. It is common when separate services own separate data and communicate asynchronously.

For example, an order service may mark an order as placed before a reporting service updates its read model. A user may briefly see the old reporting total while the event is being processed.

Practical techniques

Define which service owns each business entity.
Publish domain or integration events after committed changes.
Use read models designed for the queries the application needs.
Expose processing status when updates are asynchronous.
Use version numbers or timestamps to detect stale updates.
Do not assume a distributed read is immediately consistent.

Eventual consistency is a deliberate trade-off, not an excuse for unclear data ownership or permanent disagreement between services.

Caching and Distributed Cache

Caching stores a reusable copy of data to reduce repeated work and improve response time. A distributed cache such as Redis can be shared by multiple application instances, unlike a process-local memory cache.

Common cache patterns

Cache-aside: Application checks the cache, loads from the data store on a miss, then populates the cache.
Read-through: A cache abstraction loads data from the source when needed.
Write-through: Writes update the cache and source through a coordinated strategy.
Write-behind: Cache writes are persisted asynchronously; this needs careful durability design.

Define expiration, invalidation, key naming, serialization, and behavior during cache outages. Avoid caching authorization-sensitive data under keys that can be shared across users or tenants. Cache is an optimization, not the source of truth unless the system is explicitly designed that way.

Service Discovery, API Gateway and Load Balancing

As systems grow, callers need a reliable way to reach service instances. Service discovery helps locate available instances. Load balancing distributes requests across healthy instances. An API gateway can provide a common entry point for routing, authentication integration, rate limiting, and cross-cutting policies.

Component	Responsibility
Service discovery	Find service endpoints or instances.
Load balancer	Distribute traffic across available targets.
API gateway	Route and apply edge policies for API traffic.
Reverse proxy	Forward client requests to backend servers.

Do not put all business logic in the gateway. Keep domain rules in the owning service and avoid making the gateway a new single point of failure.

Health Checks, Readiness and Liveness

Health checks help platforms and operators understand whether an application is running and whether it is ready to receive traffic. A liveness check answers whether the process should be restarted; a readiness check answers whether it should receive new work.

ASP.NET Core provides health-check support through Microsoft.Extensions.Diagnostics.HealthChecks and endpoint mapping.

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHealthChecks();

var app = builder.Build();

app.MapHealthChecks("/health");

app.Run();

This basic endpoint confirms that the application can respond. Add dependency checks thoughtfully: if every health probe fails whenever an optional service is unavailable, orchestration may remove all application instances at once. Separate liveness from readiness and avoid exposing sensitive diagnostic details publicly.

Observability: Logs, Metrics and Traces

Observability is the ability to understand a system's internal state from its emitted telemetry. Distributed systems need more than isolated log files because one user request may cross several services.

Three pillars

Logs: Records of discrete events, ideally structured with timestamp, severity, service, trace ID, and useful context.
Metrics: Numeric measurements over time, such as request rate, error rate, latency, queue depth, and resource usage.
Traces: A connected view of one operation as it crosses services and dependencies.

Use correlation IDs or W3C trace context to connect related work. Avoid logging passwords, access tokens, payment details, or unnecessary personal data. Define alerts around user impact and service-level objectives rather than alerting on every log entry.

Service-Level Indicators and Objectives

A Service-Level Indicator (SLI) is a measured aspect of service performance, such as successful request ratio or latency. A Service-Level Objective (SLO) is the target for that indicator over a defined period. A Service-Level Agreement (SLA) is a formal commitment, often with business consequences.

Example: an API might target 99.9% successful eligible requests over a rolling 30-day period and a defined percentage of requests under a latency threshold. The exact SLO must reflect business needs and measurement rules.

An error budget is the amount of unreliability permitted by the SLO. It helps teams balance feature delivery and reliability work. Define exclusions, measurement windows, and what actions occur when the budget is consumed.

Resilience in ASP.NET Core HTTP Clients

When an ASP.NET Core application calls another HTTP service, use IHttpClientFactory rather than creating and disposing a new HttpClient for every request. The factory manages handler lifetimes and supports named or typed clients.

For modern .NET applications, Microsoft's Microsoft.Extensions.Http.Resilience package provides resilience handlers built on the Polly resilience APIs.

dotnet add package Microsoft.Extensions.Http.Resilience

var builder = WebApplication.CreateBuilder(args);

builder.Services
    .AddHttpClient<InventoryClient>(client =>
    {
        client.BaseAddress =
            new Uri("https://inventory.example.com/");
    })
    .AddStandardResilienceHandler();

var app = builder.Build();

The standard handler composes strategies such as total timeout, retry, circuit breaker, and per-attempt timeout. Defaults can change by package version; inspect and configure them for your workload. The handler's standard retry behavior should not be blindly applied to requests with unsafe side effects. For such operations, use idempotency keys or configure an appropriate method-specific strategy.

Package distinction: Microsoft.Extensions.Http.Resilience is the .NET HTTP integration package. Polly provides the underlying resilience strategy APIs. For most HTTP client scenarios, start with the Microsoft integration instead of manually rebuilding a standard pipeline.

Polly Resilience Pipelines

Polly is a .NET resilience library that provides composable strategies including retry, timeout, circuit breaker, rate limiter, fallback, and hedging. The newer resilience pipeline APIs are designed to compose strategies into a pipeline.

Use Polly directly when you need resilience around operations beyond outgoing HTTP requests, or when you need a custom pipeline. For standard HTTP client resilience, consider the Microsoft HTTP resilience integration shown in the previous section.

using Polly;
using Polly.Timeout;

var pipeline = new ResiliencePipelineBuilder()
    .AddTimeout(TimeSpan.FromSeconds(3))
    .Build();

await pipeline.ExecuteAsync(async cancellationToken =>
{
    await PerformOperationAsync(cancellationToken);
});

This is a minimal illustration of a timeout pipeline. Production pipelines should select strategies and options based on the operation, define telemetry, and handle cancellation and exceptions deliberately. Do not stack multiple retry handlers without understanding how attempts multiply.

Resilience Testing and Chaos Engineering

A resilience design is only a hypothesis until it is tested under realistic failure conditions. Test not only successful responses but also dependency timeouts, throttling, partial failures, duplicate messages, delayed events, and process restarts.

Testing techniques

Unit tests: Verify retry decisions, idempotency rules, and compensation logic.
Integration tests: Test database transactions, outbox publishing, and message consumers.
Load tests: Observe latency, throughput, saturation, and rate-limit behavior.
Fault injection: Simulate network errors, slow dependencies, and broker outages.
Chaos experiments: Introduce controlled failures in a safe environment to test recovery assumptions.

Start with a hypothesis, limit the blast radius, define abort conditions, monitor the experiment, and document the result. Never run uncontrolled failure experiments against production systems.

Security and Reliability Together

Resilience is not only availability. A system that remains online by bypassing authorization, accepting untrusted messages, or exposing internal diagnostics is not reliable in a meaningful sense.

Authenticate service-to-service calls and authorize each operation.
Use managed identities or a secure secret-management system where supported.
Validate message schemas and input size before processing.
Apply rate limits and quotas to protect shared resources.
Encrypt traffic and protect data at rest according to requirements.
Keep dead-letter queues and replay tools access-controlled.
Use dependency allowlists and secure configuration for outbound calls.

Design operational recovery so that only authorized operators can replay messages, change circuit settings, or trigger compensating actions.

Common Distributed-System Anti-Patterns

Infinite retries: Can amplify an outage and consume resources.
Retrying every status code: Repeats permanent failures such as validation or authorization errors.
Chatty service calls: Creates excessive network latency and cascading dependencies.
Shared database across every microservice: Weakens service ownership and couples deployments.
Distributed monolith: Services deploy separately but require synchronized releases and tightly coupled calls.
No idempotency: Duplicate requests or messages create repeated business effects.
Unbounded queues: Backlogs grow until storage, latency, or recovery becomes a problem.
Silent fallback: Hides failures and returns misleading data or success.
Missing correlation: Makes cross-service diagnosis unnecessarily difficult.

End-to-End Example: Resilient Order Processing

Consider a business application where a user places an order. The API validates the request and writes the order plus an outbox event in one database transaction. A publisher sends the event to a broker. Inventory and payment workflows process their steps, and an orchestrator or event-driven workflow coordinates the result.

Client
  |
  v
ASP.NET Core Orders API
  |  Validate + idempotency key
  v
Database Transaction
  |-- Order = Pending
  |-- Outbox event
  v
Outbox Publisher
  |
  v
Message Broker
  |
  +---- Inventory consumer
  |       |-- reserve stock
  |       |-- idempotent handler
  |
  +---- Payment workflow
  |       |-- authorize payment
  |       |-- retry only safe transient faults
  |
  v
Order workflow
  |-- confirm or compensate
  v
Order status available to client

Resilience controls in this example

Request idempotency prevents duplicate order creation.
Database transaction keeps order and outbox record consistent.
Queue decouples API intake from background processing.
Consumers handle duplicate delivery safely.
Retries and circuit breakers protect remote dependencies.
Saga compensation handles partial workflow failure.
Tracing and metrics reveal delays and stuck messages.

The exact design depends on payment, inventory, regulatory, and user-experience requirements. A simple application may not need every component in this diagram.

Choosing the Right Resilience Strategy

Problem	Useful strategy	Important caution
Temporary dependency failure	Retry with bounded backoff	Only retry operations that are safe to repeat.
Dependency repeatedly failing	Circuit breaker	Define meaningful fallback or failure response.
One workload consumes too many resources	Bulkhead / concurrency limit	Choose limits based on capacity tests.
Too many incoming requests	Rate limiting / throttling	Consider identity, tenant fairness, and multiple instances.
Long-running work	Queue and background worker	Monitor backlog and dead-letter messages.
Duplicate request or message	Idempotency and deduplication	Enforce uniqueness transactionally.
Multi-service business workflow	Saga	Define compensation and recovery operations.
Database update plus event publish	Transactional outbox	Consumers still need idempotency.
Need to trace cross-service latency	Distributed tracing	Propagate context and avoid sensitive telemetry.

Practical .NET Developer Checklist

Use IHttpClientFactory for outbound HTTP clients.
Set explicit timeouts and propagate cancellation tokens.
Use Microsoft.Extensions.Http.Resilience for standard HTTP resilience where appropriate.
Retry only transient failures and only when repeating the operation is safe.
Use idempotency keys for retryable business commands with side effects.
Bound concurrency, queues, and background processing.
Use ASP.NET Core rate-limiting middleware and understand per-instance behavior.
Use outbox/inbox or deduplication patterns for reliable message processing.
Define dead-letter monitoring and safe replay procedures.
Instrument logs, metrics, traces, and health checks.
Test partial failure, duplicate delivery, and recovery—not only happy paths.
Prefer the simplest architecture that meets the actual requirements.

Interview Questions and Answers

1. What is a distributed system?

A set of independent processes or services that communicate over a network and work together to provide a capability.

2. What is partial failure?

A situation where one component fails while other components continue running, which is common in networked systems.

3. What is the difference between retry and circuit breaker?

Retry attempts an operation again after a qualifying failure. A circuit breaker temporarily blocks calls when a dependency is unhealthy.

4. Why are timeouts important?

They bound waiting time and prevent slow dependencies from consuming resources indefinitely.

5. What is idempotency?

Repeating an operation with the same intent produces the same final effect as performing it once.

6. What is a bulkhead?

A resource-isolation technique that prevents one workload or dependency from consuming all resources.

7. What is eventual consistency?

Distributed data may be temporarily different but converges after updates propagate, assuming successful processing and recovery.

8. What is the Transactional Outbox pattern?

It stores a business update and an event record in the same database transaction, then publishes the event asynchronously.

9. What is a Saga?

A sequence of local transactions coordinated across services, with compensating actions when a later step fails.

10. Why do message consumers need to be idempotent?

Messages can be delivered more than once, so duplicate processing must not repeat the business effect.

11. What is a dead-letter queue?

A destination for messages that cannot be processed successfully under the configured delivery policy.

12. What is graceful degradation?

Keeping essential functionality available while reducing or disabling non-critical features during a dependency failure.

13. What is the difference between rate limiting and concurrency limiting?

Rate limiting controls how many operations are admitted over time; concurrency limiting controls how many operations run simultaneously.

14. What is distributed tracing?

Tracing follows a request across services and dependencies using correlated spans and trace context.

15. What is the purpose of Microsoft.Extensions.Http.Resilience?

It integrates standard and configurable resilience handlers into .NET HTTP clients using IHttpClientFactory.

Official Documentation and Further Learning

Use official documentation for package compatibility, runtime support, defaults, and implementation details. Resilience settings should be reviewed against the exact package and .NET version used by your application.

What’s Next?

You have now explored how distributed applications communicate, fail, recover, and remain observable. The next step is to apply these ideas in a small system: build an ASP.NET Core API, add a typed HTTP client with bounded resilience, introduce a background worker and message broker, and implement idempotent message handling.

Start with one real reliability problem, measure the current behavior, add the smallest useful pattern, and test how the application behaves when a dependency fails. Reliable systems are built through deliberate design and continuous verification—not by adding every pattern to every project.

Programming

Distributed Systems & Resilience Engineering – Complete Developer Guide