Monitoring, Logging & Observability – Complete Developer Learning Guide

Monitoring, Logging & Observability

A practical developer guide to understanding application behavior — from structured logs and metrics to distributed tracing, OpenTelemetry, Prometheus, Grafana, Application Insights, and actionable alerting.

.NET 10 LTSC# 14OpenTelemetryPrometheusGrafanaBeginner to Advanced

Learning Roadmap

1. Introduction to Observability 01
2. Monitoring, Logging and Observability 02
3. The Three Pillars: Logs, Metrics and Traces 03
4. What Is Structured Logging? 04
5. Log Levels and When to Use Them 05
6. Logging in ASP.NET Core 06
7. Scopes and Correlation Properties 07
8. Logging Configuration and Filtering 08
9. Log Enrichment and Sensitive Data 09
10. What Are Metrics? 10
11. Counters, Gauges and Histograms 11
12. Metric Dimensions and Cardinality 12
13. The Four Golden Signals and RED Method 13
14. What Is Distributed Tracing? 14
15. Trace, Span, Trace ID and Span ID 15
16. What Is OpenTelemetry? 16
17. OpenTelemetry Architecture 17
18. OpenTelemetry Signals and OTLP 18
19. Create a .NET 10 Web API for Observability 19
20. Configure OpenTelemetry in Program.cs 20
21. Instrument Application Code with ActivitySource 21
22. Create Custom Metrics in .NET 22
23. Export Logs with OpenTelemetry 23
24. Context Propagation and Log-Trace Correlation 24
25. Sampling and Telemetry Volume 25
26. What Is the OpenTelemetry Collector? 26
27. Collector Configuration Example 27
28. What Is Prometheus? 28
29. Prometheus Scraping and Targets 29
30. PromQL Fundamentals 30
31. Percentiles and Histograms 31
32. Alertmanager and Prometheus Alert Rules 32
33. What Is Grafana? 33
34. Build Useful Grafana Dashboards 34
35. Grafana Alerting Principles 35
36. Azure Application Insights 36
37. Use OpenTelemetry with Application Insights 37
38. Query Telemetry with Kusto Query Language 38
39. Monitoring SQL Server and EF Core Dependencies 39
40. Observability for Background Services and Queues 40
41. Health Checks vs Observability 41
42. SLI, SLO and SLA 42
43. Design Actionable Alerts and Runbooks 43
44. Sampling, Retention, Cost and Data Governance 44
45. Security and Privacy in Telemetry 45
46. End-to-End Architecture for a .NET Observability Stack 46
47. Mini Project: Observe an Orders API 47
48. Common Mistakes and Troubleshooting 48
49. Interview Questions and Answers 49
50. Official Documentation and Further Learning 50
51. Final Thought 51

01. Introduction to Observability

A production application can return HTTP 200 responses and still be unhealthy. Requests may be slower than usual, background jobs may be falling behind, SQL queries may be timing out, or one downstream service may be failing intermittently. Monitoring helps detect symptoms; observability helps investigate why they are happening.

An instrumented application emits signals; a backend stores and presents them so teams can understand system behavior.

This guide explains the concepts and a practical toolchain for ASP.NET Core applications: OpenTelemetry for instrumentation and transport, Prometheus for metrics, Grafana for dashboards and alerting, and Azure Application Insights for managed application monitoring.

02. Monitoring, Logging and Observability

Practice	Main question	Example
Logging	What happened in this event?	Order 481 failed validation
Monitoring	Is the system behaving within expected limits?	HTTP 5xx rate increased
Observability	Can we investigate an unfamiliar failure from available evidence?	Trace a slow request across API, SQL and payment service

These practices complement each other. A dashboard may tell you that latency increased; metrics show the pattern, traces expose the slow dependency, and logs provide the event details.

03. The Three Pillars: Logs, Metrics and Traces

Think of a checkout request: a metric shows that 95th-percentile latency rose; a trace shows the inventory call took 2.4 seconds; a correlated log explains that the downstream database connection pool was exhausted.

04. What Is Structured Logging?

Structured logging stores named properties rather than relying only on a formatted sentence. It makes fields searchable and filterable across log systems.

logger.LogInformation(
    "Order {OrderId} created for customer {CustomerId}",
    orderId,
    customerId);

Prefer message templates with stable property names. Avoid string interpolation for high-volume structured logs because it can make values harder for providers to capture as separate fields.

05. Log Levels and When to Use Them

Level	Meaning	Example
Trace	Very detailed diagnostic flow	Fine-grained internal steps
Debug	Information useful during development	Selected branch or state transition
Information	Normal significant application events	Background job completed
Warning	Unexpected situation that may need attention	Retry after transient dependency failure
Error	An operation failed	Order processing failed
Critical	Severe failure affecting the application	Service cannot start or continue safely

Choose levels consistently. Do not log every successful request at Error, and do not hide actionable failures at Debug.

06. Logging in ASP.NET Core

ASP.NET Core provides Microsoft.Extensions.Logging and dependency injection for logging. Inject ILogger<T> into controllers, services, and hosted workers.

public sealed class OrdersService(
    ILogger<OrdersService> logger)
{
    public async Task ProcessAsync(
        int orderId,
        CancellationToken cancellationToken)
    {
        logger.LogInformation(
            "Processing order {OrderId}", orderId);

        try
        {
            // Perform the operation.
            await Task.CompletedTask;
        }
        catch (Exception ex)
        {
            logger.LogError(ex,
                "Failed to process order {OrderId}", orderId);
            throw;
        }
    }
}

Log exceptions with the exception object so providers can capture type, stack trace, and inner exception details.

07. Scopes and Correlation Properties

A logging scope attaches shared properties to a group of log events. It is useful for values such as a request identifier, job identifier, or tenant identifier when it is safe to include them.

using (logger.BeginScope(
    new Dictionary<string, object>
    {
        ["OrderId"] = orderId,
        ["Operation"] = "OrderProcessing"
    }))
{
    logger.LogInformation("Starting processing");
    await ProcessOrderAsync(orderId);
}

Use stable, non-sensitive identifiers. Never place passwords, access tokens, payment data, or unnecessary personal information into scopes.

08. Logging Configuration and Filtering

Logging configuration controls providers, minimum levels, and category-specific filters. Keep local development verbose enough to diagnose issues, but use controlled levels in production.

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning",
      "Microsoft.EntityFrameworkCore.Database.Command": "Warning"
    }
  }
}

Configuration shown here is a starting point. Tune EF Core command logging and framework categories carefully; SQL text and parameter values may expose sensitive data or create high log volume.

09. Log Enrichment and Sensitive Data

Useful log fields include service name, environment, version, trace ID, span ID, request route, and a safe business identifier. Enrichment makes it easier to search across instances.

• Use allow-listed fields rather than dumping entire request or response objects.
• Redact authorization headers, cookies, secrets, and sensitive query parameters.
• Do not log raw personal or payment information unless a documented legal and security requirement exists.
• Set retention and access policies appropriate to the data classification.
• Protect log pipelines from injection by using structured fields and safe rendering.

10. What Are Metrics?

Metrics are numeric measurements recorded over time. They help answer questions about rate, volume, duration, saturation, and resource consumption. Metrics are efficient for dashboards and alerting because they summarize behavior without storing every individual event.

Common examples include request count, request duration, active requests, queue depth, CPU usage, memory, garbage collection, and dependency failure count.

11. Counters, Gauges and Histograms

Instrument	What it represents	Example
Counter	A value that only increases during its lifetime	Orders processed
Gauge / observable gauge	A value that can increase or decrease	Queue depth or active sessions
Histogram	Distribution of measured values	Request duration or payload size
UpDownCounter	A value that can move up and down	In-flight operations

Choose an instrument based on the meaning of the measurement, not on the chart you want to draw.

12. Metric Dimensions and Cardinality

Dimensions (attributes or labels) split a metric into time series. Useful dimensions include HTTP method, normalized route, status class, service, and deployment environment.

Cardinality risk: Avoid labels such as user ID, email, order ID, raw URL, or exception message. Each unique label combination can create a separate time series and increase storage and query cost.

Use low-cardinality dimensions for metrics. Put high-detail identifiers in traces or logs where access and retention can be controlled.

13. The Four Golden Signals and RED Method

The four golden signals are latency, traffic, errors, and saturation. The RED method applies a similar lens to request-driven services: Rate, Errors, and Duration.

• Latency / Duration: how long requests take, including percentiles.
• Traffic / Rate: how many requests arrive per second.
• Errors: failed requests or operations.
• Saturation: how close a constrained resource is to its limit.

For an ASP.NET Core API, monitor request rate, 5xx rate, p50/p95/p99 duration, active requests, thread-pool pressure, database connection pool behavior, and queue backlog.

14. What Is Distributed Tracing?

A distributed trace follows one logical operation across multiple processes or services. It is composed of spans. A span represents an operation and can contain timing, attributes, events, status, and links to related spans.

A trace shows parent-child relationships and timing across a request path; the diagram is illustrative, not a real trace.

15. Trace, Span, Trace ID and Span ID

• Trace ID: identifies the end-to-end trace.
• Span ID: identifies one operation within the trace.
• Parent span: records the operation that initiated a child span.
• Attributes: key-value details such as HTTP method or database system.
• Events: timestamped occurrences within a span.
• Status: indicates whether the operation succeeded or failed.

Trace context can propagate between services using W3C Trace Context headers such as traceparent. Avoid manually inventing IDs when standard instrumentation already manages context propagation.

16. What Is OpenTelemetry?

OpenTelemetry (OTel) is an open standard and collection of APIs, SDKs, instrumentation libraries, semantic conventions, and tools for generating and exporting telemetry. It is not itself a monitoring dashboard or storage backend.

In .NET, OpenTelemetry integrates with existing platform APIs: ILogger for logs, System.Diagnostics.Metrics for metrics, and ActivitySource/Activity for traces. This lets application code remain relatively independent of a particular observability vendor.

17. OpenTelemetry Architecture

The Collector is optional for simple setups but valuable for centralized processing, routing, batching, and backend changes.

18. OpenTelemetry Signals and OTLP

OpenTelemetry defines three primary signals: traces, metrics, and logs. The OpenTelemetry Protocol (OTLP) transports telemetry between SDKs, Collectors, and compatible backends. OTLP supports gRPC and HTTP/protobuf transports.

Use a common resource identity such as service name, service version, and deployment environment. Consistent resource attributes help group telemetry from different instances of the same service.

service.name = "orders-api"
service.version = "1.4.0"
deployment.environment.name = "production"

19. Create a .NET 10 Web API for Observability

Install the .NET 10 SDK and create a sample API. Use the current servicing patch of the .NET 10 SDK/runtime in your environment.

dotnet new webapi -n OrdersApi
cd OrdersApi
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Instrumentation.Runtime
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

For SQL client spans, add the matching OpenTelemetry.Instrumentation.SqlClient package. Package versions should be aligned and checked against the current OpenTelemetry .NET release notes before deployment.

20. Configure OpenTelemetry in Program.cs

This example configures traces and metrics for an ASP.NET Core application and exports them using OTLP. The OTLP endpoint can be supplied through environment variables or configuration.

using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);
var serviceName = "orders-api";

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(serviceName: serviceName,
                    serviceVersion: "1.0.0"))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddOtlpExporter())
    .WithMetrics(metrics => metrics
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddRuntimeInstrumentation()
        .AddOtlpExporter());

var app = builder.Build();

app.MapGet("/health", () => Results.Ok("Healthy"));
app.Run();

For production, configure endpoint, protocol, authentication, sampling, resource attributes, and shutdown behavior deliberately. Do not send telemetry to an unauthenticated public endpoint.

21. Instrument Application Code with ActivitySource

Automatic instrumentation covers common framework operations. Add manual spans when you need visibility into meaningful business operations that framework instrumentation cannot infer.

using System.Diagnostics;

public static class Telemetry
{
    public static readonly ActivitySource Source =
        new("OrdersApi.OrderProcessing");
}

public async Task ProcessOrderAsync(int orderId)
{
    using var activity =
        Telemetry.Source.StartActivity("ProcessOrder");

    activity?.SetTag("order.operation", "process");
    activity?.SetTag("order.id", orderId);

    await ValidateAndSaveAsync(orderId);
}

Register the source with the tracing provider using .AddSource("OrdersApi.OrderProcessing"). Be careful with business identifiers: only add them to traces if policy permits, and never add secrets.

22. Create Custom Metrics in .NET

Use Meter and instruments from System.Diagnostics.Metrics for application-specific measurements.

using System.Diagnostics.Metrics;

public static class OrderMetrics
{
    public static readonly Meter Meter =
        new("OrdersApi.Metrics");

    public static readonly Counter<long> OrdersCreated =
        Meter.CreateCounter<long>(
            "orders.created",
            unit: "{order}");

    public static readonly Histogram<double> ProcessingDuration =
        Meter.CreateHistogram<double>(
            "orders.processing.duration",
            unit: "s");
}

// When an order is created:
OrderMetrics.OrdersCreated.Add(1);

Register the meter with .AddMeter("OrdersApi.Metrics"). Keep metric attributes low-cardinality; do not attach an order ID or customer ID to every measurement.

23. Export Logs with OpenTelemetry

OpenTelemetry can bridge .NET logging into its logs pipeline. For example, an OTLP exporter can send logs to a Collector or compatible backend. Configure log filtering and resource identity consistently with traces and metrics.

using OpenTelemetry.Logs;

builder.Logging.AddOpenTelemetry(options =>
{
    options.IncludeFormattedMessage = true;
    options.IncludeScopes = true;
    options.ParseStateValues = true;
    options.AddOtlpExporter();
});

This logging pipeline is separate from the tracing and metrics builder registrations. Ensure the required OTLP exporter package is referenced and avoid initializing multiple vendor distributions that compete to configure the same pipeline.

24. Context Propagation and Log-Trace Correlation

Correlation connects a log event to the trace and span active when the event was written. In .NET, Activity.Current exposes the current activity, and OpenTelemetry logging integrations can include trace context in exported log records.

Across HTTP services, standard trace context headers carry trace identity. When an incoming request calls another service through an instrumented HttpClient, the child span is normally linked automatically. Avoid manually forwarding trace headers unless you are implementing a supported propagation boundary.

25. Sampling and Telemetry Volume

Sampling controls which traces are recorded and exported. Recording every trace may be useful in development, but high-volume production systems may need sampling to control cost and volume.

• AlwaysOn: record all traces; simple but potentially expensive.
• TraceIdRatioBased: sample a configured proportion of traces.
• ParentBased: respect the sampling decision made by the parent context.
• Tail sampling: decide after observing a trace, often in a Collector, so errors or slow traces can be retained selectively.

Sampling should preserve consistent decisions across a distributed trace. Metrics generally remain unsampled aggregates; do not assume trace sampling is equivalent to metric sampling.

26. What Is the OpenTelemetry Collector?

The OpenTelemetry Collector is a separate process that receives telemetry, processes it, and exports it to one or more destinations. Its pipelines are composed of receivers, processors, and exporters.

• Receivers: accept OTLP, Prometheus scrape data, and other supported formats.
• Processors: batch, filter, enrich, transform, or sample telemetry.
• Exporters: send telemetry to Prometheus-compatible endpoints, trace stores, logs backends, or OTLP destinations.

The Collector can centralize credentials and routing, reduce application coupling to vendors, and provide a common place for telemetry policy.

27. Collector Configuration Example

This illustrative Collector configuration receives OTLP and exposes metrics for Prometheus to scrape. The exact component availability depends on the Collector distribution you install; verify that your binary includes the configured components.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_mib: 512
  batch:

exporters:
  prometheus:
    endpoint: 0.0.0.0:9464
  debug:
    verbosity: basic

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheus, debug]
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [debug]

Production note: Bind telemetry ports to trusted network interfaces, apply authentication and TLS where required, and configure resource limits. A debug exporter is for diagnostics, not long-term trace storage.

28. What Is Prometheus?

Prometheus is an open-source monitoring system and time-series database. It commonly collects metrics by scraping HTTP endpoints at configured intervals, stores samples with labels, and provides PromQL for querying and alert rules.

Prometheus is primarily a metrics backend. It is not a general-purpose log store or distributed trace database. Pair it with appropriate log and trace backends when you need all three signals.

29. Prometheus Scraping and Targets

Prometheus periodically requests metrics from configured targets. A target may be an application exposing a Prometheus metrics endpoint, an exporter, or another Prometheus-compatible service.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "otel-collector"
    static_configs:
      - targets: ["otel-collector:9464"]

In a container network, use the service DNS name and internal port rather than assuming localhost points to another container.

30. PromQL Fundamentals

PromQL is Prometheus's query language. It can select time series, aggregate across labels, and calculate rates over time windows.

# Select a metric
http_server_request_duration_seconds_count

# Request rate over the last 5 minutes
rate(http_server_request_duration_seconds_count[5m])

# Aggregate request rate by service
sum by (service_name) (
  rate(http_server_request_duration_seconds_count[5m])
)

Metric names depend on the instrumentation and exporter translation. Inspect the actual series in your environment before copying a query into a dashboard.

31. Percentiles and Histograms

Latency percentiles help describe the experience of slower requests. The average can look healthy while a small but important group of requests is very slow.

# Example classic histogram query (metric names/labels vary)
histogram_quantile(
  0.95,
  sum by (le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

Use histogram buckets that fit your latency objectives. Understand whether your backend exports classic histogram buckets or native histograms and adapt the query accordingly.

32. Alertmanager and Prometheus Alert Rules

Prometheus alert rules evaluate expressions and create alerts when conditions remain true. Alertmanager groups, routes, deduplicates, silences, and sends notifications.

groups:
  - name: api-alerts
    rules:
      - alert: HighHttpErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum(rate(http_requests_total[5m])) > 0.05
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "HTTP 5xx rate is above 5%"

This is a teaching example. Validate metric names, zero-traffic behavior, label matching, and the intended evaluation window before using an alert in production.

33. What Is Grafana?

Grafana is an analytics and visualization platform. It connects to data sources such as Prometheus and supported log or trace systems, allowing teams to build dashboards, explore data, and configure alerting.

Grafana does not automatically collect every signal from an application. Instrumentation and storage backends must be configured, then Grafana queries those data sources.

34. Build Useful Grafana Dashboards

Design dashboards around operational questions rather than filling them with every available metric. A practical API overview might show:

• Request rate and traffic by route group.
• HTTP error rate and status code distribution.
• Latency percentiles such as p50, p95 and p99.
• CPU, memory, garbage collection and thread-pool indicators.
• Dependency latency and failure rates.
• Queue depth and background job processing delay.

Use consistent service and environment filters, sensible time ranges, clear units, and links from overview panels to detailed traces or logs.

35. Grafana Alerting Principles

Grafana can evaluate alert rules from supported data sources and route notifications through contact points and notification policies. Good alerts describe a user-impacting condition or a condition that requires a clear operator action.

• Alert on sustained symptoms, not every brief fluctuation.
• Use labels for service, environment, and severity.
• Include a runbook link and useful context in annotations.
• Group related alerts to avoid notification storms.
• Test notification routing and ownership regularly.

Dashboards are for exploration; alerts are for conditions that need attention. Do not turn every dashboard panel into a page.

36. Azure Application Insights

Application Insights is an application performance monitoring capability within Azure Monitor. It supports request and dependency telemetry, exceptions, traces, metrics, distributed transaction diagnostics, dashboards, and query-based investigation.

It is a managed option for teams already operating on Azure. OpenTelemetry provides a standard instrumentation approach, while Azure Monitor/Application Insights supplies managed ingestion, storage, querying, and visualization capabilities.

37. Use OpenTelemetry with Application Insights

For new .NET applications adopting OpenTelemetry for Azure Monitor, Microsoft's current guidance points to the Microsoft OpenTelemetry Distro package. Existing applications already using a supported Azure Monitor OpenTelemetry Distro should follow the relevant upgrade guidance rather than initializing two distributions together.

dotnet add package Microsoft.OpenTelemetry

Configure the Application Insights connection string through a secure environment variable such as APPLICATIONINSIGHTS_CONNECTION_STRING or a managed configuration provider. Follow the current Microsoft Learn setup for the exact SDK and exporter configuration that matches your application type.

Do not mix distributions: Do not initialize the Microsoft OpenTelemetry Distro and Azure Monitor OpenTelemetry Distro together in the same application. Choose the supported setup for the project and migration path.

38. Query Telemetry with Kusto Query Language

Application Insights data can be explored with Kusto Query Language (KQL). Table names and available columns depend on the workspace-based schema and telemetry configuration.

// Recent failed requests
requests
| where timestamp > ago(1h)
| where success == false
| project timestamp, name, resultCode,
          duration, operation_Id
| order by timestamp desc

// Slow requests by operation
requests
| where timestamp > ago(24h)
| summarize p95=percentile(duration, 95),
            total=count()
  by name
| order by p95 desc

Use operation IDs to correlate related telemetry. Apply time filters early and avoid querying unnecessarily broad time ranges.

39. Monitoring SQL Server and EF Core Dependencies

For a .NET application using SQL Server, monitor both application-level behavior and database health. Useful signals include query duration, dependency failures, connection pool pressure, timeout counts, deadlocks, CPU, storage latency, and blocking.

OpenTelemetry SQL client instrumentation can capture dependency spans for supported client libraries. Be cautious about capturing SQL statements and parameters: they may contain sensitive values and can create large telemetry payloads.

EF Core command logging is useful for local diagnostics, but production database observability should be designed with query sampling, access controls, and data redaction.

40. Observability for Background Services and Queues

HTTP request metrics alone do not describe a worker-based system. For hosted services and message consumers, instrument work completion, failure, retry, processing duration, queue depth, oldest message age, and dead-letter count.

For a queue consumer, create a trace span around each message operation and propagate trace context in message metadata where the messaging technology supports it. Avoid using message IDs as metric labels; they belong in controlled logs or traces.

41. Health Checks vs Observability

Health checks answer whether an instance or dependency is ready or alive according to defined criteria. They are useful for orchestrators and load balancers, but they are not a replacement for telemetry.

• Liveness: should the process be restarted?
• Readiness: should this instance receive traffic?
• Startup: has initialization completed?

A service can be ready while experiencing elevated latency or partial failures. Combine health endpoints with metrics, traces, logs, and synthetic checks.

42. SLI, SLO and SLA

An SLI is a measured indicator of service behavior, an SLO is a target for that indicator over a time window, and an SLA is a formal agreement that may include consequences when targets are missed.

Example: “99.9% of eligible API requests complete successfully over a rolling 30-day window” is an SLO-style target. Define eligibility, success, measurement source, and maintenance exclusions precisely.

Error budgets translate reliability targets into an operational decision: when too much of the budget is consumed, a team may prioritize reliability work over risky feature changes.

43. Design Actionable Alerts and Runbooks

An alert should answer: what is wrong, which service is affected, how severe is it, what user impact is likely, and what should the on-call engineer do next?

A runbook should include dashboard links, relevant queries, safe diagnostic steps, mitigation options, escalation contacts, and recovery verification. Keep it versioned with the service and test it during incident exercises.

Prefer alerts on symptoms such as sustained error-budget burn, high request failure rate, or queue age beyond an agreed threshold. Use infrastructure saturation alerts as supporting signals.

44. Sampling, Retention, Cost and Data Governance

Observability has operational cost. High-cardinality metrics, verbose logs, long trace retention, and duplicate instrumentation can increase ingestion and storage costs.

• Set retention by signal and investigation needs.
• Sample high-volume traces while preserving error and slow-path visibility where possible.
• Keep metric labels low-cardinality.
• Filter noisy logs at source where safe, not by hiding important failures.
• Set budgets, quotas, and telemetry volume alerts.
• Define data residency, access, retention, and deletion policies.

45. Security and Privacy in Telemetry

Telemetry can contain sensitive data even when it is not intended to. Request URLs, SQL statements, exception messages, user identifiers, and custom attributes should be reviewed before export.

• Redact secrets and personal data before data leaves the process.
• Use TLS and authenticated OTLP endpoints across trust boundaries.
• Restrict dashboard and query access by role.
• Separate production and non-production telemetry where needed.
• Audit access to sensitive telemetry and set retention limits.

46. End-to-End Architecture for a .NET Observability Stack

One possible architecture. Select storage and visualization backends according to your hosting, security, and operational requirements.

For a small system, direct OTLP export to a managed backend may be sufficient. For a multi-service environment, a Collector tier can centralize routing and processing. Prometheus, Grafana, and Application Insights are options—not mandatory components that must all be deployed together.

47. Mini Project: Observe an Orders API

Build a small .NET 10 Orders API and add an end-to-end observability workflow.

1. Create endpoints to create and retrieve orders.
2. Add structured logs with safe order and operation properties.
3. Enable ASP.NET Core and HttpClient instrumentation.
4. Add an ActivitySource span around order processing.
5. Add a counter for orders created and a histogram for processing duration.
6. Export telemetry through OTLP to a local Collector.
7. Expose Collector metrics for Prometheus to scrape.
8. Create a Grafana dashboard for request rate, error rate, and latency.
9. Configure one actionable alert and document its runbook.
10. Optionally connect the same application to Azure Monitor Application Insights using the supported Microsoft OpenTelemetry setup.

Test normal traffic, an intentional validation failure, a slow dependency, and a transient dependency error. Confirm that the resulting logs, metrics, and traces tell a consistent story.

48. Common Mistakes and Troubleshooting

Symptom	What to investigate
No telemetry arrives	Check SDK registration, exporter package, endpoint, protocol, network, and Collector receiver.
Metrics appear but traces do not	Check tracing pipeline, sampler, instrumentation source, and exporter routing.
Traces stop at a service boundary	Check HTTP/message context propagation and downstream instrumentation.
Duplicate spans or logs	Look for multiple instrumentation agents or duplicate SDK/provider initialization.
Prometheus target is down	Check target address, port, path, container network, and scrape endpoint.
Grafana panel is empty	Check data source, time range, metric name, labels, and query syntax.
Telemetry costs spike	Review cardinality, sampling, log volume, duplicate exporters, and retention.
Alert never fires	Validate expression, evaluation interval, labels, threshold, and pending duration.

49. Interview Questions and Answers

1. What is observability?

The ability to understand internal system behavior from emitted telemetry, especially when investigating unfamiliar problems.

2. What are the three pillars?

Logs, metrics, and distributed traces. They provide event detail, numeric trends, and request execution paths.

3. What is structured logging?

Logging named properties as fields rather than only writing a formatted text message.

4. What is a metric cardinality problem?

Too many unique label combinations create too many time series, increasing resource use and cost.

5. What is a trace and a span?

A trace represents an end-to-end operation; spans represent individual timed operations within it.

6. What is OpenTelemetry?

An open standard and set of APIs, SDKs, instrumentation, and tools for generating and exporting telemetry.

7. Is OpenTelemetry a backend?

No. It instruments and transports telemetry; storage and analysis are provided by compatible backends.

8. What does the OpenTelemetry Collector do?

It receives telemetry, processes it, and exports it to configured destinations.

9. How does Prometheus collect metrics?

Commonly by scraping metrics endpoints at configured intervals and storing time-series samples.

10. What is PromQL?

The query language used to select, aggregate, and calculate Prometheus time-series data.

11. What is Grafana used for?

Querying supported data sources, building dashboards, exploring telemetry, and configuring alerts.

12. What is Application Insights?

An Azure Monitor application performance monitoring capability for requests, dependencies, exceptions, traces, and related telemetry.

13. What is trace context propagation?

Passing trace identity across service boundaries, commonly using W3C Trace Context headers.

14. What is sampling?

Selecting which traces are recorded/exported to manage volume and cost while preserving useful diagnostic evidence.

15. What is the difference between SLI, SLO and SLA?

An SLI is a measurement, an SLO is a target for it, and an SLA is a formal service agreement.

50. Official Documentation and Further Learning

• OpenTelemetry .NET documentation
• .NET observability with OpenTelemetry
• OpenTelemetry Collector
• Prometheus documentation
• PromQL basics
• Grafana documentation
• Application Insights overview
• Configure OpenTelemetry in Application Insights
• .NET support policy

Always check current package release notes and backend compatibility before adopting versions in production.

51. Final Thought

Good observability is not about collecting the maximum amount of data. It is about collecting the right evidence, connecting it across services, and making it useful when something goes wrong.

Start with structured logs, service-level metrics, and request tracing. Adopt OpenTelemetry to keep instrumentation portable, choose storage and visualization tools that fit your environment, and build alerts around user impact and clear actions. Review telemetry quality, privacy, and cost as part of normal application engineering.

Skills Tutorials – Learn • Build • Grow Together

Skills Tutorials

Learn • Build • Grow Together