dotnet-structured-logging
Log pipeline design and operations for .NET distributed systems. Covers log aggregation architecture (ELK, Seq, Grafana
Loki), structured query patterns for each platform, log sampling and volume management strategies, PII scrubbing and
destructuring policies, and cross-service correlation beyond single-service log scopes. This skill addresses what
happens after log emission -- the pipeline, query, and operations layer.
Scope
- Log aggregation architecture (ELK, Seq, Grafana Loki)
- Structured query patterns per platform
- Log sampling and volume management strategies
- PII scrubbing and destructuring policies
- Cross-service correlation and distributed context
Out of scope
- Log emission mechanics (Serilog/NLog/MEL, LoggerMessage, sinks, OTel export) -- see [skill:dotnet-observability]
- Application configuration and options pattern -- see [skill:dotnet-csharp-configuration]
- Distributed tracing setup and trace context propagation -- see [skill:dotnet-observability]
Cross-references: [skill:dotnet-observability] for log emission, Serilog/MEL configuration, and OpenTelemetry logging
export, [skill:dotnet-csharp-configuration] for appsettings.json configuration patterns used in log pipeline setup.
Log Aggregation Architecture
Architecture Options
| Platform |
Ingest |
Storage |
Query |
Best for |
| ELK (Elasticsearch, Logstash, Kibana) |
Logstash / Filebeat |
Elasticsearch |
KQL in Kibana |
Large-scale, flexible schema, full-text search |
| Seq |
HTTP API / Serilog sink |
Built-in |
Seq signal expressions |
.NET-native, developer-friendly, structured queries |
| Grafana Loki |
Promtail / OTel Collector |
Loki (label-indexed) |
LogQL |
Cost-effective, Grafana ecosystem, label-based queries |
| Azure Monitor |
OTel Collector / Application Insights SDK |
Log Analytics workspace |
KQL (Kusto) |
Azure-native, integrated alerting, cost management |
Recommended Pipeline Patterns
Pattern 1: OTel Collector as central router
App (OTLP) --> OTel Collector --> Elasticsearch / Loki / Azure Monitor
|
+--> Sampling / filtering / PII scrub
```text
The OpenTelemetry Collector acts as a vendor-neutral log router. Applications emit logs via OTLP; the collector handles filtering, sampling, enrichment, and routing to one or more backends. This decouples applications from backend choice.
```yaml
# otel-collector-config.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
http:
endpoint: "0.0.0.0:4318"
processors:
batch:
timeout: 5s
send_batch_size: 1024
filter:
logs:
exclude:
match_type: strict
bodies:
- "Health check endpoint hit"
exporters:
elasticsearch:
endpoints: ["https://es-cluster:9200"]
logs_index: "app-logs"
loki:
endpoint: "http://loki:3100/loki/api/v1/push"
service:
pipelines:
logs:
receivers: [otlp]
processors: [batch, filter]
exporters: [elasticsearch, loki]
```text
**Pattern 2: Direct sink (smaller deployments)**
```text
App (Serilog) --> Seq / Elasticsearch sink
```text
For smaller systems or development environments, Serilog sinks write directly to the aggregation platform. This avoids the OTel Collector but couples the application to the backend.
### .NET Application OTLP Configuration
For .NET application-side OTLP log export configuration (`builder.Logging.AddOpenTelemetry()`), see [skill:dotnet-observability]. The OTLP endpoint is configured via environment variables (`OTEL_EXPORTER_OTLP_ENDPOINT`), keeping application code backend-agnostic.
---
## Structured Query Patterns
Structured logs store each property as a queryable field. The query syntax differs by platform but the concepts are consistent: filter by property name, value, severity, and time range.
### Kibana KQL (Elasticsearch / ELK)
```text
# Find errors for a specific order
level: "Error" AND OrderId: "abc-123"
# Find slow operations (custom Duration property)
Duration > 5000 AND ServiceName: "order-api"
# Wildcard on message template
message: "Failed to process*" AND NOT level: "Debug"
# Time-scoped with correlation
TraceId: "0af7651916cd43dd8448eb211c80319c" AND @timestamp >= "2025-01-15T10:00:00"
```text
### Seq Signal Expressions
```text
# Find errors for a specific order
@Level = 'Error' and OrderId = 'abc-123'
# Find slow operations
Duration > 5000 and Application = 'order-api'
# Free-text search combined with structured filter
@Message like '%timeout%' and @Level in ['Warning', 'Error']
# Correlation across services
TraceId = '0af7651916cd43dd8448eb211c80319c'
```text
Seq signals are saved queries that trigger alerts. Define signals for recurring patterns (e.g., "Payment failures > 10/min") and attach notification channels.
### Grafana LogQL (Loki)
```text
# Filter by labels then regex on log line
{service_name="order-api"} |= "Error" | json | OrderId="abc-123"
# Structured field extraction and filtering
{service_name="order-api"} | json | Duration > 5000
# Count errors per service over time (for dashboards)
sum(rate({service_name=~".+"} |= "Error" [5m])) by (service_name)
```json
### Azure Monitor KQL (Kusto)
```kusto
// Find errors for a specific order
traces
| where severityLevel >= 3
| where customDimensions.OrderId == "abc-123"
| order by timestamp desc
// Slow operations
traces
| where toint(customDimensions.Duration) > 5000
| where cloud_RoleName == "order-api"
// Cross-service correlation
union traces, exceptions
| where operation_Id == "0af7651916cd43dd8448eb211c80319c"
| order by timestamp asc
```text
---
## Log Sampling and Volume Management
High-throughput systems can generate millions of log events per minute. Without sampling, storage costs and query performance degrade rapidly.
### Sampling Strategies
| Strategy | How it works | Use when |
|----------|-------------|----------|
| **Head-based** | Decide to sample before processing | Consistent per-request; simple to implement |
| **Tail-based** | Decide to sample after processing | Keep all errors/slow requests, drop routine logs |
| **Level-based** | Sample by severity | Always keep Warning+, sample Debug/Info |
| **Dynamic** | Adjust rate based on volume | Handle traffic spikes without config changes |
### OTel Collector Log Filtering
The `filter` processor in the OTel Collector drops log records at the pipeline level before they reach exporters. Use it to exclude noisy low-severity logs and reduce storage volume.
Note: The `tail_sampling` processor operates on **traces** (spans), not logs. For log volume management, use the `filter` and `transform` processors instead.
```yaml
processors:
filter:
logs:
exclude:
match_type: regexp
# Drop Debug and Trace logs at the collector level
severity_texts: ["DEBUG", "TRACE"]
exclude:
match_type: strict
# Exclude health check noise
bodies:
- "Health check endpoint hit"
transform:
log_statements:
- context: log
conditions:
# Keep all Warning+ logs unconditionally
- severity_number >= SEVERITY_NUMBER_WARN
statements: []
```text
### Application-Level Sampling with Serilog
```csharp
// Serilog.Expressions package for conditional log filtering
builder.Host.UseSerilog((context, loggerConfiguration) =>
{
loggerConfiguration
.ReadFrom.Configuration(context.Configuration)
// Drop health check logs entirely
.Filter.ByExcluding("RequestPath = '/health/ready'")
// Sample Debug logs at 10%
.Filter.ByExcluding(
"@Level = 'Debug' and Hash(@i) % 10 != 0");
});
```text
**Key packages:**
```xml
<PackageReference Include="Serilog.Expressions" Version="5.*" />
```xml
### Volume Management Checklist
1. **Set retention policies** per index/stream (e.g., 30 days for Info, 90 days for Error)
2. **Use log level filtering** to suppress noisy framework categories at the source
3. **Exclude health check endpoints** from request logging
4. **Apply index lifecycle management** (ILM in Elasticsearch, retention policies in Loki)
5. **Monitor ingestion rates** and set budget alerts on storage costs
---
## PII Scrubbing and Destructuring Policies
Logs must not contain personally identifiable information (PII) in production. GDPR, HIPAA, and SOC 2 require that sensitive data is masked or excluded from log storage.
### Property-Level Masking with Enrichers
```csharp
// Enricher that masks known-sensitive properties on every log event
public sealed class PiiMaskingEnricher : ILogEventEnricher
{
private static readonly HashSet<string> s_sensitiveKeys = new(
StringComparer.OrdinalIgnoreCase)
{
"Email", "PhoneNumber", "IpAddress",
"CreditCard", "SSN", "Password"
};
public void Enrich(LogEvent logEvent, ILogEventPropertyFactory factory)
{
var propertiesToMask = logEvent.Properties
.Where(p => s_sensitiveKeys.Contains(p.Key))
.Select(p => p.Key)
.ToList();
foreach (var key in propertiesToMask)
{
logEvent.AddOrUpdateProperty(
factory.CreateProperty(key, "***REDACTED***"));
}
}
}
// Registration
loggerConfiguration.Enrich.With<PiiMaskingEnricher>();
```text
### OTel Collector Attribute Processing
```yaml
processors:
attributes:
actions:
# Mask email addresses using regex
- key: user.email
action: update
value: "***@redacted.com"
# Remove sensitive attributes entirely
- key: http.request.header.authorization
action: delete
- key: user.password
action: delete
```text
### PII Scrubbing Checklist
1. **Identify PII fields** -- email, phone, IP, SSN, credit card, auth tokens, cookies
2. **Apply at the earliest point** -- enricher or OTel processor, not at query time
3. **Audit log templates** -- ensure structured log templates do not capture PII as named properties
4. **Test with compliance team** -- validate scrubbing rules against regulatory requirements
5. **Use separate retention** for audit logs that legitimately require PII (with encryption at rest)
---
## Cross-Service Correlation
In distributed systems, a single user request may traverse multiple services. Correlation enables tracing a request across all services and reconstructing the full event timeline.
### W3C Trace Context Correlation
The primary correlation mechanism is the W3C `traceparent` header, which propagates automatically through `HttpClient` when OpenTelemetry instrumentation is configured (see [skill:dotnet-observability]). All log events emitted within a traced request include `TraceId` and `SpanId` properties.
```csharp
// Query all logs for a distributed operation across services
// In Seq:
TraceId = '0af7651916cd43dd8448eb211c80319c'
// In Kibana:
TraceId: "0af7651916cd43dd8448eb211c80319c"
// In Azure Monitor:
traces | where operation_Id == "0af7651916cd43dd8448eb211c80319c"
```text
### Custom Correlation IDs
When trace context is insufficient (e.g., async workflows spanning message queues, batch jobs, or external system callbacks), add custom correlation IDs:
```csharp
// Propagate a business correlation ID through Serilog LogContext
public sealed class CorrelationIdMiddleware(RequestDelegate next)
{
private const string CorrelationHeader = "X-Correlation-Id";
public async Task InvokeAsync(HttpContext context)
{
var correlationId = context.Request.Headers[CorrelationHeader]
.FirstOrDefault() ?? Guid.NewGuid().ToString("N");
context.Response.Headers[CorrelationHeader] = correlationId;
using (LogContext.PushProperty("CorrelationId", correlationId))
{
await next(context);
}
}
}
// Registration
app.UseMiddleware<CorrelationIdMiddleware>();
```text
### Message Queue Correlation
For asynchronous messaging (Azure Service Bus, RabbitMQ), propagate correlation through message properties:
```csharp
// Producer -- attach correlation to message
var message = new ServiceBusMessage(payload)
{
CorrelationId = Activity.Current?.TraceId.ToString()
?? Guid.NewGuid().ToString("N"),
ApplicationProperties =
{
["BusinessCorrelationId"] = orderId.ToString()
}
};
// Consumer -- restore correlation in log scope
processor.ProcessMessageAsync += async args =>
{
using var scope = logger.BeginScope(new Dictionary<string, object>
{
["CorrelationId"] = args.Message.CorrelationId,
["BusinessCorrelationId"] =
args.Message.ApplicationProperties["BusinessCorrelationId"]
});
logger.LogInformation("Processing message {MessageId}", args.Message.MessageId);
await ProcessAsync(args.Message, args.CancellationToken);
};
```text
### Correlation Best Practices
| Practice | Rationale |
|----------|-----------|
| Always include `TraceId` in log output | Enables log-to-trace joins in observability platforms |
| Use `CorrelationId` for business flows | Survives async gaps where trace context resets |
| Store correlation IDs in message headers | Enables end-to-end tracing through queues |
| Include correlation in error responses | Enables support teams to look up the full trace |
| Use Serilog `LogContext.PushProperty` or MEL `BeginScope` | Automatically attaches to all log events in scope |
---
## Agent Gotchas
1. **Do not conflate log emission with log pipeline** -- this skill covers pipeline, query, and operations. For Serilog/MEL configuration, enrichers, sink registration, and source-generated LoggerMessage, see [skill:dotnet-observability].
2. **Do not store PII in production logs** -- apply masking enrichers or OTel processor rules at the pipeline level. Redacting after storage is insufficient for compliance.
3. **Do not skip log sampling for high-throughput services** -- unsampled Debug/Info logs in a service handling thousands of requests per second will overwhelm storage and degrade query performance. Use tail-based sampling to keep all errors and slow requests.
4. **Do not hardcode aggregation platform endpoints in application code** -- use environment variables (`OTEL_EXPORTER_OTLP_ENDPOINT`) or configuration so the same image works across environments.
5. **Do not rely solely on TraceId for business correlation** -- trace context resets at async boundaries (message queues, scheduled jobs). Add explicit business correlation IDs for workflows that span these boundaries.
6. **Do not forget retention policies** -- logs without retention policies accumulate indefinitely, increasing costs and slowing queries. Set per-severity retention (e.g., 30 days for Info, 90 days for Error).
---
## References
- [OpenTelemetry Collector configuration](https://opentelemetry.io/docs/collector/configuration/)
- [Seq documentation](https://docs.datalust.co/docs)
- [Seq signal expressions](https://docs.datalust.co/docs/the-seq-query-language)
- [Grafana Loki LogQL](https://grafana.com/docs/loki/latest/query/)
- [Elasticsearch KQL syntax](https://www.elastic.co/guide/en/kibana/current/kuery-query.html)
- [Serilog.Expressions](https://github.com/serilog/serilog-expressions)
- [Serilog PII masking](https://github.com/serilog/serilog/wiki/Structured-Data#masking-sensitive-data)
- [Azure Monitor KQL reference](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/)
- [W3C Trace Context specification](https://www.w3.org/TR/trace-context/)
Code Navigation (Serena MCP)
Primary approach: Use Serena symbol operations for efficient code navigation:
- Find definitions:
serena_find_symbol instead of text search
- Understand structure:
serena_get_symbols_overview for file organization
- Track references:
serena_find_referencing_symbols for impact analysis
- Precise edits:
serena_replace_symbol_body for clean modifications
When to use Serena vs traditional tools:
- Use Serena: Navigation, refactoring, dependency analysis, precise edits
- Use Read/Grep: Reading full files, pattern matching, simple text operations
- Fallback: If Serena unavailable, traditional tools work fine
Example workflow:
# Instead of:
Read: src/Services/OrderService.cs
Grep: "public void ProcessOrder"
# Use:
serena_find_symbol: "OrderService/ProcessOrder"
serena_get_symbols_overview: "src/Services/OrderService.cs"
1---2name: dotnet-structured-logging-83category: developer-experience4subcategory: cli5description: Designs log pipelines. Aggregation, structured queries, sampling, PII scrubbing, correlation.6license: MIT7targets: ['*']8tags: [architecture, dotnet, skill]9version: '0.0.1'10author: 'dotnet-agent-harness'11invocable: true12claudecode:13 allowed-tools: ['Read', 'Grep', 'Glob', 'Bash', 'Write', 'Edit']14codexcli:15 short-description: '.NET skill guidance for architecture tasks'16opencode:17 allowed-tools: ['Read', 'Grep', 'Glob', 'Bash', 'Write', 'Edit']18copilot: {}19geminicli: {}20antigravity: {}21metadata:22 source: https://skillmd.com/skills/rudironsoni/dotnet-structured-logging-823---2425# dotnet-structured-logging2627Log pipeline design and operations for .NET distributed systems. Covers log aggregation architecture (ELK, Seq, Grafana28Loki), structured query patterns for each platform, log sampling and volume management strategies, PII scrubbing and29destructuring policies, and cross-service correlation beyond single-service log scopes. This skill addresses what30happens _after_ log emission -- the pipeline, query, and operations layer.3132## Scope3334- Log aggregation architecture (ELK, Seq, Grafana Loki)35- Structured query patterns per platform36- Log sampling and volume management strategies37- PII scrubbing and destructuring policies38- Cross-service correlation and distributed context3940## Out of scope4142- Log emission mechanics (Serilog/NLog/MEL, LoggerMessage, sinks, OTel export) -- see [skill:dotnet-observability]43- Application configuration and options pattern -- see [skill:dotnet-csharp-configuration]44- Distributed tracing setup and trace context propagation -- see [skill:dotnet-observability]4546Cross-references: [skill:dotnet-observability] for log emission, Serilog/MEL configuration, and OpenTelemetry logging47export, [skill:dotnet-csharp-configuration] for appsettings.json configuration patterns used in log pipeline setup.4849---5051## Log Aggregation Architecture5253### Architecture Options5455| Platform | Ingest | Storage | Query | Best for |56| ----------------------------------------- | ----------------------------------------- | ----------------------- | ---------------------- | ------------------------------------------------------ |57| **ELK** (Elasticsearch, Logstash, Kibana) | Logstash / Filebeat | Elasticsearch | KQL in Kibana | Large-scale, flexible schema, full-text search |58| **Seq** | HTTP API / Serilog sink | Built-in | Seq signal expressions | .NET-native, developer-friendly, structured queries |59| **Grafana Loki** | Promtail / OTel Collector | Loki (label-indexed) | LogQL | Cost-effective, Grafana ecosystem, label-based queries |60| **Azure Monitor** | OTel Collector / Application Insights SDK | Log Analytics workspace | KQL (Kusto) | Azure-native, integrated alerting, cost management |6162### Recommended Pipeline Patterns6364#### Pattern 1: OTel Collector as central router6566````text6768App (OTLP) --> OTel Collector --> Elasticsearch / Loki / Azure Monitor69 |70 +--> Sampling / filtering / PII scrub7172```text7374The OpenTelemetry Collector acts as a vendor-neutral log router. Applications emit logs via OTLP; the collector handles filtering, sampling, enrichment, and routing to one or more backends. This decouples applications from backend choice.7576```yaml7778# otel-collector-config.yaml79receivers:80 otlp:81 protocols:82 grpc:83 endpoint: "0.0.0.0:4317"84 http:85 endpoint: "0.0.0.0:4318"8687processors:88 batch:89 timeout: 5s90 send_batch_size: 102491 filter:92 logs:93 exclude:94 match_type: strict95 bodies:96 - "Health check endpoint hit"9798exporters:99 elasticsearch:100 endpoints: ["https://es-cluster:9200"]101 logs_index: "app-logs"102 loki:103 endpoint: "http://loki:3100/loki/api/v1/push"104105service:106 pipelines:107 logs:108 receivers: [otlp]109 processors: [batch, filter]110 exporters: [elasticsearch, loki]111112```text113114**Pattern 2: Direct sink (smaller deployments)**115116```text117118App (Serilog) --> Seq / Elasticsearch sink119120```text121122For smaller systems or development environments, Serilog sinks write directly to the aggregation platform. This avoids the OTel Collector but couples the application to the backend.123124### .NET Application OTLP Configuration125126For .NET application-side OTLP log export configuration (`builder.Logging.AddOpenTelemetry()`), see [skill:dotnet-observability]. The OTLP endpoint is configured via environment variables (`OTEL_EXPORTER_OTLP_ENDPOINT`), keeping application code backend-agnostic.127128---129130## Structured Query Patterns131132Structured logs store each property as a queryable field. The query syntax differs by platform but the concepts are consistent: filter by property name, value, severity, and time range.133134### Kibana KQL (Elasticsearch / ELK)135136```text137138# Find errors for a specific order139level: "Error" AND OrderId: "abc-123"140141# Find slow operations (custom Duration property)142Duration > 5000 AND ServiceName: "order-api"143144# Wildcard on message template145message: "Failed to process*" AND NOT level: "Debug"146147# Time-scoped with correlation148TraceId: "0af7651916cd43dd8448eb211c80319c" AND @timestamp >= "2025-01-15T10:00:00"149150```text151152### Seq Signal Expressions153154```text155156# Find errors for a specific order157@Level = 'Error' and OrderId = 'abc-123'158159# Find slow operations160Duration > 5000 and Application = 'order-api'161162# Free-text search combined with structured filter163@Message like '%timeout%' and @Level in ['Warning', 'Error']164165# Correlation across services166TraceId = '0af7651916cd43dd8448eb211c80319c'167168```text169170Seq signals are saved queries that trigger alerts. Define signals for recurring patterns (e.g., "Payment failures > 10/min") and attach notification channels.171172### Grafana LogQL (Loki)173174```text175176# Filter by labels then regex on log line177{service_name="order-api"} |= "Error" | json | OrderId="abc-123"178179# Structured field extraction and filtering180{service_name="order-api"} | json | Duration > 5000181182# Count errors per service over time (for dashboards)183sum(rate({service_name=~".+"} |= "Error" [5m])) by (service_name)184185```json186187### Azure Monitor KQL (Kusto)188189```kusto190191// Find errors for a specific order192traces193| where severityLevel >= 3194| where customDimensions.OrderId == "abc-123"195| order by timestamp desc196197// Slow operations198traces199| where toint(customDimensions.Duration) > 5000200| where cloud_RoleName == "order-api"201202// Cross-service correlation203union traces, exceptions204| where operation_Id == "0af7651916cd43dd8448eb211c80319c"205| order by timestamp asc206207```text208209---210211## Log Sampling and Volume Management212213High-throughput systems can generate millions of log events per minute. Without sampling, storage costs and query performance degrade rapidly.214215### Sampling Strategies216217| Strategy | How it works | Use when |218|----------|-------------|----------|219| **Head-based** | Decide to sample before processing | Consistent per-request; simple to implement |220| **Tail-based** | Decide to sample after processing | Keep all errors/slow requests, drop routine logs |221| **Level-based** | Sample by severity | Always keep Warning+, sample Debug/Info |222| **Dynamic** | Adjust rate based on volume | Handle traffic spikes without config changes |223224### OTel Collector Log Filtering225226The `filter` processor in the OTel Collector drops log records at the pipeline level before they reach exporters. Use it to exclude noisy low-severity logs and reduce storage volume.227228Note: The `tail_sampling` processor operates on **traces** (spans), not logs. For log volume management, use the `filter` and `transform` processors instead.229230```yaml231232processors:233 filter:234 logs:235 exclude:236 match_type: regexp237 # Drop Debug and Trace logs at the collector level238 severity_texts: ["DEBUG", "TRACE"]239 exclude:240 match_type: strict241 # Exclude health check noise242 bodies:243 - "Health check endpoint hit"244 transform:245 log_statements:246 - context: log247 conditions:248 # Keep all Warning+ logs unconditionally249 - severity_number >= SEVERITY_NUMBER_WARN250 statements: []251252```text253254### Application-Level Sampling with Serilog255256```csharp257258// Serilog.Expressions package for conditional log filtering259builder.Host.UseSerilog((context, loggerConfiguration) =>260{261 loggerConfiguration262 .ReadFrom.Configuration(context.Configuration)263 // Drop health check logs entirely264 .Filter.ByExcluding("RequestPath = '/health/ready'")265 // Sample Debug logs at 10%266 .Filter.ByExcluding(267 "@Level = 'Debug' and Hash(@i) % 10 != 0");268});269270```text271272**Key packages:**273274```xml275276<PackageReference Include="Serilog.Expressions" Version="5.*" />277278```xml279280### Volume Management Checklist2812821. **Set retention policies** per index/stream (e.g., 30 days for Info, 90 days for Error)2832. **Use log level filtering** to suppress noisy framework categories at the source2843. **Exclude health check endpoints** from request logging2854. **Apply index lifecycle management** (ILM in Elasticsearch, retention policies in Loki)2865. **Monitor ingestion rates** and set budget alerts on storage costs287288---289290## PII Scrubbing and Destructuring Policies291292Logs must not contain personally identifiable information (PII) in production. GDPR, HIPAA, and SOC 2 require that sensitive data is masked or excluded from log storage.293294### Property-Level Masking with Enrichers295296```csharp297298// Enricher that masks known-sensitive properties on every log event299public sealed class PiiMaskingEnricher : ILogEventEnricher300{301 private static readonly HashSet<string> s_sensitiveKeys = new(302 StringComparer.OrdinalIgnoreCase)303 {304 "Email", "PhoneNumber", "IpAddress",305 "CreditCard", "SSN", "Password"306 };307308 public void Enrich(LogEvent logEvent, ILogEventPropertyFactory factory)309 {310 var propertiesToMask = logEvent.Properties311 .Where(p => s_sensitiveKeys.Contains(p.Key))312 .Select(p => p.Key)313 .ToList();314315 foreach (var key in propertiesToMask)316 {317 logEvent.AddOrUpdateProperty(318 factory.CreateProperty(key, "***REDACTED***"));319 }320 }321}322323// Registration324loggerConfiguration.Enrich.With<PiiMaskingEnricher>();325326```text327328### OTel Collector Attribute Processing329330```yaml331332processors:333 attributes:334 actions:335 # Mask email addresses using regex336 - key: user.email337 action: update338 value: "***@redacted.com"339 # Remove sensitive attributes entirely340 - key: http.request.header.authorization341 action: delete342 - key: user.password343 action: delete344345```text346347### PII Scrubbing Checklist3483491. **Identify PII fields** -- email, phone, IP, SSN, credit card, auth tokens, cookies3502. **Apply at the earliest point** -- enricher or OTel processor, not at query time3513. **Audit log templates** -- ensure structured log templates do not capture PII as named properties3524. **Test with compliance team** -- validate scrubbing rules against regulatory requirements3535. **Use separate retention** for audit logs that legitimately require PII (with encryption at rest)354355---356357## Cross-Service Correlation358359In distributed systems, a single user request may traverse multiple services. Correlation enables tracing a request across all services and reconstructing the full event timeline.360361### W3C Trace Context Correlation362363The primary correlation mechanism is the W3C `traceparent` header, which propagates automatically through `HttpClient` when OpenTelemetry instrumentation is configured (see [skill:dotnet-observability]). All log events emitted within a traced request include `TraceId` and `SpanId` properties.364365```csharp366367// Query all logs for a distributed operation across services368// In Seq:369TraceId = '0af7651916cd43dd8448eb211c80319c'370371// In Kibana:372TraceId: "0af7651916cd43dd8448eb211c80319c"373374// In Azure Monitor:375traces | where operation_Id == "0af7651916cd43dd8448eb211c80319c"376377```text378379### Custom Correlation IDs380381When trace context is insufficient (e.g., async workflows spanning message queues, batch jobs, or external system callbacks), add custom correlation IDs:382383```csharp384385// Propagate a business correlation ID through Serilog LogContext386public sealed class CorrelationIdMiddleware(RequestDelegate next)387{388 private const string CorrelationHeader = "X-Correlation-Id";389390 public async Task InvokeAsync(HttpContext context)391 {392 var correlationId = context.Request.Headers[CorrelationHeader]393 .FirstOrDefault() ?? Guid.NewGuid().ToString("N");394395 context.Response.Headers[CorrelationHeader] = correlationId;396397 using (LogContext.PushProperty("CorrelationId", correlationId))398 {399 await next(context);400 }401 }402}403404// Registration405app.UseMiddleware<CorrelationIdMiddleware>();406407```text408409### Message Queue Correlation410411For asynchronous messaging (Azure Service Bus, RabbitMQ), propagate correlation through message properties:412413```csharp414415// Producer -- attach correlation to message416var message = new ServiceBusMessage(payload)417{418 CorrelationId = Activity.Current?.TraceId.ToString()419 ?? Guid.NewGuid().ToString("N"),420 ApplicationProperties =421 {422 ["BusinessCorrelationId"] = orderId.ToString()423 }424};425426// Consumer -- restore correlation in log scope427processor.ProcessMessageAsync += async args =>428{429 using var scope = logger.BeginScope(new Dictionary<string, object>430 {431 ["CorrelationId"] = args.Message.CorrelationId,432 ["BusinessCorrelationId"] =433 args.Message.ApplicationProperties["BusinessCorrelationId"]434 });435436 logger.LogInformation("Processing message {MessageId}", args.Message.MessageId);437 await ProcessAsync(args.Message, args.CancellationToken);438};439440```text441442### Correlation Best Practices443444| Practice | Rationale |445|----------|-----------|446| Always include `TraceId` in log output | Enables log-to-trace joins in observability platforms |447| Use `CorrelationId` for business flows | Survives async gaps where trace context resets |448| Store correlation IDs in message headers | Enables end-to-end tracing through queues |449| Include correlation in error responses | Enables support teams to look up the full trace |450| Use Serilog `LogContext.PushProperty` or MEL `BeginScope` | Automatically attaches to all log events in scope |451452---453454## Agent Gotchas4554561. **Do not conflate log emission with log pipeline** -- this skill covers pipeline, query, and operations. For Serilog/MEL configuration, enrichers, sink registration, and source-generated LoggerMessage, see [skill:dotnet-observability].4572. **Do not store PII in production logs** -- apply masking enrichers or OTel processor rules at the pipeline level. Redacting after storage is insufficient for compliance.4583. **Do not skip log sampling for high-throughput services** -- unsampled Debug/Info logs in a service handling thousands of requests per second will overwhelm storage and degrade query performance. Use tail-based sampling to keep all errors and slow requests.4594. **Do not hardcode aggregation platform endpoints in application code** -- use environment variables (`OTEL_EXPORTER_OTLP_ENDPOINT`) or configuration so the same image works across environments.4605. **Do not rely solely on TraceId for business correlation** -- trace context resets at async boundaries (message queues, scheduled jobs). Add explicit business correlation IDs for workflows that span these boundaries.4616. **Do not forget retention policies** -- logs without retention policies accumulate indefinitely, increasing costs and slowing queries. Set per-severity retention (e.g., 30 days for Info, 90 days for Error).462463---464465## References466467- [OpenTelemetry Collector configuration](https://opentelemetry.io/docs/collector/configuration/)468- [Seq documentation](https://docs.datalust.co/docs)469- [Seq signal expressions](https://docs.datalust.co/docs/the-seq-query-language)470- [Grafana Loki LogQL](https://grafana.com/docs/loki/latest/query/)471- [Elasticsearch KQL syntax](https://www.elastic.co/guide/en/kibana/current/kuery-query.html)472- [Serilog.Expressions](https://github.com/serilog/serilog-expressions)473- [Serilog PII masking](https://github.com/serilog/serilog/wiki/Structured-Data#masking-sensitive-data)474- [Azure Monitor KQL reference](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/query/)475- [W3C Trace Context specification](https://www.w3.org/TR/trace-context/)476````477478## Code Navigation (Serena MCP)479480**Primary approach:** Use Serena symbol operations for efficient code navigation:4814821. **Find definitions**: `serena_find_symbol` instead of text search4832. **Understand structure**: `serena_get_symbols_overview` for file organization4843. **Track references**: `serena_find_referencing_symbols` for impact analysis4854. **Precise edits**: `serena_replace_symbol_body` for clean modifications486487**When to use Serena vs traditional tools:**488489- **Use Serena**: Navigation, refactoring, dependency analysis, precise edits490- **Use Read/Grep**: Reading full files, pattern matching, simple text operations491- **Fallback**: If Serena unavailable, traditional tools work fine492493**Example workflow:**494495```text496# Instead of:497Read: src/Services/OrderService.cs498Grep: "public void ProcessOrder"499500# Use:501serena_find_symbol: "OrderService/ProcessOrder"502serena_get_symbols_overview: "src/Services/OrderService.cs"503```