No API-to-API dependency
Ingestion and consolidation are independent boundaries. Reads must not depend on write API availability.
Architectural case study · public laboratory
Reliable asynchronous processing with visible failures and verifiable evidence.
dotnet-observability-lab is an educational reference architecture that makes the reasoning behind its design easy to inspect: problem → constraints → quality attributes → decisions → implementation → failures → observability → evidence.
This page provides a portfolio overview. The detailed technical documentation remains in the lab repository.
Problem
Accept a write now, consolidate it later, and preserve correctness when failures occur.
An API receives a value and must accept it durably and idempotently. Another service boundary must consolidate that value asynchronously, without direct API-to-API calls or applying the same business change twice when the broker redelivers a message.
The real difficulty is not the decimal value. It is the interval between persisting, publishing, consuming, acknowledging, and recovering — including cases where the previous step has an uncertain outcome.
Constraints and quality attributes
Decisions start with what must remain true under failure.
At-least-once delivery
RabbitMQ may redeliver. Duplicates are part of the contract and are treated as normal behavior.
Durable authoritative state
PostgreSQL protects idempotency, Outbox, Inbox, and aggregate state. Redis is only a best-effort shortcut.
Portable observability
OpenTelemetry defines traces, metrics, and logs. Aspire composes and displays the local environment but does not route business traffic.
ReliabilityConsistencyObservabilityOperabilityRecoverabilityEvolvability
Decisions and trade-offs
Patterns are responses to concrete failure modes.
Asynchronous messaging instead of synchronous calls
RabbitMQ decouples availability and latency between ingestion and consolidation.
Trade-off: eventual consistency and additional operational complexity.
ADR 0004 ↗Transactional Outbox
The value and pending event commit in one transaction; a worker publishes later with broker confirmation.
Trade-off: polling, retries, quarantine, and possible repeated publication.
Implementation ↗Durable consumer Inbox
Message identity and aggregate update are committed together in PostgreSQL.
Trade-off: every unique message requires durable work and a future retention policy.
Implementation ↗At-least-once + idempotency
The design does not claim exactly-once transport. It makes repetition safe at the business-effect level.
Trade-off: every consumer must model identity and duplication explicitly.
Event contract ↗PostgreSQL authoritative, Redis optional
Cache reduces duplicate HTTP work, but eviction, restart, or outage cannot alter correctness.
Trade-off: cache misses and failures may still hit the database.
ADR 0003 ↗OpenTelemetry as the standard
W3C context crosses the asynchronous boundary; Aspire is one local inspection surface.
Trade-off: propagation and instrumentation must be maintained explicitly.
ADR 0005 ↗Normal processing flow
The happy path exists, but it is not the whole architecture.
- POST /valuesidempotent request
- PostgreSQLatomic value + Outbox
- Outbox Workerclaim + confirmed publish
- RabbitMQat-least-once delivery
- Consolidation Workeratomic Inbox + aggregate
- GET /consolidatedlater independent read
Relevant failures
The value of the design becomes clear when the normal flow breaks.
Duplicate HTTP request
Same key + same value returns the original receipt; conflicting reuse returns 409. PostgreSQL uniqueness settles concurrent races.
Redelivered message
The Inbox prevents a second business effect. A message may be consumed again without incrementing the aggregate again.
Broker unavailable
The API can still accept and persist the write. Outbox work remains pending for later publication.
Redis unavailable
The fast path disappears, but correctness does not. Idempotency falls back to PostgreSQL.
Publish confirmed, commit not completed
The event may be published again with the same identity; the consumer safely handles the duplicate.
Consolidation failure
Inbox and aggregate roll back together. No false processed marker remains after a failed transaction.
Observability
Telemetry must follow the business operation across service boundaries.
The W3C context started by the request is persisted with the Outbox, resumed by the publisher, and propagated through AMQP headers to the consumer. Low-cardinality metrics distinguish acceptance, publication, processing, and duplication; structured logs capture relevant operational outcomes.
The goal isn't simply to have dashboards. It's to explain why an accepted write hasn't reached the read model and identify where processing is blocked or repeated.
See trace and metric tests ↗Public evidence
An architectural claim should point to something another engineer can inspect.
Canonical case studyFull traceability from decision to code, tests, docs, and CI.
Interactive LikeC4Versioned context, containers, components, and dynamic flow.
ADRsDecisions covering Aspire, PostgreSQL, Redis, RabbitMQ, and OpenTelemetry.
TestsIdempotency, Outbox, Inbox, contracts, and independent reads.
CIBuild, tests, coverage, ADR Guard, LikeC4, and architecture guardrails.
Deliberate limits
A reference architecture is not a production recipe.
The lab keeps one local PostgreSQL server with separate logical databases, does not prescribe a production deployment platform, and does not implement a complete bounded consumer retry/DLQ policy in v1. It also makes no event-ordering or exactly-once claim.
Those limits are part of the evidence: they show what was decided, what was simplified for educational use, and which choices would need to be reopened in a real system.
Explore the full repository ↗