Microservice chassis

This article covers implementation details of Reliability, Observability, and Cross-cutting concerns patterns in .NET Core APIs. Inspired by the patterns covered in https://microservices.io.

Overview:

Download the code from GIT Repo. Please read ReadMe to understand the code structure in the GIT Repo.

Distributed tracing:

Http header “X-Correlation-ID” is added for every outbound call in the consumer API and also in all downstream APIs.

The correlation ID is also used in API log events using Serilog. To add Correlation ID in every log entry, a .NET core custom middleware is implemented in CorrelationHeaderMiddleware.cs.

Implementing CorrelationID

Add the following Nuget packages in a project. Refer to Core.Web project in GIT repo.

Add Correlate middleware to the request execution pipeline in startup class.

Register the Correlate service in ConfigureServices of the startup class so that it is consumed across App Via DI.

Until now, we have configured the Correlate library, below code snippet shows how to use it while registering the services in DI which adds correlation ID for each outgoing HttpClinet request. Check out the full code in files { StartupBase.cs & Registry.cs}

Custom Correlation Middleware

The purpose of this middleware is to add a Correlation ID in every API log event. We are using Serilog LogContext object to add the Correlation ID for every log event in the API. The scope of the context object is the current logical thread. Have look at the middleware code here.

Download the code and invoke the “weatherforecast” endpoint DemoAPI project using postman with the “X-Correlation-ID” HTTP header. The below images depicts the behavior that we have discussed so far.

Capture the external Http/Https traffic going out of the client service using Fiddler as shown below. Every external HTTP/HTTPS call will be automatically sent with the “X-Correlation-ID” header received by the client application.

Correlation ID in the API log event:

Log events are written to the flat file in the bin directory as well as EKL stack. Even though we are logging as shown below in WeatherForecastController, it gets logged along with Correlation ID due to the Serilog LogContext properties in Custom Middleware.

Log event in a flat-file associated with the Correlation ID.

Transient fault handling:

In a distributed environment, calls to remote resources and services can fail due to transient faults, such as slow network connections and timeouts, or if resources are responding slowly or are temporarily unavailable. These faults typically correct themselves after a short time, and a robust cloud application should be prepared to handle them by using a strategy like a Retry pattern, Circuit breaker, etc. Polly policies are used for Wait &Retry, Circuit breaker, and timeout.
Wait & Retry: The “Retry pattern” enables an application to retry an operation in the expectation that the operation will eventually succeed. Wait before resending the request. Don’t Retry immediately as it will hamper downstream systems.

Circuit breaker: Stops all requests to faulty service. The Circuit Breaker pattern prevents an application from performing an operation that’s likely to fail. You can combine these two patterns in an application to improve resiliency.

Polly policies structure: The policies are wrapped one in another like the below format. PollyFallback=>Polly timeout=>Polly retry=>CircuitBreaker=> HTTP client request.

Implementation details:

Policies are defined in PolicyBuilder.cs as shown below

Policies are consumed in a startup.cs as follows.

Health Monitoring

You can add a health check endpoint to a microservice which checks internal dependencies of the microservice. You can configure the health check endpoint HTTP response code based on the internal dependencies status. For Ex. internal dependency is a DataBase, External services consumed in the microservice and RabbitMQ/SQS, etc. Refer below code/HealthChecksBuilder.cs for configuring health check endpoint.

ForRegister the health check-in startup class as follows.

The health check endpoints are used in Loadbalancers to route traffic only to the healthy instances of a microservice. The health check endpoint to the above product service will be as follows.

Centralized & Structured logging

Serilog is configured in Chassis and integrated with Correction ID. Please go through the Distributed tracing topic for additional details.

Externalized Application configuration

As per The 12-factor App guidance, Application configuration has to be separated from code. Refer this for a detailed explanation.

Exception handling & Error response structure

It is very important to plan for a consistent error response structure. Refer to this for additional details.

API Validation error structure is handled as follows in StartupBase.cs

and the structure of the above BadRequest error is as follows.

The above structure is similar to the HTTP 500 error is as shown below. If the same validation code is used across all microservices then consumers of the APIs can easily handler the error in a consistent way.

All exceptions are logged in a centralized logging system with unique id returned to the client as shown below.

HTTP Handler For Response Decompression

Http message handler is added to HttpClinet for response decompression in Registry.cs as shown below.

Conclusion:

Topics discussed in this article are common requirements in every API. Hope it helps you.