Microservices Observability with Jaeger
As many applications have started the journey of modernisation from monolithic to microservices architecture, observability tool has now become essential to troubleshoot and understand issues happened in application. Business logic will no longer sit within the monolithic code, but has now spread across multiple microservices which can be hard to trace if we don't have a proper tool.
Jaeger (https://www.jaegertracing.io/) is an open source project developed by Uber Technologies which is meant for us to perform distributed tracing across APIs. In today's sharing, I am going to implement Jaeger distributed tracing via microservices developed using Node.JS. I am going to use threadfin-http ( https://github.com/joshua-lim/threadfin-http) ; a simple Node.JS http server to illustrate this. If you are not a developer, just want to explore the features of Jaeger without doing much coding, you can download the hot r.o.d example (https://github.com/jaegertracing/jaeger/tree/master/examples/hotrod) from github, which can give you a feel of how distributed tracing can be done through Jaeger.
Without further delay, let me share the example that I am going to show in today's sharing. In today's scenario, I am going to develop a student result calculation service which consist of 3 components developed by Node.JS threadfin-http library (Please refer to the below diagram - Flow of Calculate Result microservices):
- Calculate result API
- English result API
- Mathematics result API
As what you can see from the below diagram, Calculate Result (calc-result-service) will be the main API to receive incoming request from UI / other application. This API will calls English (english-result-service) and Mathematics result (maths-result-service) API to compute the average score of English and Mathematics based on the exam and assignment marks. After Calculate Result service received the average score of English and Mathematics, the API will call a internal sub process to compute the final result and GPA of an individual student. The business logic will flow across microservices and sub-process and they can be easily monitored through Jaeger.
Flow of the Calculate Result microservices
Before creating the 3 Node.JS microservices, we need to first setup Jaeger. To have a simple setup, I decided to ease the work by using Jaeger all-in-one Docker image. If you already have docker installed in your machine, you can just execute "docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14268:14268 -p 14250:14250 -p 9411:9411 jaegertracing/all-in-one:1.24" to start a Jaeger container. Launch http://localhost:16686/ through browser and you should see the UI (please see below) after Jaeger container successfully started.
Jaeger UI Screen
As we need to create 3 microservices, I am going to create 3 folders (please see below) to separately run the Node.JS instances (Calculate Result API, English Result API and Mathematics Result API) at port 8885, 8886 and 8887.
To start the 3 Node.JS application at 3 different ports (8885, 8886 and 8887), we need 3 config.json (you may find the sample config.json at threadfin-http github test folder) in these 3 folders and set the value of server_port attribute accordingly.
You also need a main program (app.js) at each folder to start the Node.JS application. Please see below for the source code of app.js. (You should modify the path of config.json according to the folder name)
source code of app.js
We can now install the dependency libraries in these 3 folders. Besides threadfin-http (npm install threadfin-http) core library, We also need 2 additional libraries - jaeger-client (https://www.npmjs.com/package/jaeger-client) and opentracing (https://www.npmjs.com/package/opentracking) to support the implementation. Simply run "npm install <library>", you should be able to download the neccessaries library into the folders.
Calculate Result - threadfin-calculate API
After we are done with the pre-requisite, we can start to build our first Node.JS microservices - threadfin-calculate API. We will create a threadfin-http handler file called calculate_handler.js. To effective track the API in Jaeger, we need to use Jaeger client objects to define "serviceName". (please see below) In this example, we are going to call our service 'calc-result-service'. Besides the "serviceName", We also need to define the "collectorEndpoint" (localhost://localhost:14268/api/traces), so jaeger-client can send the statistic back to Jaeger collector after every API execution.
jaeger config in calculate_handler.js
Function process is the core implementation of the API logic. This is required by every threadfin-http handler. In function process, we need to define the start and finish of span which represents a unit of work in Jaeger. (We called the main span "incoming_request"). In addition, we should also use "addTags" method to include more information such as HTTP URL and Method so the trace can be effectively search in Jaeger UI. (Please see below)
As the tracing need to happen across multiple APIs, we need to create a trace context using tracer.inject(span, opentracing.FORMAT_HTTP_HEADERS, traceContext) method to inject the tracer information to traceContext object.
Span, addTags and traceContext in calculate_handler.js
We will then place the traceContext object to HTTP header. This will allow the trace references available for the subsequent APIs which will be called later. In function Process, we will make 2 API calls; one to English result (http://localhost:8886/calc-eng) and another to Mathematics result (http://localhost:8887/calc-maths) microservices. In this example, we will use another 3rd party library - Request to simplify the implementation (You may download the library by executing "npm install request") of API call. After we received the result from both APIs, we will then invoke a sub process called calcFinalResult to compute the final result and GPA.
API calls in calculate_handler.js
calcFinalResult is a simple function in calculate_handler.js. To allow Jaeger to trace the sub-process, we included a child span and linked it as a child of the original span ( childOf: span ). Similar to the main span, we need to call startSpan and finish method to track the start and end time of the invocation.
calcFinalResult in calculate_handler.js
English Result - threadfin-calc-eng API & Maths Result - threadfin-calc-maths API
After we created Calculate Result API, we can proceed to create English and Maths Result APIs. threadfin-calc-eng and threadfin-calc-maths are the microservices which responsible for the computation of English and Mathematics result. To differentiate them from the core service, we labeled them as 'english-result-service' and 'math-result-service' using serviceName. The name will reflect in Jaeger when we submit the statistic to Jaeger collector.
jaeger config in calc-eng_handler.js & calc-maths_handler
In Function Process, we will use trace.extract(opentracing.FORMAT_HTTP_HEADERS,req,headers) method to retrieve the parent span injected to header earlier and assigned the value to an object called "parentSpan". We will then create another span object by referencing it back to "parentSpan". we labelled these spans as http_request; so we differentiate them from the incoming_request we defined earlier calculate_handler.js
Finally, a simple function calcAvgScore will be created to compute the English / Maths result. The computed result will then be returned to caller as a result attribute in JSON object.
Extract trace in calc-eng_handler.js & calc-maths_handler
extract trace in calc-eng_handler & calc-maths_handler
In the above screenshots, I am using threafin-calc-eng as an illustration. You do need to implement similar logic for threadfin-calc-maths with different calculation logic set for the mathematics result.
Testing the microservices
After we are done with the 3 APIs, we can now trigger "node app.js" to start the microservices (assume the main program for the 3 Node.JS application is app.js.)
We will use Postman to fire the API (http://localhost:8885/calculate) with payload contains the following fields (engExScore, engEssay, mathExScore, mathAssignment, SciExScore). The API should returned a result of Score and GPA based on the scores of individual subject submitted.
screenshot of Postman
After we triggered the API calls through Postman, we need to navigate back to Jaeger UI to check the trace result. In the service column, we should see calc-result-service (service name defined under serviceName) displayed in the dropdown. If we click on "Find Trace" button, the trace result should be displayed at the right panel. (Please see below)
screenshot of Find Trace
When you click on the trace result, the detailed trace timeline will be displayed. You should see 4 spans recorded; with calc-result-service as parent span calling english-result-service & maths-result-service as external service and a sub_process is invoked within calc-result-service.
Different services are indicated with different colours. All the useful tags we included such as HTTP URL and Method will be available as we inspect the trace result.
Screenshot of Trace Result
Jaeger can also give us a pictorial view of how APIs are invoked. You just need to toggle from Trace Timeline to Trace Graphs, you should see the API invocation in a nice pictorial view. (Please see below)
Screenshot of Trace Graphs
This brings me to the end of my sharing. I hope you enjoy my sharing and hopefully you find it useful when you need to apply Jaeger in your project. Do leave me a comment if you have any question. Thank you!