Microservices Observability with Jaeger

 



As many applications have started the journey of modernisation from monolithic to microservices architecture, observability tool has now become essential to troubleshoot and understand issues happened in application. Business logic will no longer sit within the monolithic code, but has now spread across multiple microservices which can be hard to trace if we don't have a proper tool.

Jaeger (https://www.jaegertracing.io/) is an open source project developed by Uber Technologies which is meant for us to perform distributed tracing across APIs. In today's sharing, I am going to implement Jaeger distributed tracing via microservices developed using Node.JS. I am going to use threadfin-http ( https://github.com/joshua-lim/threadfin-http) ; a simple Node.JS http server to illustrate this. If you are not a developer, just want to explore the features of Jaeger without doing much coding, you can download the hot r.o.d example (https://github.com/jaegertracing/jaeger/tree/master/examples/hotrod) from github, which can give you a feel of how distributed tracing can be done through Jaeger.

Business Scenario

Without further delay, let me share the example that I am going to show in today's sharing. In today's scenario, I am going to develop a student result calculation service which consist of 3 components developed by Node.JS threadfin-http library (Please refer to the below diagram - Flow of Calculate Result microservices):
  • Calculate result API
  • English result API 
  • Mathematics result API

As what you can see from the below diagram, Calculate Result (calc-result-service) will be the main API to receive incoming request from UI / other application. This API will calls English (english-result-service) and Mathematics result (maths-result-service) API to compute the average score of English and Mathematics based on the exam and assignment marks. After Calculate Result service received the average score of English and Mathematics, the API will call a internal sub process to compute the final result and GPA of an individual student. The business logic will flow across microservices and sub-process and they can be easily monitored through Jaeger.

Flow of the Calculate Result microservices

Pre-requisite

Before creating the 3 Node.JS microservices, we need to first setup Jaeger. To have a simple setup, I decided to ease the work by using Jaeger all-in-one Docker image. If you already have docker installed in your machine, you can just execute "docker run -d --name jaeger -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 -p 5775:5775/udp -p 6831:6831/udp -p 6832:6832/udp -p 5778:5778 -p 16686:16686 -p 14268:14268 -p 14250:14250 -p 9411:9411 jaegertracing/all-in-one:1.24" to start a Jaeger container. Launch http://localhost:16686/ through browser and you should see the UI (please see below) after Jaeger container successfully started.


Jaeger UI Screen


As we need to create 3 microservices, I am going to create 3 folders (please see below) to separately run the Node.JS instances (Calculate Result API, English Result API and Mathematics Result API) at port 8885, 8886 and 8887.
  • threadfin-calculate
  • threadfin-calc-eng
  • threadfin-calc-maths
To start the 3 Node.JS application at 3 different ports (8885, 8886 and 8887), we need 3 config.json (you may find the sample config.json at threadfin-http github test folder) in these 3 folders and set the value of server_port attribute accordingly.

config.json

You also need a main program (app.js) at each folder to start the Node.JS application. Please see below for the source code of app.js. (You should modify the path of config.json according to the folder name)

source code of app.js

We can now install the dependency libraries in these 3 folders. Besides threadfin-http (npm install threadfin-http) core library,  We also need 2 additional libraries - jaeger-client (https://www.npmjs.com/package/jaeger-client) and opentracing (https://www.npmjs.com/package/opentracking) to support the implementation. Simply run "npm install <library>", you should be able to download the neccessaries library into the folders.


Calculate Result - threadfin-calculate API

After we are done with the pre-requisite, we can start to build our first Node.JS microservices - threadfin-calculate API. We will create a threadfin-http handler file called calculate_handler.js. To effective track the API in Jaeger, we need to use Jaeger client objects to define "serviceName". (please see below) In this example, we are going to call our service 'calc-result-service'. Besides the "serviceName", We also need to define the "collectorEndpoint" (localhost://localhost:14268/api/traces), so  jaeger-client can send the statistic back to Jaeger collector after every API execution.

jaeger config in calculate_handler.js


Function process is the core implementation of the API logic. This is required by every threadfin-http handler. In function process, we need to define the start and finish of span which represents a unit of work in Jaeger. (We called the main span "incoming_request"). In addition, we should also use "addTags" method to include more information such as HTTP URL and Method so the trace can be effectively search in Jaeger UI. (Please see below) 

As the tracing need to happen across multiple APIs, we need to create a trace context using tracer.inject(span, opentracing.FORMAT_HTTP_HEADERS, traceContext) method to inject the tracer information to traceContext object. 

Span, addTags and traceContext in calculate_handler.js


We will then place the traceContext object to HTTP header. This will allow the trace references available for the subsequent APIs which will be called later. In function Process, we will make 2 API calls; one to English result (http://localhost:8886/calc-eng) and another to Mathematics result (http://localhost:8887/calc-maths) microservices. In this example, we will use another 3rd party library - Request to simplify the implementation (You may download the library by executing "npm install request") of API call.  After we received the result from both APIs, we will then invoke a sub process called calcFinalResult to compute the final result and GPA.

API calls in calculate_handler.js

calcFinalResult is a simple function in calculate_handler.js. To allow Jaeger to trace the sub-process, we included a child span and linked it as a child of the original span ( childOf: span ). Similar to the main span, we need to call startSpan and finish method to track the start and end time of the invocation.

calcFinalResult in calculate_handler.js

English Result - threadfin-calc-eng API & Maths Result - threadfin-calc-maths API

After we created Calculate Result API, we can proceed to create English and Maths Result APIs. threadfin-calc-eng and threadfin-calc-maths are the microservices which responsible for the computation of English and Mathematics result. To differentiate them from the core service, we  labeled them as 'english-result-service' and 'math-result-service' using serviceName. The name will reflect in Jaeger when we submit the statistic to Jaeger collector.

jaeger config in calc-eng_handler.js & calc-maths_handler

In Function Process, we will use trace.extract(opentracing.FORMAT_HTTP_HEADERS,req,headers) method to retrieve the parent span injected to header earlier and assigned the value to an object called "parentSpan". We will then create another span object by referencing it back to "parentSpan". we labelled these spans as http_request; so we differentiate them from the incoming_request we defined earlier calculate_handler.js
 
Extract trace in calc-eng_handler.js & calc-maths_handler

Finally, a simple function calcAvgScore will be created to compute the English / Maths result. The computed result will then be returned to caller as a result attribute in JSON object.
 
extract trace in calc-eng_handler & calc-maths_handler

In the above screenshots, I am using threafin-calc-eng as an illustration. You do need to implement  similar logic for threadfin-calc-maths with different calculation logic set for the mathematics result.


Testing the microservices

After we are done with the 3 APIs, we can now trigger "node app.js" to start the microservices (assume the main program for the 3 Node.JS application is app.js.)

We will use Postman to fire the API (http://localhost:8885/calculate) with payload contains the following fields (engExScore, engEssay, mathExScore, mathAssignment, SciExScore). The API should returned a result of Score and GPA based on the scores of individual subject submitted.

screenshot of Postman

After we triggered the API calls through Postman, we need to navigate back to Jaeger UI to check the trace result. In the service column, we should see calc-result-service (service name defined under serviceName) displayed in the dropdown. If we click on "Find Trace" button, the trace result should be displayed at the right panel. (Please see below)


screenshot of Find Trace

When you click on the trace result, the detailed trace timeline will be displayed. You should see 4 spans recorded; with calc-result-service as parent span calling english-result-service & maths-result-service as external service and a sub_process is invoked within calc-result-service. 
Different services are indicated with different colours. All the useful tags we included such as HTTP URL and Method will be available as we inspect the trace result.

Screenshot of Trace Result

Jaeger can also give us a pictorial view of how APIs are invoked. You just need to toggle from Trace Timeline to Trace Graphs, you should see the API invocation in a nice pictorial view. (Please see below)

Screenshot of Trace Graphs


This brings me to the end of my sharing. I hope you enjoy my sharing and hopefully you find it useful when you need to apply Jaeger in your project. Do leave me a comment if you have any question. Thank you!

Comments

Post a Comment

Popular posts from this blog

API Versioning with Node.JS

Image
 API versioning is one of the key concept when we consider API implementation. It's useful when we need to maintain coexistence of multiple versions of the API implementation. It happens when you need to enhance the functionalities and at the same time support existing API consumer which still rely on the current implementation.  Typically, you do not need to maintain API versioning when the change can be gracefully handled by the application logic and the implementation won't result to any breaking change. Breaking change refers to implementation which affects the request or response of your existing API. API consumer can no longer consume the API as the API integration contract has been changed.     Implementation of API versioning There are many approaches can be used to handle API versioning. In generally, API service provider needs an indicator in API request to understand which version the API consumer need. I will cover two of the common approaches that y...
Read more

Build Retry-able API using idempotency key

Image
 If you have been working on API implementation for awhile and try to make your API resilient & bulletproof, the topic today might be something you will find interesting. Today, nobody can guarantee the microservices that you built will not run into trouble. When issue occurred, we often hope that the simplest way to solve the issue is to retry and invoke the API again. Retry can be an easy mechanism handled by your middleware/ API orchestration product. The challenge for doing so will always be on the business logic of the API if the nature of the record can gracefully managed the duplication through something like Primary Key. (e.g. create a car record with car registration number) You probably won't hit a problem when performing record update (if multiple updates is not a problem. e.g. update status of your car record from "available" to "sold"),  but it will be tricky for scenario of record creation.(e.g. create a sales record for a car) Making microserv...
Read more

Build a Batch Framework with Go

Image
Build a Batch Framework with Go Recently, I have just picked up Golang and decided to use what I learnt to build a Batch Framework with Go. Batch Framework - Batch103 I called this project Batch103 ( S tarted as Batch101. 103 is the 3rd version I decided to publish ) ; the project is a simple batch framework inspired by SpringBatch - a popular batch framework in Java world. If you familiar with SpringBatch, you may find many similar features you used to see in SpringBatch. The source code of the framework is available at github  (https://github.com/pengyeng/batch103.git )  I will illustrates the high level  architecture of Batch103 using a diagram. (Please refer the overall architecture diagram) The framework comes with a Job Launcher which provides  a implementation class to launch the batch job. Job Launcher needs 3 key components to operate; which are : Reader - handling the input of batch job Processor - handling the logic to process the data Writer - handling th...
Read more