Build a Batch Framework with Go


Build a Batch Framework with Go

Recently, I have just picked up Golang and decided to use what I learnt to build a Batch Framework with Go.


Batch Framework - Batch103

I called this project Batch103 (Started as Batch101. 103 is the 3rd version I decided to publish) ; the project is a simple batch framework inspired by SpringBatch - a popular batch framework in Java world. If you familiar with SpringBatch, you may find many similar features you used to see in SpringBatch. The source code of the framework is available at github (https://github.com/pengyeng/batch103.git) 

I will illustrates the high level  architecture of Batch103 using a diagram. (Please refer the overall architecture diagram) The framework comes with a Job Launcher which provides  a implementation class to launch the batch job. Job Launcher needs 3 key components to operate; which are :
  • Reader - handling the input of batch job
  • Processor - handling the logic to process the data
  • Writer - handling the output of batch job.
Data retrieved from Reader will flow from Processor and Writer using BatchData; which is a common data structure to carry the data. At the end of batch execution, Job Launcher will generate a statistic report to summarise the number of records processed & rejected in each process.


Overall Architecture

Batch Job Example 

To build a simple batch job using Batch103, you need to first create a Go module (example: c:\go\batchsample2) by running "go mod init example.com/batchsample2" in your project folder. Subsequently, you need to use "go get github.com/pengyeng/batch103" to include batch103 library into your Go project - batchsample2.

In this example, I am going to create an inbound file processing batch job to read a csv file (laptop.csv), perform validation check and then insert the retrieved data into mySQL database. In my project folder, I will create the following folder structures in batchsample2 project for my model, processor, reader and writer (please see the below picture) : 
  • model folder
  • processor folder
  • reader folder
  • writer folder

Directory Structure

In addition, I will also create a csv file called (laptop.csv) under project folder which will be used as my inbound file.

laptop.csv      

Before we create a reader, we need to first create a data model (laptop.go) to hold the data that is going to process. In this example, my data is a laptop record received from CSV which includes brand name, model, cpu core and memory size  (Please see below)





source code of laptop.go

Reader Component

With the data model, We can now create our reader component (laptopreader.go). Our reader component (laptopdreader.go) will need to import Batch103 framework and the data model we created earlier.

 

source code of laptopreader.go

As this example is a file processing reader, we will implement Batch103.FileReader and the Read() method in Batch103. (Please see above) The Read method needs to define the csv filename (laptop.csv), use csvFileReader object (from io library) to read the content and then insert the records into a Slice of BatchData which is a common model to flow the data throughout the entire batch execution. (Slice is a dynamic array in Golang) 


source code of laptopreader.go


Processor Component

The next component we need to create is processor component. Processor has a simple logic. It consists of a for loop to loop through each single record in BatchData. In this example, I built a simple validation to check if Model attribute is empty field. If validation failed, I will trigger Reject[batch103.stgProcess] to mark the record as Rejected at Processor stage (batch103.stgProcess indicates that the validation is at processor stage)  The rejected record will not be pickup in the subsequent process (In this case is writer)


source code of laptopprocessor.go


Writer Component

After we created the processor component, we can now create our writer component. Writer component will implement batch103.DBWriter and Write method. In the writer component, we need one extra library github.com/go-sql-driver/mysql as we need the database driver to connect with MySQL database. You can include the library by executing a "go get" command. 

source code of laptopwriter.go


source code of laptopwriter.go

Similar to processor, we will mark all records which are failed to be inserted as Rejected. we will trigger Reject(batch103.StgWrite) method to mark the records as Rejected at Writer Stage. To successfully execute the batch job, you need to have a mySQL database setup. you may use the below DDL to create the laptop table in go-data schema.

DDL of tbl_laptop


After all the 3 components are in placed, we can now create our main Go program (main.go) and launch our JobLauncher in the main function. (Please see below)


source code of main.go


Job Launcher allows us to define multiple processors and writers in case your batch job needs separate set of processing logic or output the data to multiple writers. Hence, we need to create a slice of ProcessorList and and slice of WriterList to append our LaptopProcessor and LaptopWriter into the list.

After we are all set, we can trigger the Run method to execute the batch. To test the program (main.go),  go to any terminal at project folder (batchsample2) and trigger "go run main.go". You should see a statistic report generated after the execution. You will find the number of records processed and rejected at Reader, Processor and Writer.

That's all I have in this example. I hope you find this article useful when you need to build a batch execution program using Go. Do drop me comment if you need further clarification. Thank you.


output of the batch execution




Comments

Post a Comment

Popular posts from this blog

API Versioning with Node.JS

Image
 API versioning is one of the key concept when we consider API implementation. It's useful when we need to maintain coexistence of multiple versions of the API implementation. It happens when you need to enhance the functionalities and at the same time support existing API consumer which still rely on the current implementation.  Typically, you do not need to maintain API versioning when the change can be gracefully handled by the application logic and the implementation won't result to any breaking change. Breaking change refers to implementation which affects the request or response of your existing API. API consumer can no longer consume the API as the API integration contract has been changed.     Implementation of API versioning There are many approaches can be used to handle API versioning. In generally, API service provider needs an indicator in API request to understand which version the API consumer need. I will cover two of the common approaches that y...
Read more

Build Retry-able API using idempotency key

Image
 If you have been working on API implementation for awhile and try to make your API resilient & bulletproof, the topic today might be something you will find interesting. Today, nobody can guarantee the microservices that you built will not run into trouble. When issue occurred, we often hope that the simplest way to solve the issue is to retry and invoke the API again. Retry can be an easy mechanism handled by your middleware/ API orchestration product. The challenge for doing so will always be on the business logic of the API if the nature of the record can gracefully managed the duplication through something like Primary Key. (e.g. create a car record with car registration number) You probably won't hit a problem when performing record update (if multiple updates is not a problem. e.g. update status of your car record from "available" to "sold"),  but it will be tricky for scenario of record creation.(e.g. create a sales record for a car) Making microserv...
Read more