Saturday 11 September 2021

Design and Implement Rate Limiter

Rate limiter is one of the most important aspects when designing a microservice system. It is used to limit the usage of an api upto a particular threshold. When an api exceeds its TPS (Transactions Per Second) threshold limit, it should ignore those extra requests to save all downstream services and should send proper http status to its client.

Let's take an example to understand this. Let's say there is an api whose maximum TPS it can handle is 500 requests per second. If we apply a rate limiter on this api upto 500 request per second and we experience 700 request per second in a particular time window, then 200 extra requests will be ignored with an http status 429 i.e. TOO_MANY_REQUEST. Before going into detail of how to implement a Rate limiter in a distributed environment, let's understand various advantages of using it.

Tuesday 20 July 2021

Streaming NRT data with kafka connect and Debezium

Event driven architecture principle is very popular for past few years in microservice systems. Providing Near Real Time (NRT) data to other services as soon as possible is core problem for all organisations. 
This is also known as Change data capture Design pattern where capturing each database change event, then wrapping it and pushing it to other pipeline or services. 
This is Duality of Streams and Tables where database table change events are pushed over a stream to other systems by an order of timestamp just to make a snapshot of database at other systems. 

Wednesday 27 January 2021

Useful kafka commands

For past few months I have been working with many kafka production issues and I am still trying to learn many new things about it. But while working on any kafka issue, you always want some commands to be very handy and just one click away in a file or in a blog.

Here are few kafka commands that may be very helpful to you while learning or solving any of your kafka issues. 

Here I would be referring KAFKA_HOME as your path to kafka home.