Basics of Publish Subscribe Model

publish - publishing a message (data) to a partition. used by the publisher or producer. publisher or producer creates a message to the partition. topic is a folder like filesystem where these messages are stored in partitions. partition is sub modules of a topic. a single topic can have multiple partitions. pub/sub system uses hash map (key/ value pair) to store a message. after publishing a message to a topic topic creates a hash of the key and store it to a particular partition and the publisher does not care about the partition where the message will be stored. often message is stored in JSON or XML format. there is another tool or schema management framework is called Avro. this is used to serialize the schema. an offset is an integer value which continously increases and added to the end of a message to uniquely identify and find the position of it. every message has an unique offset at the end of it which is coming from last consumed message. messages are often produced/published and consumed/subscribed in batch to manage storage.

subscribe - Messages are pushed to consumers, which means that consumers are delivered messages without having to request them. Messages are exchanged through a virtual channel called a topic. A topic is a destination where producers can publish, and subscribers can consume, messages. Messages delivered to a topic are automatically pushed to all qualified consumers.

broker/cluster - broker is a single kafka server which contains topics and partitions. this is used to produce a message that is it stores the message to a storage disk and getting the hash for a message. this also fetches messages to consume. a cluster is a group of brokers. here one broker will be automatically selected as controller which manages the administrative things. a broker can have a single partition that is called the leader of that partition. this is used to maintain redundancy. having control of a partition is called ownership. this can be taken over by another broker if a broker failure occures. in kafka there is a functionality called retension. a topic can be configured to retain the partitions and messages for a certain condition, that may be a certain number of time or certain size of the topic. this can be also configured for a particular partition or messages.

Multiple datacentre - if there is multiple datacentre and multiple clusters are there in the datacentres and all the modification of a message in a datacentre
needs to be copied to other datacentres then it cannot be done my kafka redundancy. it uses a middleware tool named mirrormaker. this acts as producer/consumer. it consumes messages from a cluster and produces it to another.

Comments

Post a Comment

Please Share Your Views

Popular posts from this blog

SSO — WSO2 API Manager and Keycloak Identity Manager

Image
In this article, we are going to see how to implement Single Sign-On (SSO) for WSO2 API Manager using Keycloak as a Federated Identity Provider. Also, we'll choose a deep-dive showing how debugging works with the WSO2 API Manager code to see what happens inside when it's configured with a third-party identity provider (i.e Keycloak in this example). High-Level Architecture This is what we are going to do in this tutorial. High-level architecture Software Needed Following softwares will be needed: WSO2 API Manager (2.6.0) Keycloak (6.0.1) Note : We aren’t getting to use WSO2 Identity Server as a middleman for this tutorial. The article is predicated on pure WSO2 API Manager 2.6.0 (all in one). Before We Start The tutorial will be divided into two parts. In the first part, I will explain how to install and configure Keycloak. In the second part, we will see how to install and configure the WSO2 API Manager. Before we start, let's modify the  /etc/hosts  file ...
Read more

Garbage Collectors - Serial vs. Parallel vs. CMS vs. G1 (and what’s new in Java 8)

Image
The 4 Java Garbage Collectors – How the Wrong Choice Dramatically Impacts Performance When Garbage collection is one of those things that are generally known to impact performance, but beyond that – in terms of how it actually works – it’s pretty much a mystery to most of us. So, I thought I’d take a whack at covering the basics of GC, especially since this is an area that has seen some major changes and improvements with Java 8, especially with the removal of the PermGen and some new and exciting optimizations (more on this towards the end). When we speak about garbage collection, the vast majority of us know the concept and employ it in our everyday programming. Even so, there’s much about it we don’t understand, and that’s when things get painful. One of the biggest misconceptions about the JVM is that it has one garbage collector, where in fact it provides  four different ones , each with its own unique advantages and disadvantages. The choice of which one to use isn’t automati...
Read more

Recommendation System Using Word2Vec with Python

Image
Overview Recommendation engines are ubiquitous nowadays and data scientists are expected to know how to build one Word2vec is an ultra-popular word embeddings used for performing a variety of  NLP  tasks We will use word2vec to build our own recommendation system. Curious how NLP and recommendation engines combine? Let’s find out!   Introduction Be honest – how many times have you used the ‘Recommended for you’ section on Amazon? Ever since I found out a few years back that machine learning powers this section – I have been hooked. I keep an eye on that section each time I log into Amazon. There’s a reason companies like Netflix, Google, Amazon, Flipkart, etc. spend millions perfecting their recommendation engine. It is a powerful acquisition channel and enhances the customer’s experience. Let me use a recent example to showcase their power. I went to a popular online marketplace looking for a recliner. There were hundreds of them. From Traditional two-position recliners ...
Read more