Kafka overview

Zookeeper General definition of distributed system: a software system that is composed of independent computing entities linked together by a computer network whose components communicate and coordinate with each other to achieve a common computational goal. Implementing coordination among components of a distributed system is hard. For example, designated master node becomes single point of … Read moreKafka overview

Performance Analysis

Overview In 2015, Brendan Gregg posted two great articles on Netflix blog: Linux Performance Analysis in 60 seconds, and Linux Perfomrance Tools. They have great value when I was in a urgency to spot performance issues. The articles cover the essential tools for performance troubleshooting, including: Check out load averages: w or uptime Print kernel … Read morePerformance Analysis

Balloons steal memory from virtual machines

I came across an ElasticSearch server (ESXi guest with 32GB physical memory) where the main process keeps dying of OOM. Even worse, after the OOM event, the free memory left is about 10G and Elastic Search cannot start because its JVM is set with -Xms16g in /etc/elasticsearch/jvm.options. So I need to address the OutOfMemory error. … Read moreBalloons steal memory from virtual machines

How memory usage adds up in Linux

There are too many metrics that describes some aspects about memory in Linux. Further to an old article on this site, where several concepts were discussed, this posting will make sense of those common metrics in Linux, CentOS as an example. The most fundamental command is free and my favourite switch is -h for human … Read moreHow memory usage adds up in Linux

High Availability and Load Balancer

Overview Fault tolerance and high availability are two architectural characteristics that people often confuse with each other. High availability focuses on minimizing downtime. It guarantees uptime, but not performance in the event of component failures. Fault tolerance, on the other hand, focuses on stable capacity event in the event of component failures. Fault tolerance is … Read moreHigh Availability and Load Balancer

Nginx as a reverse proxy for Nifi web UI and Kibana

Nginx can act as a application neutral proxy. One example is to front Nifi. The nifi default configuration provides an HTTP access point, specified in the following entries in nifi.properties: Nifi can provide secure port by commenting out the lines above and provide the followings: However, it does require configuring JKS keystore for Java, as … Read moreNginx as a reverse proxy for Nifi web UI and Kibana