technology on Digi Hunch

Virtualization 1 of 4 – Hypervisor

Mon, 27 Jul 2020 22:52:00 -0400

In broad terms, virtualization of computing resource is about isolation of resources, at different levels. There are five levels of virtualization:

Application level, such as JVM, .NET CLR
Library (user-level API) level
Operating system level, such as LXC, Docker, OpenVZ
Hardware abstraction layer (HAL) level, such as VMware, Xen, etc
Instruction set architecture (ISA) level

In my context I deal mostly with OS level and HAL (hardware abstraction layer) level of virtualization. In loose terms, the word containerization refers to OS level virtualization, while the word virtualization is exclusively reserved for HAL level virtualization, also referred to as hypervisor-based virtualization. This post will just focus on this family of technology and loosely refers to it as virtualization.

Virtualization technology evolved from on-premise data centre environment and now is the backbone of cloud computing. The challenges of IT operation in the era of virtualization involves managing VM sprawling, investigating performance issues, planning capacity and addressing storage I/O block. The idea of virtualization is sharing (thus isolating) resources for better utilization, leading to better return on investment. This posting is to cover only the very basics of virtualization.

Hypervisor

Hypervisor is the software layer which provides the capability to run multiple virtual machines on the same physical host. It is broken down into two categories:

Type I hypervisor (aka bare metal hypervisor): directly run on physical hardware. They control the hardware as well as manage the virtual machines. For example, Linux KVM, VMware ESXi, Xen and Microsoft Hyper-V
Type II hypervisor: runs as an application or service on top of the host operating system, which is installed on the bare metal. Guest operating system calls need to traverse via the host operating system stack to reach hardware resource. For example, Oracle Virtual Box, VMware Fusion and Linux Containers (LXC)

Hypervisor Types

Virtualization Techniques

The most primitive form of technology that can be arguably categorized under virtualization is hardware emulation, where a piece of (more accessible) hardware imitates another (less accessible). The architecture limits itself in functional testing only, and is not built for performance or production at all.

The original virtualization technology deals with CPU and memory virtualization. In this well-written whitepaper fromVMware, there are three CPU virtualization techniques introduced for x86 architecture.

The x86 architecture offers four levels of privilege known as Ring 0,1,2 and 3 to operating system and applications to manage access to the computer hardware. User-level applications typically run in Ring 3, the OS must execute its privileged instructions in Ring 0 since it needs to have direct access to memory and hardware. The two main challenges with virtualizing x86 architecture are:

A virtualization layer between hardware operating system who expects Ring 0 privilege;
Some instructions with different semantics when not executed in Ring 0 cannot be virtualized effectively. They need to be translated at runtime.

These challenges makes true virtualization of x86 architecture impossible and thus VMware developed three alternative technologies.

Full virtualization (using binary translation): virtual machine presents a complete simulation of the actual hardware environment so that an unmodified guest OS can run in isolation. The Guest OS is not aware that the underlying environment it is running on is virtualized, and issues hardware calls to communicate with (what it thinks as) hardware. The virtual processors have to understand guest CPU instruction, and reproduce the equivalent CPU instructions of the host machine. VMware’s technology to address this is called Binary Translation. This overhead makes true full virtualization difficult to achieve. In real life, a virtual environment that provides “enough representation of the underlying hardware” can be considered to provide full virtualization as long as it allows guest OS to run without modification. Full virtualization comes with a performance penalty.

Paravirtualization (aka OS assisted virtualization): refers to communication between the guest OS and the hypervisor to improve performance and efficiency. In this technology, guest OS is modified with an interface to host hardware to be able to communicate and operate seamlessly. Since the guest OS is modified, the VM does not need to be a complete simulation of the hardware. The modified guest OS knows it is running on a virtualized environment, and (vm driver) makes API calls (known as ‘hyper calls’) to the hypervisor. This allows para-virtualization technology to achieve performance closer to non-virtualized environment. However, since paravirtualization cannot support unmodified operating systems, its compatibility and portability is poor.

Hardware-Assisted Virtualization: hardware vendors such as Intel and AMD both have developed extensions (new features) to simplify virtualization techniques, for example, the introduction of privileged instructions with new CPU execution mode feature to allow hypervisor to run in a new root mode below ring 0. This removed the need for full virtualization and paravirtualization. With VMware originally as a promoter of full virtualization and Xen for paravirtualization, most virtualization technologies today utilizes hardware-assisted virtualization feature, for example, Linux KVM, VMware workstation, VMware fusion, Xen, VirtualBox, etc. Intel’s virtualization extension is VT-x. AMD’s counterpart is AMD-V technology.

To virtualize memory, another level of memory virtualization is required (similar to the virtual memory support in Linux). Hypervisor is responsible for mapping guest physical memory to the actual machine memory, and it uses shadow page tables to accelerate the mappings, usually at a performance cost.

Popular hypervisors

On the market there are a few popular hypervisor technologies. They are all type 1 hypervisors:

Xen is an open-source hypervisor project originally developed in Cambridge University, licensed under GPLv2. . Based on that, Citrix developed its commercial product XenServer, a bare-metal virtualization platform with enterprise-grade features for x86 and AMD environments. Oracle VM is another commercial implementation of Xen. The Xen project also supports many cloud platforms such as Openstack, Cloudstac, etc. Xen project supports paravirtualization (Xen-PV) as well as hardware-assisted virtualization (Xen-HVM) for virtualization of X86, IA64, ARM and other CPU architectures. The earlier versions does not support memory overcommit (aka “dynamic memory optimization”, “memory ballooning“, or as Citrix calls it “dynamic memory control, DMC”). This delivers better performance but also has higher budgetary requirement on hardware since there isn’t room for over-subscription.

Hyper-V is a Microsoft product. It executes in high CPU privilege (Microsoft calls it ring -1 which is equivalent to root mode as Intel calls it). On the guest VM, OS kernel and drivers run in ring 0, application rin in ring 3. This eliminates the need for binary translation. Hyper-V does not support memory overcommit either. Hyper-V is well integrated with Windows platform. It supports Linux as well although with some performance penalty.

Linux KVM (Kernel-based Virtual Machine) is a full open-source virtualization solution for GNU/Linux. What makes KVM a special hypervisor is that it uses a loadable kernel module kvm.ko that turns itself into a hypervisor and provides VMs with direct access to the hardware. So it is a type 1 hypervisor despite of the presence of Linux OS. KVM also contains a processor specific module, kvm-intel.ko or kvm-amd.ko. KVM leverages qemu to access devices. Because KVM runs as a process inside of Linux OS, KVM can use many existing feature in Linux kernel. Redhat has an enterprise solution based on KVM.

Xen vs KVM

VMware ESXi is VMware’s premium hypervisor product (not open-source) and is available for ~~free download~~, although the advanced features are not free. (Update no free download link anymore.) VMware vSphere is virtualization platform built on top of ESXi, including a whole family of virtualization products.

Market segments and players

Virtualization involves many market segments such as virtual desktop infrastructure (VDI, for desktop virtualization), server virtualization is the predominant domain in the virtualization of data centre environment. This effort led to Hyper-Converged Infrastructure (HCI) where almost all the traditional hardware resources are software-defined through the virtualization layer. The management of infrastructure is abstracted away from the physical hardware management. The three most fundamental areas in HCI are:

Server (compute) virtualization: the previous section covers the virtualization of memory and x86 CPU, which are the main focus on computing resource virtualization. Additionally, graphics computing resources can be virtualized today. Example products include: VMware vShpere (compute virtualization based on ESXi hypervisor).

Storage Virtualization: the technology to abstract physical data storage resource to make them appear as if they were a centralized resource. Storage virtualization takes place at three levels depending on the use case: block-level, file-level and object level. Example products include: VMWare vSAN (vSphere-native storage), HPE 3PAR (Tier-1 storage), EMC VxRail, PureStorage Flash Array (Tier 1), etc. Storage Virtualization enables Software-Defined Storage (SDS), the provisioning and management of data storage independent of the underlying hardware.

Network Virtualization: the technology to abstract network resources that were traditionally delivered in hardware to software. Network virtualization decouples network services from the underlying hardware management and allows virtual provisioning of an entire network. VLAN is a classic example of network virtualization. There are also various overlay technologies such as VXLAN, which provides an industry framework for overlaying virtualized layer 2 network over layer 3 network (used in Docker network) using an encapsulation mechanism and a control plane. Example products include: VMware NSX Data Center (L2-L7 network and security virtualization platform), Cisco ACI, Palo Alto Panorama. Network Virtualization enables Software-Defined Network (SDN), an approach to network management that enables dynamic, programmatically efficient network configuration in order to improve network performance and monitoring, making it more like cloud computing than traditional network management.

Delivery model

Virtualization allows managed service providers (MSPs) to deliver IT service in the following three models:

Iaas (Infrastructure as a Service): MSP delivers VM to customers.
PaaS (Platform as a Service): MSP delivers environments to customers (e.g. Database as a Service, managed RabbitMQ service, etc).
SaaS (Software as a Service): MSP delivers entire application for the customer.

IT service delivery models enabled by virtualization technology

Since virtualization is the backbone of cloud computing. This model is also referred to as cloud computing delivery model.

Virtualization and Containerization

These two concepts are similar and could be confusing to beginners. Both provide a mechanism to isolate computing resource for different applications, for the purpose of higher utilization of resource. The difference lies in how and where the isolation is made. Virtualization requires a guest operating system per VM (OS level isolation), whereas the container technology isolates application processes along with its runtime into a container (dependency level isolation), using some new Linux kernel features such as namespaces and cgroups. All containers make their system calls to the container engine on the host operating system. So they share a kernel on the same host. In this sense, container engine running on OS could be considered as type 2 hypervisor.

From VMs to containers

VMware is a major player in enterprise data centre virtualization, which is facing fierce competition from public and private cloud vendors. VMware also has its own private cloud services. Docker is the most popular container technology that conforms to the specifications of Open Container Initiative (OCI), a governance structure for industry standards around container formats and runtimes.

Virtualization and Cloud

Among public cloud vendors, AWS EC2 used Xen PV and Xen HVM in its earlier implementations. It has transitioned to AWS bare metal. The history is well summarized here. Microsoft Azure runs Azure Hypervisor as the native hypervisor in Azure Cloud Services platform. It is a customized version of Microsoft Hyper-V specifically for Azure platform. With GCP, Google Compute Engine (GCE) instance runs VMs on KVM as hypervisor. It can also enable nested virtualization.

The scope of cloud computing is evolving overtime. It originally only refers to a business model of offering IT services (in one of the three delivery models outlined above) based on virtualization technology. Therefore I cannot make comparison between a technology and a business model. Today, with public cloud vendor extending their offerings (with various managed services and platforms) and people’s misuse of the terms, the buzz-word “cloud” seems to suggest anything that is offered in public cloud service. The essence still remain the same where managed services and managed platforms are built on top of virtualized compute unit under the hood, which are driven by virtualization technologies.

How RPC and NFS work

Wed, 15 Jul 2020 10:45:00 -0400

I touched on NFS in several previous postings, and here is a deeper dive of this particular protocol. NFS is built on top of Remote Procedure Call (RPC) and therefore it is important to understand RPC first. In fact NFS is one of the most prominent user of RPC and the best example for learning RPC.

RPC overview

According to Wikipedia, an RPC is when a computer program causes a procedure to execute in a different address space (commonly on another computer on a shared network), which is coded as if it were a normal (local) procedure call, without the programmer explicitly coding the details for the remote interaction. That is, the programmer writes essentially the same code whether the subroutine is local to the executing program, or remote. This is a form of client–server interaction (caller is client, executor is server), typically implemented via a request–response message-passing system. In the object-oriented programming paradigm, RPCs are represented by remote method invocation (RMI), such as Java RMI API.

RPCs are a form of inter-process communication (IPC), in that other processes have a different address spaces: if on the same host machine, they have distinct virtual address spaces, even though the physical address space is the same; while if they are on different hosts, the physical address space is different.

RPC is a request–response protocol, and therefore synchronous. An RPC is initiated by the client, which sends a request message to a known remote server to execute a specified procedure with supplied parameters. The remote server sends a response to the client, and the application continues its process. While the server is processing the call, the client is blocked (it waits until the server has finished processing before resuming execution), unless the client sends an asynchronous request to the server. There are many variations and subtleties in various implementations, resulting in a variety of different (incompatible) RPC protocols.

NFS overview

NFS is defined as a set of RPCs, including their arguments, results and effects. RPC makes NFS protocol transparent. RPC is also stateless so the server does not keep the state of RPCs once the request has been served. Each RPC contains the necessary information to complete the call. In the event of server failure, client will need to resubmit requests. NFS has several versions, with v3 and v4 most popular. We will focus on v3 in this posting and brief on v4.

Portmap

RPC makes a remote call appears to client application as a local call, with the help of portmap. The utility for portmap is rpcbind. In RedHat/CentOS 5 or prior, it was even simply called portmap but they are essentially the same service for RPC port mapper.

The rpcbind service is required on both NFS client and NFS server. On the client, it talks to client application, as well as its counterpart on the server. Its main function is query its counterpart on the server, providing a RPC program number, and expects a port number in return. On the server, the rpcbind utility listens at port 111, waiting for request with RPC program number(service), and return the TCP or UDP port number on which the requested service is hosted. RPC program number is reserved numeric indicator of services as outlined in RFC5531. For example, 100005 for mountd, 100021 for nlockmgr, and 100003 for nfs.

The port that rpcbind service itself listens on is always at port 111, which is known to both client and server. This is also referred to as portmapper daemon. Other than this fixed port, each NFS-related service (with their respect reserved program number) may be hosted on different ports on the server. The client rpcbind service must first look up for the port for the requested program number, then it directs the client to initiate connection to the specified port for the specific service.

With the following command rpcinfo command you may look up the program to port number mapping on destination server isilon.company.com

# rpcinfo -p isilon.company.com

The following command is commonly used for displaying mountpoint and troubleshooting mount. Under the hood it is using the information from rpcinfo.

showmount -e isilon.dcb.digitalhunch.com

On the NFS server side, rpcbind service must start before nfs service start. Otherwise nfs service cannot register ports to rpcbind. If you restart rpcbind, every service that had registered to rpcbind must restart to register themselves again. By default, NFS server can dynamically assign a port for mountd, nlockmgr within a port range. This makes firewall setting a pain. These dynamically assigned port can be fixed via configuration files.

File handle

NFS uses file handles (or fhandle) to represent files. It is a better mechanism to reference a file object than pathname for three reasons: 1. file handle has fixed length (32bytes); 2. If the file is renamed, the file handle reference remain the same; 3. If a file is deleted, and then a new file is created with the same path, a new file handle will be created. A file handle has three parts:

volume ID: to identify the mounted file system
inode #: to identify the file within the mounted file system
generation #: to detect when file handle refers to an older version of inode. Traditional Unix filesystems may reuse inode and thus NFS client could mistakenly use an old file handle and access a new file.

The file handle information is only meaningful to the server. New file handles are returned to client by certain procedures, such as LOOKUP, CREATE, and MKDIR. The file handle for the root of the file system, is obtained by the client when it mounts the file system, as permission allows.

Permission and Locking

When accessing a file on server, the client passes uid/gid info in RPCs, and the server performs permission checks as if the user was performing the operation locally. So users and groups are represented as integers. There are two security problems:

The mapping from uid/gid to user must be the same on all clients. This is not practical in large deployment, although can be solved via Network Information Service (NIS);
Whether the root user on the client has root access to files on the server, is a server policy configuration. This can be addressed by enabling “root squashing” on server, so that client’s uid 0 (root) is mapped to 65534 (nobody).

Unix has two locking mechanisms (fcntl and flock). NFS protocol supports fcntl but not flock. The flock function is managed by a separate service (nfslock) to allow NFS to lock files. The nfslock daemon provides the ability to lock regions of NFS files. NFS service itself is still completely stateless with locking managed separately. This is changed in NFSv4.

Procedures used in NFS service

NFS service defines a list of procedures. Here is a list with brief summary of activities. The bottom five RPCs are introduced in v3.

Procedure	Activity
GETATTR(fh)	Returns the attributes of a file, similar to stat syscall.
SETATTR(fh, attr)	Sets the attributes of a file (mode, uid, gid, size, atime, mtime); setting the size to 0 truncates the file
STATFS(fh)	Returns the status of a filesystem, such as block size, number of free blocks. e.g. df command.
LOOKUP (dirfh, name)	Returns fhandle and attributes for the named file in the directory specified by dirfh
READ (fh, offset, count)	Reads from a file, with offset and count specified. In v2, the length is up to 8192 bytes; v3 support more.
WRITE (fh, offset, count, data)	Writes to a file, with offset and count specified, as well as a separate field called data. Returns the new attributes of the file after the write.
CREATE (dirfh, name, attr)	Creates a file with the name, in directory, returns new fhandle and attributes
REMOVE (dirfh, name)	Deletes the named file in from directory dirfh and returns status.
RENAME (dirfh, name, tofh, toname)	Renames name in directory dirfh, to toname in directory tofh.
LINK (dirfh, name, tofh, toname)	Creates a hard link toname, in directory tofh, that points to name, indirectory dirfh.
SYMLINK (dirfh, name, string)	Creates a symbolic link name, in the directory dirfh, with value string.
READLINK (fh)	Reads a symbolic link and get file name of the target.
MKDIR (dirfh, name, attr)	Creates a directory name in the directory dirfh, and returns the new fh and attributes.
RMDIR(dirfh, name)	Removes a directory with the name, from parent directory dirfh.
READDIR (dirfh, cookie, count)	Reads a directory and returns up to count bytes of directory entries from the directory dirfh. The cookie is used in subsequent readdir calls to start reading at a specific entry in the directory. Cookie of zero get the server to start with the first entry in the directory.
NULL	No activity. Used for testing only.
ACCESS	Helps with client caching.
MKNOD	Makes a device special file.
FSINFO	Returns information about the server’s capabilities.
READDIRPLUS	Returns both file handle and attributes to eliminate LOOKUP calls when scanning a directory
COMMIT	In NFSv3, the server can reply to WRITE RPCs immediately without syncing to disk. When client wants to ensure that the data is on stable storage, it sends a COMMIT RPC. This is used in asynchronous writes for better performance, which is an option negotiated at mount time.

The addition of COMMIT procedure in v3 offers the option to improve write performance in place of synchronous write. However, asynchronous write requires more coordination to ensure data integrity during transmission, in the event of server crash. NFSv3 uses write verifier for this purpose. A write verifier is an 8-bye value that the server must change if it crashes.

After an asynchronous write, the reply from WRITE RPC includes a write verifier, the client must keep it for later use;
The client then sends a COMMIT RPC and the reply contains another write verifier;
The client compares the verifiers from the two returns for crash detection. If the verifiers don’t match, the client must rewrite all uncommitted data.
The client must keep all uncommitted data in case of a server crash.

Additional daemon processes

In addition to the three essential services, (nfs, rpcbind and nfslock), there are several auxiliary processes that facilitates NFS services. Their functions are listed here:

Process	Description
rpc.mountd	Used by NFS server to process MOUNT requests from NFSv3 client. It checks that the requested NFS share is currently exported by the NFS server, and that the client is allowed to access it. If the mount request is allowed, the rpc.mountd server replies with a Success status and provides the File-Handle for this NFS share back to the NFS client.
rpc.nfsd	Allows explicit NFS versions and protocols the server advertises to be defined. It works with the Linux kernel to meet the dynamic demands of NFS clients, such as providing server threads each time an NFS client connects. This process corresponds to the nfs service.
rpc.lockd	A kernel thread which runs on both clients and servers. It implements the Network Lock Manager (NLM) protocol, which allows NFSv3 clients to lock files on the server, using procedures such as NLM_NULL, NLM_TEST, NLM_LOCK, NLM_GRANTED, NLM_UNLOCK, NLM_FREE. The service is started automatically whenever the NFS server is run and whenever an NFS file system is mounted.
rpc.statd	This process implements the Network Status Monitor (NSM) RPC protocol, which notifies NFS clients when an NFS server is restarted without being gracefully brought down. rpc.statd is started automatically by the nfslock service, and does not require user configuration. This is not used with NFSv4.
rpc.rquotad	This process provides user quota information for remote users. rpc.rquotad is started automatically by the nfs service and does not require user configuration.
rpc.idmapd	provides NFSv4 client and server upcalls, which map between on-the-wire NFSv4 names (strings in the form of user@domain) and local UIDs and GIDs. For idmapd to function with NFSv4, the /etc/idmapd.conf file must be configured. At a minimum, the “Domain” parameter should be specified, which defines the NFSv4 mapping domain. If the NFSv4 mapping domain is the same as the DNS domain name, this parameter can be skipped. The client and server must agree on the NFSv4 mapping domain for ID mapping to function properly.

NFSv4

Even NFSv4 was introduced in 20 years ago, it improves access and performance of NFS on the Internet. It should be the default option for any new deployment.

NFSv4 is TCP only protocol and it is stateful.
NFSv4 combines mount and lock protocols into NFS so only one port is being used.
Users and groups are identified with strings (user@domain, or group@domain where domain represents a registered DNS domain or sub-domain), instead of integers. The access control policies are compatible with both Unix and Windows.
NFSv4 mandates strong RPC security built on cryptography, with negotiation at the time of mount
NFSv4 adopted a framework for authentication, integrity and privacy at RPC level
Introduced new RPC COMPOUND, which allows for several operations in one go. At the server, operations are evaluated in order, and each has a return value.

NFSv4.1 was release in 2010, and 4.2 in 2016. Both AWS EFS and Azure File storage supports 4.1.

EMC Isilon storage product

Wed, 08 Jul 2020 20:04:00 -0400

EMC has several product lines for different use cases in enterprise data storage. Like may other IT solutions, the website is clouded with marketing terms and slogans, and is purposefully not technical. This makes it difficult for technical staff to grasp the advantage of its product in a glimpse. I personally have to know their product (mostly with Isilon and ECS) well in order to make integration decisions. So I’m putting together this note (updated as of July 2020), with lots of details from their technical white paper.

Overview of EMC storage

At the highest level, the EMC enterprise data storage product lines are categorized into two groups: primary storage (along the lines of block-level storage) and unstructured storage (mostly file and object storage). The primary storage includes the following product:

PowerMax for OLTP database (Oracle, MicrosoftSQL and SAP)
PowerFlex: for Software defined storage, Oracle RAC, Elastic Stack, Kubernetes, Splunk
XtremIO for VMware, VDI, SAP
PowerStore for Database, VMware
PowerVault for Entry-level SAN and DAS environment

This post only expands on the unstructured storage product line, which mainly consists of PowerScale and ECS. ECS (elastic cloud storage) is EMC’s object storage. PowerScale (aka Isilon) is scale-out NAS platform for high-volume storage (up to 50 PB in a single file system), backup and archiving of unstructured data. For the rest of this post, I will still refer to PowerScale as Isilon.

Dell’s official support website is the most resourceful place to get information. For example, when I want to read about Isilon. I start with Dell support, then click on knowledgebase at the top, then go to “servers, storage and networking“, then “storage technical documents and videos“. There I can select a productline such as Isilon.

Overview of Isilon Family

Isilon is a clustered storage system consisting of three or more nodes. A node is a server with OneFS as its operating system. Based on FreeBSD, OneFS is EMC’s proprietary operating system to unify a cluster of nodes into a single shared resource. So OneFS is for Isilon only. It is the basis of Isilon. Isilon has three series of products:

F series: F200, F600, F800 and F810.
H series: typical models are H400, H500 and H600, which seeks to balance performance and capacity
A series: typical models are A200 and A2000 for active and deep archive storage

In June 2020, Dell decoupled OneFS software (with 9.0 released) from server hardware (referred to as PowerScale). Going forward EMC will refer to Isilon as PowerScale for OneFS version newer than 9.0 in spec sheets and white papers.

F200 is the cost-effective choice with SSD for remote office, small hospital, retail outlets, IOT or factory floor. F600 uses NVMe drives instead, and has more ECC memory and faster ethernet backend network. and is higher than F200 in its use case. Both F200 and F600 provide inline data compression and deduplication capabilities. F800 and F810 both use SSD and they are similar. F800 comes with InfiniBand backend network and F810 provides inline data compression and deduplication capabilities. H series tries to strike a balance between performance and capacity so they are pretty much everything in betwee. On the other end, A200 and A2000 are almost the same except for capacity difference.

Isilon’s advantage

Isilon has lots of intelligence built into its solution compared to a traditional NAS. Here are some aspects from its product white paper:

Aspects of Design	Isilon OneFS Scale-Out NAS	Traditional NAS
Network	Separation of front-end and back-end network to isolate node-to-node communication to a private low-latency network. Front-end traffic load balanced with SmartConnect	Single network for both external and internal traffic
File system structure and NameSpace	The storage is completely virtualized to users as a truly single file system with one namespace. There is no partitioning or volumes. The single file tree can grow organically without requiring planning or oversight about how the tree grows. SmartPool handles tiering of files to appropriate disk, without disrupting the single file tree.	An appearance of single namespace is typically achieved through namespace aggregation, where files are still managed in separate volumes, and a simple “veneer” layer glues individual directories to a “top-level” tree via symbolic links. LUNs and volumes, as well as volume limits are still present. Files have to be manually moved from volume-to-volume to load-balance.
Data Layout	OneFS controls the placement of file directly, down to the sector-level on any drive anywhere in the cluster. The addressing scheme for data and metadata is indexed at physical level by a tuple of {node, drive, offset}	Data are sent through RAID and volume management layers, introducing inefficiencies in data layout and providing non-optimized block access.
Redundancy Control	OneFS can flexibly control the type of striping as well as the redundancy level of the storage system at the system, directory and even file-levels.	The entire RAID volume is dedicated to a particular performance type and protection setting.

Isilon terms

The Isilon technology re-implemented the read and write path during file storage and introduced several terms along with its technology.

SmartPools – Job that runs and moves data between the tiers of nodes within the same cluster. Also executes the CloudPools functionality if licensed and configured. FilePolicy is changelist-based SmartPools file pool policy job. SmartPoolsTree enforces SmartPools file policies on a subtree.

Storage Pools – Storage pools provide the ability to define subsets of hardware within a single cluster, allowing file layout to be aligned with specific sets of nodes through the configuration of storage pool policies. The notion of Storage pools is an abstraction that encompasses disk pools, node pools, and tiers.

Disk Pools – Disk pools are the smallest unit within the storage pools hierarchy. OneFS provisioning works on the premise of dividing similar nodes’ drives into sets, or disk pools, with each pool representing a separate failure domain. Disk pools are laid out across all five sleds in each node.

Node Pools – groups of disk pools, spread across similar storage nodes (or equivalent classes). Multiple groups of different node types can work together in a single, heterogeneous cluster. For example, one node pool of all-flash F-Series anodes, one node pool of H-series, and one node pool of A-series. Each node pool only contains disk pools from the same type of storage nodes.

Tiers – groups of nodepools combined into a logical superset to optimize data storage, according to OneFS platform type. this allows customers who consistently purchase highest capacity nodes available to consolidate a variety of node styles within a single tier, and manage them as one logical group. SmartPools users typically deploy 2 to 4 tiers. different node pools under a tier needs to be compatible.

Global Namespace Acceleration (GNA)’s principal goal is to help accelerate metadata read operations by keeping a copy of a cluster’s metadata on high performance, low latency SSD media.

SmartConnect is a load balancer that works at the front-end Ethernet layer to evenly distribute client connections across the cluster. SmartConnect supports dynamic NFS failover and failback to ensure that when a node failure occurs, or preventative maintenance is performed, all in-flight reads and writes are handed off to another node in the cluster to finish its operation without any user or application interruption.

Auto Balance reallocates and rebalances data and make storage space more usable and efficient.

SmartQuotas is directory-level quota management. Note: there is no partitioning, and no need for volume creation in OneFS.

SmartRead creates a data “pipeline” from L2 cache, prefetching into a local “L1” cache, on the captain node, in order to greatly improve sequential-read performance. For high-sequential cases, SmartRead can very aggressively prefetch ahead. SmartRead can control how aggresive the pre-fetching is, and how long data stays in the cache, and optimizes where data is cached.

In-line Data Reduction – the write path involves zero block removal, in-line deduplication, and in-line compression. This is supported in some models only.

Smart Dedupe – post-process, asynchronous deduplication. Smart Dedupe scans the on-disk data for identical blcoks and then eliminate the duplicates. After duplicate blocks are discovered, SmartDedupe movees a single copy of those blocks to a special set of files known as shadow stored. With post-process deduplication, new data is first stored on the storage device and then a subsequent process analyzes the data looking for commonality. This means that initial file write or modify performance is not impacted, since no additional computation is required in the write path, as opposed to in-line deduplication. This is supported on some models only.

OneFS SSD strategy – How OneFS leverage the SSD for performance. It has these options:

L3 cache (implemented at nodepool level)
metadata read
metadata read/write
Global Namespace Acceleration (GNA)
Data on SSD

L3 cache consumes all the SSD in node pool. L3 cannot coexist with other SSD strategies, with the exception of GNA just because L3 cache node pool SSD cannot participate in GNA.

Isilon’s High Availability

The OneFS is distributed across all nodes in the cluster and is accessible by clients connecting to any node in the cluster. Metadata and locking tasks are managed by all nodes collectively and equally in a peer-to-peer architecture. This symmetry is key to the simplicity and resiliency of the architecture. There is no single metadata server, lock manager or gateway node.

The entire cluster forms a single file system with a single namespace that runs across every node equally. No one node controls or “masters” the cluster; all nodes are true peers.

During failover, clients are evenly redistributed across all remaining nodes in the cluster, ensuring minimal performance impact. If a node is brought down for any reason, including a failure, the virtual IP addresses on that node is seamlessly migrated to another node in the cluster. When the offline node is brought back online, SmartConnect automatically rebalances the NFS and SMB3 clients across the entire cluster to ensure maximum storage and performance utilization. This functionality allows for per-node rolling upgrades affording full-availability throughout the duration of the maintenance window.

There are two logical roles in processing an I/O request from client:

The initiator: the node that the client connects to with front-end protocol. The initiator acts as the ‘captain’ for the entire I/O operation.
The participant: Every node in the cluster is a participant for a particular I/O operation.

File Write in Isilon

OneFS employs a patented transaction system during write to eliminate single point of failure. In a write operation, the initiator “captains” or orchestrates the layout of data and metadata, the creation of erasure codes, and the normal operations of lock management and permission control.

When a client connects to a node to write a file, it is connecting to the Initiator. OneFS breaks the file down into atomic units. An atomic unit is a smaller logical chunk of data, also called stripe, or protection groups in the context of data protection. The size of each file chunk is referred to as the stripe unit size. After this division, OneFS then write the stripe individually to the Participant (with disks). This design ensures that data is protected at the specified level as soon as it is being written. Redundancy is built into protection groups, such that if every protection group of a file is safe, then the entire file is safe. In terms of protection mechanism, OneFS can use either Reed-Solomon erasure coding system, or simply mirroring for data protection. Erasure coding is the predominant mechanism with very high performance without sacrificing on-disk efficiency.

The initiator node uses a modified two-phase commit transaction to safely distribute writes to multiple NVRAMs across the cluster. As client initiates write to OneFS cluster, instead of immediately writing to disk, OneFS temporarily writes the data to an NVRAM-based journal cache on the initiator node and acknowledge the write the client. As outlined above, these writes are also mirrored to participant nodes’ NVRANM journals to satisfy the file’s protection requirement. Later, at a more convenient time, OneFS then flush these cached writes to disks asynchronously.

Since NVRADM journals all the transactions that are occurring across every node in the storage cluster. If a node fails mid-transaction, and then re-joins the cluster, the uncommitted cached writes are fully protected, and the only required actions for the node, are to replay its journal from NVRAM, and occasionally for AutoBalance to rebalance files that were involved in the transaction. Writes are never blocked due to a failure. There is no ‘fsck’ or ‘disk-check’ process.

OneFS file system block size is 8KB. A file smaller than 8KB will use a full 8KB block. For larger files, OneFS can maximize sequential performance by taking advantage of a stripe unit consisting of 16 contiguous blocks, for a total of 128KB per stripe unit.

Cache in Isilon

OneFS aggregates the cache present on each node in a cluster into one globally accessible pool of memory by using a messaging system similar to NUMA (non-uniform memory access). This allows all the nodes’ memory cache to be available to each and every node in the cluster. Remote memory is access over internal network with much lower latency than accessing hard disk drives. The internal network as distributed system bus, is a redundant, under-subscribed flat Ethernet up to 40Gb. The oneFS caching subsystem is coherent across the cluster, due to the use of MESI protocol to maintain cache coherency. If the same content exists in the private caches of multiple nodes, this cached data is consistent across all instances.
OneFS uses up to three levels of read cache, plus an NVRAM-backed write cache, or coalescer.

OneFS Caching Hierarchy

L1 cache – prefetches data from remote nodes. Data is prefetched per file, and this is optimized in order to reduce the latency associated with the nodes’ back-end network. The L1 cache refers to memory on the same node as the initiator. It is only accessible to the local node, and typically the cache is not the master copy of the data.

L1 is also known as remote cache because it contains data retrieved from other nodes in the cluster. It is coherent across the cluster but is used only by the node on which it resides and is not accessible by other nodes. Data in L1 cache on storage nodes is aggressively discarded after it is used. L1 cache uses file-based addressing, in which data is accessed via an offset into a file object.

OneFS also uses a dedicated inode cache in which recently requested inodes are kept. The inode cache frequently has a large impact on performance, because clients often cache data, and many network I/O activities are primarily requests for file attributes and metadata, which can be quickly returned from the cached inode.

L2 cache (backend cache) refers to local memory on the node on which a particular block of data is stored. L2 cache is globally accessible from any node in the cluster and is used to reduce the latency of a read operation by not requiring a seek directly from the disk drives.

L2 cache is also known as local cache because it contains data retrieved from disk drives located on that node and then made available for requests from remote nodes. Data in L2 cache is evicted according to a Least Recently Used (LRU) algorithm.

L3 cache, or Smart Flash, is configurable on nodes that contain solid state drives. Smart Flash (L3) is an eviction cache that is populated by L2 cache blocks as they are aged out from memory.

During I/O request, clients talk to L1 cache and write coalescer; L1 cache talks to L2 cache on all cluster nodes. L2 cache buffers to and from disks. L3 cache is optionally enabled per node pool, as an extension from L2. L3 and L2 communicate in backend network.

Name	Medium	Description
L1 Cache (aka front-end cache or remote cache)	RAM (volatile)	holds clean, cluster coherent copies of file system data and metadata blocks requested by clients via front-end network
L2 Cache (aka back-end cache or local cache)	RAM (volatile)	contains clean copies of file system data and metadata on a local node
SmartCache (Write Coalescer)	Battery-backed NVRAM (Persistent)	a persistent journal cache that buffers any pending writes to front-end files that have not been committed to disk
SmartFlash or L3 Cache	SSD (persistent)	contains file data and metadata blocks evicted from L2 cache, effectively increasing L2 cache capacity

File Read in Isilon

The high-level steps for fulfilling a read request with cache interaction involves:

Step 1 – on local node, determine whether part of the requested data is in the local L1 cache:

if so, return to client
if not, request data from remote nodes

Step 2 – on remote nodes, determine whether requested data is in the local L2 or L3 cache:

if so, return to the requesting node
if not, read from disk and return to requesting node

During a read operation, the “captain” node gathers all of the data from the various nodes in the cluster and presents it in a cohesive way to the requestor.

The cluster provides a high ratio of cache to disk (multiple GB per node) that is dynamically allocated for read and write operations as needed. This RAM-based cache is unified and coherent across all nodes in the cluster, allowing a client read request on one node to benefit from I/O already transacted on another node. As the cluster grows larger, the cache benefit increases. For this reason, the amount of I/O to disk on a cluster is generally substantially lower than it is on traditional platforms.

For files marked with an access pattern of concurrent or streaming, OneFS can take advantage of pre-fetching of data based on heuristics used by the SmartRead component

Conclusion

This post provided a high level introduction to EMC storage product line and expanded into some technical details in the read write operation in OneFS/Isilon. Some of the features can be seen in OneFS simulator which is a free tool from EMC.

Performance Analysis

Fri, 19 Jun 2020 16:47:01 -0400

Overview

In 2015, Brendan Gregg posted two great articles on Netflix blog: Linux Performance Analysis in 60 seconds, and Linux Perfomrance Tools. They have great value when I was in a urgency to spot performance issues. The articles cover the essential tools for performance troubleshooting, including:

Check out load averages: w or uptime
Print kernel ring buffer: dmesg -T
Virtual memory status: vmstat 1
Multiple processor staticstics: mpstat -P ALL 1
Task status: pidstat 1
CPU and I/O status: iostat -xz 1
Free memory check: free -m
Network Activity record: sar -n DEV 1
TCP activity record: sar -n TCP,ETCP 1
Display processes: top

We will dive into each of them in the next section.

Basic Troubleshooting

The command w is equivalent of uptime (which shows uptime since boot) and who (which shows logged-in users). It also displays load average for the last 1 minute, 5 minutes and 15 minutes. The number of load average reflects the overall system load (CPU + disks), and it is further discussed in this post with a simple take away:

If the averages are 0.0, then your system is idle.
If the 1 minute average is higher than the 5 or 15 minute averages, then load is increasing.
If the 1 minute average is lower than the 5 or 15 minute averages, then load is decreasing.
If they are higher than your CPU count, then you might have a performance problem (it depends).

When Linux load averages increase, you know you have higher demand for resources (CPUs, disks, and some locks), but you aren’t sure which. You will need to switch to other metrics. Brendan recommend don’t spend more than 5 seconds on these numbers.

[dhunch@c7v-ghintapp01 ~]$ w
 12:14:10 up 46 days, 16:41, 3 users, load average: 2.69, 2.44, 2.29
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
dhunch pts/0 w6v-ghas01 24Jun20 3days 0.36s 0.30s ssh c7v-bastion
dhunch pts/1 202.95.88.111 12:02 2.00s 0.00s 0.00s w
dhunch pts/4 w6v-ghas01 17Jun20 15days 0.15s 0.07s view readme.txt

Before moving to more insightful metrics, it is also worth a quick look into the kernel ring buffer with dmesg command (dmesg -T | less +G). This will allow us to capture obvious issues such as oom-killer or TCP request dropping.

The vmstat tool reports the statistics of virtual memory. Servers have a fixed amount of physical memory, but they can run a set of applications that use a much larger amount of virtual memory. Application tend to reserve more memory than they need, and they usually operate on only a subset of their memory. In both cases, the operating system can keep the unused parts of memory on disk, and page it into physical memory only if it is needed. For the most part, this kind of memory management works well. But it doesn’t always with Java applications due to Java heap. Once a system start swapping – moving pages of data from main memory to disk, and vice versa, the performance tend to be bad. Systems must be configured so that swapping never occurs.

[dhunch@c7v-ghintapp01 ~]$ vmstat 1
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r b swpd free buff cache si so bi bo in cs us sy id wa st
 3 0 239360 385928 0 36734692 0 0 1 200 0 0 6 1 93 0 0
 3 0 239360 387732 0 36734704 0 0 0 43 8017 8524 14 0 85 0 0
 3 0 239360 387608 0 36734904 0 0 0 57 6768 7680 14 1 86 0 0
 2 0 239360 389008 0 36734904 0 0 0 44 6366 7300 14 0 86 0 0
 3 0 239360 421728 0 36700144 0 0 0 0 8141 7957 13 0 86 0 0
 2 0 239360 421984 0 36702048 0 0 0 467 8994 8362 14 0 85 0 0

The tool prints key server statistics each line, with the first line showing the average since boot. Here lists the explanation of some columns:

r: number of processes running on CPU and waiting for a turn. This provides a better signal than load averages for determining CPU saturation, as it does not include I/O. To interpret: an “r” value greater than the CPU count is saturation.
swpd: the amount of virtual memory used. This number should align with the used column for Swap row from free command.
buff, cache: buffer and cache. They should align with the buff/cache column form Mem row from free command.
free: free memory in kilobytes. This number should align with the free column for Mem row from free command.
si, so: swap-ins and swap-outs. As mentioned, if these are non-zero, you’re out of memory.
bi, bo: blocks received from and sent to a blcok device (# of block per second)
in, cs: number of interrupt, and context switches per second.
us, sy, id, wa, st: user, system, idle, wait I/O and stolen times. These are breakdowns of CPU time, on average across all CPUs. They should add up to 100% (or close). stolen time is amount of CPU time needed by a guest virtual machine that is not provided by the host. IO wait time is the CPU time waiting for I/O activity. Idle time could be several things: the process may be waiting for something (e.g. a response from database); the process may be blocked by a thread lock; or the process simply has nothing to do. user and system times are CPU times spent on user tasks and kernel tasks, respectively.

Out of these columns, watch for columns r, free, buff, cache, us, sy, id and wa at minimum. The combination of us and sy confirms if CPUs are busy. A constant degree of wa points to a disk bottleneck with too much time spent on pending disk I/O. The sy (kernel time) is necessary for I/O processing but sy stays high (e.g. constantly over 20%), it becomes interesting. Perhaps the kernel is processing I/O inefficiently.

For further per-CPU stats, use mpstat command (-P ALL), to prind CPU time breakdowns per CPU and check for imbalance. A single host CPU can be evidence of a single-threaded application. Here is an example output from a system of 16 CPU cores.

[dhunch@c7v-ghintapp01 ~]$ mpstat -P ALL 1
Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU)

13:47:23 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
13:47:24 all 13.77 0.00 0.19 0.00 0.00 0.00 0.00 0.00 0.00 86.05
13:47:24 0 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 1 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 2 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 3 98.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00
13:47:24 4 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 5 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 6 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 7 2.94 0.00 0.98 0.00 0.00 0.00 0.00 0.00 0.00 96.08
13:47:24 8 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 9 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 10 97.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00
13:47:24 11 0.99 0.00 0.99 0.00 0.00 0.00 0.00 0.00 0.00 98.02
13:47:24 12 2.02 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 97.98
13:47:24 13 1.98 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.02
13:47:24 14 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.00
13:47:24 15 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00

For a per process summary of CPU consumption, use pidstat command. It can be thought of a periodical snapshot of top command, allowing you to watch for patterns. The %CPU column is the total across all CPUs so 5 CPUs have a maximum value of 500.

If vmstate indicates some I/O issue, iostat tool can help us understand block devices, on both the workload applied and the resulting performance. Key columns are:

r/s, w/s, rkB/s, wkB/s: delivered reads, writes, read Kbytes, and write Kbytes per second to the device. Use these for workload characterization. A performance problem may simply be due to an excessive load applied.
await: the average wait time for I/O in milliseconds. This is the time that the application suffers, as it includes both time queued and time being serviced. Larger than expected average times can be an indicator of device saturation, or malfunction.
avgqu-sz: the average number of requests issued to device. Values greater than 1 can be evidence of saturation (although devices can typically operate on requests in parallel, especially virtual devices which front multiple back-end disks.)
%util: device utilization. This is really a busy percent, showing the time each second that the device was doing work. Values greater than 60% typically lead to poor performance (which should be seen in await), although it depends on the device. Values close to 100% usually indicate saturation.

I/O problem may either be inefficiencies in application that issues I/O request, or slowing disk unable to keep up with I/O requests. We review two examples here to illustrate each situation. The first output is as follows:

% iostat -xm 5
avg-cpu: %user %nice %system %iowait %steal %idle
 23.45 0.00 37.89 0.10 0.00 38.56

 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
 sda 0.00 11.60 0.60 24.20 0.02 0.14 13.35 0.15 6.06 5.33 6.08 0.42 1.04

In the first example, the disk stat loosk up at first glance. The w_await (time to service I/O write) is fairly low at 6.08ms. However, the system is spending 37.89% of its time in the kernel. If all that system time is from the application, it suggest something inefficient is happening. The fact that the system is doing 24.2 writes per second is another clue: that is alot when writing only 0.14 MB per second (MBps). I/O has become a bottleneck, and the next step would be to look into how the application is performing its writes.

The second example output is as follows:

% iostat -xm 5
avg-cpu: %user %nice %system %iowait %steal %idle
 35.05 0.00 7.85 47.89 0.00 9.20

 Device: rrqm/s wrqm/s r/s w/s rMB/s wMB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
 sda 0.00 0.20 1.00 163.40 0.00 81.09 1010.19 142.74 866.47 97.60 871.17 6.08 100.00

In this example, it tells us that processes are spending 47.89% of their time in iowait, and the data to complete the I/O (w_await) is 871ms, the queue size is large, and the disk is writing at 81MB per second. This all points to disk I/O as a problem and that the amount of I/O in the application (or elsewhere in the system) must be reduced.

Bear in mind that poor performing disk I/O isn’t necessarily an application issue. Many techniques are typically used to perform I/O asynchronously, so that the application doesn’t block and suffer the latency directly (e.g. read-ahead for reads, and buffering for writes, also refer to “tail latency“).

Note that the acceptable utilization metric depends on the configuration of block device. If the storage is a logical disk device fronting many back-end disks (e.g. RAID 0), then 100% utilization may just mean that some I/O is being processed 100% of the time, however, the back-end disks may be far from being saturated, and may even be able to handle more work.

The free command gives the breakdown of memory usage. The right two columns are:

buffers: for the buffer cache, used for block device I/O.
cached: for the page cache, used by file systems.

We just want to check that these aren’t near-zero in size, which can lead to higher disk I/O (confirm using iostat), and worse performance. Linux uses free memory for the caches, but can reclaim it quickly if applications need it. So in a way the cached memory should be included in the free memory column. In this case, it’s included in the available column. This website has further details.

To check interface stat, nicstat is a great tool but it isn’t available by default in Linux. Instead we can run sar (-n DEV) to retrieve stats.

[dhunch@c7v-ghintapp01 ~]$ sar -n DEV 1
Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU)

16:51:25 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
16:51:26 eth0 3089.00 934.00 3815.33 834.61 0.00 0.00 0.00
16:51:26 lo 464.00 464.00 2289.07 2289.07 0.00 0.00 0.00

16:51:26 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
16:51:27 eth0 956.00 586.00 826.66 211.34 0.00 0.00 0.00
16:51:27 lo 213.00 213.00 196.00 196.00 0.00 0.00 0.00

16:51:27 IFACE rxpck/s txpck/s rxkB/s txkB/s rxcmp/s txcmp/s rxmcst/s
16:51:28 eth0 349.00 181.00 52.32 147.19 0.00 0.00 0.00
16:51:28 lo 244.00 244.00 81.13 81.13 0.00 0.00 0.00

Here rxkB/s/s and txkB/s represents receive and transmission throughput, as a measure of workload. If they reach the limit then the interface is the bottleneck.

On top of interface is the TCP layer, which can be monitored with sar again (-n ECP, ETCP). The key metrics are:

active/s: number of locally-initiated (e.g. via connect()) TCP connections per second
passive/s: number of remotely-initiated (e.g. via accept()) TCP connections per second
retrans/s: number of TCP retransmits per second

The active and passive counts are often useful as a rough measure of server load. It might help to think of active as outbound, and passive as inbound, but this isn’t strictly true (e.g. consider a localhost to localhost connection). Retransmits are a sign of network or server issue; it may be an unreliable network (e.g. public Internet), or it may be due to a server being overloaded and dropping packets.

[dhunch@c7v-ghintapp01 ~]$ sar -n TCP,ETCP 1
Linux 3.10.0-1062.12.1.el7.x86_64 (c7v-ghintapp01.digihunch.com) 08/01/20 _x86_64_ (16 CPU)

16:52:29 active/s passive/s iseg/s oseg/s
16:52:30 0.00 1.00 28.00 35.00

16:52:29 atmptf/s estres/s retrans/s isegerr/s orsts/s
16:52:30 0.00 0.00 0.00 0.00 0.00

16:52:30 active/s passive/s iseg/s oseg/s
16:52:31 8.00 8.00 200.00 317.00

16:52:30 atmptf/s estres/s retrans/s isegerr/s orsts/s
16:52:31 0.00 1.00 1.00 0.00 3.00

Last but not least is our favourite command top, which includes many of the metrics covered in previous tools. The downside to top is it is harder to see patterns over time, which may be more clear in tools like vmstat and pidstat, both of which produce rolling output.

Several tools introduced here involves sar, which is a great monitoring tool on its own that we need to be familiar with.

System Activity Report (SAR)

Further to the basic tools, the sar command is very helpful as it stores historical stat every 10 minutes. Sar keeps 18 types of reports, all stored in /var/log/sa/. When viewing the report file, you may pipe the result to less command so it only prints header once. For example, if you would like to print CPU report for the 2nd of the month:

# sar -u -f /var/log/sa/sar02 | less

If you check NFS client statistics for the 31st

 # sar -n NFS -f /var/log/sa/sar31

If you need to check network server statistics for today

# sar -n NFS -f /var/log/sa/sar31

Below are all types of reports:

-u CPU utilization
-w task creation and system switching activity
-W swapping statistics
-B report paging
-b report I/O and transfer rate statistics
-R report memory statistics
-r memory utilization
-S swap space utilization
-H huge pages utilization statistics
-v inode
-q queue length and load average
-y TTY device activity
-d activity for each block device
-n network statistics, DEV (per interface)
-n network statistics, EDEV (error per interface)
-n network statistics, NFS (NFS client)
-n network statistics, NFSD (NFS server)
-n network statistics, SOCK (socket usage)

Berkeley Packet Filter (BPF) Compiler Collection (bcc) tools

For advanced, low-level performance troubleshooting, the BCC tools provide a suite of tools. Here we only cover the installation of it.

In CentOS, install bcc-tools package with yum. When you try to run a command, such as cachestat, if you come across this error:

-bash: cachestat: command not found

Then you will need to add its path to default:

export PATH=$PATH:/usr/share/bcc/tools

Now if you run into this error:

chdir(/lib/modules/3.10.0-1062.12.1.el7.x86_64/build): No such file or directory
Traceback (most recent call last):

The file listed is a symbolic link, and if it is missing, you just need to install kernel-headers that matches the kernel version:

yum install kernel-headers

Then you may use tools in /usr/share/bcc/tools. For example, cachestat help you display page cache hit ratio; gethostlatency shows DNS resolution latency; tcpconnect prints out active tcp connections (made via connect system call):

[root@dhunch ~]# /usr/share/bcc/tools/tcpconnect -t -P 8080 | gawk '{ print strftime("%F %T  "), $0 }'
2020-06-13 00:16:57   TIME(s)  PID    COMM         IP SADDR            DADDR            DPORT
2020-06-13 02:16:57   0.000    15241  QNetworkAcce 4  10.100.22.21    10.101.84.10    8080
2020-06-13 02:16:57   0.064    15241  QNetworkAcce 4  10.100.22.21    10.101.84.10    8080
2020-06-13 02:16:57   0.438    15241  QNetworkAcce 4  10.100.22.21    10.101.84.10    8080

The command above outputs a time and pid stamped log line every time a TCP connection is made to port 8080; tcpaccept traces passive tcp connections (via accept system call). These tools are not as intrusive as tcpdump.

It is beyond the purpose of this article to get into details of each tool in the BFP suite. The tools are covered in detail in books “BPF Performance Tools” and “Linux Observability with BPF”.

Capture filter and Display filter in Network Analyzer

Wed, 10 Jun 2020 21:21:18 -0400

Capture filter is set before collecting packets. It is applied at the time of data acquisition and it impacts the size of the capture. It does not have as many variations as display filter and is usually not aware of protocols above TCP/UDP layer. A common form of capture filter is BPF (Berkerly Packet Filter) which is used in Linux Socket Filtering (e.g. tcpdump).

Basic form is:

[tcp|udp] [src|dst] host 192.168.1.2 port 1234

For example:

'tcp dst port 8080 and src host 147.206.160.9'

Display filter is set after packet collection. It is applied at the time of data manipulation. It does not impact the size of capture, but it controls how the data is presented (typically for analysis purpose). Display filter may support a variety of expressions that are interpreting data at TCP/UDP layer or above, for example HTTP.

Here are some examples:

(tcp.flags.syn == 1) || (tcp.flags.reset == 1)
(tcp.flags.reset == 1) || (http.request.method==GET) ||
(tcp.flags.reset == 1)||(http.request.uri contains "/box/url/string")
||(http.response.code == 200)

Here are some further examples provided by Wireshark.

For more details about the usage of capture filter and display filter, here is a page with cheatsheet. Example for tcpdump on the left and wireshark in the middle and on the right.

To view http packet in shell terminal, there is also a helpful tool called httpry. You can applied BPF styled filter for capture, and organize display column. The drawback is there is no display filter so you’ would have to use grep. Here is an example:

httpry -i eth0 'tcp dst port 8080 and src host 147.206.160.9' -m GET -f Timestamp,x-correlation-id,x-userid,Request-URI | grep -v -P '\t\-\t'

Networking Basics 3 of 3 – common network protocols and technologies

Fri, 20 Dec 2019 10:28:00 -0400

The 5 layer TCP/IP model (or its more rigorously defined alternative OSI model) leads to a whole world of network protocols. Understanding these new protocols requires one to map it out agains the network layers (e.g. at Layer 4 whether it is TCP or UDP, etc) .

VPN is a whole family of technologies with many flavours of implementation. The previous posting covered some basics of the idea, as well as the two common forms (remote access VPN and site-to-site VPN). The VPN implementation protocols vary a lot.

PPTP(Point-to-Point Tunnelling Protocol) is outdated and less secure
IPSec (Internet Protocol Security)
L2TP (Layer 2 Tunnelling Protocol) replacement of PPTP, more secure, more overhead and slightly slower.
OpenVPN – very secure, and reliable and supported by communities all over the world.
TLS/SSL and SSH connections may be considered VPN as well.

Phone service protocols:

VoIP (voice over IP, operating at network layer) – allows one to make and receive phone calls over the network. Communication on the IP network is perceived as less reliable in contrast to the circuit-switched public telephone network because it does not provide a network-based mechanism to ensure that data packets are not lost, and are delivered in sequential order. It is a best-effort network without fundamental Quality of Service (QoS) guarantees. Voice, and all other data, travels in packets over IP networks with fixed maximum capacity. This system may be more prone to data loss in the presence of congestion[a] than traditional circuit switched systems; a circuit switched system of insufficient capacity will refuse new connections while carrying the remainder without impairment, while the quality of real-time data such as telephone conversations on packet-switched networks degrades dramatically. Therefore, VoIP implementations may face problems with latency, packet loss, and jitter.
SIP (session initiation protocol, operating at application layer) – a VOIP signaling protocol responsible for the creation and tearing down of media connections. So it supports all types of media.

ALG (application layer gateway, aka proxy server) – a software component that manages specific application protocols such as SIP and FTP. An ALG acts as an intermediary between the Internet and an application server that can understand the application protocol. ALG proxies connection to destination on behalf of client. This adds

Application Layer Gateway

DDNS (dynamic domain name service) – a router service that assigns your device a fixed domain name even though you are using dynamic IP.

NAT – another family of technologies, usually implemented in the following

Port preservation: source port chosen by a client is the same port used by the router
Port forwarding: NAT application redirects a communication request from one address and port number combination to another while the packets are traversing a network gateway, such as a router or firewall.
Port triggering: a dynamic form of the port forwarding model. Generally, port triggering is used when the user needs to use port forwarding to reach multiple local computers. Port are close when they aren’t in use (more secure) protocol used is UPnP

DMZ – a physical or logical subnet that contains external facing service to untrusted network (e.g. Internet). The purpose is to add an additional layer of security so an external network can assess what is exposed in DMZ while the rest of network remains firewalled.

VPN passthrough – a feature that allows any device connected to the router to establish outbound VPN connections. Most modern router already have this built in.

WAN optimization – a collection of techniques for increasing data transfer efficiencies across wide-area networks.

Deduplication
Compression
Latency optimization
Caching/proxy
Protocol spoofing
Traffic shaping

Network performance tuning

Performance tuning in network covers a variety of skills. It is important to understand in which layer the problem occurs. Anything above layer 4 is more likely to be an application issue. For ethernet performance tuning, I found this page and this page to be helpful in my practices.

Networking basics 2 of 3 – Layer 4 and common network configurations

Sat, 07 Dec 2019 23:19:00 -0400

Transport Layer

Transport Layer handles multiplexing & de-multiplexing through ports. Port is more or less a virtual concept. Source port is usually ephemeral. Two dominant protocols are TCP and UDP.

TCP relies on acknowledgement. TCP control flags are SYN, ACK, FIN, URG, PSH, RST, ECE, CWR. TCP connection is established by 3-way handshake and torn down by 4-way termination.

TCP handshake and termination

Socket – the instantiation of an end-point in a potential TCP connection. A socket can be in one of the following states:

LISTEN: a TCP socket is ready and listening for incoming connections;
SYN_SENT: a SYNC request has been sent but connection hasn’t been established yet;
SYN_RECEIVED: a socket previously in a LISTEN state has received a SYNC request and sent a SYN/ACK back;
ESTABLISHED: connection is up;
FIN_WAIT: FIN sent, ACK hasn’t been received yet;
CLOSE_WAIT: connection has been closed at the TCP layer but the application that opened the socket hasn’t release the hold on the socket yet;
CLOSED: connection fully terminated;

TCP packet format

TCP is a connection-oriented protocol

Transport layer is responsible for re-sending data if data is lost
Sequence # is important because packet may arrive out of sync but receiver reassemble them in order
There is a lot of overhead (acknowledgement, establish connection first, tear down connection afterwards)

On the other hand, UDP is connectionless. A good example is video streaming, where it is okay to lose a few packet along the way, in exchange of bandwidth saving.

Firewall may operate at different layers but it is most commonly used at transport layer, to block traffic based on port.

Application Layer

There is no dominant protocol at this layer. IIS, Nginx and Apache are examples of applications operating at this layer.

Basic network configurations

Standard modern network configuration involves: IP address, subnet mask, gateway and DNS server.

DNS – global and highly distributed network service that resolves domain name into IP address. There are many steps in DNS resolution. DNS service listens on port 53. Two famous free public DNS servers are 8.8.8.8 and 8.4.4.4. DNS servers have five categories:

Caching name servers: store known domain name lookups in cache. TTL today can be a few hours, much shorter than what it used to be in early days;
Recursive name servers: perform full DNS resolution request;
Root name servers;
TLD name servers;
Authoritative name servers;

DNS resolution steps

DNS uses UDP protocol and it can generate a lot of traffic (TCP is impractical. If implemented in TCP, it would have required 44 packet for a DNS query, which is very expensive considering DNS query is just a precursor of the real traffic)

Anycast DNS – any one of a number of DNS servers can respond to DNS queries, and typically the one that is geographically closest will provide the response. This reduces latency, improves uptime for the DNS resolving service and provides protection against DNS flood DDoS attacks.

DNS record types:

A record: domain name to IP address. DNS service round robin across multiple A records
AAAA (quad A) record: domain name to IPv6 address
CNAME: redirect traffic from one domain to another (e.g. test.com to www.test.com so you can minimize IP references)
MX record
SRV record
TXT record: originally for human consumption, freeform text for configuration purpose.

A FQDN (fully qualified domain name) can have up to 127 domains, but only three in most cases. (i.e. subdomain.domain.topleveldomain)

DNS zones – allow for easier control over multiple levels of a domain. DNS zones are configured in zone files.

Domains vs zones – Domains are broken into zones for which individual DNS servers are responsible. A domain represents the entire set of names/machines that are contained under an organizational domain name. For example, all domain names ending with “.com” are part of the “com” domain. A “zone” is a domain less any sub-domains delegated to other DNS servers. A DNS server could be responsible (authoritative) for all records under the “xyz.com” domain, but by defining NS-records for “abc.xyz.com”, this part of the domain is delegated to other DNS servers – and possibly a different company/entity. A zone contains exactly one SOA-record describing the general properties of the zone, and any number of other DNS records. Entire zones can transferred from a primary DNS server to secondary DNS servers through Zone Transfers. A domain administrator would be responsible for creating zones, and delegating responsibility for these zones to an administrator and DNS server.

Reverse DNS lookup – query for FQDN by IP. This is commonly used by email servers where anti-spam mechanism on the receiver needs to validate that sender’s IP is associated with a domain as claimed. This is also used in logging application to convert IP into human-readable domains in the log data. Reverse DNS lookups query DNS server for a PTR (pointer reserve record). If the server does not have a PTR record, it cannot resolve a reverse lookup.

DHCP

DHCP operates at application layer and helps you to configure IP automatically with a lease, through automatic allocation, or fixed allocation based on MAC, etc. DHCP process involves address allocation, renewal, and release. Address allocation takes four steps:

Client sends a broadcast to discover DHCP server;
DHCP server broadcast a DHCP offer;
Client requests IP address from the DHCP server;
Server acknowledged the DHCP request;

Here is an illustration of DHCP address allocation.

DHCP can also be used to set NTP address.

NAT

Network Address Translation (NAT), can be implemented in many different ways in different OS. Essentially, it is a technology that allows a gateway, usually a router or firewall, to rewrite the source IP of an outgoing IP datagram while retaining the original IP in order to rewrite it into the response. Two categories of NAT are:

Basic NAT: provides a one-to-one translation of IP addresses, aka one-to-one NAT. Basic NATs can be used to interconnect two IP networks that have incompatible addressing.
One-to-many NAT: maps multiple private hosts to one publicly exposed IP address, aka IP masquerading. This can be a security measure so that no external host can establish to your computer without knowing your actual IP. Source port conflict can be managed in two ways:
- port preservation: When making outgoing connection, NAT preserves the ephemeral port number used by internal client that initiates the connection; if two clients happen to use the same ephemeral port, then NAT picks a random port to initiate outgoing TCP connection;
- port forwarding (port mapping): Forward traffic to certain destination based on the port of incoming request that NAT receives.

Proxy

Proxy refers to a concept rather than a specific implementation. It exists on almost every layer in the network model, and act on behalf of a client in order to access other service. For example,

Web proxy used to be used to cache web traffic data in slow Internet but it is not necessary any more because 1) there is not much speed benefit; 2) website today is much more dynamic.
Reverse proxy is a popular architecture of web server, such as Nginx, to act as a front end of web servers, as well as point of decryption so web servers are free to just serve the content.

VPN

VPN – allows for extension of a private or local network to host that might not be on that local private network by using encrypted tunnel. There are many flavours of implementation for many purposes. It is a general concept rather than a specific protocol (just like NAT). VPN client provisions the computer with a virtual interface with an IP that matches the address space of the private network, and establish a VPN tunnel to it. Most VPNs work by using the payload section of transport layer to carry an encrypted payload that actually contains an entire second set of packets: the network, the transport and the application layers of a packet intended to traverse a network. Basically, this payload is carried to the VPN’s endpoint where all the other layers are stripped away and discarded. Then, the payload is unencrypted, leaving the VPN server with the top three layers of a new packet. This gets encapsulated with the proper datalink layer information and sent across the network. This process is completed in the inverse in the opposite direction. VPN usually requires strict authentication procedures and encryption. VPN can also be used to establish site-to-site connection (aka point-to-point VPN) where individual user doesn’t have to establish connections on their own. Both sites needs specialized hardware to achieve this. Site-to-site VPN is a good alternative to WAN when two site don’t need to transfer large amount of data for very fast speed.

WAN

Wide Area Network – act like a single network, but span across multiple physical locations, it requires that you contract the link across the internet with ISP. ISP handles data link from one site to another.

Wireless Network

Wireless protocol (802.11 family) defines how Wifi operates at physical and data link layer. Wifi networks operates on 2.4GHz and 5GHz frequency bands. Wireless frame is fairly different from Ethernet frame due to the nature of wireless transmission.

Wireless frame

Wireless access point is a device that bridges the wireless and wired portions of a network. A single wired network might have many wireless access points to cover a large area.

Wireless network can be configured in a few main ways:

Ad-hoc network – nodes all directly speak to each other. No supporting infrastructure is needed but not most common. It can be powerful tool during disasters.
Wireless LAN (WLAN) – one or more access points act as abridge between wireless and wired network. This is the most common type in business world where the wired LAN provides link to the Internet.
Mesh networks – a hybrid of the two above

Wireless Security

Wireless transmission is across the air so encryption is more important. The number of bit in the encryption key corresponds to how secure the encryption is.

WEP (encryption technology) provides low level of privacy (40-bit encryption) and it is not preferred today;
WPA provides 128-bit key encryption;
WPA2 provides 256-bit key encryption and is most common today.
MAC filtering also help security in wireless

Basic networking troubleshooting

ICMP ping to test general quality of connection;
traceroute discovers the path between two nodes and give you the information along the way;
netcat checks port and host address (telnet is retiring);
nslookup: very powerful in interactive mode for resolution tools;

IPv6

IPv4 address running out in November 2019 is a major crisis. IPv6 becomes more critical to implement for many organizations. IPv6 address is very long with numbers. Numbers represents a natural way that computer thinks and operates, but not for human. So IPv6 comes with two rules to shorten the address:

You may remove any leading 0;
Any number of consecutive groups can be replaced with two colons;

IPv6 header looks like below:

IPv6 header

Coexistence – the optimal approach for existing networks is to focus not on transition but on coexistence. Coexistence may live a long period with these phases: 1) Turn on IPv6 routing in their existing IPv4 networks and start using it; 2) Contract IPv6 service with their upstream, peer, and downstream neighbours; 3) Use the IPv6 protocol in addition to IPv4 in their applications and services both on server equipment and on their clients; 4) turn off IPv4 at some point when it is no longer a business requirement.

Storage Nitty-Gritty 5 of 5 – Replication

Tue, 19 Nov 2019 00:10:23 -0400

Replication Terms

PIT (point in time) replica – snapshot of the source at some specific timestamp;
Continuous Replica – always in-sync with the production data;
Recoverability – enables restoration of data from the replica to the source if data loss or corruption occurs;
Restartability – enables restarting business operations using the replicas;

Local Replication

Use Case:

Alternative source for backup
Fast recovery
Decision-support activities such as data warehousing
Testing platform
Data migration

Consistency in file system replication
File systems buffer the data in the host memory to improve the application response time. The buffered data is periodically written to the disk. In UNIX operating systems, sync daemon is the process that flushes the buffers to the disk at set intervals. In some cases, the replica is created between the set intervals, which might result in the creation of an inconsistent replica. Therefore, host memory buffers must be flushed to ensure data consistency on the replica, prior to its creation.

Flushing the file system buffer

In the illustration above, If the host memory buffers are not flushed, the data on the replica will not contain the information that was buffered in the host. If the file system is unmounted before creating the replica, the buffers will be automatically flushed and the data will be consistent on the replica.

Consistency in database replication
When a database is replicated while it is online, changes made to the database at this time must be applied to the replica to make it consistent. A consistent replica of an online database is created by using the dependent write I/O principle or by holding I/Os momentarily to the source before creating the replica.

A dependent write I/O principle is inherent in many applications and database management systems (DBMS) to ensure consistency. According to this principle, a write I/O is not issued by an application until a prior related write I/O has completed. For example, a data write is dependent on the successful completion of the prior log write.

For a transaction to be deemed complete, databases require a series of writes to have occurred in a particular order. These writes will be recorded on the various devices or file systems.

Another way to ensure consistency is to make sure that the write I/O to all
source devices is held for the duration of creating the replica. This creates a
consistent image on the replica. However, databases and applications might time out if the I/O is held for too long.

Local Replication Technologies

Host-based Local Replication

LVM-based replication: logical volume manager (LVM) is responsible for creating and controlling the host-level logical volumes. Each logical block in a logical volume is mapped to two physical blocks on two different physical volumes. LVM-based replication is part of operating system and comes without additional license cost. However, every write generated by application translates into two writes on the disk, and thus, an additional burden is placed on the host CPU. This can degrade application performance. Presenting an LVM-based logical replica to another host is usually not possible because the replica will still be part of the volume group, which is accessed by one host at any given time. You can’t track changes on LVMs either so it does not support incremental resynchronization.

File system snapshot: a pointer-based replica that requires a fraction of the space used by the production FS. This snapshot can be implemented by either FS or by LVM. It uses the Copy on First Write (CoFW) principle to create snapshot. When a snapshot is created, a bitmap and blockmap are created in the metadata of the Snap FS. The bitmap is used to keep track of blocks that are changed on the production FS after the snap creation. The blockmap is used to indicate the exact address from which the data is to be read when the data is accessed from the Snap FS. Immediately after the creation of the FS snapshot, all reads from the snapshot are actually served by reading the production FS. In a CoFW mechanism, if a write I/O is issued to the production FS for the fi rst time after the creation of a snapshot, the I/O is held and the original data of production FS corresponding to that location is moved to the Snap FS. Then, the write is allowed to the production FS. The bitmap and blockmap are updated accordingly. Subsequent writes to the same location do not initiate the CoFW activity. To read from the Snap FS, the bitmap is consulted. If the bit is 0, then the read is directed to the production FS. If the bit is 1, then the block address is obtained from the blockmap, and the data is read from that address on the Snap FS. Read requests from the production FS work as normal.

File system snapshot

Storage Array-based local replication
the array-operating environment performs the local replication process. The host resources, such as the CPU and memory, are not used in the replication process. Consequently, the host is not burdened by the replication operations. The replica can be accessed by an alternative host for other business operations.

Full-Volume Mirroring – the target is attached to the source and established as a mirror of the source. After all the data is copied and both the source and the target contain identical data, the target can be considered as a mirror of the source. After the synchronization is complete, the target can be detached from the source and made available for other business operations. The target becomes a point-in-time (PIT) copy of the source. After detachment, changes made to both the source and replica can be tracked at some predefined granularity. This enables incremental resynchronization (source to target) or incremental restore (target to source). The granularity of the data change can range from 512 byte blocks to 64 KB blocks or higher.

Full volume mirroring

Pointer-based, Full-Volume Replication – the target is immediately accessible by the BC host after the replication session is activated. Therefore, data synchronization and detachment of the target is not required to access it.

Pointer-based, Virtual Replication – at the time of the replication session activation, the target contains pointers to the location of the data on the source. The target does not contain data at any time. Therefore, the target is known as a virtual replica. the target is immediately accessible after the replication session activation. A protection bitmap is created for all data blocks on the source device. Granularity of data blocks can range from 512 byte blocks to 64 KB blocks or greater.

Network-based local replication: the replication occurs at the network layer between host and storage arrays. By offloading replication from servers and arrays, network-based replication can work across a large number of server platforms and storage arrays, making it ideal for highly heterogeneous environments.

Continuous Data Protection: CDP provides the ability to restore data to any previous PIT. In CDP, data changes are continuously captured and stored in a separate location from the primary storage. With CDP, recovery from data corruption poses no problem because it allows going back to a PIT image prior to the data corruption incident. CDP uses a journal volume to store all data changes on the primary storage. The journal volume contains all the data that has changed from the time the replication session started. The amount of space that is configured for the journal determines how far back the recovery points can go. CDP appliance is an intelligent hardware platform that runs the CDP software and manages local and remote data replications. Write splitters intercept writes to the production volume from the host and split each write into two copies. Write splitting can be performed at the host, fabric, or storage array.

CDP Local Replication Operation: before the start of replication, the replica is synchronized with the source and then the replication process starts. After the replication starts, all the writes to the source are split into two copies. One of the copies is sent to the CDP appliance and the other to the production volume. When the CDP appliance receives a copy of a write, it is written to the journal volume along with its timestamp. As a next step, data from the journal volume is sent to the replica at predefi ned intervals.

Continuous Data Protection

Tracking Changes to Source and Replica
Changes can occur on the replica device if it is used for other business operations. To enable incremental resynchronization or restore operations, changes to both the source and replica devices after the PIT should be tracked.
This is typically done using bitmaps, where each bit represents a block of data. For example, if the block size is 32 KB, then a 1-GB device would require 32,768 bits (1 GB divided by 32 KB). The size of the bitmap would be 4 KB. If the data in any 32 KB block is changed, the corresponding bit in the bitmap is flagged. If the block size is reduced for tracking purposes, then the bitmap size increases correspondingly.
The bits in the source and target bitmaps are all set to 0 (zero) when the replica is created. Any changes to the source or replica are then fl agged by setting the appropriate bits to 1 in the bitmap. When resynchronization or restore is required, a logical OR operation between the source bitmap and the target bitmap is performed. The bitmap resulting from this operation references all blocks that have been modifi ed in either the source or replica.
This enables an optimized resynchronization or a restore operation because it eliminates the need to copy all the blocks between the source and the replica. The direction of data movement depends on whether a resynchronization or a restore operation is performed.
If resynchronization is required, changes to the replica are overwritten with the corresponding blocks from the source. If a restore is required, changes to the source are overwritten with the corresponding blocks from the replica.
If a restore is required, changes to the source are overwritten with the corresponding blocks from the replica.

Tracking Changes

Comparison of local replication technologies

Local Replication in a Virtualized Environment
Typically, local replication of VMs is performed by the hypervisor at the compute level. However, it can also be performed at the storage level using array-based local replication, similar to the physical environment. In the array-based method, the LUN on which the VMs reside is replicated to another LUN in the same array. VM Snapshot captures the state and data of a running virtual machine at a specifi c point in time. The VM state includes VM files, such as BIOS, network confi guration, and its power state (powered-on, powered-off, or suspended). The VM data includes all the files that make up the VM, including virtual disks and memory. A VM Snapshot uses a separate delta file to record all the changes to the virtual disk since the snapshot session is activated. Snapshots are useful when a VM needs to be reverted to the previous state in the event of logical corruptions. Reverting a VM to a previous state causes all settings confi gured in the guest OS to be reverted to that PIT when that snapshot was created. There are some challenges associated with the VM Snapshot technology. It does not support data replication if a virtual machine accesses the data by using raw disks. Also, using the hypervisor to perform snapshots increases the load on the compute and impacts the compute performance.

Remote Replication

Synchronous remote replication – writes must be committed to the source and remote replica (or target), prior to acknowledging “write complete” to the host. Additional writes on the source cannot occur until each preceding write has been completed and acknowledged. This ensures that data is identical on the source and replica at all times. Further, writes are transmitted to the remote site exactly in the order in which they are received at the source. Therefore, write ordering is maintained. If a source-site failure occurs, synchronous remote replication provides zero or near-zero RPO. However, application response time is increased with synchronous remote replication because writes must be committed on both the source and target before sending the “write complete” acknowledgment to the host. The degree of impact on response time depends primarily on the distance between sites, bandwidth, and quality of service (QOS) of the network connectivity infrastructure.

Synchronous replication

In asynchronous remote replication, a write is committed to the source and immediately acknowledged to the host. In this mode, data is buffered at the source and transmitted to the remote site later. Asynchronous replication eliminates the impact to the application’s response time because the writes are acknowledged immediately to the source host. This enables deployment of asynchronous replication over distances ranging from several hundred to several thousand kilometers between the primary and remote sites.

Asynchronous replication

Below are the bandwith requirement for both:

Bandwidth requirement for synchronous replication

Bandwidth requirement for asynchonous replication

Asynchronous replication implementation can also take advantage of locality of reference (repeated writes to the same location). If the same location is written multiple times in the buffer prior to transmission to the remote site, only the final version of the data is transmitted. This feature conserves link bandwidth.

Remote Replication Technologies

Host-Based Remote Replication

LVM-based remote replication: performed and managed at the volume group level. Writes to the source volumes are transmitted to the remote host by the LVM. The LVM on the remote host receives the writes and commits them to the remote volume group.
LVM-based remote replication supports both synchronous and asynchronous modes of replication. LVM-based remote replication is independent of the storage arrays and therefore supports replication between heterogeneous storage arrays.
The replication process adds overhead on the host CPUs. CPU resources on the source host are shared between replication tasks and applications. Because the remote host is also involved in the replication process, it must be continuously up and available.

LVM based remote replication

Host-Based Log Shipping
Database replication via log shipping is a host-based replication technology supported by most databases. Transactions to the source database are captured in logs, which are periodically transmitted by the source host to the remote host. The remote host receives the logs and applies them to the remote database.
RPO at the remote site is fi nite and depends on the size of the log and the frequency of log switching. Available network bandwidth, latency, rate of updates to the source database, and the frequency of log switching should be considered when determining the optimal size of the log file. Host-based log shipping requires low network bandwidth because it transmits only the log fi les at regular intervals.

Host based log shipping

Storage Array-Based Remote Replication

Synchronous replication mode
To optimize the replication process and to minimize the impact on application response time, the write is placed on cache of the two arrays. The intelligent storage arrays destage these writes to the appropriate disks later.
If the network links fail, replication is suspended; however, production work can continue uninterrupted on the source storage array. The array operating environment keeps track of the writes that are not transmitted to the remote storage array. When the network links are restored, the accumulated data is transmitted to the remote storage array. During the time of network link outage, if there is a failure at the source site, some data will be lost, and the RPO at the target will not be zero.

Array-based remote synchronous replication

Asynchronous replication mode Fig 12-8
Data is buffered at the source and transmitted to the remote site later. The source and the target devices do not contain identical data at all times. The data on the target device is behind that of the source, so the RPO in this case is not zero. Asynchronous replication writes are placed in cache on the two arrays and are later destaged to the appropriate disks. Some implementations of asynchronous remote replication maintain write ordering. A timestamp and sequence number are attached to each write when it is received by the source. Writes are then transmitted to the remote array, where they are committed to the remote replica in the exact order in which they were buffered at the source. This implicitly guarantees consistency of data on the remote replicas.

Array-based asynchronous replication

Disk-buffered replication mode: a combination of local and remote technologies. A consistent PIT local replica of the source device is fi rst created. This is then replicated to a remote replica on the target array.
At the beginning of the cycle, the network links between the two arrays are suspended, and there is no transmission of data. While production application runs on the source device, a consistent PIT local replica of the source device is created. The network links are enabled, and data on the local replica in the source array transmits to its remote replica in the target array.

Disk buffered remote replication

Network-based Remote Replication

CDP remote replication
Fig 12-10

Three site replication

Cascade/Multihop: data fl ows from the source to the intermediate storage array, known as a bunker, in the fi rst hop, and then from a bunker to a storage array at a remote site in the second hop. Replication between the source and the remote sites can be performed in two ways: synchronous + asynchronous or synchronous + disk buffered. Replication between the source and bunker occurs synchronously, but replication between the bunker and the remote site can be achieved either as disk-buffered mode or asynchronous mode.

Three-site remote replication cascade/multihop

Triangle/Multitarget: data at the source storage array is concurrently replicated to two different arrays at two different sites. The source-to-bunker site (target 1) replication is synchronous with a near-zero RPO. The source-to-remote site (target 2) replication is asynchronous with an RPO in the order of minutes. The distance between the source and the remote sites could be thousands of miles. The key benefit of three-site triangle/multitarget replication is the ability to failover to either of the two remote sites in the case of source-site failure, with disaster recovery (asynchronous) protection between the bunker and remote sites.

Three-site replication triangle/multitarget

Data migration solutions
Data mobility refers to moving data between heterogeneous storage arrays for cost, performance, or any other reason. It helps implement a tiered storage strategy.
Data migration refers to moving data from one storage array to other heterogeneous storage arrays for technology refresh, consolidation, or any other reason. The array performing the replication operations is called the control array.
Data migration solutions perform push and pull operations for data movement.
These terms are defined from the perspective of the control array. In the push operation, data is moved from the control array to the remote array.
The control device, therefore, acts like the source, while the remote device is the target.
In the pull operation, data is moved from the remote array to the control array.
The remote device is the source, and the control device is the target.

The push and pull operations can be either hot or cold. These terms apply to the control devices only. In a cold operation the control device is inaccessible to the host during replication. Cold operations guarantee data consistency because both the control and the remote devices are offl ine. In a hot operation the control device is online for host operations. During hot push and pull operations, changes can be made to the control device because the control array can keep track of all changes and thus ensure data integrity.

Remote replication and migration in a virtualized environment

In hypervisor-to-hypervisor VM migration, the entire active state of a VM is moved from one hypervisor to another. This method involves copying the contents of virtual machine memory from the source hypervisor to the target and then transferring the control of the VM’s disk fi les to the target hypervisor. Because the virtual disks of the VMs are not migrated, this technique requires both source and target hypervisor access to the same storage.

Hypervisor-to-hypervisor VM migration

In array-to-array VM migration, virtual disks are moved from the source
array to the remote array. This approach enables the administrator to move VMs across dissimilar storage arrays. Array-to-array migration starts by copying the metadata about the VM from the source array to the target. The metadata essentially consists of configuration, swap, and log files. After the metadata is copied, the VM disk file is replicated to the new location.

Array-to-array VM migration

Networking Basics 1 of 3 – Layer 1 through Layer 3

Sun, 10 Nov 2019 20:18:00 -0400

What layer model works the best? Back in university my textbook was based on OSI 7-layer model. It is rigorously defined and often used in academics. When it comes to day-to-day operation, the 5-layer TCP/IP model is more useful. It combines Application, Presentation and Session layers in OSI model into a single Application layer.

Layer	Name	Protocol	Protocol Data Unit	Addressing	Device	Description
5	Application	HTTP, FTP, etc	Message	N/A	N/A
4	Transport	TCP and UDP	Segment	Port Number	Gateway	Sort out which application on the same host receives incoming data
3	Network	IP	Datagram	IP address	Router and (layer-3) switch	Allows devices across different networks to talk to each other
2	Data Link	Ethernet, WiFi	Frame	MAC address	Bridge and (layer-2) Switch	Defines common way of interpreting signals so devices in the network can communicate
1	Physical	10baseT, 802.11	bit	N/A	Hub	Hardware: cables, signal connector

Physical Layer

Crosstalk – electrical pulse on one wire is accidentally detected on another wire. This was a common challenge when the industry started. The most common cable is UTP (Unshielded Twisted Pair) cable such as Cat 5, Cat 5e cables.

Hub – a physical layer device that allow for connectivity from many computers at once. It is up to each device to determine if incoming data is for them, or to ignore it. Because this slows down transmission, hubs are hardly used any more.

Collision domain – A network segment where only one device can communicate at a time. The device sending signal is occupying the entire media, within its time-sharing window. All devices connected to a hub are in the same collision domain. Ethernet nodes use CSMA/CD to detect collisions and re-transmit when the wire becomes available again.

Wireless Channels – individual, smaller sections of the overall frequency band used by a wireless network. Collision is very common in wireless communication. So the channel selection should minimize collision.

Data Link Layer

In a LAN, NICs talk to each other via MAC address (first 3 octets is organization unique identifier; the last 3 octets are assigned by vendor). Two NICs communicate through twisted pair cable in one of the following modes:

Simplex: data is sent in one directly only;
Half duplex: one line and transmission in each direction takes turns;
Full duplex: two lines, one for each direction, simultaneous;

Network Switch – connects to many devices as well but it determines which device the data is intended for and only send that data to that device. Switch is a layer-2 device. (However today as a network device, many switches have layer-3 capability so it is important to be specific when talking about switch)

Ethernet Frame Format

Ethernet Frame

Ethernet address types:

Unicast address: points to one receiving end; it contains a unique MAC address the frame is intended for;
Multicast address: multicast frame is identified by FF as the first 8-bit, followed by a 4-bit flag field, a 4-bit scope field, and a 112-bit group ID.
Broadcast address: for special destination such as ARP; it contains all Fs in the destination address.

Ethernet frame types include, but not limited to:

Ethernet II frame (most common type in use today used directly by the Internet Protocol)
Novell raw IEEE 802.3 non-standard variation frame
IEEE 802.2 Logical Link Control (LLC) frame
IEEE 802.2 Subnetwork Access Protocol (SNAP) frame

Virtual LAN – any broadcast domain that is partitioned and isolated in a computer network at the data link layer. It is a technique that allows you to have multiple logical LANs operating on the same physical equipment, to segregate traffic.

Network Layer

Router connects between LANs. A router needs at least two NICs. The steps to route are:

Receive data packet
Examines destination IP
Look up IP destination network in routing table
Forward traffic to destination;

Routing can be complex but it is mostly handled by ISPs now. A routing table may have millions of rows (use route command to check). Here is an example of routing tables.

Example of routing table

Autonomous system – a collection of networks that fall under the control of a single network operator (i.e. large corporation)

Routing protocol – specifies how routers communicate with each other, distributing information that enables them to select routes between any two nodes on a computer network. Interior Gateway Protocols are used by routers to share routing information within a single autonomous system. Exterior Gateway protocols are used across autonomous system.

Interior Gateway Protocol (link-state routing): OSPF, IS-IS
Interior Gateway Protocol (distance-vector): RIP, RIPv2, IGRP
Exterior Gateway Protocol: BGP (Border Gateway Protocol) – allows routers (e.g. Internet) to learn from each other about the most optimal paths to forward traffic.

IP datagram

IP datagram (payload at the bottom)

An IP datagram contains a lot more compared to Ethernet frames. One place called type of service field (8-bits) specifies priority. QoS technologies are mostly built on this field, to allow routers to determine which datagram is more important.

IP Fragmentation: an Internet Protocol (IP) process that breaks packets into smaller pieces (fragments), so that the resulting pieces can pass through a link with a smaller maximum transmission unit (MTU) than the original packet size. The fragments are reassembled by the receiving host. If a receiving host receives a fragmented IP packet, it has to reassemble the packet and pass it to the higher protocol layer. Reassembly is intended to happen in the receiving host but in practice it may be done by an intermediate router, for example, network address translation (NAT) may need to reassemble fragments in order to translate data streams.

IP address class

Non-routable IPv4 address spaces belong to no one. Any one can use them in their private network:

192.168.0.0/16
172.16.0.0/12
10.0.0.0/8

Subnetting – splitting large network into smaller ones. Incorrect subnetting setups are a common problem you might run into as an IT support. Each subnet has their ingress routers, subnet ID and subnet mask.

CIDR is a better way to describe subnet because router only need one entry in their routing table to know where to deliver the traffic.

NAT allows communicate between non-routable addresses.

ARP (address resolution protocol) table – maps IP address to MAC address. It is kept on each device (run arp -a to check) and expires after short period of time.

Clean up Git repository

Sat, 26 Oct 2019 20:33:00 -0400

A BitBucket repo has a hard limit of 2GB in size, and soft limit of 1GB. This is not expandable as per Bitbucket and contributors will start receiving warnings once soft limit is reached. We can tell the usage of a repo from the landing page of the repo in BitBucket.

Git is a distributed version control system for source code management, which implies the followings:

It is intended for source code, or configuration code; but not for storing build artifacts, or installers;
Git remembers every single commit, including the ones associated with large files;
Even a contributor deletes a large file (“git rm filename”) after commit, the large file is only removed from the HEAD. The historical commit still stores the file. After all, the whole point of version control is to survive crazy deletion.
distributed means that those large files will be pulled down to contributors laptop (waste everybody’s space although up to 2G:);

With all these implications, shrinking the size of a repo isn’t as straightforward as just removing large files from current commit. We’d have to rewrite the commit history. Here are the steps we should take once repo size grows over the soft limit.

Clean up remote orphaned branches

Removing these branches (remotes/origin/branchname) per se does not free up space. It simplifies the branch structure, leaving /remote/origin/HEAD the only branch left to cleanse for the rest of the steps.

# git push origin --delete branchname

Remove useless files in current commit (HEAD)

In this step we remove useless files in current commit. Again we should not expect much space freed because all file committed previously, even deleted, are still stored. They are just now showing up in the working directory. For this step, we can create a separate local dir on Mac:

mkdir -p /Users/digihunch/repo-cleanup
cd /Users/digihunch/repo-cleanup

Now within the new directory, we create a bare repo and then the full repo:

git clone --mirror https://gh@bitbucket.org/digihunch/source.git
git clone https://gh@bitbucket.org/digihunch/source.git

Then we dive into the full repo and identify the large files:

find . -type f -size +1000k -exec ls -lh {} \; |awk '{print $9":" $5}'

We can run “git rm ” against the files identified as too large or deletable. Then commit and push to remote repo. This removes large files from current commit.

Remove large file and the relevant commits in the history

As previously mentioned, we have to re-write the history so history forget about the large files. After this step, the historical commits that large files are associated with will all be deleted. Compare the two charts below to understand what the effect is:

We can use the bare repo created in the last step, with “git filter-branch” tool to cleanse the branch tree. Some advocate as a faster third party tool BFG Repo-Cleaner as a faster, third-party alternative but I usually lean towards native tool. This article explains the command switches.

git filter-branch -f --tree-filter "rm -rf \large_file.zip" --prune-empty -- --all

After this steps the repo should be cleansed. According to this guide from BitBucket, we still need to contact their support to run a garbage collection for us in order to see the size change. It even takes time for the size to be reflected after garbage collection. This reference also does great job explaining what we need to do.

Other contributors re-sync history

It is important to understand that the step above modifies history. Although the commit hash did not change, they are assigned with different commit-ids and you can tell from the commit history where it displays former commit id.

This activity only affects remote repository. Each contributor’s local repository still stores the old commits and should be sync’ed with the remote origin by deleting the entire repo and run “git clone” again. Although not welcomed by every individual contributors, but it is a necessary evil and better approached with explicit instruction.

Because this activity takes higher risks, changes each commit, involves vendor support and requires activities by each contributor, the support team should focus on preventing this from happening instead of fixing it.

Configure pre-commit hook as a preventive measure

As we have more Ansible tasks related, working directory becomes complicated and sometimes contributors accidentally committed large unwanted files (and pushed into the remote repo). Down the road, the best practice is to prevent contributors from committing junks.

The best spot to detect this should be a pre-receive hook on the server side, which is only available with self-hosted Bitbucket Server. Unfortunately, this is not a viable option for Bitbucket cloud. Our best bet is client-side pre-commit hook, in which a script performs size check when contributors run “git commit”. The purpose is to fail the commit if total file size is over the limit (20M), and the hook itself should be version controlled as well. Compared to (server side) pre-receive hook, the drawback of (client side) pre-commit hook is it requires initial client configuration. The upside is it captures large files before commit.

This hook can be a shell script as simple as this:

commitsizelimit=20
stagedfilelist=$(git diff --name-only --cached)
stagedfilecnt=`echo "$stagedfilelist"| sed '/^\s*$/d' |wc -l`
if [[ $stagedfilecnt -gt 0 ]]; then
 totalcommitsize=$(du -cm $stagedfilelist | tail -1 | cut -f 1)
 # Redirect output to stderr.
 exec 1>&2

 if [[ $totalcommitsize > $commitsizelimit ]]; then
 echo "Warning: Total size of all files in staging area is "$totalcommitsize"MB, exceeding the limit of "$commitsizelimit"MB."
 echo " To list files by size, run 'du -ch \$(git diff --name-only --cached)'"
 echo " To drop large ones from staging area with 'git rm -f filename'"
 echo " To bypass this limit, use 'git commit --no-verify'"
 exit 1
 fi
fi

In the repo we will have a .githook directory to store hooks (e.g. ~/source/.githooks/pre-commit) and point to the hooks directory using the following command:

git config core.hooksPath .githooks

Storage Nitty-Gritty 4 of 5 – Backup and Archive Solutions

Mon, 14 Oct 2019 19:42:00 -0400

Business Continuity

Information Availability IA = MTBF/(MTBF+MTTR), where
* MTBF (Mean Time Between Failure) – average time available for a system or component to perform its normal operations between failures.
* MTTR (Mean Time to Repair) – the average time required to repair a failed component.

Disaster Recovery – the coordinated process of restoring systems, data, and the infrastructure required to support ongoing business operations after a disaster occurs. It is the process of restoring a previous copy of the data and applying logs or other necessary processes to that copy to bring it to a known point of consistency.

Recovery-Point Objective (RPO) – the point in time to which systems
and data must be recovered after an outage. It defi nes the amount
of data loss that a business can endure. A large RPO signifi es high tolerance
to information loss in a business.

Recovery-Time Ojbective (RTO) – The time within which systems and applications must be recovered after an outage. It defi nes the amount of downtime that a business can endure and survive. Businesses can optimize disaster recovery plans after defi ning the RTO for a given system.

Strategies to meet RTO and RPO

Data Vault: a repository at a remote site where data can be periodically or continuously copied so a copy is always available in that site.
Hot site: A backup site running all the time.
Cold site: A backup site with minimum infrastructure, to be activated for operation in the event of disaster.
Server Clustering: a group of servers and relevant resources coupcled to operate as a single syste. Clusters can ensure high availability and load balancing.

Single Point of Failure – failure of a component that can terminate the availability of the entire system or IT service. To mitigate single point of failure, systems are designed with redundancy. This includes:

redundant HBA on server
NIC teaming
redundant switch
multiple storage array ports
RAID and hot spare configuration
Redundant storage array
server clustering (e.g. clustered servers exchange heartbeat to inform each other about their health. If one of the servers fails, other server can take up the workload.
VM Fault Tolerance
Multipathing software: If one path fails, I/O does not reroute unless the system recognizes that it has an alternative path. Multipathing software provides the functionality to recognize and utilize alternative I/O paths to data. Multipathing software also managees the load balancing by distributing I/Os to all available, active paths.

Backup

Backup is an additional copy of production data created and retained for the sole purpose of recovering lost or corrupted data. Backup are typically performed for the following purposes:

Disaster recovery. e.g. the backup copies are used for restoring data at an alternate site, when the primary site is incapacitated due to disaster.
Operational recovery. e.g. accidental deletion, file corruption
Archival. e.g. data is not changed or accessed any more.

Common considerations for backup includes: time interval between two backups (to meet RPO), retention period, media type (to meet RTO), granularity, compression and deduplication

Backup Granularity

Full backup: backup of the complete data on the production volumes.
Incremental backup: copies the data that has changed since the last full or incremental backup, whichever occurred more recently.
Cumulative backup: copies the data that has changed since the last full backup.

Backup Methods

hot backup/online backup: backup is completed while application is up and running;
cold backup/offline backup: backup is completed while the application is shutdown for the backup window.

The hot backup of online production data is challenging because data is actively used and changed. If a file is open, it is normally not backed up during the backup process. In such situations, an open file agent is required to back up the open file. These agents interact directly with the operating system or application and enable the creation of consistent copies of open files. In database environments, To ensure a consistent database backup, all files need to be backed up in the same state. That does not necessarily mean that all files need to be backed up at the same time, but they all must be synchronized so that the database can be restored with consistency. The disadvantage associated with a hot backup is that the agents usually affect the overall application performance. If this is not acceptable, PIT (point-in-time) copy method can be utilized to create a PIT copy from the production volume and use it as the source for the backup. PIT copy method can reduce impact on production volume.

Typical Backup Architecture

Typical Backup steps

Typical Restore steps

Backup topologies
Direct-attached backup: the storage node is configured on a backup client, and the backup device is attached directly to the client;
LAN-based backup: the clients, backup server, storage node, and backup device are connected to the LAN;
SAN-based backup (LAN-free): The SAN-based backup topology is the most appropriate solution when a backup device needs to be shared among clients;
Mixed topology: mix of LAN-based and SAN-based topologies;

NDMP protocol is for backup in NAS environment

Backup media

Tape: for long-term offsite storage due to low cost. data access is sequential which implies slowness for both backup and restore. Tapes are susceptible to wear and tear.
Disk: fast backup and retrieve to improve RPT and RTO. No offsite capability.
Virtual Tape: virtual taps are disk drives emulated and presented as tapes to the backup software. VTL (virtual tape library) has the same components as that of a physical tape library.
Fig 10-18

Data deduplication – identify and eliminate redundant data to reduce backup window and size. Common data deduplication methods:

file-level deduplication (aka. single-instance storage) detects and removes redundant copies of identical files. It enables storing only one copy of the file; the subsequent copies are replaced with a pointer that points to the original file.
subfile deduplication breaks file into smaller chunks and then uses a specialized althorithm to detect redundant data within and across the file. This eliminates duplicate data across files. This has two forms:
* fixed-length block deduplication – divides the files into fi xed length blocks and uses a hash algorithm to fi nd the duplicate data.
* variable-length segment deduplication – if there is a change in the segment, the boundary for only that segment is adjusted, leaving the remaining segments unchanged.

Data deduplication implementation

source-based data deduplication – eliminates redundant data at the source before it
transmits to the backup device. This requires less bandwidth and shortens backup window. It increases the overhead on the backup client and could impact the performance of the backup and application running on the client.
target-based data deduplication – deduplication occurs at the backup device, which offloads the backup client from the deduplication process. This takes two forms:
inline deduplication – performs deduplication on the backup data before it is stored on the backup device. this reduces storage need, but introduces time overhead to identify and remove duplication. best for large backup window
post-process deduplication – enables backup data to be stored on backup device first, and then deduplicate later. This is suitable for tighter backup windows, but requires more storage.

In virtualized environments, backup agent can be installed on the hypervisor, where the VMs appear as a set of files to the agent. VM files can be backed up by performing a file system backup from a hypervisor. For example, Image-based backup operates at hypervisor level and essentially takes a snapshot of the VM. It creates a copy of the guest OS and all the data associated with it (snapshot of VM disk files), including the VM state and application configurations. The backup is saved as a single file (an image) and mounted on a separate server as proxy, which acts as backup client.

Image Based Backup

Data archive

Archive – a repository where fixed content is stored. Fixed content can be data that were changed but will not be changed anymore.
Online archive: A storage device directly connected to a host that makes
the data immediately accessible.
Nearline archive: A storage device connected to a host, but the device where the data is stored must be mounted or loaded to access the data.
Offline archive: A storage device not ready to use. Manual intervention is required to connect, mount or load the storage device before data can be accessed.

An archiving agent is software installed on application server. The agent is responsible for identify data that can be archvied based on policy. After the data is identified for archiving, the agent sends the data to the archiving server. Then the original data on the application server is replaced with a stub file, which contains the address of the archived data.

Archiving Solution Architecture

An archiving server is software installed on a host that enables administrators to configure the policies for archiving data. An archiving storage device stores fixed content.

Cryptography basics 2 of 2

Sun, 08 Sep 2019 21:21:41 -0400

My previous post outlines several core concepts around cryptography, such as asymmetric key encryption, digital certificate, the encoding formats and relevant file extensions. In this article, we continue to explore cryptography use cases, where these concepts are connected and put into application.

The most important use case is TLS handshake. I cannot stress enough how paramount this scenario is. This use case and its variation can be found in almost every situation where connection needs to be secured.

TLS handshake: the process in which client and server establish secure connection. During the handshake, two parties agree on TLS version, decide on cipher suite, authenticate the identity of each other (although client identity authentication is less common), and generate session key for symmetric encryption after the handshake. Details steps are very important. Here is a fairly thorough reference, and here is a great diagram:

TLS handshake

Secure browser connection: The most widespread use of TLS handshake is to secure browser connection with HTTPS:

Browser initiates connection to the server;
Server sends browser its certificate (public key + digital signature signed by CA);
Browser has preloaded public key of CA and uses it to decrypt digital signature and get the digest of public key;
Browser calculate digest of received public key and compares it against the digest from the previous step;
If the result is the same, public key is trusted, a green lock is displayed; otherwise, a warning is displayed;
proceed to the rest of the steps in TLS handshake.

Secure connection in Java: Java applications manages keys and certificates through two classes: java.security.KeyStore and java.security.TrustStore. Suppose a Java application client initiates TLS connection to server. The server application will present its certificates from server’s Key Store. The client will use certificates stored in client’s Trust Store to verify the identity of the server. Once validated, the client then presents certificates stored in client’s Key Store back to the server for validation.

A KeyStore keeps keys and certificates for your own application. Typically, you store a KeyPair in a KeyStore file.
A TrustStore keeps the certificates of external systems that your application trusts.

Below is a great diagram:

Secure Java application in TLS

JKS file is Java’s version of PKCS#12 (private key + certificate, password protected). Entries in a JKS file must have an “alias” that is unique. The JKS file type can be used for both Key Store and Trust Store. When it’s used in Key Store, it contains a certificate and private key for the Java application. When it’s used in Trust Store, it only contains certificate from external trusted applications. Note that JKS is the default keystore format until Java 8. Since Java 9 the default keystore format is PKCS12.

Finally, we can introduce some tools:

Command line Tools

openssl is a versatile tool for cryptography and keytool is a similar tool for Java applications.

keytool is for store keys/certificates in Java Key Store or Trust Store.

Conclusion

I do not include any openssl or keytool command in this article in order to remain theoretical. However, once through the conceptual hurdles, one should become fairly comfortable picking up the tools and understand why each command is needed to achieve its purpose.

AWS Certified DevOps Engineer Exam Tips

Sat, 17 Aug 2019 23:07:59 -0400

The last 30 days have been exhausting for me. I studied hard on the new AWS Certified DevOps Engineer exam and thank goodness I passed (750 out of 1000 is required). This was the hardest professional certification I ever worked on. The exam was re-launched recently in March 2019 so there is still a shortage of information around the community. I was hoping to share my experience to help demystify this new exam.

I had a somewhat solid background to begin with, having taken the AWS Certified Solution Architect Professional exam (before the 2019 update), and worked quite a bit on CloudFormation, automation and Git. However, I still did not anticipate the exam to be this difficult until I was halfway through and already had the exam and materials paid for.

Without clear guideline on study material other than the white papers, I first skimmed through the ACloudGuru course, which helped me form a high level sense of exam coverage. Nonetheless it does not cover any topic in-depth and therefore by no means makes an essential part of my study. I checked out LinuxAcademy course and they are much more in-depth for the major topics indeed. I like the course material in Lucid chart. However, the LinuxAcademy course along does not cover everything you need to know.

What I found extremely helpful is the free training videos from the official training website. I strongly recommend the 7-hour course Exam Readiness: AWS Certified DevOps Engineer – Professional. The instructor did a great job outlining the services and knowledge areas in the assessment . The course also comes with quality sample questions with answers and explanations on what the thinkings are behind the correct answers or why some choices are obviously wrong. I went through these questions twice and feel much better at not only understanding the question, but also understanding the intent of the question.

Apart from the Exam Readiness course, other free introductory courses from the official training website are helpful as well especially for those services that you only need to know the basics. Most of those courses are 5 ~ 10 minutes long, with brief but sufficient introduction and a demo session. The other extremely helpful resource is the official practice questions. The practice exam is harder than the actual exam but they closely resemble the actual question style in the exam. Unfortunately, no answer is provided but they made me spend time finding answers across the documentations and blogs. It is worth-noting that the AWS blogs provides plenty of use cases that are covered in the questions.

When it comes to the real exam, it covers a lot more topics than its predecessor. Many questions are long and confusing. And I wish I could run a diff command to highlight the differences between choices. During exam preparation you really need to train yourself on reading efficiently. I found myself sometimes eyeball through all four choices at the same time, which get my mind scattered.

During my study, I divide all services into three categories based on the level of familiarity, and here is my list:

Category 1. Know these services very well, in and out:

AWS ElasticBeanstalk, OpsWorks Stacks, OpsWorks Chef Automate, CloudFormation, CloudWatch, CodeBuild, CodeCommit, CodeDeploy, CodePipeline, CodeStar, Lambda, API Gateway, Config, Trusted Advisor, CloudTrail, Systems Manager, Autoscaling Group in EC2, DynamoDB

Category 2. Know these services well, but not necessarily down to every single detail:

Amazon Kinesis Firehose, Kinesis Analytics, Kinesis Streams, Step Functions, Elastic Load Balancer, Secrets Manager, Serverless Application Model (SAM), Route53, RDS, Certificate Manager, ElasticSearch, ECS, ECR

Category 3. Know about these services at a high level, but do not skip any:

AWS Organization, X-Ray, GuardDuty, Macie, Inspector, Service Catalog, KMS, Batch, Athena, Single-Sign-On, Data LifeCycle Manager, CloudSearch, Health Dashboard, Glue, QuickSight, LightSail

Although I did not mention much about the white papers, I want to highlight their importance again. I would not attempt the exam without reading and understanding the required white papers.

If you aspire to take the AWS certified DevOps Engineer exam I hope this helps you a little bit. Good luck.

Storage Nitty-Gritty 3 of 5 – NAS and Object Storage

Sat, 13 Jul 2019 23:31:00 -0400

NAS (network attached storage)

NAS server is dedicated to file-serving. NAS device runs its own specialized operating system that is optimized for file I/O, integrated hardware and software component that meets specific file-service needs, and performs file I/O better than a general-purpose server. NAS device can serve more clients than general-purpose servers and provide the benefit of server consolidation (centralized storage).

NAS uses network and file-sharing protocols to provide access to the file data. These protocols include TCP/IP for data transfer, and Common Internet File System (CIFS) and Network File System (NFS) for network file service.

Network File Sharing – user who creates a file determines the type of access to be given to other user. When multiple users try to access a shared file at the same time, a locking scheme is required to maintain data integrity and, at the same time, make this sharing possible. Examples of file sharing method (FTP, DFS, NFS, CIFS, P2P)

Components of NAS – NAS head (CPU, memory, NIC, optimized OS, ports, applications that supports CIFS/NFS) and Storage Array

Typical NAS components

NAS I/O operation:

Client packages an I/O request into TCP/IP and forwards it through network stack. NAS head receives this request from network;
NAS head converts the I/O request into an appropriate physical storage request, which is a block-level I/O, and then performs the operation on the physical storage;
When NAS head receives data from the storage array, it processes and repackages the data into an appropriate NFS/CIFS response;
NAS head packages this response into TCP/IP again and forwards it to the client through the network

NAS I/O operation

NAS implementation

Unified NAS – consolidate NAS-based and SAN-based data access within a unified storage platform and provides a unified management interface for managing both the environments.

Unified NAS connectivity

Gateway implementation – similar to unified NAS, the storage is shared with other applications that use block-level I/O. The gateway NAS is more scalable compared to unified NAS because NAS heads and storage arrays can be independently scaled up when required. For example, NAS heads can be added to scale up the NAS device performance.

When the storage limit is reached, it can scale up, adding capacity on the SAN, independent of NAS heads. Similar to a unified NAS, a gateway NAS also enables high utilization of storage capacity by sharing it with the SAN environment.

Gateway NAS connectivity

Scale-out NAS – enables grouping multiple nodes together to construct a clustered NAS system. A scaled-out NAS provides the capability to scale its resources by simply adding nodes to a clustered NAS architecture. The cluster works as a single NAS device and is managed centrally. Scaled-out NAS creates a single file system that runs on all nodes in the cluster. All information is shared among nodes, so the entire file system is accessible by clients connecting to any node in the cluster. Scale-out NAS stripes data across all nodes in a cluster along with mirror or parity protection. As data is sent from clients to the cluster, the data is divided and allocated to different nodes in parallel. When a client sends a request to read a file, the scale-out NAS retrieves the appropriate blocks from multiple nodes, recombines the blocks into a file, and presents the file to the client. As nodes are added, the file system grows dynamically and data is evenly distributed to every node. Each node added to the cluster increases the aggregate storage, memory, CPU, and network capacity. Hence, cluster performance also increases.

Scale-out NAS use separate internal and external networks for back-end and front-end connectivity, respectively. The internal network offers high throughput and low-latency and uses high-speed networking technology, such as InfiniBand or Gigabit Ethernet.

Scale-out NAS with dual internal and single external networks

NFS protocol – originally based on UDP, uses RPC as a method of inter-process communication between two computers. NFS provides a set of RPCS to access remote file system for the following operations:

Searching files and directories
Opening, reading, writing to and closing a file
Changing file attributes
Modifying file links and directories

NFSv3 and earlier is stateless protocol. Each call provides a full set of arguments to access files on the server. NFSv3 is most commonly used version, based on UDP or TCP.

NFSv4 uses TCP and is based on stateful protocol design.

CIFS – a public, or open variation of SMB protocol. Filenames in CIFS are encoded using unicode characters. It is stateful protocol because the server maintain connection information regarding every connected client. If a network failure or CIFS server failure occurs, the client receives a disconnection notification. If application has embedded intelligence to restore the connection, then the storage solution is fault tolerant. If the embedded intelligence is missing, the user must take steps to reestablish the CIFS connection.

NAS Performance – network congestion is one of the most significant sources of latency in NAS environment. Other factors

number of hops
authentication with AD
Retransmission – speed and duplex settings on the network devices and NAS heads must match
Over-utilized routers and switches
File system lookup and metadata request – deep directory structure could cause delay.
Over-utilized NAS devices – client accessing multiple files can cause high utilization levels on a NAS device
Over-utilized clients – if a client is busy itself, it requires a longer time to process the request and responses.

NAS latency

NFS server manages privilege and does not require username and password from the client at the time of mounting. CIFS share does require username and password.

Common network optimization practices for network contestion:

A VLAN is a logical segment of a switched network or logical grouping of end devices connected to different physical networks. The segmentation or grouping can be done based on business functions, project teams, or applications. VLAN is a Layer 2 (data link layer) construct and works similar to a physical LAN. A network switch can be logically divided among multiple VLANs, enabling better utilization of the switch and reducing overall cost of deploying a network infrastructure.

The broadcast traffic on one VLAN is not transmitted outside that VLAN, which substantially reduces the broadcast overhead, makes bandwidth available for applications, and reduces the network’s vulnerability to broadcast storms.

MTU setting determines the size of the largest packet that can be transmitted without data fragmentation. Path maximum transmission unit discovery is the process of discovering the maximum size of a packet that can be sent across a network without fragmentation. The default MTU setting for an Ethernet interface card is 1,500 bytes. A feature called jumbo frames sends, receives or transports Ethernet frames with an MTU of more than 1,500 bytes. The most common deployments of jumbo frames have an MTU of 9,000 bytes. However, not all vendors use the same MTU size for jumbo frames. Servers send and receive larger frames more efficiently than smaller ones in heavy network traffic conditions. Jumbo frames ensure increased efficiency because it takes fewer, larger frames to transfer the same amount of data. Larger packets also reduce the amount of raw network bandwidth being consumed for the same amount of payload. Larger frames also help to smooth sudden I/O burst.

The TCP window size is the maximum amount of data that can be sent at any time for a connection. For example, if a pair of hosts is talking over a TCP connection that has a TCP windows size of 64KB, the sender can send only 64KB of data and must then wait for an acknowledgement from the receiver. If the receiver acknowledges that all the data has been received, then the sender is free to send another 64 KB of data. If the sender receives an acknowledgment from the receiver that only the first 32 KB of data has been received, which can happen only if another 32 KB of data is in transit or was lost, the sender can send only another 32 KB of data because the transmission cannot have more than 64 KB of unacknowledged data outstanding.

In theory, the TCP window size should be set to the product of the available bandwidth of the network and the round-trip time of data sent over the network. For example, if a network has a bandwidth of 100 Mbps and the round-trip time is 5 milliseconds, the TCP window should be as follows:

100 Mb/s x .005 seconds = 524,288 bits or 65,536 bytes

The size of the TCP window fi eld that controls the fl ow of data is between 2 bytes and 65,535 bytes

Link aggregation is the process of combining two or more network interfaces into a logical network interface, enabling higher throughput, load sharing or load balancing, transparent path failover, and scalability. Due to link aggregation, multiple active Ethernet connections to the same switch appear as one link. If a connection or a port in the aggregation is lost, then all the network traffic on that link is redistributed across the remaining active connections.

File-level virtualization

File-level virtualization eliminates the dependencies between the data accessed at the file level and the location where the files are physically stored. Implementation of file-level virtualization is common in NAS or file-server environments. It provides non-disruptive file mobility to optimize storage utilization.

It provides user or application independence from the location where the files are stored. File-level virtualization creates a logical pool of storage, enabling users to use a logical path, rather than a physical path, to access files. While the files are being moved, clients can access their files non-disruptively. Clients can also read their files from the old location and write them back to the new location without realizing that the physical location has changed. A global namespace is used to map the logical path of a file to the physical path names.

File-serving environment before and after file-level virtualization

Object-based storage

In NAS, metadata are stored as part of the file distributed throughout the environment, which adds to the complexity and latency in searching and retrieving files. Object-based storage, on the other hand, stores file data in the form of objects based on its content and other attributes, rather than the name and location.

Hierarchical File System and Flat Address Space

OSD – object-based storage devices, stores data in the form of objects using flat address space. There is no hierarchy of directories and file. Object is identified by objectID, which is usually generated using hash function.

In block storage, when file system receives the IO from an application, the file system maps the incoming I/O to the disk blocks. The block interface is used for sending the I/O over the channel or network to the storage device. The I/O is then written to the block allocated on the disk drive. When an application accesses data stored in OSD, the request is sent to the file system user component. The file system user component communicates to the OSD interface, which in turn sends the request to the storage device. The storage device has the OSD storage component responsible for managing the access to the object on a storage device.

Benefit of object storage

security and reliability: OSD can use special algorithm for strong encryption capacity. Request authentication is performed at the storage device rather than with an external authentication mechanism
platform independence: standard web access via REST or SOAP
scalability: Both storage and OSD nodes can be scaled independently in terms of performance and capacity

Block-level access vs object-level access

OSD components:

nodes: a server with OSD operating environment to provide services to store, retrieve and manage data. Two key services are metadata service (generating objectID and maintaining the mapping between objectID and file) and storage service (manage a set of disks where data are stored).
private network: provides node-to-node connectivity and node-to-storage connectivity.
storage device

OSD system components

Storage mechanism

The application server presents the file to be stored to the OSD node.
The OSD node divides the file into two parts: user data and metadata.
The OSD node generates the object ID using a specialized algorithm. The algorithm is executed against the contents of the user data to derive an ID unique to this data.
For future access, the OSD node stores the metadata and object ID using the metadata service.
The OSD node stores the user data (objects) in the storage device using the storage service.
An acknowledgment is sent to the application server stating that the object is stored.

OSD: object storage

Retrieval mechanism

The application server sends a read request to the OSD system.
The metadata service retrieves the object ID for the requested file.
The metadata service sends the object ID to the application server.
The application server sends the object ID to the OSD storage service for object retrieval.
The OSD storage service retrieves the object from the storage device.
The OSD storage service sends the file to the application server.

OSD object retrieval

OSD usage: data archival, especially long-term; and cloud storage, storage as service

CAS – content addressed storage, a special type of OSD designed for secure online storage and retrieval of fixed content. Data access in CAS differs from other OSD devices. In CAS, the application server access the CAS device only via the CAS API running on the application server. However, the way CAS stores data is similar to the other OSD systems.

CAS Use case

Healthcare: storing patient studies – size of radiology study ranges from 15MB to more than 1GB. Newly acquired studies are retained for 60 days and moved to long term storage.
Finance: storing financial records – bank stores images of cheques (~25KB each) for about 90 millions a month. Images are processed in transaction system for 5 days. For the next 60 days images are requested for verifications. After 60 days access requirements drop drastically. Retention policy manages life-cycle of the images.

Unified storage

Components

storage controller: The storage controller provides block-level access to application servers through iSCSI, FC, or FCoE protocols.
NAS head: a dedicated file server that provides file access to NAS clients
OSD node: accesses the storage through the storage controller using a FC or FCoE connection.
Storage

Unified storage platform

Cryptography Basics 1 of 2

Wed, 10 Jul 2019 20:07:18 -0400

I have been dabbling with OpenSSL commands to achieve what I needed during IT implementation, but I decided to spent some time to overcome the conceptual hurdles around cryptography. In this domain, following other people’s instructions through the project does not produce much learning value when too many concepts cloud around. Let’s take the bull by the horn.

This article is purely conceptual. There are already lots of step-by-step guideline about acquiring a website certificate. The intention is to elucidate the core concepts on IT cryptography, and then connect the dots to form the big picture in cryptography.

First, let’s distinguish three basic concepts: encoding, hashing and encryption. They are in essence all mathematical functions, but one does not need to understand the underlying algorithm in order to understand what they are.

Encoding – Transform data into a format so it is readable by external system. Encoding is about interoperability. It is not about security whatsoever. Example: ASCII, BASE64, UNICODE.

Hashing – Mathematic algorithms to generate digest of content. A digest is usually fixed-size, non-reversible and deterministic. You cannot restore content from its digest (non-reversible, one-way calculation). Two different contents results guarantees different digests (deterministic). Digest as a result of hashing is mostly about data integrity. For example, in file download you can calculate MD5 hash and compare the result against the digest given by the source. Another common use, is to hash all password in database. In that sense, hashing has to do with security. Popular algorithms are MD5 and SHA-256.

Encryption – Mathematic algorithms that only succeeds if correct parameter (key) is provided. The function is deterministic and reversible (two-way). The parameter (key) used for calculation (encryption) and reverse calculation (decryption) can either be the same or different.

Cryptography involves all three concepts, but it mainly addresses issues around encryption. We break down encryption into two categories, symmetric key encryption (aka private key encryption) and asymmetric key encryption (aka public key encryption).

Symmetric Key Encryption: using the same key (shared secret) for encryption and decryption. Both parties need to keep it secret. Popular algorithm is AES.

Asymmetric Key Encryption: involves a pair of public key and private key. Public key is given out to external systems. Private key is kept secret. Popular algorithm is RSA. You can run some experiments on this page, and I’ve made some additional notes (all based on RSA algorithm):

Public Key and Private Key are NOT interchangeable. The size of an RSA private key is usually much larger than its public key;
You can encrypt with either key, and decrypt with the other, so long as you specify the key type at the time of encryption or decryption;
You can generate public key from private key (ssh-keygen -y); but not the other way round;
Using a wrong key to encrypt or decrypt leads to failure, instead of wrong result.

Obviously symmetric key encryption is the original form of encryption and requires both parties to keep the key secret, permanently. This is not realistic in real life between organizations. This challenge leads to the adoption of asymmetric key where the public key can be published, for external party to encrypt outgoing messages, whereas the private key is kept secret within the owner, only to decrypt incoming messages.

Knowing this distinction, we can introduce two concepts that are built on top of asymmetric key encryption.

Digital Signature: if an entity signs a document digitally. The digital signature is the digest of the document (hash of the content) encrypted with the signer’s private key.

Digital Certificate: contains owner’s public key and digital signature of issuer (digest of owner’s public key encrypted by issuer’s private key). Client (e.g. browser with CA’s public key preloaded) should not trust the owner’s public key until it compares its digest against decryption result of digital signature (using CA’s public key).

The key difference here is that a digital certificate involves third party. Digital signature by itself cannot address impersonation, which is addressed by digital certificate issued by third party (certified authority). This requires digital certificate must follow some standard, and the most prevalent one is:

X.509: a standard format for digital certificates. It contains:

a public key
an identity (a hostname, or an organization, or an individual)
a signature (either signed by CA or self-signed)

Since the advent of certificate, there are tons of global organizations that need to manage keys and certificates. When we create a secure connection now, we only need to get a certificate from an intermediate CA, thanks to existing certificate chain. It involves lots of work for large organizations to maintain keys and certificates, which requires:

Public Key Infrastructure (PKI): the IT infrastructure to create, manage, distribute, use, store and revoke digital certificates and public keys.

One of the important technique to manage keys and certificates is the encoding. Keys and certificates are usually wrapped with different encoding formats based on what need to be done on them. Here is a summary of encoding formats

Format	Encoding	what is stored	Possible Suffix
PEM	DER file (binary content) encoded in Base64. Certificate files typically include clear text statement “BEGIN CERTIFICATE” and “END CERTIFICATE”	single certificate, certificate chains or private keys	.pem, .crt, .cer, .key
DER	Binary format in early days; it’s complex and don’t use unless with a specific purpose.	single certificate, certificate chains, or private keys	.der, .cer
PKCS#7	Base64 encoded ASCII file, typically include clear text statement “BEGIN PKCS7” and “END PKCS7”	only certificates or certificate chains; no private keys	.p7b .p7s
PKCS#8	Similar to PEM as base64 encoded format but for storing private key only, can be password protected	Private key	.key
PKCS#12	a binary format, heavily used by Microsoft products	certificate, certificate chains or private keys; public private key pair	.pfx .p12
OpenSSH	used by OpenSSH to store public keys (as specified in RFC4253)		.pub

Note: above are just encoding formats. Just by file extension you cannot tell whether the file is a key or a X.509 certificate. When you are configuring certificates, you may come across the following file extensions as well:

.key this extension can indicate any kind of key, but usually it is a private key (used along with .crt file)

.csr certificate signing request, including a public key and an identity required by CA. CA needs this file to issue a certificate. CSR could be encoded in Base-64 or DER

.cer or .crt a certificate, usually in X.509 v3 (public key + identity + signature), the encoding could be PEM or DER.

.jks -> java key store file type. It can be either a key store (private key along with certificate) or trust store (certificate) for Java application. Refer to the section for Java applications.

In the next article, we will examine some use case involving the concepts introduced in this post.

Git Explained 2 of 2

Tue, 18 Jun 2019 17:06:00 -0400

This is a continuation from Git Explained 1 of 2 where the fundamental concepts are covered. In this article we introduce some tools for customization and maintenance.

As for Git configuration, there are two files to dictate your Git configuration. ~/.gitconfig and .git/config in project directory. Running `git config –list –show-origin` shows all config entries and where they are from. For example you can custom your Git hooks location. Neither of the two files are being version controlled, so the configuration is only effective in the client environment,

Server side Git hooks

Git implementation supports server side hooks (pre-receive, update, post-receive). They are bash scripts placed in .git/hooks with specific names, fired upon event occurrence. Exit code of 1 from the scripts fails the event. Since server side Git hook consumes server resources, many repository hosting vendors (e.g. BitBucket Cloud) do not support it. You will need to enable it in self-hosted servers (e.g. BitBucketServer).

Client side Git hooks

Since server side hooks are not widely supported in every vendor, client-side Git hooks is good alternative places to implement functions such as code style check, commit size check, etc

The default directory for hooks is .git/hooks/ under the project directory is not version controlled and not easy to share with the team. If the hook needs to be shared among project contributors, we can place hooks files in .githooks/ under project directory. This will make the hook files version controlled. In addition, we need to point the hooks to this directory in configuration, by running `git config core.hooksPath .githooks` from project directory.

Web hooks

Web hooks can be thought of as an event notification mechanism. It is a common feature provided by VCS repository hosting providers. If a certain type of event occurs to the repo, web hook will fire an RESTful API call. The HTTP Endpoint, authentication secret and event payload are pre-configured in the repo settings. Web hooks are commonly supported by Git-based repo implementation, such as BitBucket cloud, GitHub, GitLab or AWS CodeCommit. It can also be enabled in self hosted Git repo. Web hook is a powerful tool to drive downstream event, such as Jenkins to start building the code. The major difference between server side hook and web hook is web hook is RESTful API driven, whereas server side hook is executing a script.

Pipelines

Some repo hosting vendor also provide a feature named pipeline. BitBucket has Pipeline as a CI/CD tool, AWS has AWS Code Pipeline, and GitLab offers CI/CD pipeline as well. These pipelines are usually in the form of a YAML file in the repo with a special name. The YAML spell out the steps to perform along the pipeline.

Squash Commits

For small projects I was in the habit of committing to main branch. I often need to squash a number of commits into one to “clean up”. Usually a Pull Request (e.g. in GitHub) or Merge Request (e.g. in GitLab) have such option during approval. We can squash a few commits with git command as well (suppose we want to squash the most recent 16 commits):

git rebase -i HEAD~16
git push origin +main

The command will open text editor to allow you to mark what to do with each commit. You can mark all except one commit as squash. For the commit to keep, mark it as pick. Then save the text editor. Git rebase will perform the squash for you. However, since this is a rebase, do not do this if there are other collaborators working on the same branch.

Cleanse a repository

Take BitBucket cloud for example, the size of a remote repo has a non-expandable hard limit of 2GB, and a soft limit of 1GB. Once the soft limit is reached, a warning will be displayed on Bitbucket’s landing page as well as when contributors pushes changes. Once the hard limit is reached, the entire repo will turn read-only mode.

Space consumption can be caused by accidental committing of large file. As covered in the previous post, files are stored as blob objects in .git directory. If a file was deleted by `git rm` command, it simply means it is de-referenced from the next commit and on. After all, Git as a distributed version control system, has the ability to magically restore the deleted file when we want. The cost of that magic, is that deleted file is permanently stored in the repo, in the form of blob object, although not present in the working directory. It consumes space not only in remote repo, but also in the local repo of each contributor.

The purpose of Git repository is to store source code which are fairly small texts. However if a contributor pushed in large files, it can be tricky to cleanse a Git repo on the remote side. Here is some guidelines:

Before cleansing, identify large unwanted files in current working directory, delete them with `git rm` and then commit this change in master branch.
Consolidate branches (e.g. delete useless remote branches with `git push origin –delete branch_name`). This step itself does not free up space in remote repo but it simplifies the branches;
Remove large blob objects and commit objects that reference them. This step essentially is re-writing the commit history of repo. Given the risk, it is recommended to perform this step from a separate local project directory with bare repo only without working directory. This article presents some good advices and illustrations. The command recommended is `git filter-branch -f –tree-filter “rm -rf \large_file.zip” –prune-empty — –all` ; and if that is too slow here is an alternative named BFG Repo-Cleaner.
After this step, each commit will have a new hash id. For the repo size to reduce, either wait until the next garbage collection on the server side, or request cloud vendor to run a manual garbage collection.

Git Explained 1 of 2

Sat, 08 Jun 2019 22:39:00 -0400

In a nutshell, Git is a distributed version control system, commonly used as source control management. It places files in one of three logical areas (working dir, staging, repo) below depending where it is in its lifecycle.

There are many cheetsheets out there but this article will just sort through some concepts unique to Git. To understand how Git works it is crucial to think in terms of Git data model.

Working directory a single checkout of one version of the project. These files are pulled out of the object database in the Git directory (upon checkout) and placed in the project directory on disk, for you to use or modify;

Index a file contained in your Git directory (stored as binary data in file .git/index) that keeps information about what will go into your next commit. To display what’s in the index, run `git ls-files –stage`. Read this post for further details

Repository: where Git stores the metadata and object database for your project. The local repository is in .git/ under the project directory.

Add – register one or more modified files to staging area. You may edit several files with only a few needed registered for future commit. Add activity ensures the file edited are recorded in the index (as a preview of next commit). You technically need to run add against each file. But the command syntax with * or . allows you to capture all edits in the same directory or under.

Commit – persist the staged file edits to the repository (so they are stored in Git object database). A commit represents all the file edits that were staged by add command in previous steps.

Branch – a branch is simply a movable pointer to a commit. Default branch name created by git init is called “master”. Other than the name, there is nothing special about master branch. Everytime you commit, the master branch pointer moves forward automatically. Branch pointers are kept in .git/refs directory. Read this post for further details.

HEAD – the pointer to the current branch reference, which is in turn a pointer to the last commit made on that branch. Git use HEAD pointer to know what branch you’re currently on. HEAD will be the parent of the upcoming commit.

Tag – an annotated tag contains the SHA of the commit being tagged. Alias of a commit.

Merge – choose current commit of other branch and apply it onto your branch.

Rebase – copy all commits from other branch to your branch. Compared to merge, rebasing forms a cleaner commit history.

Cherrypick – choose a previous commit from other branch and apply it onto your branch.

Stash – temporarily stashes changes you’ve made to working tree so you can work on something else, and then come back and re-apply them later on.

Reset – at a high level, reset is to revert some operations. After pulling code, developer usually follow three steps: editing->add->commit. reset is to reverse these steps, based on different modes. The Pro Git reference has further details on the three different modes:

soft mode (reverse operation of commit): based on what branch HEAD points to, move where that branch points to (e.g. from latest commit, to a different commit several steps up the link);
mixed mode (default; reverse operation of commit and add) – in addition to soft mode, also update index;
hard mode (reverse operation of commit, add and file editing)- in addition to mixed mode, also update working directory. Edit on files are discarded.

Git object model – In Git database, files, commits and directories are stored as objects, In Git object model, there are three types (to tell object type, run `git cat-file -t`). Read this post for further details:

blob object – stores file data with metadata; use `git show` to examine blob object;
tree object – represents a directory. It references other tree objects (sub-directories) or blob objects (files under the directory, of a certain version); use `git ls-tree` to examine tree object;
commit object – represents a commit. It references its parent commit, as well as a tree object that represents the entire project directory. use `git cat-file -p` to inspect commit object;

This diagram from from Git Pro outlines the interactions amongst these types of objects.

For more details, the official documentation is actually the most helpful reference with illustrations. In addition, I find on Hakcermoon three excellent articles with thorough explanation on data model, branching and index.

Storage Nitty-Gritty 2 of 5 – SAN

Mon, 06 May 2019 22:06:02 -0400

In direct attached storage (DAS), storage is server centric and the host owns the storage. The storage is fully dedicated to the server that owns it.

With DAS, storage is server-centric

Storage has evolved into information centric model. In this model, when a new server is deployed in the environment, storage is assigned from the same shared storage pool to the new server.

Network based centralized storage solution

A network-based storage solution is the centralized storage pool. No single host owns the entire storage pool. The storage solution consists of two categories based on the interface with the host: SAN (storage area network) and NAS (network attached storage). To a client OS on the host, SAN typically appears as a local disk, allowing block-level access from the client OS, and therefore is more suited for structured workload such as database storage. It operates on its own storage network independent of the host network. NAS on the other hand, typically appears as a file share to the client OS, identified by an IP address and path. This is because NAS operates on the same TCP/IP network where the hosts are operated on. The client has file level access to storage, therefore NAS is better for unstructured data such as video and medical images. It is very important to understand the difference between SAN and NAS.

SAN deployment consists of two categories based on the connection technology. FC SAN is based on Fibre Channel network; and IP SAN is based on Internet protocol (iSCSI, FCIP, FCoE).

Fibre Channel SAN

Cable types: MMF (multimode fibre, usually for short distance within data centre because of signal attenuation due to modal dispersion) and SMF (single mode fibre, carries a single ray of light, used for long-distance cable runs;
Connector: standard connector (SC), lucent connector (LC) and straight tip connector (ST);
Interconnect device
- FC hub – for FC-AL implementation, but no longer in use
- FC switch – directly route data from one physical port to another (more intelligent than hub)
- Director – high end switches with redundant components to provide high availability
FC connectivity
- point-to-point: two devices connected directly to each other;
- arbitrated loop (FC-AL): devices are attached to a shared loop;

FC- AL (rarely used today)

switched fabric (FC-SW): uses switches that can switch data traffic between nodes directly through switch ports. Frames are routed between source and destination by the fabric

Fibre Channel Switched Fabric

Protocol: Fibre Channel Protocol (FCP): defines protocol stack (five layers, FC-0 through FC-4), addressing, identification (world wide name), frame, data structure, flow control, fabric services

Fibre Channel Protocol stack

FC frame

Topology: mesh topology and core-edge fabric topology
Block-level virtualization: aggregates block storage devices (LUNs) and enables provisioning of virtual storage volumes, independent of underlying physical storage. The virtualization layer maps the virtual volumes to the LUNs on the individual arrays. Block-level storage virtualization not only enables extending the storage volumes online; it consolidates heterogeneous storage arrays and enables transparent volume access. It also provides the advantage of non-disruptive data migration, where the virtualization layer handles the back-end migration of data, which enables the LUNs to remain online during migration.

Block-level virtualization (classic)

Federation of block storage across data centers (new generation)

Virtual SAN (VSAN, aka virtual fabric) – a logical fabric on an FC SAN, which enables communication among a group of nodes regardless of physical location in the fabric.

IP SAN

iSCI (one of the IP SAN protocols) – an IP based protocol that establishes and manages connections between host and storage over IP. iSCSI encapsulates SCSI commands and data into an IP packet and transport them using TCP/IP. It is relatively inexpensive and easy to implement so widespread in environments without FC SAN.

Topology
- Native connectivity (without FC components)
- Bridged connectivity (including FC components in the configuration)
- Combined connectivity (most common because a storage array usually comes with both FC and iSCSI ports)

iSCSI topologies

Protocol
- stack: SCSI is the command protocol that works at the application layer of OSI model; iSCSI is session-layer protocol that initiates a reliable session between devices that recognize SCSI commands and TCP/IP. The iSCSI session-layer interface is responsible for handling login, authentication, target discovery, and session management.

iSCSI protocol stack

iSCSI session and PDU encapsulation

PDU encapsulation

iSCSI discovery – an initiator must discover the location of its targets on the network and the names of the targets available to it before session establishment. Two types are SendTargets discovery and internet Storage Name Service
iSCSI names:
- IQN, iSCSI Qualified Name such as iqn.2008-02.com.example:optional_string;
- EUI, extended unique identifier such as eui.0300732A32598D26
iSCSI command sequencing – A command sequence may generate multiple PDUs. A command sequence number (CmdSN) within an iSCSI session is used for numbering all initiator-to-target command PDUs belonging to the session. This number ensures that every command is delivered in the same order in which it is transmitted, regardless of the TCP connection that carries the command in the session.

FCIP (one of the IP SAN protocols) – transporting FC block data over the IP infrastructure.

protocol stack and packet encapsulation

FCIP protocol stack

FCIP encapsulation

Topology (FCIP gateway involved):

FCIP topology

FCoE (one of the IP SAN protocols) – consolidation of LAN and SAN traffic over a single physical interface infrastructure. FCoE helps organizations address the challenges of having multiple discrete network infrastructures.

CNA (converged network adapters) replaces both HBAs and NICs in the server and consolidates both the IP and FC traffic
Special requirement on cables and switches
protocol stack and encapsulation

FCoE field mapping

CEE (converged enhanced Ethernet, or lossless Ethernet) provides new specification to existing Ethernet standard that eliminates the lossy nature of Ethernet. This makes 10Gb Ethernet a viable storage networking option, similar to FC. It features the following functionalities as part of IEEE 802.1:
- PFC (priority-based flow control)
- ETS (enhanced transmission selection)
- CN (congestion notification)

Application I/O Characteristics

Sat, 20 Apr 2019 22:09:00 -0400

There are many unknown factors and randomness when a solution architect determines storage requirement for an application. However, this process should still be as scientific as it can be and here are some of the important considerations regards application I/O characteristics.

Random and Sequential

	Random I/O	Sequential I/O
Description	Successive read/write operations from noncontiguous addresses – accesses that are spread across the addressable capacity of the LUN.	Successive read/write operations from contiguous addresses: one logical block address after another. In sequential I/O access, disk seek time is reduced because the read/write head moves little to access the next block.
Example	Messaging OLTP application	Data Backup

Reads and Writes

Another aspect of the I/O workload is the ratio of read I/Os to write I/Os generated by application. The sum of the read and write rate is the I/O rate (number of I/O operations per second). The application’s I/O rate is one of the important factors that determine the minimum number of disks required for application. In storage systems, cache plays an important role to improve the system performance. The table below summarizes how read I/O and write I/O interact with cache.

I/O Type	READ	WRITE
Random	Hard to effectively cache because of difficulty in predicting prefetch; Requires multiple fast disks for good performance	Caching is effective, resulting in a response time better than disk response time.
Sequential	Caching is extremely effective due to predictability of prefetch; Reads are done at cache speeds;	Caching is effective; cache is flushed quickly because entire disk stripe can be written.

Here are some typical read versus write ratio for common business applications:

OLTP: 67% reads and 33% writes
DSS (decision support, aka data warehouse or business intelligence): 80%~90% reads to data tables including frequent table scans (sequential reads)
Backup: As long as the file system is not fragmented, file-based backups are sequential

I/O Request Size

The size of I/O generated by an application may vary depending upon the type of the application. Some of the overhead to execute an I/O is fi xed. If data exists in large chunks, it is more effi cient to transmit larger blocks because a host can move data faster by using larger I/Os than smaller I/Os. The response time of each large transaction is longer than the response time for a single small transaction, but the combined service time of many smaller transactions is greater than a single transaction that contains the same amount of data.

Application	Seek Type	I/O Request Size	Proportion of I/O as writes
Microsoft Exchange	Random	32KB	Moderate to high
SAP/Oracle Applications	Random	~8KB	Depends on application
RDBMS: Data entry/OLTP	Random	Database or file system page size	Moderate to high
RDBMS: Online transaction logs	Sequential	512 byte+	High, except for archiving process
RDBMS: Temp Space	Random	Database or file system page size	Very high
Web file server	75% random, 25% sequential	4KB, 8KB, 64KB	95% read, 5% write
Web server log	100% sequential	8KB	100% write
Media Streaming	100% sequential	64KB	98% write; 2% read
OS paging	100% sequential	64KB	98% write; 2% read

Lightsail – create a WordPress site in one hour

Wed, 10 Apr 2019 01:30:44 -0400

My challenge with my just revived wordpress.com site is the plug-ins. Even paid premium users cannot install plug-ins for diagram, etc. I don’t want to upgrade to business plan so I decided to build my own.

14 years ago in university, I prototyped an intranet site using the Windows, Apache, MySQL, and PHP. It took a few weeks. Nowadays, this is referred to as WAMP stack with WordPress. AWS has a post on the best practices for hosting WordPress. However, I just need a single all-in-one server and there are some instruction on that. With Amazon Lightsail it was made a lot easier. The annual cost to host a WordPress site would be $55 in US, given that virtually nobody visits it except myself. The cost consists of:

Domain registration for $13 a year
Lightsail blueprint for $3.5 a month

There are several benefit of using this Lightsail blueprint, such as:

You can actually SSH into the EC2 instance. This is important to myself in the habit of probing into how things work;
Resources are all packaged into a fixed price plan. Remember the pesky accidental AWS charges?
You have the whole LAMP stack pre-installed, with the bitnami image for WordPress;
Start small but with scalability.

To start with Lightsail is extremely intuitive. Just click and launch:

There is a good instruction here on YouTube. Once launched successfully, there are some post configurations:

Request a static IP and assign it to the EC2 instance;
Register a domain (within Lightsail, with Route53 or any other domain registrar) and create an A record referencing the IP address;
Create a certificate and set up anto-renewal. Here is an instruction but the steps differ if DNS is managed outside of Lightsail;
Redirect http to https. Refer to the instruction from bitnami;
Export from wordpress.com and import the xml into this little site.

Notice that all services relevant to hosting are packaged into a single service called bitnami. In order to restart service, instead of restarting php, httpd and mysql individually, you can run the following:

# service bitnami stop
# service bitnami start

In fact, in /etc/rc5.d or /etc/init.d, I do not find the individual services for httpd or mysql. There is actually an instance of MySQL database in the Instance:

# mysql -u root -p
mysql > show databases;

There you go! Now the site is live. This little instruction will come in handy for rebuilding when this site is blown away 🙂

Migrate to new LightSail instance

I have been using the same LightSail instance for 3.5 years and it’s been stable. By Dec 2022, the version of PHP (7.2.17) has been outdated and I decided to launch a new LightSail instance based on new version of image, which comes with newer version of PHP (8.1.13). Here is a note of my steps:

Create a new LightSail instance with the same SSH key, assign a new static IP to it. As soon as the new instance is created, we can access it by IP on HTTP, using the initial credential provided.
We need the latest version of All-in-One WP Migration plugin to perform the migration. Ensure its latest version is installed on both old and new instance.
Export site to file using All-in-One WP Migration plugin. It should produce a single file with wpress extension. For this site, the site is 450MB by Dec 2022
For the new site, try to upload the file and notice that the max size allowed is 80MB. I made the following changes:
- Change PHP attributes. Edit /opt/bitnami/php/etc/php.ini at the following attributes:
  - post_max_size = 512M
  - upload_max_filesize = 512M
- Change WordPress attributes. Create /opt/bitnami/wordpress/.htaccess with the following attributes:
  - php_value upload_max_filesize 512M
  - php_value post_max_size 512M
  - php_value memory_limit 512M
  - php_value max_execution_time 0
  - php_value max_input_time 300
- Restart services related to wordpress
Now upload the .wpress file and it should take less than 5 minutes to upgrade. After the process, the new site is imported. Delete useless plugins.
Port the certificate files, including keys, certificate and certificate chains. In my case, I had to edit the following files:
- /opt/bitnami/apache/conf/vhosts/wordpress-https-vhost.conf
- /opt/bitnami/apache/conf/vhosts/wordpress-vhost.conf
- /opt/bitnami/apache/conf/bitnami/bitnami.conf
- /opt/bitnami/apache/conf/bitnami/bitnami-ssl.conf
Configure HTTP->HTTPS redirect on the new server. Revisit the files above.
Change DNS record to point to the new server.
Browse the site, check page and posts and pay attention to: images, tables, code blocks.
One common issue is images do not display. Check the image URL. The URL might have been replaced by bad values during import. Use a plugin (e.g. Go Live Update Urls) to correct the URLs in tables.
Use a plugin to find out broken links such as image, etc and fix the broken links. This should be an ongoing maintenance activity but is particularly worth doing on newly migrated site.

Other little things

There are many little things to fix. For example, I need to give transparent images white background. This is because when the site’s background is black and the transparent area in the diagram will be black too, making diagrams (authored in white background) illegible. To do this, I have to add the following section to the additional CSS setting of the active theme:

WordPress is dynamic site and if you don’t edit the site often, and don’t want to start a static site from scratch (e.g. using Hugo or Jekyll frameworks), you can use WP2Static plugin. Make sure to configure S3 bucket and CloudFront accordingly.

Now, enjoy blogging.

Interesting terms and principles

Sat, 26 Jan 2019 18:10:05 -0400

It is enlightening to find out that several things that I struggled at different times in professional life are actually experienced by many predecessors. Some smart people actually coined terms for these phenomena. Although they are somewhat negative, they won’t go away simply because not being mentioned. Both technical staff and project management professions should be wary of them. Some came out of software project management and some grew out of behavioural economics context.

Software Delivery

Here we go the terms, mostly excerpts from Wikipedia:

Death march project – a project that the participants feel is destined to fail, or that requires a stretch of unsustainable overwork. The general feel of the project reflects that of an actual death march because project members are forced by their superiors to continue the project against the member’s better judgement. Death marches of the destined-to-fail type usually are the result of unrealistic or overly optimistic expectations in scheduling, feature scope, or both, and often include lack of appropriate documentation or relevant training and outside expertise that would be needed to accomplish the task successfully. Often the death march will involve desperate attempts to right the course of the project by asking team members to work especially grueling hours, or by attempting to “throw (enough) bodies at the problem”, often causing burnout.

Software Peter Principle – in software engineering, software Peter principle describes a dying project which has become too complex to be understood even by its own developers. It is well known in the industry as a silent killer of projects, but by the time the symptoms arise it is often too late to do anything about it. Good managers can avoid this disaster by establishing clear coding practices where unnecessarily complicated code and design is avoided. This term is derived from the Peter Principle – a theory about incompetence in hierarchical organizations. There are mainly three causes of this:

Loss of conceptual integrity: The conceptual integrity of software is a measure of how well it conforms to a single, simple set of design principles. When done properly, it provides the most functionality using the simplest idioms. It makes software easier to use by making it simple to create and learn. Conceptual integrity is achieved when the software design proceeds from a small number of agreeing individuals. For software to maintain conceptual integrity, the design must be controlled by a single, small group of people who understand the code in depth. In projects without a strong architecture team, the task of design is often combined with the task of implementation and is implicitly delegated among the individual software developers. Under these circumstances, developers are less likely to sacrifice personal interest in favour of the interests of the product. The complexity of the product grows as a result of developers adding new designs and altering earlier ones to reflect changes in fashion and individual taste.
Programmer incompetence: the best developer should understand what details need to be communicated with people.
Programmer inexperience: programmers sometimes make implementation choices that work but have unintended negative consequences. Over time, many such implementation choices degrade the software’s design, making it increasingly difficult to understand.

Brooks’ law – an observation about software project management according to which “adding human resources to a late software project makes it later”. It was coined by Fred Brooks, according to who, there is an incremental person who, when added to a project, makes it take more, not less time. This is similar to the general law of diminishing returns in economics. Brooks admit the law is an outrageous oversimplification, but it captures the general rule. Brooks points the the main factors:

It takes some time for the people added to a project to become productive. Brooks calls this the “ramp up” time. Software projects are complex engineering endeavours, and new workers on the project must first become educated about the work that has preceded them; this education requires diverting resources already working on the project, temporarily diminishing their productivity while the new workers are not yet contributing meaningfully. New worker may even make negative contributions, for example, if they introduce bugs that move the project further from completion.
Communication overhead increases as the number of people increase. Due to combinatorial explosion, the number of different communication channels increases rapidly with the number of people. Everyone working on the same task needs to keep in sync, so as more people are added they spend more time trying to find out what everyone else is doing.
Adding more people to a highly divisible task, such as cleaning rooms in a hotel, decreases the overall task duration (up to the point where additional workers get in each other’s way). However, other tasks including many specialties in software projects are less divisible; Brooks points out this limited divisibility; Brooks points out this limited divisibility with another example: while it takes one woman nine months to make one baby, nine women can’t make a baby in one month.

Escalation of commitment – a human behaviour pattern in which an individual or group facing increasingly negative outcomes from a decision, action, or investment, nevertheless continues the behaviour instead of altering course. The actor maintains behaviours that are irrational, but align with previous decisions and actions.

Cowboy coding – the development process where autonomous developers in control of schedule, algorithms, frameworks and coding style work with minimal process or discipline. Usually it occurs when there is little participation by business users or fanned by management that controls only non-development aspects of the project, such as the broad targets, timelines, scope and visuals (the “what”, but not the “how”)

Conway’s law – organizations design systems which mirror their own communication structure. The law is based on the reasoning that in order for a software module to function, multiple authors must communicate frequently with each other. Therefore, the software interface structure of a system will reflect the social boundaries of the organization(s) that produced it, across which communication is more difficult.

Goodhart’s law – When a measure becomes a target, it ceases to be a good measure.

Murphy’s law – Anything that can go wrong will go wrong.

KISS principle – “Keep it simple, stupid”, as a design principle, it states that most systems work best if they are kept simple rather than made complicated; therefore, simplicity should be a key goal in design, and unnecessary complicity should be avoided.

Boondoggle – a project that is considered a waste of both time and money, yet is often continued due to extraneous policy or political motivations.

Optimism bias – a cognitive bias that causes someone to believe that they themselves are less likely to experience a negative event.

Planning fallacy – a phenomenon in which predictions about how much time will be needed to complete a future task display an optimism bias and underestimate the time needed. This phenomenon sometimes occurs regardless of the individual’s knowledge that past tasks of a similar nature have taken longer to complete than generally planned. The bias only affects predictions about one’s own tasks. When outside observers predict task completion times, they show a pessimistic bias, overestimating the time needed.

Separation of concerns – a design principle for separating a computer program into distinct sections such that each section addresses a separate concern. A concern is a set of information that affects the code of a computer program. A concern can be as general as “the details of the hardware for an application”, or as specific as “the name of which class to instantiate”. A program that embodies SoC well is called a modular program. Modularity, and hence separation of concerns, is achieved by encapsulating information inside a section of code that has a well-defined interface. Encapsulation is a means of information hiding. Layered designs in information systems are another embodiment of separation of concerns (e.g., presentation layer, business logic layer, data access layer, persistence layer).

DRY principle – Don’t Repeat Yourself. Every piece of knowledge must have a single, unambiguous, authoritative representation within a system. The principle has been formulated by Andy Hunt and Dave Thomas in their book The Pragmatic Programmer. They apply it quite broadly to include “database schemas, test plans, the build system, even documentation”.[3] When the DRY principle is applied successfully, a modification of any single element of a system does not require a change in other logically unrelated elements. Additionally, elements that are logically related all change predictably and uniformly, and are thus kept in sync. This principle is aimed at reducing repetition of software patterns, replacing it with abstractions or using data normalization to avoid redundancy.

YAGNI principle – You Aren’t Gonna Need It. Always implement things when you actually need them, never when you just foresee that you need them.

Golden Hammer – aka Law of Instrument, a cognitive bias that involves an over-reliance on a familiar tool. It is a form of narrow-minded instrumentalism.

Shturmovshchina – last minute rush, a common Soviet work practice of frantic and overtime work at the end of a planning period in order to fulfill the planned production target. The practice usually give rise too poor quality.

Principles more relevant software development

Least Knowledge (aka Law of demeter): this principle states that an object should never know the internal details of other objects. The dependencies between packages should be in the direction of the stability of the packages. A package should only depend upon packages that are more stable than it is.

Stable Dependencies: The dependencies between packages should be in the direction of the stability of the packages. A package should only depend upon packages that are more stable than it is. In Stable Dependencies principle, “Stable” roughly means “hard to change”, whereas “instable” means “easy to change”.

SOLID: SOLID is an acronym for five principles for OOP. The five principles are: Single-responsibility, Open-closed, Liskov Substitution, Interface Segregation, Dependency Inversion.

Inversion of Control (IoC): Inversion of control (IoC) is a design pattern in which custom-written portions of a computer program receive the flow of control from a generic framework.

Boy Scout Rule: Always leave the code better than you found it.

Persistence Ignorance (PI): The principle of Persistence Ignorance (PI) holds that classes modeling the business domain in a software application should not be impacted by how they might be persisted. For example, application code should not be affected by the chosen technology for underlying database or persistence storage. Business Logic should be independent of underlying technology for persistent storage.

Bounded Context: Bounded Context is a concept in domain driven design that is often used in microservice design. It provides a way of tackling complexity in large applications or organizations by breaking it up into separate conceptual modules. Each conceptual module then represents a context that is separated from other contexts (hence, bounded), and can evolve independently. Each bounded context should ideally be free to choose its own names for concepts within it, and should have exclusive access to its own persistence store.

Cohesion: cohesion refers to the degree to which the elements inside a module belong together.

That’s it for now. There are some more software development philosophies on Wikipedia.

Redhat Firewall configuration: from iptables to firewalld

Fri, 12 Oct 2018 17:56:00 -0400

Tools to manage firewall

Packet filter rules in Linux Kernel is managed by an user-space application named iptables in CentOS and RedHat. Since CentOS 7, firewalld is introduced as an alternative to iptables. Firewalld can be installed and executed as a systemd service, and it is supposed to replace iptables. This article describes how to configure both.

There are several advantages in firewalld. One is is the support of zones. Here are some useful information. Also, iptables involves three different services for IPv4(iptables), IPv6(ip6tables), and software bridging (ebtables), whereas firewalld only involves a single service to manage all three. Firewalld allows user to add or remove rules/ports from running firewall, without restarting firewall. Unless you have specific reason to use iptables, always use firewalld service to manage firewall. Here is an instruction to firewalld service. In this posting however, we will be focusing on iptables to understand firewall managment. We also go through an example of opening a TCP port.

How does iptables work

When working with iptables, it is important to understand that its related concepts (tables->chains->rules->criteria and targets) and how the order of rules plays a factor. There are five independent tables, each contains a number of chains, either built-in or user-defined. Administrators mostly deals with built-in chains in filter and nat tables. The five tables are:

filter: If -t isn’t specified, this is the default table. It contains built-in chains:
- INPUT: for packet destined to local sockets
- FORWARD: for packets being routed through the box
- OUTPUT: for locally-generated packets
nat: this table is consulted when a packet that creates a new connection is encountered. It has three built-in chains:
- PREROUTING: for altering packets as soon as they come in
- OUTPUT: for altering locally generated packets before routing
- POSTROUTING: for altering packets as they are about to go out
mangle: this table is used for specialized packet alternation, with five built-in chains (since kernel 2.4.18): PREROUTING and OUTPUT, INPUT, FORWARD, and POSTROUTING
raw: this table is mainly for configuring exceptions from connection tracking with two built-in chains: PREROUTING and OUTPUT
security: for Mandatory Access Control (MAC) networking rules, with three built-in chains: INPUT, OUTPUT, and FORWARD.

Under the table (e.g. filter, nat), each chain (e.g. INPUT, OUTPUT, etc) consists of list of firewall rules. Each rule is made up of two parts defined for the packets:

Criteria: if the packet does not match the criteria, the next rule in the chain is examined; if it does match, then the next rule is specified by the value of the target.
Target: what to do if criteria is met. The target can be:
- user-defined chain,
- one of the target described in iptables-extensions, or
- in most cases, one of the special values ACCEPT, DROP or RETURN
  - ACCEPT – let the packet through
  - DROP – drop the packet on the floor
  - RETURN – stop traversing this chain, and resume at next rule in the previous (calling) chain

The rules, defined in each chain under their tables, can be found in file /etc/sysconfig/iptables. You can find tables (prefix with asterisk *), chains (prefix with colon :), rules under their chains and a statement COMMIT after each table. The iptables process flow illustrates how a packet interact with all these rules under different chains and tables defined in this file:

iptables Process Flow

Although this big picture looks formidable, an administrator commonly only deals with the green and purple blocks (filter and nat), with the big picture in mind. Here is an example of /etc/sysconfig/iptables file from a newly installed system:

# Generated by iptables-save v1.4.21 on Fri Sep 11 23:15:32 2017
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [132:17200]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
# Completed on Fri Sep 11 23:15:32 2017

The rule simply allows SSH traffic. This file will be loaded up on every reboot (specifically, restart of iptables service). So if you have made some changes to rules and you want the change picked up on reboot. The rules should be saved to this file:

$ sudo iptables-save > /etc/sysconfig/iptables

Other than saving rule for reboot, if you simply want to edit the rules (e.g. order of rules is incorrect), you can save the rules to file, modify the file and restore the rule from file:

$ sudo iptables-save > ~/iptables.txt
$ sudo vi ~/iptables.txt
$ sudo iptables-restore &lt; ~/iptables.txt

Here is some further reading about iptables architecture.

Anatomy of a rule

The man page for iptables species the following synopsis:

iptables [-t table] {-A|-C|-D} chain rule-specification
rule-specification = [matches...] [target]
match = -m matchname [per-match-options]
target = -j targetname [per-target-options]

So when you append (-A), delete (-D), insert (-I) or replace (-R) a rule, you need to specify rule specification. The man page further explains that the following parameters make up a rule specification:

protocol (-p): the protocol of the rule of the packet to check. value can be tcp, udp, icmp, all or any name defined in /etc/protocols.
match (-m): specifies the name of a match to use and is followed by match options. The match refers to an extension module that tests for a specific property. Those extension modules are documented in the man page of iptables-extensions. You may specify -m multiple times for different match names, which together make up the condition under which a target is invoked. Matches are evaluated first to last as specified. We often use extensions tcp and state. According to iptables-extensions man page, we can specify –dport followed by port number for the tcp extension, and –state followed by value such as NEW or ESTABLISHED for the state extension.
jump (-j): specifies the target of the rule, such as ACCEPT, REJECT or DROP.
source and destination (-s and -d): source and destination IP address or masks. Hostname will work but not recommended since resolution is needed.
inbound and outbound interface (-i and -o): name of interface via which the packet was received and is going to be sent.
goto (-g): processing should continue in a user specified chain
Other parameters: -4/–ipv4, -6/–ipv6, -c/–set-counters, -f/–fragment

When we run iptables command to view rules, we need to specify the table (e.g. filter, nat, etc) followed by -S or –list-rules:

$ iptables -t nat -S

If you do not specify -t switch, the default (-t filter) is applied. Be aware that in this case, you’re only seeing rules under filter table, and not all rules under tall tables!

In the result, for example one line from command “iptables -S” may say:

-A INPUT -p tcp -m state --state NEW -m tcp --dport 9200 -j ACCEPT

The interpretation: appending a rule to INPUT chain of filter table (implicitly specified). The protocol is tcp. The first match extension is state, and the state value shall be NEW. The second match extension is tcp, and the dport value shall be 9200. If the packet is a match, then the target (action) is ACCEPT.

Managing rules

As mentioned earlier, rules can be dumped to any file or /etc/sysconfig/iptables, in which the rules are assessed in order. Below is a real life iptables file with a nat table as well.

# Generated by iptables-save v1.4.21 on Wed Jan 15 13:58:39 2017
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [4:208]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 7000:7001 -j ACCEPT
-A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 7199 -j ACCEPT
-A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 9042 -j ACCEPT
-A INPUT -s 10.100.160.56/32 -p tcp -m state --state NEW -m tcp --dport 9160 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 8080 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 161 -j ACCEPT
-A INPUT -p udp -m state --state NEW -m udp --dport 162 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT

*nat
:PREROUTING ACCEPT [1:328]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
-A PREROUTING -p tcp -m tcp --dport 2392 -j REDIRECT --to-ports 2398
-A PREROUTING -p tcp -m tcp --dport 2393 -j REDIRECT --to-ports 2398
-A OUTPUT -o lo -p tcp -m tcp --dport 2392 -j REDIRECT --to-ports 2398
-A OUTPUT -o lo -p tcp -m tcp --dport 2393 -j REDIRECT --to-ports 2398
COMMIT
# Completed on Wed Jan 15 13:58:39 2017

In this example, the nat table defines traffic forwarding: traffic arriving at TCP port 2392 and 2393 are forwarded to port 2398; outgoing traffic to port 2392 and 2393 are also redirected to port 2398. These rules do not overlap each other so the rules probably don’t matter.

On the other hand, the tcp filter table lists the rules to open certain TCP and UDP ports. Its block starts with a couple accepting rules and ends with a couple reject rules (regardless of protocols or ports). This is a good way to close a chain of rules with security. However, if you need to add additional rules to open more TCP ports, the new rule should not be appended after the reject rules at the bottom since the order matter here!

Correct way to open a TCP port

It’s a common task for developers to open a TCP port simply for the purpose of bring up a web service and make it accessible to client. If we simply add a new rule to existing list, for example:

# iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 9870 -j ACCEPT
# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9870 -j ACCEPT
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

# systemctl reload iptables

You will notice that the rule is appended to the end of INPUT block, below the INPUT REJECT rule, which will never take effect.

To address this, you can use iptables-save and iptables-restore to export, edit to correct order and reload the rule, as illustrated above, instead of using iptables command to modify the rule directly.

# iptables-save > /tmp/rule.list
# vi /tmp/rule.list
# iptables-restore < /tmp/rule.list
# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 9870 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited

Alternatively, you could use some advanced iptables command switches to add the new rule to certain line number with –line-number switch. Here is more information.

A review of vSphere virtualization technologies

Sat, 21 Jul 2018 15:54:00 -0400

This post is a summary of VMware’s white paper Introduction to VMware vSphere.

ESXi is the hypervisor (virtualization layer) on bare metal servers that abstracts processor, memory, storage and networking resources into multiple virtual machines. It was previously known as ESX and VMware discontinued ESX at version 4.1 so only ESXi is provided at and above version 5.0. vSphere is the platform to view, configure and manage the key aspects of virtualization, including:

computing and memory resources (hosts, clusters and resource pools)
storage resources (data stores)
networking resources (networks)
virtual machines

Under vSphere product family, vCenter Server is the central point for configuring, provisioning and managing the virtual environment. vShphere client is a client application to connect remotely to vCenter Server, or ESXi from any Windows PC. There is also vSphere Web Access for users from non-Windows environment.

For each aspects of virtualization, there is some vSphere features.

Computing

A host is a virtual representation of of the computing and memory resources of a physical machine running ESXi. When two or more physical machines are grouped to work and be managed as a whole, the aggregate computing and memory resources form a cluster. Physical machines can be dynamically added to or removed from a cluster. A cluster acts and can be managed as a single entity. It represents the aggregate computing and memory resources of a group of physical x86 servers sharing the same network and storage arrays. Computing and memory resources from hosts and clusters can be finely partitioned into a hierarchy of resource pools.

You can dynamically change resource allocation policies without shutting down the associated VMs. When reserved resources are not being used by a resource pool or a VM, the resources can be shared. This helps to maximize resource use while also ensuring that reservations are met and resource policies enforced.

ESXi provides a memory compression cache to improve VM performance when you use memory overcommitment. Memory compression is enabled by default. When a hosts memory becomes overcommitted, ESXi compresses virtual pages and stores them in memory. This is because accessing compressed memory is faster than accessing memory that has been swapped out to disk. Memory compression in ESXi allows you to overcommit memory without hindering performance. When a virtual page needs to be swapped, ESXi first attempts to compress the page. Pages that can be compressed to 2KB or smaller are stored in the VM’s compression cache, increasing the capacity of the host.

High Availability

VMware vMotion enables the migration of running VMs from one physical server to another without service interruption. The effect is a more efficient assignment of resources across physical servers.

Storage vMotion enables the migration of VMs from one datastore to another datastore without service interruption. This allows administrators to off-load VMs from one storage array to another.

VMware DRS (distributed resource scheduler) helps you manage a cluster of physical hosts as a single compute resource. You can configure DRS to execute VM placement, VM migration, and host power actions. When you create a VM on a cluster, DRS places the VM in such a way as to ensure that load across the cluster is balanced, and cluster-wide resource allocation policies (e.g. reservations, priorities, and limits) are enforced. When you add a new physical server to a cluster, DRS enables VMs to immediately take advantage of the new resources. When a VM is powered on, DRS performs an initial placement of the VM on a host. As cluster conditions (e.g. load and available resources) change over time, DRS migrates (using vMotion) VMs to other hosts as necessary.

VMware DRS

When DPM (distributed power management) is enabled, the system compares cluster-level and host-level capacity to the demands of VMs running in the cluster. If the resource demands of the running VMs can be met by a subset of hosts in the cluster, DPM migrates the VMs to this subset and powers down the hosts that are not needed. When resource demands increase, DPM powers these hosts back on and migrates the VMs to them. This dynamic cluster right-sizing that DPM performs reduces the power consumption of the cluster, without sacrificing VM performance or availability.

Storage I/O control congestion management allows cluster-wide storage I/O prioritization and enables administrator to set congestion thresholds for I/O shares.

VMware HA enables quick automated restart of virtual machines on a different physical server within a cluster if a host fails. HA monitors all physical hosts in a cluster and detects host failures. An agent placed on each physical host maintains a heartbeat with the other hosts in the resource pool. Loss of a heartbeat initiates the process of restarting all affected VMs on that host. HA also provides a VM monitoring feature that monitors the status of VM in an HA cluster. If a VM does not generate heartbeats within a specified time, VM monitoring identifies it as having failed and restarts it. HA is configured centrally through vCenter Server and once configured, it operates continuously and in a distributed manner on every ESXi host without needing vCenter Server to stay up.

VMware HA

VMware vLockstep technology and VMware Fault Tolerance provides continuous availability by protecting a VM with a shadow copy that runs in virtual lockstep on a separate host. Inputs and events performed on the primary VM are recorded and replayed on the secondary VM to ensure identical state. The secondary VM in virtual lockstep can take over execution at any point without interruption or loss of data.

Storage

Datastores are virtual representations of combinations of underlying physical storage resources in the data center. These physical storage resources include:

Local SCSI, SAS, or SATA disks attached to the physical machines
Fibre Channel or iSCSI SAN disk arrays
Network Attached Storage (NAS) arrays

Storage subsystem appears as a virtual SCSI controller connected to one or more virtual SCSI disks. These virtual controllers (BusLogic Parallel, LSI Logic Parallel, LSI Logic SAS and VMware Paravirtual) are the only types of SCSI controllers that a VM can see and access. The virtual SCSI disks are provisioned from datastore. This datastore abstraction is a model that assigns storage space to VMs while insulating the guest from the complexity of the underlying physical storage technology. The guest VM however, is not exposed to Fibre Channel SAN, iSCSI SAN, direct attached storage or NAS.

Storage Architecture

Each datastore is a VMFS volume on a storage device. Datastore can span multiple physical storage subsystems. A single VMFS volume can contain one or more LUNs from a local SCSI disk array on a physical host, a Fibre Channel disk farm, or iSCSI SAN disk farm. New LUNs added to any of the physical storage subsystems are detected and made available to all existing new datastores. Storage capacity on a previously created datastore can be extended without powering down physical hosts or storage subsystems. If any of the LUNs within a VMFS volume fails, only VMs that use that LUN are affected.

Each VM is stored as a set of files in a directory in the datastore. The disk storage associated with each VM is a set of files within the guest’s directory. You can operate on the guest disk storage as an ordinary file, which can be copied, moved, or backed up. New virtual disks can be added to a virtual machine without powering it down. In that case, a virtual disk file (.vmdk) is created in VMFS to provide new storage for the added virtual disk

VMFS is a clustered file system that leverages shared storage to allow multiple physical hosts to read and write the same storage simultaneously. VMFS provides on-disk locking to ensure that the same virtual machine is not powered on by multiple servers at the same time. If a physical host fails, the on-disk lock for each VM is released so that VMs can be restarted on other physical hosts. VMFS also features failure consistency and recovery mechanisms, such as distributed journaling, a failure-consisten VM I/O path, and VM state snapshots. These mechanisms can aid quick identification of the cause and recovery from VM, physical host and storage subsystem failures. VMFS also supports raw device mapping (RDM), which is a mechanism for a VM to have direct access to a LUN on the physical storage subsystem (Fibre Channel or iSCSI only). An RDM is a symbolic link from a VMFS volume to a raw LUN. The mapping makes LUNs appear as files in a VMFS volume. The mapping file, not the raw LUN, is referenced in the VM configuration. When a LUN is opened for access, the mapping file is read to obtain the reference to the raw LUN. Thereafter, reads and writes go directly to the raw LUN rather than going through the mapping file.

Raw Device Mapping

Networking

Each VM has one or more vNICs (virtual network interface cards). The guest OS and application program communicate with a vNIC through either a commonly available device driver or a VMware device driver optimized for the virtual environment. In either case, communication in the guest OS occurs just as it would with a physical device. On the network, the vNIC responds to standard Ethernet protocol as would a physical NIC. An outside agent does not detect that it is communicating with a virtual machine.

A virtual switch (vSwitch) works like a layer 2 physical switch. Each server has its own virtual switches. One one side of the virtual switch are port groups that connect to virtual machines. On the other side are uplink connections to physical Ethernet adapters on the physical server where the virtual switch resides. VMs connect to the outside world through the physical Ethernet adapters that are connected to the virtual switch uplinks. A virtual switch can connect its uplinks to more than one physical Ethernet adapter to enable NIC teaming.

Port group is a unique concept in the virtual environment. A port group is a mechanism for setting policies that govern the network connected to it. A vSwitch can have multiple port groups. A VM connects its vNIC to a port group instead of to a particular port on the vSwitch, for better network segmentation.

A vNetwork Distributed Swtich (vDs) function as a single virtual switch across all associated hosts. This functionality allows VMs to maintain consistent network configuration as they migrate across multiple hosts. Like vSwitch, each VDS is a network hub that VMs can use and it can route traffic internally between VMs or link to an external network by connecting to physical Ethernet adapters. Each vDS can also hae one or more dvPort groups assigned to it. dvPort groups aggregate multiple ports under a common configuration and provide a stable anchor point for VMs connecting to labeled networks.

When network resource management is enabled, vDS traffic is divided into six network resource pools: FT traffic, iSCSI traffic, vMotion traffic, management traffic, NFS traffic, and VM traffic. You can control the priority of each of these network resource pools.

vCenter

vCenter Server provides centralized managed for data centers. It communicates with the ESXi host agent through the VMware vSphere API. When you first add a host to vCenter Server sends a vCenter Server agent to run on the host. The vCenter Server agent acts as a small vCenter Server to perform many fundamental management functions.

DICOM data encoding

Sat, 30 Jun 2018 20:42:36 -0400

DICOM is a standard for medical imaging exchanges, originally in radiology, but later expanded into other departments where mass imaging data are acquired, such as cardiology. One part of the DICOM standard defines how to lay out the data without providing any official code implementation. It is up to each vendor to implement their application and declare what parts of DICOM standard they are compliant to in their individual conformance statement. Therefore it is common that different vendors have different perspectives of whether each other’s implementation is compliant.

Many arguments of this topic revolves around the encoding of DICOM data (or in strict term, Information Object Definition, IOD). In this article I try to clarify DICOM IOD encoding, best practices from vendor support. A more comprehensive coverage of this topic is chapter 5 of Oleg Pianykh’s book, which discussed the basics such as implicit vs explicit VR, big vs little endian.

The very purpose of standard is to determine a common protocol in which two application communicate with each other. A strong standard leaves no room for ambiguity in implementation and unfortunately, many healthcare IT standards are weak standards. When two devices from two vendors fail to communicate properly, the healthcare provider (device buyer) should take the lead of moderation, because they suffer the most pain from proprietary implementation and they benefit the most from good interoperability. In reality however, healthcare organizations with insufficient technical competency in their information technology team, usually leave it in Vendors’ hands to configure integration, with minimum supervision on standard conformance. This allows vendor to put in technologies that are just made to work, but not fully up to standard. This is not optimal. Remember: Proprietary technology = Vendor Lock-In

From vendor’s perspective, the implementation should just comply with DICOM standard. They should not accommodate to third party application that are incorrectly implemented. Vendor’s responsibility with customer is simply to proof that the data encoding is compliant with DICOM; or if otherwise is discovered, escalate to engineering with low level technical detail. It is a courtesy in the discretion of vendor’s support operation, to investigate and advise on the integrity of externally sourced DICOM data. In reality however, vendors are pressured to just make it work.

DICOM Objects

In DICOM encoding, an IOD, either in a C-Store or a part 10 file consist of hundreds of data elements. A data element (uniquely identified by a tag) can be either:

A single item;
A sequence (SQ) of multiple items;

Transfer Syntax

Below is the table that summarizes the metadata and pixel data encoding under different transfer syntax UID specified in (0020,0010). It is not meant to be a completed list.

transfer syntax UID	transfer syntax name	Metadata encoding	Pixel data encoding
1.2.840.10008.1.2	Implicit VR Endian: Default Transfer Syntax for DICOM	Implicit VR Little Endian	Implicit VR Little Endian
1.2.840.10008.1.2.1	Explicit VR Little Endian	Explicit VR Little Endian	Explicit VR Little Endian
1.2.840.10008.1.2.4.70	JPEG Lossless, Nonhierarchical, First- Order Prediction (Processes 14 [Selection Value 1]): Default Transfer Syntax for Lossless JPEG Image Compression	Explicit VR Little Endian	JPEG Lossless Compression
1.2.840.10008.1.2.4.80	JPEG-LS Lossless Image Compression	Explicit VR Little Endian	JPEG-LS Lossless Compression

Practically, Big Endian encoding is rarely used in DICOM. So is .99 so they are not covered. Little Endian simply refers to the reverse ordering of each pair of bytes. The rest of this article only discusses metadata encoding.

Encoding of Data Element of single item

Most DICOM parsers out in the market don’t have a problem with data elements of single items. However it is important to understand the encoding of single item before trying to understand sequence. Regardless of implicit or explicit VR, big or little endian, a single item is always encoded in the following sequence:

	Length	Data format	Example
Tag	Group number	2-byte	unsigned integer
Element number	2-byte	unsigned integer	0010,0010
VR (present only for explicit VR)	2-byte	2 ASCII characters	PN
Length of Value	2-byte	an even integer	0x000A
Value	determined by length of value	determined by VR	Smith^Joe

Note that in the example, length of value is 10 in decimal, and the value “Smith^Joe ” contains a trailing space to make up for 10 character length. It is required by DICOM that the length be even number of characters, which sometimes omitted and tolerated by different implementations. The corresponding DICOM’s guideline is here. On this page please understand Figure 7.1-1, Table 7.1-1 and 7.1-2 before reading on.

Encoding of data element with SQ type

When it comes to SQ (sequence), there’s much confusion about what are valid options for sequence encoding. There is also a good chance that a third party DICOM interpreter is incompletely implemented, and mistakenly complains correctly-encoded sequence as bad data. Symptoms include, but not limited to, A-ABORT an association, silence a TCP connection, complaining in their logs that the data is “corrupted”.

Here is the reference to DICOM standard as to the valid options for sequence encoding. The language is fairly abstract and I’m making some addition to elucidate it:

When determining the sequence encoding, DICOM needs to address two problems:
1. Define how to start and end a data item;
2. Define how to start and end the entire sequence;
You can explicitly specify the length of a data item, or leave it undefined; Similarly for the entire sequence, you can explicitly specify the length upfront, or leave it undefined. This leads to four possible combinations but one of them is invalid. The following table points to an example of each based on the tables in DICOM document:

	Sequence Length is explicit	Sequence Length is undefined
Data item length is explicit	Valid Format A Exemplified in Table 7.5-1, when sequence length is explicit, length of each individual data item must be explicit as well. – Sequence length is 0F00H, 3840 bytes – Data Item length is 04F8H, 1272 bytes – No delimiter (FFFE,E0DD or FFFE, E00D) is needed for sequence of data element – Sum of unit length equals total length: (1272 + 4 + 4 ) x 3 = 3840 Even though this particular example is implicit VR, the parser should know this is a sequence by the length calculation a. the length of data element (sequence) is 0F00H	Valid Format B Exemplified in Table 7.5-2, as well as the first Item in Table 7.5-3 Sequence length is undefined, marked by (FFFF,FFFF) as the length value of data element Data Item length is explicit defined, as follows: – 98A5 and B321 for the two items in Table 7.5-2 – 17B6 for the first item in Table 7.5-3 – FFFE,E000 marks the start of a data item
Data Item length is undefined	This is NOT a valid encoding option. It would be error prone, if the total length is explicitly defined but the unit length is not.	Valid Format C Exemplified in the second Item in Table 7.5-3 – Sequence length is undefined, marked by (FFFF,FFFF) as the length value of data element – Data ltem length is also undefined – FFFE,E00D followed by 00000000H marks the end of the data item – FFFE,E0DD followed by 00000000H marks the end of the sequence

As shown above, there are multiple valid options (A, B and C) to encode data item in a sequence and the entire sequence. If sequence length is undefined, explicit and undefined data item length can even co-exist within the same sequence. (Table 7.5-3)
if the length is left undefined at the beginning, you must clearly mark the end of the data item, or sequence using one of the special data elements.

Special Data Element used in SQ encoding:

FFFE,E000 (Data Item) – marks the start of each data item inside of SQ element; it shall be followed by a 4-byte field to indicate the length of the data item (either an explicit value or FFFFFFFFH to indicate undefined length)

FFFE,E00D (Item Delimitation) – marks the end of each data item only if the length of that data item is undefined; it shall follow the data item immediately and the length of itself shall be set to 00000000H

FFFE,E0DD (Sequence Delimitation) – marks the end of an entire sequence only if the length of that sequence is undefined; it shall follow the last item of the SQ element and the length of itself shall be set to 00000000H

cron and anacron in RedHat Linux (How logrotate works)

Tue, 15 May 2018 14:11:00 -0400

Cron and anacron

We all know cron is a job scheduler. Many admin uses crontab to manage scheduled task. It is also important to know that crontab works at different levels as well, as well as the distinction between cron and anacron. They are similar, but different, managed by different sets of files.

Below is a brief description of how cron works from this article.

After Cron starts, it searches its spool area to find and load crontab files into the memory. It additionally checks the /etc/crontab and or /etc/cron.d directories for system crontabs.
After loading the crontabs into memory, Cron checks the loaded crontabs on a minute-by-minute basis, running the events which are due.
In addition to this, Cron regularly (every minute) checks if the spool directory’s modtime (modification time) has changed. If so, it checks the modetime of all the loaded crontabs and reloads those which have changed. That’s why we don’t have to restart the daemon when installing a new cron job.

Basically, in cron, you specify a particular time at which a job will run. These jobs are managed by files in /var/spool/cron/ directory. In this directory, each file is named by the username that owns the crontab file. These files shall not be edited directly by respective users. Instead, they are edited by crontab by each user. To understand the syntax, one can refer to RedHat document for automating system tasks. Note that you can specify periodical jobs here with special syntax. For example, */5 at the minute slot indicates every five minutes.

Running cron jobs can be allowed or disallowed for different users. For this purpose, use the /etc/cron.allow and /etc/cron.deny files. If the cron.allow file exists, a user must be listed in it to be allowed to use cron If the cron.allow file does not exist but the cron.deny file does exist, then a user must not be listed in the cron.deny file in order to use cron. If neither of these files exists, only the super user is allowed to use cron.

In addition, there is a system-wide crontab file in /etc/crontab, in which you need to not only specify tasks, but also the user to run those tasks. By default, the schedule in this file is empty.

The limitation of cron is it assumes the servers is up all the time, if a script misses the schedule while the server is down, it will not be executed when the server comes back up. This is where anacron comes in handy. Although anacron can only be used by superuser, it doesn’t expect system to be running 24×7. If a job is scheduled at a time system is down, it starts the job when system comes back up.

Both cron and anacron are run by systemd service named crond.service. Although they require different packages installed (cronie vs cronie-anacron). They are also managed by different sets of files as explained below:

package	file or directory	purpose	example
cronie	/var/spool/cron/	this directory accommodates files that represents cron jobs for each individual users.	if a file named digihunch contains a valid line, it means that Linux user digihunch has a scheduled task for the time specified.
cronie	/etc/crontab	This file keeps system-wide cronjob entries. Each line needs to sepcify users.	if a line specifies schedule, user and command, it means that at the scheduled time, that user will execute the command.
cronie-anacron	/var/spool/anacron/	This directory accommodates files such as cron.daily, in which a timestamp is kept to indicate last execution time.	if cron.daily in this directory reads 20180418, it indicates last daily execution time stamp is 20180418
	/etc/anacrontab	This file tells anacron where in the file system to go for directories for periodical jobs.	Example: `#period in days delay in minutes job-identifier command 1 5 cron.daily nice run-parts /etc/cron.daily 7 25 cron.weekly nice run-parts /etc/cron.weekly @monthly 45 cron.monthly nice run-parts /etc/cron.monthly` The file usually also indicates RANDOM_DELAY and START_HOURS_RANGE
	/etc/cron.hourly/ /etc/cron.daily/ /etc/cron.weekly/ /etc/cron.monthly/	These directories stores script files that anacron needs to execute at different intervals. this is configured in /etc/anacrontab	if script logrotate is present in /etc/cron.daily/, it means the script is to be executed daily

An anacron example: logrotate

Rotating logs is a common task in Linux that can be done by logrotate. To understand how this works, first, make sure cronie-anacron package is installed and crond.service is up. Then examine the /etc/anacrontab file:

# /etc/anacrontab: configuration file for anacron

# See anacron(8) and anacrontab(5) for details.

SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22

#period in days delay in minutes job-identifier command
1 5 cron.daily nice run-parts /etc/cron.daily
7 25 cron.weekly nice run-parts /etc/cron.weekly
@monthly 45 cron.monthly nice run-parts /etc/cron.monthly

This indicates that daily, weekly and monthly jobs are active. Go into /etc/cron.daily/, and examine script logrotate:

#!/bin/sh

/usr/sbin/logrotate -s /var/lib/logrotate/logrotate.status /etc/logrotate.conf
EXITVALUE=$?
if [ $EXITVALUE != 0 ]; then
 /usr/bin/logger -t logrotate "ALERT exited abnormally with [$EXITVALUE]"
fi
exit 0

This indicates that logrotate loads configuration from /etc/logrotate.conf, the man page of logrotate explains how this configuration works, along with an example. If you have any custom application where the log file needs rotated, it can be configured in this file.

Basics Terms in Linux OS

Tue, 24 Apr 2018 19:16:00 -0400

These are the things quite confusing or abstract while I was at school but now makes lots of sense after many years working with different flavours of OS.

GPL and BSD as software license types

The main difference is that BSD (Berkeley Software Distribution) is a permissive (non-protective) license, while GPL(GNU General Public License) is a copyleft (protective) license.

Permissive licenses do not protect the code from being used in non-open source apps and apply no restrictions on the derivatives, while copyleft licenses force the creator of derivatives or re-distributor of the software to open the modified code. Under GPL you can’t sub-license, meaning, you can’t change any of the original license terms or introduce any of your own. You’re also required to state all the changes you make to the original code. That is why components licensed under GPL and other copyleft licenses should be avoided in commercial products that would later be distributed under proprietary licenses.

The BSD license family (including the Modified BSD License), on the other hand, doesn’t compel you to do any of the above. They have fairly relaxed redistribution terms.

GNU/Linux and BSD as operating systems

Unix – the name of the original system designed at AT&T in the 1970s. At the time, it featured a great deal of novelties such as multi-tasking, multi-user support, time sharing, etc. It was made portable by using C language.

Linux Kernel – a free, open-source, monolithic, Unix-like OS kernel. It was conceived and created in 1991 by Linus Torvalds for his personal computer. The word Linux, technically, is just the kernel. By itself, Linux (Kernel) has no place for user to land (e.g. no apps, no commands)

GNU/Linux – GNU project has developed a comprehensive set of free software tools for use with Unix and Linux. GNU/Linux involves the software tools along with the Linux Kernel.

GNU/Linux distribution – ready-to-use full OS, including Linux Kernel, GNU library and tools, whose developers have made a commitment to follow GNU GFSD, X Window and desktop environment (e.g. KDE, GNOME) and includes many pieces of software. GNU/Linux distro is what many people refer to as “Linux”, and it includes Debian, Ubuntu, RedHat and CentOS.

BSD (Berkeley Software Distribution) – an OS based on Research Unix, originally developed at Bell Labs, eventually grown into a complete operating system. Today, “BSD” often refers to its decendants, such as FreeBSD, OpenBSD, NetBSD, or DragonFly BSD. Each of these are both a kernel and an operating system. Another famous BSD descendant is Darwin, which is what Mac OS X based on.

Comparing “BSD” with “Linux” – Linux is more popular and tends to support new hardware sooner. Typical users usually don’t feel the difference between them. FreeBSD as desktop OS uses the same GNOME, KDE, or Xfce desktop environments that many flavours of Linux use as well. Although you need to install the desktop environment yourself. Another important difference is the licensing model as mentioned above between GPL and BSD. This article is a good reference about all the differences bewteen “BSD” and “Linux”
https://www.educba.com/linux-vs-bsd/

Swap, Cache and Buffer

Swap – swap file or swap partition. The primary function is to substitute disk space for RAM memory when real RAM fills up and more space is needed. The kernel uses a memory management program that detects blocks, aka pages, of memory in which the contents have not been used recently. The memory management program swaps enough of these relatively infrequently used pages of memory out to a special partition on the hard drive specifically designated for “paging”, or swapping. This frees up RAM and makes room for more data to be entered into your spreadsheet. Those pages of memory swapped out to the hard drive are tracked by the kernel’s memory management code and can be paged back into RAM if they are needed.

Swapping moves entire process between main memory and secondary storage; this is the original Unix method and can cause severe performance loss;
Paging moves small unites of memory (i.e. pages with 4Kbytes). It is more efficient and was added to BSD

In both cases, least recently used memory is moved to secondary storage and back to main memory only when needed again. In Linux, the term swapping is used to refer to paging. Older Unix-style swapping of entire thread and process is no longer supported.

CPU Cache – a hardware cache used by CPU to reduce the average time to access data from the main memory. A cache is a smaller, faster memory, located closer to a processor core, which stores copies of the data from frequently used main memory locations. Most CPUs have different independent caches, inclusing insructions and data caches, where the data cache is usually organized as a hierarchy of more cache levels (L1, L2, L3, L4, etc)

Page Cache (or disk cache) – kept by the OS in computer’s main memory and controlled by the computer. The OS keeps a page cached in otherwise unused portions of the main memory, resulting in quicker access to the contents of cached pages and overall performance.

Disk Buffer – (or ambiguously called disk cache or cache buffer) the embedded memory in a hard disk drive acting as a buffer between the rest of the computer and the physical hard disk platter that is used for storage. Modern hard disk drives come with 8 to 256 MiB of such memory. Disk buffer is physically distinct from and is used differently from page cache. It is controlled by the microcontroller in the hard disk drive.

GRUB (GNU GRand Unified Bootloader) – a boot loader package from GNU project. It is predominantly used for Unix-like systems. Current version is GRUB2

System call – In computing, a system call is the programmatic way in which a computer program requests a service from the kernel of the operating system it is executed on. This may include hardware-related services (for example, accessing a hard disk drive), creation and execution of new processes, and communication with integral kernel services such as process scheduling. System calls provide an essential interface between a process and the operating system. Below is a list of key system calls:

System Call	Description
read()	read bytes
write()	write bytes
open()	open a file
close()	close a file
fork()	create a new process
exec()	execute a new program
connect()	connect to a network host
accept()	accept a network connection
stat()	fetch file statistics
ioctl()	set I/O properties, or other miscellaneous functions
mmap()	map a file to the memory address space
brk()	extend the heap pointer

strace is the tool to trace system calls and signals in Linux.

sysVinit, runit and systemd

init – in Unix-based OS, init is the first process started during booting of OS. Init is a daemon process that continues running until the system is shutdown. It is the direct or indirect ancestor of all other processes and automatically adopts all orphaned processes. Init is started by the kernel during the booting process; a kernel panic will occur if the kernel is unable to start it. Init is typically assigned process identifier 1 and its job is to start other programs that are essential to the operation of your system. All other processes are descended from init.

init systems
Linux has several options as init systems. For example: sysvinit, runit, systemd and upstart. Here is a comparison of commands involved in managing each.
http://unix.cafe/wp/en/2017/07/howto-manage-a-service-in-systemd-sysvinit-upstart-runit-and-openrc/

SysV init – System initialization process is handled by the init daemon. One of the original daemon is SysVinit, which is a collection of System V-style init programs. init process starts serially. It is a run-once process during the start of the OS. One task starts only after the last task startup was successful and it was loaded in the memory. This often resulted in delayed and long booting time.

Runit – an init scheme for Unix-like operating systems that initializes, supervises, and ends processes throughout the operating system. It is a replacement of sysvinit and features brevity and simplicity.

Systemd – A init replacement daemon designed to start process in parallel, implemented in a number of standard distribution – Fedora, OpenSuSE, Arch, RHEL, CentOS, etc. Its flexibility comes with more complexity. It is an event driven init system, that not only starts stuff at boot (hence managing dependencies), but also after that. It also keeps track of many things after boot, such as mounts, availability of services, integration with resource management, etc. Because of that, systemd is also good at logging and monitoring. Systemd allows services to start when:

the system boots
a hardware components attaches to the system
other service started
a timer fires

To determine which system initialization method your current Linux distribution is using (SysVinit or systemd), simply check process id 1:

$ ps -p 1

Soft link and hard link

inode – stores the attributes and disk block locations of a file or directory

Soft/Symbolic link – essentially a shortcut to another file. The link itself is a separate file, pointing to the destination file or directory. The inode of the file is different from that of the symbolic link. Deleting the destination file will leave the symbolic link file a “dangling link”. Symbolic file may also have different permissions from the destination file or directory.

Hard link – essentially an alias of a file. The link itself is not a separate file, and the destination can only be a file (no directory). The inode of the link the the same as the file itself. So there is actually no distinction between destination file and link. Both files are equal. If you delete the file, the link continue to work until the number of hard links to the file becomes zero.

Soft link points to a file by name whereas hard link points by inode number.

Linux Admin Basics 1 of 3 – Bash

Fri, 02 Feb 2018 21:35:00 -0400

This article summarized my time-saving Linux tips, mostly with CentOS environment.

Bash Shortcuts

Bash shortcut illustration

These shortcuts save a lot of arrow keystrokes. Some of them may require some tweaking to work on MacOS.

ctrl – w	delete word
ctrl – a	cursor to end of line
ctrl – e	cursor to beginning of line
ctrl – k	delete to end of line
ctrl – u	delete to beginning of line
ctrl – l	clear screen
cd –	change to previous directory
alt – b	move cursor back by a word
alt – f	move cursor forward by a word
alt – .	type last parameter of previous command

Job Control

We need to first understand the following job states:

	Actively running	Displayed in Active Session
foreground job	Yes	Yes
suspended job	No	No
background job	Yes	No

These states can be managed by the following shortcut keys:

ctrl – z	send active job to suspended
ctrl – c	send SIGINT to active job to kill it
jobs	list jobs with id
bg	bring a job to background. the current shell still “owns” the job
fg	bring a job to foreground (by id)
disown	remove job from current shell’s job table. the job is still running and can be found by ps command
kill	kill a job by id. the job is no longer running and won’t be found by ps command

Note 1: running a command with ampersand(&) at the end starts the process and pushes it to the background, so you can continue typing;
Note 2: an example is to start vim with a file, ctrl-z to push to background, jobs to view, fg + job id to bring it back

Different persons may share screen on Linux shell and interact with each other. To do so, everyone need to log on the same server as the same linux user. Then the first person runs:

screen -S screen_name

Then the second (and third, etc) person runs:

screen -x screen_name

Now everyone can collaborate by seeing what each other is doing and type at the same time.

I/O redirection

	file descriptor	input or output expression
stdin	0	to read file: 0< or < for short to read a single line: <<< to read multiple lines: <<
stdout	1	to overwrite: 1> or > for short to append: 1>> or >> for short
stderr	2	to overwrite: 2> to append: 2>>

For example, here is how you display all lines between “ANYWORD” (aka “here document”):

cat <<ANYWORD
paragraph
after
paragraph
ANDWORD

In addition, to redirect between commands, pipe (|) is used. You may also redirect stdout into stderr or the other way round. Below are several examples:

command1 \| command2	redirect stdout of command1 to stdin of command2
command3 > /dev/null	get rid of stdout from command3
command4 2> file1	redirect stderr of command4 to file1
command5 \| tee file2	redirect stdout of command5 to stdin for tee command, which display stdout and write to file2 at the same time
command6 2>&1	redirect command6’s stderr into stdout.
command7 1>&2	redirect command7’s stdout into stderr

Note: in the last two examples, the ampersand just indicates the following number is file descriptor instead of file name.

For example, in the first command below, stderr is redirected to stdout, and stdout goes to file. The second and third commands are just two forms of shortcut for the first:

command > file 2>&1
command >& file
command &> file

Here is an advanced example to compare two files from two different servers:

diff <(ssh user@host1 'cat /tmp/file1.txt') <(ssh user@host2 'cat /tmp/file2.txt')

With the knowledge of I/O redirection, let’s compare the efficiency of the following two commands in dealing with a huge file:

cat huge.file | mycmd
mycmd < huge.file

In the first command, cat first causes an I/O read of huge.file, then stdout gets written into pipe buffer. Lastly, the mycmd reads it from its stdin. This is an example of inefficiency.

In the second command, huge.file is provided as stdin to my cmd. Only one read is involved as compared with two read and one write in the previous command. The second command is expected to be three times as fast as the first.

Here is a diagram to illustrate command and I/O redirection:

Check out the page for more scenarios.

Operators

First, let’s introduce some special shell variable and operators:

IFS	A special variable indicating internal field separator
;	The ; token just separates commands to run. Use it when you’d like to combine multiple lines of commands into a single line
&&	Logical AND operator. When you run command1 && command2, command2 ONLY runs if command1 returns true (success, or exit status = 0).
\|\|	Logical OR operator. When you run command1 && command2, command2 ONLY runs if command1 does not return false (fail, or exit status != 0)
{}	command combination operator. e.g. {command1;command2}
()	precedence operator

Note the use of && and || are fairly common. && is used when you want to use the first command to provide a good status for the second. || is used when you want to report error about the first command. For example:

test -d "/tmp/newdir" || mkdir -p "/tmp/newdir"
test -f "/var/run/app/app.pid" || echo >&2 "ERROR: cannot detect pid"

Attributes

I’ve been using chown and chmod to manipulate owners and file permissions against owners, groups and other users. I recently realized another command chattr (change attribute) which controls file attributes, regardless of users or groups. In other words, if you change the attribute of a file to immutable, then even the owner isn’t able to change it. Its manual has full list of attributes, but common ones are:

a -> Append only
i -> Immutable
c -> File automatically compressed in kernel.

For example, the append only attribute is commonly used for log file to prevent any other users from modifying the file. Sometimes we want to keep system files from being changed as well:

chattr -i /etc/resolv.conf
echo "nameserver 8.8.8.8" > /etc/resolv.conf 
chattr +i /etc/resolv.conf

Whatever attribute that we set with chattr, we can use lsattr to list them. These two commands are in Linux not BSD.

Loop

If you can tell how many iterations, then use for loop. For example, you want to execute a command for each file in a directory:

for file in /tmp/*.hl7
do
 echo "Picking File $file" >> output.txt
 hl7snd -f "$file" -d destionatiohost:2398 >> output.txt
 sleep 2
done

or you may specify range and steps for a for loop:

for i in {1..11..2} {13..23..2}
do
 echo "hostmachine"$i
 scp local.file "hostmachine"$i:/tmp/
done

Note that the way we use step here is only available on Bash 4.x (check version through variable $BASH_VERSION, reference)

If it’s hard to tell how many iterations, then use while loop. Here are some examples

Example 1. You want to execute the same command multiple times, each with a line in a file as parameter:

while IFS= read -r var
do
 echo $(date +%Y%m%d:%H%M%S) displaying line "$var" >> resend.log
 sleep 2
done < list.txt

Example 2. One step further, if the input file is a CSV format like this:

acc1,pid1
acc2,pid2
acc3,pid3

And you want to repeat a command multiple times, each execution with a column 1 as parameter 1, column 2 as parameter 2, etc. Then you can run something like this:

IFS=,$(echo -en "\r\n\b"); while read -r f1 f2; 
do 
echo processing $f1, $f2
curl -i -X PUT -H "Content-Type:application/json" -d '{"userId":"dhunch"}' --url http://localhost:9876/monitoring/document/$f1/$f2/publish;
done < input.txt >>out.txt

In this example IFS assignment is done before while loop. So it will take effect outside of while loop. This is why \r, \n and \b are specified in addition to comma. If this is not something you like, you may put IFS assignment after while and only specify comma as field separator.

Example 3. If you just want to repeat the same command every 5 seconds, apart from watch command, you can leverage while loop doing something like this:

while true; do df -Ph | grep "sda1"; sleep 10; done

In fact in all these examples while loop can be expressed in a single line, with semicolon to indicate where you would have typed enter.

find, with xargs and -exec {}

The find command offers flexibility searching files with certain conditions.

Example 1. Find files from within last 5 * 24 hours in current directory:

find -maxdepth 1 -type f -mtime -5

Example 2. Find tar files older than 6 * 24 hours:

find . -type f -name "*.tar" -mtime +6

Notes:

the switch -iname is similiar to -name but case insensitive
the switch -mtime goes by modified datetime; -atime goes by access time
the switch -mmin measures in minutes
the swtich -daystart makes it measure time from the beginning of current day (instead of 24 hours from current time of current day)

Here are more information about find command, and more examples.

To execute command per result from find, we have the options of xargs and -exec {}. Both build command based on parameter input, instead of I/O redirection. xargs is considered more efficient and it also works with commands other than find. -exec {} only works with find command.

Here’s a basic example of -exec {}

find . -iname '*.dcm' -exec dcm2txt {} \;

On the other hand, xargs works with any command followed by a pipe. It takes input from previous command, split it by space or carriage return into a list, then build command with each item on the list. To help understand parameter passing with xargs, we examine two examples. The first example is a directory with three files in it: x.a, y.a, z.a

$ ls
x.a x.b x.c

If we want to prefix each file with pre, we can do the following:

ls | xargs -I aa echo "mv aa prefix_aa"

In the second example, suppose find command produces the following result:

firstdir
seconddir

Compare the following two commands:

find . -type d | xargs ls -l
find . -type d | xargs -n 1 ls -l

In the first command, xargs invokes “ls -l firstdir seconddir” whereas in the second command, xargs invokes “ls -l firstdir” and then “ls -l seconddir”. The first command requires that the utility takes multiple parameters (in this case, ls does. Other commands such as wc, grep also do). The second command is particular helpful when the utility only takes one parameter. This is because the switch -n sets the maximum number of arguments taken from standard input for each invocation of utility.

Last but not least, operators are quite useful in Bash scripting. Here is a good reference.

technology on Digi Hunch

Virtualization 1 of 4 – Hypervisor

Hypervisor

Virtualization Techniques

Popular hypervisors

Market segments and players

Delivery model

Virtualization and Containerization

Virtualization and Cloud

How RPC and NFS work

RPC overview

NFS overview

Portmap

File handle

Permission and Locking

Procedures used in NFS service

Additional daemon processes

NFSv4

EMC Isilon storage product

Overview of EMC storage

Overview of Isilon Family

Isilon’s advantage

Isilon terms

Isilon’s High Availability

File Write in Isilon

Cache in Isilon

File Read in Isilon

Conclusion

Performance Analysis

Overview

Basic Troubleshooting

System Activity Report (SAR)

Berkeley Packet Filter (BPF) Compiler Collection (bcc) tools

Capture filter and Display filter in Network Analyzer

Networking Basics 3 of 3 – common network protocols and technologies

Networking basics 2 of 3 – Layer 4 and common network configurations

Transport Layer

Application Layer

Basic network configurations

Storage Nitty-Gritty 5 of 5 – Replication

Replication Terms

Local Replication

Remote Replication

Related Postings

Networking Basics 1 of 3 – Layer 1 through Layer 3

Physical Layer

Data Link Layer

Network Layer

Clean up Git repository

Clean up remote orphaned branches

Remove useless files in current commit (HEAD)

Remove large file and the relevant commits in the history

Other contributors re-sync history

Configure pre-commit hook as a preventive measure

Storage Nitty-Gritty 4 of 5 – Backup and Archive Solutions

Business Continuity

Backup

Data archive

Related Postings

Cryptography basics 2 of 2

AWS Certified DevOps Engineer Exam Tips

Storage Nitty-Gritty 3 of 5 – NAS and Object Storage

NAS (network attached storage)

Object-based storage

Related Postings

Cryptography Basics 1 of 2

Git Explained 2 of 2

Server side Git hooks

Client side Git hooks

Web hooks

Pipelines

Squash Commits

Cleanse a repository

Git Explained 1 of 2

Storage Nitty-Gritty 2 of 5 – SAN

Fibre Channel SAN

IP SAN

Related Postings

Application I/O Characteristics

Random and Sequential