Google Cloud Platform on Digi Hunch

The Leanest Web and Email Hosting

Thu, 27 Feb 2025 22:14:11 -0500

This site has been quiet for a while. During this time, I migrated the hosting platform again, and refactored email solution. This post, is another note about how I finally came to the most cost-effective web and email solution for a small business, with a solid security posture, and at nearly no cost.

For background, I have had this website for more than five years now, and revamped it a couple times. I have been with Amazon Lightsail at an already low cost. I didn’t have a lot of emails. So I have been relying on the email forwarding feature from my old domain registrar. There have been a few challenges. The content has grown a lot. The speed gradually slows down. WordPress needs reinstall. The outgoing emails get flagged as unverified.

WordPress Hosting

Worried about too many plugins, I debated about changing content management platform. Unfortunately, it’s not easy. There are some alternatives but none has all the features from all the plugins that I’ve been using. Even if they do, it would be too much work: I’d have to migrate the code snippets, highlights, etc, find similar themes and verify the new theme and new plugins work together. Despite of the criticism, WordPress today is still a predominant content management platform. So I stick to it.

Staying with WordPress isn’t maintenance-free. When PHP version has major update once in a while, I still had to re-install and migrate to a new server. Since I’ll exercise migration anyways, why not also shop around for a new virtual private server. Surprisingly I learned that Google Cloud Platform has an always-free tier including a decent size virtual machine.

	Amazon Lightsail 7USD/mo	GCP Free tier
Class	micro	e2-micro
CPU	2 vCPUs	2 vCPUs
Memory	1 GB	1 GiB
Disk	40GB SSD	30 GB (standard persistent disk HDD)
Data Transfer	1TB	1GB egress

The vCPU in different platforms do not represent exactly the same compute capacity, but should be generally comparable. The egress data transfer usage was below 1GB on Lightsail so the new service is sufficient. By going with GCP’s free tier, the only downgrade is the disk speed.

The GCP platform provides Bitnami package image and it’s fairly simple to spin up a new VM. The setup is as simple as Amazon Lightsail and I don’t have to deal much with networking. It doesn’t work out to be completely free but the net cost is negligible. I also picked the Nginx-based WordPress image because I had wasted a lot of time dabbling with Apache which I never use elsewhere.

WordPress Migration

With the new server up in a few minutes the next step is migration. The All-in-One WP Migration and Backup plugin (by ServMask) is the Swiss army knife. There are some pitfalls though.

First, the plugin has a size limitation in the free version. Luckily I can bypass it with changes to the PHP and Nginx configurations. For PHP configuration, change the line in /opt/bitnami/php/etc/php.ini, to upload_max_filesize = 512M. For the other change on Nginx as the reverse proxy, in /opt/bitnami/nginx/conf/nginx.conf, under http configuration, add:

client_max_body_size 512M;

The second pitfall is trickier. I have to temporarily host the new site, either by an public IP, or by a different DNS name (such as new.digihunch.com), so I can log in and verify the functions. Once I log on to the new site, the new DNS name or IP will start to be written to the database. I’m not sure which exact service does that. But once I put the new site on the production domain name (i.e. www.digihunch.com), the database still contains many entries referencing the temporary DNS name. This causes many insidious broken links.

There is a plugin for this kind of issue: Better Search Replace by WP Engine. The plugin goes through all tables in the database and allows you to find and replace strings. It also supports dry-run mode and I managed to fix tens of thousands of bad references. In addition, I managed to change username using a plugin called Easy Username Updater.

Another important activity after migration, is to ensure the MFA by WordFence plugin continues to work, and re-scan the entire file system.

CloudFlare Security

With less egress data and an HDD, the VPS in GCP is slower. I need to speed it up with CDN. It is hard to resist the freebies by CloudFlare. I’ve already been using it for security. Now it’s time to dig deeper.

CloudFlare greatly simplifies TLS certificates. It manages your certificate automatically. On the other side, it issues an origin certificate (and key, with long expiry) for communication with origin server. In full encryption mode, all I need to do is deploy origin certificate to Nginx. There is no manual renewal effort.

Note this setup requires CloudFlare to manage DNS so I just transferred my domain to CloudFlare. Therefore I consolidate domain registrar, certificate vendor and CDN to one vendor. CloudFlare can manage DNS record in proxy mode, to hide the public IP of the origin server from clients. I can also control the security group of the origin server to only open port 443 to CloudFlare IPs.

Another annoying but necessary configuration is the redirect of zone apex and http requests. Specifically most websites needs the following redirects:

If request from client is http://www.digihunch.com, enforce https;
If request is https://digihunch.com, redirect to https://www.digihunch.com;
If request is http://digihunch.com, enforce https, then redirect following rule #2.

I used to manage the these redirects in the reverse proxy (Apache). Now that I changed to Nginx, I offload these settings to CloudFlare, instead of configuring another reverse proxy again. For #1, I simply turned on “Always use HTTPS” and “Automatic HTTPS Rewrites” under edge certificates. For #2, I need a CNAME for zone apex to alias to www, along with a redirect rule. As a result, there is no need to ever open port 80 on the VPS, just to let the reverse proxy redirect URL.

CloudFlare Cache Rules

As to caching, I had to turn on a couple Cache rules to enable the caching. Cache hit was at about 50% and there are rooms to go higher if I enable reserve cache. Once cache is on, there are other considerations, such as expiring the cache, and by passing the cache during server deployment. CloudFlare allows you to purge cache by rules. There is also a Development Mode that temporarily allows you to bypass all cache for testing. With cache enabled, and the WordPress plugin activated, the page load time for recent posts appear to be faster than before.

One challenge is that the admin bar of WordPress went into the CDN and is served to visitors. To get rid of this behaviour I have two catch rules:

A cache-everything rule at order 1, to make everything eligible for cache;
A bypass rule at order 2 to bypass cache conditionally. The condition expression reads:

(http.cookie wildcard "wp-.*") or (http.cookie wildcard "wordpress_logged_in_*") or (http.cookie wildcard "wordpress.*")

To make use of the cache space. It is important to have small image sizes. Last year I added many AI generated feature images, most of which are 2MB in PNG format. I have to convert many PNG images into webp format, which only takes a fraction of the space and is widely accepted by most browsers today.

Sometimes there are other annoyances when it comes to speeding up page loading. For example, I noticed that with integration between Google Tag Manager and Microsoft Clarity, the home page issues a call to https://www.clarity.ms/tag/tag-id-xyz. This call blocks the rendering of a big chunk of home page by a minute! I didn’t notice it because my Brave browser’s ad-block removes that call. Nonetheless, the issue has been there for any visitor without built-in ad-block in their browser. The lessons learnt is that you always test with standard browsers, or at least in Brave browser with shields down.

SMTP service

I have been using email forwarding for incoming emails, and Gmail’s feature to send email from a different address or alias, for outgoing emails. This feature works but oftentimes the email gets marked as unverified on the recipient side. That gives it a good chance to appear as spam.

I need similar email forwarding mechanism and sending service. The free tier of CloudFlare goes a long way. Receiving is simply about creating Email routing rules, with a few MX and TXT records on DNS created and managed by CloudFlare. These records ensure CloudFlare routing rules captures all emails coming to the domain.

For outgoing emails, I have to get rid of the alias-based mechanism by Gmail. I need a proper SMTP server but CloudFlare doesn’t offer one. Resend seems like a popular choice with a good amount of free tier usage. However I’m hesitant to introduce another single-purpose platform so I just resort to Amazon SES for a full-feature, low-cost SMTP service. I do have to request production access, stating the server is for transactional email. This allows the SMTP service to arbitrary recipient address.

The other benefit of having my own SMTP server is to enable email integration in WordPress, which enables other important features such as web form, WordFense security alert and admin password reset by email.

Sending Reputation

To keep outing email from being marked as spam, it is important to understand what mechanisms are at play (SPF, DKIM, DMARC) to maintain sending reputation. At minimum, we configure SPF, DKIM and DMARC. SPF (Sender Policy Framework) identifies which mail servers are allowed to send mail on behalf of your custom MAIL FROM domain through a DNS TXT record that is used by DNS. The receiving mail server checks the SPF record of the sender’s domain to see if the email came from an authorized server. For example, when I use the SMTP server by Amazon SES, and customize the MAIL FROM field to mail.digihunch.com, I need an MX and a TXT record for mail.digihunch.com to tell receivers that it authorized amazonses.com to send email on its behalf.

DKIM (DomainKeys Identified Mail) is an email authentication method used to verify that an email message was sent by an authorized mail server and that the message content hasn’t been altered in transit. When an email is sent, the sending mail server generates a unique digital signature (based on the content of the email) using a private key. This signature is added to the email header. DNS Lookup: The receiving mail server looks up the sender’s domain in DNS to find the corresponding public key for DKIM authentication. Amazon SES configuration requires three TXT records to store the sender’s key information.

With SPF and DKIM configurations, the Amazon SES setup complies with DMRAC authentication protocol, making it less likely to be flagged as spam. The email comes off as mailed by amazonses.com and signed by digihunch.com.

In addition, there is Brand Indicators for Message Identification (BIMI) protocols that enables logo for your email on the recipient’s inbox. I managed to set up BIMI with SES, without Verified Mark Certificate (VMC), a protocol that requires evidence of ownership of the logo.

Although SPF, DKIM and DMARC cannot guarantee the email is not marked as junk, this is the best thing you can do. There are other techniques with diminishing return on effort.

Summary

This post summarizes my journey to land on the leanest web and email hosting solutions for small business. After five years of improvement, the tech stack that brings me low cost, solid security posture and lightening-fast speed, consists of: virtual machines from Google Cloud Platform (or Lightsail from AWS), web security and CDN by CloudFlare with Email supported by Gmail and Amazon SES (or Resend).

Initially with GCP, I tried to stick to the free tier but I did notice the standard persistent disk struggling, especially when I run system scan from WordFence. To overcome that I upgraded the disk to balanced disk (SSD backed) out of the free tier. There might be even cheaper alternatives but the marginal value isn’t worth the effort. Also, I configured the caching in CloudFlare for read, adjusted the home page, and schedule IO-intensive activities such as bi-weekly snapshot to quiet hours. The combination gave the site a 100 score in the CloudFlare speed testing.

Computing services: from PaaS to Serverless

Fri, 21 Oct 2022 19:31:00 -0400

Silicon Valley startups in mid-2000s likely do not run their own IT operations (i.e. renting their own data centre spaces, purchasing their own rack-mounted servers). Since the launch of EC2, AWS has been renting extra computing capacity to those startups, in the IaaS model. The leased infrastructure requires maintenance work, and AWS realized that many of these customers cannot afford specialized database admins, network admins, storage admins, or even server admins. As a result, they created a handful of managed services aiming to cut out admin overhead and let their customer focus on coding. This is how Platform-as-a-service (PaaS) came about.

Let’s take a look at what are exactly operation activities.

Ops activities

IT operation team manages server provisioning, installation of operating system, tuning performance, configuring auto scaling and load balancing, configure networking and storage systems, etc. Networking can be so complex that many infrastructure teams have a dedicated Network Operation Center (NOC), who along with security team, manages key aspects of networking, such as segmentation, router configuration, load balancing, firewall configuration.

For client-server application, the client-side code will make outgoing connections, utilizing the TCP/IP stack on the host through an ephemeral port. The server-side code has to be wrapped as a service. A daemon ensures the process running this service stays up and listens to a TCP port in order to respond to request by invoking the functions. Application team usually assumes these activities.

If database is involved, then the patching, upgrade, replication, data protection are all Ops problems. If storage is involved, then Ops has to manage mass data accumulated over years, the integration between storage and database and applications, performance, replication, etc. Some larger organizations have full-time database administrator and storage administrators.

Then comes container. Containers have their benefits but it increases the operation overhead by an order of magnitude. Running container application at scale warrants its own platform, most likely a Kubernetes platform, to address all of the problems above again at the cluster level. Some organization created platform team to manage container and VM platforms.

It is the Ops, that turns functional code into a running business. It is also the Ops, that becomes a pain point as a startup scales. With IaaS and PaaS models, AWS managed to convince many small businesses to delegate their IT operations to AWS. This is the humble start of cloud computing.

Elastic Beanstalk

At first, I wasn’t too impressed with Elastic Beanstalk, since it abstracts away too many details. However, I later realized that it has been surprisingly popular in the developer community, especially with individual developers and SMBs. It simplifies deployment to the point that their users don’t need to know other AWS services, allowing them to focus on coding application logic.

You configure Applications and Environments (one application may have multiple environments). In the Environment layer, you can specify code platform (e.g. Python 3.8 on 64bit Amazon Linux 2, Java, Go, PHP, Ruby) and even container platform (Docker on EC2 or ECS). Behind the scene, Elastic Beanstalk configures EC2 instances, Elastic Load Balancers, etc on the selected VPC and integrate with logging and monitoring services. In the console, Elastic Beanstalk exposes a list of configurations options (e.g. AMI, instance type). This centralized configuration page is dummied down for those who don’t want to deal with Ops.

The downside of Elastic Beanstalk is it takes away a lot of flexibility. Many developers find Elastic Beanstalk limit their choices of deployment, as their applications scale. Elastic Beanstalk does not suit for applications that demand extensive operation efforts. Its niche market is individual developers and SMB. Few enterprise applications run on Elastic Beanstalk.

Containerization with ECS and EKS

When we containerize an application, we build container images. Then we run these images with container runtimes, is a core feature of container platform. Container platform also provides orchestration engine since we frequently take containers up and down. In addition, container platform provides mechanisms for container networking and storage.

AWS has a couple options for container platform. ECS (Elastic Container Service) came out earlier. It organizes a group of EC2 instances as a cluster. You can manage autoscaling, networking, and persistent storage (EFS, FSx etc) on ECS. EKS (Elastic Kubernetes Service) is the managed Kubernetes service by AWS. Just like AKS, it provides a managed control plane along with computing nodes.

I see ECS as a proprietary and simplified container platform, and Kubernetes as an open-source standard for full-fledged container platform with an entire ecosystem. EKS includes an upstream-certified Kubernetes distribution with a set of tools specific to AWS. Since Kubernetes is the de-facto standard container platform, I prefer EKS by default, unless I can justify the use of ECS. In fact, ECS and Kubernetes have many concepts in common. For example, a “Task” in ECS is equivalent to a Pod in Kubernetes.

Whether it is ECS or EKS, right-sizing the computing node group is always challenging especially when the application traffic load is irregular. AWS Fargate is a technology that provides on-demand, right-sized compute capacities. It works with ECS and EKS. When integrated with EKS, we delegate the node management (e.g. scaling) to Fargate and forget about sizing the node pool.

Using ECS and Fargate involves quite a bit of configurations. To simplify that, we can use App Runner, which builds ECS cluster and uses Fargate to execute the container behind the scenes. App Runner helps client in a way similar to Elastic Beanstalk, but concentrate on Container workload.

Serverless with Lambda and API Gateway

The services above have their limitations when it comes to scaling capability. First, they cannot scale to zero. You still pay for idling resources. Also, it is not easy to find the optimal autoscaling setting. Lambda and API Gateway together solves these challenges. AWS refers to it as serverless, which has since become a buzzword. To understand what it is, let’s examine two concepts:

Function as a Service: service with the ability to execute code on demand. Users only pay for code execution time and do not care where the underlying runtime is
Backend as a Service: service with the ability to listen to a port and respond to web request

Lambda itself is a function as a service. Triggered by events, it only incurs a charge when it’s invoked. It does not stay up and listening to a TCP port for incoming web request, as does a backend service. In order to act as a backend, Lambda needs to pair up with API gateway. In this configuration, API gateway listens to a web request, and it fires an event to trigger the execution of Lambda function. Lambda and API gateway together makes a backend as a service. In AWS, the coupling of API gateway and Lambda function ensures an idle service does not incur computing cost.

Since Lambda supports many types of events as trigger, it is also used in event-driven architecture, either standalone or from a VPC. Under the hood, Lambda runs code in a container (with a quick startup time relative to a VM).

Developers can release Lambda code by uploading zip package to S3 bucket, or just packaging code into container image. For deployment, apart from AWS console and CLI, one can leverage CloudFormation, SAM (serverless application model), or CDK.

Lambda vs Fargate

Both Lambda and Fargate are serverless capabilities, at least from a marketing perspective. Both can be used to back web service but there are differences. They provision computing resource at different granularity.

In a web service, the execution duration of a Lambda function is the response duration to an API request, in terms of seconds. While the server is waiting for a request, there is no usage of the computing resource so you’re not paying for waiting for a request. However, this also creates the delay of cold-start, especially when the code size is large. There are several ways to optimize the cold start (e.g. SnapStart for Java), but none of those can completely get rid of the cold-start delay with a once-after-a-while request. A light GET call could take 5 seconds with cold start.

On the Fargate side, the resource provisioning is based on container lifecycle, instead of request lifecycle. As a result, you’re still paying for wait time, and it is not per-request billing. Since the container remains up, your request is not going to experience the cold-start if it’s been idle for a while. Although, Fargate saves you from the effort to right-sizing the computing nodes for container execution, it is not quite the idea of “scale-to-zero when idle” by itself.

Serverless Architecture

In the white paper AWS Serverless Multi-Tier Architectures with Amazon API Gateway and AWS Lambda, AWS advocates the serverless architecture as a modern alternative to the traditional widely adopted three-tier architecture (presentation, logic and data tiers). In the three tier architecture, the scalability of three tier are managed separately. The modern serverless architecture that AWS whitepaper proposes uses API Gateway and Lambda function in place of Load Balancer and EC2 instances (e.g. in an Auto Scaling Group), as illustrated below:

Both API Gateway and Lambda scale automatically to support the need of application workload. It assumes the role of logic tier in three-tier architecture but requires minimal maintenance work. For presentation tier, AWS has serverless alternatives such as CloudFront, S3. For data tier, AWS has serverless alternatives such as Amazon Aurora for relational database and DynamoDB for NoSQL. However, the “no request, no pay” model for Lambda does not apply to the data tier in serverless architecture.

Whilst this paradigm benefits small shop IT who wants to minimize infrastructure cost, it has downsides. There is no ability for infrastructure optimization. Since you do not manage where the code runs, client may have concerns over security (e.g. multi-tenant runtime). As business grows, keep using Lambda can result in technology lock-in. Also, a less used application usually requires warm-up time. A code start (downloading the code and preparing the environment behind the scene) can take 100ms to over a second.

Conclusion

PaaS attempts to help startups simplify the “grunt work” of IT operation. Serverless takes it even further. The semantics of serverless computing is confusing and the Wikipedia page acknowledges it as a misnomer. The nature of serverless model, is the cloud users delegate server capacity management to cloud platforms. The users don’t need to manage servers, VMs, instances, containers, etc on their own. In a previous post, I discussed the ability to scale to zero, which is just one of the many enabling technologies of serverless. Also, “no request, no pay” is neither an inherent nature of serverless model. Serverless service may involve storage (e.g. data service, S3, Aurora serverless) which incurs storage cost. There is a whitepaper on choosing the right AWS service to deploy your website or web application, with a decision tree.

AWS pioneered serverless with Lambda release in 2014 but competitor follows. In the Azure landscape, there is an entire suite of computing services from virtual machine to serverless (also with a decision tree in documentation). Azure’s counterpart for serverless architecture is Azure Function (released in 2016 for GA) with API Management. As for GCP, the serverless suite includes the event-driven Cloud Function (introduced in 2017) and Knative-based FaaS Cloud Run (introduced in 2019).

There are voices in advocacy of standardization of serverless model, and CNCF had since made minuscule efforts such as CloudEvents. The status quo, unfortunately, is anything but standardized.

Lastly, here is a table that summarizes the pros and cons of each computing service model.

Computing Service Model	Pro	Con
EC2	– Most straightforward and widespread legacy model – Legacy	– Ops tasks can be heavy (e.g. patch and vulnerability management of OS) – Utilization can be low
ECS	– Container orchestration is managed – Convenient to scale – Well integrated with other AWS services	– Limited advanced features – Vendor lock-in
EKS	– Highly scalable and flexible – Advanced, platform-neutral deployment tools available (e.g. Helm, ArgoCD, etc) – Custom configurations (e.g. operators)	– Significant operation overhead – Steep learning curve (especially for teams)
Lambda	– automatically scale – low ops overhead – pay per use	– limited choices of runtime – subject to latency due to cold start; yet warm start incurs cost – not suitable for long running tasks (batch processing jobs etc)

In summary, the Lambda-based serverless model is good for stateless server-side workload with short response time (<15s), tolerance of cold-start, no need for portability across platforms, and no complex package dependency.

Cloud storage overview

Wed, 12 Aug 2020 22:19:00 -0400

In a narrow sense, cloud storage refers to object storage. In a broader sense, it refers to any storage service (block, file or object level) provided by cloud vendors, in a cloud business model. The underlying technology of storage, is the same be it in the cloud or on-premise.

	Block storage	File storage	Object
Interaction with OS	OS has direct byte-level access to disk blocks.	OS manages storage by file, or byte range of file. Files are organized in POSIX hierarchy.	OS reads and writes the entire object, or a byte range, via rest API calls.
Metadata	N/A	Stored in file system, for directory or file	customizable metadata
Common protocol	N/A	NFS	S3
Implementation	SAN (bock device is typically dedicated to a single VM) or DAS	NAS, file storage is usually shared amongst multiple VMs. Locking mechanism is usually in place to keep access in order.	S3
Workload	database storage, scratch data, etc	persistent data, content management, etc	archive data, media streaming, data analytics, static asset serving, etc

Below is a list of common storage services provided by public cloud vendors to day.

	Block Storage	File Storage	Object Storage	Other managed storage service
AWS	Elastic Block Store (EBS)	Elastic File System (EFS) FSx for Windows FSx for Lustre	Simple Storage Service (S3)	Storage Gateway Snow Family DataSync
Azure	Azure Managed Disks	Azure Files	Azure Blobs	Azure Table Azure Queues
GCP	Persistent Disk local SSD	Filestore	Cloud Storage	Cloud Storage for Firebase Data Transfer
Digital Ocean	Volumes Block storage local SSD	N/A	Space object storage (S3 compatible)	Content Delivery Network

Storage Products from common public cloud vendor

Since AWS is the first vendor that provides a full suite of storage service, this post will focus on the storage product lines, as a refresher of AWS cloud storage options: Simple Storage Service, Elastic File Storage and Elastic Block Storage). There will be some overlap with the AWS storage service whitepaper.

Before getting further to details, here’s a reminder of two types of policies in AWS:

	IAM policy	Resource-based policy
Principal	Must be attached to individual user, group, or role to take effect	Needs to be explicitly specified, can be ARN under other AWS account
Element	Action/NotAction Resource/NotResource Effect (Allow/Deny) Condition	Principal/NotPrincipal Action/NotAction Resource/NotResource Effect (Allow/Deny) Condition
Example	Managed policy, custom policy	File system policy, S3 bucket policy, access point policy, etc

Two types of policies

Although the resource is usually assumed in a resource-based policy, the policy usually target a sub-section of a resource (e.g. object with certain prefix), so resource section is still required in resource-based policy. In storage services, we may use S3 bucket policy, access point policy, or file system policy for EFS.

Below we go over the three families of storage service in AWS.

EBS (Elastic Block Storage)

EBS is a distributed system. Each volume is a logical volume, made up of multiple physical devices. EBS data is persistent, and access is dedicated to a single EC2 instance at a time. If EC2 instance failed, the attached EBS volume can be detached, and then re-attached to other instance, in the same Availability Zone. There are two types of EBS:

EC2 Instance store: ephemeral, block-level storage for EC2 instance, no replication by default, no snapshot support. Used as buffers, caches, scratch data, temporary content.
EBS volume (persistent) : used for database, dev/test, enterprise application, etc. There are two sub-categories:
- SSD-backed volumes:
  - Optimized for transnational workloads that requires very low latency
  - Dominant performance attribute is IOPS
  - For frequent, read/write with small size and random I/O
  - Typical use case include relational database (PostgresQL, MySQL) and NoSQL (Cassandra, Mongo)
  - gp2 (general purpose) and io1 (provisioned IOPS)
- HDD-backed volumes:
  - Optimized for large streaming workloads demanding throughput
  - Dominant performance attribute is throughput
  - For workloads with lots of sequential I/O
  - Typical use case icnlude big data, analytics (Kafka, Splunk, Hadoop, data warehousing), file/media server
  - st1 (throughput optimized0 and sc1 (cold HDD)

The four types of EBS are compared here:

Note that the volume can be modified (change type, increase size) after creation. However, you cannot decrease size. If you increase the size, the file system must be extended after the increase.

Another way to deliver better performance is to use EBS-optimized instances. These instances have dedicated network bandwidth for its I/O traffic to and from EBS. Without EBS-optimized instance, the traffic between EBS volume and EC2 instance uses shared network link with EC2, which is subject to latency during heavy traffic. This distinction is similiar to the difference between iSCSI SAN and FC SAN. Also, you may increase read-ahead buffer in OS for better EBS performance.

On EBS, users can create snapshot, a point-in-time incremental backup. When snapshot is restored to a volume, data is loaded lazily in the background, so that volume is available immediately. This also means that initial read of data that is not yet loaded will be subject to latency, known as first read penalty. To achieve target performance, user may run an initialization on the volume, by reading all blocks with data upfront.

In a newly created snapshot, only the data blocks modified since the previous snapshot are stored as is. The rest are pointers to unchanged data blocks in the original snapshot. When a previous snapshot is deleted, AWS ensures changes are reconciled into the newer snapshot so there is no loss of data. Creation of snapshots on many volumes can be automated with Data Lifecycle Manager (DLM).

As far as encryption goes, the best practice is to create your own master key. KMS uses envelop encryption, where the data key encrypts the data, and the master key encrypts the data key. The encryption key is stored in EC2 instance memory only and never written to disk, for security and performance considerations.

EFS (Elastic File Storage)

EFS is a managed implementation of file storage that supports NFS 4.0 and 4.1, with strong data consistency and file locking. An EFS includes a single mount target in (one subnet of) each availability zone. EC2 instance, or on-premise client via Direct Connect, can mount EFS volumes using amazon-efs-utils yum package. EC2 instance can also be configured to automatic mount EFS volume in launch wizard. EFS also has a lifecycle management policy, and a storage class for infrequent access.

Performance wise, EFS has two performance modes and two throughput modes. The two performance modes are:

General Purpose: for latency-sensitive applications and general-purpose workloads. limit of 7k ops/sec, best choice for most workloads
Max I/O: for large-scale and data-heavy applications, with virtually unlimited ability to scale out throughput/IOPS, but with slightly higher latencies. consider this for large scale-out workloads

The two throughput modes are:

Bursting throughput: recommended for the majority of workload. Since file system workload is typically spiky, aws use credit system to determine when the file system throughput can burst. credit accumates idle time, and consumed in retrieval
Provisioned throughput: recommended for higher throughput to storage ratio workload, can increase the provisioned throughput afterwards. but it incurs separate throughput charge

Other ways to achieve higher performance, include parallelization of file operation (e.g. multiple threads, more instances); and increase I/O size for better throughput.

In terms of security, EFS encryption at rest must be selected at the time of file system creation. There is an TLS mount option to encrypt traffic in transit. EFS involves its own resource-based policy called file system policy to manage file-level POSIX permissions. IAM policy is used to manage NFS administration access and client access. EFS access points is also a means to enforce the use of a specific operating system user, and group to access EFS.

S3 (Simple Storage Service)

S3 is one of the earliest and maturest AWS services for object storage. It is very cheap and easy to use, and supports user-defined metadata on objects as well as many peripheral features. There is no limit to the number of objects in a bucket. As the object in bucket increases, S3 scales to request rate by automatically creating more partitions to meet the target number of request per partition. There used to be a performance trick, that requires client to make object key naming pattern distribute across multiple prefixes. It is not required any more as of July 2018.

Versioning can be enabled at bucket level, and suspended afterwards. New version of object is created on every upload, without performance penalty.

S3 integrate well with other event-driven AWS services, such as SNS, SQS, Lambda, etc. Event can fire on request such as PUT, POST, COPY.

Object tags (not to be confused with object metadata) can help categorize storage. It also facilitates access control (i.e. by being referenced in bucket policy or IAM policy), lifecycle policy, analysis and CloudWatch configurations.

S3 select is a way to retrieve only a subset of data from an object based on a SQL expression, to reduce amount of data and help with performance. The input can be json or CSV and output will be in CSV.

S3 Inventory is a tool to audit object replication status and encryption status. It generates CSV report with all objects in the given bucket name, including: key name, version id, islatest, size, last modified date, etag, storage class, multipart upload flag, delete marker, replication status, encryption status. For storage-class analysis, S3 inventory is much faster than list-object API call which parses through all objects.

S3 also has access point, similar to EFS, with unique hostnames that customers create to enforce distinct permissions and network controls for any request made through the access point.

S3 transfer acceleration take advantage of edge locations (at additional charge) to speed up transfer of large object over long distance, by providing a separate end point. It is also helpful for faster uploads over long distances. Apart from transfer acceleration, for faster uploads for large object, user may also consider multi-part upload API when the object reaches 100MB. Orphaned uploaded parts can be cleaned up in lifecycle configuration. For better download performance, take advantage of CloudFront and byte range request.