With Kubernetes becoming the most popular container orchestrator for shipping and running containerized applications, developers can focus more on the application’s logic while delivering value.

In Kubernetes, containerized applications can be either stateful or stateless. Stateless applications do not have any persistent state, and they lose their data once the containerized application shut down or accidentally crashed. However, for stateful applications, Kubernetes needs to attach persistent external volumes for the storage of states.

Storage for stateful applications has always been a challenge as they have to be always provisioned with the volume which stays connected with pods through remote storage. Kubernetes community is trying to fill the gap for managing stateful apps for years by providing many abstractions and features. But containers by default are stateless, causing significant challenges for stateful workloads.

To mitigate this problem and adequately support stateful workloads in Kubernetes, we must understand how Kubernetes storage works. In this blog post, we will attempt to explain Kubernetes storage architecture by looking at different abstractions, challenges, and solutions for managing stateless and stateful workloads.

Kubernetes Storage Architecture

Volume-Based Abstraction

The architecture of Kubernetes storage is based on the abstraction of volumes. Volumes are defined as basic entities that containers can use to access storage in Kubernetes. They can be mounted on different storage infrastructure types, whether it’s local storage devices, NFS, and cloud storage service.

Custom storage plugins can also be created to support specific storage types depending upon how it is accessed from a pod. Pod utilizing more than two containers can have volumes mounted to particular paths within that container image.

Volumes in Kubernetes can be accessed directly in two ways. Ephemeral/non-persistent volumes can be used for short-term storage, whereas persistent volumes are defined for long-term/permanent storage.

Non-Persistent Volumes

There are many more differences between the two. By default, Kubernetes uses non-persistent storage as a basic storage unit that provides temporary storage until the pod exists. Non-persistent storage is defined as a container component inside a Kubernetes pod, which makes them unmanageable after the pod’s deletion.

Ephemeral volumes are mostly useful in scenarios where caching of temporary information is required. A simple example could be where Admins are deploying a logging agent within the pods to share logs with container agents, so implementation can stay separated from the main application image while providing continued access to application logs.

Persistent Volumes

Persistent storage on the other hand in Kubernetes is supported by various storage models including block/object storage and cloud services. Persistent storage is defined from outside of the pods which require a particular interface to connect storage devices.

In early support for persistent volumes in Kubernetes. Storage drivers are directly integrated into Kubernetes code to enable persistent storage devices for pods. But with an addition of Container Storage Interface(CSI) in Kubernetes. Storage connectivity and management have become simplified.

CSI has provided an extensible, pluggable architecture that enables vendors to provide drivers as plugins for easy integration into Kubernetes and guarantees cleaner abstraction between storage and Kubernetes.

It’s important to know that CSI only provides methods/plugins to connect storage devices to clusters. Persistent Volume (PV) and Persistent Volume Claims (PVC) have to be defined to get access to a persistent storage object.

Persistent Volume provides a storage instance/object that can be added to cluster volumes to define persistent storage. PV points to the actual physical storage, which is connected through CSI. Mounting a PV to a Kubernetes application requires a matching PersistentVolumeClaim (PVC) request that provision persistent storage with a specific type and configuration.

Both PV and PVC storage objects separate users and applications from storage configuration requirements. Administrators can securely back up and define details such as credentials, policies, and routes while staying completely independent from the attached pod.

In fact, PV’s helps taking backup in different storage types(NFS, GCEPersistentDisk, CSI..etc.) which is useful when you need to share the same data with a cluster spanned across various regions. PVs are considered essential when deploying a stateful application in Kubernetes as data has to be revived from outside of the pods through backup and replication.

PVC, on the other hand, is required by the application to identify the storage available in the defined PV. If it matches the requirements in the PVC request, it binds that particular storage. PVC can also specify some or all of the storage parameters defined in the PV.

Persistent Volume Provisioning: Static vs Dynamic

Static provisioning provides the freedom of manually creating a persistent volume that cluster admins can make available to pods for persistent storage. Static provisioning is usually done by administrators who have access to the underlying storage provider, supported disk configurations, and mounting options.

There are many use cases for static provisioning of persistent storage. Many times admins have to retain the data available in the cluster after the removal of PVC. They want to provision persistent storage directly into their cluster and share PV across clusters and namespaces in the same zone.

Dynamic provisioning of persistent volumes, on the other hand, utilizes a StorageClass API object that allows cluster admins to define storage type and configuration for on-demand volumes while eliminating the need to pre-provision them.

Storage class API in dynamic provisioning configures a provisioner that provisions a volume on a particular set of parameters. Once these values are specified, Kubernetes allocates a persistent volume depending upon the PVC request specification.

It is mandatory to create a storage class for dynamic provisioning of storage. Depending upon the type of storage options infrastructure can support, admins can create different storage classes for supporting specific storage needs.

Dynamic provisioning is essential and is required in workflows where you want to give developers the freedom to provision storage when they required it. Many organizations also want to automate cluster admins’ workflows so that there is no zero manual intervention; dynamic provisioning can be especially helpful in these contexts.

Storage Challenges in Kubernetes

Traditional storage solutions natively do not support Kubernetes

Kubernetes enables rapid scaling of production containers to accommodate a large number of users. However, scaling the containers with traditional storage can pose challenges.

Most of the traditional storage solutions available right now statically map to a cluster and are hard to assign to containerized workloads. Storage is assigned during the creation of hosts and stays consistent during the lifetime of the host.

While re-assigning storage volumes to hosts or clusters are possible, the design and architecture do not provide workload mobility, making traditional storage unsuitable for elastic resource scaling and automated provisioning required in Kubernetes workloads.

Kubernetes also promotes an automated approach to everything, whether it’s requesting new storage, changing the storage size, or dynamic mapping to different storage volumes. Traditional storage solutions were never built to support these workflows or continuous changes in storage configuration and provisioning.

Driver Specific Integration of Volumes

Kubernetes provides a wide variety of volume storage options to statically and dynamically provision volumes. However, volume storage options utilize different drivers depending upon the underlying infrastructure of the cloud platform.

Platform-specific drivers for storage options also require enterprises to purchase a complete platform for that particular functionality, which will increase the costs and time spent by enterprises to properly evaluate the storage requirements for your application.

To overcome volume driver integration issues, implement solutions such as OpenEBS and StorageOS, which fully integrates CSI for various storage plugins while avoiding platform lock-in.

Avoiding platform lock-in when dealing with Stateful applications is essential as storage data is written to storage provider infrastructure, and each storage provider has a different implementation for their infrastructures.

Solutions that avoid platform lock-in utilizes a data abstraction layer that allows easy migration of storage volumes across solutions in case of changing organization needs.

Lack of Enterprise-class features

Containers are natively designed to be non-persistent so that they can be easily spun up or down based on organizations’ workflows. Still, any application requires persistence to store and operate with the data.

This mismatch between the design of Kubernetes and enterprise requirement forces them to develop their own data availability mechanisms. So, enterprises who want to address global namespace or multi-cloud scenarios can easily do it through data reduction mechanisms (deduplication, compression, thin-provisioning) and data availability (snapshots, failover, replication) features.

Also, container deployment rapidly evolves with the changing elasticity of different cloud storage models. In contrast, many enterprises who look forward to seamless integration with these models lack the resources resulting in bottlenecks, performance and compatibility issues.

Storage Solutions in Kubernetes

The last few years have seen a significant increase in the number of tools and projects released to deal with storage issues on a container-based architecture. The most popular and evolved projects regarding storage in Kubernetes are OpenEBS, Rook, and Portworx.

OpenEBS

OpenEBS is the leading container-native storage solution that is very easy to install and use because of its Container Attached Storage (CAS) approach. In OpenEBS, each Kubernetes volume has a dedicated storage controller, which helps achieve granular control on persistent storage operations for Kubernetes applications.

OpenEBS allows persistent workloads to be deployed as just another container, enabling storage management on a per pod basis. Monitoring these persistent workloads is possible through OpenEBS volumes, which provide metrics on throughput, latency, and data patterns.

With such granular metrics, OpenEBS can easily debug storage for stateful applications by observing the traffic data patterns while adjusting the storage policy parameters.

Rook & Ceph

Rook is an open-source container storage orchestrator that provides storage management capabilities for different backends, such as Ceph and EdgeFS. Rook and Ceph are among the most popular cloud-native storage solutions accepted as the first storage project into CNCF with over 16M downloads.

Rook makes it easy to use Ceph, which provides storage management capabilities by providing a logical abstraction over Kubernetes storage components. Ceph has advanced distributed storage features that can make scalability easier for stateful applications by eliminating the point of failure while providing unified access to persistent storage objects.

Portworx/PX-Storage

Portworx is a company specialized in providing Kubernetes-based products ranging from storage, security, and monitoring. PX-Store is known for its performance and data replication features, which clones data at least two other nodes. So, in case of a node failure, loss of volume replicas continues to be available at the same performance levels.

Portworx defines a software-defined storage overlay that allows stateful applications data to be highly available across multiple nodes, data centers, or clouds. It implements HA through synchronous replication, which becomes especially useful when enterprises want to build highly available stateful applications through local disks on cloud providers.

Rancher Longhorn

Longhorn being a fully open-source platform, provides enterprise-grade persistent storage implementations for various Kubernetes workflows. At the core, Longhorn is a distributed block storage system that simplifies cloud-native storage by reducing the cost overhead through easy backups and incremental snapshots.

Scheduled backups of persistent storage volumes are taken through Longhorn’s intuitive and easy to manage UI, which also provide maximum granularity over storage components.

For disaster recovery, Longhorn implements a dedicated storage controller for each block device volume, synchronously replicating volumes across multiple replicas stored on different nodes. The storage controllers and replicas can be easily managed within the Kubernetes environment compared to externally deployed replication solutions, which can take days to replicate the entire data store.

Longhorn also creates a disaster recovery volume in another Kubernetes cluster in case your primary cluster fails; enterprises can quickly bring up the app stored in the DR cluster without any data losses.