The container ecosystem market has seen quite a rise in the last few years. As enterprises shifted their business-critical applications to container-based architecture, the demand for developing, running, and operating containers has increased. Many of the core components for deploying containers, including Docker, Kubernetes, and Linux itself, have been summoned together to provide a complete container-specific platform in the form of a Container operating system.

Container specific OS provides an ideal platform for enterprises to run containerized applications and do not want to worry about security, updates, or OS support from their cloud providers. Various flavors of Linux optimized container OS such as Container Linux, DCOS, and RancherOS have already been around for a few years.

In this blog, we will discuss Amazon’s latest container-based OS Bottlerocket, which is purposely built for hosting Linux containers. We will also discuss how Bottlerocket enhances container functionality and how it provides leverage to container orchestration platforms such as Kubernetes and various other cloud-based Kubernetes offerings.

Before we dig deep into the feature set of BottleRocket, we want to give a general overview of how containers specific operating systems are typically used with container orchestrator such as Kubernetes.

Kubernetes provide tools and mechanisms for managing applications and many different applications on the same set of computers. They provide features like service discovery, network policy management, load balancing, application tracing, and more - key capabilities to successfully run microservices based applications.

However, running containers at a larger scale, across many computers, requires consistent and secure management. With the use of container based OS, enterprises can easily leverage the positive qualities of Kubernetes and use that in an operating system that hosts those containers.

AWS Bottlerocket

Amazon introduced Bottlerocket in 2020 as a Linux based operating for hosting containers in Amazon infrastructure. It runs natively in Amazon Elastic Kubernetes Service (EKS), AWS Fargate, and Amazon Elastic Container Service (ECS).

Bottlerocket is essentially a Linux based kernel having the additional functionality to run containers. Written primarily with memory-safe languages like Rust and Go. Bottlerocket strongly focuses on security and durability, making it the optimal solution for running orchestrated containers at scale.

Bottlerocket integrates well with various container orchestrators such as Kubernetes and provides the most efficient way to package and scale large container-based applications. It also mitigates all predominant challenges, such as keeping up with OS vulnerabilities and patching potentially large instances, updating packages while dealing with dependencies, and other OS processes.

The Bottlerocket project is a stripped-down version of feedback and lessons Amazons learned by running production services such as Elastic Container Service (ECS) and Elastic Kubernetes Service(EKS) at scale. Amazon combined, which was earlier provided in ECS and EKS AMI separately in one package to run containers and Kubernetes pods at scale easily.

Bottlerocket Features

Enhanced Security

Bottlerocket’s security approach reduces the attack surface to protect against outside attackers, minimize the impact that unverified software has on the system, and provide pod/container isolation by enforcing permission boundaries.

Bottlerocket runs all services in containers. If a container is compromised, there is a very slim chance that the entire system will be breached; updates are automatically applied when running the Bottlerocket using the Kubernetes operator that comes preinstalled with the OS.

Bottlerocket also reduces the attack surface of applications managed by container orchestrator tools by applying software hardening techniques such as building position-independent executables (PIE), making the loading of all the binary files and the dependencies of containerized applications into random locations within virtual memory each time they are executed.

Bottlerocket provides further protection by notably eliminating shell components: Bottlerocket doesn’t have SSH, any interpreters like Python, making it harder for an attacker to gain unauthorized access in the system. Bottlerocket also has SELinux enabled by default in enforcing mode to prevent modification of settings or containers from privilege/OS containers.

Bottlerocket provides cryptographic verification of the disk image on boot; any unexpected changes to the disk image contents will cause the operating system to fail to boot. Bottlerocket makes use of its own software updater rather than Linux package manager. All of the Bottlerocket updates are taken from a repository that follows The Update Framework (TUF) specification, which mitigates common classes of attacks against software repositories present in traditional package management systems.

Bottlerocket supports Kubernetes security monitoring tools such as Alicide out of the gate. Alcide is known to provide security for pre-deployment and production environments for Kubernetes misconfigurations. Bottlerocket seamlessly integrates with Alicide to minimize the attack surface by covering additional security layers and providing must-have guardrails in terms of Kubernetes security. Users can easily leverage Alcide’s kAudit and Runtime Protection(ART) capabilities in Bottlerocket, focusing on real-time Kubernetes glitches by scanning Kubernetes configuration.

Consistent Deployments

Enforcing consistency in your Kubernetes environment is critical when you want to run the same workloads on a different cluster of computers. Bottlerocket provides this consistency through three ways: image-based updates, a read-only root filesystem, and API-driven configuration.

Generally, Linux based distributions make use of an integrated package manager for installing and updating software. That provides flexibility for distributions, but when managing a large number of hosts, it can be a downside as different packages, and different versions of packages can be installed on each host, making them inconsistent with each other.

Bottlerocket makes use of a preconstructed image that contains the software for the operating system. This image-based mechanism ensures that all the Bottlerocket hosts in the environment can run the same software, and it is assured that each component’s specific versions included in a Bottlerocket image have been tested together. Updates for the new disk image are applied through a simple reboot.

Bottlerocket uses the same image-based model for a rapid and complete rollback if necessary. That removes opportunities for conflicts and breakage and makes it easier for you to apply cluster-wide updates with confidence using orchestrators such as Kubernetes and EKS.

Bottlerocket operating system is also configured with a read-only root filesystem. That is another mechanism that enforces consistency and reduces drift so that applications in Kubernetes are unable to modify the contents of the disk image and introduce changes from one host to another.

When updates are downloaded using Bottlerocket, they are first written to a secondary partition. During the reboot, the bootloader understands how to boot into the correct partition, changing the primary partition and leaving the old version of the container image as secondary.

This process is also used during the rollback procedure when you experience issues with the update. Bottlerocket makes use of a separate, writable portion of the filesystem that is designed for container volumes and images.

Bottlerocket has also implemented an API-driven configuration for consistency and secure management. Commonly in Linux based distributions, software configuration settings are stored in a particular directory, allowing applications to mutate operating systems and containers configurations.

But in Bottlerocket, an API regenerates this directory on every boot to supports secure migrations and storage of configurations. The API is accessed from the Bottlerocket control container via AWS for interactive changes, but can also be configured programmatically.

When using Kubernetes with Bottlerocket, a helper program also runs to configure Kubernetes-specific settings like the cluster DNS settings of the container image, which can be overridden using the API if you are running Bottlerocket on EC2.

Ease of Operations and Management

When it comes to operability, Bottlerocket is different from a traditional Linux-based operating system and has mechanisms for performing automatic updates, including Kubernetes integration for reducing node disruption.

Bottlerocket’s Updog, which is an on-host tool for interacting with the repository and retrieving updates, has the ability to look for the latest updates and apply updates to Bottlerocket immediately. Bottlerocket also comes with a Kubernetes operator to perform cluster updates. The operator ensures that only one host in your cluster gets updated at a time and will handle the draining of the pods before the update is applied.

Bottlerocket Updog tool also makes use of a wave-based update strategy that implements a mechanism to make updates available to different hosts in the cluster at different times rather than providing updates to hosts together. Such a strategy helps reduce the chance of all the hosts attempting to update at the same time, decreasing disruption to your container-based workloads, and allowing you to stop updates if you find out that there is an issue.

As we already discussed, Bottlerocket does not have SSH installed, and it is not intended to be managed using a shell. To access a Bottlerocket instance of Kubernetes managed containers. Bottlerocket uses two container runtimes, a control container for maintenance tasks like changing settings, and an admin container for emergency use.

A control container is managed by a separate instance of containerd and is launched on boot. It contains the AWS SSM agent, which allows interaction with AWS Systems Manager API. The control container is also enabled by default and facilitates communication with the admin container for shell access and emergency purposes.

On the other hand, the administrative container runs on the instance’s internal control plane and launch with full privileges and is unconstrained. The admin container is based on Amazon Linux 2 container image and has features that you would expect with a general-purpose Linux distribution. It has an SSH server that allows you to connect to Bottlerocket’s primary network interface using the SSH key specified during the launch of an instance.

There are many reasons why Bottlerocket implements this control-admin container mechanism:

1. The Kubernetes containers and host containers can have separate security requirements enforced by separate SELinux profiles.
2. The orchestrated containers can be launched by a different runtime than the host container.
3. The orchestrated containers and host containers can have separate fault domains for configuration changes and container runtime failures.

Fully Open-Source

Bottlerocket components are 100 percent open-source. All the components which make the OS like the Linux kernel and 50 other packages are written specifically for Bottlerocket. They all are licensed under their own original licenses, while all the specific components are licensed under the Apache 2.0 license.

Amazon, with Bottlerocket, has an intent to be a collaborative community project so that everyone can have the ability to contribute directly and to make their own customized versions. The code’s workflows are made available to the developer using GitHub, and they are encouraged to create new variants, which opens support for multiple orchestration platforms.

Also, to keep the OS footprint as small as possible, each Bottlerocket variant uses a specific orchestration plane. Amazon has included predefined versions for Kubernetes, and local development builds. For example, a new developer can create his own update operator or own control container by easily changing the container components.