Setting Up a Containerised App Pipeline

In my previous article, I described a quick and easy way to get a Kubernetes cluster up and running on your local machine. Hopefully by now, you’ve had a chance to try this out, created and deployed a few Docker containers, and are now convinced that containers are the way to go!

You probably noticed that the whole process of creating and deploying containers adds a few more steps to your development workflow. The good news is that we can automate these steps by setting up a Continuous Delivery pipeline, allowing us to build, test and run (in production) our applications using a single codebase and minimal intervention.

Introduction

If you’re about to start the development of a new application or move an existing one to a container-based hosting environment, then you will likely come up against the challenge of how to do so while maintaining the ability to iterate efficiently.

The initial challenge is to decide whether to make use of one of the existing, off-the-shelf, solutions for this or to just bite the bullet and build our own automated build pipeline. For example:

Option 1Just use OpenShift and save yourself some time.

Option 2 – Use Cloud Foundry, if you’re able to conform to their patterns.

Option 3 – Build it yourself.

If OpenShift or Cloud Foundry aren’t viable options, or if you just like the open source way of doing things and want to find out how it all works ‘under the hood’, then this article is for you.

Reinventing the Wheel?

If you’re blown away by the scope of PaaS solutions like OpenShift or Cloud Foundry, don’ worry – we aren’t going to try and develop anything on that scale. Instead, we will just be making use of existing frameworks and tools to build a pipeline solution that meets our needs.

The main areas, which our pipeline will be concerned with are the following:

  1. Building application source code – which tools we should use here will depend on the programming languages and frameworks used in the app. Our example will be a simple Node.js web application, written in JavaScript, so the only build process needed will be to obtain all of the required libraries needed to run it.
  2. Packaging application binaries inside Docker containers, along with any dependent libraries that may be required. The pipeline will also automate versioning and pushing new containers to a suitable repository. We will need to carefully consider which operating system image to use as a base for our containers – unlike OpenShift, for example, we won’t be aiming to automate this decision.
  3. Running automated tests (unit tests, integration tests, UI tests, etc) against newly-built applications, running in their containers or outside (dev environment only).
  4. Application monitoring for each of the environments
  5. Some kind of workflow management, which will allow us to promote container builds from the development, through test, to production environments according to the results of testing and any other gates or checks that we may wish to put into the process.

In other words, we will not be aiming to develop such a broadly-applicable solution as OpenShift, but we will be aiming to automate most tasks that we believe may be performed repeatedly in our particular context. We may also need to setup a very similar kind of pipeline again in future, for other projects perhaps, so we will also aim to make the environment setup as automated or scripted as is practical.

Continuous Delivery Pipeline

The figure below, adapted from Bryant[1], illustrates a typical application delivery pipeline, adapted to accommodate containers.

Containerised Application Delivery Pipeline.png

We’ll now consider each environment separately.

Dev Environment

The dev environment will be self-contained, allowing applications to be developed and tested locally, without any external dependencies.

Developers will, as usual, have the ability to build and run automated tests locally, both before and after container-wrapping an update, using existing tools such as JSUnit or Cucumber. Numerous IDEs and text editors now come with at least some degree of support for building Docker containers, so developers should be free to choose whichever one they prefer to work with, including the Docker command line tools.

Although containers will be built in this stage, the output of it is not a container, but rather the application source code including a Dockerfile. This will be hosted in a git repository of some sort, and it will need to be made accessible to the CI tools described in the next section.

CI Tools

We will make use of a Jenkins server, which will either poll for changes in the git repository or start builds whenever it receives a push trigger (e.g. from GitHub). One of the main aims of this effort is to make setting up new environments quickly and easily, so we will be treating our Jenkins servers as cattle rather than as pets.

As usual, Jenkins follows the same scripts for building, testing and code reviewing the app as are used in the dev environments. Assuming these tests all pass, a container build will then be triggered by Jenkins, using the CloudBees Docker Pipeline Plugin. If this build is successful, the container is uploaded to a container registry server.

This CI-built container image now becomes the single source of truth for the new version of the app, as it is promoted through the pipeline and into production.

QA

At this stage, the containerised app is run in a more integrated environment, most likely along with other application containers hosting any required dependencies, such as databases, web services or service stubs. This will enable performance testing and any non-automated acceptance testing to be carried out, before the new app container is promoted to Staging. In our example here, we will form the QA, Staging and Production environments using Kubernetes hosted on Google Cloud, but of course any other hosting (internal, external or cloud-based) or container orchestration solution could be used, such as Docker Swarm or Apache Mesos, as long as all three environments are as similar as possible.

Staging

In Staging, the new version of the containerised app is made available to a wider audience of users. Some organisations may choose to skip this environment, or combine it with QA into a single Test environment. Others may wish to have more than one Staging environment available, perhaps for different user groups to evaluate and test.

Production

Traditionally, only Operations teams had access to applications running in Production. With DevOps, we aim to bring the teams responsible for developing and running applications together. We will provide tools to enable new versions of an application to be deployed automatically, without any service disruptions. We will also allow developers to view live application logs, so they can see how well their code is running with real users and real data.

In the next article in this series, we will begin the process of setting up our delivery pipeline, by creating a CI environment.

Running Kubernetes Locally – why and how

In the beginning there were containers. Mostly Docker containers, but there were others too. Containers were great, but left to themselves, they were no more than individual musicians, without a conductor to hold them together or even a real orchestra to perform in. It was easy enough to get one container to do one thing well, but it was difficult to get them to play together.

Then came along container orchestration. Tools mostly developed mostly by companies with a need for massively scalable applications, such as Google and Amazon. Eventually, even Docker themselves saw the need, and also jumped in with Compose and Swarm.

Containers provide numerous benefits to developers, including dependency management, increased testability and increased scalability. Applications can now be developed and tested in environments very similar to the production environment they will be running in.

In this series of articles, we will see how to create a development pipeline using Docker containers. We will make use of Kubernetes as our container orchestration platform, as well as other freely-available tools such as Ansible and Vagrant, to help us automate the setup and running of the pipeline as much as possible.

Why are we doing this?

Kubernetes includes a command line tool, Minikube, which is a tool that makes it easy to run Kubernetes locally. Minikube runs a single-node Kubernetes ‘cluster’ inside a VM on your local computer. It’s mostly for users looking to try out Kubernetes or develop a simple application with it. While Minikube is easy to use and will help you get going quickly, it is restricted to running just one server node, and so won’t allow you to really test your application in a concurrent, multi-node environment.

Our local cluster

This article will detail the major steps required to setup a Kubernetes cluster running Ubuntu Linux. We will setup the cluster using several virtual machines hosted locally, using the free VirtualBox application. This cluster may be setup on any Mac or PC supported by VirtualBox and Vagrant.

Our cluster will enable us to test application containers in a multi-node environment, so that we can see how well they respond to the challenges of scaling. In particular, it will help us identify any issues related to concurrency even while in the development environment, so they can be resolved as early as possible in the development process.

The cluster will consist of 3 hosts:

  • one Kubernetes Master
  • two Kubernetes Nodes

Each host will consist of a virtual machine, running an identical copy of Ubuntu Linux. There must always be one master available in our cluster, but we can vary the number of nodes if required. This will allow us to create an environment as similar as possible to production, right on our local machine.

Prerequisites

If you are already familiar with Docker, then by all means, go ahead and dive straight into setting up the cluster by following the steps below. If not, then I recommend you begin by taking some time to learn the basics of Docker, as some of the concepts and terminology may be new to you. Here are some suggested resources to help you get up to speed quickly:

  • The Get Started guide for Docker is a good place to start. Go through the first 2 sections of this first. They cover installing Docker, creating containers and running simple containerised applications locally. Once you are clear on this, come back here.
  • The Kubernetes page on Wikipedia provides a quick intro to the design and architecture of Kubernetes.
  • Kubernetes Tutorials, to help you become familiar with the Kubernetes architecture, features and command line.

Once you have a basic grasp of Docker and Kubernetes, you should be ready to continue.

Download

  • VirtualBoxInstall this to run virtual machines on your local Mac or PC.
  • VagrantInstall this to allow quick and easy setup of the virtual machines we will be using in this article.

 

Install this on your local Mac/PC, to allow you to control your cluster and access the Kubernetes dashboard through a web browser.

Configuring and Running the Virtual Machines

By making use of Vagrant, we will spin up 3 virtual machines (VMs) using just one configuration file. One of these VMs will take the role of Master and the other two VMs will act as Nodes in our Kubernetes cluster.

  1. Download the required Vagrantfile and save it in a new, empty folder on your Mac or PC. You can either clone my GitHub repository or just download the file and save it to an empty folder on your local machine.
  2. Open a terminal or command line window and change to the folder that you saved the Vagrantfile to.
  3. Start up the VMs in one go
    $ vagrant up

    You will then see a number of messages, starting with:

    Bringing machine 'master' up with 'virtualbox' provider...
    Bringing machine 'node1' up with 'virtualbox' provider...
    Bringing machine 'node2' up with 'virtualbox' provider...

    as Vagrant downloads the ‘box’ image (the image of the basic VM we will be using) and sets up each of the 3 VM instances. Once the box image has been downloaded, numerous additional packages will be downloaded and installed automatically, including those required for Docker and Kubernetes.

    This process will take approximately 15 minutes to complete.

  4. Get the configuration for our new Kubernetes cluster so we can access it directly from our local machine
    $ export KUBECONFIG="$KUBECONFIG:`pwd`/admin.conf"
    
  5. Optionally, proxy the admin console to your local Mac/PC
    $ kubectl proxy

    Leaving the above command running, access the Kubernetes Admin Console in your web browser.

You should now have a fully working Kubernetes cluster running on your local machine, to which you can deploy containers using either the admin console or the kubectl command line tool.

In the next article in this series, we will see how to make use of Docker and Kubernetes to setup an automated container pipeline, with separate environments for continuous integration, testing and production.

In the mean time, have fun playing with containerised applications on your local machine!