DevOps

From Localhost To Production Using The DevOps Way - Part 1

This post covers:

  • Making a case for automation
  • Defining what is a Pipeline
  • Defining a game plan to automatically ship an application to a container repository
  • Setting up a Continuous Integration Platform
  • Breaking down the implementation

Making a Case for Automation

Building a compiler is not an easy task. It involves the development of many modules such as a lexer, parser and a virtual machine. It also involves the processing of variable declarations and the evaluation of simple arithmetic expressions all the way to calling returning functions with parameters such as a factorial recursive function.

Along the process we found out a lot of things during the development phase that were not considered during the design phase. As a result, we were constantly moving the grammar to make sure edge cases worked. That is why we decided to take a step back and take advantage of the unittest module that python3 provides. For each .tl file, our programming language file extension, we created a test. This allowed us to make changes faster while eliminating the fear of breaking previous working code.

Tests were focused on three different categories:

  1. Correctness of the intermediate code generation (quadruples)
  2. Expected failures of .tl files with the expected number of errors
  3. Correctness in expected executed output

As a result, doing make test runs 50+ tests. But what happens if a developer forgets to run the tests before doing a pull request or merging to master? Thankfully we use GitHub, so going back to a specific commit is possible, but is it practical? Do we really want to go through that painful way?

Furthermore, trendlit, our compiler, is meant to be cloud-based. So that means that we also need an easy way to deploy it. On top of that, since trendlit runs inside a Docker container, we also need a way to build the Docker image, check if the current version already exists in the container repository (e.g. Docker Hub), and eventually push that image to the registry. Therefore, we decided to do it the DevOps way.

But First, What is a Pipeline?

A pipeline is a set of automated processes that allow Developers and DevOps professionals to reliably and efficiently compile, build and deploy their code to their production compute platforms. There is no hard and fast rule stating what a pipeline should look like and the tools it must utilize, however the most common components of a pipeline are; build automation/continuous integration, test automation, and deployment automation.

Implementation Roadmap

As we previously mentioned, we want to manage trendlit the DevOps way. To do this, we will implement steps of Continuous Integration (CI), Continuous Delivery (CD) and Infrastructure as a Service (IaaS). For this post, we are going to focus on the Continuous Integration part.

Why would we even want CI?

Continuous Integration is a development practice that requires developers to integrate code into a shared repository several times a day. Each check-in is then verified by an automated build, allowing teams to detect problems early.

As a result, errors can be detected faster and can be located more easily. This allows an organization to deliver software more rapidly, while reducing the risk for each release.

For this example, the idea is that a developer, while adding or fixing a feature, can go from localhost to the application being pushed to a registry within minutes, while relying on an automated process. The workflow looks something like this:

  1. A developer adds new feature to the application.
  2. The application goes through an automated testing process.
  3. When the developer makes a pull request, the application goes under the four-eyes principle and if it is approved by another peer and the automated tests pass, the new feature is merged to the master branch of the code repository.
  4. A new version of the application is automatically built and packaged into a container.
  5. The container is pushed to a container repository.

Getting Started

In order to run the implementation roadmap described above, we chose to use the following tech stack:

Code Repository

We will use GitHub in order to host the application’s source code.

CI Platform

When it comes down to pipelines, we can find them in all colors and flavors. For this project, we chose to go with CircleCI because it integrates easily with GitHub and allows SSH access to build instances, which is handy for debugging the build steps. Is worth mentioning that we could have replaced CircleCI with another automation server such as Jenkins, but it takes longer to setup because it is self-hosted.

Container Repository

We’ll use the repository that is provided by Docker Hub.

Setting up CircleCI

This is the application’s code repository: trendlit-tutorial-1

Here you can also find all the scripts that the pipeline is going to run in order to test, build, check, and push the application to Docker Hub.

Steps

  1. Fork the project, so you can create your own pipeline.
  2. Go to hub.docker.com and create an account and a repository with trendlit-tutorial-1 as namespace.
  3. Sign up to CircleCI and link your GitHub account.
  4. Select the forked version of trendlit-tutorial-1 and build the project. The build will fail because you need to put your Docker Hub’s credentials.
  5. In your CircleCI’s dashboard in the project list go to trendlit-tutorial-1 settings.
  6. Under Build Settings you will see Environment Variables. Add DOCKER_USER and DOCKER_PASS as variables with their respective values.
  7. Rerun the workflow and this time it should succeed.

Afterwards, go to your Docker Hub and you should see three new tags. One with the trendlit’s RELEASE that was specified at the top of the Makefile, other one with a shortened version of the commit hash and the other one named latest.

How does it work?

Webhooks

The pipeline is able to listen each time there is a change in the repository thanks to a webhook. According to GitHub, webhooks allow external services to be notified when certain events happen. When the specified events happen, GitHub sends a POST request to each of the URLs provided.

In other words, when you link your GitHub account with CircleCI, you allow the latter to perform some actions in your repositories. One of them is automatically configuring a webhook. So every time there is an event such as a pushed commit or a pull request, this webhook sends a notification to CircleCI in order to trigger the build.

Decomposing the Config File

Once the build is triggered by the webhook, CircleCI goes to the config.yml file located inside the .circleci directory.

1. Jobs - The pipeline consists of 2 main jobs: Test and Deploy.

2. Tests - Here we select Docker as an executor type. Then, we use the python:3.7-alpine image in order to run some tests.

3. Deploy - Here we use an official Docker image called docker:stable-git, because we need a Docker image that installs Docker and has git.

Then we use setup_remote_docker, because we should think twice before using Docker inside Docker. As a result, all the Docker commands such as docker build and docker push will be safely executed in this new environment.

4. Workflows - Here we defined a workflow with 2 jobs: Test and Deploy. Test will occur each time a change is made to the application in any given branch, while Deploy will always wait for the Test job to finish and will only run in the master branch.

Wrapping it up

Although we finished the Implementation Roadmap, the implementation as a whole is far from done. Now we have a fully automated process that tests our application and pushes it to a container repository without any human interaction. It’s worth pointing out that some important aspects such as security were neglected, but this is only one part of the implementation’s MVP (Continuous Delivery is coming soon).

References

Originally published on Medium.