Table of contents

• Introduction
  • LendARead's Goal
  • Architecture
    • Development enviroment
    • QA enviroment
    • Production enviroment
  • CI-CD Setup
  • CI-CD Environments:
  • CI-CD Flow
  • CI-CD Secrets:

Introduction

This is a project that seeks to provide a continuos integration and deployment solution for a full-stack web application, as well as a AWS architecture defined in terraform.

The CI-CD pipeline tests, deploys and notifies the different type of developers through Slack and Email when approval is required to move a build from development to quality assurance (qa), and from qa to production. More details regarding the CI-CD flow can be found at the end of the article.

LendARead's Goal

The goal of the web application is to create a community of readers, where all of them can upload their books and request loans from each other. The aim is to allow readers to access books that would not normally be easily accessible.

For example, let's imagine the user Marco, who enjoys reading philosophy in multiple languages. Thanks to other readers with German heritage (for example), he can borrow a philosophy book from a German author that he could not have read by going to a library in Argentina.

The Problem

The deployment flow before the CI/CD pipeline involved 6+ steps noted in Figure 0. It was hard to reproduce, it needed domain knowledge, it was risky in that missing a step (for instance dumping the database) could result in big trouble, and it was time consuming.

Moreover, this initial workflow was designed for the initial version of the LendARead webapp i.e. a single server running Tomcat and serving a MVC webapp. Because we had already migrated this architecture to a three-tier AWS one we decided that we could harness it to develop a CI/CD pipeline. (The CI/CD could be done for the original MVC architecture, but that would be re-tracing our steps, moreover, the current architecture allowed for a more robust pipeline)

Architecture

The LendARead architecture is deployed through terraform as part of a 3-tier architecture. In addition, the current project builds three versions: dev, qa, prod thus providing a devops approach to the deployment of LendARead.

The diagram shows the three different versions of the same architecture design.

Development enviroment

The dev enviroment optimizes costs in terms of high-availability i.e. we do not need to have a highly available system by keeping the postgres in two availability zones (AZs) and two instances of the REST backend in ECS.

QA enviroment

The qa enviroment optimizes costs in terms of high-availability (only a single instance of the database is present on an AZ), but it does have two instances of the ECS task because in this stage the overall resilience of the system to requests is usually tested.

Production enviroment

The prod enviroment is highly available both in terms of the backend docker instances in the ECS and the database replicas in two different AZ.

CI-CD Setup

To begin the pipeline one should first initialize the s3 bucket and dynamo table to hold both the terraform state as well as a lock to prevent incosistencies in the terraform state stored in the s3. To initialize the architecture one should push the tag init

git tag init                                                                                          
git push origin init

This will trigger a Github action that will create the s3 bucket and dynamo table. After this is complete one can push to the main branch, this will run the terraform init and terraform apply using the state defined in the last step. Whatever changes are done to the application, or the infraestructure itself, will be updated in the deployment. The key here is that because the state is stored in a s3 only the changes to the current architecture will be made, if nothing of the infraestructure changed then only the docker image will be updated.

CI-CD Environments:

To run the demo, the repository needs 3 environments to run:

• dev
• qa

• prod

  Configure each environment with the corresponding approval permits

CI-CD Flow

To trigger a workflow for the deployment, the creation of a Release with a new tag is necessary. The tag must follow the regex: v[0-9]+.[0-9]+.[0-9]+ and follow semantic versioning.

After that the CI-CD Flow is described in Figure 5:

• A job to extract the value of the newly created tag will run.
• After that, a job that will check all the tests in the application will run. If it runs successfully, it will notify the authors (through Email and Slack, see Secrets section) and continue with the deployment
• If the tests fail, the tag will be deleted from the repo. This way, all tags in the repo will contain a valid version
• Once the tests finished, a new job that will create the docker image and push it to the registry (currently setup to use AWS ECR, see Secrets)
• After the image is created approval is needed to promote the build to the next enviroment, this is sent through Slack and Email although it can also be approved directly from Github. After the approval, the ECS task's docker image in the corresponding enviroment will be updated.

An important aspect of the pipeline is requiring approval for each stage, Figure 6 shows the approvals made by the admin of the project (which are recorded to have accountability throughout the process).

CI-CD Secrets:

• AWS_ACCESS_KEY_ID: Key ID for accessing the AWS account
• AWS_REGION: Region where the application is deployed
• AWS_SECRET_ACCESS_KEY: Value for accessing the AWS account
• AWS_SESSION_TOKEN: AWS session token
• DYNAMODB_TERRAFORM_LOCK_NAME: Lock to prevent race conditions on the Terraform state stored in the S3 bucket
• ECR_URL: ECR URL once deployed
• S3_TERRAFORM_STATE_NAME: Bucket for storing the Terraform state
• SLACK_WEBHOOK_URL: Slack webhook for notifications
• SMTP_SERVER: SMTP server URL
• SMTP_USERNAME: Username for the SMTP server
• SMTP_PASSWORD: Password for the SMTP server
• SMTP_PORT: Port for the SMTP server