Project 2 ECS Tutorial

Contents

Introduction_ 1

ECS Features 2

Demonstration of Working application_ 5

Special Issue 1: Instance Stops Running_ 12

Special Issue 2: Instance is rebooted_ 13

Technologies Used_ 14

Non-working parts of project(if any) 15

Introduction

Objective

The objective of this project is to demonstrate the working of Amazon ECS by deploying a simple nodeJS app and scaling it.

Goal

The goal of this project is fully demonstrate the usage of ECS container service to deploy a web app but using docker containers. The tutorial aims to give insight on the simplicity of scaling a web app when using Amazon ECS.

Amazon ecs overview

Amazon EC2 Container Service (Amazon ECS) is a highly scalable, fast, container management service that makes it easy to run, stop, and manage Docker containers on a cluster of Amazon Elastic Compute Cloud (Amazon EC2) instances. Amazon ECS lets you launch and stop container-based applications with simple API calls, allows you to get the state of your cluster from a centralized service, and gives you access to many familiar Amazon EC2 features.

benefits of using ecs

Some of the benefits of using ec2 are:

_       Elastic web-scale computing

_       Completely controlled

_       No additional charges

_       Supports docker-compose style yaml files

_       Multiple web apps under same cluster

ECS Features

ECS, just like EC2, is a regional service that leverages the underlying technology of docker and makes it simpler to deploy applications that span multiple availability zones in a highly scalable manner.

A brief overview of ECS architecture is as bellows:

This diagram gives a well-defined view of how ECS feature work together.

The following sections dive into these individual elements of the Amazon ECS architecture in more detail.

CONTAINERS and images

ECS makes use of docker as core underlying technology to drive the deployment engine and mechanism using docker containers and images. A container is like a running instance of app whereas an image is immutable form of container, when its not started yet.

To deploy applications on Amazon ECS, your application components must be architected to run in containers. A Docker container is a standardized unit of software development, containing everything that your software application needs to run.

Images are typically built from a Dockerfile, a plain text file that specifies all of the components that are included in the container. These images are then stored in a registry from which they can be downloaded and run on your container instances.

Task definitions

Task definitions are textual description of the actual tasks that need to run for your webapp. You define various parameters in a YAML or JSON file about what docker images its going to use, what ports should be open, what data volumes need to be mounted etc.  To prepare your application to run on Amazon ECS, you create a task definition. The task definition is a text file in JSON format that describes one or more containers that form your application. It can be thought of as a blueprint for your application.

Clusters

Clusers are a logical grouping of EC2 instances, where you running tasks are placed by ECS. An application images are downloaded from whatever the registry specified to the EC2 instances within the cluster and containers are then started from those images.

tasks and scheduling

ECS task scheduler is responsible for placing tasks on container instances. A task is simply an instantiated, running form of task definition on a container instance.

There are several different scheduling options available. For example, you can define a service that runs and maintains a specified number of tasks simultaneously.

container agent

A container agent is a process that runs on each instance with the cluster and is responsible for sending information to ECS about our tasks. It runs in the form of a container and is responsible for sending information received from client about start/stop of tasks.

 

Demonstration of Working application

In this demo, we will show you how to deploy a scalable,load-balanced and high available  simple nodejs app. The demo will have multiple instances of nodejs app each running inside a container and load-balanced through haproxy.

The basic architecture looks like following:

The app as a whole consists of four components that will be deployed:

_       Haproxy(for load-balancing)

_       App(Simple node JS app that prints out the port it started on)

_       MondoDB as backend

The demo will first walk-through setting ECS cluster from Web UI of AWS and later how to use ECS CLI.

i) Creating a ECS Cluster:

First step in deploying any web application to create an ECS cluster from aws console. This is a fairly simple step as show in picture below:

ii) Install & Configure ECS CLI:

Now that a bare minimum cluster is created its time to use ECS CLI. ECS CLI is a command line tool that you can use with given credentials to manage you cluster and prepare to run tasks and services on it.

Installing the ECS CLI:

Configure the ECS-CLI with credentials:

ecs-cli configure --region us-west-2 --access-key <key> --secret-key <secret> --cluster cs6320

INFO[0000] Saved ECS CLI configuration for cluster (cs6320)

iii) Creating the ECS Cluster from CLI:

iv) Use a docker-compose YAML file for task definition:

Declaring tasks can be done from the Console or CLI. ECS CLI has a neat feature of supporting docker-compose , which are docker native way of declaring services. We avoided the hassle by reusing same compose based definitions as ECS CLI already supported them

The compose for our app looks like:
version: '2'

services:

    mongo:

        build: mongo

        image: hayderimran7/mongo:cs6320

        container_name: mongo

        volumes:

            - /tmp/mongo_logs:/logs

            - /tmp/mongo_data:/data

        ports:

            - "27017:27017"

    haproxy:

        build: haproxy

        image: hayderimran7/haproxy:cs6320.4

        container_name: haproxy

        volumes:

            - /tmp/haproxy_logs:/logs

        ports:

            - "80:80"

    redis:

        build: redis

        image: hayderimran7/redis:cs6320

        container_name: redis

        volumes:

            - /tmp/redis_data:/data

        ports:

            - "6379:6379"

    app1:

        build: app

        image: hayderimran7/app:cs6320

        container_name: app1

        ports:

            - "81:80"

        links:

            - haproxy

    app2:

        build: app

        image: hayderimran7/app:cs6320

        container_name: app2

        ports:

            - "82:80"

        links:

            - haproxy

    app3:

        build: app

        image: hayderimran7/app:cs6320

        container_name: app3

        ports:

            - "83:80"

        links:

            - haproxy

    app4:

        build: app

        image: hayderimran7/app:cs6320

        container_name: app4

        ports:

            - "84:80"

        links:

            - haproxy

v) Start a task and services from YAML file:

Now is the time to finally start our webapp from the compose file.

Once that’s successful, you will see the status using ecs cli.

vi) Verify cluster status and containers:

Now we verify the status of containers by running “ps” command from ecs-cli as:

Name                                          State                  Ports                            TaskDefinition

a48573dc-7961-4d9a-afb5-8b24f1aec12f/redis    RUNNING                54.202.144.143:6379->6379/tcp    ecscompose-cs6320:2

a48573dc-7961-4d9a-afb5-8b24f1aec12f/app3     RUNNING                54.202.144.143:83->80/tcp        ecscompose-cs6320:2

a48573dc-7961-4d9a-afb5-8b24f1aec12f/app2     RUNNING                54.202.144.143:82->80/tcp        ecscompose-cs6320:2

a48573dc-7961-4d9a-afb5-8b24f1aec12f/haproxy  RUNNING                54.202.144.143:80->80/tcp        ecscompose-cs6320:2

a48573dc-7961-4d9a-afb5-8b24f1aec12f/mongo    RUNNING                54.202.144.143:27017->27017/tcp  ecscompose-cs6320:2

a48573dc-7961-4d9a-afb5-8b24f1aec12f/app1     RUNNING                54.202.144.143:81->80/tcp        ecscompose-cs6320:2

vii) Verify services and task from Console:

We can also verify that all services are up and cluster status from console.

viii) Scaling to multiple web apps from ECS CLI:

Scaling a webapp to multiple instances is simply one command. Lets say we want to scale to 5 , we can then issue the command as:
ecs-cli compose  --file docker-compose.yml --project-name cs6320_1 scale 5

Resources and Links:

The tutorial was possible due to several helpful resources we found online as cited below:

[1]. ECS CLI Tutorial http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ECS_CLI_tutorial.html

Synopsis: this is the official link to how to run ecs cli.

Reliability: Its from official AWS so its reliable

[2]. Amazon ECS Overview http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html

Synopsis: this is the official link about the overview of ECS

Reliability: Its from official AWS so its reliable

[2]. Amazon ECS Overview http://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html

Synopsis: this is the official link about the overview of ECS

Reliability: Its from official AWS so its reliable

Technologies Used

We  have made use of following technologies for this project

1.1     NodeJS

1.2     Amazon ECS

1.3     Docker

1.4     Docker-compose

1.5     Mongo