mesos: cluster management system

Cluster Management Systemmesos.apache.org

• to understand target problem domain.

• to know the operating system for the Datacenter.

• to gain an architectural overview of a Mesos cluster.

• to learn how to run applications in a Mesos cluster.

• to understand the Service Discovery & Load Balancing

• to get an overview on Mesos Frameworks.

http://mesos.apache.org Agenda

Your Expectations

Mesos is the name of the cluster management framework and the Apache Project.

Mesosphere (Mesosphere, Inc.) is the company which is commercializing the open source project Apache Mesos, and provides consulting services.

http://mesos.apache.org Mesos

Mesos vs Mesosphere

http://mesos.apache.org Problem Domain

Static Partitioning

http://mesos.apache.org Problem Domain

Static partitioning Dynamic resource sharing

Resource Management

http://mesos.apache.org Problem Domain

Unified API

• Realignment of the partitions in the clusters means un-deploy/redeploy of the running software on the instances and requires error-prone and tedious manual effort.

• It is hard to predict the workload and fluctuations of the load makes an efficient resource utilisation impossible.

• Changes in infrastructure may cause changes in the applications so the coupling between Developers and Devops does remain strong.

http://mesos.apache.org Problem Domain

Problem Description

Datacenter as a Computer

• Process scheduling.

• Fairly resource (on the same CPU, Memory, Disk, etc) sharing.

• Package managers (aptitude, yum, etc.)

• Unified API between apps and hardware.

http://mesos.apache.org Datacenter OS

DC as a Computer

• Cluster-wide application scheduling.

• Fairly resource sharing in a distributed environment.

• Docker as package manager tool.

• Unified API between apps and hardware.

Operating Systems Mesos as a DC OS

• orchestrates the infrastructure so that it can be extended by adding new hardware or shrunk to downscale, dynamically.

• abstracts the underlying infrastructure while providing a unified API to the applications.

• employs the LXC (Linux Containers) interface to maximize resource utilisation via operating system level virtualization (like Docker or using Docker).

http://mesos.apache.org Problem Domain

Apache Mesos

http://mesos.apache.org Architecture

Architecture

Coordination over Zookeeper

Mesos CLI

Ports are ranged resources

http://mesos.apache.org Datacenter OS

How does it work ?

Mesos Framework = Scheduler + Executor

http://mesos.apache.org Datacenter OS

How does it work ?

Resource negotiation between components

• You can develop your own custom Frameworks for Mesos.

• or you can deploy open source Mesos Frameworks like Spark, Hadoop, etc.

• or, getting even better, you can deploy Docker containers.

• Mesos can orchestrate Docker containers in a Mesos cluster, as well.

http://mesos.apache.org Frameworks

Applications on Mesos

http://mesos.apache.org Frameworks

Mesos Frameworks

OSS Frameworks as Mesos Frameworks

http://mesos.apache.org Architecture

Marathon

http://mesos.apache.org Marathon

Marathon

• Marathon is the PaaS on Apache Mesos.

• Marathon provides REST API to automate your workflows.

• Marathon UI allows Dev/Ops to manage the frameworks by point-n-click.

http://mesos.apache.org Architecture

Load Balancing thru HAProxy

User Requests

http://mesos.apache.org Architecture

Service Discovery

User Requests

Mesos DNS provides its clients a

REST endpoint for Service Discovery

http://mesos.apache.org Hands-on

Hands-on

• Introduction: Mesos Dashboard

• Introduction: Marathon UI and REST API.

• Case Study: Elastic Jenkins on Mesos.

http://mesos.apache.org End

QA

any Questions?