Getting started

From ControlTier

After reading this Getting Started page, you should have the general pattern to using ControlTier to define and manage your automation. It begins with the installation of the ControlTier server and client across your hosts. Then you create a project where you will manage your work. Each step of automation is specified in a new or existing script. Each script can then be exposed as a defined command in a type. Automation steps are combined into a workflow. Any script or type-defined command can be executed across the network via the command dispatcher via the GUI or CLI. Dispatching flags control how the commands are executed across the network.

Requires Version 3.4.9 (help?)

Download and Install

• Download the desired package type (e.g., ZIP installer or RPM installer)
• Set the CTIER_ROOT environment variable
• Install the ControlTier Server package to the chosen ControlTier server.
• Install the Client package on each additional client host. (optional)
  • Be sure to run ctl-setup for each client host, to configure the server's hostname.
• Once the server is running access the server landing page and login.
• Set your environment

If ControlTier is working correctly you should now be able to type ctl --version in the terminal window and see output below:

ctl --version	
	.
	.
	.
[CTL version 1.4.9 (20091117)]	

Create a project

A ControlTier "project" provides a space to manage related automation activities.

The ctl-depot shell command is used to create the space (aka "depot") for each project.

Run the ctl-depot command and specify a project name, here we use "demo":

ctl-depot -a create -p demo

Do this first on the server, and then on each of the client hosts.

This will create the Project in the server, and register each client as a Node in the Project's Resource Model.

Node metadata

Each ControlTier client is called a "Node" within the Resource Model. You may typically have one Node per physical or virtual "Host" computer.

When the client hosts were setup for the new project, a bit of metadata about each host was registered to the server. You can view information about the registered nodes via the ctl-exec -v command. Use the -p option to specify the Project name:

 $ ctl-exec -p demo -v	
 strongbad:
    hostname: strongbad
    os-arch: x86_64
    os-family: unix
    os-name: Mac OS X
    os-version: 10.6.2
    tags: []
    ---- Attributes ---- 

Node metadata can be extended but one of the most useful pieces of metadata is the "tags" property. A tag is a text label that you give to the Node, perhaps denoting a classification, a role the node plays in the environment, or group membership. The output above shows the strongbad node currently has an empty tags property: tags: []

It is useful to start thinking about node tagging for your own nodes because you will use it later when specifying #Command_dispatcher_options to drive distributed command execution.

A node's tags can be maintained with the graphical interface, with the XML definition or via the command line.

Here we use a command called "tag-add" to add two tags to the "strongbad" node:

 $ ctl -p demo -m modelutil -c tag-add -- -type Node -name strongbad -tags admin,simple

Running ctl-exec -v again shows the tags property now contains "admin,simple":

 $ ctl-exec -p demo -v	
 strongbad:
    hostname: strongbad
    os-arch: x86_64
    os-family: unix
    os-name: Mac OS X
    os-version: 10.6.2
    tags: [admin, simple]
    ---- Attributes ---- 

The tag-remove command can remove a tag. Here the "admin" tag is removed:

 $ ctl -p demo -m modelutil -c tag-remove -- -type Node -name strongbad -tags admin

Now the node has only one tag, "simple":

 $ ctl-exec -p demo -v	
 strongbad:
    hostname: strongbad
    os-arch: x86_64
    os-family: unix
    os-name: Mac OS X
    os-version: 10.6.2
    tags: [simple]
    ---- Attributes ----

Dispatching commands

The ControlTier Command dispatcher provides a means to execute scripts or commands across the network either through command line or graphical interface. Dispatcher options control how the command or scripts will execute and lets you coordinate actions across hosts. The Scripters cookbook page offers several examples on how to take advantage of various features and useful utilities that cover common scripting use cases.

Run shell scripts

An "ad hoc" command is system script or executable that you might type at an interactive terminal. Ad hoc commands can be executed via the ctl-exec shell command. Use ctl-exec to execute individual commands or shell script files. Using the command dispatcher to run ad hoc commands and scripts is a great way to avoid the disadvantages of one off looping SSH scripts.

Chances are, you have your routine steps already in scripts maintained from a central administrative host. For example, assume there is a script on the server called "listening.sh" that checks if the 8080 port is in the listen state. If it is not in the listen state, the script exits with exit code 1.

#!/bin/sh
netstat -an | grep 8080 | grep -q LISTEN
if [ "$?" != 0 ]; then
echo "not listening on 8080"
exit 1;
fi
echo  listening port=8080, host=`hostname`;

You can execute the "listening.sh" script across all the hosts via the ctl-exec command line tool:

ctl-exec -s listening.sh

You can also run an ad hoc command or a script with Jobcenter by exposing it as a "job". Jobs can either be created directly with the GUI, or with a XML file that gets loaded into Jobcenter.

The main Jobcenter page lists all of the defined jobs:

You can then run the Job to dispatch the script to a set of target nodes:

Run defined commands

Defined commands are procedures formally defined in a Type and that are callable by the command dispatcher. A Type represents a control module comprised of one or more defined commands, each of which can have a set of formally defined command options. ControlTier provides a range of core types organized into several layers - base, utility, deployment. These types provide a variety of useful commands.

Packaging your own procedures as commands allows you to use ControlTier to easily distribute them to your nodes as modules, compose them into workflows, and provide clear input options.

Defined commands have a uniform calling interface and are a standard way to distribute and deploy your automation procedures. The command dispatcher also has a command-line listing feature that can help users to learn what commands are defined and even show their supported options. Defined commands ultimately form the basis of your own automation framework, where one command can call another either directly or via workflows.

Here's an example that shows off executing a defined command called "listening" declared in the the netutil type. The listening command checks if the specified port is listening and accepting connections. Any type-defined commands can be invoked using the ctl command line tool.

ctl -m netutil -c listening -- -port 8080

Defined commands can also be executed via the GUI as a Job. The job definition sample below shows the same module and command chosen, and port 8080 entered as an option:

Save the job with a name to define it.

Dispatcher options

The command dispatcher supports a number of flags that can control how commands are executed across nodes. The command line tools, ctl, ctl-exec as well as job definitions support the notion of command dispatching flags.

• The include and exclude flags filter the set of nodes the command will execute. The ctl and ctl-exec commands do this via "-I" and "-X" flags.

Below, the "-I os-name=Linux" option specifies to include all nodes that have the operating system name "Linux". The "-X centos54" excludes the node named "centos54":

ctl -I os-name=Linux -X centos54 -m netutil -c listening -- -port 8080

Here's an example that runs the who am i unix command on all hosts that tagged "simple":

ctl-exec -I tags=simple -- who am i

• The thread count and keepgoing and retry flags control concurrency and errors via "-C" and "-K/-F" options.

Here's an example that runs the "listening.sh" script across all unix hosts. The "-C 3" option specifies to run three threads of execution. The "-K -F /tmp/failednodes" will specify to keep going if a command fails on a node but to save the list of failed nodes in the "/tmp/failednodes" file. If an error occurs, the user can later repeat the same command line but only nodes contained in that file will be targeted.

ctl-exec -I os-family=unix -C 3 -K -F /tmp/failednodes -s listening.sh

• The level of logging is controlled via the "-l" flag

Here only messages of "warning" level or higher are displayed to the console. Choose from: debug|verbose|info|warning|error

ctl -l warning -m netutil -c listening -- -port 80

• Job definitions can also specify dispatch and node filtering. When defined with XML, use the dispatch and nodefilters tags.
• When defined via the GUI, enter the filter option values in the form.

This example selects all nodes with the operating system name "Linux" but excludes the one named "strongbad":

Create a controller type

In ControlTier, multi-step processes are easily defined as a "workflow". A workflow is a command defined as a sequence of one or more commands. Like any command, workflows are defined in a type. Types can be developed in XML with the help of ProjectBuilder or they can be defined graphically using the Graphical type editor.

You can run ProjectBuilder commands on any node, as it can upload type definitions to the server. For the sake of convenience of this document, assume the commands are run on the server host.

• The ProjectBuilder create-type command will create a new type definition, generating a directory of source files:

ctl -m ProjectBuilder -c create-type [-- options]

• The build-type command processes the type definition and turns it into an executable artifact.

ctl -m ProjectBuilder -c build-type [-- options]

A SimpleApp controller example

Imagine the goal is to create a controller type and set of defined commands to manage the deployment process for a fictitious application named SimpleApp distributed over several hosts. The plan is to run a workflow that will execute each step of the deployment cycle for the SimpleApp application. This example will show how to define a new type that defines a workflow command named "deploy" along with a command to manage each step of the deploy process: stop, install, configure, start, listening.

Control scripts

The ProjectBuilder create-type includes a feature that will bootstrap command definitions for a new type based on a directory of scripts. A command name corresponding to the script file's name (minus the extension) will be defined for each file found inside the chosen script directory.

For the SimpleApp, imagine a set of scripts exist in a directory named "~/tmp/SimpleApp/scripts". The directory contains a separate script for each step of the process (i.e., reusable building blocks).

[ctier@training src]$ ls ~/tmp/SimpleApp/scripts/
configure.sh  install.sh  listening.sh  start.sh  stop.sh

Of course for a real application, each script implementation would do the actual procedure for each step, but for purposes to help get started, each script just contains code comments that suggest their intent:

configure

Customizes the configuration for the client node

File listing: configure.sh

#!/bin/sh
echo "Running configure ..."
# if [ -d /usr/local/tomcat ]; then
#   echo tomcat not installed; exit 1;
# fi;
# sed 's/8080/28080/g' /usr/local/tomcat/conf/server.xml > /tmp/server.xml
# mv /tmp/server.xml /usr/local/tomcat/conf/server.xml

install

Gets, extracts the tomcat packages

File listing: install.sh

#!/bin/sh
echo "Running install ..."
#
# Download and extract the SimpleApp packages to the correct installation directory.
#

listening

Check if SimpleApp is listening on its port

File listing: listening.sh

#!/bin/sh
echo "Running listening ..."
# if [ -d /usr/local/tomcat ]; then
#   echo tomcat not installed; exit 1;
# fi;
# netstat -an | grep 8080 | grep LISTENING

start

Runs the SimpleApp startup script

File listing: start.sh

#!/bin/sh
echo "Running start ..."
# if [ -d /usr/local/tomcat ]; then
#   echo tomcat not installed; exit 1;
# fi;
# /usr/local/tomcat/bin/startup.sh

stop

runs the SimpleApp shutdown script

File listing: stop.sh

#!/bin/sh
echo "Running stop ..."
# if [ -d /usr/local/tomcat ]; then
#   /usr/local/tomcat/bin/shutdown.sh
#   exit $?
# fi;

Create the type

Run the ProjectBuilder create-type command using the "-scriptdir" and "-initialize" options to automatically generate command definitions for each of the scripts.

[ctier@training tmp]$ cd $CTL_BASE
[ctier@training ctl]$ mkdir src
[ctier@training ctl]$ cd src
[ctier@training src]$ ctl -m ProjectBuilder -c create-type -- -type SimpleAppCtl -scriptdir ~/tmp/SimpleApp/scripts -initialize
Description of type: 
Simple application module
Choose a supertype:  ([Module], Package, Builder, Deployment, Service, Site)
 
Directory where module files will be created:  [/opt/ctier/ctl/src]
 
Creating module definition files in directory: /opt/ctier/ctl/src ...
Creating directory structure...
Created dir: /opt/ctier/ctl/src/modules/SimpleAppCtl
Created dir: /opt/ctier/ctl/src/modules/SimpleAppCtl/bin
Created dir: /opt/ctier/ctl/src/modules/SimpleAppCtl/commands
Created dir: /opt/ctier/ctl/src/modules/SimpleAppCtl/objects
Created dir: /opt/ctier/ctl/src/modules/SimpleAppCtl/templates
Initializing type module from template dir: /opt/ctier/ctl/modules/ProjectBuilder/templates/boilerplate ...
Copying 2 files to /opt/ctier/ctl/src/modules/SimpleAppCtl
Define commands and attributes in this file: /opt/ctier/ctl/src/modules/SimpleAppCtl/type.xml
Matching scripts in the scripts directory...
Found 5 matching scripts. Generating commands from scripts...
Created dir: /opt/ctier/ctl/src/modules/SimpleAppCtl/scripts
Copying 1 file to /opt/ctier/ctl/src/modules/SimpleAppCtl/scripts
Copying 1 file to /opt/ctier/ctl/src/modules/SimpleAppCtl/scripts
Copying 1 file to /opt/ctier/ctl/src/modules/SimpleAppCtl/scripts
Copying 1 file to /opt/ctier/ctl/src/modules/SimpleAppCtl/scripts
Copying 1 file to /opt/ctier/ctl/src/modules/SimpleAppCtl/scripts
Copying 1 file to /opt/ctier/ctl/src/modules/SimpleAppCtl
Copying 1 file to /opt/ctier/ctl/src/modules/SimpleAppCtl
Commands have been generated for SimpleAppCtl: /opt/ctier/ctl/src/modules/SimpleAppCtl

Accept the defaults if prompted.

The command will generate a new directory structure of source files that will be later processed into an executable module that can be distributed to the ControlTier clients.

Directory convention

The type source directory structure created by create-type follows a convention:

• SimpleAppCtl - top-level directory for type sources
  • type.xml - The XML file that defines the type and its commands and their options
  • commands - Generated dispatcher code for each defined command
  • lib - optional resource files
  • scripts - Your scripts

Define the workflow

A workflow command defines a sequence of commands to execute. The goal is here is to define a workflow that manages the deployment process for the SimpleApp. The example below describes a workflow command named "deploy" that calls each of the commands defined during the create type step in the desired order. Cut and paste the following into your SimpleAppCtl type definition (eg, CTL_BASE/src/modules/SimpleAppCtl/type.xml).

<command name="deploy" description="run the app deploy cycle"		
	       command-type="WorkflowCommand" is-static="true"
	       error-handler-type="FAIL">
  <workflow threadcount="1">
    <command name="stop"/>
    <command name="install"/>
    <command name="configure"/>
    <command name="start"/>
    <command name="listening"/>
  </workflow>
  <opts>
      <opt parameter="argline" required="false" property="opts.argline" default="simple" type="string" 
    description="Arguments passed to the script"/>
  </opts>
</command>

Workflows also support error handling and notification. Reference: Type-v10.xml#workflow

Build the type

Use the ProjectBuilder build-type command to process the type sources and upload it to the server as a deployable control module.

[ctier@training src]$ pwd
/opt/ctier/ctl/src
[ctier@training src]$ ctl -m ProjectBuilder -c build-type -- -type SimpleAppCtl -upload -deploy
Base directory where module source files reside [/opt/ctier/ctl/src]
 
Target directory where build files are generated [/opt/ctier/ctl/target]
 
Building type using the buildmodule.xml via classloader
converting type.xml for module: SimpleAppCtl
generating handlers...
packaging module: SimpleAppCtl
Copying 1 file to /opt/ctier/ctl/src/modules/SimpleAppCtl
Copying 1 file to /opt/ctier/ctl/src/modules/SimpleAppCtl
Deleting: /opt/ctier/ctl/src/modules/SimpleAppCtl/module.properties.temp
Building jar: /opt/ctier/ctl/target/modules/SimpleAppCtl-1.jar
Uploading built module to server ...
processing files in directory: /opt/ctier/ctl/target/modules
scanning for files matching pattern: (SimpleAppCtl)-([0-9]+)\.jar
Uploading jar: /opt/ctier/ctl/target/modules/SimpleAppCtl-1.jar to server: 'training' ...
Installing new build of "SimpleAppCtl" module from server ...
Getting: http://training:8080/jackrabbit/repository/controltier/projects/demo/artifacts/modules/SimpleAppCtl-head.jar
To: /opt/ctier/ctl/projects/demo/artifacts/modules/SimpleAppCtl-head.jar
Created dir: /opt/ctier/ctl/projects/demo/modules/SimpleAppCtl
Expanding: /opt/ctier/ctl/projects/demo/artifacts/modules/SimpleAppCtl-head.jar into /opt/ctier/ctl/projects/demo/modules/SimpleAppCtl

Accept the defaults if prompted. The "-upload" option takes the built artifact and stores it on the server's WebDAV repository while the "-deploy" option subsequently downloads and installs the module into the depot.

Note that you can distribute and install the new SimpleAppCtl controller type on other nodes using Install-Module command. Use include/exclude dispatch flags to run the command across all hosts:

ctl -I '.*' -m Managed-Entity -c Install-Module -- -module SimpleAppCtl

Run commands

With the SimpleAppCtl type built and installed on the client hosts, it is possible to run the newly defined commands.

The general usage for running commands in the new SimpleAppCtl type is:

[ctier@training src]$ ctl -p demo -m SimpleAppCtl -c deploy
Start: "run the app deploy cycle" commands: stop,install,configure,start,listening
begin workflow command (1/5) -> "stop -argline simple" ...
Running stop ...
end workflow command (1/5) -> "stop -argline simple"
begin workflow command (2/5) -> "install -argline simple" ...
Running install ...
end workflow command (2/5) -> "install -argline simple"
begin workflow command (3/5) -> "configure -argline simple" ...
Running configure ...
end workflow command (3/5) -> "configure -argline simple"
begin workflow command (4/5) -> "start -argline simple" ...
Running start ...
end workflow command (4/5) -> "start -argline simple"
begin workflow command (5/5) -> "listening -argline simple" ...
Running listening ...
end workflow command (5/5) -> "listening -argline simple"
[command.timer.demo.SimpleAppCtl.deploy: 1.523 sec]
Workflow completed. execution time: 1.523 sec

Here's how to run the "deploy" workflow command on all nodes tagged "simple":

ctl -I tags=simple -m SimpleAppCtl -c deploy 
Start: "run the app deploy cycle" commands: stop,install,configure,start,listening
begin workflow command (1/5) -> "stop -argline simple" ...
end workflow command (1/5) -> "stop -argline simple"
begin workflow command (2/5) -> "install -argline simple" ...
end workflow command (2/5) -> "install -argline simple"
begin workflow command (3/5) -> "configure -argline simple" ...
end workflow command (3/5) -> "configure -argline simple"
begin workflow command (4/5) -> "start -argline simple" ...
end workflow command (4/5) -> "start -argline simple"
begin workflow command (5/5) -> "listening -argline simple" ...
end workflow command (5/5) -> "listening -argline simple"
[command.timer.demo.SimpleAppCtl.deploy: 5.234 sec]
Workflow completed. execution time: 5.234 sec

From the output you can see each of the steps was run in the order specified.

Define job for the workflow

The SimpleAppCtl "deploy" command can also be run as a job. Save the following job.xml definition into a file:

<joblist>
  <job>
    <name>deploy</name>
    <description></description>
    <additional></additional>
    <loglevel>INFO</loglevel>
    <group>SimpleAppCtl</group>
    <context>
      <depot>demo</depot>
      <type>SimpleAppCtl</type>
      <command>deploy</command>
	  <options>
	     <option name="argline" value="simple"/>
      </options>
    </context>
 
    <dispatch>
      <threadcount>1</threadcount>
      <keepgoing>false</keepgoing>
    </dispatch>
  </job>
</joblist>

Load the job into the server with load-jobs:

ctl -m ProjectBuilder -c load-jobs -- -file job.xml

Go to the GUI and run the job.

The result will look something like the following screenshot: