Skip to main content

Mosix simulation cluster

Telecommunication Network Group MOSIX cluster

Contents

Description

Currently TWO machines are part of the cluster

  • mosix1.polito.it (130.192.9.218) - Intel Xeon CPU 2.53GHz - 8 cores - 12MB of cache - 16GB of RAM

  • mosix2.polito.it (192.168.10.123) - Intel Xeon CPU 2.53GHz - 8 cores - 12MB of cache - 16GB of RAM

How to access the cluster

Since processes migration is automatically provided in MOSIX environment, only one MOSIX machine is needed to act as an interface to launch processes. That is mosix1.polito.it. To access it, it is necessary to go use YOUR ORTO login and password (if you do not have a login account, ask florin [dot] bota [at] polito [dot] it (Florin Bota)):

ssh user@mosix1.polito.it

user@mosix1:~$

How to run processes on the cluster

Single processes

If you want to execute a single process, simply put command mosrun -l -q -M -b in front of the process you wish to run:

mosrun -l -q -M -b my_process --my_parameters 1> stdout_file 2> stderr_file &

This means that the process will be inserted in a queue and executed as soon as a CPU becomes available.

Multiple processes (batches)

If you want to run a batch of processes, you MUST follow a different approach:

  1. generate the list of processes you want to run and put it in a file, e.g., my_batch_to_run.sh:

    exec ./my_process --my_parameters-1 1> stdout-file-1 2> stderr_file-1
    ...
    exec ./my_process --my_parameters-N 1> stdout-file-N 2> stderr_file-N

  2. run the list of processes with mosrun:

    mosrun -l -q -M -b -S16 my_batch_to_run.sh

If the processes inmy_batch_to_run.share going to last more than 30 minutes each, use a maximum parallelism equal to 8 for a fair CPU utilization:

mosrun -l -q -M -b -S8 my_batch_to_run.sh

If you receive the following message:

MOSRUN: Shared memory (MAP_SHARED) not supported under MOSIX. (try 'mosrun -e' or 'mosrun -E')

you must add -e after mosrun.

mosrun -l -q -M -b -e -S16 my_batch_to_run.sh

or

mosrun -e -q my_process --my_parameters 1> stdout_file 2> stderr_file &

If you prefer to run your processes with some mosrun specific parameter, please, take care of using "-q" option in order to respect the queue and read carefully this guide first, or ask Paolo Giaccone or Stefano Traverso how to use it. Processes not using the mosix queue might get killed at any time by an admin with no further notice.

How to monitor processes on the cluster

To monitor the load on machines (to be run from mosix1):

mon

It displays basic information (tty format) about resources in the local cluster.

mosps -aH

It provides information about your MOSIX processes.

mosq listall

It lists both the running jobs and the jobs waiting in the queue.

How to kill processes on the cluster

The default command to kill processes running on the cluster is moskillall. Take care that this command will send a termination signal to all of your MOSIX processes (all those that were started by the mosrun utility, including batch and queued processes).
A good point of moskillall command is that you can kill also subsets of jobs: adding the "
- J{jobid}" argument causes the signal to be sent only to your processes that were started with mosrun -J{jobid}. Here is an example:

mosrun -l -q -b -M –J20 myprog

Runs myprog with Job_ID = 20.

mosps –J20

Lists only my processes with Job_ID = 20.

moskillall –J20

Kills all my processes with Job_ID = 20.

If you want to remove a specific process from the waiting queue list, get its PID through mospsall command and remove it with mosq delete PID. Example:

mospsall

PID   USER MEM(MB) GRID PRI FROM   COMMAND

21666 user   -      NO  RUN here   mosrun -b testload

21667 user   -      NO  50  here   mosrun -b testload

21155 user   -      NO  50  mosix2 mosrun testload

mosq delete 211667

Important notes

FileSystem

All users' home folders are stored on a network hard drive which is mounted on all UNIX machines in TNG network and on mosix1.polito.it as well. Since space is limited on that hard drive, it is strictly recommended to store there only files necessary to programs (code, etc.) and to store temporary files (like output files) somewhere else. Mosix1.polito.it machine provides some local space (750 GB in total) in /home/ where you can store your local files.
There is a COMMON folder in which any user has right to create/delete/change all files. Simply create a folder there, change the ownership if you like, and work there.

cd /home
mkdir my_dir
chmod 755 my_dir
cd my_dir
touch my_file
...

I/O considerations

MOSIX programs that issue a large number of system-calls or perform intensive I/O relative to the amount of computation are expensive because those operations are emulated. When a process is running in a remote node, there is also overhead of sending those operations to the home-node over the network. Such processes will automatically be migrated back to the home-node, to eliminate the communication overhead.
System calls and I/O operations are monitored and taken into account in automatic migration considerations, tending to pull processes towards their home-nodes.  The "-c" flag tells mosrun not to take system calls and I/O operations in the migration considerations. So, to improve I/O performance use mosrun -c if the I/O phase is expected to be short.

If the I/O operation are supposed to be intensive,

  • Spread your input files across the cluster (in /tmp or /scratch).

  • Use mosrun –M flag to spread the processes home nodes across different nodes in the cluster, to balance the I/O.

Compiling  programs

Since the cluster is a pure 64bit, the kernels installed on all machines are 64bit. Therfore it is needed to compile source code as 64bit. The best is to do that on mosix1 directly.

Further information about MOSIX

MOSIX is a management system for Linux clusters and organizational grids that provides a single-system image (SSI), i.e. the equivalent of an operating system for a cluster as a whole. In a MOSIX cluster/grid there is no need to modify or to link applications with any library, to copy files or login to remote nodes, or even to assign processes to different nodes – it is all done automatically, like in an SMP.

Main features

  • Provides aspects of a single-system image:

    • Users can login on any node and do not need to know where their programs run.

    • No need to modify or link applications with special libraries.

    • No need to copy files to remote nodes.

  • Automatic resource discovery and workload distribution by process migration:

    • Load-balancing.

    • Migrating processes from slower to faster nodes and from nodes that run out of free memory.

  • Migrable sockets for direct communication between migrated processes.

  • Secure run time environment (sandbox) for guest processes.

  • Live queuing – queued jobs preserve their full generic Linux environment.

  • Batch jobs.

  • Checkpoint and recovery.

  • Tools: automatic installation and configuration scripts, on-line monitors.

Contacts

For any problem or suggestion, please contact html [dot] paolo [dot] giaccone [at] polito [dot] it (Paolo Giaccone) or html [dot] stefano [dot] traverso [at] polito [dot] it (Stefano Traverso.)