User login
Learn how to use Qlustar!

Qlustar Installation Guide

This document describes the necessary steps to install the Qlustar operating system. The Qlustar installer has been kept simple, yet it should include all necessary installation options a HPC cluster head-node needs. A setup with redundant (HA) head-nodes is supported, but such a configuration is a lot more complex and beyond the scope of this guide. We'll provide a separate manual for such installations, but note that this will require (a) a carefully chosen hardware configuration, (b) quite some manual configuration and (c) a lot of testing to ensure reliable operation.


To install the Qlustar OS you need:

  • A DVD or a USB flash-drive (minimum size 2GB) loaded with the Qlustar installer
  • A 64bit x86 server/PC (or virtual machine) with
    • at least two network adapters
    • at least one disk with a minimum size of 160GB
    • optionally a second (or more) disk(s) with a minimum size of 160GB
    • CPU supporting virtualization (for virtual front-end and demo nodes)
    • Internet connection

Note that the only virtual machine (VM) we test the installation process on is KVM. Hence, this is the only VM officially supported by Qlustar. Furthermore, nesting of VMs is currently not supported by our kernel. This means that you won't be able to start a virtual front-end node or a virtual demo-cluster directly from within a head-node that is itself running as a VM.

Note also, that the Qlustar kernel is configured for cluster server hardware. Hence, it's possible, that drivers for some network or storage adapters on Laptops or other non-server PCs are missing. If you run into such problems, please post it in the installation forum. Qlustar also has no support for secure boot (UEFI). If your hardware uses UEFI by default, you need to configure your BIOS to allow so-called legacy boot.

Loading the installer to a USB flash-drive

Starting form Qlustar 9, we provide an ISO install image, that can be burned onto a DVD or be loaded onto a USB flash-drive. This section explains the USB method. If you already have an up-to-date Qlustar USB drive you can skip this step. But make sure that you have the latest installer to be found at Important: After downloading the installer image, verify its integrity, by executing the following command:

sha256sum qlustar-installer.iso.gz

and compare the such-generated checksum with the value that is published at the above download page. If the checksums are not identical, do not proceed with the installation, since that means the installer is corrupt (either due to hardware errors or a malicious attack). In this case, you should download the image once more. If the problem persists, also after downloading from a different computer, please post the incidence in the support forum.

The instructions in this guide only apply to the latest version. With the downloaded disk image you can easily create a bootable drive under Linux as follows (note that the USB drive should have a minimum size of 2GB): Given that the downloaded installer image file is named qlustar-installer.iso.gz and that the device name of the drive is /dev/sdb, the following command (executed as root) will load the image to the flash-drive:

gunzip -c qlustar-installer.iso.gz | dd of=/dev/sdb

Depending on the speed of your flash-drive, this can take up to 20 minutes. Attention: The above command will destroy all data on the USB drive or to be more precise, it will destroy the data of the device named /dev/sdb! So make sure to find out the correct device name of the drive. This can be achieved e.g. by executing the command

tail -f /var/log/kern.log

at the shell prompt before you insert the drive. Just after you have inserted it, a message like [sdb] Attached SCSI removable disk should appear. In this case your USB drive would have the device name /dev/sdb. If the above command finished successfully, you can use the drive to install the Qlustar OS on the target machine. This is described in the next paragraph.

Booting from the USB drive

Insert the drive in the server that you want to install the Qlustar OS on and boot. Make sure to select the USB drive as the boot device, either by entering the BIOS and changing the boot order or by entering a boot menu and then selecting the drive. On some BIOSes the latter is achieved by pressing the key F7 or F12.

Installation process

If the server boots from your drive it will present a menu with the following entries:

Qlustar Installation
Qlustar Installation (ttyS0)
Qlustar Installation (ttyS1)
Qlustar Installation (ttyS2)
Boot OS of first partition on first disk

of which the first is highlighted. If you install via a serial console (e.g. IPMI serial-over-LAN) select the correct entry (ttyS0/1/2) depending on your hardware setup. Press enter to start the installation. After the kernel was loaded you will see the kernel messages on the screen followed by some messages of the installation system. Finally the installer will present a welcome screen and you can start configuring the system by pressing enter.

Localization settings

In the first screen you can select the timezone and keyboard mapping. After you have made your choices, selected OK and pressed enter, the keyboard mapping will be configured according your choice. If you have selected a wrong mapping the keyboard will not function properly. If this is still the case after changing the mapping to the correct value, you will have to reboot the server and start the installation from scratch.

Disk Configuration

In this screen you can select the disk(s) to install Qlustar on. If there is more then one disk in the system you can select multiple disks and create a RAID array (different RAID levels can be chosen depending on the number of the disks available). The chosen disk (or created RAID device) will then be used as a LVM physical volume to create a LVM volume group (the name of which can be specified). Four ext4 filesystems (root, var, tmp, apps) and swap space will automatically be created on equally named LVM logical volumes within this volume group.

Home Filesystem

A separate file system is used for the home directories. If there are additional unused disks in the system you are offered the choice to use them in the next screen. If not, you can use the previously configured volume group for this additional file system. If you choose "Other" then no configuration for a home filesystem will be done. In this case, you won't be able to add any cluster users without further manual steps.

Home Filesystem on separate Disks

This screen only appears if you have selected "Separate disks" in the previous step. You can choose the disks and possibly the raid level to use for the home file system similar to the previous dialog.

Network Settings (1)

In this screen, you configure the IP addresses for the cluster and public network. The installer has some logic to check for valid network configuration parameters as well as suggesting reasonable values. The chosen number of compute nodes NK in the cluster determines the suggested cluster network address and other parameters. It doesn't have to be accurate, just an approximate number will do.

Note that Qlustar requires cluster-internal IP networks to be in a non-public range. If you're unhappy with the range suggested, you can change NK to another number (e.g. NK=1000 suggests 10.x.x.x, NK=200 172.16.x.x and NK=16 192.168.x.x).

You can also set the hostname (associated with the public IP) for the new head-node. The name of the cluster-internal interface is beosrv-c. Note that two network adapters are necessary for proper operation of your cluster. It's important that you correctly identify external (public/access to the Internet) and internal (cluster) network adapters. If you select DHCP as the configuration method for the public interface, make sure to add its IP address later to the NIS and ssh related header configurations in the graphical QluMan management interface. If you don't have any special reason for changing the suggested address/netmask of the internal adapter, we recommend to leave them unchanged.

Postfix is running on the head-node as the central mail transfer agent. You need to specify a mail relay host (either by name if resolvable via DNS or by IP) in your LAN, where mail of the cluster will be accepted and redistributed. Root mails generated on the cluster, will be redirected to a list of e-mail addresses, that you can enter next (multiple addresses should be separated by a comma).

Network Settings (2)

Here you have the possibility to configure optional Infiniband (IB) and/or IPMI network parameters, if you have the corresponding hardware in your cluster.

If there are nodes with IB in the cluster, you first need to set the option whether the head-node itself also has IB hardware or not. If it doesn't, you only need to choose the IB TCP/IP network address parameters (defaults are again recommended). Otherwise, you have the additional choice to run OpenSM (IB subnet manager) on the head-node. Note, that at least one node (or switch) in an IB fabric has to run a subnet manager. Given the value of the IB network address, the host part of the IB adapters IP address will be set from the host part of the Ethernet address of the node in the cluster network. Example: IP cluster network - IP IB network

As with IB, if some of the nodes of your cluster have an IPMI adapter, that is connected to a switch, you can enter the network address parameters that will be used to configure its IP address. Again, the host part of the IPMI adapters IP will be derived from its cluster network IP (see above). Additional, host specific IPMI configuration options can be set in QluMan. Please note, that the configuration of IPMI adapters is hardware dependent. The method, we use, works on most systems, but we can't give any guarantee, that it will work on yours. Please contact us in case of problems.

Additional Settings

For the sake of system stability and performance, it is usually a good idea to separate user from system activities as much as possible. To achieve this, we suggest to run only system-related tasks directly on the cluster's head-node(s) and have physical or virtual front-end (FE) nodes for user access/activity. In this dialog you can choose to setup a virtual FE node as a virtual machine running on the head-node. If you decide not to do so, we advise to assign a physical machine as a FE node at a later stage. The suggestions for the network parameters of the FE VM are derived from the head-nodes values. Again, if you select DHCP as the configuration method for the public interface, make sure to add its IP address later to the NIS and ssh related header configurations in the graphical QluMan management interface. The name of the FEs cluster-internal interface would be login-c.

You also have the option to create a virtual demo-cluster on the head-node, provided the latter has enough main memory. When choosing the number of demo-nodes, keep in mind, that each one of them will be configured with 1GB of main memory. Even if configured, the VMs will not be started automatically though, so the head-node will be able to boot, even if an insane number of demo VMs are chosen.

Please note, that if you choose neither the option for the virtual FE node, nor the virtual demo-cluster, the necessary network pre-configurations (bridge setup, etc.) for starting VMs will not be done. This means, that you'll have to do a number of non-trivial manual steps after installation, in case you find out, that you need support for VMs later on.

If you need a proxy to access the WWW, you can enter the relevant proxy parameters here. To speed up the installation of additional packages later on, you could copy the software packages on the USB drive to the hard-drive of the installed machine. But note that the advantage of this fades away over time, once more and more packages will have newer versions/updates and hence will have to be downloaded from the Qlustar repository on the Internet anyway. To use a serial device for console access, select the next option and enter the device name. Additional kernel parameters for the head-node can be specified next (typically, none are needed). Finally type in the password for the root user twice.

Edge Platform Selection

Starting from version 8.1.1 Qlustar supports multiple so-called Edge Platforms. Currently the choices are Ubuntu/Precise and Debian/Wheezy. Please select your favorite platforms. Multiple choices are possible. At least one must be selected.

In essence, this option corresponds to the choice of what Linux distribution you (or your users, if you're an administrator) like to work with/develop for. Note that you can always add support for a previously unselected edge platform later on by creating corresponding chroots and Qlustar images using QluMan.

Technically, choosing a specific edge platform (e.g. Debian/Wheezy) will cause Qlustar images based on Debian/Wheezy to be created along with an associated chroot. The latter will hold a complete Debian/Wheezy installation in which you can later add packages. By assigning such an image to some nodes later on, you will have a full-featured Debian Wheezy installation available on these nodes, once they have booted over the network. Since multiple edge platforms are available, it is fully supported and easily achieved to have subsets of nodes running Ubuntu/Precise and Debian/Wheezy simultaneously in the same cluster.

Package Bundle Selection

The default installation will only install a minimal set of basic packages on the head-node of your cluster. Here you can select additional package bundles to install. Among others, you have the choice of two workload managers: Slurm and Torque. If you don't have a strong preference, we suggest to use Slurm. In particular, if you intend to follow all the instructions in the First Steps Guide, Slurm is a requirement and is also sufficient for a basic evaluation setup.

Beware, that you should choose Lustre as an option here, only if the head-node itself is supposed to run Lustre services (MGS/MDTs/OSTs). If you merely intend to try out Lustre, it is recommended to configure an image with Lustre support in QluMan, and assign it to some nodes of the virtual demo-cluster.

Note, that apart from Nvidia Cuda software, the packages selected here will only be installed on the head-node. Selecting and generating boot-image modules as well as the process of general package installation for compute/storage nodes booting from the network must be done elsewhere. This is discussed in the First Steps and the QluMan Guide.

Reviewing Installation Settings

Before the actual installation will begin, the final screen shows a summary of the settings, that have been selected/will be used (you will have to move back with the left arrow key in order to scroll down in the summary screen). If there are any parameters, you want to change, just go back and make the required modifications.

Actual Installation

The actual installation process will take a few minutes to complete. Press enter when requested at the end of the process. Remove the USB drive and press enter again to reboot. Please note that the head-node has a text-only console. An X server (graphical interface) is not installed, in order to keep resource consumption low and eliminate package overhead.

glqxz9283 sfy39587stf02 mnesdcuix8