QluMan Architecture
QluMan is Qlustar’s control center. Apart from the initial base installation of the head-node(s), cluster configuration, management and operation is mainly performed via the simple to use QluMan GUI based on the QT toolkit .
The QluMan GUI is multi-user as well as multi-cluster capable: Different users are allowed to work simultaneously with the GUI. Changes made by one user are updated and visible in real-time in the windows opened by all the other users. On the other hand, it is possible to manage a virtually unlimited number of clusters within a single instance of the QluMan GUI at the same time. Each cluster is shown in a tab or in a separate main window.
QluMan has a distributed architecture. It is designed for limitless scalability and efficiency without compromise. Its configuration space is backed by a MariaDB database. The central switch board is qlumand the admin service to which all other components connect at least initially. All QluMan node operations are executed via the execd component, an instance of which runs on any node. Whenever necessary and possible, QluMan’s components operate asynchronously.
Fig.1 - The QluMan architecture
QluNet: QluMan Communication Platform
Communication Reliability and Scalability
One of the most import design goals for QluMan’s communication platform is the requirement for reliable and scalable messaging including guaranteed delivery. To prevent an inconsistent cluster state, we need to make sure that communication messages between different QluMan components are not lost. We have to take into account node failures and network package loss. Furthermore, in case that communication channels (network connections) are blocked, we want to be noticed and resume message transfer as soon as the channels are functional again.
Here are some of the reasons, why these requirements are so important for cluster consistency and reliability:
- Administrators need to know, whether commands executed on some nodes or changes to the cluster configuration were successful.
- Large clusters require highly efficient and scalable communication architecture.
- Parallel execution of a command on 1000+ nodes including an efficient analysis of output/command status per node is a daunting task.
- The Internet is lossy, hence integration and communication with public clouds is a challenge. * QluMan addresses these issues by its messaging-based communication platform QluNet implemented on top of ZeroMQ including a custom-tailored mechanism for guaranteed delivery of messages.
Communication between different QluMan components uses intelligent heartbeat and message acknowledgment techniques with adaptive timeouts and resend intervals to detect failure of nodes and/or to ensure delivery of messages.
| Transport type | Reliable Transport | Scalable | Reliable Messaging | Guaranteed Delivery |
|---|---|---|---|---|
| UDP | clear | done | clear | clear |
| TCP | done | clear | clear | done |
| ZeroMQ | done | done | done | clear |
| QluMan/ZeroMQ | done | done | done | done |
Communication between different QluMan components uses intelligent heartbeat and message acknowledgment techniques with adaptive timeouts and resend intervals to detect failure of nodes and/or to ensure delivery of messages.
Fig.2 - QluMan guaranteed delivery
QluMan Communication Security
QluMan implements highest security through CurveZMQ, a ZeroMQ component based on CurveCP (invented by D.J. Bernstein). We use public/private key certificates for encryption between admind, execd and the QluMan GUI. No security-relevant communication between any two QluMan components is unencrypted!
QluMan technology provides perfect forward-security since key certificates are unique for each session. QluMan user authentification also works with public keys, so there is no /weak password problem/. In particular, QluMan communications are protected by design against replay, man-in-the-middle and hijacking attacks, an absolute requirement for secure cloud computing.
Hierarchical Configuration Space
Having a simple and flexible yet highly precise mechanism to configure cluster nodes and group them is one of the most crucial components a smart cluster management must provide. To keep the administrative effort as low as possible, it should allow to generalize as much as possible, while at the same time offer the possibility to assign specific configurations to nodes when needed. The design goal is to support arbitrary complex cluster configurations each with a minimal amount of settings to be defined by an admin.
QluMan meets this design goal by a) providing three different configuration property categories:
- Generic Properties are simple on/off options or key+value pairs applicable to individual or groups of nodes, e.g. to schedule reformatting of the local disks at the next boot, add options to Infiniband subnet manager, etc.
- Config Classes handle more complex configurations like boot/disk configs, DHCP, Slurm, etc.
- The node’s Hardware Properties describe their hardware configuration (# CPUs, RAM size, etc.) and are auto-detected during the boot phase. They are of importance e.g. for the configuration of the Slurm workload manager and/or inventory management.
and b) by supplementing these with a hierarchical configuration space consisting of four hierarchy levels ranging from least to most specific:
- The global template consist of the global Hardware/Generic Property and Config Set. Parameters coming from the global template have lowest priority when conflicting settings exist for a given node parameter.
- Host Templates consist of exactly one selectable Hardware Property, Generic Property and Config Set. Assigning a Host Template to a node is the fastest and most straight-forward option to specify all its specific and relevant parameters. Parameters coming from Host Templates override global ones when conflicting settings exist for a given node parameter.
- Property/Config Sets allow to combine a collection of Generic Properties, Hardware Properties or Config Classes into a corresponding set. In addition to be being part of a Host Template, such sets can also be individually assigned to nodes thereby adding to or overwriting parameters coming from templates.
- Finally, it is possible to directly assign single properties or config classes to nodes. Such assignments have the highest priority and the corresponding settings overwrite any conflicting value coming from a template or set.
Fast Deployment with Setup Wizards
Given all the options and tweaks QluMan offers to create an optimal cluster setup, for less experienced cluster admins, it can take a while, to grasp some of the ideas and concepts involved with advanced configurations. QluMan’s setup wizards are created for exactly this reason.
Using auto-detected node properties, the node setup and GPU wizards e.g. guides newcomers through the creation of host templates and the corresponding property sets such that machines with different hardware and functionality will be grouped and configured in the most effective way. Whenever multiple alternative configuration options are selectable, the wizards make suggestions about which ones will make most sense in a given context. As a consequence, admins are enabled to bootstrap and expand arbitrarily complex clusters in hours rather than days or weeks.
Remote Execution Engine
QluMan provides a powerful graphical remote command execution engine (RXEngine). It allows to run shell commands on any number of hosts in parallel and analyze their output/status in real-time. Commands fall into two categories: Pre-defined commands and custom commands.
- The command can be a single one or a series of commands in shell syntax.
- The hosts are specified in Hostlist format or through a filter, so that even large groups can be represented by a short string.
- Commands are executed in parallel on all hosts.
- The network connection used for remote execution is both encrypted and authenticated. It employs the same high-speed/high-security elliptic-curve cryptography that is used for the connection between the QluMan server and the QluMan GUI.
- The output is analyzed and updated in short intervals during the execution phase.
- In the command display window, hosts with identical command output are consolidated so the same output coming from different nodes is not shown multiple times. This results in a much clearer overview of results.
- The output can further be filtered by the return code of the command and by (de)selecting stdout and/or stderr.
Flexible User and Rights Management
Large clusters are often managed by a group of administrators each with different skill levels. To prevent damage by wrong configurations or other operational mistakes, it is important to have the capability to limit the actions less experienced admins are allowed to tackle.
QluMan has all the features required to handle such scenarios. It comes with multi-user support and provides an interface to setup its users together with their permissions within the QluMan framework. Note, that QluMan users are not connected to system users in any way.
The concept of user roles is implemented for an efficient permission management. User roles are defined as a collection of permissions to perform various QluMan operations. Once generated, they can be assigned to a user. User roles can also be assigned to RXEngine commands, thereby allowing to restrict the list of users that have the permissions to execute a specific command on cluster nodes.