AN!CDB - Cluster Dashboard
AN!Wiki :: AN!CDB - Cluster Dashboard
AN!CDB, the Alteeve's Niche! Cluster Dashboard, is a management tool for clusters built following the AN!Cluster_Tutorial_2.
Its first and foremost goal is to be extremely easy to use. No special skills or understanding of HA is required!
To achieve this ease of use, the cluster must be built to fairly specific requirements. Simplicity of use requires many assumptions be made.
- A single view of all cluster components and their current status.
- Control of the cluster nodes. Nodes can be:
- Powered On, Powered Off and Fenced
- Join to and withdrawn from the cluster
- Control of the virtual servers. Servers can be:
- Booted up, gracefully shut down and forced off
- Migrated between nodes
- Create, modify and delete servers;
- Create and upload installation and driver media
- Provision new servers, installing from media just like bare-iron servers
- Insert and Eject CD/DVD images
- Change allocated RAM and CPUs
- Delete servers that are no longer needed.
- Control of the cluster nodes. Nodes can be:
AN!CDB is designed to run on a machine outside of the cluster. The only customization needed is for the cluster name and the name of the nodes be added to the program. Once done, AN!CDB will collect and cache everything needed to control the cluster, even when both nodes are offline.
|Note: AN!CDB is a constantly evolving, changing and improving project. It is highly likely that the version you install will differ somewhat from what is shown in this manual.|
- Download v1.0.2
Installing AN!CDB is pretty easy. Please jump to the tutorial below and return here once AN!CDB is installed.
Feel free to jump to the section you are interested in. The order below was compiled as a "story line" of how someone might use AN!CDB to "cold boot" a new Anvil!, create some servers, take a node out for maintenance, manage existing servers, boot a windows server off of the install disk for recovery, "cold stop" and Anvil! and delete servers.
All screen shots shown here were taken in late July 2013. AN!CDB is always evolving so what you see today may differ somewhat from what you see here.
There are two main buttons that you will see and use often. They are the "Back" button (not to be confused with your browser's "Back" button) and the "Refresh" button. In addition, the main dashboard logo is click-able and will either return you to the Anvil! selection screen or, if you have just one Anvil!, return you to the main dashboard page.
- The "Back" button, #1, will return you to the status and control screen for the currently selected Anvil!.
- The "Refresh" button, #2, will update the current screen to ensure that what you are seeing is accurate and up to date. It is a good habit to always click "Refresh" when ever you return to AN!CDB after a period of time away.
- The main logo, #3, will return you to the Anvil! selection screen. If you have just one Anvil, it will be automatically selected causing it to behave like the "Back" button.
Many terms in the computer world are a little fuzzy. To make sure we're all on the same page, this is how we at AN! define a few core terms;
|Node||The physical hardware machines. These are members in the cluster stack and a node can host any of your servers.|
|Server||This "virtual" server that you install your operating system and software on. You can think of servers as being essentially identical to traditional "base-iron" servers you are used to using.|
|Bare-iron||This is a term often used to describe traditional servers where the operating system and software are installed on a dedicated hardware machine.|
|Hypervisor||This is the software that emulates the "virtual hardware" that your servers see and use.|
Connecting to the Dashboard
|Connecting to the Dashboard|
Open a web browser and browse to the dashboard. Here we will use the actual dashboard's desktop, so we will browse to http://localhost. If you connect from another computer, browse to the IP address or host name you setup for the dashboard.
When you first connect, you will be asked to log into the dashboard. The default user name is admin. There is no default password, but it usually set to the same password used when logging into the dashboard's desktop as the alteeve user.
If you do not know what the password is, please contact support. We can not recover your password, but we can help you reset it.
Choosing the Anvil! to Work On
|Choosing an Anvil!|
If you have configured your dashboard to work on two or more Anvil!s, the first screen you see will ask you to select the one you wish to work on. This list is created by reading your /etc/an/an.conf file on the dashboard. By default, four sample entries are created. You will need to edit the an.conf configuration file to point to your Anvil!(s) before you can use the dashboard to control them.
If you have only one Anvil!, you will not see this screen.
If the dashboard sees only one Anvil! in the configuration file, it will automatically be selected.
The Loading Screen
|Choosing an Anvil!|
The dashboard does not cache or record an Anvil!'s state. Each time the dashboard needs to display information about the Anvil! and before performing an action on an Anvil!, the dashboard calls the nodes directly and checks their current state.
What this means is that you will frequently see the dashboard loading screen. This is an animated pinwheel asking you to wait while it gathers information.
When you see this screen, please be patient. The dashboard will wait up to 1 hour and 40 minutes, by default, for the cluster to return information. This is far longer than any action should take, of course. If you wait several minutes and the loading screen is still displayed, feel free to click on the "reload" icon to the right of the top logo.
"Cold Start" the Anvil!
|"Cold Start" process|
If you Anvil! nodes are both powered off, you will see the "Cold Start" screen. If your dashboard is connected to the Anvil!'s "Back-Channel Network" (BCN), you will be given the option to turn on both nodes or just one node.
In almost all cases, you will want to start both nodes at the same time.
To start both nodes, simply click on the "Power On Both" button. Should you ever want to turn on just one of the nodes, click on the "Power On" button that is on the same line as the name of the node you wish to boot.
Once you click on "Power on Both" or "Power On" for a single node, you will be asked to confirm the action. Click on the "Confirm" button and the dashboard will connect to each node's IPMI interface and ask it to boot the nodes.
After confirming the action, the dashboard will do an initial scan to make sure that the nodes can still be turned on. If someone else turned them on already, the dashboard will abort and tell you why. These checks are done because multiple people can use the dashboard at one time. In most cases though, your action will complete successfully.
If you go back to the main page before the nodes are booting, you will see one of two screens;
- "No Access"; This tells you that the dashboard can see that the node is powered up, but it was unable to connect via ssh to the node. This means that the node is still powering up, a process that can take several minutes on most modern hardware servers.
- "Connection Refused"; This tells you that the node's operating system is booting up, but not quite ready to accept ssh connections. The node should become accessible very soon. Click on the "refresh" icon at the top-right and you should be able to connect to the Anvil! in just a moment.
Start the Cluster Stack
|Starting the Cluster Stack|
To prevent a possible condition called a "Fence Loop", the cluster nodes are not allowed to automatically start the cluster stack when they boot.
After a cold start of the cluster, your Anvil! will not yet be able to run servers. The next step that is needed is to start the cluster stack by pressing the "Start Cluster" button. You will be asked to confirm the action, press "Confirm".
The two nodes in the cluster must have their stacks started at the same time (as opposed to one after the other). For this reason, the data coming back from the nodes can be mixed together. The main effect of this is that, sometimes, the formatting of the output can be skewed, as seen in the third screen shot above. This is harmless and safe to ignore.
When you click on the "back" button, you will see that the cluster stack is either starting or fully started. The fourth screen shot above shows the cluster after it has fully started. If you see anything that isn't green, please wait a moment and then click on the "refresh" button. Within a minute or two, everything should become green and you will be ready to proceed.
The Media Connector is used to make installation, driver and software media (CD and DVD images) available for use in servers. There are three ways that media can be added to an Anvil!;
- Create an image of a physical CD or DVD disk.
- Download from a website address.
- Upload a file from your computer.
Once an image file is on you Anvil!'s shared storage, you will be able to use it to install a server's operating system or mount it in an existing server's (virtual) DVD-ROM drive.
Creating an Image from a Physical CD or DVD
|Note: This step requires a physical CD or DVD drive be installed or plugged into the dashboard server.|
|Creating an image of a physical CD or DVD|
- If your dashboard has a DVD or CD drive, insert your disc into it. If your dashboard does not have an optical drive, plug a USB drive in and insert your DVD or CD.
- Once the disc has been inserted, refresh the Media Connector window and the "Upload" button will become available. Click on Upload to go to the confirmation page.
- The label on the disk will be used to name the disc image file. Often this is a fairly cyptic name, so you can change the name if you wish. Once you are happy with the name, click on Confirm.
- PLEASE BE PATIENT! This can take a while for anything to show up on your screen. Please give it some time.
- After a bit, you will see a screen telling you that the image is being created. How long this takes to complete depends on the speed of the CD or DVD-ROM and the size of the disc being imaged.
- Once the image has been created, it will be uploaded from the dashboard machine to the Anvil!. How long this takes will depend on the speed on the Back-Channel Network connection and the size of the image file.
- Once the upload is complete, you will see Done!. Click on the back AN!CDB "Back" button to return to the Media Connector page.
- On the main Media Connector page's Files already on the shared space section, you should see the image file you just created!
Downloading an Image From a Website or FTP Server
|Download an ISO from a website or FTP server directly to shared storage|
If you have an ISO image available for download from a website, you can pass the URL directly to Media Connector and it will connect to the address and download the image directly to the Anvil!'s shared storage. This is one of the most efficient ways to make CD/DVD images available to your servers.
- Copy the website or FTP server address and paste it into the Address field and then click the Download button.
- You will be asked to confirm the download. Note that, depending on the size of the image and the network speed, the download might take a long time to complete.
- Once the download begins, a progress message will be displayed every 5% of the download.
- Once the download is complete, you will see a summary of the download and a Done! message.
- Click on the "Back" icon and you will see the new image in the file list!
|Warning: Once the download starts, do not browse away from the download screen! Doing so could result in a partial download and unusable image file. If this happens, delete the file and download the file again.|
Uploading an Image From Your Computer
|Upload an ISO from your computer to shared storage.|
There are two common cases where you will need to upload an ISO from your personal computer;
- You created an image file of a CD or DVD using an application on your computer.
- You had to download an ISO from a website that uses temporary, session-based links to ISO files.
In either case, you will need to upload the ISO file to your Anvil!'s shared storage.
|Note: Browsers are very cautious about links that open your file browser. For this reason, when you click on Upload, it might take a while before the progress screen appears. Note in the second screen shot how the browser shows that the upload is in progress by showing a pin-wheel in the tab and a notification message at the bottom (other browsers will do some variation of this). Please be patient!|
- Click on the Browse button, locate the CD/DVD image file you wish to upload and select it. Once done, you will see the path to the file in the File field.
- Click on the Upload button and the upload will begin. Note that it could take a while for the page to change!. The browser should show a message and an icon indicating that the upload is in progress.
- After a time, the browser will refresh and an "in-progress" message will be displayed. Please continue to wait.
- When the upload is complete, a message indicating this will be displayed.
- Click on the "Back" icon and you will see the new image in the file list!
|Warning: Once the upload starts, do not browse away from the download screen! Doing so could result in a partial download and unusable image file. If this happens, delete the file and download the file again.|
This section will cover how to install various operating systems. The first one will use Microsoft's Windows 7. This first example will cover all of the installation steps in detail. The OS instructions after that will be somewhat condensed, so please review the Windows 7 install even if you plan to install a different operating system.
Starting the Build
|Note: The Build a New Server will turn grey once you run out of disk space or memory. If this has happened, you will need to free up some resources before you can build a new server.|
|Connect to 'Virtual Machine Manager' and then click on Build a New Server|
To build a new server, we will need direct access to the new server just as if we had a monitor, keyboard and mouse plugged into a traditional bare-iron server. This is needed because when the operating system install starts, there is no network connection. This makes remote installation (almost always) impossible. To do this, the first step is to double-click on the Virtual Machine Manager icon on the dashboard server's desktop. In order to do this, you will need to be using the dashboard server directly.
If you enabled auto connect when you added your nodes to Virtual Machine Manager, then the connection to the servers should immediately be established. If you see Not Connected beside the node names, then simply double-click on the node names in the Virtual Machine Manager window. This will connect to the nodes and you will then be ready to start the new server build!
Single-click on the Build a New Server button at the bottom-left of the dashboard. This will take you to the New Server form. There are seven sections you will need to fill out.
- Server Name: This is the name of your server. It must be unique from any existing servers on your Anvil!(s). Beyond that, you can use just about any name you wish.
- Optimize for: The hypervisor adjusts how it creates the virtual hardware to try and get the best performance possible. To do this, you will need to select the operating system that most closely matches the operating system you plan to install. If you can't find the exact entry for your operating system, then choose the one that is closest. It does not have to match exactly.
- Install From: This is a drop-down list of installation media. Select the disc image that you wish to install your operating system from.
- Driver Disc: This is the same drop-down list found by Install From. In this case, you can select a second disc image to use during the operating system install. This disc will not be used to start the operating system install, but it will be available during the install process. This is ideal if your operating system will need extra drivers, such as the virtio drivers needed for all Windows installs. It can also be used if your operating system needs a "Disc 2" in order to complete. If you don't need a second disc, you can leave this empty.
- Memory (RAM): This is where you can choose how much RAM to give your new server. Please note the amount of RAM free and do not try to set a value higher than what is available. There is a select field to the right of the text field where you can change the capacity from GiB to another size type. Generally this can be left as-is.
- Number of CPUs: This is a drop-down list that lets you choose how many CPUs to allocate to the new server.
- Storage Space: This section serves two jobs, which we'll cover in more detail below;
- It allows you to choose how much hard drive space to assign to the new server.
- It assigns the new server to a given node.
For the Server Name, we will use the format vmXX-description. The prefix vmXX- is a simple way to ensure that all servers are uniquely named. If you have two or more Anvil!'s, you will want to ensure that the server names are unique across all dashboards. This way, if you want to move a server from one Anvil! to another, you won't ever have to worry about conflicting names. In the end, you can use whatever makes sense to you.
The Optimize for section contains many, many operating systems. Even so, it is not a complete list. If you do not see your operating system in the list, try to choose the closest one. For example; At the time of writing, there is not an entry for Microsoft Windows Server 2012. So you could choose "Microsoft Windows Server 2008" as it is the closest match.
Above Install From is a button called Add a new Disc. If you don't see the install or driver disc you wish to use in the drop down lists, you can click on this button. It will open the Media Connector in another browser window or tab. You can add the missing disc(s) and then, when you are done, close the Media Connector window. Then refresh the Build a New Server window and the new disc(s) should show up in the list. Most modern browsers will preserve any fields you've already filled out, so you should not have to start over.
The Storage Space section, as mentioned, serves two roles.
Each new server gets assigned to a "preferred" host node. This is done because, usually, you will have two storage pools. Whenever possible, you will want all of the servers on a given storage pool to be running on the same node at the same time. This is done as a fail-safe to simplify recovery in some failure scenarios.
You will have to consider two things when choosing a host node;
- Do both nodes have enough free space for the new server? If not, then your decision will be limited to the storage pool with enough free space.
- If both storage pools have enough space, then look at the existing servers on each host. If your new server will have a high CPU load, then you will want to add it to the node with servers that have relatively lower-CPU loads. This way, you can spread out the processing load when both nodes are healthy and online to gain the best performance possible.
Once you have filled out the fields, click on the Confirm button and your new server will be created and the install will begin!
Once the install starts, you will see your new server appear in Virtual Machine Manager under the node whose storage pool you used. Double-click on the server and you will get direct access to the server. From there, you can install your operating system just like you would if your were installing the same OS on a bare-iron server.
Building a Windows 7 Server
If you are familiar with installing Microsoft Windows 7, this section will be very familiar to you. There are only two steps of note;
- Loading the virtio network driver.
- Loading the virtio storage driver.
In order to provide maximum network and storage performance possible, Anvil!'s use Red Hat's virtio technology. From the perspective of Windows, the network card and "SCSI" controller are just another type of hardware network interface card and storage controller. Behind the scenes though, these emulated hardware devices behave much more efficiently than tradition network and storage emulated devices because they know that they are virtualized.
This means that a lot of system calls that make sense on real hardware, but are inefficient on virtual hardware, are removed. The end result is that your server will have much better network and storage performance!
Microsoft windows 7 does not natively support virtio, so when windows initially starts to install, it will not be able to see these devices. So we will need to download the virtio drivers. You can use the Media Connector to either download the ISO file directly to your Anvil! or you can download the ISO on your computer and then upload it to the Anvil!. In either case, you can then select it as the driver disc. When the OS install reaches the partitioning stage, you can load both drivers. This will be covered in detail shortly.
Windows 7; New Server Form
|Create the new Windows 7 server|
In this example, we will fill out the New Server fields this way;
|Optimize for||Microsoft Windows 7|
|Install From||Windows7-64bit-SP1.iso (Windows 7 DVD image)|
|Driver Disc||virtio-win0.1-59.iso (Driver disc downloaded earlier)|
|Memory (RAM)||4, GiB|
|Number of CPUs||2|
|Storage Space||100, GiB drawn from an-c05n01's storage pool|
This will create a server with 2 CPUs, the most Windows 7 will support, 4 GiB of RAM and have a 100 GiB hard drive. This new server will have an-c05n01 as it's preferred host.
Click on Confirm to create the new server and begin the operating system install. Once the new server is created, double-click on it when it appears in Virtual Machine Manager.
|Note: Do not add the new server to the cluster until after the first reboot. If you do, the server will try to boot off the install media every time you try to boot it, until you manually change the boot order.|
Windows 7; First Steps
|Installing Windows 7|
- The first step is to select install language, time and currency and keyboard type. Change the language and regional settings to whatever you wish, but leave the "Keyboard" as US. Click Next.
- Windows 7 shows a couple of options and a large Install Now button. Click it to proceed.
- Please read the EULA for Windows 7 and, if you agree with it, click to check the I accept the license terms and then click the Next button.
This will take you to the storage, which we'll cover in it's own section directly below.
Windows 7; Loading the virtio Drivers
|Note: This is the only non-standard part of the windows install. Please pay close attention here and you can probably skim the rest if you are already familiar with installing Windows 7.|
|Loading the virtio drivers|
Technically, we only have to install the virtio SCSI driver in order to see the hard drive and complete the operating system install. However, this would leave the network card unrecognised until you manually loaded the drivers after the OS finished installing. This can make the install of Windows a little slower because it makes use of an Internet connection during the install, if it is available.
So the trick we will do is, when asked to load the storage driver, we will first load the network card's driver. This loads the driver into memory so that Windows will recognise the network card and configure it for you automatically. To do this;
- Click on the Custom (advanced) install mode.
- No hard drive will be found. Click on Load Driver at the bottom-left.
- Click on Browse. This will allow you to browse the files on the driver disc we used when we created the server.
- Expand the E: drive. This is the virtio driver disc.
- Here we are installing the 64-bit version of Windows 7, so we will want to browse to the E:\Win7\AMD64 directory and then click Next. If you wanted to install a 32-bit version of Windows 7, browse instead to E:\Win7\X86.
- This will load three drivers; One for the network card, one for the storage controller and one called a "balloon" driver, which we won't use. As we discussed, we will load the network card driver first so click to select Red Hat VirtIO Ethernet Adapter and click Next.
- This returns us to the storage screen and, of course, the storage is not yet visible. So click on Load Driver at the bottom-left.
- Browse again to E:\Win7\AMD64, click Next and this time click to select the Red Hat VirtIO Ethernet Adapter driver. This is the actual storage driver.
- Now Windows can see and use the 100 GiB hard drive we created!
When you click Next here, the actual first stage of the OS install will begin. That is the topic for the next section.
Windows 7; Finishing The First Stage of the Install
|Finishing the first stage of the install.|
This stage is totally autonomous and will not require you to do anything.
Windows 7; Final OS Configuration
|Final OS Configuration|
If you have ever installed Windows 7 before, this section will be very familiar to you.
You will be asked to set the computer name, initial administrative user name and password. Once done, you will have to enter the Windows 7 license key, which is a 25-character string of letters and numbers affixed to your install DVD or license card that you purchased from Microsoft.
You will then be asked how you want to update windows. It is strongly recommended that you do not enable automatic updates! There are two reasons for this;
- If you initiate a graceful shut down of the windows server and forget to first install the updates, the updates will begin and may not complete in time. If the Anvil! doesn't see the server power off within two minutes, it will treat the server as having failed and will force-off the server.
- Secondly, some operating system updates will cause windows to reboot without any human confirmation. This could result in the Windows 7 based server rebooting in the middle of a working day.
Of course, if you do disable automatic updates, be sure to periodically, once a week ideally, manually check for and install the OS updates. Be sure to choose a time where you can reboot the server as many windows updates will require this.
|Note: The Anvil! nodes set their time against a time server and all servers get their time from the hosts. So please do not set the time in windows or tell windows to sync against a time server. It is not needed.|
Next, set the date, time and time zone for your server. Lastly, tell windows whether the network it is on is trusted or not. Generally speaking, you will always want to select Work Network.
After this, windows will finish configuring the operating system and bring up your desktop for the first time. We show here two common post-install tasks that you may wish to do. First is increase the screen resolution to 1024 x 768, which will give you more working space when you connect to the desktop using Virtual Machine Manager. The second is to enable Remote Desktop, which is a very popular and built-in method of accessing Windows 7 remotely. It is entirely up to you if you actually want to make these changes.
From this point forward, you can use your new server exactly the same as you would use any other windows 7 machine!
Windows 7; Adding the New Server to the Anvil!
|Adding the New Server to the Anvil!|
In the screen shots above, I closed the vm01-win7 window entirely just to make it easier to see what was going on. There is no need to actually close or even minimize the window though, it's up to you. If you plan to keep working directly on your windows 7 machine, then it's fine to leave it open.
Click on the click here button at the bottom-right of the dashboard and then please be patient. It can take a minute to finish. If you see a warning like the one shown in the screen shot, you can safely ignore it.
Click on the dashboard's back button and you will return to the main page. You should see that the new server has been added!
Window7; Graceful Shut Down and Boot Up
|Gracefully shutting down a server and then booting it back up.|
|Note: If you just want to reboot your system, you do not need to use the dashboard.|
You can reboot your server from inside the operating system. If you power off the server, the Anvil! will boot it back up for you. You only need to follow this if you want to shut the server down and have it stay off. This might be the case if you're planning to completely power off your Anvil! for a prolonged power outage, for example.
So, if you want to power off your server and keep it off, please keep reading.
|Warning: Please read the next section carefully.|
The Anvil! draws a very strong line between it and the servers that run on it. There is no way for it to "look inside" any of the servers to see what is happening. So when you ask the Anvil! to shut down a server, it will initiate the shut and then start a two minute timer. If the server hasn't shut off after two minutes, it treats the server as failed and forces it off.
The concern here is how windows holds off installing some updates until the shut-down starts. So windows could well take over two minutes to install the queued up updates.
So with windows machines, it is recommended that you set automatic updates to download, but not automatically install. This way, you can install updates and reboot at your leisure. Whether on an Anvil! or not, Windows 7 may automatically reboot your machine to install updates, so this is a good practice in any case. If you do this, then orderly shut down of the server from the dashboard should always be safe.
If you leave automatic updates running though;
Always log into windows and make sure there are no pending updates. If there are, reboot the server from within the OS so that windows can finish the install. Wait for the system to boot back up and make sure a second reboot is not needed. Once there are no pending updates, it is safe to proceed.
When you are ready to shut down the server, click on Orderly Shut Down beside the server. Confirm the action and then you should see windows shutting down.
When you want to power the server back up, click on Boot Up, confirm the action and you should be on your way!
Installing CentOS version 6 is nearly identical to RHEL version 6 which is covered in the next section. The main differences; Registering with Red Hat Network and adding entitlements to RHEL, are not covered in this guide.
Red Hat Enterprise Linux 6
|Licensing, Updates and Network|
KVM/QEMU maintained list of tested guest OS.
Windows 7 Professional has a limit of two sockets. The hypervisor presents each core as a socket, so Windows 7 will only use up to two cores.
There is a harmless but verbose bug when using Solaris 11. You will see errors like:
WARNING: /pci@0,0/pci1af4,1100@1,2 (uhci0): No SOF interrupts have been received , this USB UHCI host controller is unusable
Ignore this until the install is complete. Once the OS is installed, run:
The errors will no longer appear.
Solution found here.
|Any questions, feedback, advice, complaints or meanderings are welcome.|
|Us: Alteeve's Niche!||Support: Mailing List||IRC: #clusterlabs on Freenode||© Alteeve's Niche! Inc. 1997-2019|
|legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions.|