Networking in Xen 4.x
This section is based on this link, which is fairly old. I am trying to sort out what exactly has changed in Xen 4.0 and may not have all the details right. Please feel free to jump in and make changes if you see a mistake or a place that could be improved.
Definitions
dom0
The "real" OS on a Xen server is not real at all, but in fact just another virtual machine with special access to the hardware. This first virtual machine is known as dom0.
domU
Any virtual machines created on a Xen server other than the special dom0 is called a domU server. Generally this is used in documentation to describe any particular virtual machine. When a specific virtual machine is to be referenced, it is called dom1, dom2, ..., domN. The number of the virtual machine is important as it relates to the underlying network setup. The given number is not user assignable, but is set when the virtual machine is created. Please note that, even if a virtual machine is destroyed, it's number is not recycled. Thus, you may well have something like 'dom0', 'dom1', 'dom3', 'dom5' if, at some point, you created a 'dom2' and 'dom4' and then destroyed them.
ethX
Normally, this is the name of a "main" ethernet device. However, under Xen, this does NOT reflect the real hardware once the xend daemon has started!
Under Xen, what 'ethX' refers to depends on where you are. Under dom0, 'ethX' is a virtual ethernet device which has a copy of the real ethernet device's MAC and IP addresses. On domU, 'ethX' is a virtual ethernet device connecting to a matching 'vifX.Y device created on dom0 by Xen when domU was started. See the vifX.Y definition below.
If you followed the HowTo on creating a [Node Xen cluster on Fedora 13], then ethX also acts as the bridge.
Replace 'X' in 'ethX' with a number reflecting a given ethernet device. For example, 'eth0' is the first ethernet device, 'eth1' the second and so on.
pethX
The pethX device(s) exist only on dom0.
When xend starts, it renames the "real" 'ethX' to 'pethX' in order to create a virtual copy of it. This is done to facilitate bridging and routing. So then, pethX devices are in fact the interfaces to the real hardware ethernet devices.
Replace 'X' in 'pethX' with a number reflecting a given ethernet device. For example, 'peth0' is the first ethernet device, 'peth1' the second and so on.
vethX
The vethX device(s) exist only on dom0 and only for a short time during xend startup.
During the Xen startup process, ethX is brought down and it's MAC and IP addresses are copied to vethX. Once done, ethX is renamed to pethX to move it out of the way. Then vethX is renamed to ethX to take the place of the pethX interface. As such, vethX devices exist for only a short time. You should never need to worry about or factor in these devices into your networking plans.
Replace 'X' in 'vethX' with a number reflecting a given virtual ethernet device. For example, 'veth0' is the first virtual ethernet device, 'veth1' the second and so on.
vifX.Y
The vifX.Y devices exist only on dom0.
Note; in this section, 'X' matches the domU number, 'Y' matches a given ethX device on dom0 and 'Z' matches a given ethX device inside a domU. I know this is a little odd, but you need to get this clear in order to see how things connect here.
These are Xen's virtual interfaces used to connect an ethX device on dom0 to a pethZ device on a domU. Please be aware that there need not be any correlation between eth0 on dom0 and eth0 on domU! This is why 'X' and 'Z' are used separately here.
Specifically, when Xen starts a domU, that virtual machine's ID# plus a given dom0 ethX device number are used to create one or more virtual interfaces inside dom0. For example, if you want to connect eth0 on dom0 to eth0 on dom4, you will do so via vif4.0 where '4' matches the domU ID# and '0' matches the ethernet device number in dom0. The actual routing goes through xenbr0. If this is still confusing, take a look at the network map below.
virbrX
The virbrX device(s) exist only on dom0.
Xen uses software bridges to route traffic within it's virtual network. These can be thought of as a "switch" your would find in any normal network. They serve to connect a given ethernet device on dom0 to a given ethernet device in a domU virtual machine. A different bridge is created for each ethX device on dom0.
If you followed the HowTo on creating a [Node Xen cluster on Fedora 13], then virbrX is not used. Instead, the virtual ethX also acts as the bridge.
Network Map - Putting it all together
This map is a sample of a four-VM setup in a single Xen host. You will notice that eth1 is not connected to any domU's because, in this case, it is used for the DRBD channel. All virtual machines have a connection to eth0 because that is the virtual machine network's "back channel". In practice, there is no internet connection on eth0, and thus on the xenbr0 bridge, except through the firewall, dom1. The dom6 shows a link to eth2, which would give it a direct connection to the Internet.
_________________________________________________________
| dom0 _______________ |
|_______ _______ |______ dom1 | |
= peth0 |=-| eth0 |-----------[vif1.0]-= eth0 | | |
|-______| | |---------\ |-_____| | |
| | B |-------\ | | | |
| | r |-----\ | | |______ | |
| | i | /-+-+-+-[viv1.2]-= eth1 | | |
| | d | | | | | |-_____| | |
| | g | | | | | |_______________| |
| | e | | | | | |
| |_______| | | | | _______________ |
|_______ _______ | | | | |______ dom2 | |
= peth1 |=-| eth1 | | | | \-[viv2.0]-= eth0 | | |
|-______| | | | | | |-_____| | |
| | B | | | | | | |
| | r | | | | | | |
| | i | | | | | | |
| | d | | | | | | |
| | g | | | | |_______________| |
| | e | | | | _______________ |
| |_______| | | | |______ dom4 | |
| _______ | | \---[viv4.0]-= eth0 | | |
|_______ | eth2 |---/ | |-_____| | |
= peth2 |=-| |---\ | | | |
|-______| | B | | | | | |
| | r | | | | | |
| | i | | | | | |
| | d | | | |_______________| |
| | g | | | _______________ |
| | e | | | |______ dom6 | |
| |_______| | \-----[viv6.0]-= eth0 | | |
| | |-_____| | |
| | | | |
| | |______ | |
| \-------[viv6.2]-= eth1 | | |
| |-_____| | |
| | | |
| |_______________| |
|_________________________________________________________|
Real World Example
Note: These have not been updated for Xen 4.0 yet.
This is one an example of networking inside a two node cluster.
| Any questions, feedback, advice, complaints or meanderings are welcome. | |||
| Alteeve's Niche! | Enterprise Support: Alteeve Support |
Community Support | |
| © Alteeve's Niche! Inc. 1997-2024 | Anvil! "Intelligent Availability®" Platform | ||
| legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions. | |||