Networking in Xen 3.x
In Xen3, there are two possible ways that networking is handled. That is, bridging connections between domU's and the real world. The tutorials on AN!Wiki use the method adopted in newer Xen versions and which is recommended by Xen. That is, bridges are built in the system, outside the scope of Xen.
However, the old method, and still the default method on EL5, has Xen create and manage the bridge. In this scenario, the real interface is moved off to the side and the dom0 interface becomes a virtual interface which connects to the bridge in the same manner as the domU VMs.
This section is based on this link. There are a lot of other good topics on Xen networking and general Xen topics here.
Definitions
Some of the terms below will only exist in the tradition Xen 3 networking, and some will be used in both new and old type networks but have different meanings. Where there is uniqueness, the following prefixes will be used:
- Old Type: Refers to the Xen default configuration, where Xen creates and manages the bridges.
- New Type: Refers to the new, recommended method of managing the bridges outside of Xen.
dom0
The "real" OS on a Xen server is not real at all, but in fact just another virtual machine with special access to the hardware. This first virtual machine is known as dom0.
domU
Any virtual machines created on a Xen server other than the special dom0 is called a domU server. Generally this is used in documentation to describe any particular virtual machine. When a specific virtual machine is to be referenced, it is called dom1, dom2, ..., domN. The number of the virtual machine is important as it relates to the underlying network setup. The given number is not user assignable, but is set when the virtual machine is created. Please note that, even if a virtual machine is destroyed, it's number is not recycled. Thus, you may well have something like 'dom0', 'dom1', 'dom3', 'dom5' if, at some point, you created a 'dom2' and 'dom4' and then destroyed them.
ethX
In either new or old type configurations, ethX devices are found in both dom0 and domU VMs. All other interfaces are found in dom0 only.
ethX; Old Type
Normally, this is the name of a "main" ethernet device. However, under Xen, this does NOT reflect the real hardware once the xend daemon has started!
Under Xen, what 'ethX' refers to depends on where you are. Under dom0, 'ethX' is a virtual ethernet device which has a copy of the real ethernet device's MAC and IP addresses. On domU, 'ethX' is a virtual ethernet device connecting to a matching 'vifX.Y device created on dom0 by Xen when domU was started. See the vifX.Y definition below.
Replace 'X' in 'ethX' with a number reflecting a given ethernet device. For example, 'eth0' is the first ethernet device, 'eth1' the second and so on.
ethX; New Type
When bridges are built outside of xen, the ethX devices are, in fact, the actual ethernet device rather than copies. However, the one difference is that the IP address that had been assigned to the ethX gets moved to the bridge's xenbrX interface.
pethX
pethX; Old Type
The pethX device(s) exist only on dom0.
When xend starts, it renames the "real" 'ethX' to 'pethX' in order to create a virtual copy of it. This is done to facilitate bridging and routing. So then, pethX devices are in fact the interfaces to the real hardware ethernet devices.
Replace 'X' in 'pethX' with a number reflecting a given ethernet device. For example, 'peth0' is the first ethernet device, 'peth1' the second and so on.
pethX; New Type
There is no pethX device created when the bridge built outside of Xen.
vethX
vethX; Old Type
The vethX device(s) exist only on dom0 and only for a short time during xend startup.
During the Xen startup process, ethX is brought down and it's MAC and IP addresses are copied to vethX. Once done, ethX is renamed to pethX to move it out of the way. Then vethX is renamed to ethX to take the place of the pethX interface. As such, vethX devices exist for only a short time. You should never need to worry about or factor in these devices into your networking plans.
Replace 'X' in 'vethX' with a number reflecting a given virtual ethernet device. For example, 'veth0' is the first virtual ethernet device, 'veth1' the second and so on.
pethX; New Type
There is no vethX device created when the bridge built outside of Xen.
vifX.Y
 |
Note: In this section, X matches the domU ID number, Y matches a given ethY device on dom0 and Z matches a given ethX device inside a domU. I know this is a little odd, but you need to get this clear in order to see how things connect here. |
|
Note: Under the new-type configuration, a vif0.Y is not needed as the eth devices are directly connected to the bridge. |
These are Xen's virtual interfaces used to connect an ethX device on dom0 to a pethZ device on a domU. Please be aware that there need not be any correlation between eth0 on dom0 and eth0 on domU! This is why 'X' and 'Z' are used separately here.
Specifically, when Xen starts a domU, that virtual machine's ID# plus a given dom0 ethX device number are used to create one or more virtual interfaces inside dom0. For example, if you want to connect eth0 on dom0 to eth0 on dom4, you will do so via vif4.0 where '4' matches the domU ID# and '0' matches the ethernet device number in dom0. The actual routing goes through xenbr0. If this is still confusing, take a look at the network map below.
xenbrX
Xen uses software bridges to route traffic within it's virtual network. These can be thought of as a "switch" your would find in any normal network. They serve to connect a given ethernet device on dom0 to a given ethernet device in a domU virtual machine. A different bridge is created for each ethX device on dom0.
xenbrX; Old Type
When Xen creates these bridges, they are brought up when the xend daemon starts. Specifically, they are created by the (network-script network-bridge) script, where network-bridge is the default, but replaceable, xen script. This is set in the /etc/xen/xend-config.sxp script. In this case, the xenbrX devices have no IP address.
xenbrX; New Type
When the xenbrX devices are managed outside of Xen, they are created using /etc/sysconfig/network-scripts/ifcfg-xenbrX interface configuration files. The corresponding ethX devices have their IP values moved into the bridge configuration files, then are connected to the bridge with the BRIDGE=xenbrX option.
With this, the IP address is no longer assigned to the interface, but is instead bound to the bridge itself. However, all programs can continue using the ethX interfaces as they always would.
Network Map - Putting it all together
Example; Old Type
This map is a sample of a four-VM setup in a single Xen host. You will notice that eth1 is not altered because, in this case, it is used for the DRBD channel.
In this scenario, xenbr0 would connect to a private server network, called the Back Channel Network in some tutorials. All virtual machines have a connection to xenbr0. By contrast, xenbr2 would be the Internet Facing Network, thus only a restricted subset of machines would have access to this bridge.
Assuming that dom1 is a firewall, routing Internet traffic to the rest of the domU VMs, connections can be routed in from the internet via xenbr0, routed and sent on the the destination domU via the xenbr2 bridge. This works in clusters even when the firewall VM is on a different node from the target server. In this situation, the traffic from the firewall would hit it's own xenbr2, cross the physical switch and travel on over the other node's xenbr2 before reaching the destination server.
For a little variety, dom6 has been given a direct access to xenbr0, akin thus exposing it directly to the Internet.
___________________________________________________________________
| dom0 _______________ |
|_______ ________ |______ dom1 | |
= peth0 |=----------| xenbr0 |-----------[vif1.0]-= eth0 | | |
|-______| | |---------\ |-_____| | |
| ______ | |-------\ | | | |
| | eth0 =-[vif0.0]-| |-----\ | | |______ | |
| |______| | | /-+-+-+-[viv1.2]-= eth1 | | |
| | | | | | | |-_____| | |
| |________| | | | | |_______________| |
| | | | | |
| | | | | _______________ |
|_______ | | | | |______ dom2 | |
= eth1 | | | | \-[viv2.0]-= eth0 | | |
|-______| | | | |-_____| | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | |_______________| |
| | | | _______________ |
|_______ ________ | | | |______ dom4 | |
= peth2 |=----------| xenbr2 |---/ | \---[viv4.0]-= eth0 | | |
|-______| | |---\ | |-_____| | |
| ______ | | | | | | |
| | eth2 =-[vif0.2]-| | | | | | |
| |______| | | | | | | |
| | | | | | | |
| |________| | | |_______________| |
| | | _______________ |
| | | |______ dom6 | |
| | \-----[viv6.0]-= eth0 | | |
| | |-_____| | |
| | | | |
| | |______ | |
| \-------[viv6.2]-= eth1 | | |
| |-_____| | |
| | | |
| |_______________| |
|___________________________________________________________________|
Example; New Type
This is the same scenario described above, but showing the lack of vifX.Y and pethX devices seen in the Xen-managed setup.
__________________________________________________________
| dom0 _______________ |
|_______ ________ |______ dom1 | |
= eth0 |=-| xenbr0 |-----------[vif1.0]-= eth0 | | |
|-______| | |---------\ |-_____| | |
| | IP |-------\ | | | |
| | addy |-----\ | | |______ | |
| | | /-+-+-+-[viv1.2]-= eth1 | | |
| | | | | | | |-_____| | |
| |________| | | | | |_______________| |
| | | | | |
| | | | | _______________ |
|_______ | | | | |______ dom2 | |
= eth1 | | | | \-[viv2.0]-= eth0 | | |
|-______| | | | |-_____| | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | | | |
| | | | |_______________| |
| | | | _______________ |
|_______ ________ | | | |______ dom4 | |
= eth2 |=-| xenbr2 |---/ | \---[viv4.0]-= eth0 | | |
|-______| | |---\ | |-_____| | |
| | IP | | | | | |
| | addy | | | | | |
| | | | | | | |
| | | | | | | |
| |________| | | |_______________| |
| | | _______________ |
| | | |______ dom6 | |
| | \-----[viv6.0]-= eth0 | | |
| | |-_____| | |
| | | | |
| | |______ | |
| \-------[viv6.2]-= eth1 | | |
| |-_____| | |
| | | |
| |_______________| |
|__________________________________________________________|
| Any questions, feedback, advice, complaints or meanderings are welcome. | |||
| Alteeve's Niche! | Enterprise Support: Alteeve Support |
Community Support | |
| © Alteeve's Niche! Inc. 1997-2024 | Anvil! "Intelligent Availability®" Platform | ||
| legal stuff: All info is provided "As-Is". Do not use anything here unless you are willing and able to take responsibility for your own actions. | |||