bridge for virtualisation stays in state DOWN - unable to get it up

Running Rocky 9.3, attempting to get a bridge for a public Range of IPs for use with Virtual machines.

Same hardware used to run Centos 7.8 machine with it all working, am now trying to set it up the same with Rocky 9.3

# /etc/sysconfig/network-scripts/ifcfg-brpub0 (pointopoint, Hetzner Standard)
DEVICE=brpub0
ONBOOT=yes
TYPE=Bridge
BOOTPROTO=none
IPADDR=xxx.yyy.89.1
NETMASK=255.255.255.240
STP=off
DELAY=0

gives

3: br0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 6a:1f:e1:44:ec:b8 brd ff:ff:ff:ff:ff:ff
    inet xxx.yyy.89.1/28 brd xxx.yyy.89.15 scope global noprefixroute br0
       valid_lft forever preferred_lft forever

I installed vagrant and Virtualbox same as before and although I can ping the bridge on xxx.yyy.89.1 I can't ping the public IP address allocated to the vm.

For reference, on another machine that is still on centos 7, the equivalent is

3: brpub0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
    link/ether 5e:ce:35:cf:d2:90 brd ff:ff:ff:ff:ff:ff
    inet aa.bb.207.145/28 brd aa.bb.207.159 scope global brpub0
       valid_lft forever preferred_lft forever
    inet6 fe80::5**e:35**:f**f:d290/64 scope link
       valid_lft forever preferred_lft forever

Why is my bridge down? whatever I do in terms of nmcli and link/device commands, it stays down. How do I get it up?

--- Edit ---

As requested by @larsks, ip link show master brpub0 on the working machine - has no output

[root@prod-vmhost-114 ~]# ip link show master brpub0
[root@prod-vmhost-114 ~]#

Very detailed description here

What happened between CentOS 7 and Rocky Linux 9? A newer systemd version.

With older systemd, systemd didn't peek on bridges, they were left alone when created. With newer versions of systemd, systemd monitors virtual interface creations and will assign a "stable random" MAC address on it instead of leaving the (fully) random MAC address set by default, as soon as its creation is detected. This has a side effect of immediately setting the bridge's operational state (aka operstate) to DOWN, because this has changed its default mode of operation and it has no bridge port (yet).

So if you create twice a bridge on CentOS 7, you can expect its MAC address to change, similar to this:

# ip link add name testbr0 up type bridge
# ip link show dev testbr0
3: testbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether de:6b:22:95:cf:76 brd ff:ff:ff:ff:ff:ff
# ip link del dev testbr0
# ip link add name testbr0 up type bridge
# ip link show testbr0
4: testbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether 02:75:85:6b:a8:d4 brd ff:ff:ff:ff:ff:ff

In this example: first time de:6b:22:95:cf:76 second time 02:75:85:6b:a8:d4

Note how the examples above show the bridge operstate is UNKNOWN.

Now if a MAC address is set on the bridge:

# ip link set dev testbr0 address b6:00:00:00:00:01
# ip link show dev testbr0
3: testbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether b6:00:00:00:00:01 brd ff:ff:ff:ff:ff:ff

This changes the mode of the bridge from "no MAC address assigned" to "been assigned a MAC address".

Notice how the bridge operstate when from UNKNOWN to DOWN.

Now in any Linux system running a newer systemd, including Rocky Linux 9, this assignation is systematically done (this could be verified by leaving a ip monitor link running).

# ip link add name testbr0 up type bridge
# ip link show dev testbr0
21: testbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 92:45:24:9e:58:8e brd ff:ff:ff:ff:ff:ff
# ip link del dev testbr0
# ip link add name testbr0 up type bridge
# ip link show testbr0
22: testbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 92:45:24:9e:58:8e brd ff:ff:ff:ff:ff:ff

Notice how the supposed-to-be-random MAC address is twice the same: systemd, whatever the Network Manager in use, assigned twice the same MAC address on it as soon as it detected it was created without assigning an address.

As above this makes the operstate DOWN instead of UNKNOWN. While the operstate was UNKNOWN, it was temporarily considered as UP, until the first bridge port is assigned to it. Then the behavior would become the same.

As the bridge is operstate DOWN, it won't setup a link-local IPv6 address, as seen in this Q/A: linux ipv6 bridge address does not work when mac address is forced and as seen in OP's examples.

For IPv6 there will be issues, for IPv4 I'm not sure there are visible issues, unless an interface flag such as ignore_routes_with_linkdown is enabled on this interface.

If wanting the bridge to be always in operstate UP, to get a consistent behavior, add a bridge port that is UP. If you don't want to use any, still add a dummy one. Continuing the example above:

# ip link add name testbr0p0 up master testbr0 type dummy
# ip link show testbr0
22: testbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 92:45:24:9e:58:8e brd ff:ff:ff:ff:ff:ff

The bridge interface's operstate is now UP.

More information about systemd's change of behavior and the change of the behavior of the bridge when explicitly assigned or not assigned a MAC address can be seen in these UL SE, SF and SU Q/A where I made an answer:

bridge for virtualisation stays in state DOWN - unable to get it up

1 answer

0 question comments

Your perspective belongs here.