Setup multiple vNICs on an OpenStack-based CentOS VM

In some cases, you may need to setup an external Internet-facing floating IP (FIP) for an existing OpenStack-based CentOS VM that already has an internal network interface. Because the VM is already being used, you may not be able to just get rid of the old internal-facing network interface because data is flowing to it or the VM needs to remain attached to its current internal subnet for configuration reasons. In these cases you can add a second network interface tying the VM to an external Internet-facing network.

  1. First thing's first: warn your users that network connectivity may be lost during the configuration and that you will need a maintenance window. Then create a snapshot of the VM in case you bonk the setup.

  2. Add the second, external-facing NIC to the VM (through Horizon dashboard is fine but you can also use the OpenStack CLI if you are so inclined).

  3. Associate a floating IP with the NIC (again, either through Horizon dashboard or CLI).

  4. Setup a new OpenStack security group to include the incoming ports your users have requested. At this time, it doesn't hurt to add SSH and All ICMP rules for your own IP address for testing. Remember, generally these requests are to open up the VM to all Internet-facing traffic so exercise discretion when setting allowable ports/port ranges.

  5. Add the new OpenStack security group to the VM.

  6. SSH into the VM on the existing internal interface.

  7. Check the current network configuration:

    ifconfig -a

    The -a argument will list all detected network interfaces, including those that are down

  8. You should see the second network interface at this point. Copy its MAC address.

  9. CentOS 7 cloud images don't appear to consistently come with /lib/udev/write_net_rules (maybe because they use cloud-init) so we will modify the persistent net rules configuration by hand. Add an entry for the new network interface (eth1 in this example). It doesn't appear from my experience that order matters in this file so eth1 should be able to be after eth0 if you're picky about those types of things:

    [centos@opst-centos-vm ~]$ sudo vi /etc/udev/rules.d/70-persistent-net.rules

    SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?", ATTR{address}=="fa:16:3e:6f:75:c0", NAME="eth1"

    SUBSYSTEM=="net", ACTION=="add", DRIVERS=="?", ATTR{address}=="fa:16:3e:35:b1:51", NAME="eth0"

    [centos@opst-centos-vm ~]$

  10. Open /etc/sysconfig/network-scripts/ifcfg-eth0 (or whatever interface name is currently the main interface). We will ignore the file's comment which says "..., do not edit."

    NOTE: METRIC may be deprecated in a future release so we will have to find a new method of setting NIC priority at that point:

    [centos@opst-centos-vm ~]$ sudo vi /etc/sysconfig/network-scripts/ifcfg-eth0

    # Created by cloud-init on instance boot automatically, do not edit.

    Add the following two lines, write the file to disk and exit the file:

  11. Copy the file you just edited to a new file in the same directory but append the second network interface's name to the command (eth1 in our case):
    sudo cp /etc/sysconfig/network-scripts/ifcfg-eth0

  12. Open the new file, replace the existing MAC address with the new interface's MAC address and change METRIC to 20:
    [centos@opst-centos-vm ~]$ sudo vi /etc/sysconfig/network-scripts/ifcfg-eth0
    # Created by cloud-init on instance boot automatically, do not edit.

    Then add the following lines to the new config file, write the file to disk and exit the file:
    NAME="System eth1"

  13. At this point, you should be ready to reboot to ensure the network configuration comes up properly. Confirm that no users are still logged in. If there are users still logged in send them a shutdown message:
    sudo shutdown -r +5 "The system is going down for maintenance, please save all work ASAP"

  14. After reboot, attempt to ping the new IP address. You should receive completed ping responses. Then, attempt to SSH into the new address. If you're successful, confirm that the route table is setup appropriately. If you're unsuccessful skip to step 16:
    [centos@opst-centos-vm ~]$ ip r
    default via dev eth1 metric 20
    default via dev eth0 metric 121 dev eth0 proto kernel scope link src dev eth1 proto kernel scope link src
    [centos@opst-centos-vm ~]$

  15. Finally, confirm that the external-facing NIC is handling Internet-bound traffic. If it is, you will know because the FIP will be the IP address that is returned by the command:
    [centos@opst-centos-vm ~]$ curl
    [centos@opst-centos-vm ~]$

  16. If you are not able to SSH into the VM through the FIP, then attempt to SSH in through the internal interface. If you are unable to access through the internal interface, you may have to rebuild the VM (VM rebuild using same IP address tutorial will be published at a later date). If you are able to SSH in through the internal interface, confirm your route table and which NIC is handling outbound Internet traffic:
    ip r

  17. If the internal interface appears as the default interface and/or has a higher listing in the route table than the new interface or the IP address that is returned from your curl command gives a different IP address than your FIP or whatever is expected (your VPN IP, for example), check that METRIC in your eth## config files are set to the appropriate values. If this is not the issue, additional debug beyond the scope of this tutorial will be required.

  18. Once the FIP is up and working, remove the SSH and All ICMP rules from your new security group.

  19. Notify your users that the VM is ready for use.