Fragmented floating IP pools and multiple AS hack

When an OpenStack Havana cluster is deployed on hardware rented from OVH and Hetzner, IPv4 are rented by the month and are either isolated ( just one IP, not a proper subnet ) or made of a collection of disjoint subnets of various sizes.

91.121.254.238/32
188.165.144.248/30
...

OpenStack does not provide a way to deal with this situation and a hack involving a double nat using a subnet of floating IP is proposed.
A L3 agent runs on an OVH machine and pretends that 10.88.15.0/24 is a subnet of floating IPs, although they are not publicly available. Another L3 agent is setup on a Hetzner machine and uses the 10.88.16.0/24 subnet.
When an instance is created, it may chose a Hetzner private subnet, which is connected to a Hetzner router for which the gateway has been set to a network providing the Hetzner floating IPs. And the same is done for OVH.
A few floating IP are rented from OVH and Hetzner. On the host running the L3 agent dedicated to the OVH AS, a 1 to 1 nat is established between each IP in the 10.88.15.0/24 subnet and the OVH floating IPs. For instance the following /etc/init/nat.conf upstart script associates 10.88.15.3 with the 91.121.254.238 floating IP.

description "OVH nat hack"

start on neutron-l3-agent

script
  iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
  ip addr add 10.88.15.1/24 dev br-ex
  while read private public ; do
    test "$public" || continue
    iptables -t nat -A POSTROUTING -s $private/32 -j SNAT --to-source $public
    iptables -t nat -A PREROUTING -d $public/32 -j DNAT --to-destination $private
  done <<EOF
10.88.15.3      91.121.254.238
EOF
end script

Fragmented floating IP pools and routing

Each floating IP ( also called failover IP ) provided by either OVH or Hetzner is uniquely associated to the MAC of the ethernet interface of a given hardware, using the proprietary web interface provided by OVH and Hetzner. The packets destined to the floating IP are routed to the interface even if the interface does not answer to arp who-has. The subnet to which a given floating IP belong is unknown and it is not possible to figure out if there is a gateway in the same subnet as a floating IP. If an instance is associated with such a floating IP, the outgoing packets are expected to be routed via the same gateway as the host. For instance on an OVH host:

root@bm0015:~# ip addr show dev eth0
2: eth0:  mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 70:54:d2:1a:9d:76 brd ff:ff:ff:ff:ff:ff
    inet 188.165.223.81/24 brd 188.165.223.255 scope global eth0
root@bm0015:~# ip route
default via 188.165.223.254 dev eth0  metric 100

The IP address of the bm0015.the.re hardware ( it is not a floating IP ) is 188.165.223.81, is in a /24 subnet and 188.165.223.254 is its router. The 91.121.254.238 floating IP is routed to bm0015.the.re and although it is in a completely different subnet, it is expected to use the same gateway, that is 188.165.223.254.

AS segregation

An external network is defined for OVH:

neutron net-create ovh --router:external=True

The L3 agent for OVH is configured to only handle this network so that multiple agents can be run.

# neutron net-list --name ovh
+--------------------------------------+------+----------------------------------------------------+
| id                                   | name | subnets                                            |
+--------------------------------------+------+----------------------------------------------------+
| 033b4851-af21-478e-9a57-e624ff0b1340 | ovh  | 9cb918b6-8737-4416-a5e0-e4bc5a5e6718 10.88.15.0/24 |
+--------------------------------------+------+----------------------------------------------------+
# grep 033b4851 /etc/neutron/l3_agent.ini
gateway_external_network_id = 033b4851-af21-478e-9a57-e624ff0b1340

It is also set to be the default for internal routers

# grep internal /etc/neutron/l3_agent.ini
handle_internal_only_routers = True

The subnet that will act as if it was a public subnet is created within the ovh net:

# neutron subnet-create --name ovh --disable-dhcp \
              --allocation-pool=start=10.88.15.2,end=10.88.15.254 \
              ovh 10.88.15.0/24
Created a new subnet:
+------------------+------------------------------------------------+
| Field            | Value                                          |
+------------------+------------------------------------------------+
| allocation_pools | {"start": "10.88.15.2", "end": "10.88.15.254"} |
| cidr             | 10.88.15.0/24                                  |
| dns_nameservers  |                                                |
| enable_dhcp      | True                                           |
| gateway_ip       | 10.88.15.1                                     |
| host_routes      |                                                |
| id               | 9cb918b6-8737-4416-a5e0-e4bc5a5e6718           |
| ip_version       | 4                                              |
| name             | ovh                                            |
| network_id       | 033b4851-af21-478e-9a57-e624ff0b1340           |
| tenant_id        | 2a2365c4031d47d890bb403db7e92583               |
+------------------+------------------------------------------------+

The –disable-dhcp prevents running a dnsmasq process that is not going to be used anyway. The allocation pool is not served by dnsmasq for floating IPs.
A router is created and the ovh network is set to be its gateway, implicitly meaning the subnet is going to be used when allocating floating IPs.

# neutron router-create ovh
# neutron router-gateway-set ovh ovh
Set gateway for router ovh

A private subnet is created and connected to the router. All instances that intend to get a floating IP from the ovh pool must be connected to this subnet, otherwise there will be no route between them and the floating IP.

# neutron net-create ovh-lan
# neutron subnet-create --name ovh-lan ovh-lan 10.0.15.0/24
# neutron router-interface-add ovh ovh-lan

Double NAT hack

The first nat is established by OpenStack between 10.88.15.0/24 and 10.0.15.0/24. Another nat is maintained outside of OpenStack to map the IPs from the subnet 10.88.15.0/24 to actual public IPs. The map is maintained manually from an upstart script that runs after the neutron L3 agent.
The instances that have no associated public IPs are masqueraded behind 10.88.15.1 ( the gateway_ip picked by default when the ovh subnet was created above ). The line

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

will masquerade them once more, using whatever public IP is the default for eth0. The full script is added in /etc/init/nat.conf and can be run manually with start nat.

description "OVH nat hack"

start on neutron-l3-agent

script
  iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
  ip addr add 10.88.15.1/24 dev br-ex
  while read private public ; do
    test "$public" || continue
    iptables -t nat -A POSTROUTING -s $private/32 -j SNAT --to-source $public
    iptables -t nat -A PREROUTING -d $public/32 -j DNAT --to-destination $private
  done <<EOF
10.88.15.3      91.121.254.238
EOF
end script

Alternatives

After discussing with Edouard Thuleau, Chux Uzoeto, Mathieu Rohon, Pierre-Andre Morey, Christophe Sauthier, Erwan Gallen, Carl Perry, Sylvain Afchain, Emilien Macchi I’ve not been able to find a simpler or less ad-hoc way. It is not possible to attach eth0 to br-ex because OVH will raise alerts if it sees unexpected MAC addresses. It is possible to route a floating IP to br-ex. However it is not possible to subnet-create a single IP ( there needs to be at least one other IP used as a gateway and there is no way to specify a gateway that is no in the same subnet as the IP ). It is also not possible to update the allocation pools using neutron subnet-update because it is a read only attribute. Although it is possible to hack routes and IP directly in the net namespace of the router, the end result is more contorted than relying on a double nat.

12 Replies to “Fragmented floating IP pools and multiple AS hack”

  1. An alternative, that I originally tried pushing into nova-network, is to use BGP to announce your floating IPs. In that situation you don’t need subnets at all. My nova-network change was rejected due to quantum (now neutron) existing, but neutron at the time didn’t even have floating IP support, so I just gave up…

    The awesome thing about BGP announced floating IPs is that with a little work on the quantum side, it would be possible to move IPs not just between instances but also between networks, regions, datacenters, etc..

  2. Loic,
    That is an interesting and amazing hack to work around the issues .. I am still trying to understand it fully in a logical manner. So, the limitation seems not on physical ethernet devices as I worried about, but more on networking address blocks and how they get used ? .. Please, can you elaborate on this a wee bit.

    It seems the best way forward with these multisite setups would be to look towards SDN based solutions like Open Daylight .. SDN is rapidly gaining traction, but as far as I am aware OpenDaylight is not ready for production use right now.

    Again due to the huge allocation blocks of ipv6 (/64) available with these service providers, ipv6 may represent an immediate solution for splitting networking the way neutron expects them to be setup .. We need to look more seriously at ipv6 as a way out.

  3. The limitation is indeed about using multiple network address blocks as part of a single floatting IP allocation pool. It is not a common use case.
    I don’t know anything about OpenDaillight 🙂 As far as I understand, neutron provides a SDN and as soon as it is able to conveniently handle the use case described here the hack won’t be necessary.
    Using IPv6 is definitely a good idea. The problem with OVH and Hetzner is that they only provide a single IPv6 ( i.e. a /64 such as 2001:41d0:2:aa70::/64 ). You would need to run an IPv6 proxy on the machine to further divide it. At the moment only tetaneutral provides a /56 which could be used with OpenStack.

  4. Loic,
    Thank you for your great post, very useful to understand how to setup OpenStack with the OVH network.
    Since a few months, OVH added a new feature called VRACK. I think it can simplify your (and a lot of people btw) setup even if OpenStack doesn’t provide a way to deal with this new feature right now (or I didn’t found how…).
    Available if you want to talk about it 🙂

  5. Hello Dachary, thanks for the blog.
    I approached in a different way, and I managed to let VMs (without floating IP) ping to the Internet.
    However, when I assigned floating IP to VM, the ping (both way) can’t get through. I debugged and find out OVH reply ping echo with the MAC address of floating IP, then packets came to router-getway, and were dropped (perhaps due to MAC address differences with router-gateway-inteface). No iptables rule were applied.
    did you encounter the same situation?
    Do you have any idea about this problem.

    1. I managed to bridge br-ex with eth0 (by changing MAC address of br-ex = MAC eth0). Then I change MAC/IP external-router-interface = one MAC/IP of floating IP range. So VM without floating IP can get internet access.

    2. I managed to make it work by adding all MAC addresses of floating IP to router gateway. My approach won’t solve the Fragmented floating IP pools, but it work well with OVH MAC check. One drawback is that I have to manually add all RIPE MAC addresses to Router gateway interface.

      About fragmented floating IP range, as I remember, there was a guy tried to inject multiple IP range to neutron db, and he was success.

  6. Take a look to this other approach:

    https://github.com/mangelajo/ovh-ebtables-agent

    It requires at least a patch to allow on-link routes, but it basically does MAC-NAT in an intermediate bridge.

    I guess we could do this with openflow over OVS, probably with a mechanism driver on ML2 would be the best, but this is the best I could come up with in a 2-day weekend hack. I use it myself as a development environment on OVH, it works well.

    1. Thank you Miguel Angel Ajo. I would like to try the ovh-ebtables-agent but I dont really know how to apply this to my system. Can you tell us ?

Comments are closed.