Why do I care about IPv6?

Let me spend some lines, just to introduce why I care about this topic.
As many of my friends know, I’m an IPv6 lover and I like to give presentations to teach how it works and how to setup it.
The IPv6 standard was published nearly 20 years ago and until ~10 years ago there were very few sites enabling it, as well as few autonous systems able to route it.
February 2011 marked the date when IANA (no-profit organization managing the IPv4/IPv6 address space) exhausted its free pool of IPv4 address /8 blocks for assignment to regional registries. This didn’t changed much the availability of IPv4 for the normal customers, but many regional registries and big corporations started to ask themselves about the future of IPv4 address availability.
So, the IPv6 deployment drastically increased the next year after the IPv6 world launch in 2012, when several big companies (Facebook, Google, Cisco, …) deployed and enabled IPv6 for all the customers. Many network operators started to follow the wave and the number of deployments continued to increase, especially when ARIN (the north american Internet registry), completely exhausted its free pool of IPv4 addresses assignable to customers.
Google is continously measuring the percentage of visitors connecting to google.com via IPv6. Google’s data shows how the deployment of IPv6 is currently (February 2016) 10% worldwide, ~23% in the United States alone, and continously increasing exponentially.
This means that, willing or not, we have to move to IPv6 now. It’s not a matter of setting up shitty networks just to put the “IPv6 ready” logo on your homepage. It’s a matter of getting it working.

IPv6 is not complicated. It has some similarities to IPv4 on many things, but after all it’s a new protocol with new features and new methods of address assigning, multicast etc…

While some hosting providers (Online.net) did understand how to setup and operate it flawlessly; others are really doing an awful job (OVH).

So, let’s see what’s wrong with OVH and why Online.net is better.

OVH

First, let me be very clear: I’ve been an OVH customer for more than 6 years with dedicated servers, RPS, kimsufi (old one, new one), domains and I’ve even tested new beta services. I enjoyed many of their services and offers. However they deeply disappointed me for the following pitfalls, expecially considering that they were one of the first hosting providers in Europe to offer IPv6.
IPv6 on OVH works. The problem is how “it works” and the number of limitations that you have (expecially compared to a different hosting provider).

Kimsufi

It all started when I bought two kimsufi servers on OVH. One was personal (I was hosting a research website: www.rpki.me), another one was for my university club.

/128

The first thing that I’ve noted was the subnet size: all kimsufi servers are sold with a /128 IPv6 prefix.
OVH is currently announcing on public BGP the following prefix for the european region: 2001:41d0::/32 .The latest RFC regarding IPv6 address assignments suggests to assign a /48 prefix to each end-site. Let’s assume that OVH would assign a /48 for each server. This means that there are 2^(48-32) = 65’000 subnets available.
Has OVH more than 65k servers? Probably yes. Then they could buy a new /32 prefix and address 65k more servers. A /32 IPv6 prefix is not so expensive and if they have so many servers, money should not be an issue. In fact they have three /32, one for Asia, one for Europe and one for Canada.

OK, so maybe /48 should be reserved to big customers, while others may use a  /56 instead, which is less than the best RFC suggestion, but still enough to allow each server to split the subnet in multiple /64s (for example for VM networks, VPNs, etc..). In this case they would have 2^(56-32) = 16’777’216 subnets /56. Has OVH more than 16M servers? I don’t think so. So a /56 could be reasonable, maybe even a larger allocation for each one.

But let’s say that they completely ignore the possibility for someone to be interested in having mutiple subnets, but of course you may use multiple addresses on your server. Maybe they would choose a /64 assignment for each site in order to change the address assignment method in the future to SLAAC.
In case of a /64 assignment, they would have 2^(64-32) = 4’294’967’296 subnets /64. Has OVH more than 4G servers? No. Will they ever have that much? No.

What did they do? A /128. A single IPv6 address. Do you need another address on your server to handle mutiple websites or more services on the same port? They don’t care. Do you need to setup a VM using IPv6? Why don’t you choose the idiot idea of doing NAT on IPv6? Why not? (Just kidding, but NAT is bad).

RFC6177 also specifically says: “It is no longer recommended that /128s be given out. While there may be some cases where assigning only a single address may be justified, a site, by definition, implies multiple subnets and multiple devices“. You may argue that a dedicated server is not “a site”. However I would consider it a site since the datacenter is not your too and you get addresses as a service from another authority (which doesn’t know what you want to do with those addresses).

The only possible idea I have regarding the /128 is this: someone did generate a lot of traffic using a lot of addresses in a /64, and some crappy router that they have crashed or had problems due to fill of NDP cache table. So they thought it was a good idea to proceed in OVH-style to solve the problem. This problem could have been solved instead, by buying decent networking gear, setting limits on caches or delegating the subnet instead of generating NDP neighbor solicitation requests (as we will see later).

Hopefully, it is a lie. For some reason all kimsufi servers are setupped with a /128 IPv6 address, but you can use the whole /64 of the provided address just by changing the prefix length on your interface. It’s easy to understand that it’s a lie because they assign the IPv6 address ending with ::1 to each customer. So how could they use the same /64 for two customers, if both of them need that specific address of the subnet?

Why did they lie? I don’t know. Probably to let kimsufi customers know that they don’t deserve a decent service and that those servers are only for “testing purposes”,  as they like to underline.

Static addresses and stange routers

While on IPv4 you are stuck with static addresses or DHCP, in IPv6 you have more choice. You could use static, DHCPv6 for addresses and router advertisements for routes, SLAAC, both DHCPv6 and SLAAC or force one of the two, or you could use DHCPv6 PD (prefix delegation).
What did OVH choose? Everything static. It’s a server, could make sense. It’s annoying because you have to search for your own address on the manager, but you have to do only once.
OK, but given that I know my address, which is the address of the router?
A guide on the OVH website illustrate how to assign the address and the default route. Let’s ignore the fact that all the commands are based on the deprecated ifconfig tool. In order to setup a default route, you have to create a static route to a specific address (of the router) and then route all the traffic through that.
The router address is computed by picking your own /64, computing the /56 (even though the “/56” concept is not even discussed in the guide) containing it and adding ff:ff:ff:ff:ff, or more properly writing, by adding ff:00ff:00ff:00ff:00ff to the /56 prefix.
The example on the page is misleading because it’s considering a particular case (where the host’s address have the 4th group with a leading zero).
Moreover the page is even more misleading by saying “5x FF”, which is not correct, otherwise the address would be ….:ffff:ffff:ff00:0000.

My understanding is that each server has its own /64, there is a /56 containing some of these /64s and there is a single IPv6 router per /56 subnet. The router is located in the last /64 of the /56. Each host is able to reach the router directly on layer 2.

Why not using the address “all-zeros” or ending in “::1” of the last /64 for the router?
If the router is in the same layer 2, answering to neighbor solicitations, why not use a link-local address such as fe80::?

As a side note, OVH completely forgot to mention IPv6-reachable DNS servers to use.

Security? Isn’t that only for v4?

A friend of mine wrote me “you told me that the /128 is a /64, but I’m actually able to use the whole /56 !“. My friend was partially correct. You don’t have the whole /56, but nobody will stop you from stealing it. All servers of the same /56 seem to be in the same layer 2 network, and there is no protection in place preventing you to assign any address in the /56. The router will send a neighbor request for that address and you can reply to that. This is not an IPv6 security issue. This is a shitty setup issue.
I’m not sure if the same problem is present on IPv4 for the same network.

NDP vs. delegating the prefix

One of the main issues with the OVH IPv6 setup is that they do not delegate the IPv6 prefix. I will later show how this is valid also for servers other than the kimsufi.

When you send an IPv6 packet from the Internet to one of the IPv6 addresses of your server, the router responsable of forwarding the packet to your server will generate a neighbor solicitation request (part of the NDP protocol), asking which is the MAC address associated to the given destination address of the packet. Your server will receive the solicitation and reply with a neighbor advertisement.

This approach might be fine for an access network, but it’s not good for a server. A server might receive traffic on a lot of addresses, and this process will not scale and could eventually give problems to the NDP cache of the router (expecially if not properly implemented).
Let’s say your server has 2001:41d0:8:17d8::1/64 and you want to run several virtual machines inside the server using IPv6. There are only two ways for doing this: bridging the server’s network interface with the VM one (probably a bad idea), or route the traffic for a slice of the subnet to the vm (better idea).
So you split a slice of your subnet, in order to use it only for the VMs. You add a static route pointing 2001:41d0:8:17d8:1000:/65 to the hypervisor interface.
However, when a packet destinated to the VM arrives from the Internet through the OVH router, the router will send a neighbor solicitation in order to find who has the address. No one will reply, since the VM is routed behind the server and do not receive neighbor solicitations.

The dirty workaround is to use the NDP proxy function of the Linux kernel, in order to allow the server to reply on behalf of the VM.
The Linux kernel allows us to proxy only single addresses. So if we want to reply to neighbor solicitation for the whole /65 we have to compile and configure a NDP proxy daemon program, just to reply to these useless requests.

If we want to setup a VPN providing IPv4 and IPv6 connectivity, using the address space of the server, we will run in the exact same problem.

The solution to this problem, and also the previously mentioned security problem, is to get the server’s prefix delegated by the router. This is possible through DHCPv6 PD or more simply by creating a static route on the router in the datacenter, routing all the traffic for the /64 prefix to the server’s link-local address for example. Setupping the static route with an automatic script is matter of minutes to setup. This would also solve the problem of filling the NDP cache which probably conviced OVH to use /128s.

Reverse DNS delegation

This could be useful for a number of things, such as email servers which need the correct reverse DNS on the address they listen to.

From the kimsufi administration web interface it’s possible to configure a reverse DNS for each IPv4 address. It is also possible to configure a reverse DNS, but only for a single IPv6 address, not the whole subnet. OK. The server is sold as a /128, so it could make sense, but we will see later that this does not change much also for a server sold with a /64.

Kimsufi broken IPv6 connectivity

As I said, I had more than one kimsufi, purchased in different time frames. Sadly the second one came with IPv6 connectivity broken.

A friend told me “yes, probably it’s broken, but nobody will notice except you“.
Wrong: as soon as I’ve booted the fresh-installed system pre-configured by OVH, and I run apt-get update , the command started and immediately halted waiting for connection to repositories. APT by default is trying over IPv6, which was broken, so I was not even able to install anything, unless forcing the use of IPv4.

The problem

After some troubleshooting I’ve quickly found out that the IPv6 gateway (…ff:ff:ff:ff:ff) was not replying to neighbor solicitations. I beleve this was very likely to be a VRRP/HSRP or loopback address configuration error on their router.

The ticket

I did the “standard” process. On the kimsufi you can’t open a proper ticket, but you can send a message to the technical assistence. So I did, and I’ve been waiting for 5 days. No reply.
I’ve sent again.
After 6 days they reply “Please include the complete IP configuration in your reply. Once we can rule out any configuration errors I can then forward this on to our technicians and have them correct the issue.”

I thought it was clear enough already. Anyway I copy the output of all the commands clearly showing where the problem is.

In the meantime there is a thread on the kimsufi forum, with several people reporting the exact same issue. Many people wrote to the support and got no reply.

After 12 days they reply “The IPv4 configuration is incorrect. Your IP is isolated /32 shoud should be […]”. My problem was on IPv6, what are they talking about? I double check what they wrote and then I reply immediately saying that they didn’t understood the problem, and I also link the thread on the forum.

After few hours they reply “I can ask a technician to have a look into this but first I need to confirm you have a full backup of your data and can authorise an intervention before we proceed?”. I agree and wait.

2 days later: “Please configure the IP as described in the following link. It will then work fine. http://help.ovh.co.uk/Ipv4Ipv6 “. Of course they didn’t solve anything and following the guide was useless. I decide to ignore the problem and I setup an IPv6 tunnel with hurricane electric (perfectly working with a /48, but slower because tunneled far away).

So: IPv6 broken for more than 1 month and I gave up on trying again.

OVH-style solution

In the meantime, the number of people experiencing the same problem with IPv6 was increasing. Probably someone at OVH noticed the problem and decided to apply an OVH-style solution: filter all AAAA DNS records in the DNS replies sent out from OVH’s DNS server. In this way everyone can buy the server, boot and run apt-get update without noticing IPv6 being completely broken.

# host -t AAAA google.com
google.com has no AAAA record

# host -t AAAA google.com 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases:
google.com has IPv6 address 2a00:1450:4007:807::1001

I have no words.

SoYouStart

Now you may think: “OK. But you are speaking about Kimsufi servers. Those are test servers completely unreliabile, tought for tests, not real stuff”.
Sadly I’ve had to manage also a SoYouStart E3-SAT-1 , which is a production-grade server with SLA and assistance.

So a /48 finally?

No. The SoYouStart IPv6 setup is basically identical to the kimsufi’s one, except for the fact that they officially provide a /64. Yes: you are paying 30€/month of rent and you only get a single /64.

SLAAC? What is that? Ah yes, but it’s broken

When I was setting up the server I did the partitioning and distro install by myself in order to tune a couple of things, so the IPv6 setup was not there ready.

After booting the server, I’ve noticed that as soon as the network interface came up, I had a default route in the IPv6 routing table. Good! So they enabled SLAAC? No, because I got only the route, not the automatically assigned address. Anyway I got a route advertisment from the router, so I just have to assign the IPv6 address and I’m ready.

Everything works great. For about 30 minutes. After 30 minutes I don’t have a default route anymore. I bring down and up the interface, and now I have again the default route.
I notice however that, the default route learned through neighbor advertisments, has an expire time: it can only be used for 30 min. In that period of time the route should be reannounced periodically by the router. However, for some reason, OVH’s routers are not sending router advertisements periodically. They reply only when a router solicitation is sent.
However this is not DHCPv4. This is IPv6: the host, complying the standard, will only send a router solicitation when it brings up the interface. This should be enough since router advertisements are send periodically.

So after all you can’t use router advertisements , because they are broken in OVH.

So I disabled accepting router advertisments via sysctl and configured the static address as explained in the old guide.

The question here is: why on earth sending broken router advertisements if you don’t support them? Just disable them completely, otherwise people without the correct sysctl might get IPv6 broken by OVH routers.

Security

Security didn’t change from the Kimsufi. You can still use addresses of other people’s /64s.

NDP vs. Prefix delegation

Does OVH delegate the /64 IPv6 prefix instead of using NDP, on the soyoustart? No. Same as kimsufi.

Reverse DNS delegation

Do they allow you to delegate the reverse DNS zone of your /64? No. If you want you can add each address to the web interface, or using APIs, but they don’t delegate.

 

Online.net

I recently purchased a couple of servers from Online.net and I’ve experienced a much better setup instead:

Addressing and prefix delegation

Each user holding an online.net account can request a /48 IPv6 prefix. This is possible for all users, both if you take a 30€/month server or if you take a 5€/month one. This prefix can be splitted from the manager into several /56. Each /56 that you slice from the /48 , is associated with a special unique DUID.

The server receive the default route normally via router advertisements, then sends a DHCPv6 request containing the DUID of the chosen subnet to be used. Online.net DHCPv6 server will lookup which is the subnet associated to the DUID and will reply delegating the entire prefix to the host (as discussed before). So it’s possible to setup VPNs, VMs and other things without having to deal with NDP proxying.

The DHCPv6 daemon is continously running to renew the delegation of the prefix, and router advertisements are continously coming to the routing table.

Online.net also have good documentation explaining how to setup the DHCPv6 client correctly.

Reverse DNS delegation

In the manager interface is possible to specify two addresses of DNS servers authoritative for the reverse DNS of the /48.

Security

As far I can tell, because of the absence of neighbor solicitations, it’s not possible to steal addresses of other servers, nor I can see any “public” NDP traffic on the network interface.