Load balancing services with AWS Route53 DNS health checks

0 Flares Twitter 0 Facebook 0 LinkedIn 0 Email -- 0 Flares ×

DNS is a great technology that everyone uses over internet. How would you reach a given website if you weren’t able to solve its name to the IP address? Would you memorize the public IP addresses of any website you want to reach? No, and with IPv6 coming in the future, DNS will become even more important for internet consumption. But DNS has one drawback: its records are usually static, and if a platform is dynamic and spawn/removes instances on the fly, it needs to have a way to modify the DNS records that are published, so that a non-reachable instances is not even listed.

The use case: Veeam Cloud Connect

I’ve started to look at these types of services when working on Veeam Cloud Connect. I wrote both the Reference Architecture for v8, and the new one for v9. One of the suggested way to publish Veeam Cloud Connect has always been DNS Round Robin: it’s a easy solution to publish multiple instances using the same DNS hostname, like in my lab:

These three gateways can load balance between them without the need of any external load balancer, and they are all reachable with a common DNS hostname:

that is mapped to all the three addresses. The client component only needs to talk with one of the gateways, and from there on the software is able to switch to any of the gateways if needed. But what if I want to immediately remove a gateway that’s not reachable, like during a maintenance period, without having to manually edit and update my DNS configuration? And obviously re-add the same record once the gateway is up again?

Here’s where Health Checks in AWS Route 53 come into play.

AWS Route53 DNS health checks

All my domains and their dns zones are already hosted into AWS Route 53. I think it’s a really compelling dns service, and it’s so easy to edit dns zones. So, when I first built my Veeam Cloud Connect I created my new records into the virtualtothecore.com hosted zone:

DNS round robin

The default routing policy to rotate the records is set to Simple: this is what’s usually known as DNS Round Robin. Round-robin DNS works by responding to DNS requests not only with a single potential IP address, but with one out of a list of potential IP addresses corresponding to several servers that host identical services. The order in which IP addresses from the list are returned is the basis for the term round robin. With each DNS response, the IP address sequence in the list is permuted. Usually, basic IP clients attempt connections with the first address returned from a DNS query, so that on different connection attempts, clients would receive service from different providers, thus distributing the overall load among servers.

The limit of round robin DNS is mainly record caching in the DNS hierarchy itself, as well as client-side address caching and reuse. This is why my records have a TTL (Time To Live) of 60 seconds. But we can do even better using Health Checks. With Health Checks, Amazon Route 53 checks the health of the resources and respond to DNS queries using only the healthy resources. For my scenario, I’m choosing an Active-active failover: this failover configuration is to be used when you want all of your resources to be available the majority of the time. When a resource becomes unavailable, Amazon Route 53 can detect that it’s unhealthy and stop including it when responding to queries.

Health Checks configuration

In the Health Checks area of Route 53, we select “Create health check”, and we configure it like this:

Health Check for a Veeam Cloud Gateway

With this setup, the Veeam Cloud Gateway will be tested every 30 seconds, and after 3 failures (that is, 90 seconds) it will be considered unavailable. You can also receive an alarm via e-mail when the check is triggered:

Health Check alarm

We repeat the same configuration for the other two gateways, and as soon as the checks are configured, we can immediately trace information like health and latency of the monitored services:

Health check latency measures

Changing DNS configuration to use health checks and weighted resources

Now that we have the correct health checks in place, it’s time to modify our DNS configuration to use them. If you look again at the DNS configuration I captured in the first screenshot, a single A record is holding all the three IP addresses. In order to switch to the new configuration, we need to split those IPs over thre different records, and remove the previous one. The final result of each new record will be like this:

New weighted record

Let’s see what are the selected options:

– the record is a type A, and the nam for each is cc.virtualtothecore.com. As we will have one record for each IP, nothing changes from what we had before
– TTL is again very low, 60 seconds. In this way the cache at the client side will always be clean and when a failover operation will occur, the time needed to switch from one record to another will be short
– Value is the IP used on each gateway, one for each new record set
– Routing policy: the previous policy was set to “Simple”, which is the default policy. But you cannot create multiple records with the same name using the simple policy. So, we switch to the Weighted policy: each record has a value of 10, and the sum of them all is 30. Each record will be passed to a client doing a dns resolution with a frequency determined by the weight. Since each of them is 10, the percentage will be 10 / (10*3) = 33%. So, exactly as before, DNS round robin will cycle through the three records with no special preference for any.
– Set ID is only a description of the record. I’ve just wrote in this field the name of the cloud gateway associated with the public IP listed in the record
– Health Check: here I linked each record set with the corresponding Health Check.

Health checks in action

From now on, Health Checks verify each 30 seconds that a cloud gateway can be reached over TCP port 6180. After 3 failed attempts, the record will be disabled, and this will be done automatically by Route53 by setting the weight of the record to 0. The overall failover time should be as short as 150 seconds (2,5 minutes): 90 seconds for the 3 failed health checks to happen, plus a worst case of a client dns cache that was just updated before the last check and it would take additional 60 seconds to expire the TTL of the record. These values can be lowered even more by using fast checks (each 10 seconds).

Let’s do some tests. Under normal conditions, if I check the resolution of cc.virtualtothecore.com from my computer I obtain:

nslookup cc.virtualtothecore.com

Name:cc.virtualtothecore.com

Address: 185.62.37.98

nslookup cc.virtualtothecore.com

Name:cc.virtualtothecore.com
Address: 185.62.37.100

 
This is totally expected. There have been two minutes of delay between the first check and the second one, and surely some other tenant I’m hosting has refreshed the information in between my two checks, and has received the .99 record.
Now, I power down gtw1. Veeam lists the gateway as unavailable:
GTW1 is down
AWS notifies me about the failure:
Health check failure notification
But what’s most important, DNS server doesn’t uses anmore the ip .98 as a possible answer. Any DNS resolution I try, using different sources, always end up with the .99 and the .100 response. And this will be the result as long as the health checks are in this state:
Health check status
As soon as I restart the gateway GTW1, everything starts working again as expected, also also the .98 record is passed to DNS clients.
Thanks to solutions like this, for example I can patch and reboot at will my different gateways (or any other service I’m publishing…) without being worried everytime of updating my DNS entries. I’ve always believed that Route53 is an amazing DNS service, but the additional combination of conditional routing and health check makes it an incredible service.

4 thoughts on “Load balancing services with AWS Route53 DNS health checks

  1. Your article is much better than AWS official pages. Thanks!

  2. Luca, great article.

    Just a quick question, is there anyway with Amazon we can create a floating IP like digital ocean and point it to the outside amazon servers? they have an Elastic IP but does not have the option to point to an outside amazon server.

    • Not sure what you mean by outside amazon server, if it’s a server NOT running in AWS, then the answer is no, their ip can only be used against amazon services.

Comments are closed.