DNSSEC Validation at Spark NZ

Introduction

DNSSEC is a technology that adds security to the DNS, by having the DNS data cryptographically signed. Deployment of DNSSEC helps the whole Internet ecosystem by providing means to authenticate websites and other resources, and to fight against attacks like cache poisoning. DNSSEC deployment can be measured in two different areas: by the number of domains signed and by the number of resolvers validating DNSSEC data. Tracking deployment at the zone level has been done by several parties, like Internet Society and this beautiful map, Verisign Labs and the adoption in .net and .com, and ICANN tracking the number of Top Level Domains signed. At the validation side, APNIC publishes their DNSSEC Validation Map to track the number of users covered by DNSSEC validation per country.

NZRS, as registry and DNS operator for .nz, signed the .nz Top Level Domain back in 2011, and all .nz Second Level Domains later in 2012.

Changes in the landscape

During the last NZNOG meeting in Rotorua, Geoff Huston presented an interesting result from APNIC's continuous research in DNSSEC validation deployment in New Zealand: the country reached a level of 15% of validation among a sample of users. Although these numbers are well below the level of adoption in other countries, like the US with 21.6%, these are at the same level of Australia and a lot better than the UK with 5%, this result carries a surprise: the same metric was sitting at the 8% level in December 2014.

New Zealand nearly doubled their DNSSEC validation adoption in a single month? That's too good to be true.

Geoff's report indicates that apparently Spark NZ started doing DNSSEC validation during December 2014.

NZRS is in a privileged position to confirm the nature of this change, given we operate the DNS for .nz and as Spark NZ being a major ISP provider, they generate a large amount of traffic to our DNS servers. So we dug into the large dataset of DNS traffic, to understand what the change was about.

Let's establish first how traffic normally looks like. We identified the source addresses that represent Spark NZ resolvers, and count the number of non-DNSSEC queries per day for each one. In the figure below, each line represents the number of queries sent by an specific address. Before Dec 15th, there are four addresses sending between half a million to a million queries per day, but after that day an extra set of four addresses were added, and the total query load was redistributed. This kind of change is normal in a large ISP, to distribute the DNS traffic among multiple servers for resilience.

The next step was to take a look at the number of DNSSEC-related queries coming from the same set of addresses in the same period. Now things get interesting:

Before Dec 15th, Spark NZ resolvers were hardly sending DNSSEC queries for .nz to our nameservers, but with the flick of a button, they ramp up to half a million queries in a day per address, roughly two million queries a day. Apparently, the new addresses from Spark NZ started doing validation, as well as balance the DNS load from their customers. Merging the two datasets clarifies this question:

From our data, we can conclude on Dec 15th, Spark NZ added four new DNS resolvers that load balance the DNS traffic coming from their customers, and those new DNS resolvers do DNSSEC validation, the old four addresses don't do validation.

The view from inside Spark

To detect how validation is operating for Spark customers, we need to be inside Spark network to use their resolvers, and have a DNSSEC signed domain for testing. Fortunately in NZRS we have a UFB Spark NZ connection in demo, and we can use radionz.co.nz a fully signed domain name. According to the information in Spark NZ website, the main DNS resolver is 122.56.237.1

So, once connected to Spark, we query their resolver using dig, command line tool.

dig a radionz.co.nz @122.56.237.1 +dnssec ; <<>> DiG 9.8.3-P1 <<>> a radionz.co.nz @122.56.237.1 +dnssec ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38841 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 4000 ;; QUESTION SECTION: ;radionz.co.nz. IN A ;; ANSWER SECTION: radionz.co.nz. 1215 IN A 103.14.3.1
;; Query time: 25 msec ;; SERVER: 122.56.237.1#53(122.56.237.1) ;; WHEN: Wed Feb 25 12:38:06 2015 ;; MSG SIZE rcvd: 58

This result is a surprise! We were expecting the string ad (for Authenticated Data) as part of the flags for the response (line starting with ;; flags), and also DNSSEC data (called signatures), but none of those were returned. Did we run it wrong? The same query, ran exactly three seconds later, yields this result:

dig a radionz.co.nz @122.56.237.1 +dnssec ; <<>> DiG 9.8.3-P1 <<>> a radionz.co.nz @122.56.237.1 +dnssec ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43278 ;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 4, ADDITIONAL: 6 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags: do; udp: 4096 ;; QUESTION SECTION: ;radionz.co.nz. IN A ;; ANSWER SECTION: radionz.co.nz. 3367 IN A 103.14.3.1
radionz.co.nz. 3367 IN RRSIG A 8 3 3600 20150325141206 20150223131206 45573 radionz.co.nz. T7lBeF35xVZp4pnxwD/5z9W2pce+qXrCpx2ITdUB+/U7GP7EeOsY9Kmm rvTBwgpyn27dECetiBhCzBWYOA2vdzPW4zsy9A5kI7qQ+nhGsPFBKy2f 44ozDPMMDTdZTvVgAfqBh/PWEgh9KGJ5nuHmluYerdIzQbncVsC7kMgn Mb0=
;; AUTHORITY SECTION: radionz.co.nz. 1650 IN NS ns1.metaname.net.
radionz.co.nz. 1650 IN NS ns3.metaname.net.
radionz.co.nz. 1650 IN NS ns2.metaname.net.
radionz.co.nz. 1650 IN RRSIG NS 8 3 3600 20150325141206 20150223131206 45573 radionz.co.nz. SQRDtz1z/kFpqGbEGnxUjZX7Qx1ClSsqv3ko/fz0AZdzw85O8indt78X mAd0c2pdsd7lDau/D1u/5vCGf9/dEAlqtlMTHQqCEGXCDyabobYHRqlY uQ8A2LRvcW/V8Cs6Ef8KX7frye2ayxZ1Xi5os6MFgJDvrGDAI9M6W0uN JYg=
;; ADDITIONAL SECTION: ns1.metaname.net. 2686 IN AAAA 2001:4428:200:80f6::1
ns2.metaname.net. 106559 IN A 103.11.126.252
ns2.metaname.net. 150766 IN AAAA 2001:470:67:c8::1
ns3.metaname.net. 106559 IN A 103.11.126.244
ns3.metaname.net. 25705 IN AAAA 2001:4428:200:8125::1
;; Query time: 25 msec ;; SERVER: 122.56.237.1#53(122.56.237.1) ;; WHEN: Wed Feb 25 12:38:09 2015 ;; MSG SIZE rcvd: 586

Now we do get DNSSEC data and the mark indicating this has been cryptographically validated! A few more attempts (not shown) give us the idea our queries are being load balanced, with some of them hitting the validating resolvers, and some hitting the "old" resolvers.

Can we automate this testing from different Spark NZ customers? Fortunately for Internet researchers, the RIPE Atlas project (https://atlas.ripe.net) exists, and there are three probes hosted in the Spark NZ network. Using RIPE Atlas API to create measurements, we ask those three probes to send DNS queries for < A, www.radionz.co.nz > to Spark NZ main resolver every 60 seconds for 10 minutes, and check if the answers obtained are consistent.

The table below summarises the result of the measurement. Three probes were used, identified by the probe_id column, and queries are ordered by the time they were sent (NZ localtime). It's clear there is no consistency in the responses, with some being authenticated and some are not. The presence of load balancing is also detected by comparing the time and TTL difference between consecutive queries getting the same DNSSEC validation status. In case where both differences are equal or very similar, means we hit the same server. In cases where there is a big gap between differences, means we hit different servers that yield the same validation result. We can infer this based on a property of DNS TTL in caches, that gets decremented monotonically by the server as time passes by.

Probe ID

Time of query

DNSSEC validated?

TTL in response

Time difference

TTL difference

Hit same server?

74

2015-02-25 16:20:55

False

928

N/A

N/A

N/A

74

2015-02-25 16:22:01

True

184

N/A

N/A

N/A

74

2015-02-25 16:23:01

True

124

60

60

Yes

74

2015-02-25 16:23:58

False

690

183

238

Likely not

74

2015-02-25 16:24:52

False

636

54

54

Yes

74

2015-02-25 16:25:55

False

628

63

8

Likely not

74

2015-02-25 16:26:52

True

2370

231

-2246

Likely not

74

2015-02-25 16:27:51

True

2311

59

59

Yes

17594

2015-02-25 16:20:53

False

798

N/A

N/A

N/A

17594

2015-02-25 16:21:58

False

500

65

298

Likely Not

17594

2015-02-25 16:23:06

False

433

68

67

Yes

17594

2015-02-25 16:24:00

True

3432

N/A

N/A

N/A

17594

2015-02-25 16:24:53

False

325

107

108

Yes

17594

2015-02-25 16:26:01

True

1741

121

1691

Likely not

17594

2015-02-25 16:27:02

False

3581

129

-3256

Likely not

17594

2015-02-25 16:27:54

False

3529

52

52

Yes

22769

2015-02-25 16:20:36

True

2745

N/A

N/A

N/A

22769

2015-02-25 16:21:34

False

833

N/A

N/A

N/A

22769

2015-02-25 16:22:37

False

769

63

64

Yes

22769

2015-02-25 16:23:43

False

704

66

65

Yes

22769

2015-02-25 16:24:42

True

22

246

2723

Likely not

22769

2015-02-25 16:25:41

False

585

118

119

Yes

22769

2015-02-25 16:26:40

True

3506

118

-3484

Likely not

22769

2015-02-25 16:27:43

False

464

122

121

Yes

Global Impact

The reader may think a change like this only produces effects within New Zealand, but in an interconnected world, that's no longer true. Working together with SIDN, the registry for .nl domain names, they were able to repeat our analysis and count the number of DNSSEC queries they received from Spark NZ addresses in the same period. .nl is a good option to check this, because they have over 2.2 million signed domains.

Although not at the same scale as .nz, mainly because popularity of .nl domain names in New Zealand and smaller data set used for this query, traffic from Spark NZ resolvers went from a few queries per day, up to 2500 queries per day including DNSSEC traffic.

Conclusions

NZRS is a big advocate of DNSSEC adoption, at both signing and validation side. We are very happy to see Spark NZ taking this step to enable validation, that together with the use of DANE, will enable a broad set of application to securely exchange traffic across the Internet. We are looking forward to have this service deployed widely to all Spark customers, and have consistently validated answers to DNS inside their network.