DNS over HTTPs and Unbound

, ,

TL;DR: I wanted to secure my internal DNS-traffic, which I did with Unbound - and I ended up writing an Ansible role for that purpose. This proved to be much easier than I anticipated, however getting my clients to use said secure resolver turned out to be a lot harder than I originally thought.

I recently decided to run my own recursing resolver for my servers and my laptop. Not only would that enhance the privacy within my network, but it would also allow me to log queries and responses. That in turn opens up a whole new world of possibilities, including feeding the logged data into an Elastic-stack to, combined with DHCP-logs, build a poor man’s SIEM or, if I open up the resolver to more people, build a passive DNS-database.

All modern Linux distributions have Unbound in their repositories, and the configuration is so simple that you can easily get a simple resolver up and running within a couple of minutes.

After doing so I took a step back and asked myself what could be improved. The obvious answer: cryptography. Getting Unbound to validate DNSSEC, at least on Debian, is a matter of basically setting a configuration switch. Encrypting requests to the resolver, and to the upstream servers used for recursion, requires a bit more work.

There are three major technologies to secure DNS-traffic:

DNSCrypt used to be the “hot shit” at the time when there were massive improvements made to privacy on the Internet, fueled by the impact of the leaks by Edward Snowden about global mass surveillance.

But despite its popularity it never became standardized, and while dnscrypt-proxy is a mature piece of software (and I truly like some of the ideas about how to enhance the privacy of DNS-requests, the Internet is filled with a ton of reports of issues during installation, updates or operation. And while that’s something I could deal with on my laptop, I don’t want one of the central services responsible for ensuring other vital services (like my mailserver) of mine to be anything other but stable.

After some reading I decided to go with DNS over HTTPs. Once you have your own certificate (in my case I’m my own CA) the configuration is pretty straightforward, all you need are the following lines in /etc/unbound/unbound.conf:

port: 443
tls-service-key: /etc/unbound/private.key
tls-service-pem: /etc/unbound/public.crt

The above handles the configuration for the resolver itself, for the upstream connection it makes sense to also enable forward-ssl-upstream: yes - either on a global or per-upstream level. I mean, you technically could use plaintext DNS for the forwarded queries, but that would kind of defeat the point.

After a restart of the service I tried querying a domain from one of the systems that were whitelisted in the ACL, without success. Checking the status of Unbound I noticed that the service was running successfully, but not listening on any port. What?

(Sidenote: dig doesn’t support DoH or DoT. Debian has kdig in its sources, in the knot-dnsutils package. That works.)

The reason became clear when I took a look at journalctl -u unbound:

Feb  7 18:27:46 rs1 unbound: [39059:0] warning: Unbound is not compiled with nghttp2. This is required to use DNS-over-HTTPS.

There’s an open bug report in the Debian bugtracker, but it has been stale for months, so my hopes of DoH being supported by the package provided by Debian aren’t high. In theory I could, with relative ease, recompile Unbound to support DoH. In practice the effort isn’t worth the time in my case:

  • I don’t need the privacy gains DoH would provide, because the goal is ensuring that no plaintext queries and responses are sent around the Internet
  • I don’t see any danger of anyone, such as my ISP, deciding to specifically block port 853
  • I don’t need any of the modern features that DoH would come with, such as the ability to ‘push’ DNS-data - in fact I actively don’t want those features

So I turned my attention to DNS-over-TLS, with the base configuration being nearly identical:

tls-service-key: /etc/unbound/private.key
tls-service-pem: /etc/unbound/public.crt
tls-port: 853

Once more, it makes sense to use upstream resolvers that support DoT as well. And this time I was not sabotaged by the package being compiled ‘in the wrong way’, DNS-resolution was working. Time to get my clients to use it!

Apple has been aggressivelly marketing themselves as a privacy-oriented company for years now, including high-profile court cases against federal authorities. Since they value the privacy of its customers, there surely is an easy way to get the network manager of macOS to encrypt DNS-queries.

Before anyone tries to lynch me for mentioning “Apple” and “Privacy” in the same sentence: That was a joke. I am under no illusion that as a user of Apple products I’m as much the product as Google users, except that I don’t even get the benefit of cheaper devices.

So it didn’t catch me fully by surprise that Apple doesn’t support any of the ways to secure DNS-traffic, neither DoT nor DoH, at all. Luckily there is software that runs on macOS that does - meet Stubby.

I wanted to make a joke about “wait, wrong link, I’m sorry”, but as it turns out, that’s actually the inspiration for the name of stubby, a small application that acts as a local stub resolver for forwarding DNS-queries, with a focus on privacy. And luckily, it’s available via Homebrew.

After installing it I modified /opt/homebrew/etc/stubby/stubby.yml to my liking by adding my resolver:

############################ DEFAULT UPSTREAMS  ################################
# My resolver
  - address_data: $myaddress
      tls_auth_name: "$myservername"

(On a sidenote: This configuration file contained the most comprehensive list of available resolvers that claim to respect your privacy that I have seen so far. Nice.) After that I ran a quick brew services run stubby to test my configuration and whas promptly greeted by this:

Bootstrap failed: 5: Input/output error
Try re-running the command as root for richer errors.
Error: Failure while executing; `/bin/launchctl bootstrap gui/501 /Users/gmmi/Library/LaunchAgents/homebrew.mxcl.stubby.plist` exited with 5.

After some searching on the net I learned that this was because there have been some changes and deprecations to launchctl, which is what Homebrew services are using under the hood, with MacOS Monterey, which effectively broke them to some extent. And there’s pretty much nothing I can do about it. I decided on a brutish solution, adding the following line to my ~/.zshrc:

if ! pgrep "stubby" &> /dev/null; then
    sudo /opt/homebrew/opt/stubby/bin/stubby -C /opt/homebrew/etc/stubby/stubby.yml -g &> /dev/null
        echo "[*] Starting stubby .."
	fi

Next in line were the critical variables in this equation, my servers. All of them run Debian, mostly Stable, but with some Testing mixed in here and there.

Without trying to start a flamewar heated discussion, I am not the biggest fan of systemd. But since I am a fan of using onboard-tools I opted to use systemd-resolved, which should have supported DoH for a while now. I added the following to /etc/systemd/resolved.conf, pointed /etc/resolv.conf to 127.0.0.1 and restarted the service:

[Resolve]
DNS=$myresolver
DNSSEC=yes
DNSOverTLS=yes

In theory I should now have working resolution, including DNSSEC-validation and DoT. But once more, the emphasis is on the word “theory”.

sigok.verteiltesysteme.net: resolve call failed: All attempts to contact name servers or networks failed

At this point I started to get annoyed by the constant roadblocks I was hitting. Despite the urge to just walk away I took a peek at the logs:

Feb 07 21:53:16 wfh systemd-resolved[371099]: Failed to invoke gnutls_handshake: Error in the certificate verification.

This was the day that I learned systemd-resolved was using GnuTLS instead of OpenSSL. And that was one of those learned lessons that created a lot of question marks in my head, because I also learned that the trust store of GnuTLS differed from the regular trust store in Debian. A blog post from a couple of years ago gave me the feeling that this was going to be tricky, and several other posts on different sites didn’t give me too much confidence either.

Also OpenSSL and GNUTLS (the most widely used certificate processing libraries used to handle signed certificates) behave differently in their treatment of certs which also complicates the issue.

The official documentation for GnuTLS wasn’t really helpful either. There are a few references about about system-wide configurations and files, but nothing that pointed me towards anything specifically applicable in practice. I tried to use the tools contained in gnutls-bin, but while certtool had a command line switch to manually load a certificate authority (--load-ca-certificate) neither the help nor the manpage contained any references to a default path.

The German messageboard ubuntuusers.de claimed that /etc/ssl/ssl/certs/ca-certificates.crt would be used by GnuTLS, which would make sense - but that’s not the case, since my certificate authority has been imported to the system-wide trust store.

What eventually got me a breakthrough was a bugreport in the Ubuntu-bugtracker:

On the other hand if I set dnsovertls to opportunistic, things seem to work, but the log reports that systemd-resolved is “Using degraded feature set UDP for DNS server”.

It didn’t solve my problem of GnuTLS being unable to verify my certificate, but it presented a viable alternative in my case - my resolver was the only one configured, and the only thing it served was DNS over TLS. There was no plaintext resolving available, thus making a fallback impossible.

Nonetheless, I wasn’t satisfied. It felt wrong to put effort into a clean, dependable solution on the end of the resolver, only for my client-side setup to be half-hearted.

A short trip to my mental drawing board later I decided to simply replicate what I am doing on my laptop - stubby, which was one apt install and a confiugration-copy-paste-orgy away. And my now beloved stubby didn’t disappoint, resolution worked like a charm. Yeah!

As mentioned in the TL;DR at the beginning of this post, the endeavour to secure my DNS-requests turned out to be way more adventurous than I had anticipated when I started. My expectation was that, in 2021, it should be a relatively straightforward process.

And while getting a DNSSEC-aware, encrypted connection providing DNS-resolver up and running required a bit of debugging, it was the client-side that surprised me. I understand that vendors would want the default setting to be ye good ol' plaintext DNS. But deciding to actively not support an encrypted alternative was not something that I would have thought to be common.

Even more surprising was the fact that Windows, of all the operating systems out there, supports DNS-over-HTTPS out of the box for nearly two years now. While Apple doesn’t support it at all and the built-in support for Linux, through systemd-resolved is half-assed. Remember the times when Microsoft was way behind when it came to implementing security technologies?

Pepperidge Farm remembers