Tag Archives: docker

Quirks of DNS traffic with Docker Compose

Recently I had a scenario where I wanted to restrict the network traffic coming out of certain processes I started inside a container, so they could only do the minimum required for them to work, and not reach anything else outside the container. In order to explain what I found, let’s imagine that my process only wants to make a HEAD HTTP request to http://www.google.com (on port 80, not 443).

It will obviously need to send packets with destination port 80, and packets with destination port 53 so it can make DNS requests to resolve http://www.google.com. So let’s implement a quick setup with iptables to accomplish this. We’ll use the following Dockerfile that installs curl, iptables, and dnsutils on top of the default Ubuntu image, so we can test our scenario.

Dockerfile

FROM ubuntu:latest
RUN apt-get update && apt-get install -y curl iptables dnsutils

And the following docker-compose.yml file to help us build and run our container.

docker-compose.yml

version: "3.4"
services:
  my-container:
    build:
      context: .
    image: <your-docker-repo>/my-custom-image
    cap_add:
      - NET_ADMIN
    command: tail -f /dev/null

The scenario I want to talk about only happens when starting services with Docker Compose, not when starting containers directly with docker run, so using a docker-compose.yml file is necessary even if it feels a bit overkill. Note we specify the NET_ADMIN capability for the container, which we need so we can use iptables, and a command that will keep the container running, so we can connect to it after Docker Compose starts it.

Now we run docker-compose -p test up -d in the folder that contains both our files, Docker Compose builds the image and starts a container. We can then connect to that container with docker exec -it test_my-container_1.

Let’s start by verifying that we can make our HEAD request to http://www.google.com:

HEAD request works

Great. Now let’s set up the iptables rules discussed above and make sure they look right.

iptables --append OUTPUT --destination 127.0.0.1 --jump ACCEPT
iptables --append OUTPUT --protocol tcp --dport 80 --jump ACCEPT
iptables --append OUTPUT --protocol udp --dport 53 --jump ACCEPT
iptables --append OUTPUT --jump DROP
iptables -L -v
Set up iptables rules

We add the rule for localhost just to make sure that we don’t break anything that’s connecting to the machine itself (without it, the rest of this scenario won’t work as expected).

Now we test curl --head http://www.google.com again to make sure everything’s fine… but it says it cannot resolve the host! Furthermore, nslookup http://www.google.com times out. And checking the iptables rules we see 5 packets dropped by the last rule, but none accepted by the rule for UDP port 53. How come?

CURL does not resolve host, nslookup times out

Well, it turns out that when Docker Compose creates a service, it creates iptables rules in another table (the NAT table) to reroute certain things through the Docker infrastructure. In particular, it changes the port of DNS requests from 53 to something else. You can see this by running iptables -L -v -t nat:

iptables rules in the NAT table

Here we can see that there’s a rule mapping UDP port 53 to 53789, when the request is going to IP 127.0.0.11 (where Docker hosts its DNS resolver). So if we now add another iptables rule for that port to our setup, we’ll see that our curl command works again!

CURL works again after adding new iptables rule

However, that port is not static, so the approach that I ended up taking was to create a rule to allow any packet with destination IP 127.0.0.11, which is the one where Docker hosts its DNS server, and the only one for which it maps ports.

Conclusion

If you plan to mess with DNS network traffic in your containers and you use Docker Compose to start them, be aware that Docker sets up rules to change the destination port for DNS requests.

Visual Studio, Docker Cloud hooks, and UTF-8 with signature

Today I ran into an issue trying to make custom hooks in Docker Cloud work. I first tried the default post-push hook to add another tag to the built image, straight from the documentation:

#!/bin/bash
docker tag $IMAGE_NAME $DOCKER_REPO:$SOURCE_COMMIT
docker push $DOCKER_REPO:$SOURCE_COMMIT

But the logs in Docker Cloud had the following error: Could not execute hook at 'hooks/post_push'. Is it missing a #! line?

Error in Docker Cloud build log

It was not missing that line, and I got the same error whether I used Windows or Unix end-of-line characters.

After a while I remembered an issue I’ve had with Visual Studio at other times which is that it will save files with encoding “UTF-8 with signature”. The “with signature” part is why sometimes other consumers of the file (in this case Docker Cloud) cannot read it correctly.

What “UTF-8 with signature” does is add an invisible sequence of bytes called a BOM (Byte Order Mark) to the beginning of the file.

However, let’s look at this answer in Stack Overflow:

The UTF-8 BOM is a sequence of bytes (EF BB BF) that allows the reader to identify a file as being encoded in UTF-8.

Normally, the BOM is used to signal the endianness of an encoding, but since endianness is irrelevant to UTF-8, the BOM is unnecessary.

According to the Unicode standard, the BOM for UTF-8 files is not recommended:

Even if it was recommended, I was almost certain that the BOM was the culprit of my issue. I needed the file to be saved as “UTF-8 without signature” (without that invisible sequence of bytes at the beginning of the file). The way to do that is to open your file in Visual Studio and go to File -> Save As. When the dialog comes up, first make sure Visual Studio didn’t add .* to the end of your filename (it probably does this for files that have no extension), otherwise nothing will happen when you try the next step. Now click on the arrow on the right side of the “Save” button, select “Save with Encoding…” (if you didn’t remove .* from the end of your filename, nothing will happen here) and set the Encoding option to “Unicode (UTF-8 without signature) – Codepage 65001”.

How to save a file as UTF-8 without signature – Step 1

How to save a file as UTF-8 without signature – Step 2

Save your file, and retry whatever it is you were doing, it should work this time. At least the Docker Cloud build did :).