How to Build a Tiny Httpd Container

TL;DR: Smith can shrink a 69MB apache httpd container down to under 2MB, resulting in space savings of 97%, while making it more secure and easier to operate. Constructing microcontainers can require some tricky modifications, however, so some practice with the tool is useful.

Smith is a microcontainer builder that I have been working on for the past few months. The readme shows how to build an ultra-simple microcontainer, but some software can be trickier to “microize”. Apache httpd is a good example of a tricky application because it includes configuration files, has some complex permissions requirements, and loads dynamic libraries. All of these things require some extra steps beyond the basics. Below I detail the process of building a working httpd microcontainer.

First Steps

Smith can build directly from yum repositories and rpms using mock, but it can also “microize” existing docker containers. For the purposes of this process we will use the official httpd container on Docker Hub as our base. The latest tag is built from debian jessie and weighs in at 69MB compressed. There is also an alpine image at a much leaner 28MB. I will use the debian-based latest image, but a similar process could be performed with the alpine image as well.

The first step is to download and install smith:

# before starting ensure you have golang 1.6 or later installedgit clone github.com/oracle/smithcd smithmakesudo make install

Now we create a new directory for our build:

mkdir smith-httpdcd smith-httpd

Next, we download the httpd image so we can use it as a base. (Note that you can also specify the package as the full download path in the smith.yaml, but the image is not cached so it will re-download every time we do a build. Since we are going to be building quite a few times, it is best to download the image manually.)

smith download -r https://registry-1.docker.io/library/httpd \-i httpd.tar.gz

The next step is to create a base yaml file. I have no idea where the httpd binary is installed in the image so we will just guess at its location. The arg of -DFOREGROUND tells httpd not to daemonize so that docker can pick up logs in the usual manner:

cat << EOF > smith.yamltype: ocipackage: httpd.tar.gzpaths:- /usr/bin/httpdcmd:- /usr/bin/httpd- -DFOREGROUNDEOF

Now we can try a build:

smith -i smith-httpd.tar.gz# …# WARN[0002] chmod failed: chmod: cannot access ‘/usr/bin/httpd’: No such file or directory# WARN[0002] Could not make paths readable: Process chmod -R go+rX /usr/bin/httpd exited with status 1# …

It looks like /usr/bin/httpd is not the right location. Fortunately, smith unpacks the layers from our source image into a directory for us, so we can look around. The directory is /tmp/smith-unpack-$(id -u). Let’s find the right binary:

find /tmp/smith-unpack-$(id -u) -name httpd# /tmp/smith-unpack-1000/usr/local/apache2/bin/httpd

It looks like the binary has been installed into /usr/local/apache2/bin/httpd. In fact if we look around a bit in the unpacked image, it looks like everything has been installed under /usr/local/apache2. Let’s take a bit of a shortcut and just grab the whole directory:

cat << EOF > smith.yamltype: ocipackage: httpd.tar.gzpaths:- /usr/local/apache2/cmd:- /usr/local/apache2/bin/httpd- -DFOREGROUNDEOF

Now we can build again. Since we haven’t changed our source image, we can skip the unpack step by passing in -f when we build:

smith -f -i smith-httpd.tar.gz

Testing our Image

Docker doesn’t yet support importing an oci image directly via docker load, so we have to push the image to a repository to use it. Docker Hub works just fine. Once you create an account and make a new repository, you can push your image as follows:

smith upload -r \https://<user>:<pass>@registry-1.docker.io/<user>/<repo> \-i smith-httpd.tar.gz

You can optionally append :<tag> to the repository if you want to tag the image as something other than latest. Once the upload is complete, you can run the image via docker run -it --rm <user>/<repo> or you can follow along by using the images that i have built and uploaded:

docker run -it --rm vishvananda/smith-httpd:first# …# /usr/local/apache2/bin/httpd: error while loading shared libraries: libuuid.so.1: cannot open shared object file: No such file or directory

Hmm, looks like it doesn’t quite work.

Fixing our Image

Smith is smart enough to download any libraries needed by the binaries selected for the image as long as they are not opened via dlopen at runtime. Unfortunately, apache does dlopen some libraries, including libuuid.so.1 as seen in the error message above. Sometimes you can find libraries that might be opened by running strings on the binary, but the best method I have found is to try to run the binary and see what fails. Usually the error message will tell you exactly what file is needed.

In this case, we need to add /lib/x86_64-linux-gnu/libuuid.so.1 to our image. This is easily done by appending that filename to paths in our yaml file. After rebuilding and re-pushing the image, remember to use a new tag or use docker pull <user>/<repo> to force it to get the updated version before docker run. If you run the new image, you will see a similar error message complaining about librt.so.1. To save you some time, here is a new yaml with the full list of dynamically loaded libraries which you will need:

cat << EOF > smith.yamltype: ocipackage: httpd.tar.gzpaths:- /usr/local/apache2/- /lib/x86_64-linux-gnu/libuuid.so.1- /lib/x86_64-linux-gnu/librt.so.1- /lib/x86_64-linux-gnu/libdl.so.2- /lib/x86_64-linux-gnu/libexpat.so.1- /lib/x86_64-linux-gnu/libgcc_s.so.1- /lib/x86_64-linux-gnu/libcrypt.so.1cmd:- /usr/local/apache2/bin/httpd- -DFOREGROUNDEOF

After another build/upload/pull we can run it again:

docker run -it --rm vishvananda/smith-httpd:second# …# AH00543: httpd: bad user name daemon

To understand what is going wrong here, we need to digress a bit. In most installations apache httpd will run as root. It generally needs to run as root so that it can bind a port below 1024. Of course, it is a security risk to have your web server running as root, so it then drops privileges to a user/group that is specified in the config file.

In an ideal world we would remove the User and Group commands from the httpd config file and run the binary as an unprivileged user. This can be done, but it requires us to work around the lack of binding low ports in one of the following ways:

1. Change the config to make httpd listen on an unprivileged port
2. pass --sysctl net.ipv4.ip_unprivileged_port_start=0 to docker run (Requires kernel >= 4.11)
3. pass --cap-add NET_BIND_SERVICE to docker run and also set the proper capability bit on the httpd binary (There was briefly support for ambient capabilities in docker which would obviate the need for setting the capability bit, but it was reverted and has not yet made it back in to docker).

These options aren’t ideal for everyone, so the creators of the httpd image use the tried-and-true method of starting as root and dropping to another user. In general we do not want microcontainers to run as root, but smith supports it for cases like this. In order to enable the feature we are going to want to set two booleans in our yaml: root and nss. root sets up the image to run as root, and nss installs the required linux bits to make username and group lookups work in the container. Let’s make a new yaml with these settings:

cat << EOF > smith.yamltype: ocipackage: httpd.tar.gznss: trueroot: truepaths:- /usr/local/apache2/- /lib/x86_64-linux-gnu/libuuid.so.1- /lib/x86_64-linux-gnu/librt.so.1- /lib/x86_64-linux-gnu/libdl.so.2- /lib/x86_64-linux-gnu/libexpat.so.1- /lib/x86_64-linux-gnu/libgcc_s.so.1- /lib/x86_64-linux-gnu/libcrypt.so.1cmd:- /usr/local/apache2/bin/httpd- -DFOREGROUNDEOF

We also want apache to drop to the user that owns the files in the image in order to avoid any permissions issues. The default user:group for all files in microcontainers built with smith is smith:smith. We are going to have to modify the httpd config to use these names. Additionally, by convention, we put config files for microcontainer processes in /read, so let’s create an overlay that does that:

# make an overlay read directorymkdir -p rootfs/read# copy the config file out of the imagecp /tmp/smith-unpack-$(id -u)/usr/local/apache2/conf/httpd.conf \rootfs/read/# replace the file in our image with symlink to our filemkdir -p rootfs/usr/local/apache2/conf/ln -s /read/httpd.conf rootfs/usr/local/apache2/conf/httpd.conf

Now we can replace the User and Group with the desired settings:

sed -i -e ‘s/User daemon/User smith/’ \-e ‘s/Group daemon/Group smith/’ rootfs/read/httpd.conf

After a build/upload/pull, we can run the container and expose a port on the host:

docker run -it -p 80:80/tcp --rm vishvananda/smith-httpd:third

And a quick curl from another shell verifies that httpd is working:

curl localhost# <html><body><h1>It works!</h1></body></html>

Getting Truly “Micro”

Now that the container is working we can work on minimizing it further. There is actually quite a bit of extraneous stuff in the apache2 directory. The only bin file we are using is httpd itself. We do need the modules, but we can pick out just the ones that are enabled in httpd.conf. This is our final build, so we can also switch to pulling the image directly from Docker Hub:

cat << EOF > smith.yamlpackage: https://registry-1.docker.io/library/httpdnss: trueroot: truepaths:- /usr/local/apache2/bin/httpd# dynamic dependencies- /lib/x86_64-linux-gnu/libuuid.so.1- /lib/x86_64-linux-gnu/librt.so.1- /lib/x86_64-linux-gnu/libdl.so.2- /lib/x86_64-linux-gnu/libexpat.so.1- /lib/x86_64-linux-gnu/libgcc_s.so.1- /lib/x86_64-linux-gnu/libcrypt.so.1# modules- /usr/local/apache2/modules/mod_authn_file.so- /usr/local/apache2/modules/mod_authn_core.so- /usr/local/apache2/modules/mod_authz_host.so- /usr/local/apache2/modules/mod_authz_groupfile.so- /usr/local/apache2/modules/mod_authz_user.so- /usr/local/apache2/modules/mod_authz_core.so- /usr/local/apache2/modules/mod_access_compat.so- /usr/local/apache2/modules/mod_auth_basic.so- /usr/local/apache2/modules/mod_reqtimeout.so- /usr/local/apache2/modules/mod_filter.so- /usr/local/apache2/modules/mod_mime.so- /usr/local/apache2/modules/mod_log_config.so- /usr/local/apache2/modules/mod_env.so- /usr/local/apache2/modules/mod_headers.so- /usr/local/apache2/modules/mod_setenvif.so- /usr/local/apache2/modules/mod_version.so- /usr/local/apache2/modules/mod_unixd.so- /usr/local/apache2/modules/mod_status.so- /usr/local/apache2/modules/mod_autoindex.so- /usr/local/apache2/modules/mod_dir.so- /usr/local/apache2/modules/mod_alias.socmd:- /usr/local/apache2/bin/httpd- -DFOREGROUNDEOF

To follow the microcontainer conventions, we also want to make a few more tweaks to the conf and rootfs:

# move the mime.types config file into the read directorycp /tmp/smith-unpack-$(id -u)/usr/local/apache2/conf/mime.types \rootfs/read/ln -s /read/mime.types rootfs/usr/local/apache2/conf/mime.types# create a directory for serving filesmkdir -p rootfs/read/www# copy the index file into our new directorycp /tmp/smith-unpack-$(id -u)/usr/local/apache2/htdocs/index.html \ rootfs/read/www/# change the serve directory in the config# (note we could also have accomplished this with a symlink)sed -i ‘s;/usr/local/apache2/htdocs;/read/www;’ \ rootfs/read/httpd.conf# store the pidfile in /runsed -i ‘/Group smith/a PidFile /run/httpd.pid’ rootfs/read/httpd.conf

After a build/upload/pull, we run our microcontainer as securely as possible:

docker run -it -p 80:80/tcp --read-only --tmpfs /run \-v httpd-read:/read:ro --rm vishvananda/smith-httpd

We have access to the served directory on the host so we can make changes:

sudo sed -i ‘s/It works/Hello world/’ \/var/lib/docker/volumes/httpd-read/_data/www/index.htmlcurl localhost# <html><body><h1>Hello world!</h1></body></html>

Building From a Yum Repository

The yum based build using mock is surprisingly similar. After setting up mock, you can simply replace the package line in the yaml with package: httpd. The files will be in different locations than in the docker image, of course, and in this case there is no unpacked image to explore.

To find the files you want to pull out, use /usr/bin/mock --shell after the first build to look around in the mock chroot. You can also use /usr/bin/mock --copyout to copy files from the chroot into your local directory. The build/upload/run process is the same, and I encourage you to try to generate a microcontainer without explicit instructions.

Conclusion

The first working version of our microcontainer clocked in at a svelte 6MB and the final version is under 2MB. That is less than 3% of the size of the original container! In addition we have removed all extraneous binaries and made the filesystem of the container read-only, improving the operational characteristics and security of the container itself. I hope this inspires you to build some other microcontainers with smith.