A few years ago I spent the Labor Day weekend building a platform for using Docker for local development on an OSX host. Surprisingly, I still find this to be this solution to be the best, most complete solution. But I haven’t ever written about the project and figure it is about time.

If you want to jump to the punchline, the project described here is located at https://github.com/adamaig/dev-with-docker-on-ubuntu. It has continued to evolve and hold up as my preferred development platform since late 2016.

The History

Back in 2016, the projects I worked on were all RubyOnRails, deployed via Capistrano on shared EC2 instances. My team hadn’t been involved in the setup of the boxes and they had been provisioned by hand. In the short time, I had been at the company we’d suffered a from the following avoidable missteps: another team decided to modify a shared NodeJS install, without communication, which broke the npm installation we depended on; the OSX (SSL) + rvm + Brew + rubyracer + V8 combination had cost devs days of finding the right incantation to fix the build when setting up one of the projects or coming back to a project that they hadn’t touched in a while; and finally, one dev introduced code that somehow only worked under chruby installed ruby, but segfaulted with rvm installed rubies. So with Docker becoming a hot-topic in 2016, and in collaboration with Ben, a new DevOps hire at the company, advocating for it, I thought it might be time to try to stabilize our development and deployment environments.

Initially, we started building against the DockerMachine implementation. I got the first app set up as a container quickly and started to develop a small feature using DockerMachine and shared filesystem mapping. I ran two instances of the app image: one for the app, and another with the test watcher. But it was horrifically slow! Something like 43 seconds to load the first page. I tried modifying the mount mapping to use NFS, and that got it down to 18 seconds, but as a development flow that is still a really long time to wait for a page to refresh. Beyond the slow page loads, both of these solutions also made it so that my test watcher container didn’t work, because filesystem events aren’t supported on those mount types.

So, I spent a little time working on a file watcher that used rsync to execute a 2-way sync, but it was only triggered by changes on the host. Using rsync this way meant that file events would be generated in the DockerMachine filesystem, and the test runner executing in a docker container would execute as expected. This meant that I needed to disable the direct mount for the filesystem into the DockerMachine instance, otherwise the system might sync a file across the mount first, defeating the rsync-generated events.

With this rsync script, the page load time from the docker container was now within milliseconds of the native OSX execution, and my test runner was executing as soon as a file was modified, getting me back to my BDD bliss. Score! But still, the rsync solution was kludgy, and it wasn’t really developer-friendly.

The “Solution”

I really, really didn’t like any of the solutions I had explored, but I really didn’t want to continue to suffer from the problems that ad hoc developer setups created. I had spent a fair bit of time working with Vagrant and Ansible as part of a previous project and as I settled into my Labor Day weekend hack session, I thought I’d try to assemble all the learnings of the last few months into a new VM provisioning script. The output would be a repeatable development environment (code as infrastructure), that enabled using native host tooling when desired.

I selected an Ubuntu box as the base and set to work. I wanted to be able to distribute the setup as a single Vagrantfile, so I stuck with inline shell provisioning. This kept the requirements to a minimum of VirtualBox + Vagrant. Because I am an avid Vim + Tmux user, I started with the thought that I could just ssh into the guest VM and do my development there. Because the Docker engine would be running in the VM and any file system events would be in the same machine both the test runner and app performance issues would be solved.

This solution was great for all the programming tasks. But it still lacked a few things I felt would provide a better experience for development.

Personally, I really hate having to remember what project is running on which port, and I hate the kinds of problems that are created by mixing shared deployments, like a dev instance of a service, with a local development instance. In the prior iterations of a solution, Ben had been exploring Consul for service discovery and had provided a nice consul + registrator docker-compose project that would provide DNS for any container running on the machine. By adding a DNS Resolver entry on the OSX host, any service on the host machine could find the docker based services, making it easy to use native browsers and other tools to work against the container applications. The complete solution required adding a routing table entry, but that just required a simple script that could be hooked into the vagrant upcommand.

Now that I had service discovery with DNS and routing working, I turned to the last few details. I knew some of my colleagues weren’t as at home working in a remote Linux context or had a preference for GUI editors like RubyMine. While NFS wasn’t a good solution for mounting the filesystem _into_ the guest machine for the reasons given above, I thought it would be fast enough for the human experience of editing files and would enable developers to use native OSX editors without significant pain–This proved to be mostly true. To test this out I added a provisioning script that will export a project directory from the guest to the host. This enables easy access to files across the two systems, without imposing a significant penalty for most tasks. The only context I’ve seen this solution be insufficient so far has been with RubyMine circa early-2017. When running a specific spec, the entire project would first be indexed by the editor, which added an unacceptable upfront overhead of more than 20 seconds on that project relative to the same project running entirely in OSX.

A Little Polish

I was already a fan of having a dotfiles repo for quickly setting up new boxes, and I wanted to have the experience of working within the guest OS be as seamless as possible. So I added a few more provisioning scripts to the Vagrantfile to

1. create a user in the guest with the same name as on the host,
2. copy the ssh configs and keys into the guest,
3. install common tools
4. enable customization of the user account

This made it even more convenient to delete a VM and rebuild without too much delay. Over the next few months, we found that it took maybe 30 minutes to fully set up a new developer’s machine with the projects they needed, and have them running and ready to start learning the codebase.

I also decided that I wanted to have DNS working on the guest. Not every project is going to be in docker, and I wanted to be able to use CLI tools like curl in the guest to hit the docker apps. I solved this by adding dnsmasq into the provisioning.

Conclusion

The project described above is located at https://github.com/adamaig/dev-with-docker-on-ubuntu. I continue to develop and evolve the project to suit my needs, and hopefully the needs of some other developers. The provisioning scripts are reasonably documented and should provide a guide to anyone looking to build their own system from these ideas. I’ve appreciated the uptake of the project to-date as validation that this was a real-itch, and it has bothered a fair number of people.

While this solution isn’t bullet-proof or quite as push-button as it could be, it has been very solid and gets out of my way to let me focus on programming efforts. The latest pleasure that I’m getting from this project is that on my company provided system, working in the DockerDesktop cause the security scanning software to cripple my development tooling performance because the files were constantly choking the scanner, and network traffic was bottlenecking. With the Vagrant guest solution neither is a problem.

The outstanding problem on my mind right now is that with all files being owned by the guest, it is easier to lose work due to a corrupted VM disk, or accidental VM deletion. It also requires having the VM running if you want to access the files. I think this will be easily solved by a Vagrant hook and rsync, but I haven’t spent time on it yet.

I’m also looking to add a local Kubernetes development tool-chain. I’ll post about both efforts in the next few months.

Coda

A few months after I developed this project, DockerDesktop hit beta. I was excited, curious, and ultimately let down. The way DockerDesktop broke (and still breaks) networking defeats the ability to do DNS resolution between the host and guest and the filesystem mapping is still noticeably slower than the native access my solution provides. This continues to be the case in mid-2020.

I’ve also explored Ubuntu’s Multipass briefly but found the documentation lacking, and didn’t feel like I had the same level of control of the system Vagrant + VirtualBox provided, and giving up the future flexibility to change distros is also something to consider.