docker on CodeGoalie

Cleaning Up Docker Images

Mon, 30 Dec 2019 16:13:36 -0500

After a while of using docker locally for development, you may notice that you have a lot of images and containers laying around on your disk that you no longer use. This post shows you an effective way to clean up 50 of GB or more of disk space.

The first, best thing to do is clean up dangling images. These are docker images that are not referenced by a container. These can be orphaned when containers are removed or intermediate steps that don’t get cleaned up. One quick command can remove them:

$ docker image prune

Typically, if you’ve been using docker for a while, this will clear up quite a bit of space. But, there’s more we can do.

The next step you might think to do is remove other images that you know you don’t use. We can list them with the docker images command. However, you won’t be able to delete them directly because they are being referenced by containers (otherwise they would have been removed by the prune command). So, let’s look at containers:

$ docker ps -a

Note: the a flag shows all containers not just those running. You probably don’t want to remove a container that’s currently running anyway.

Wow. That’s a long list. Way down at the bottom I noticed some old and temporary projects and experiments. Here’s an example of some less notables from my list:

68585b76a752 46d00b64e6cb "/bin/sh -c yarn" 13 months ago Exited (1) 13 months ago upbeat_tesla
3cb298f62318 3bc06921fcd0 "/bin/sh -c yarn" 14 months ago Exited (1) 14 months ago charming_benz
31e91aaa9111 3bc06921fcd0 "/bin/sh -c yarn" 14 months ago Exited (1) 14 months ago upbeat_lovelace
79428b687b22 6533ab88e0ff "basj" 16 months ago Created welcomely_web_run_75

Ok, so now copy and paste the container ID (first column) for each one you… J/K.

I also noticed that this output has the created time in the fourth column. This is helpful, but not particularly useful. I do have some long running projects and would prefer not to simply remove containers that I’ve created long ago.

However, the very next column shows the last time the container was executed. I would like to remove containers that I haven’t run in a while. We can grep for rows of the output which exited many months ago:

 $ docker ps -a | grep -E "Exited \([0-9]{1,3}\) 2[0-3] months ago"

This looks scary. But, I promise I actually wrote this by hand and it’s pretty simple. If I’m being honest, it’s probably too simple, but it worked for my case.

First, we use the E flag to enable the regex engine. Then we want to match rows with the string like “Existed () <some number more than 20> months ago”. Most of that is plain strings. Really just for the numbers we want to use the regex. The first is the exit code in the parenthesis. Most containers had 0 or 1 exit codes. But, a few had 137. So I just matched on any up to 3 digit number within the parenthesis: [0-9]{1,3}.

Second, we match a plain 2 then any units digit of months between 0 and 3, inclusively: 2[0-3]. My results had the oldest container stopping 23 months ago. You mileage may vary and I’ll leave it up to you to modify and re-run to suit your situation.

Now that we’ve got a good list of containers to delete, let’s use some command-line fu to save a ton of copy/pasting. Let’s pipe this filtered list into awk to get rid of the rest of the text save for the first hash (field).

awk '{print $1}'

Then, pass each row (which now only contains the ID hash of the container) into docker rm. To do this, we can use xargs:

xargs -i@ docker rm @

Putting it all together:

$ docker ps -a | grep -E "Exited \([0-9]{1,3}\) 2[0-3] months ago" | awk '{print $1}' | xargs -i@ docker rm @

Then, go back to the top and remove dangiling images:

$ docker image prune

There you have it. I removed containers I hadn’t run for more than 8 months or so and freed up almost 50GB. I hope you have similar or better results! Let me know how you fare: chris@codegoalie.com

Happy docker rm-ing!

Working with Containers: beyond the tutorial

Thu, 04 Feb 2016 15:11:50 -0500

Over the past few months, I’ve been getting my hands dirty with containers, Docker, and Kubernetes in an effort to get some hands on experience working toward microservices. I’ve been building up a small application to generate and serve Sudoku puzzles. My original goal was to see how many puzzles with unique solution I could actually find, but that’s another post entirely. There are already many great tutorials about Docker and Kubernetes on Google Container Engine, so I won’t go into getting started details here. This post is meant to explain some of the stickier points that I had to pickup the hard way. I hope to save you one or two headaches.

Turn Key Kubernetes

Typically, I’ve used AWS for my infrastructure needs, but Google Container Engine runs kubernetes for you and can spin up a new cluster with one command. Again, there are many tutorials about the specifics but here are a few things I didn’t know going in which would have been helpful.

Keeping costs down

By default your cluster is composed of 3 n1-standard-1 instances. When just getting started this is probably more than you need in both resources and cost. Since I was more interested in playing with containers than messing with compute engine instances (that’s the whole point of containers), I didn’t pay any attention to them, but you should. It’s super easy and quick to change both the number of instances and the instance types in your cluster.

Google Container Engine clusters are built on the idea of instance groups, which are just one or more instance defined by the same instance template. The template described the instances you want in your group with attributes like machine type and base image, etc. In the Google Cloud Console, you can design a new template right in the interface (pick a new machine type from a drop down) and use smaller or larger instances.

Scaling the number of instances in your cluster is even easier. Edit the Instance Group and change the “Number of Instances” textbox value. You can even enter 0 to stop getting billed for the instance time. This works great when you don’t need the instances to be up 24/7 while just playing around.

Persistent Storage

When a pod is restarted or rescheduled, it gets recreated from scratch. Anything that was stored on disk is gone. This can cause unexpected issues. For example, I was using redis for storing Sudoku puzzles and if the redis-master pod restarted, I lost all contents of the database. This can be unexpected because when running redis locally, the disk snapshots always remain even when restarting the redis-server process. With containers, this is not the case. If you need to store anything to disk that you care about, you’ll need to store that in a volume backed by a Google Persistent Disk. Once again, there are many tutorials on the specifics.

Stuck in CreatingContainer

When using a Google persistent disks, I was running into pods getting rescheduled onto new nodes (instance), but the disk remained attached to the previous node. Since Google Cloud persistent disks can only be attached to one node at a time, this caused the pod to hang during creation and never start. The quick fix to this is just detach the disk from it’s current node. Kubernetes will attach it to the correct node for you.

gcloud compute instances detach-disk <instance-name> --disk <disk-name>

There’s currently an effort to redesign the volumne mount/dismount system in Kubernetes to address this issue. I also think that as redis was maxing out the memory on the nodes and crashing, it was causing unnecessary havok on my cluster.

“Restarting” a pod

Early on my Sudoku generator was very naive and had exponention performance, so it would get “stuck” trying to come up with new puzzles. Locally, I was just restarting the process. But when I got it into the cluster, it wasn’t always clear if it was broken, or just getting held up (I did fix this and went from a puzzle every few minutes to a few puzzles a second). The great feature of using replication controllers is that it will keep the specified number of pods running. So to “restart” a pod that’s being managed by a replication controller, just delete it. The replication controller will bring up a new one. If you aren’t using a replication controller to manage your pods, do so.

Hopefully these tips will make your container journey a little smoother.