Self-Reference on CodeGoalie

How shell scripting saves my side projects

Tue, 19 Jul 2016 17:58:03 -0400

I work on a lot of little side projects. Many of them are short-lived and small scale. I start them to learn a new technology, to learn a new concept, try an idea, or just to see if I can actually build something cool. I should also mention that my memory is terrible. I “remember” things by figuring them out again. Rote memorization is basically non-existent to me. I’ve recently come up with a new best practice for my side projects which has saved my butt several times now.

It’s a deploy.sh script and I put one in every project that I do which needs to be built, deployed, or executed beyond the typical compile step. This very blog has one. There are also a few best practices I’ve developed for making this file useful and not just another headache.

Runable

This seems like a no-brainer, but hear me out. This file should be executable on its own. It should not be deploy.rb. It should not need to be compiled or otherwise prepared to run. You are writing a script. Script away all of that tedium. If you need to deploy with ruby, make your deploy.sh simply contain ruby deploy.rb. The point here is that you can cd into your project and run this script. Period. If you have to remember to initialize your Python virtualenv first, you’ve missed the point. Set it and forget it.

Helpful

Not everything can be scripted away. Sometimes a deploy requires some unique information, such as an environment or version number. Every microservice I create is meant to run in a docker container on kubernetes. In order to automate building, tagging, pushing the image, and then updating the deployment, the deploy script needs to know the version number. Since I’ve deployed 10 times today to “fix” that one bug, I can easily remember to pass the version to the script: ./deploy.sh v1.0.1337. However, I won’t be so hip to that fact tomorrow. Good thing we are writing a script. Just add some checks and helpful error messages to guide your future self down the right path. Here’s an easy one for requiring an argument to your script:

#!/bin/bash
if [ -z "$1" ]
then
echo "Please provide a version number, like v35"
exit 1
fi
# other deploy code here

It may seem like a drag, but you’ll be so happy that you took the time to take care of your own self now.

Complete

Lastly, your deploy script needs to do every step on its own. It’s a grown-up script now and can handle it. If you need to do just one more thing after the deploy finishes, you might as well just rm deploy.sh. Even steps which you might consider “optional” now, need to be included. For example, this blog is hosted on AWS S3 and CDN’ed by CloudFront. It really is enough to only upload (read: sync; save those bits) the files. Eventually the cache will expire and the new content will be shown. However, in just one more line in the deploy.sh, I can invalidate the CloudFront distribution and see this awesome new post right away. If you think it’s optional, it’s required.

Well, those are my three tips for keeping your side project deployments sane.

Happy deploying!!

P.S. I’ve recently had a resurgence of shell scripting in general. In the past, I’ve certainly been one to (ab)use zsh’s history substring search to re-run the same set of commands over and again. See also: “Up 12 times then enter. 6 times.” By committing those mini-scripts to ~/bin/, it really has made me much more efficient. A practical example is streamlining my workflow at work. We use Github for code hosting and reviews. We also use YouTrack for our project management. I wrote a command line client for YouTrack and combined all the commands I was doing to checkout branches by story number or create new branches as well as marking stories In Progress/Under Review/etc. into shell scripts like review_story c-1234 or

$ start_story -t bug -s c-1234 -b this-is-a-heisenbug
Applying "Assignee: me In Progress" to story c-1234
Switched to new branch 'bugs/c-1234-this-is-a-heisenbug'

Get creative and really push yourself to save some time. Again, you’ll thank yourself, and then me. ;)

Delete multiple branches with git

Wed, 25 Jun 2014 00:00:00 +0000

When working on projects with multiple developers following a feature branching workflow, you can end up with a lot of branch laying around. I’ve been trying to optimize my branch cleanup workflow for a while now and had a big breakthrough recently.

UPDATE: I now use a shell script which has the meat of the git-sweep functionality. Thanks to @pengwynn for letting me steal that.

Before we get to that, I must mention git-sweep. It has been revolutionary to keeping branches (both locally and remote) clean. If you care at all about the output of git branch, install that now.

However, there’s another simpler trick I just recently discovered. Deleting multiple local branches with one command.

$ git branch -D old-branch-1 old-branch-2

That’s it! Simply pass as many branches into branch -D as you want and git will gladly delete them all.

But why do you need this?

I just said that git-sweep cleans up local and remote branches after they are merged into master. So, why would we ever need to delete local branches? For one, you might not merge branches into master, but still want to delete them. But more likely, you’ll need to do this after pulling down a colleague’s branch to review. Once you give them the green light, it’s best practice for them to rebase onto master (interactively to prevent accidentally merging !fixup commits). However, you will have one state of the branch locally, but they’ve merged a different, rebased state. This is just enough of a change to fool git-sweep, and git herself, into missing the fact that these branches are the same. Luckily, there is a quick way to identify and remove those branches too.

Handily, git-branch has a super verbose mode which can be shown with the “double v” flag (-vv). This displays the branch name, HEAD commit, remote tracking branch, and HEAD commit message.

$ git branch -vv
* gh-pages 123456 [origin/gh-pages] Show new features on the project website
master 654321 [origin/master] Merge features/super-new-feature into master

Additionally, super verbose mode will also show us when a remote tracking branch disappears (is deleted). Immediately following the tracking branch name, we’ll notice a “gone”.

$ git branch -vv
* master 654321 [origin/master] Merge features/super-new-feature into master
old-feature 523621 [origin/old-feature: gone] Users can see the brand new feature

I like to grep for that and filter out the branches which I don’t want to delete. Basically, my whole cleanup procedure goes like this:

$ git branch -vv | grep gone]
old-feature-1 523621 [origin/old-feature-1: gone] Link to the new account page
old-feature-2 856956 [origin/old-feature-2: gone] Toggle turtle visibility with feature flag
$ git branch -D old-feature-1 old-feature-2
Deleted branch old-feature-1 (was 523621).
Deleted branch old-feature-2 (was 856956).

And there you have it, removing multiple branches in git with a bonus of finding branches who’ve lost their remote tracks.

Happy branching!!

– Chris

Zeus gem error: "panic: runtime error: invalid memory address or nil pointer dereference"

Tue, 12 Feb 2013 00:00:00 +0000

I was getting this error and the excellent Zeus rails environment loader gem was crashing making it a huge bummer to dev.

Luckily, I found a fix in one of the open issues on the project.

It was easy to fix, just run a bundle install to add the missing gems:

Easy as that. Hope this helps someone else!

– Chris

Ruby on Rails/ActiveSupport: To delegate or not to delegate

Fri, 12 Aug 2011 00:00:00 +0000

This is an account of my adventure with delegation from problem to apparent solution to bug in ActiveSupport to you’re doing it wrong!

TL;DR

Delegation is used when you want to forward a method call from on class/model to an associated class/model. The Ruby on Rails Guides use the example of a User model which has a Profile associated with it. This is a great example with a little subtlety which I overlooked. If you aren’t famaliar with delegation, go ahead and get yourself learned. I won’t wait for you though. This is text just come back when you’re up to speed.

# app/views/drivers/show.html.haml
.truck_number
-if @driver.truck
= @driver.truck.number
-else
Not Driving any Truck

NBD. Here’s our problem:

The Problem

As when any good application which acts as a font-end for a database, we track history on all edits, etc. So, when we update the Driver in a Truck, we want to add history for the Truck and the Driver. If the Driver is already in another Truck we want to show, in his history, that he moved from Truck 1001 into Truck 2002. We’ll say we have a create_history method included into our Entity classes whose declaration looks something like:

def create_history(attribute, prev_value, new_value)

So, we want to call something like:

@driver.create_history('Truck Number', @driver.truck.number, @new_truck.number)

Jackpot! Works perfectly. We get the old Truck number (1001) from the @driver instance and the new Truck number (2002) directly from the @truck instance. We are geniuses. Hold the phone though. What if that Driver didn’t have a previous truck… Spoiler Alert: NoMethodError: undefined method ‘number’ for nil:NilClass I guess we could check that a Truck exists each time we want to access any attributes for Driver.truck. But, really, what do we expect Driver.truck.number to be if the Driver is not in a Truck? Nil would work…

Delegate: for science!

I’ll allow it. Let’s delegate and allow_nil:

# app/models/driver.rb
class Driver < ActiveRecord::Model
delegate :number, :to => :truck, :prefix => true, :allow_nil => true
# ...
end

Works! Now, we’ve got history showing: Driver had no Truck and moved into Truck 2002. Geniusness times 2. Delegation to perferction.

Sometime later…

Instead of the Truck number, maybe we want to store the id of the Truck in the history. Perhaps we know the truck number may change over time and want to link up the correct Truck record. Seems easy:

delegate :id, to: :truck, prefix: true, allow_nil: true

But, when our Driver doesn’t have a ‘previous’ truck:

RuntimeError - Called id for nil, which would mistakenly be 4 -- if you really wanted the id of nil, use object_id

WTF? I allowed nil. I know he doesn’t have a previous truck. I just want my nil back and I’ll go about my business. Let’s get our hard hats on and get digging into the source. I’ll save you some googling and just tell you that the Delegation module is in ActiveSupport in the Core Extensions under Module. Scroll, scroll, scroll. There! Line 136! They(we)(whomever) are only rescuing from NoMethodErrors. Calling id on nil raises a RuntimeError, which isn’t rescued and bubbles up to us. This is me stomping my feet and wining. My knee jerk reaction was to just rescue RuntimeErrors too:

rescue NoMethodError, RuntimeError

We’re gonna be Famous

It works! AND, how cool are we? We get to submit a bug fix to Rails. Uber-cool. Uh-oh, uneasyness…crap… It just doesn’t feel right to just catch any old RuntimeError and possibly, silently fail to nil. However, the urge to try and submit a simple ‘fix’ to Rails was strong. Instead, I started doing more research into delegate and Ruby Exception handling. Here’s what I’ve come up with:

The Moral of the Story

You should delegate when you want to hide an architectural aspect and expose the expected abstraction. In the Ruby on Rails Guides example, we expect a user to have a name through @user.name even through name actually resided in Profile. However, in our example, a Driver isn’t expected to have a truck_id. A truck has an id, but a driver does not have a truck_id.

One of the classic ‘bad smells’ in code for refactoring is using a temporary variable in place a a query. Ruby and its optional parantheses, blurs the lines between variable, attribute and method. It’s easy to forget what you are doing and assume everything is an attribute. Why shouldn’t a Driver have a truck_id? The same reason a Truck doesn’t have a CDL. But, we can query the truck_id for the kTruck associated with a Driver with a method.

After all that, I should have just written a query method in Driver as such:

def truck_id
self.truck.id if self.truck
end

Too easy…

UPDATE 3/16/2013

As I convert this blog to Octopress and reread these posts, I’m see some imporvements. Instead of the code above, the try method is really what we want to use here:

 def truck_id
self.truck.try(:id)
end

On Object, try will attempt to send the parameter as a message to the invoking object. So, it works just like calling the method. However, on nil, try simply returns nil. It effectively exactly replaces the first implementation nicely.

– Chris

Ruby Auto-Responder for Postfix/Vmail

Tue, 15 Feb 2011 00:00:00 +0000

Not knowing perl, I set out to write my own script in Ruby to send auto-responses from e-mail addresses setup in a vmail folder structure. You don’t need to use postfix, just have the ‘new’ folder where new messages are stored.

I recently moved our company’s e-mail server to a new VPS on slicehost and started using postfix. I followed this great tutorial on How-ToForge (except for the squirrelMail part) to setup a mysql database with the vmail folders to store the mail. I couldn’t get the Autoresponder in the tutorial to function correctly. Actually, it was ‘eating’ random mail, which was a weeks worth of headache in itself. So, we rolled out without any auto-response capabilities.

I’ve been half-heartedly looking for another solution every few weeks, but haven’t found anything other than a few Perl scripts (I’m not really famaliar with perl). So, yesterday I decided to write a ruby script to send auto-responses. I wanted to use text files to hold each autoresponse. Then, I’d just move them int and out of a folder to activate and deactivate them. I also took advantage of the ‘new’ folder within the vmail structure to reply to mail newer than the autoresponder config file.

The format of the config files is:

The message can be multiple lines until the end of the file.

For each config file in the directory, the script parses the file for the above information. Then it will check that user’s new mail folder for files. Any new mail file which was modified (sent) after the modification date of the config file is parsed to get the sender’s address(es)*. A new e-mail is composed using benprew’s pony gem and sent to each of the senders. Finally, the script ‘touches’ the config file to move the modification date newer than the messages for which it just responded.

The nifty parts of the script are iterating over a directory listing of files:

and using the stat method of a file object to get the modified dateTime:

and lastly, using the FileUtils module to ‘touch’ a file:

Here’s the full source:

– Chris

Keep ssh sessions from timing out

Thu, 20 May 2010 00:00:00 +0000

The Ubuntu Blog has a nice lil' article about keeping SSH sessions alive It basically boils down to editing your /etc/ssh/ssh_config file and adding the following:

# /etc/ssh/ssh_config
ServerAliveInterval 5

The number is the number of seconds to send the small keep alive which keeps the connection open. Ubuntu Blog suggests changing it from 5 to 240 or 300 (4 or 5 minutes).

– Chris

Installing PianoBar - Console client for Pandora Radio

Tue, 26 Jan 2010 00:00:00 +0000

PianoBar is a console client for Pandora Internet Radio. For me, this is a huge discovery. If you run Ubuntu, the flash player for Pandora can be a pain in the ass to install. Plus, on my netbook, the flash plugin for Chrome usually eats about 40-55% CPU constantly. A console client for Pandora will resolve all these issues.

However, the install instructions in the readme at the PianoBar Github Repo are very sparse and for someone pretty new to linux (like me) may be a little daunting. In the next few paragraphs, I will try to flesh out the installation instructions for PianoBar on Ubuntu linux.

1. Get the PianoBar source.

I downloaded the .tar.gz from http://6xq.net/html/00/17.html and extracted it with

tar -xzf PromyLOPh-pianobar-e079b45.tar.gz

2. Try to Make the PianoBar source.

I say try because for me it didn’t work. I was missing some of the required dependencies. But, you might not be missing the same ones as me. So, try to make it and then follow the instructions below for the dependencies you are missing.

cd PromyLOPh-pianobar-e079b45/
cmake .

3. Get and install libao

Libao is a cross-platform audio library for playing audio. I followed these instructions from Linux from Scratch to install it. Download the source from that page. Extract the .tar.gz:

tar -xzf libao-0.8.8

Then, from withing the libao-0.8.8 directory, configure the library with:

./configure --prefix=/usr && make

Finally, install it as root:

sudo make install && sudo install -v -m644 README /usr/share/doc/libao-0.8.8

Then, I retried step 2 to see that I didn’t need the LIBAO anymore.

4. Install FAAD2

Download the bootstrapped .tar.gz from AudioCoding.com. Extract the .tar.gz with:

tar -xzf f@d2-2.7

There are installation instructions in the INSTALL file in the extracted folder. I’ll summarize what worked for me. Configure it:

./configure

Make it:

make

Install it:

sudo make install

Then, I tried to make pianobar again. It still said it couldn’t find libmad, but it turned out I didn’t need it.

5. Install PianoBar.

After running the cmake from the extracted PianoBar folder, you should see that the object files were made. Then, run make:

make

And install:

sudo make install

If all goes well, you will be able to run it:

./src/pianobar

Enter your pandora username/email address and password. Choose your station. Hear music!

UPDATE:

I recently used this post to install pianobar on another computer and was getting pianobar: error while loading shared libraries: libf@d.so.2: cannot open shared object file: No such file or directory I needed to update the Shared Libraries (You can read more about Linux Shared Libraries from this page, specifically 3.5). I tried ldd /usr/local.bin/pianobar to see which shared libraries where being used by PianoBar and did see that libf@d/so/2 was not found. linux-gate.so.1 => (0x0044c000) libf@d.so.2 => not found libao.seeo.2 => /usr/lib/libao.so.2 (0x00edb000) libpthread.so.0 => /lib/tls/i686/cmov/libpthread.so.0 (0x00507000) libm.so.6 => /lib/tls/i686/cmov/libibm.so.6 (0x00794000) libc.so.6 => /lib/tls/i686/cmov/libc.so.6 (0x00982000) libdl.so.2 => /lib/tls/i686/cmov/libdl.so.2 (0x00f20000) /lib/0x00f20000ld-linux.so.2 (0x00f4e000) But I could find the libf@d.so.2 file in the /usr/local/lib/directory. Therefore, I needed to run ldconfig: sudo ldconfig Then I ran the ldd again and saw that libf@d.so.2 was now found in /usr/local/lib/ and PianoBar was now running again. Also, I learned how to automatically sign in and change the keyboard short cuts.

6. PianoBar config

You can change the configuration of PianoBar to switch up the keyboard short cuts and automatically login on run. To do this, you need to create and edit a file named ‘config’ in the .config/pianobar/ within your home directory.

vim ~/.config/pianobar/config

To automatically login, add the following two lines

user = pandoraUserName
password = pandoraPassword

I also found that changing the “Loving Song” keyboard short cut from + to = worked out better, since I didn’t have to press Shift+=.

act_songlove = =

– Chris

Notepad++ Color Configurator and styler.xml

Fri, 08 Jan 2010 00:00:00 +0000

Before I start talking about colors, let’s talk font. Consolas. That’s all you need to know. By far the best programming font. Get it. Use it. Love it.

Recently I started learning Ruby on Rails and I found RailsCasts with a ton of Rails focused screencasts. I believe he use TextMate for mac. I hadn’t really thought much about color in my code before watching some of these screencasts. But, it actually helps, a lot! I use Notepad++ in Windows and they have a great styling system. You can use the in program Style Confirurator or you can directly modify the XML style sheet.

Style Configurator

The Style Configurator is a dialog within Notepad++ accessible through the Settings menu. The really great thing about the Configurator is that the styles you choose are updated live in your code. So, open up a file with a lot of code. Then open the Configurator and move it over to another screen. If you don’t have more than one monitor, check the Transparency box in the lower right and you will be able to see through the Configurator. The slider there sets the Transparency level. Now you can see all of your code and pick styles for them.

Start with the very first ‘language,’ Global Styles. These not only include default styles of text but many of the stuff around your code, such as line number margin, inactive and active tab color, and the edge. The edge is great to keep your lines of code to a certain length for printing or displaying in a terminal. I also set the global font to Consolas. Also, be sure to set the current line color and selected text background color.

Now you are ready to style for you language. I usu PHP mostly at work so I’ll go though that one. Hopefully, I’ll be familiar with Ruby soon enough to write a post about Ruby styles.

QUESTION MARK - This is the style of the opening and closing php tags ('')
DEFAULT - Text that doesn’t match any of the other styles will be this.
STRING - Any string enclosed with double quotes
STRING VARIABLE - variables enclosed withing literal strings: “The Count is $count”
SIMPLESTRING - These are string enclosed with single quotes. (I used a very similar green to STRING)
WORD - These are the keywords of PHP. There are a bunch defined. You can also define some of your own in the blank box.
NUMBER - Number literals, including array indices
VARIABLE - Any word started with a $
COMMENT - Text between matching multi-line comments ('/' and ‘/')
COMMENTLINE - Text on a line after and including two forward slashes (//)
OPERATOR - Operators (+,=,-,etc.), matched parenthesis, brackets, curly brackets and logical operators

Stylers.xml

The stylers.xml file contains all the settings from the Configurator used by Notepad++. In addition to using the Configurator, you can manually edit this file. Or, download one from the web. The Notepad++ site has some under Theme Files. Joy Boner (read the about page) has 60 themes. I didn’t check any of these out, but it might be worth a shot if you really don’t want to set your own styles.

I also put my personal stylers.xml as a gist on github. Feel free to fork it.

If you have any other Notepad++ styling tips, comment below!

–Chris