Bugs with Non-Alphanumeric Credentials for AWS

I've recently had a chance to be working with AWS and S3 and encountered a somewhat frustrating bug that I wanted to share. I was using boto3, an AWS SDK for Python. I had created a script that would list the EC2 instances that make up an EMR cluster. It worked fine with my credentials, but another user would be a SignatureDoesNotMatch error every time. We spent a lot of time comparing the environment setup and permissions on the two accounts, but everything lined up. We started guessing a little more wildly when we noticed that my AWS credentials (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) were entirely alphanumeric, but the other user's were not. My first thought was that there was some kind of encoding error, but I had no luck with that. After some searching I found this longstanding bug with the same issue. As it turns out having non-alphanumeric credentials can cause this error and has for at least a couple of years. The only fix is to regenerate the credentials until getting a set that is entirely alphanumeric.

I later ran into the same signature error trying to distcp some data from HDFS to S3, but I knew what to look for this time. HADOOP-3733 is a very similar issue to that in boto3, but this is fixed in Hadoop 2.8.0.

This was a troublesome bug for me, so hopefully this post helps someone else who encounters the same issue.

Blogging More Regularly

It has been a while since I have written anything substantial here. General busyness with life and work have kept me from getting around to writing anything. However, I am going to try and blog more regularly going forward. Starting out I'm going to try and post something once a week and see how that goes.

Writing Setup

For my first post in this attempt I am going to talk a little about my writing setup.

I've found that I really like distraction-free writing tools as opposed to a standard text editor or word processor. Being able to focus only on the text helps keep me in the zone. I've tried a few of these and I've come to like Ulysses the best. It's less minimalistic than some others that I have tried but has a lot of really useful features. You write with Markdown, which I'm already familiar with, and it has great export functionality. It also syncs via iCloud so I can work on either my iPad or my desktop computer.

I used to write exclusively on a computer at my desk, but I found this an impediment. If I have an idea of something to write about I don't necessarily want to go sit at my desk to put down the words. Instead I'm now usually using a keyboard attached to my iPad to write. It's a little smaller than a full-sized keyboard so my typing speed isn't up to par yet, but it's great to be able to sit on the couch and write.

Suppressing ad-handle-definition Warnings in Emacs

As I have been tweaking my Emacs configuration, I noticed that I was getting warnings like the following in *Messages* during Emacs startup:

I looked into it and these warnings are generated when functions are redefined with defadvice. Most of the warnings I received were coming from third-party packages I had installed, not my own configuration. In any case, they were not helpful, just noise during Emacs startup.

Like everything in Emacs, this is configurable. Turning off the warnings is as easy as adding:

To your .emacs file. I found I had to add it fairly early to ensure that all the warnings were removed.

New Year, New Tools

New Year, New Tools

It’s a new year, so I thought it would be a good time to review some new tools I started using over the last year.


Mosh is a replacement for SSH that handles intermittent connectivity. It will warn you if your connection drops and automatically reconnect. This handles situations like putting your laptop to sleep or changing between different connections (e.g. switching from Ethernet to Wifi). I’m connected to a bunch of different servers over the course of the day, so being able to put my laptop to sleep and have it wake back up with my remote sessions intact is incredibly useful. It’s hard to believe I used to work without it.

Mosh doesn’t support X11 forwarding or non-interactive uses of SSH, like port forwarding. However, I don’t use these particularly often, so in practice I’m not bothered by this. In any case, regular SSH is still available!


I had resisted changing shells for a long time. Bash is available pretty much everywhere. I had a nice .bashrc that I could put anywhere and have the environment I was used it. zsh had been recommended to me several times, but I was concerned about the effort needed to get up and running on a new machine. I liked being able to just drop a few dotfiles in my home directory – would I also have to install zsh?

I decided to try the switch after getting fed up with an issue with my bash prompt and long lines – characters would occasionally get doubled. I am really glad I made the switch. Needing to install zsh wasn’t as big of a deal as I had feared. I used oh-my-zsh to get started. This gave me a lot of the configuration I had done manually in my .bashrc out of the box. Plus it provides plugins for just about anything you might use in the shell. My favorite feature of zsh itself is the improved tab completion. It’s hard to explain without experiencing it – just give zsh a try.


Much like bash, I had used screen for a while and it just worked. I had a configuration I liked and I didn’t need to think about it. When I was switching to zsh, I decided to reevaluate screen as well and give tmux a try. There aren’t a lot of major differences here, but I’m sticking with tmux. The documentation is better, window splitting feels nicer, and the memory usage is lower. The biggest gain from switching is the configuration format. I find tmux’s configuration file much easier to understand and modify. Just compare the status line from my .screenrc to the same status line from my .tmux.conf.


The scope of customization for Emacs is astoundingly broad. It’s one of my favorite parts of using Emacs – anything I want to do can be made better with the judicious application of some Lisp. The downfall of this is that it can be hard to keep up with all the options. When I first starting using Emacs in 2008, I was looking for a library for completing file names. I tried out two candidates: ido-mode and anything.el. ido was easy to set up and use, so I went with it. I kept using ido because it worked great. I hadn’t even considered switching until I came across this article on Helm. Helm is a fork of anything.el that cleans up the code and makes it more modular.

I read through that article and was immediately sold on using Helm. Switching to Helm has been the best change to my text editing workflow since switching to Emacs! It provides a consistent interface for completing anything. The fuzzy matching is far ahead of anything else I’ve used. Despite this power, it’s still incredibly fast. Even with very large lists of completion candidates there is no noticeable slowdown.

I encourage you to try out Helm rather than just reading about it. It is a bit different than other completion libraries for Emacs, and it did take me a couple of days to really get used to it. It was time well-spent, however.

Configuring Emacs for Go Development

Setting Up Emacs for Go Development

I’ve recently started learning Go, and, as with any language, I want to configure Emacs as much as possible for Go development. After scouring the Internet, this is what I’ve come up with.


You’ll need Emacs and Go installed. I’m using OS X, so I use Emacs from http://emacsformacosx.com/ and this Go package.

Once you’ve installed everything, you’ll need to set a few environment variables:

You can set $GOPATH to a different directory if you like.


The first thing you’ll need for Go development in Emacs is go-mode. You can install this easily in Emacs 24 and up with M-x package-install go-mode.

Installing go-mode will also let you pull up documention on the standard library or third-party packages with M-x godoc. To make godoc work correctly, you’ll need to make sure Emacs has the correct values of $PATH and $GOPATH. This shouldn’t be an issue unless you’re using GUI Emacs on OS X. If you are, I’d recommend using my env-var-import package to import their values from the shell. That package will also set exec-path correctly from your $PATH, which will be important in later steps.


If you’ve done any Elisp development, you’re probably familiar with eldoc. If not, eldoc shows you the argument list of the function at your point in the minibuffer. go-eldoc is a package that does this for Go. You can install it with M-x package-install go-eldoc. Once you’ve done that, you’ll need to enable it:


One of the things I really like about go is the automatic enforcement of a consistent coding style via gofmt. You could manually run this on your code, but go-mode lets you configure Emacs to run it automatically when you save a buffer:


There is also a tool called goimports that not only formats your code like gofmt but also automatically updates all your imports. I’d recommend using this instead of gofmt.

You’ll need Mercurial installed first in order to install it. Then you can run go get code.google.com/p/go.tools/cmd/goimports. Now you can tell Emacs to use goimports instead of gofmt:


Installing go-mode will also give you godef, which will let you jump to the definition of a function like in an IDE. You can jump with M-x godef-jump and then return to the original point with M-*. I bound godef-jump to M-. for ease of use:

Custom compile command

You can use M-x compile to compile and test your code from Emacs. You’ll just need to set the command that compile runs:

I also bound C-c C-c to run compile.


The last step is to set up autocompletion for Go. I use the auto-complete package for other languages, so it’s natural to use it here too. For Go-aware autocomplete, you’ll need to install the gocode tool first with go get -u -v github.com/nsf/gocode. Then you can install auto-complete with M-x package-install auto-complete and go-autocomplete with M-x package-install go-autocomplete. Once you’ve installed everything add the following to your Emacs configuration:

Now you’re ready to start writing some Go in Emacs!