Modernizing my Emacs Configuration

I’ve been using Emacs for well over a decade at this point. Over the years my configuration for Emacs has grown and evolved. I’ve always appreciated the ability to change basically any aspect with some Lisp, and I am an inveterate tinkerer.

The end result of 10+ years of tinkering was an Emacs config that, while functional and tuned to my needs, was hard to understand and change. While I have changed it continuously over the years, I had not revisited the basic structure of my configs since Emacs 24 shipped ELPA in 2012. As such I decided to undertake the effort to modernize and reorganize things.

Previous Setup

My Emacs configs were previously split up across several files:

  • ~/.emacs: The core init file. Here I would enable various packages and set configuration variables.
  • ~/.emacs.d/elisp/keyboard-shortcuts.el: Here I would define any custom keybindings, both global and for specific modes.
  • ~/.emacs.d/elisp/mode-hooks.el: Here I would define custom hooks to change the behavior of various modes. Any add-hook call using built-in hooks went here too.
  • ~/.emacs.d/elisp/functions.el: Here I defined any custom functions other than mode hooks.

This setup was basically unchanged since I first split up my original .emacs file (probably circa 2010 though I don’t have the git history from that time). This did have the advantage of mostly making sense to me for code organization. If I wanted to add something new or find an existing setting, it was easy for me to know where I would have put it. The downsides were basically everything else.

In particular I had the following problems:

  1. No real structure. While finding the right file was easy, within that file there was no real organization. While I tried to group related things together, this did not work out in practice. Finding a particular setting was a matter of searching. In a couple cases I had redundant settings because I missed that I had configured something previously.
  2. Configuration sprawl. The settings for a particular package were not always in one place. Complex packages, like Helm, had configuration spread across all the files listed above. This made it hard to understand how I had configured each package.
  3. No documentation. My Emacs configs had been built up organically over years, and I was not always good at documenting why I did things. I had started keeping the configs in git around 2009, but for various reasons I won’t go into here my current history only goes back to 2014. Consequently I had many settings where it was unclear whether or not it was still useful or important to me. This was exacerbated by configs accumulated from past jobs that were no longer relevant.
  4. Not up to modern standards. The Emacs community continues to evolve the best practices for configuration. I was more active in the community during undergrad, but had fallen away from it since. As such I was still using patterns from almost a decade ago. In particular I wanted to address using ~/.emacs.d/init.el in favor of ~/.emacs and using use-package for package management.

Reorganization and Cleanup

The first step I undertook was to clean up and reorganize my configs. First off I did two basic things:

  1. Move my ~/.emacs to ~/.emacs.d/init.el. This was nothing more than moving the file.
  2. Remove unused settings. I had accumulated configuration for various programming languages from past jobs and projects. If I wasn’t actively using it, I decided to rip it out. I’d likely want something new anyway if I did come back to it. I know I didn’t get everything unused here, but I wanted to at least remove the biggest offenders.

This got my configs to a slightly cleaner place. The next step was to reorganize into new modules. My goal was to pull as much as possible out of init.el and put everything into a small number of new modules. I settled on the following structure:

  1. appearance.el: All configuration related to changing the Emacs UI
  2. base.el: Core Emacs configuration independent of any particular mode or workflow
  3. editor.el: Core text/code editing configuration
  4. modes.el: Specific configuration for various modes
  5. work.el: Any configuration specific to my work setup that I didn’t use for personal projects.

I started out just reorganizing things. Once I got everything into the new locations, I took another pass to clean up unused settings, since the new layout gave me a fresh view of things.

The next step was to start using use-package. This is a very nice library that solves several problems with Emacs package configuration:

  • Automatically installs missing packages
  • Standard layout and location for package configuration
  • Defer loading certain packages for faster startup

This was a somewhat tedious process of going through each (require 'package) in my configuration and converting it to use-package by moving all the related configuration into use-package’s structure. Once I had done that I took a pass through my list of installed packages in package-selected-packages to confirm they were all explicitly loaded, and removed everything unused. Altogether I think I removed ~20% of the packages I had installed previously and got my Emacs startup time down to ~1.5 seconds. I don’t usually care about this much since I only start Emacs once a day, but this was a big improvements from the 5-10 second range it was in previously.

You can see what all of these changes looked like if you compare afcc501c15d05200e39b5e833854d9f12b3c7c1a to 3c43236132c45b1d022bc4917dccf25ad0b743c7.

Originally I was going to stop with this new organization. However, I felt it hadn’t totally addressed the documentation problem, so I decided to keep tinkering.

Org-Babel

As I had been researching the changes I wanted to make to my configs, I came across many people online who used org-babel to structure their Emacs configuration. I was attracted to this because org-mode provides good structure, is well-integrated with Github, and allowed for more detailed documentation.

This did mean I needed to redo the structure of my configs yet again, but since I had cleaned things up in the previous move, it was much easier this time around. To begin, ~/.emacs.d/README.org is now my core init file. ~/.emacs.d/init.el becomes just a single line:

(org-babel-load-file (expand-file-name "README.org" user-emacs-directory))

Almost all other configuration goes into README.org, with two exceptions:

  1. ~/.emacs.d/early-init.el contains some settings that need to happen at the very start of loading (mostly GC/performance related).
  2. ~/.emacs.d/custom.el is managed by the built-in Custom system. I don’t use this directly, but the package manager does. I don’t want these messing up my core configs and there is nothing I care about in them, so they can go in a separate file.

org-babel allows mixing org-mode markup and code. I used org-mode’s standard headings and list to structure and organize the file. My actual config goes into code blocks. A simple example of this looks like so:

Next I disable the toolbar, scrollbar, and menu as I find them unncessary:
#+BEGIN_SRC emacs-lisp
(tool-bar-mode -1)
(scroll-bar-mode -1)
(menu-bar-mode nil)
#+END_SRC

Everything between the BEGIN_SRC/END_SRC lines is actual Emacs Lisp code that makes up my configuration.

Results

All in all I’m happy with the results. You can see the final project at https://github.com/ajsquared/home-osx/tree/master/.emacs.d. There’s more I could do here, like linking to package documentation or adding more detailed reasoning for each setting, but I don’t think this is particularly pressing. I now have a much more well-structured and explainable configuration, and think it is also much more easily shared.

I’d definitely recommend the org-babel approach and use-package to anyone who uses Emacs.

Restoring tmux Sessions

I’ve been a big fan of tmux since I started using it in 2014. I use it locally for multiplexing my terminal, and in combination with mosh it’s also fantastic for work on remote servers. My main issue with it is losing my sessions when a box restarts. Luckily there’s plugins for tmux that solve this problem!

Prerequisites

  • tmux (brew install tmux on OS X)
  • TPM (tmux plugin manager). See https://github.com/tmux-plugins/tpm#installation for installation instructions.

Setup

There are two plugins I recommend: tmux-resurrect and tmux-continuum. tmux-resurrect enables saving and restoring tmux sessions manually, while tmux-continuum saves automatically and allows for automatically restoring from the last save when tmux is started. To enable these settings, add the following to your ~/.tmux.conf:

Usage

With this configuration, there’s no need to do anything manually. Your tmux sessions will be automatically saved every 15 minutes and automatically restored when tmux is started (e.g. after a reboot). You can manually save with prefix-Ctrl-s and manually restore with prefix-Ctrl-r if desired.

Note that this won’t restore running applications. tmux-resurrect has an optional configuration for doing so, but I’ve left this off for the time being.

HTTPS Support

I didn’t keep up with my promise to blog more regularly from summer 2016, but it’s a new year and I’m giving it another go.

To start things off, I decided to finally enable HTTPS for this site and my separate site for short fiction. Even though these sites are low-traffic and don’t deal with any sensitive information, it’s still a good idea to support HTTPS. See https://www.eff.org/encrypt-the-web for some background on why.

I had looked at doing this when I first started running this site,  but at the time it was a pretty tedious process. Today it’s straightforward, even though I’m on shared hosting.

First I needed to get certificates. I chose to use Let’s Encrypt, since it’s free, highly automated, and a joint, open effort. To actually interact with Let’s Encrypt, I installed the Acme PHP client on my web hosts. This tool allowed me to obtain the certificates via one shell command:

Then I modified the .htaccess file to redirect all HTTP traffic to HTTPS:

And that’s it!

The only piece I don’t have totally working is automatic renewal of my certificates. I’ve got the renewal happening on a cron but it’s not yet updating the one used by the site.

 

Bugs with Non-Alphanumeric Credentials for AWS

I've recently had a chance to be working with AWS and S3 and encountered a somewhat frustrating bug that I wanted to share. I was using boto3, an AWS SDK for Python. I had created a script that would list the EC2 instances that make up an EMR cluster. It worked fine with my credentials, but another user would be a SignatureDoesNotMatch error every time. We spent a lot of time comparing the environment setup and permissions on the two accounts, but everything lined up. We started guessing a little more wildly when we noticed that my AWS credentials (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) were entirely alphanumeric, but the other user's were not. My first thought was that there was some kind of encoding error, but I had no luck with that. After some searching I found this longstanding bug with the same issue. As it turns out having non-alphanumeric credentials can cause this error and has for at least a couple of years. The only fix is to regenerate the credentials until getting a set that is entirely alphanumeric.

I later ran into the same signature error trying to distcp some data from HDFS to S3, but I knew what to look for this time. HADOOP-3733 is a very similar issue to that in boto3, but this is fixed in Hadoop 2.8.0.

This was a troublesome bug for me, so hopefully this post helps someone else who encounters the same issue.

Blogging More Regularly

It has been a while since I have written anything substantial here. General busyness with life and work have kept me from getting around to writing anything. However, I am going to try and blog more regularly going forward. Starting out I'm going to try and post something once a week and see how that goes.

Writing Setup

For my first post in this attempt I am going to talk a little about my writing setup.

I've found that I really like distraction-free writing tools as opposed to a standard text editor or word processor. Being able to focus only on the text helps keep me in the zone. I've tried a few of these and I've come to like Ulysses the best. It's less minimalistic than some others that I have tried but has a lot of really useful features. You write with Markdown, which I'm already familiar with, and it has great export functionality. It also syncs via iCloud so I can work on either my iPad or my desktop computer.

I used to write exclusively on a computer at my desk, but I found this an impediment. If I have an idea of something to write about I don't necessarily want to go sit at my desk to put down the words. Instead I'm now usually using a keyboard attached to my iPad to write. It's a little smaller than a full-sized keyboard so my typing speed isn't up to par yet, but it's great to be able to sit on the couch and write.

Suppressing ad-handle-definition Warnings in Emacs

As I have been tweaking my Emacs configuration, I noticed that I was getting warnings like the following in *Messages* during Emacs startup:

I looked into it and these warnings are generated when functions are redefined with defadvice. Most of the warnings I received were coming from third-party packages I had installed, not my own configuration. In any case, they were not helpful, just noise during Emacs startup.

Like everything in Emacs, this is configurable. Turning off the warnings is as easy as adding:

To your .emacs file. I found I had to add it fairly early to ensure that all the warnings were removed.

New Year, New Tools

New Year, New Tools

It’s a new year, so I thought it would be a good time to review some new tools I started using over the last year.

Mosh

Mosh is a replacement for SSH that handles intermittent connectivity. It will warn you if your connection drops and automatically reconnect. This handles situations like putting your laptop to sleep or changing between different connections (e.g. switching from Ethernet to Wifi). I’m connected to a bunch of different servers over the course of the day, so being able to put my laptop to sleep and have it wake back up with my remote sessions intact is incredibly useful. It’s hard to believe I used to work without it.

Mosh doesn’t support X11 forwarding or non-interactive uses of SSH, like port forwarding. However, I don’t use these particularly often, so in practice I’m not bothered by this. In any case, regular SSH is still available!

zsh

I had resisted changing shells for a long time. Bash is available pretty much everywhere. I had a nice .bashrc that I could put anywhere and have the environment I was used it. zsh had been recommended to me several times, but I was concerned about the effort needed to get up and running on a new machine. I liked being able to just drop a few dotfiles in my home directory – would I also have to install zsh?

I decided to try the switch after getting fed up with an issue with my bash prompt and long lines – characters would occasionally get doubled. I am really glad I made the switch. Needing to install zsh wasn’t as big of a deal as I had feared. I used oh-my-zsh to get started. This gave me a lot of the configuration I had done manually in my .bashrc out of the box. Plus it provides plugins for just about anything you might use in the shell. My favorite feature of zsh itself is the improved tab completion. It’s hard to explain without experiencing it – just give zsh a try.

tmux

Much like bash, I had used screen for a while and it just worked. I had a configuration I liked and I didn’t need to think about it. When I was switching to zsh, I decided to reevaluate screen as well and give tmux a try. There aren’t a lot of major differences here, but I’m sticking with tmux. The documentation is better, window splitting feels nicer, and the memory usage is lower. The biggest gain from switching is the configuration format. I find tmux’s configuration file much easier to understand and modify. Just compare the status line from my .screenrc to the same status line from my .tmux.conf.

Helm

The scope of customization for Emacs is astoundingly broad. It’s one of my favorite parts of using Emacs – anything I want to do can be made better with the judicious application of some Lisp. The downfall of this is that it can be hard to keep up with all the options. When I first starting using Emacs in 2008, I was looking for a library for completing file names. I tried out two candidates: ido-mode and anything.el. ido was easy to set up and use, so I went with it. I kept using ido because it worked great. I hadn’t even considered switching until I came across this article on Helm. Helm is a fork of anything.el that cleans up the code and makes it more modular.

I read through that article and was immediately sold on using Helm. Switching to Helm has been the best change to my text editing workflow since switching to Emacs! It provides a consistent interface for completing anything. The fuzzy matching is far ahead of anything else I’ve used. Despite this power, it’s still incredibly fast. Even with very large lists of completion candidates there is no noticeable slowdown.

I encourage you to try out Helm rather than just reading about it. It is a bit different than other completion libraries for Emacs, and it did take me a couple of days to really get used to it. It was time well-spent, however.