Emacs Lisp: Toggle Between a Clojure String and Keyword

When I was doing a fair bit of Ruby I often used the TextMate's shortcut (Ctrl+:) to convert a Ruby String to a Symbol or a Ruby Symbol to a String. It's something I've periodically missed while doing Clojure, and yesterday I found myself in the middle of a refactoring that was going to force the conversion of 5+ Clojure Keywords to Strings.

The following emacs lisp is my solution for toggling between Clojure Strings and Keywords. The standard disclaimers apply - it works on my machine, and I've never claimed to know emacs lisp well.

<script src="https://gist.github.com/jaycfields/5502257.js"></script> A quick video of the behavior:

<iframe width="560" height="315" src="http://www.youtube.com/embed/UQU7xasn3eA" frameborder="0" allowfullscreen=""></iframe>

strong_parameters are good and you should feel good

Besides moving attribute whitelisting to the controller rather than the model, Rails 4’s move to Strong Parameters over attr_accessible provides great documentation about the data with which records are being created.

Here is an example of a controller many of us have written, using strong_parameters:

class CommentsController < ApplicationController
  respond_to :html

  def create
    @comment = Comment.create!(comment_params)
    respond_with @comment
  end

  private

  def comment_params
    params.
      require(:comment).
      permit(:body).
      merge(user: current_user, commentable: commentable)
  end

  def commentable
    # find and return a commentable record
  end
end

Notice how the comment_params method tells you at a glance what object’s parameters this controller/action cares about (comment), the specific data being used (body), and the extra information being added. After glancing at the method, you hardly have to concern yourself with the rest of the class: everything just makes sense.

strong_parameters will be standard in Rails 4.0, but they can be used now in Rails 3.*.

Written by Caleb Thompson

P.S. You can include ActiveModel::ForbiddenAttributesProtection on a model-by-model basis, but given the level of awesomeness provided I wouldn’t recommend it.

Deploying WebP via Accept Content Negotiation

Deploying new image formats on the web is a challenge, but not an unsolvable one, and one that would pay high dividends in terms of performance. In fact, there are many ways to tackle the problem: client-side logic, server negotiation, and hybrid strategies. There is a time and place for each one - they are not exclusive.

Having said that, I'm partial to server negotiation as it requires the least amount of work to move the web forward in a significant way: a single flip of a switch by a CDN or a large hosting provider, instantly enabling significant byte savings for all of their users. Computers are great at performing routine negotiation work, humans are not: we get bored, we get lazy, we routinely forget to optimize our images.

WebP deployment checklist

As a hands on example, let's take a look at what we would need to enable Accept negotiation for deploying WebP:

1. User agents which support WebP should advertise it in the Accept header
2. Servers and caches should use the Accept header to serve appropriate assets
3. Origin servers must specify Vary: Accept in generated responses

Most importantly, notice what's missing: there is no mention of having to modify the markup or logic of millions of individual web applications. Having said that, the above steps still require some work, let's dig a bit deeper.

Configuring Accept negotiation in Nginx

The great news is that Chrome Canary is now advertising image/webp in its Accept header for all image requests - with that in place, we can check off the first item on our checklist above. Now, we need to configure our Nginx server to automatically serve the right file. To start, let's assume we have a pre-generated a WebP asset on disk:

location / {
  # check Accept header for webp, check if .webp is on disk
  if ($http_accept ~* "webp") { set $webp T; }
  if (-f $request_filename.webp) { set $webp "${webp}T"; }

  # if WebP is supported, and WebP is on disk, serve it!
  if ($webp = TT) {
    add_header Vary Accept;
    rewrite (.*) $1.webp break;
  }
  # ...
}

First, we check if the Accept header is advertising WebP. Then we check if there is a corresponding file with a .webp extension on disk. If both conditions match, we serve the WebP asset and add Vary: Accept - done.

Now, let's consider the case where Nginx acts as a proxy cache. Here we need to teach Nginx to cache and serve multiple variants of the same image (WebP and non-WebP versions) based on the value of the Accept header:

server {
  location / {
    # lookup inbound requests with exta WebP flag if advertised in Accept
    if ($http_accept ~* "webp") { set $webp T; }
    proxy_cache_key $scheme$proxy_host$request_uri$webp;

    # proxy requests to remote servers
    proxy_pass http://backend;
    proxy_cache my-cache;
  }
}

Not much different from the previous example! Instead of checking the local disk for the asset, we pass the request through to some set of backends, and append the WebP bit to our cache key - that's all there is to it.

Accept fragmentation and Key

Wait, why didn't we just append the $http_accept variable to our proxy cache key above? Unfortunately, the Accept header varies from browser to browser, which means that using the raw header would unnecessarily fragment the cache. Hence, we scan for WebP support in the header and add an additional bit to the cache key.

However, wouldn't it be nice if we had a standard mechanism to define a custom cache key? Well, good news, there is work underway on the new Key response header, which will act as a smart replacement to Vary:

The 'Key' header field for HTTP responses allows an origin server to describe the cache key for a negotiated response: a short algorithm that can be used upon later requests to determine if the same response is reusable.

IE is special, as always!

Unfortunately, Vary support in IE, even in the latest versions, remains very poor: IE does not cache outbound headers, which means it cannot perform the appropriate matching algorithm, and is forced to issue a revalidation request on any asset with Vary. As a result, to avoid the extra requests, we can create a special case for IE:

  if ($http_user_agent ~* "(?i)(MSIE)") {
    # drop Vary for IE users, mark resource as private
    proxy_hide_header Vary;
    add_header Cache-Control private;
  }

With the above configuration in place, IE users won't see Vary, and resource is marked as private: it can still be cached on the client, but it won't be cached by any other intermediate proxy.

The "action plan", continued...

Small steps, but we're heading in the right direction. We've addressed the issue on the client, we know how to configure our servers, now we need to push for better support by proxies and CDN's - ideally, by deploying support for Key! The good news is, this requires work, but it is not rocket science. As a case in point, CDNConnect recently enabled transparent Accept+WebP negotiation for all of their users.

Finally, let's take a look at Facebook's recent experiment with WebP, as summarized by ArsTechnica:

It turns out that Facebook users routinely do things like copy image URLs and save images locally... As a result, users were left with unusable URLs and files; files they couldn't open locally, and URLs that didn't work in Internet Explorer, Firefox, or Safari.

URL sharing is not an issue if Accept negotiation is configured correctly - see instructions above. Next, once a file is saved locally, you need a local WebP codec or viewer. The good news is, Chrome just landed a patch to register itself as a default WebP handler across all platforms - if you have Chrome, you can view local WebP files, and there is a growing list of editors, viewers, and plugins. Also, there are ongoing discussions about enabling the browser to save a "safe copy" of the asset to mitigate the problem entirely. One step at a time, we're getting there!

Embedding Canvas and SVG charts in emails

In our app Freckle Time Tracking we’re sending out weekly reports to users by email, reporting to them what they achieved last week.

Our emails not only include a report, listing all entries they logged, but also our “Mini-Pulse”, a graphical representation of how much they worked for each project.

Here’s how a typical email we send out looks like:

Now, generating a table and styling it for the HTML email is (relatively) easy, and beyond the scope of this article, but let’s have look at the charts in the email. We already generate the Mini-Pulse graph in our web app—we use an ancient version of Raphael.js to generate SVG, and this gets the job done nicely (SVG works on practically every modern browser, and Raphael falls back to VML on older Internet Explorer versions). Of course, you could use Canvas or any other HTML supported by WebKit just as well.

This is all great in your web browser, but not for emails. First off, obviously, JavaScript
is disabled in HTML emails, so we can’t use that to generate the SVG on the fly; moreover
SVG only works in a handful of email clients.

The only image formats that reliably work in HTML emails are GIF, PNG and JPEG, which means we have to dynamically generate such an image containing the charts and refer to it from the email.

There’s two possible ways to do this:

1. Reimplement the logic and rendering with a tool specifically made for generating chart images
2. Reuse the existing JavaScript/SVG code in a headless web browser and make “screenshots”

We chose to reuse the code we have, so we can easily adapt and extend both the web app and the HTML emails in the future (plus no need to learn yet another tool!).

A great way to create screenshots of the graphs is to use PhantomJS, which is a headless WebKit with an API that has support for taking screenshots.

We also have the following requirements for our report emails:

• Don’t generate the Mini-Pulse if the email is never opened, to conserve server resources
• Cache the generated image once the email was opened once
• Securely serve the image and use encrypted URLs with embedded authentication (the user the email was sent to may no longer have permission to access Freckle at the time the email is opened)
• Charts should be retinafied (your HTML emails are retinafied, are they?)
• File size of image should be small so it loads fast on mobile email clients

To fulfill these requirements, here’s what happens when a user opens an email that has a chart embedded:

1. HTML email is shown
2. Email client or browser accesses URL in the form of https://app.letsfreckle.com/m/xxxx/yyy.gif
3. If there’s a cached version of the image serve it and go to step 10, otherwise continue to step 4
4. Rails app decrypts account ID, user ID, chart type and date range from the given encrypted URL*
5. Rails app calls internal PhantomJS web service with a URL to call to generate the chart
6. PhantomJS web service calls the Freckle Rails app internally
7. Rails app serves Raphael.js, and our chart generation code and the data needed for it. (We use a special, stripped-down layout that only serves the chart and doubles resolution on everything to simulate rendering on a high-density screen (a feature that Raphael.js doesn’t yet directly support).
8. PhantomJS renders the page
9. PhantomJS returns a GIF to Rails (the call from step 4)
10. Rails returns the GIF and caches it into a file (Rails page caching)
11. Email client or web browser renders the GIF

All this sounds pretty complex, but it’s actually implemented in just about a hundred lines of code.

There’s a few tricky things you have to deal with when installing Phantom.js on a Linux server, such as adding fonts that may not be part of your default Linux server setup, but it’s pretty easy to get going. For Ubuntu 10.04, you can check out this gist with instructions on getting decent rendering quality.

*To accommodate passing parameters along from an HTML email to our Rails app, I’ve released URLcrypt, an open-source, MIT-licensed Ruby library for ‘elegant’ encrypted URLs. Alternatively, you could also use a table the holds tokens, but I find encrypting the account/user id information a more scalable solution.

Year Five

The average lifespan for a software engineering job is 4 years. Okay, I've never actually seen proof (or contradiction), but that's the general feeling in the groups I associate with. Perhaps that's selection bias - my employer has generally changed on year 3 or 4. Perhaps this is the exception and not the rule, in that case feel free to simply read this as an experience report. However, I do think it's somewhat common for developers to leave around year 3 or 4. This entry contains speculation on why they leave, and offers one idea on what employers can do to break that cycle.

My 4 year employment cycle generally looks like this

• Year One: "I'm in over my head. My semi-bluff was in-fact a bluff. They're going to fire me any day."
• Year Two: "It's nice to feel like a productive team member"
• Year Three: "This is fun, and I'm not bad at it. It's satisfying to pass on knowledge to teammates."
• Year Four: "This feels repetitive, that grass over there sure looks greener"

• A company you don't work at always seems to have infinite possibilities; however, after a few years with an employer, it's extremely clear what your options are. More importantly, it's very clear what limitations will likely always be there.
• A company you don't work at contains no code you're responsible for. Conversely, any company you've been with for 4 years probably has plenty of code you're not proud of. If you're responsible for that code, it's a constant reminder of your previous limitations. If you're not responsible for it, your co-workers aren't likely to let you forget about it anytime soon.
• There's always someone willing to pay you more than you're worth. After several years with a company it's likely that they're going to pay you what you're worth, but not what some other company thinks you're worth. I'm surprised that more companies don't pay (the employees they want to keep) what their "flawless market value" would be. In other words, what would you pay them if they interviewed, you determined what they knew, you determined what value they would bring, and you were completely ignorant of their flaws? That's what your competition is likely doing. That's what you're fighting against if you want to keep them around.
• A new job often offers a new challenge. Once you feel like you've given that challenge your best shot, what remains? If you did a great job, it's likely that you'll have plenty of other options. However, if you've done a good job, you may be stuck in a spot where there aren't as many open doors and challenges to choose from - not nearly as many as a position at another company will appear to offer.

We're at RailsConf 2013!

Winter is finally over, spring is here, and thoughtbot folks have flown in from Boston, San Francisco, Stockholm, and Boulder to be at RailsConf 2013!

Do you have any questions about our open source projects? Do you need some help setting up your project on Trajectory, or do you want to Learn something? We would love to chat with you!

We’re also setting up a temporary studio for our podcast in room D136. Feel free to stop by to say hi and watch us record live!

Adarsh Pandit, Ben Orenstein, George Brocklehurst, Harlow Ward, and Prem Sichanugrist are giving talks:

Tuesday at 11:30am: TDD Workshop: Outward-in Development, Unit Tests, and Fixture Data
Tuesday at 12:20pm: TDD Workshop: Mocking, Stubbing, and Faking External Services with Harlow Ward and Adarsh Pandit:

thoughtbot are creators of the open-sourced testing tools FactoryGirl and Shoulda Matchers.

We recognize Test-Driven Development (TDD) can be difficult to practice as features increase in complexity. Testing is often skipped when developers feel uncomfortable with TDD or have not yet seen certain approaches in practice.

We’ll describe specific techniques used in TDD which touch on: Integration testing with RSpec+Capybara, Model Associations and Data Validations, Asynchronous Jobs, Emails, 3rd Party Services, and JSON API endpoints.

Tuesday at 2:00pm: Sleeping with the enemy with George Brocklehurst:

In this session we’ll go off the Rails and take a look at what our Pythonista cousins are doing with Django.

I’ll start with some live coding: recreating DHH’s infamous 15 minute blog demo using Django and explaining the building blocks of a Django app along the way. I’ll then take that app and use it to look at some design decisions Django makes, and how they compare to Rails. You’ll see convention over configuration in places you didn’t expect it, why Django doesn’t need attr_accessible or strong parameters, and how the template method pattern could change your life.

Why talk about Python at a Rails conference? Seeing another way of doing things forces us to think about what we’re doing, challenges or validates the assumptions we make about our work, and inspires us to try new things.

Wednesday at 2:00pm: Zero-downtime payment platforms with Prem Sichanugrist and Ryan Twomey (from LevelUp):

When you’re building a payment platform, you want to make sure that your system is always available to accept orders. However, the complexity of the platform introduces the potential for it to go down when any one of the moving parts fails. In this talk, I will show you the approaches that we’ve taken and the risks that we have to take to ensure that our platform will always be available for our customers. Even if you’re not building a payment platform, these approaches can be applied to ensure a high availability for your platform or service as well.

Thursday at 10:30am: How to Talk to Developers with Ben Orenstein:

Nearly every developer will be asked to present to his or her peers at some point. Those that do it well will tend to have an outsized influence on their team, company, and community.

This talk will demonstrate (mostly by example) the straightforward techniques for giving excellent presentations, from a veteran conference speaker and teacher. Topics to cover include: * Phrases that turn your audience against you.
* Basic body language tips that affect perception.
* How to be more interesting than the internet.
* The power of live coding and demos.
* Being funny without resorting to reddit memes.
* How to get plenty of questions during Q&A.
* How to get an unfair amount of talk acceptances (aka ‘Bribing conference organizers’).

Ruby at GitHub

Here’s the video and slides for my recent talk “Ruby at GitHub” at MountainWest RubyConf.

<iframe width="560" height="315" src="http://www.youtube.com/embed/5vUD9Zg7gp0" frameborder="0" allowfullscreen=""></iframe>

Episode 46: We don't have a monopoly on being unhealthy

Episode 46: We don't have a monopoly on being unhealthy:

Ben Orenstein is joined by Joe Kutner, programmer and author of ‘The Healthy Programmer’. Ben and Joe discuss how the demands of a development job lead to unhealthy habits, and ways to address the issues. They discuss specifics like standing desks, walking desks, the pomodoro technique, exercise, vitamin D, and much more.

• Episode Notes and Links
• Subscribe via iTunes
• Subscribe via RSS
• Direct Download

It's for the orphans!

Too often, I come across GitHub Pages branches (gh-pages) branches that are simply forks from the master branch of the repository. This is not ideal.

There are several problems with this strategy:

• Unless you manually update the gh-pages branch every time you add a commit to master, you will be out of date.
• It makes for a much more difficult to understand source folder for the jekyll or HTML site, since people have to mentally categorize every file and folder into either “part of the site” or “an artifact from the master branch”.

• It introduces the need to have lines like this in _config.yml to prevent unrelated files from being included in the gh-pages site:

  exclude: ./app, ./lib, ./project.gemspec, ./Gemfile, ./LICENSE, ./Rakefile, ./readme.md, ./features, ./spec

git checkout --orphan gh-pages

When it comes time to create a GitHub Page for your project, use the above command. This creates a branch that is completely separate from the history of the rest of the repository. You will need to delete the files by running “git rm -rf .”:

If you want to start a disconnected history that records a set of paths that is totally different from the one of , then you should clear the index and the working tree right after creating the orphan branch by running "git rm -rf ." from the top level of the working tree. Afterwards you will be ready to prepare your new files, repopulating the working tree, by copying them from elsewhere, extracting a tarball, etc.

— git checkout --help

After you have an empty branch, you’re ready to get started on your jekyll or HTML site with a clean history and a clear conscience.

Marbled Godwit

It was a gorgeous spring morning to set up low along the banks of the pond and watch the birds do their thing.

5 useful tips for a better commit message

You’re already writing decent commit messages. Let’s see if we can level you up to awesome. Other developers, especially you-in-two-weeks and you-from-next-year, will thank you for your forethought and verbosity when they run git blame to see why that conditional is there.

1. The first line should always be 50 characters or less and that it should be followed by a blank line. Vim ships with syntax, indent, and filetype plugins for Git commits which can help here.

2. Add this line to your .vimrc to add spell checking and automatic wrapping at the recommended 72 columns to you commit messages.

   autocmd Filetype gitcommit setlocal spell textwidth=72

3. Never use the -m <msg> / --message=<msg> flag to git commit.

   It gives you a poor mindset right off the bat as you will feel that you have to fit your commit message into the terminal command, and makes the commit feel more like a one-off argument than a page in history:

   git commit -m "Fix login bug"

   A more useful commit message might be:

   Redirect user to the requested page after login
   
   https://trello.com/path/to/relevant/card
   
   Users were being redirected to the home page after login, which is less
   useful than redirecting to the page they had originally requested before
   being redirected to the login form.
   
   * Store requested path in a session variable
   * Redirect to the stored location after successfully logging in the user
   

4. Answer the following questions:

   1. Why is this change necessary?

      This question tells reviewers of your pull request what to expect in the commit, allowing them to more easily identify and point out unrelated changes.

   2. How does it address the issue?

      Describe, at a high level, what was done to affect change. “Introduce a red/black tree to increase search speed” or
      “Remove <troublesome gem X>, which was causing <specific description of issue introduced by gem>” are good examples.

      If your change is obvious, you may be able to omit addressing this question.

   3. What side effects does this change have?

      This is the most important question to answer, as it can point out problems where you are making too many changes in one commit or branch. One or two bullet points for related changes may be okay, but five or six are likely indicators of a commit that is doing too many things.

      Your team should have guidelines and rules-of-thumb for how much can be done in a single commit/branch.

5. Consider making including a link to the issue/story/card in the commit message a standard for your project. Full urls are more useful than issue numbers, as they are more permanent and avoid confusion over which issue tracker it references.

   This is generally done as the first paragraph after the summary, on line 3.

Having a story in your git log will make a huge difference in how you and others perceive your project. By taking great care in commit messages, as you do in your code, you will help to increase overall quality.

Special thanks to Tim Pope, whose Note About Git Commit Messages literally sets the standard for a good commit message.

Additional thanks to the creator of Git and a real stickler for a good commit message, Linus Torvalds.

Better commit messages

Caleb Thompson:

Answer the following questions:

• Why is this change necessary?
• How does it address the issue?
• What side effects does this change have?

Redis partial word match -- you (auto)complete me

We can use partial word matching to rapidly search strings of text such as Names, Cities, States, etc.

We can do this by indexing strings into Redis sets based on partial matches of the string. The indexing process takes a string and breaks it into left-bound substrings which are placed into the appropriate Redis sets.

Partial word matching movie titles

In this example we’ll enable partial word matching queries of movie titles.

The movie titles will be stored in plain text. The keys are simple key-value pairs that are stored with the key structure movies:id, where id is incremented:

$ redis-cli get movies:1
"Bad Santa"
$ redis-cli get movies:2
"Batman"
$ redis-cli get movies:3
"Bad Company"

The movie Batman would be decomposed into the following sets (assuming that the indexing started at two characters):

ba
bat
batm
batma
batman

We’ll index the movie titles into sets based on partial match of their titles. To prevent any key collisions in Redis we’ll create keys with the following structue index:abc:movies.

View the indexed ids using Redis smembers command:

$ redis-cli smembers index:ba:movies
1) "1"
2) "2"
3) "3"

$ redis-cli smembers index:bat:movies
1) "2"

Note: The keys for the movie titles have been parameterized (lowercase and spaces replaced with dashes).

We’ll use a Ruby class Autocomplete to wrap the Redis calls:

# autocomplete.rb
require 'redis'

class Autocomplete
  def initialize(partial_word)
    @partial_word = partial_word
  end

  def movies
    redis.sort "index:#{partial_word}:movies", by: :nosort, get: 'movies:*'
  end

  private

  def redis
    Redis.current
  end

  attr_reader :partial_word
end

movies = Autocomplete.new('bat').movies

We’ll use the sort command with the by: :nosort parameter to query the partial word match index and return the matched movie names.

Written by Harlow Ward

Related Reading

Redis Set Intersection - Using sets to filter data

Redis Pub/Sub…how does it work?