Sass !default

Posted 5 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

From the Sass documentation:

You can assign to variables if they aren’t already assigned by adding the !default flag to the end of the value. This means that if the variable has already been assigned to, it won’t be re-assigned, but if it doesn’t have a value yet, it will be given one.

This is what it looks like:

$example: 'value' !default;

So using Sass !default is like adding an "unless this is already assigned" qualifier to your variable assignments. But what's the use case for this?

Example: custom styles for a white label app

Recently, a client asked us to create a white label app: they wanted their customers to be able to customize things like colors, logo, background, etc., but to have fallbacks for customers who wouldn't use custom branding. How do you handle this with Sass? Let's step through it.

First, all of these customizable parts are defined in variables.

$brand: company-name;
$brand-color: #0074BE;
$brand-color-alt: #E2EAF2;
$brand-background-1: hsla(0, 0%, 97%, 1.0);
$brand-background-2: transparent;

The variable names are broad enough to use for any customer. If a company doesn't customize anything, this is what they get.

For each customer, we'll create a file with their custom variables. It will use the same variable names, but replace the values. Normally to override variables, you have to define the new value below the original value:

$var_name: 'val';
$var_name: 'new val';

With !default, it's the other way around: we include the brand specific SCSS file first, then we add !default at the end of all our default brand values. This is what our fallback variables look like now:

$brand: company-name !default;
$brand-color: #0074BE !default;
$brand-color-alt: #E2EAF2 !default;
$brand-background-1: hsla(0, 0%, 97%, 1.0) !default;
$brand-background-2: transparent !default;

Optimization

If every client has a customized stylesheet with at least their company name, we need uniquely named files for each of them since client-name-1.scss will live in the same directory as client-name-2.scss.

Files included for client-name-1's account:

@import 'client-1-overrides';
@import 'base-variables';
@import 'header';
@import 'body';
@import 'footer';

To reduce repeated code, we take all of the general imports after our overrides and move them into _application.scss.

Files included for client-name-1's account:

@import 'client-1-overrides';
@import 'application';

So, we've used !default to define the actual default values, and we've overridden them with brand specific values where needed.

What's next?

If you found this useful, you might also enjoy:

Using Polymorphism to Make a Better Activity Feed in Rails

Posted 5 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

You won't find "polymorphic partials" in Rails' API documentation. However, the general programming technique of polymorphism can be applied to Rails partials.

Example: an activity feed for an eBay marketplace

Imagine you are signed into an application similar to eBay. You can buy or sell items on the marketplace. In order to buy an item, you need to bid on it. You see an activity feed that includes bids, comments, and other things that are relevant to you.

Models

We'll use a polymorphic association:

class Activity < ActiveRecord::Base
  belongs_to :subject, polymorphic: true
  belongs_to :user
end

When subjects like Bids and Comments are created, we'll create Activity records that both the seller and bidder will see in their activity feeds.

class Bid < ActiveRecord::Base
  belongs_to :item
  belongs_to :user

  after_create :create_activities

  def bidder
    user
  end

  def seller
    item.user
  end

  private

  def create_activities
    create_activity_for_bidder
    create_activity_for_seller
  end

  def create_activity_for_bidder
    Activity.create(
      subject: self,
      name: 'bid_offered',
      direction: 'from',
      user: bidder
    )
  end

  def create_activity_for_seller
    Activity.create(
      subject: self,
      name: 'bid_offered',
      direction: 'to',
      user: seller
    )
  end
end

In a production app, we'd create those records in a background job. We've simplified here for example's sake. The further benefit of creating them in the background can be seen when we create activities for each of the commenters, which may be a large number for an active marketplace item:

class Comment < ActiveRecord::Base
  belongs_to :item
  belongs_to :user

  after_create :create_activities

  def seller
    item.user
  end

  private

  def create_activities
    (commenters_on_item + [seller]).uniq.each do |user|
      Activity.create(
        subject: self,
        name: 'comment_posted',
        direction: 'to',
        user: user
      )
    end
  end

  def commenters_on_item
    Comment.where(item_id: item.id).map(&:user).uniq
  end
end

Now that we have a clean set of activities in a database table, the SQL lookup is simple:

class User < ActiveRecord::Base
  has_many :activities

  def recent_activities(limit)
    activities.order('created_at DESC').limit(limit)
  end
end

This is the core benefit of structuring our data this way. At runtime, we find the data via a single indexable foreign key, user_id:

create_table :activities, do |t|
  t.timestamps null: false
  t.integer :subject_id, null: false
  t.string :subject_type, null: false
  t.string :name, null: false
  t.string: direction, null: false
  t.integer: user_id, null: false
end

add_index :activities, :subject_id
add_index :activities, :subject_type
add_index :activities, :user_id

Anti-pattern

We've seen alternative implementations that look something like this:

class User < ActiveRecord::Base
  def recent_activities(limit)
    [comments, items.map(&:comments), bids].
      flatten.
      sort_by(&:created_at).
      first(limit)
  end
end

There are a couple of problems with that approach:

  • the number of ActiveRecord objects loaded into memory is large
  • sorting is done in Ruby, which is slower than SQL

Controller

We make our fast lookup:

@activities = current_user.recent_activities(20)

Polymorphic Rails partials

Now, let's show the activity feed in a view:

%ul
  - @activities.each do |activity|
    %li.activity
      = render "#{activity.name}_#{activity.direction}_current_user", subject: activity.subject

Here we render partials with polymorphism. Through the single "#{activity.name}_#{activity.direction}_current_user" interface, we're able to render multiple partials:

  • bid_offered_to_current_user
  • bid_offered_from_current_user
  • comment_posted_to_current_user

When we write upcoming features, we'll be able to render even more partials representing many other interactions, using the same simple structure:

  • bid_accepted_from_current_user
  • bid_rejected_to_current_user
  • etc.

In turn, each partial is small, contains no conditional logic, and results in copy text that makes sense for the user's context:

The Old Man offered a bid of $100 for your Red Ryder BB gun with a compass in the stock, and this thing which tells time.

We can style each partial differently, perhaps showing an image of the items being offered or the avatars of the users who commented.

Nowhere do we do anything like this:

%ul
  - @activities.each do |activity|
    %li.activity
      - if activity.subject_type == 'Bid' && activity.direction == 'to'
        = render "bid_offered_to_current_user", subject: activity.subject
      - elsif
        = # ...

We've replaced an ugly conditional with polymorphism and used a couple of naming conventions to made it easier to add subject types without changing the view logic.

What's next?

If you found this useful, you might also enjoy:

  • Ruby Science to read more about replacing conditionals with polymorphism in Ruby
  • Simple Made Easy to hear about a rigorous definition of "simple" applied to software

The Kid

Posted 5 months back at Mike Clark

The Kid

The kid's a chip off the ol' block.

Improving the Command-Line Postgres Experience

Posted 5 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

Understanding the ~/.psqlrc configuration file, its options, and reading others' ~/.psqlrc files makes working with command-line Postgres more enjoyable.

Changing the prompt

By default, the prompt is a little blah:

$ psql my_database
my_database=#

Let's jazz it up.

There are a lot of options for changing the prompt. You can add colors, domain names, the name of the current database, and more.

\set PROMPT1 '%[%033[1m%]%M %n@%/%R%[%033[0m%]%# '
  • The %[..%] sets the default prompt color to a bold black.
  • %M is "The full host name (with domain name) of the database server, or [local] if the connection is over a Unix domain socket".
  • %n is the database user name.
  • %/ is the database name.
  • %R is "normally =, but ^ if in single-line mode, and ! if the session is disconnected from the database". It's nice to see when you're disconnected.
  • The final %[...%] resets the color to non-bold black.
  • %# is # if you're a superuser, otherwise >.

Here's how it looks on a local database:

[local] gabe@my_database=# SELECT * FROM users;

Changing the prompt, again

Being the smart cookie you are, you've probably inferred that a setting called PROMPT1 implies that there's a PROMPT2. And you're right! You go, Glen Coco. PROMPT2 for you:

\set PROMPT2 '[more] %R > '

PROMPT2 is used when psql is waiting for more input, like when you type SELECT * FROM then hit enter - you haven't typed a semicolon, so psql patiently displays PROMPT2 and waits.

Here's the rundown:

  • [more] is the literal string [more].
  • %R in PROMPT2 has a different meaning than in PROMPT1 - "in prompt 2 the sequence is replaced by -, \*, a single quote, a double quote, or a dollar sign, depending on whether psql expects more input because the command wasn't terminated yet, because you are inside a /\* ... \*/ comment, or because you are inside a quoted or dollar-escaped string."

Here's a contrived example:

[local] gabe@my_database=# SELECT
[more] - > '
[more] ' > name
[more] ' > '
[more] - > FROM users;

Nice.

There's more to life than prompts

Now your prompt is spiffed up, but your ~/.psqlrc can bring still more joy to your life. Here are some options I set, with comments:

-- By default, NULL displays as an empty space. Is it actually an empty
-- string, or is it null? This makes that distinction visible.
\pset null '[NULL]'
-- Use table format (with headers across the top) by default, but switch to
-- expanded table format when there's a lot of data, which makes it much
-- easier to read.
\x auto
-- Verbose error reports.
\set VERBOSITY verbose
-- Use a separate history file per-database.
\set HISTFILE ~/.psql_history- :DBNAME
-- If a command is run more than once in a row, only store it once in the
-- history.
\set HISTCONTROL ignoredups
-- Autocomplete keywords (like SELECT) in upper-case, even if you started
-- typing them in lower case.
\set COMP_KEYWORD_CASE upper

What's next?

If you found this useful, I recommend:

Caching API Requests

Posted 5 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

When making requests to an external service's API, some requests will frequently occur with the same parameters and return the same result. If we cache our request or response, we can reduce HTTP requests, which can improve performance and avoid hitting rate limits.

The APICache Ruby gem is a good choice for caching the API responses (typically JSON) in any Moneta store, such as Memcache or Redis.

However, we don't always need to cache the entire API response. We can save space, avoid adding the operational overhead of Memcache or Redis, and avoid repeating the JSON parsing step if we cache only the URL requested.

In the following example, our app only needs a venue's name, latitude, longitude, and street address. We'll get the data from Foursquare's venue search API by category ("restaurant") and neighborhood ("The Mission").

url = Foursquare.new(category, neighborhood).venue_search_url

ApiRequest.cache(url, Foursquare::CACHE_POLICY) do
  # make a GET to the URL
  # parse JSON
  # create or update venue name, lat, lon, street address
end

The first time this code runs for a venue search for restaurants in the Mission, ApiRequest will save the URL to the database and the block will be executed.

Whenever this runs again for a venue search for restaurants in the Mission, as long as it is within Foursquare's 30 day cache policy, the block won't be executed and expensive work will be avoided.

The internals

It's a pretty simple pattern and the code to make it happen is also straightforward.

Here's the database migration:

class CreateApiRequests < ActiveRecord::Migration
  def change
    create_table :api_requests do |t|
      t.timestamps null: false
      t.text :url, null: false
    end

    add_index :api_requests, :url, unique: true
  end
end

The index improves performance of future lookups and enforces uniqueness of the URL.

Here's the ApiRequest model:

class ApiRequest < ActiveRecord::Base
  validates :url, presence: true, uniqueness: true

  def self.cache(url, cache_policy)
    find_or_initialize_by(url: url).cache(cache_policy) do
      if block_given?
        yield
      end
    end
  end

  def cache(cache_policy)
    if new_record? || updated_at < cache_policy.call
      update_attributes(updated_at: Time.zone.now)
      yield
    end
  end
end

We've kept this model generic enough that it can be used with other APIs, not just Foursquare. We inject a cache policy dependency that must respond to call. This allows us to pass in ActiveSupport's nice Numeric methods like 30.days.ago and have them execute at runtime.

Here's the Foursquare model:

class Foursquare
  BASE_URL = 'https://api.foursquare.com/v2/'
  CACHE_POLICY = lambda { 30.days.ago }

  attr_reader :category, :neighborhood

  def initialize(category, neighborhood)
    @category = category
    @neighborhood = neighborhood
  end

  def venue_search_url
    BASE_URL + 'venues/search?' + {
      categoryId: category_id,
      client_id: ENV['FOURSQUARE_CLIENT_ID'],
      client_secret: ENV['FOURSQUARE_CLIENT_SECRET'],
      limit: 50,
      ll: lat_lon,
      radius: 800,
      v: '20130118'
    }.to_query
  end

  private

  def category_id
    category.foursquare_id
  end

  def lat_lon
    "#{neighborhood.lat},#{neighborhood.lon}"
  end
end

In this example, we chose to build the URL ourselves, using ActiveSupport's Hash#to_query and pulling our client ID and secret in from environment variables.

What's next?

If you found this useful, you might also enjoy:

At The Ready

Posted 5 months back at Mike Clark

Great Gray At The Ready

With talons and wings at the ready, this great gray scopes out the fields for a meal. I went in tight because I loved the contrast between the soft feathers, the face detail, and the razor-sharp talons.

A case against the module design pattern

Posted 5 months back at A Programmer's Perspective

The module design pattern is a particularly popular design pattern used in JavaScript these days. The main reasons for its use are: “Private” variables via closure Global imports A simple example of a module design pattern implementation looks like this: (…)

Read the rest of this entry »

Happy Thanksgiving!

Posted 5 months back at Mike Clark

Great Gray Takeoff

Today I'm also thankful for the birds who aren't turkeys.

NPM needs our help, and why you should donate

Posted 5 months back at Phusion Corporate Blog


NPM must be one of the most invaluable tools for Node.js users. It’s used by tons of users all around the world every second. But there have been various issues with the NPM registry lately. One thing is clear: running the NPM registry is not easy and requires a lot of administration time, effort and money. It is for this reason that Nodejitsu, the party who runs the NPM registry, has launched a funding campaign to improve the NPM registry infrastructure.

As active participants in the open source community, we fully understand how difficult it is to run such a service. The sheer amount of resources needed to properly run and maintain something like this, is easily underestimated by a lot of people. The hardware, the bandwidth, shouldn’t cost that much, should they? Once they’re in place, just configure a few things and the thing runs, right?

But setting up a reliable infrastructure and keeping it healthy is by no means a small task. Properly configuring a high-traffic cluster requires a lot of time and expertise. And there will always be unexpected problems. What happens if there’s a network problem? What happens if an OS upgrade broke things? What happens if there’s a disk failure? Even with all the automatic failover you configure, Murphey’s law will always find you. And when it does, you better be prepared and have the man power to solve the issues. Man power costs time and money.

One might think that Nodejitsu, being a commercial, VC-funded company can easily pay for this themselves. So why should anybody donate? Scalenpm.org explains this: their investors want their money to be used for making more money, not for providing free community services. All donations go toward running the NPM registry.

We all rely on NPM a lot. Even though NPM is open source, it is important not to equate open source with free. It is not free for the party that has to run and maintain it. So we believe it’s time to give something back as well, which is why we have donated. You may want to consider donating as well.

5 things I’ve learned in 5 years of running a SaaS

Posted 5 months back at mir.aculo.us - Home

5045502202_41476791a4_o

Photo Credit: Will Clayton cc

Freckle Time Tracking is turning five on December 1. In 5 years of being a co-founder of Freckle I’ve learned a lot of things, but here are 5 important takeaways. Maybe it helps you on your path to product nirvana:

1. You’re not a “tech company”—you’re a “make customers awesome” company
People don’t pay you because you have amazing programming skills and can write nginx configurations blindfolded. People pay you money because the product you sell to them saves them time, money, effort and nerves. It’s your job to make your customer more awesome. Every decision you make for your product and business should revolve around that.

2. Never promise dates for a feature launch
Just don’t promise launch dates for a feature. Ever. Trust me on this. People will ask you all the time when “feature X” is ready. A good way to answer that question is (if you plan on doing it), “We’re considering this feature for a future version. I can’t give you a date on when it will be ready.”. Just be honest to your customers—you don’t know yourself if and when a feature will really be ready.

3. Spend money on things that help you stay productive
This includes obvious stuff like a laptop that doesn’t suck (upgrade often), a good working chair and desk, and less obvious things like software that allows you to concentrate on developing your application’s features rather than configuring servers.

4. Do not work too much
Overworking yourself is the first step to failure in business. You can’t do your best if you’re permanently stressed out. Don’t check email in the evenings. If you’re only 1 or 2 people, don’t provide 24/7 support. It’s ok. Customers understand. It helps to not have a mission-critical product (if Time Tracking goes down it’s annoying but people can take a note on paper).

You didn’t start a company to die of exhaustion. Your health, family and social life is more important than 5 minute support response times and a 100% uptime guarantee.

By the way, one way to keep on top of this is to keep track on how you spend your time.

5. Don’t believe the hype
People are good at getting excited. And people are good at believing the hype™ about new technologies, frameworks, programming languages and was to deploy. People will tell you what to do and what to plan for. That you need to scale to millions of users, and you’re doomed if you don’t plan for that. That generating HTML on the server is so 1994. That node.js will cure cancer.

The fact is that you need to be pragmatic—your goal is to run a business. Use technology that is proven (to you), and that you know how to work with. My “litmus test” for technology is if the people that provide it are in a similar situation as you are: having to rely on it to run their own business (this quickly weeds out cool-but-academic-only stuff). You need to optimize for shipping. That includes writing less code, having broad test coverage, and concentrate on getting things out in order of long-term profitability for your business.

Good luck with your business! :)

Configuring & Optimizing WebSocket Compression

Posted 5 months back at igvita.com

Good news, browser support for the latest draft of “Compression Extensions” for WebSocket protocol — a much needed and overdue feature — will be landing in early 2014: Chrome M32+ (available in Canary already), and Firefox and Webkit implementations should follow.

Specifically, it enables the client and server to negotiate a compression algorithm and its parameters, and then selectively apply it to the data payloads of each WebSocket message: the server can compress delivered data to the client, and the client can compress data sent to the server.


Negotiating compression support and parameters #

Per-message compression is a WebSocket protocol extension, which means that it must be negotiated as part of the WebSocket handshake. Further, unlike a regular HTTP request (e.g. XMLHttpRequest initiated by the browser), WebSocket also allows us to negotiate compression parameters in both directions (client-to-server and server-to-client). That said, let's start with the simplest possible case:

GET /socket HTTP/1.1
Host: thirdparty.com
Origin: http://example.com
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Extensions: permessage-deflate
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Access-Control-Allow-Origin: http://example.com
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
Sec-WebSocket-Extensions: permessage-deflate

The client initiates the negotiation by advertising the permessage-deflate extension in the Sec-Websocket-Extensions header. In turn, the server must confirm the advertised extension by echoing it in its response.

If the server omits the extension confirmation then the use of permessage-deflate is declined, and both the client and server proceed without it - i.e. the handshake completes and messages won't be compressed. Conversely, if the extension negotiation is successful, both the client and server can compress transmitted data as necessary:

  • Current standard uses Deflate compression.
  • Compression is only applied to application data: control frames and frame headers are unaffected.
  • Both client and server can selectively compress individual frames: if the frame is compressed, the RSV1 bit in the WebSocket frame header is set.

Selective message compression #

Selective compression is a particularly interesting and a useful feature. Just because we've negotiated compression support, doesn't mean that all messages must be compressed! After all, if the payload is already compressed (e.g. image data or any other compressed payload), then running deflate on each frame would unnecessarily waste CPU cycles on both ends. To avoid this, WebSocket allows both the server and client to selectively compress individual messages.

How do the server and client know when to compress data? This is where your choice of a WebSocket server and API can make a big difference: a naive implementation will simply compress all message payloads, whereas a smart server may offer an additional API to indicate which payloads should be compressed.

Similarly, the browser can selectively compress transmitted payloads to the server. However, this is where we run into our first limitation: the WebSocket browser API does not provide any mechanism to signal whether the payload should be compressed. As a result, the current implementation in Chrome compresses all payloads - if you're already transferring compressed data over WebSocket without deflate extension then this is definitely something you should consider as it may add unnecessary overhead on both sides of the connection.

In theory, in absence of an official API, or a per-message flag to indicate a compressed message, the UA could run a “data randomness” test to see if the data should be compressed. However, this by itself can add non-trivial processing overhead.

Optimizing and scaling Deflate compression #

Compressed payloads can significantly reduce the amount of transmitted data, which leads to bandwidth savings and faster message delivery. That said, there are some costs too! Deflate uses a combination of LZ77 and Huffman coding to compress data: first, LZ77 is used to eliminate duplicate strings; second, Huffman coding is used to encode common bit sequences with shorter representations.

By default, enabling compression will add at least ~300KB of extra memory overhead per WebSocket connection - arguably, not much, but if your server is juggling a large number of WebSocket connections, or if the client is running on a memory-limited device, then this is something that should be taken into account. The exact calculation based on zlib implementation of Deflate is as follows:

        compressor = (1 << (windowBits + 2)) + (1 << (memLevel + 9))
      decompressor = (1 << windowBits) + 1440 * 2 * sizeof(int)
             total = compressor + decompressor

Both peers maintain separate compression and decompression contexts, each of which require a separate LZ77 window buffer (as defined by windowBits), plus additional overhead for the Huffman tree and other compressor and decompressor overhead. The default settings are:

  • compressor: windowBits = 15, memLevel = 8 → ~256KB
  • decompressor: windowBits = 15 → ~44KB

The good news is that permessage-deflate allows us to customize the size of the LZ77 window and thus limit the memory overhead via two extension parameters: {client, server}_no_context_takeover and {client, server}_max_window_bits. Let's take a look under the hood...

Optimizing LZ77 window size #

A full discussion of LZ77 and Huffman coding is outside the scope of this post, but to understand the above extension parameters, let's first take a small detour to understand what we are configuring and the inherent tradeoffs between memory and compression performance.

The windowBits parameter is customizing the size of the “sliding window” used by the LZ77 algorithm. Above video is a great visual demonstration of LZ77 at work: the algorithm maintains a “sliding window” of previously seen data and replaces repeated strings (indicated in red) with back-references (e.g. go back X characters, copy Y characters) - that's LZ77 in a nutshell. As a result, the larger the window, the higher the likelihood that LZ77 will find and eliminate duplicate strings.

How large is the LZ77 sliding window? By default, the window is initialized to 15 bits, which translates to 215 bits (32KB) of space. However, we can customize the size of the sliding window as part of the WebSocket handshake:

GET /socket HTTP/1.1
Upgrade: websocket
Sec-WebSocket-Key: ...
Sec-WebSocket-Extensions: permessage-deflate;
  client_max_window_bits; server_max_window_bits=10
  • The client advertises that it supports custom window size via client_max_window_bits
  • The client requests that the server should use a window of size 210 (1KB)
HTTP/1.1 101 Switching Protocols
Connection: Upgrade
Sec-WebSocket-Accept: ...
Sec-WebSocket-Extensions: permessage-deflate;
  server_max_window_bits=10
  • The server confirms that it will use a 210 (1KB) window size
  • The server opts-out from requesting a custom client window size

Both the client and server must maintain the same sliding windows to exchange data: one buffer for client → server compression context, and one buffer for server → client context. As a result, by default, we will need two 32KB buffers (default window size is 15 bits), plus other compressor overhead. However, we can negotiate a smaller window size: in the example above, we limit the server → client window size to 1KB.

Why not start with the smaller buffer? Simple, the smaller the window the less likely it is that LZ77 will find an appropriate back-reference. That said, the performance will vary based on the type and amount of transmitted data and there is no single rule of thumb for best window size. To get the best performance, test different window sizes on your data! Then, where needed, decrease window size to reduce memory overhead.

Optimizing Deflate memLevel #

The memLevel parameter controls the amount of memory allocated for internal compression state: when set to 1, it uses the least memory, but slows down the compression algorithm and reduces the compression ratio; when set to 9, it uses the most memory and delivers the best performance. The default memLevel is set to 8, which results in ~133KB of required memory overhead for the compressor.

Note that the decompressor does not need to know the memLevel chosen by the compressor. As a result, the peers do not need to negotiate this setting in the handshake - they are both free to customize this value as they wish. The server can tune this value as required to tradeoff speed, compression ratio, and memory - once again, the best setting will vary based on your data stream and operational requirements.

Unfortunately, the client, which in this case is the browser user-agent does not provide any API to customize the memLevel of the compressor: memLevel = 8 is used as a default value in all cases. Similar to the missing per-message compression flag, perhaps this is a feature that can be added to a future revision of the WebSocket spec.

Context takeover #

By default, the compression and decompression contexts are persisted across different WebSocket messages - i.e. the sliding window from previous message is used to encode content of the next message. If messages are similar — as they usually are — this improves the compression ratio. However, the downside is that the context overhead is a fixed cost for the entire lifetime of the connection - i.e. memory must be allocated at the beginning and must be maintained until the connection is closed.

Well, what if we relaxed this constraint and instead allowed the peers to reset the context between the different messages? That's what “no context takeover” option is all about:

GET /socket HTTP/1.1
Upgrade: websocket
Sec-WebSocket-Key: ...
Sec-WebSocket-Extensions: permessage-deflate;
  client_max_window_bits; server_max_window_bits=10;
  client_no_context_takeover; server_no_context_takeover
  • Client advertises that it will disable context takeover
  • Client requests that the server also disables context takeover
HTTP/1.1 101 Switching Protocols
Connection: Upgrade
Sec-WebSocket-Accept: ...
Sec-WebSocket-Extensions: permessage-deflate;
  server_max_window_bits=10; client_max_window_bits=12
  client_no_context_takeover; server_no_context_takeover
  • Server acknowledges the client “no context takeover” recommendation
  • Server indicates that it will disable context takeover

Disabling “context takeover” prevents the peer from using the compressor context from a previous message to encode contents of the next message. In other words, each message is encoded with its own sliding window and Huffman tree.

The upside of disabling context takeover is that both the client and server can reset their contexts between different messages, which significantly reduces the total overhead of the connection when its idle. That said, the downside is that compression performance will likely suffer as well. How much? You guessed it, the answer depends on the actual application data being exchanged.

Note that even without “no_context_takeover” negotiation, the decompressor should be able to decode both types of messages. That said, the explicit negotiation is what allows us to know that it is safe to reset the context on the receiver.

Optimizing compression parameters #

Now that we know what we're tweaking, a simple ruby script can help us iterate over all of the options (memLevel and window size) to pick the optimal settings. For the sake of an example, let's compress the GitHub timeline:

$> curl https://github.com/timeline.json -o timeline.json
$> ruby compare.rb timeline.json
Original file (timeline.json) size: 30437 bytes
Window size: 8 bits (256 bytes)
   memLevel: 1, compressed size: 19472 bytes (36.03% reduction)
   memLevel: 9, compressed size: 15116 bytes (50.34% reduction)

Window size: 11 bits (2048 bytes)
   memLevel: 1, compressed size: 9797 bytes (67.81% reduction)
   memLevel: 9, compressed size: 8062 bytes (73.51% reduction)

Window size: 15 bits (32768 bytes)
   memLevel: 1, compressed size: 8327 bytes (72.64% reduction)
   memLevel: 9, compressed size: 7027 bytes (76.91% reduction)

The smallest allowed window size (256 bytes) provides ~50% compression, and raising the window to 2KB, takes us to ~73%! From there, it is diminishing returns: 32KB window yields only a few extra percent (see full output). Hmm! If I was streaming this data over a WebSocket, a 2KB window size seems like a reasonable optimization.

Deploying WebSocket compression #

Customizing LZ77 window size and context takeover are advanced optimizations. Most applications will likely get the best performance by simply using the defaults (32KB window size and shared sliding window). That said, it is useful to understand the incurred overhead (upwards of 300KB per connection), and the knobs that can help you tweak these parameters!

Looking for a WebSocket server that supports per-message compression? Rumor has it, Jetty, Autobahn and WebSocket++ already support it, and other servers (and clients) are sure to follow. For a deep-dive on the negotiation workflow, frame layouts, and more, check out the official specification.

P.S. For more WebSocket optimization tips: WebSocket chapter in High Performance Browser Networking.

Faster Grepping in Vim

Posted 5 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

We often search our projects for specific text within Vim like so:

:grep sometext

However, we have a need for speed. How can we grep faster?

Override to use The Silver Searcher

grep is a built-in command of Vim. By default, it will use our system's grep command. We can overwrite it to use The Silver Searcher's ag command instead by putting this in our ~/.vimrc:

" The Silver Searcher
if executable('ag')
  " Use ag over grep
  set grepprg=ag\ --nogroup\ --nocolor

  " Use ag in CtrlP for listing files. Lightning fast and respects .gitignore
  let g:ctrlp_user_command = 'ag %s -l --nocolor -g ""'

  " ag is fast enough that CtrlP doesn't need to cache
  let g:ctrlp_use_caching = 0
endif

Search for the word under the cursor

This searches for the text under the cursor and shows the results in a "quickfix" window:

" bind K to grep word under cursor
nnoremap K :grep! "\b<C-R><C-W>\b"<CR>:cw<CR>

It looks like this when we hit K with our cursor over SubscriptionMailer in the Learn repo:

Cursor over each search result, hit Enter, and the file will be opened.

Using ag arguments

This defines a new command Ag to search for the provided text and open a "quickfix" window:

" bind \ (backward slash) to grep shortcut
command -nargs=+ -complete=file -bar Ag silent! grep! <args>|cwindow|redraw!

We can map it to any character, such as \:

nnoremap \ :Ag<SPACE>

When \ is pressed, Vim waits for our input:

:Ag

Standard ag arguments may be passed in at this point:

:Ag -i Stripe app/models

Hitting Enter results in:

What's next?

If you found this useful, you might also enjoy:

Episode #422 – November 26th, 2013

Posted 5 months back at Ruby5

Ruby fixes a heap overflow, Rack::Attack protects your app from abusive clients, a proposal for changing Rails Session Storage, learn about Rake File Tasks and Demystify the Ruby GC.

Listen to this episode on Ruby5

This episode is sponsored by Top Ruby Jobs
If you're looking for a top Ruby job or for top Ruby talent, then you should check out Top Ruby Jobs. Top Ruby Jobs is a website dedicated to the best jobs available in the Ruby community.

Heap overflow in Ruby's Floating Point Parsing
A security vulnerability has been found in MRI late last week, which exploits a heap overflow when converting strings to floating point. All versions of MRI Ruby are affected. The Ruby core team has already fixed the issue and released new versions for Ruby 1.9, 2.0 and the 2.1 preview.

Rack::Attack
Rack::­Attack is a rack middleware from the Kickstarter team for blocking & throttling abusive clients.

Rails Session Storage
Matt Aimonetti started an interesting discussion regarding the default session storage serialization in Rails. He proposes a way to specify a custom serializer to be used for the Cookie Session Store, instead of having it hardcoded to use Marshal.

Rake File Tasks
Rake file tasks offer a handy way to work with files and Jacob Swanner wrote a blog post describing how they work. Basically the name of a file task is the same as a file’s name. When invoked, Rake determines that a file task needs to be run if the file doesn’t exist or if any of the prerequisite file tasks have a more recent timestamp.

Demystifying the Ruby GC
Sam Saffron published a post on the official Discourse forum which demystifies the Ruby Garbage Collector. Specifically, the MRI Ruby 2.0 GC. He introduces heaps and RVALUEs, touches on how different architectures vary with regard to them, and he walks through the information you can get from a GC.stat command.

Anti-Pattern: Iteratively Building a Collection

Posted 5 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

Ruby comes with many fantastic Enumerable methods, but the two most useful ones come through Smalltalk via LISP: #map and #inject. What follows are some lengthy method definitions followed by rewrites that are not only more concise but also more clear in their intentions.

Building an array

Requirement:

As a user with a PGP key I want to see the list of key ids for all my signers so I can quickly import them from the keyserver.

The initial implementation is a little lengthy and overly explicit:

def signer_key_ids
  result = []

  signers.each do |signer|
    result << signer.key_id
  end

  result
end

But a simple use of #map more clearly illuminates what this method does:

def signer_key_ids
  signers.map { |signer| signer.key_id }
end

Building an array from multiple arrays

Another requirement comes in:

As a user with a PGP key I want to see the list of all UIDs for all my signers so I can see their names and where they work.

We can write this in a structured way using #each and #flatten:

def signer_uids
  result = []

  signers.each do |signer|
    result << signer.uids
  end

  result.flatten
end

But a #map makes it more clear. Note the use of Symbol#to_proc here:

def signer_uids
  signers.map(&:uid).flatten
end

An #inject combined with Array#+ removes the need to call #flatten at the end:

def signer_uids
  signers.inject([]) do |result, signer|
    result + signer.uids
  end
end

Build a hash from an array

Another requirement comes in from above:

As a user with a PGP key I want to see a mapping of all key ids to their UIDs for each signer so I can build my own keyserver.

Well we need to build a hash, and we need to build it from each element in an array. At least, that's one way to phrase it:

def signer_keys_and_uids
  result = {}

  signers.each do |signer|
    result[signer.key_id] = signer.uids
  end

  result
end

But another way to phrase it is: given an empty hash, #inject a hash from key id to UIDs for each element in the array of signers:

def signer_keys_and_uids
  signers.inject({}) do |result, signer|
    result.merge(signer.key_id => signer.uids)
  end
end

Build a Boolean from an array

One last requirement, they swear:

As a user with a PGP key I want to confirm that all my signers are signed by me so I can always feel mutually complete.

With the hash above we were dealing with another Enumerable. Here it's a Boolean, so let's try it the long way:

def mutually_signed?
  result = true

  signers.each do |signer|
    result = result && signer.signed_by?(self)
  end

  result
end

Though, now that we've seen that, it looks a bit familiar:

def mutually_signed?
  signers.inject(true) do |result, signer|
    result && signer.signed_by?(self)
  end
end

But if that's too obtuse, we can always think of it as an array of Booleans that must all be true:

def mutually_signed?
  signers.map(&:signed_by?).inject(:&)
end

As Rubyists we also know that we have other fantastic abstractions up our Enumerable sleeve:

def mutually_signed?
  signers.all?(&:signed_by?)
end

What's next?

To get a comfortable intuition with #map, #inject, and other Enumerable methods, I recommend going outside of Ruby for a bit. Some amazing books on the topic of functional programming are:

If you want to read more about #inject in Ruby, check out these articles:

Giving thanks

Posted 5 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

Happy Thanksgiving! We're breaking out the Thanksgiving pants and remembering what we're thankful for this year.

Our clients and customers

Thank you to our clients Abundant Capital LLC, Advanced Energy Economy, AllTrails, Arcadia Solutions, Arthritis Foundation, BabelBase, Babelbase, Blue Sky Broadcast, BodyMap+, Brewster, Business Intelligence Associates, CareZone, Charity, Chilling Effects, CoachUp, Constant Contact, CountIt, Darby Smart, DealRoom, Elkins Kalt, Funding Gates, Gift of Knowledge, Great Engagements, Groupize, Hampstead Capital, Hampstead Capital, Highland Capital, InTheKnow, Induro Capital, Induro Capital, Klarna, LevelUp, Locus Control, Mag+, ManhattanPrep, Merge.io, Moser Silver and Hoak, Neon, Omakase, Omni, Omni, Overture Media, Pixbi, Propel Marketing, Redis to Go, Resonance Labs, RosterWrangler, Schoolkeep, Sermo, Skillable, Skillable, SnapEngage, Spogo, T1D Exchange, TDDium, TeamWork Online, Threadflip, Thrively, Valence, WoodSnap, Wootric, and Yammer, for trusting us to work on your products.

Thank you to all of our workshop alumni, those of you who have bought one of our ebooks or screencasts, and everyone else who is a Learn customer.

Software as a service providers

Thank you to Heroku for your awesome support and services hosting our applications.

Thank you to GitHub for hosting all of our open source and private code.

Thank you to Fog Creek Software for Trello.

Thank you to Dribbble for inspiring us.

Thank you to CodeClimate for providing insight about the quality of our code.

Thank you to TDDium for making our tests run faster.

Thank you to Rubygems.org for hosting software which makes our lives easier.

Thank you to 37signals for Campfire and Basecamp.

Thank you to New Relic for making performance monitoring pleasant.

Thank you to Amazon for S3.

Thank you to Fastly for speeding up the delivery of our images, stylesheets, JavaScripts, HTML, and JSON.

Thank you to Dropbox for hosting large design assets and important files.

Thank you to Hoefler & Frere-Jones for their typography, which we're using on this very blog.

Thank you to Typekit for serving up fonts for us and our clients.

Thank you to Google for GMail, Analytics, Adwords, Hangouts, and search.

Business partners

Thank you to Gesmer Updegrove for handling our legal needs.

Thank you to AccountingDepartment.com for handling our accounting and bookkeeping.

Thank you to WeWork for our previous home in San Francisco.

Thank you to Galvanize in Denver and Fuse in Boulder for coworking and meetup space.

Thank you to WebAssign in Raleigh for hosting Ruby meetups.

Thank you to Justin Dziama, CBRE, and Richards Barry Joyce and Partners for helping us find our current home in San Francisco.

Thank you to TMF Group and Newcomers for helping us expand into Stockholm.

Open source contributors

Thank you to every person who submits a pull request to our open source projects, even the ones we don’t merge.

Thank you to Linus Torvalds for Git.

Thank you to Matz for Ruby.

Thank you to DHH, the Rails core team, and the Rails community for Rails.

Thank you to Yehuda Katz and Carl Lerche for Bundler.

Thank you to John Resig for jQuery.

Thank you to Jonas Nicklas for Capybara, an intuitive browser simulation framework.

Thank you to Travis CI for letting us know when our code is working (or broken) on various versions of Ruby.

Thank you to KDE, Apple, Google, Trolltech, and Nokia for Webkit and QtWebKit, which enabled our capybara-webkit.

Thank you to Google and Paul Irish for Chrome Developer Tools.

Thank you to Bill Joy, Bram Moolenaar, and Tim Pope for making and improving Vim, our preferred text editor since forever.

Thank you to the many Postgres committers for a rock-solid and always-improving database.

Thank you to Mattt Thompson for AFNetworking and NSHipster.

Thank you to Allen Ding, Marin Usalj, Peter Kim, and all the contributors to Kiwi and Specta/Expecta for making TDD in Objective-C a viable option.

Thank you to Elloy Durán, Fabio Pelosin, and Orta Therox for CocoaPods, which brings dependency management in Objective-C into the modern age.

Thank you to Hampton Catlin, Nathan Weizenbaum, and Chris Eppstein for Haml and Sass.

Thank you to David Chelimsky and Myron Marston for RSpec, which we use on all our apps.

Thank you to Max Howell for making it simple to install dependencies like C compilers, Postgres, Ack, Exuberant Ctags, tmux, ImageMagick, Redis, and more with Homebrew.

Thank you to Wayne E. Seguin, Michal Papis, and 37signals for making it easy to manage Ruby versions with RVM and rbenv.

Thank you Nicholas Marriott for making it easier to manage various terminals with tmux.

Teachers

Thank you to Alan Kay for object-oriented programming.

Thank you to Martin Fowler for refactoring.

Thank you to the Gang of Four for design patterns.

Thank you to Uncle Bob Martin for clean code.

Thank you to Steve Jobs and Jony Ive for setting the bar high.

Thank you to Gerard Meszaros for xUnit testing patterns.

Thank you to Sandi Metz for Principles of Object-Oriented Design in Ruby.

Event organizers and hosts

Thank you to Brian Cardarella, Patrick Robertson, and all of the Boston Ruby Group for a fantastic local Ruby community with great talks and a calendar full of hackfests.

Thank you to Brightcove, ZenDesk, and ApartmentList for hosting fun Ruby and vim meetups in Boston and San Francisco.

Thank you Wrapp, The Park, SUP46, and Spotify for hosting meetups in Stockholm.

Thank you to Microsoft NERD for being a classy, dependable event host.

Thank you to Chrome Dev Summit, DotRB, Eurucamp, Future of Web Apps London, GoGaRuCo, Growth Hacker's Conference, Lean Startup Conference, Nickel City Ruby, Nordic Ruby, RailsConf, RealtimeConf, Rocky Mountain Ruby, RubyConf, SSWC, STHLM Tech, Seedcamp, Smashing Conference, Steel City Ruby, Stockholm Ruby, Tel-Aviv Ruby, Ultimate Developer Event Boston, Warm Gun, Waza, and Wicked Good Ruby, for bringing together those passionate about design, Ruby, iOS, open source, vim, Unix, and more.

And thank you, too, for reading, commenting, and making us think.