Happy Thanksgiving!

Posted 5 months back at Mike Clark

Great Gray Takeoff

Today I'm also thankful for the birds who aren't turkeys.

NPM needs our help, and why you should donate

Posted 5 months back at Phusion Corporate Blog


NPM must be one of the most invaluable tools for Node.js users. It’s used by tons of users all around the world every second. But there have been various issues with the NPM registry lately. One thing is clear: running the NPM registry is not easy and requires a lot of administration time, effort and money. It is for this reason that Nodejitsu, the party who runs the NPM registry, has launched a funding campaign to improve the NPM registry infrastructure.

As active participants in the open source community, we fully understand how difficult it is to run such a service. The sheer amount of resources needed to properly run and maintain something like this, is easily underestimated by a lot of people. The hardware, the bandwidth, shouldn’t cost that much, should they? Once they’re in place, just configure a few things and the thing runs, right?

But setting up a reliable infrastructure and keeping it healthy is by no means a small task. Properly configuring a high-traffic cluster requires a lot of time and expertise. And there will always be unexpected problems. What happens if there’s a network problem? What happens if an OS upgrade broke things? What happens if there’s a disk failure? Even with all the automatic failover you configure, Murphey’s law will always find you. And when it does, you better be prepared and have the man power to solve the issues. Man power costs time and money.

One might think that Nodejitsu, being a commercial, VC-funded company can easily pay for this themselves. So why should anybody donate? Scalenpm.org explains this: their investors want their money to be used for making more money, not for providing free community services. All donations go toward running the NPM registry.

We all rely on NPM a lot. Even though NPM is open source, it is important not to equate open source with free. It is not free for the party that has to run and maintain it. So we believe it’s time to give something back as well, which is why we have donated. You may want to consider donating as well.

5 things I’ve learned in 5 years of running a SaaS

Posted 5 months back at mir.aculo.us - Home

5045502202_41476791a4_o

Photo Credit: Will Clayton cc

Freckle Time Tracking is turning five on December 1. In 5 years of being a co-founder of Freckle I’ve learned a lot of things, but here are 5 important takeaways. Maybe it helps you on your path to product nirvana:

1. You’re not a “tech company”—you’re a “make customers awesome” company
People don’t pay you because you have amazing programming skills and can write nginx configurations blindfolded. People pay you money because the product you sell to them saves them time, money, effort and nerves. It’s your job to make your customer more awesome. Every decision you make for your product and business should revolve around that.

2. Never promise dates for a feature launch
Just don’t promise launch dates for a feature. Ever. Trust me on this. People will ask you all the time when “feature X” is ready. A good way to answer that question is (if you plan on doing it), “We’re considering this feature for a future version. I can’t give you a date on when it will be ready.”. Just be honest to your customers—you don’t know yourself if and when a feature will really be ready.

3. Spend money on things that help you stay productive
This includes obvious stuff like a laptop that doesn’t suck (upgrade often), a good working chair and desk, and less obvious things like software that allows you to concentrate on developing your application’s features rather than configuring servers.

4. Do not work too much
Overworking yourself is the first step to failure in business. You can’t do your best if you’re permanently stressed out. Don’t check email in the evenings. If you’re only 1 or 2 people, don’t provide 24/7 support. It’s ok. Customers understand. It helps to not have a mission-critical product (if Time Tracking goes down it’s annoying but people can take a note on paper).

You didn’t start a company to die of exhaustion. Your health, family and social life is more important than 5 minute support response times and a 100% uptime guarantee.

By the way, one way to keep on top of this is to keep track on how you spend your time.

5. Don’t believe the hype
People are good at getting excited. And people are good at believing the hype™ about new technologies, frameworks, programming languages and was to deploy. People will tell you what to do and what to plan for. That you need to scale to millions of users, and you’re doomed if you don’t plan for that. That generating HTML on the server is so 1994. That node.js will cure cancer.

The fact is that you need to be pragmatic—your goal is to run a business. Use technology that is proven (to you), and that you know how to work with. My “litmus test” for technology is if the people that provide it are in a similar situation as you are: having to rely on it to run their own business (this quickly weeds out cool-but-academic-only stuff). You need to optimize for shipping. That includes writing less code, having broad test coverage, and concentrate on getting things out in order of long-term profitability for your business.

Good luck with your business! :)

Configuring & Optimizing WebSocket Compression

Posted 5 months back at igvita.com

Good news, browser support for the latest draft of “Compression Extensions” for WebSocket protocol — a much needed and overdue feature — will be landing in early 2014: Chrome M32+ (available in Canary already), and Firefox and Webkit implementations should follow.

Specifically, it enables the client and server to negotiate a compression algorithm and its parameters, and then selectively apply it to the data payloads of each WebSocket message: the server can compress delivered data to the client, and the client can compress data sent to the server.


Negotiating compression support and parameters #

Per-message compression is a WebSocket protocol extension, which means that it must be negotiated as part of the WebSocket handshake. Further, unlike a regular HTTP request (e.g. XMLHttpRequest initiated by the browser), WebSocket also allows us to negotiate compression parameters in both directions (client-to-server and server-to-client). That said, let's start with the simplest possible case:

GET /socket HTTP/1.1
Host: thirdparty.com
Origin: http://example.com
Connection: Upgrade
Upgrade: websocket
Sec-WebSocket-Version: 13
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Sec-WebSocket-Extensions: permessage-deflate
HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Access-Control-Allow-Origin: http://example.com
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
Sec-WebSocket-Extensions: permessage-deflate

The client initiates the negotiation by advertising the permessage-deflate extension in the Sec-Websocket-Extensions header. In turn, the server must confirm the advertised extension by echoing it in its response.

If the server omits the extension confirmation then the use of permessage-deflate is declined, and both the client and server proceed without it - i.e. the handshake completes and messages won't be compressed. Conversely, if the extension negotiation is successful, both the client and server can compress transmitted data as necessary:

  • Current standard uses Deflate compression.
  • Compression is only applied to application data: control frames and frame headers are unaffected.
  • Both client and server can selectively compress individual frames: if the frame is compressed, the RSV1 bit in the WebSocket frame header is set.

Selective message compression #

Selective compression is a particularly interesting and a useful feature. Just because we've negotiated compression support, doesn't mean that all messages must be compressed! After all, if the payload is already compressed (e.g. image data or any other compressed payload), then running deflate on each frame would unnecessarily waste CPU cycles on both ends. To avoid this, WebSocket allows both the server and client to selectively compress individual messages.

How do the server and client know when to compress data? This is where your choice of a WebSocket server and API can make a big difference: a naive implementation will simply compress all message payloads, whereas a smart server may offer an additional API to indicate which payloads should be compressed.

Similarly, the browser can selectively compress transmitted payloads to the server. However, this is where we run into our first limitation: the WebSocket browser API does not provide any mechanism to signal whether the payload should be compressed. As a result, the current implementation in Chrome compresses all payloads - if you're already transferring compressed data over WebSocket without deflate extension then this is definitely something you should consider as it may add unnecessary overhead on both sides of the connection.

In theory, in absence of an official API, or a per-message flag to indicate a compressed message, the UA could run a “data randomness” test to see if the data should be compressed. However, this by itself can add non-trivial processing overhead.

Optimizing and scaling Deflate compression #

Compressed payloads can significantly reduce the amount of transmitted data, which leads to bandwidth savings and faster message delivery. That said, there are some costs too! Deflate uses a combination of LZ77 and Huffman coding to compress data: first, LZ77 is used to eliminate duplicate strings; second, Huffman coding is used to encode common bit sequences with shorter representations.

By default, enabling compression will add at least ~300KB of extra memory overhead per WebSocket connection - arguably, not much, but if your server is juggling a large number of WebSocket connections, or if the client is running on a memory-limited device, then this is something that should be taken into account. The exact calculation based on zlib implementation of Deflate is as follows:

        compressor = (1 << (windowBits + 2)) + (1 << (memLevel + 9))
      decompressor = (1 << windowBits) + 1440 * 2 * sizeof(int)
             total = compressor + decompressor

Both peers maintain separate compression and decompression contexts, each of which require a separate LZ77 window buffer (as defined by windowBits), plus additional overhead for the Huffman tree and other compressor and decompressor overhead. The default settings are:

  • compressor: windowBits = 15, memLevel = 8 → ~256KB
  • decompressor: windowBits = 15 → ~44KB

The good news is that permessage-deflate allows us to customize the size of the LZ77 window and thus limit the memory overhead via two extension parameters: {client, server}_no_context_takeover and {client, server}_max_window_bits. Let's take a look under the hood...

Optimizing LZ77 window size #

A full discussion of LZ77 and Huffman coding is outside the scope of this post, but to understand the above extension parameters, let's first take a small detour to understand what we are configuring and the inherent tradeoffs between memory and compression performance.

The windowBits parameter is customizing the size of the “sliding window” used by the LZ77 algorithm. Above video is a great visual demonstration of LZ77 at work: the algorithm maintains a “sliding window” of previously seen data and replaces repeated strings (indicated in red) with back-references (e.g. go back X characters, copy Y characters) - that's LZ77 in a nutshell. As a result, the larger the window, the higher the likelihood that LZ77 will find and eliminate duplicate strings.

How large is the LZ77 sliding window? By default, the window is initialized to 15 bits, which translates to 215 bits (32KB) of space. However, we can customize the size of the sliding window as part of the WebSocket handshake:

GET /socket HTTP/1.1
Upgrade: websocket
Sec-WebSocket-Key: ...
Sec-WebSocket-Extensions: permessage-deflate;
  client_max_window_bits; server_max_window_bits=10
  • The client advertises that it supports custom window size via client_max_window_bits
  • The client requests that the server should use a window of size 210 (1KB)
HTTP/1.1 101 Switching Protocols
Connection: Upgrade
Sec-WebSocket-Accept: ...
Sec-WebSocket-Extensions: permessage-deflate;
  server_max_window_bits=10
  • The server confirms that it will use a 210 (1KB) window size
  • The server opts-out from requesting a custom client window size

Both the client and server must maintain the same sliding windows to exchange data: one buffer for client → server compression context, and one buffer for server → client context. As a result, by default, we will need two 32KB buffers (default window size is 15 bits), plus other compressor overhead. However, we can negotiate a smaller window size: in the example above, we limit the server → client window size to 1KB.

Why not start with the smaller buffer? Simple, the smaller the window the less likely it is that LZ77 will find an appropriate back-reference. That said, the performance will vary based on the type and amount of transmitted data and there is no single rule of thumb for best window size. To get the best performance, test different window sizes on your data! Then, where needed, decrease window size to reduce memory overhead.

Optimizing Deflate memLevel #

The memLevel parameter controls the amount of memory allocated for internal compression state: when set to 1, it uses the least memory, but slows down the compression algorithm and reduces the compression ratio; when set to 9, it uses the most memory and delivers the best performance. The default memLevel is set to 8, which results in ~133KB of required memory overhead for the compressor.

Note that the decompressor does not need to know the memLevel chosen by the compressor. As a result, the peers do not need to negotiate this setting in the handshake - they are both free to customize this value as they wish. The server can tune this value as required to tradeoff speed, compression ratio, and memory - once again, the best setting will vary based on your data stream and operational requirements.

Unfortunately, the client, which in this case is the browser user-agent does not provide any API to customize the memLevel of the compressor: memLevel = 8 is used as a default value in all cases. Similar to the missing per-message compression flag, perhaps this is a feature that can be added to a future revision of the WebSocket spec.

Context takeover #

By default, the compression and decompression contexts are persisted across different WebSocket messages - i.e. the sliding window from previous message is used to encode content of the next message. If messages are similar — as they usually are — this improves the compression ratio. However, the downside is that the context overhead is a fixed cost for the entire lifetime of the connection - i.e. memory must be allocated at the beginning and must be maintained until the connection is closed.

Well, what if we relaxed this constraint and instead allowed the peers to reset the context between the different messages? That's what “no context takeover” option is all about:

GET /socket HTTP/1.1
Upgrade: websocket
Sec-WebSocket-Key: ...
Sec-WebSocket-Extensions: permessage-deflate;
  client_max_window_bits; server_max_window_bits=10;
  client_no_context_takeover; server_no_context_takeover
  • Client advertises that it will disable context takeover
  • Client requests that the server also disables context takeover
HTTP/1.1 101 Switching Protocols
Connection: Upgrade
Sec-WebSocket-Accept: ...
Sec-WebSocket-Extensions: permessage-deflate;
  server_max_window_bits=10; client_max_window_bits=12
  client_no_context_takeover; server_no_context_takeover
  • Server acknowledges the client “no context takeover” recommendation
  • Server indicates that it will disable context takeover

Disabling “context takeover” prevents the peer from using the compressor context from a previous message to encode contents of the next message. In other words, each message is encoded with its own sliding window and Huffman tree.

The upside of disabling context takeover is that both the client and server can reset their contexts between different messages, which significantly reduces the total overhead of the connection when its idle. That said, the downside is that compression performance will likely suffer as well. How much? You guessed it, the answer depends on the actual application data being exchanged.

Note that even without “no_context_takeover” negotiation, the decompressor should be able to decode both types of messages. That said, the explicit negotiation is what allows us to know that it is safe to reset the context on the receiver.

Optimizing compression parameters #

Now that we know what we're tweaking, a simple ruby script can help us iterate over all of the options (memLevel and window size) to pick the optimal settings. For the sake of an example, let's compress the GitHub timeline:

$> curl https://github.com/timeline.json -o timeline.json
$> ruby compare.rb timeline.json
Original file (timeline.json) size: 30437 bytes
Window size: 8 bits (256 bytes)
   memLevel: 1, compressed size: 19472 bytes (36.03% reduction)
   memLevel: 9, compressed size: 15116 bytes (50.34% reduction)

Window size: 11 bits (2048 bytes)
   memLevel: 1, compressed size: 9797 bytes (67.81% reduction)
   memLevel: 9, compressed size: 8062 bytes (73.51% reduction)

Window size: 15 bits (32768 bytes)
   memLevel: 1, compressed size: 8327 bytes (72.64% reduction)
   memLevel: 9, compressed size: 7027 bytes (76.91% reduction)

The smallest allowed window size (256 bytes) provides ~50% compression, and raising the window to 2KB, takes us to ~73%! From there, it is diminishing returns: 32KB window yields only a few extra percent (see full output). Hmm! If I was streaming this data over a WebSocket, a 2KB window size seems like a reasonable optimization.

Deploying WebSocket compression #

Customizing LZ77 window size and context takeover are advanced optimizations. Most applications will likely get the best performance by simply using the defaults (32KB window size and shared sliding window). That said, it is useful to understand the incurred overhead (upwards of 300KB per connection), and the knobs that can help you tweak these parameters!

Looking for a WebSocket server that supports per-message compression? Rumor has it, Jetty, Autobahn and WebSocket++ already support it, and other servers (and clients) are sure to follow. For a deep-dive on the negotiation workflow, frame layouts, and more, check out the official specification.

P.S. For more WebSocket optimization tips: WebSocket chapter in High Performance Browser Networking.

Faster Grepping in Vim

Posted 5 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

We often search our projects for specific text within Vim like so:

:grep sometext

However, we have a need for speed. How can we grep faster?

Override to use The Silver Searcher

grep is a built-in command of Vim. By default, it will use our system's grep command. We can overwrite it to use The Silver Searcher's ag command instead by putting this in our ~/.vimrc:

" The Silver Searcher
if executable('ag')
  " Use ag over grep
  set grepprg=ag\ --nogroup\ --nocolor

  " Use ag in CtrlP for listing files. Lightning fast and respects .gitignore
  let g:ctrlp_user_command = 'ag %s -l --nocolor -g ""'

  " ag is fast enough that CtrlP doesn't need to cache
  let g:ctrlp_use_caching = 0
endif

Search for the word under the cursor

This searches for the text under the cursor and shows the results in a "quickfix" window:

" bind K to grep word under cursor
nnoremap K :grep! "\b<C-R><C-W>\b"<CR>:cw<CR>

It looks like this when we hit K with our cursor over SubscriptionMailer in the Learn repo:

Cursor over each search result, hit Enter, and the file will be opened.

Using ag arguments

This defines a new command Ag to search for the provided text and open a "quickfix" window:

" bind \ (backward slash) to grep shortcut
command -nargs=+ -complete=file -bar Ag silent! grep! <args>|cwindow|redraw!

We can map it to any character, such as \:

nnoremap \ :Ag<SPACE>

When \ is pressed, Vim waits for our input:

:Ag

Standard ag arguments may be passed in at this point:

:Ag -i Stripe app/models

Hitting Enter results in:

What's next?

If you found this useful, you might also enjoy:

Episode #422 – November 26th, 2013

Posted 5 months back at Ruby5

Ruby fixes a heap overflow, Rack::Attack protects your app from abusive clients, a proposal for changing Rails Session Storage, learn about Rake File Tasks and Demystify the Ruby GC.

Listen to this episode on Ruby5

This episode is sponsored by Top Ruby Jobs
If you're looking for a top Ruby job or for top Ruby talent, then you should check out Top Ruby Jobs. Top Ruby Jobs is a website dedicated to the best jobs available in the Ruby community.

Heap overflow in Ruby's Floating Point Parsing
A security vulnerability has been found in MRI late last week, which exploits a heap overflow when converting strings to floating point. All versions of MRI Ruby are affected. The Ruby core team has already fixed the issue and released new versions for Ruby 1.9, 2.0 and the 2.1 preview.

Rack::Attack
Rack::­Attack is a rack middleware from the Kickstarter team for blocking & throttling abusive clients.

Rails Session Storage
Matt Aimonetti started an interesting discussion regarding the default session storage serialization in Rails. He proposes a way to specify a custom serializer to be used for the Cookie Session Store, instead of having it hardcoded to use Marshal.

Rake File Tasks
Rake file tasks offer a handy way to work with files and Jacob Swanner wrote a blog post describing how they work. Basically the name of a file task is the same as a file’s name. When invoked, Rake determines that a file task needs to be run if the file doesn’t exist or if any of the prerequisite file tasks have a more recent timestamp.

Demystifying the Ruby GC
Sam Saffron published a post on the official Discourse forum which demystifies the Ruby Garbage Collector. Specifically, the MRI Ruby 2.0 GC. He introduces heaps and RVALUEs, touches on how different architectures vary with regard to them, and he walks through the information you can get from a GC.stat command.

Anti-Pattern: Iteratively Building a Collection

Posted 5 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

Ruby comes with many fantastic Enumerable methods, but the two most useful ones come through Smalltalk via LISP: #map and #inject. What follows are some lengthy method definitions followed by rewrites that are not only more concise but also more clear in their intentions.

Building an array

Requirement:

As a user with a PGP key I want to see the list of key ids for all my signers so I can quickly import them from the keyserver.

The initial implementation is a little lengthy and overly explicit:

def signer_key_ids
  result = []

  signers.each do |signer|
    result << signer.key_id
  end

  result
end

But a simple use of #map more clearly illuminates what this method does:

def signer_key_ids
  signers.map { |signer| signer.key_id }
end

Building an array from multiple arrays

Another requirement comes in:

As a user with a PGP key I want to see the list of all UIDs for all my signers so I can see their names and where they work.

We can write this in a structured way using #each and #flatten:

def signer_uids
  result = []

  signers.each do |signer|
    result << signer.uids
  end

  result.flatten
end

But a #map makes it more clear. Note the use of Symbol#to_proc here:

def signer_uids
  signers.map(&:uid).flatten
end

An #inject combined with Array#+ removes the need to call #flatten at the end:

def signer_uids
  signers.inject([]) do |result, signer|
    result + signer.uids
  end
end

Build a hash from an array

Another requirement comes in from above:

As a user with a PGP key I want to see a mapping of all key ids to their UIDs for each signer so I can build my own keyserver.

Well we need to build a hash, and we need to build it from each element in an array. At least, that's one way to phrase it:

def signer_keys_and_uids
  result = {}

  signers.each do |signer|
    result[signer.key_id] = signer.uids
  end

  result
end

But another way to phrase it is: given an empty hash, #inject a hash from key id to UIDs for each element in the array of signers:

def signer_keys_and_uids
  signers.inject({}) do |result, signer|
    result.merge(signer.key_id => signer.uids)
  end
end

Build a Boolean from an array

One last requirement, they swear:

As a user with a PGP key I want to confirm that all my signers are signed by me so I can always feel mutually complete.

With the hash above we were dealing with another Enumerable. Here it's a Boolean, so let's try it the long way:

def mutually_signed?
  result = true

  signers.each do |signer|
    result = result && signer.signed_by?(self)
  end

  result
end

Though, now that we've seen that, it looks a bit familiar:

def mutually_signed?
  signers.inject(true) do |result, signer|
    result && signer.signed_by?(self)
  end
end

But if that's too obtuse, we can always think of it as an array of Booleans that must all be true:

def mutually_signed?
  signers.map(&:signed_by?).inject(:&)
end

As Rubyists we also know that we have other fantastic abstractions up our Enumerable sleeve:

def mutually_signed?
  signers.all?(&:signed_by?)
end

What's next?

To get a comfortable intuition with #map, #inject, and other Enumerable methods, I recommend going outside of Ruby for a bit. Some amazing books on the topic of functional programming are:

If you want to read more about #inject in Ruby, check out these articles:

Giving thanks

Posted 5 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

Happy Thanksgiving! We're breaking out the Thanksgiving pants and remembering what we're thankful for this year.

Our clients and customers

Thank you to our clients Abundant Capital LLC, Advanced Energy Economy, AllTrails, Arcadia Solutions, Arthritis Foundation, BabelBase, Babelbase, Blue Sky Broadcast, BodyMap+, Brewster, Business Intelligence Associates, CareZone, Charity, Chilling Effects, CoachUp, Constant Contact, CountIt, Darby Smart, DealRoom, Elkins Kalt, Funding Gates, Gift of Knowledge, Great Engagements, Groupize, Hampstead Capital, Hampstead Capital, Highland Capital, InTheKnow, Induro Capital, Induro Capital, Klarna, LevelUp, Locus Control, Mag+, ManhattanPrep, Merge.io, Moser Silver and Hoak, Neon, Omakase, Omni, Omni, Overture Media, Pixbi, Propel Marketing, Redis to Go, Resonance Labs, RosterWrangler, Schoolkeep, Sermo, Skillable, Skillable, SnapEngage, Spogo, T1D Exchange, TDDium, TeamWork Online, Threadflip, Thrively, Valence, WoodSnap, Wootric, and Yammer, for trusting us to work on your products.

Thank you to all of our workshop alumni, those of you who have bought one of our ebooks or screencasts, and everyone else who is a Learn customer.

Software as a service providers

Thank you to Heroku for your awesome support and services hosting our applications.

Thank you to GitHub for hosting all of our open source and private code.

Thank you to Fog Creek Software for Trello.

Thank you to Dribbble for inspiring us.

Thank you to CodeClimate for providing insight about the quality of our code.

Thank you to TDDium for making our tests run faster.

Thank you to Rubygems.org for hosting software which makes our lives easier.

Thank you to 37signals for Campfire and Basecamp.

Thank you to New Relic for making performance monitoring pleasant.

Thank you to Amazon for S3.

Thank you to Fastly for speeding up the delivery of our images, stylesheets, JavaScripts, HTML, and JSON.

Thank you to Dropbox for hosting large design assets and important files.

Thank you to Hoefler & Frere-Jones for their typography, which we're using on this very blog.

Thank you to Typekit for serving up fonts for us and our clients.

Thank you to Google for GMail, Analytics, Adwords, Hangouts, and search.

Business partners

Thank you to Gesmer Updegrove for handling our legal needs.

Thank you to AccountingDepartment.com for handling our accounting and bookkeeping.

Thank you to WeWork for our previous home in San Francisco.

Thank you to Galvanize in Denver and Fuse in Boulder for coworking and meetup space.

Thank you to WebAssign in Raleigh for hosting Ruby meetups.

Thank you to Justin Dziama, CBRE, and Richards Barry Joyce and Partners for helping us find our current home in San Francisco.

Thank you to TMF Group and Newcomers for helping us expand into Stockholm.

Open source contributors

Thank you to every person who submits a pull request to our open source projects, even the ones we don’t merge.

Thank you to Linus Torvalds for Git.

Thank you to Matz for Ruby.

Thank you to DHH, the Rails core team, and the Rails community for Rails.

Thank you to Yehuda Katz and Carl Lerche for Bundler.

Thank you to John Resig for jQuery.

Thank you to Jonas Nicklas for Capybara, an intuitive browser simulation framework.

Thank you to Travis CI for letting us know when our code is working (or broken) on various versions of Ruby.

Thank you to KDE, Apple, Google, Trolltech, and Nokia for Webkit and QtWebKit, which enabled our capybara-webkit.

Thank you to Google and Paul Irish for Chrome Developer Tools.

Thank you to Bill Joy, Bram Moolenaar, and Tim Pope for making and improving Vim, our preferred text editor since forever.

Thank you to the many Postgres committers for a rock-solid and always-improving database.

Thank you to Mattt Thompson for AFNetworking and NSHipster.

Thank you to Allen Ding, Marin Usalj, Peter Kim, and all the contributors to Kiwi and Specta/Expecta for making TDD in Objective-C a viable option.

Thank you to Elloy Durán, Fabio Pelosin, and Orta Therox for CocoaPods, which brings dependency management in Objective-C into the modern age.

Thank you to Hampton Catlin, Nathan Weizenbaum, and Chris Eppstein for Haml and Sass.

Thank you to David Chelimsky and Myron Marston for RSpec, which we use on all our apps.

Thank you to Max Howell for making it simple to install dependencies like C compilers, Postgres, Ack, Exuberant Ctags, tmux, ImageMagick, Redis, and more with Homebrew.

Thank you to Wayne E. Seguin, Michal Papis, and 37signals for making it easy to manage Ruby versions with RVM and rbenv.

Thank you Nicholas Marriott for making it easier to manage various terminals with tmux.

Teachers

Thank you to Alan Kay for object-oriented programming.

Thank you to Martin Fowler for refactoring.

Thank you to the Gang of Four for design patterns.

Thank you to Uncle Bob Martin for clean code.

Thank you to Steve Jobs and Jony Ive for setting the bar high.

Thank you to Gerard Meszaros for xUnit testing patterns.

Thank you to Sandi Metz for Principles of Object-Oriented Design in Ruby.

Event organizers and hosts

Thank you to Brian Cardarella, Patrick Robertson, and all of the Boston Ruby Group for a fantastic local Ruby community with great talks and a calendar full of hackfests.

Thank you to Brightcove, ZenDesk, and ApartmentList for hosting fun Ruby and vim meetups in Boston and San Francisco.

Thank you Wrapp, The Park, SUP46, and Spotify for hosting meetups in Stockholm.

Thank you to Microsoft NERD for being a classy, dependable event host.

Thank you to Chrome Dev Summit, DotRB, Eurucamp, Future of Web Apps London, GoGaRuCo, Growth Hacker's Conference, Lean Startup Conference, Nickel City Ruby, Nordic Ruby, RailsConf, RealtimeConf, Rocky Mountain Ruby, RubyConf, SSWC, STHLM Tech, Seedcamp, Smashing Conference, Steel City Ruby, Stockholm Ruby, Tel-Aviv Ruby, Ultimate Developer Event Boston, Warm Gun, Waza, and Wicked Good Ruby, for bringing together those passionate about design, Ruby, iOS, open source, vim, Unix, and more.

And thank you, too, for reading, commenting, and making us think.

Ruby and Python Libraries for Yammer's API

Posted 5 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

We've worked closely with folks at Yammer to help developers build applications on Yammer's API.

For Rubyists, we've written the Yam Gem. It's a light API wrapper:

# gem 'yam'

access_token = 'abcdefghijklmn'
yammer_endpoint = 'https://www.yammer.com/api/v1/'
yam = Yam.new(access_token, yammer_endpoint)

# Post a status message to your Yammer network
yam.post('/messages', body: 'status update')

For Pythonistas, we've written the Yampy package. It features a nice DSL:

from yampy import Yammer

access_token = 'abcdefghijklmn'
yam = Yammer(access_token)

yam.messages.post('Hello world!')

Both repositories are open sourced under Yammer's GitHub account, maintained by folks at Yammer, and are considered official. Contributions are welcome. Enjoy!

Ram-alanche Video

Posted 5 months back at Mike Clark

<iframe src="//player.vimeo.com/video/80163262?title=0&amp;byline=0&amp;portrait=0" width="700" height="394" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe>

When the big ram decided he wanted to get off the mountain, they all came thundering down after him. I love how it started slow and quickly snowballed. Indeed, it was a ram-alanche!

(Apologies for the sloppy panning work. I was totally taken by surprise and not really prepared to shoot video.)

Docking Maneuver

Posted 5 months back at Mike Clark

Docking Maneuver

Great gray owl dials in the thrusters for a precision docking maneuver.

Handling API Rate Limits

Posted 5 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

Has your app ever encountered a 429 (Too Many Requests) status code when making requests to a third-party API? Getting rate limited can be a nuisance and if not handled properly can result in a negative user experience. While one solution is to catch the exception and ignore it, a better solution is to retry the request.

Let's take a look at how we can alleviate rate-limiting woes by utilizing a background job system. In this example we'll use delayed_job, since it provides the ability to retry failed jobs.

We are going to assume that we are accessing an API of a Popular Website. First, we'll create a background job that makes a request to that API.

class MyCustomJob < Struct.new(:username)
  def perform
    PopularSiteApi.get("/feed/#{username}")
  end
end

When this job gets executed a bunch of times in the row, we will potentially reach a limit to how many requests we can make, which is provided by the Popular Website. When that happens, an exception will be raised and our backround job will fail. That's okay, delayed_job will retry any failed job (up to 25 times by default).

Rate limiting can vary from amount of requests per day to amount of requests per minute. For the sake of example, let's assume the latter. Now, delayed_job retries failed jobs in the following manner (from the docs):

On failure, the job is scheduled again in 5 seconds + N ** 4, where N is the number of retries.

In our case, we want to retry our jobs every minute if they fail due to rate limiting. delayed_job provides a method called error which we can define to inspect the exception.

def error(job, exception)
  if is_rate_limit?(exception)
    @rate_limited = true
  else
    @rate_limited = false
  end
end

def is_rate_limit?
  exception.is_a?(Faraday::Error::ClientError) && exception.response[:status] == 429
end

Now, we can retry this job at our known time interval by overriding the reschedule_at method. delayed_job uses reschedule_at to calculate when to re-run the particular job. We can also override the number of times we retry the job (if we want it to be different than the default 25 times).

def reschedule_at(attempts, time)
  if @rate_limited
    next_rate_limit_window
  end
end

def max_attempts
  if @rate_limited
    10
  else
    Delayed::Worker.max_attempts
  end
end

def next_rate_limit_window
  1.minute.from_now
end

Once our custom job is configured thusly, we will retry it every minute, ten times in a row until it works. If the job is still encountering a 429 status code after our retries, it will fail completely. At this point, we'll send out a notification of the failure (using Airbrake) and consider upgrading our API rate plan.

Here's the full code example:

class MyCustomJob < Struct.new(:param1, :param2)
  def perform
    PopularSiteApi.get('/posts')
  end

  def error(job, exception)
    if is_rate_limit?(exception)
      @rate_limited = true
    else
      @rate_limited = false
    end
  end

  def reschedule_at(attempts, time)
    if @rate_limited
      next_rate_limit_window
    end
  end

  def failure(job)
    Airbrake.notify(error_message: "Job failure: #{job.last_error}")
  end

  def max_attempts
    if @rate_limited
      10
    else
      Delayed::Worker.max_attempts
    end
  end

  private

  def is_rate_limit?(exception)
    exception.is_a?(Faraday::Error::ClientError) && exception.response[:status] == 429
  end

  def next_rate_limit_window
    1.minute.from_now
  end
end

Look inside of app/jobs of this open source repository for a real world example.

This week's changelog

Posted 5 months back at entp hoth blog - Home

Hi everyone,

Time flies it seems, and while our little hands are never idle, we don’t always share what we are working on. This is unfortunate as many bugfixes and features, big and small, go into production every day. So let’s mix things up a bit and showcase this week’s changelog.

Bugfixes

  • When posting a comment to a discussion that had changed category since the page was opened, the comment would seemingly disappear (it still existed, but not at the right place). This is now fixed.

  • Tender always had a “duplicate” detection mechanism that trashes multiple copies of the same message (people hitting the form a little too much, or sending the same email multiple times). The detection was a little too eager and would look for messages in a large timeframe. We reduced this timeframe to 2h which is what feels reasonable to catch those accidentals duplicates, while still allowing some customers to keep you on your toes.

Visible Features

  • The sticky queue now provides pagination:

    You can choose which queue you want to always have around in the dashboard:

  • The miniprofile now shows the number of replies:

  • The “attached files” section doesn’t show the C:\fakepath\ anymore:

  • You can now press ? on a discussion page to get a list of available shortcuts on that page.

  • We added some basic validation of all fields when creating a discussion or posting a comment. When you click the submit button, fields are checked and focus will be put back on incorrect values. Fields will turn a light green or red to indicate where the error is:

Invisible Features

  • Improved spam detection for certain types of evil emails.

  • Improved failover mechanism for some of our servers. We had a few minutes of disturbance last week because of a connectivity issue in one datacenter. If this happens again we should now automatically switch to a different server in a different datacenter.

  • Ongoing sysadmin work to improve our infrastructure.

Mobile

The move to mobile is still ongoing.

You can now create new discussions and browse forums:

Only the discussion page is left, but it’s a big chunk. We want to provide all or most of the current functionality to supporters. This will give you more time to adjust the CSS of your site :)

As we make progress on mobile, we are rewriting a lot of the frontend code from Prototype JS (I know…) to Knockout. This is overall a good thing as it will make a lot of things faster and allows us to iterate more quickly in the future. But if you use custom JS, this may break things. So please be on the lookout for errors in your JavaScript console. If we removed code you were using, we will provide you with an alternative.

Finally, still as part of the move to Knockout, we introduced jQuery in mode no conflict to the front part of the site. So $ and $$ still refer to Prototype, but you can now use jQuery. Once the rewrite is complete, we will remove Prototype entirely.

Conclusion

I hope this changelog is helpful and provides a better insight into what keeps us busy. As usual, if you have comments, ideas, hit us up at support@tenderapp.com or post a comment on the discussion.

Cheers!

Short Beaver Video

Posted 5 months back at Mike Clark

<iframe src="//player.vimeo.com/video/80102940?title=0&amp;byline=0&amp;portrait=0" width="700" height="393" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe>

Found this young beaver up on the bank late in the evening just happily munching away.

Wet Behind the Ears

Posted 5 months back at Mike Clark

Wet Behind the Ears

This elk calf didn't really want his ears cleaned... but then it felt SO good!