Intent to Add

Posted about 1 month back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

The git add command runs blind, but can be controlled with more fine-grained precision using the --patch option. This works great for modified and deleted files, but untracked files do not show up.

$ echo "Hello, World!" > untracked
$ git status --short
?? untracked
$ git add --patch
No changes.

To remedy this, the --intent-to-add option can be used. According to git-add(1), --intent-to-add changes git add’s behavior to:

Record only the fact that the path will be added later. An entry for the path is placed in the index with no content. This is useful for, among other things, showing the unstaged content of such files with git diff and committing them with git commit -a.

What this means is that after running git add --intent-to-add, the specified untracked files will be added to the index, but without content. Now, when git add --patch is run it will show a diff for each previously staged untracked file with every line as an addition. This gives you a chance to look through the file, line by line, before staging it. You can even decide not to stage specific lines by deleting them from the patch using the edit command.

$ echo "Hello, World!" > untracked
$ git status --short
?? untracked
$ git add --intent-to-add untracked
$ git status --short
AM untracked
$ git add --patch
diff --git a/untracked b/untracked
index e69de29..8ab686e 100644
--- a/untracked
+++ b/untracked
@@ -0,0 +1 @@
+Hello, World!
Stage this hunk [y,n,q,a,d,/,e,?]?

In my .gitconfig I alias add --all --intent-to-add to aa and add --patch to ap which means that for most commits, I type:

$ git aa
$ git ap

Or in gitsh:

& aa
& ap

Linked Development: Linked Data from CABI and DFID

Posted about 1 month back at RicRoberts :

In March this year we launched the beta version of Linked Development. It’s a linked open data site for CABI and the DFID, which provides data all about international development projects and research.

Linked Development site screenshot

This is a slightly unusual one for us: we’re taking it on after others have worked on it in the past. It’s currently in beta release and we’re hosting it on our PublishMyData service so users can easily get hold of the data they want in both human, and machine readable, formats. So it’s comes with most of the usual benefits we offer: thematic data browsing, a SPARQL endpoint and Linked Data APIs. And, because the data’s linked, each data point has a unique identifier so users can select and combine data from different data sources to get the exact information they’re after.

We’ve also rebuilt the site’s custpom Research Documents API, that was offered by the alpha version of the site to make it faster and more robust (it’s backward-compatible with the previous version).

Linked Development custom API screenshot

This site illustrates what’s possible for government organisations using linked data: it allows for collective ownership of data and data integration whilst aiming to improve audience reach and data availablility. It’s great to see linked data being embraced by an increasing number of public bodies.

DNS to CDN to Origin

Posted about 1 month back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

Content Distribution Networks (CDNs) such as Amazon CloudFront and Fastly have the ability to “pull” content from their origin server during HTTP requests in order to cache them. They can also proxy POST, PUT, PATCH, DELETE, and OPTION HTTP requests, which means they can “front” our web application’s origin like this:

DNS -> CDN -> Origin

Swapping out the concepts for actual services we use, the architecture can look like this:

DNSimple -> CloudFront -> Heroku

Or like this:

DNSimple -> Fastly -> Heroku

Or many other combinations.

Without Origin Pull or an Asset Host

Let’s first examine what it looks like serve static assets (CSS, JavaScript, font, and image files) from a Rails app without a CDN.

We could point our domain name to our Rails app running on Heroku using a CNAME record (apex domains in cloud environments have their own set of eccentricities):

www.thoughtbot.com -> thoughtbot-production.herokuapp.com

We’ll also need to set the following configuration:

# config/environments/{staging,production}.rb
config.serve_static_assets = true

In this setup, we’ll then see something like the following in our logs:

no asset host

That screenshot is from development mode but the same effect will occur in production:

  • all the application’s requests to static assets will go through the Heroku routing mesh,
  • get picked up by one of our web dynos,
  • passed to one of the Unicorn workers on the dyno,
  • then routed by Rails to the asset

This isn’t the best use of our Ruby processes. They should be reserved for handling real logic. Each process should have the fastest possible response time. Overall response time is affected by waiting for other processes to finish their work.

How Can We Solve This?

AssetSync is a popular approach that we have used in the past with success. We no longer use it because there’s no need to copy all files to S3 during deploy (rake assets:precompile). Copying files across the network is wasteful and slow, and gets slower as the codebase grows. S3 is also not a CDN, does not have edge servers, and therefore is slower than CDN options.

Asset Hosts that Support “Origin Pull”

A better alternative is to use services that “pull” the assets from the origin (Heroku) “Just In Time” the first time they are needed. Services we’ve used include CloudFront and Fastly. Fastly is our usual default due to its amazingly quick cache invalidation. Both have “origin pull” features that work well with Rails' asset pipeline.

Because of the asset pipeline, in production, every asset has a hash added to its name. Whenever the file changes, the browser requests the latest version as the hash and therefore the whole filename changes.

The first time a user requests an asset, it will look like this:

GET 123abc.cloudfront.net/application-ql4h2308y.css

A CloudFront cache miss “pulls from the origin” by making another GET request:

GET your-app-production.herokuapp.com/application-ql4h2308y.css

All future GET and HEAD requests to the CloudFront URL within the cache duration will be cached, with no second HTTP request to the origin:

GET 123abc.cloudfront.net/application-ql4h2308y.css

All HTTP requests using verbs other than GET and HEAD proxy through to the origin, which follows the Write-Through Mandatory portion of the HTTP specification.

Making it Work with Rails

We have standard configuration in our Rails apps that make this work:

# Gemfile
gem "coffee-rails"
gem "sass-rails"
gem "uglifier"

group :staging, :production do
  gem "rails_12factor"
end

# config/environments/{staging,production}.rb:
config.action_controller.asset_host = ENV["ASSET_HOST"] # will look like //123abc.cloudfront.net
config.assets.compile = false
config.assets.digest = true
config.assets.js_compressor = :uglifier
config.assets.version = ENV["ASSETS_VERSION"]
config.static_cache_control = "public, max-age=#{1.year.to_i}"

We don’t have to manually set config.serve_static_assets = true because the rails_12factor gem does it for us, in addition to handling any other current or future Heroku-related settings.

Fastly and other reverse proxy caches respect the Surrogate-Control standard. To get entire HTML pages cached in Fastly, we only need to include the Surrogate-Control header in the response. Fastly will cache the page for the duration we specify, protecting the origin from unnecessary requests and serving the HTML from Fastly’s edge servers.

Caching Entire HTML Pages (Why Use Memcache?)

While setting the asset host is a great start, a DNS to CDN to Origin architecture also lets us cache entire HTML pages. Here’s an example of caching entire HTML pages in Rails with High Voltage:

class PagesController < HighVoltage::PagesController
  before_filter :set_cache_headers

  private

  def set_cache_headers
    response.headers["Surrogate-Control"] = "max-age=#{1.day.to_i}"
  end
end

This will allow us to cache entire HTML pages in the CDN without using a Memcache add-on, which still goes through the Heroku router, then our app’s web processes, then Memcache. This architecture entirely protects the Rails app from HTTP requests that don’t require Ruby logic specific to our domain.

Rack Middleware

If we want to cache entire HTML pages site-wide, we might want to use Rack middleware. Here’s our typical config.ru for a Middleman app:

$:.unshift File.dirname(__FILE__)

require "rack/contrib/try_static"
require "lib/rack_surrogate_control"

ONE_WEEK = 604_800
FIVE_MINUTES = 300

use Rack::Deflater
use Rack::SurrogateControl
use Rack::TryStatic,
  root: "tmp",
  urls: %w[/],
  try: %w[.html index.html /index.html],
  header_rules: [
    [
      %w(css js png jpg woff),
      { "Cache-Control" => "public, max-age=#{ONE_WEEK}" }
    ],
    [
      %w(html), { "Cache-Control" => "public, max-age=#{FIVE_MINUTES}" }
    ]
  ]

run lambda { |env|
  [
    404,
    {
      "Content-Type"  => "text/html",
      "Cache-Control" => "public, max-age=#{FIVE_MINUTES}"
    },
    File.open("tmp/404.html", File::RDONLY)
  ]
}

We build the Middleman app at rake assets:precompile time during deploy to Heroku, as described in Styling a Middleman Blog with Bourbon, Neat, and Bitters. In production, we serve the app using Rack, so we are able to insert middleware to handle the Surrogate-Control header:

module Rack
  class SurrogateControl
    # Cache content in a reverse proxy cache (such as Fastly) for a year.
    # Use Surrogate-Control in response header so cache can be busted after
    # each deploy.
    ONE_YEAR = 31557600

    def initialize(app)
      @app = app
    end

    def call(env)
      status, headers, body = @app.call(env)
      headers["Surrogate-Control"] = "max-age=#{ONE_YEAR}"
      [status, headers, body]
    end
  end
end

CloudFront Setup

If we want to use CloudFront, we use the following settings:

  • “Download” CloudFront distribution
  • “Origin Domain Name” as www.thoughtbot.com (our app’s URL)
  • “Origin Protocol Policy” to “Match Viewer”
  • “Object Caching” to “Use Origin Cache Headers”
  • “Forward Query Strings” to “No (Improves Caching)”
  • “Distribution State” to “Enabled”

As a side benefit, in combination with CloudFront logging, we could replay HTTP requests on the Rails app if we had downtime at the origin for any reason, such as a Heroku platform issue.

Fastly Setup

If we use Fastly instead of CloudFront, there’s no “Origin Pull” configuration we need to do. It will work “out of the box” with our Rails configuration settings.

We often have a rake task in our Ruby apps fronted by Fastly like this:

# Rakefile
task :purge do
  api_key = ENV["FASTLY_KEY"]
  site_key = ENV["FASTLY_SITE_KEY"]
  `curl -X POST -H 'Fastly-Key: #{api_key}' https://api.fastly.com/service/#{site_key}/purge_all`
  puts 'Cache purged'
end

That turns our deployment process into:

git push production
heroku run rake purge --remote production

For more advanced caching and cache invalidation at an object level, see the fastly-rails gem.

Back to the Future

Fastly is really “Varnish as a Service”. Early in its history, Heroku used to include Varnish as a standard part of its “Bamboo” stack. When they decoupled the reverse proxy in their “Cedar” stack, we gained the flexibility of using different reverse proxy caches and CDNs fronting Heroku.

Love is Real

We have been using this stack in production for thoughtbot.com, robots.thoughtbot.com, playbook.thoughtbot.com, and many other apps for almost a year. It’s a stack in real use and is strong enough to consider as a good default architecture.

Give it a try on your next app!

Avoid AngularJS Dependency Annotation with Rails

Posted about 1 month back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

In AngularJS, it is a common practice to annotate injected dependencies for controllers, services, directives, etc.

For example:

angular.module('exampleApp', [])
  .controller('ItemsCtrl', ['$scope', '$http', function ($scope, $http) {
    $http.get('items/index.json').success(function(data) {
      $scope.items = data;
    });
  }]);

Notice how the annotation of the injected parameters causes duplication ($scope and $http appears twice). It becomes the responsibility of the developer to ensure that the actual parameters and the annotation are always in sync. If they are not, problems will occur that can cause lost time while head-scratching. As the list of parameters grows, it gets even harder to maintain.

The reason for needing to annotate the injected dependencies is documented in the AngularJS docs under “Dependency Annotation”:

To allow the minifiers to rename the function parameters and still be able to inject right services, the function needs to be annotated…

So, JavaScript minifiers will rename function parameters to something short and ambiguous (usually using just one letter). In that case, AngularJS does not know what service to inject since it tries to match dependencies to the parameter names.

Variable mangling and Rails

When using AngularJS with Rails, developers typically rely on the asset pipeline to handle the minification of JavaScript. Namely, the uglifier gem is the default, and most commonly used. With uglifier we are given an option to disable the mangling of variable names. JavaScript will still be minified – whitespace stripped out and code nicely compacted – but the variable and parameter names will remain the same.

To do this, in your Rails project disable the mangle setting for uglifier in the production (and staging) environment config, like so:

# config/environments/production.rb
ExampleApp::Application.configure do
  ...
  config.assets.js_compressor = Uglifier.new(mangle: false)
  ...
end

With that in place, you can write your AngularJS code without needing to annotate the injected dependencies. The previous code can now be written as:

angular.module('exampleApp', [])
  .controller('ItemsCtrl', function ($scope, $http) {
    $http.get('items/index.json').success(function(data) {
      $scope.items = data;
    });
  });

With this trick, there is no duplication of parameter names and no strange array notation.

The catch

Here are a couple of screenshots that show the difference in HTTP responses between mangling and non-mangling variable names, on a project of about 500 lines of production AngularJS code:

With full uglifier minification (variables are mangled): full-minification-screenshot

With variable mangling disabled (no variable mangling): mangling-disabled-screenshot

Disabling variable name mangling comes at the cost of about 200KB more. The size difference between the two settings can be more significant on larger projects with a lot more JavaScript code. The dilemma is to decide whether the convenience gained during development outweighs the size cost. Keep in mind that HTTP compression of web requests can help reduce the size difference. Benchmarking and comparison is advised on a per project basis.

What’s next?

If you found this useful, you might also enjoy:

KB Ratings

Posted about 1 month back at entp hoth blog - Home

Howdy!

What is this new section that just appeared in the sidebar for KB articles?

Screenshot of the sidebar showing a 'Is this article helpful, thumbs up/down' section

Well, starting today, users can now rate your KB articles! If you are logged in as a regular user, you will see the rating widget, and if you are logged in as staff, you will see the actual rating:

Same section showing the actual rating

Click through and you will be able to see all ratings and comments for the article, as well as the version they are associated with, so that you can keep track of your progress when improving articles:

Or head over to Knowledge Base > Ratings to see all ratings for all articles.

I hope you enjoy the change, and let us know if you have any feedback ;)

Cheers!

Efficient JSON in Swift with Functional Concepts and Generics

Posted about 1 month back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

A few months ago Apple introduced a new programming language, Swift, that left us excited about the future of iOS and OS X development. People were jumping into Swift with Xcode Beta1 immediately and it didn’t take long to realize that parsing JSON, something almost every app does, was not going to be as easy as in Objective-C. Swift being a statically typed language meant we could no longer haphazardly throw objects into typed variables and have the compiler trust us that it was the actually the type we claimed it would be. Now, in Swift, the compiler is doing the checking, making sure we don’t accidentally cause runtime errors. This allows us to lean on the compiler to create bug free code, but means we have to do a bit more work to make it happy. In this post, I discuss a method of parsing JSON APIs that uses functional concepts and Generics to make readable and efficient code.

Request the User Model

The first thing we need is a way to parse the data we receive from a network request into JSON. In the past, we’ve used NSJSONSerialization.JSONObjectWithData(NSData, Int, &NSError) which gives us an optional JSON data type and a possible error if there were problems with the parsing. The JSON object data type in Objective-C is NSDictionary which can hold any object in its values. With Swift, we have a new dictionary type that requires us to specify the types held within. JSON objects now map to Dictionary<String, AnyObject>. AnyObject is used because a JSON value could be a String, Double, Bool, Array, Dictionary or null. When we try to use the JSON to populate a model we’ve created, we’ll have to test that each key we get from the JSON dictionary is of that model’s property type. As an example, let’s look at a user model:

struct User {
  let id: Int
  let name: String
  let email: String
}

Now let’s take a look at what a request and response for the current user might look like:

func getUser(request: NSURLRequest, callback: (User) -> ()) {
  let task = NSURLSession.sharedSession().dataTaskWithRequest(request) { data, urlResponse, error in
    var jsonErrorOptional: NSError?
    let jsonOptional: AnyObject! = NSJSONSerialization.JSONObjectWithData(data, options: NSJSONReadingOptions(0), error: &jsonErrorOptional)

    if let json = jsonOptional as? Dictionary<String, AnyObject> {
      if let id = json["id"] as AnyObject? as? Int { // Currently in beta 5 there is a bug that forces us to cast to AnyObject? first
        if let name = json["name"] as AnyObject? as? String {
          if let email = json["email"] as AnyObject? as? String {
            let user = User(id: id, name: name, email: email)
            callback(user)
          }
        }
      }
    }
  }
  task.resume()
}

After a lot of if-let statements, we finally have our User object. You can imagine that a model with more properties will just get uglier and uglier. Also, we are not handling any errors so if any of the steps don’t succeed, we have nothing. Finally, we would have to write this code for every model we want from the API, which would be a lot of code duplication.

Before we start to refactor, let’s define some typealias’s to simplify the JSON types.

typealias JSON = AnyObject
typealias JSONDictionary = Dictionary<String, JSON>
typealias JSONArray = Array<JSON>

Refactoring: Add Error Handling

First, we will refactor our function to handle errors by introducing the first functional programming concept, the Either<A, B> type. This will let us return the user object when everything runs smoothly or an error when it doesn’t. We can implement an Either<A, B> type in Swift like this:

enum Either<A, B> {
  case Left(A)
  case Right(B)
}

We can use Either<NSError, User>as the type we’ll pass to our callback so the caller can handle the successfully parsed User or the error.

func getUser(request: NSURLRequest, callback: (Either<NSError, User>) -> ()) {
  let task = NSURLSession.sharedSession().dataTaskWithRequest(request) { data, urlResponse, error in
    // if the response returned an error send it to the callback
    if let err = error {
      callback(.Left(err))
      return
    }

    var jsonErrorOptional: NSError?
    let jsonOptional: JSON! = NSJSONSerialization.JSONObjectWithData(data, options: NSJSONReadingOptions(0), error: &jsonErrorOptional)

    // if there was an error parsing the JSON send it back
    if let err = jsonErrorOptional {
      callback(.Left(err))
      return
    }

    if let json = jsonOptional as? JSONDictionary {
      if let id = json["id"] as AnyObject? as? Int {
        if let name = json["name"] as AnyObject? as? String {
          if let email = json["email"] as AnyObject? as? String {
            let user = User(id: id, name: name, email: email)
            callback(.Right(user))
            return
          }
        }
      }
    }

    // if we couldn't parse all the properties then send back an error
    callback(.Left(NSError()))
  }
  task.resume()
}

Now the function calling our getUser can switch on the Either and do something with the user or display the error.

getUser(request) { either in
  switch either {
  case let .Left(error):
    // display error message

  case let .Right(user):
    // do something with user
  }
}

We will simplify this a bit by assuming that the Left will always be an NSError. Instead let’s use a different type Result<A> which will either hold the value we are looking for or an error. It’s implementation might look like this:

enum Result<A> {
  case Error(NSError)
  case Value(A)
}

Replacing Either with Result will look like this:

func getUser(request: NSURLRequest, callback: (Result<User>) -> ()) {
  let task = NSURLSession.sharedSession().dataTaskWithRequest(request) { data, urlResponse, error in
    // if the response returned an error send it to the callback
    if let err = error {
      callback(.Error(err))
      return
    }

    var jsonErrorOptional: NSError?
    let jsonOptional: JSON! = NSJSONSerialization.JSONObjectWithData(data, options: NSJSONReadingOptions(0), error: &jsonErrorOptional)

    // if there was an error parsing the JSON send it back
    if let err = jsonErrorOptional {
      callback(.Error(err))
      return
    }

    if let json = jsonOptional as? JSONDictionary {
      if let id = json["id"] as AnyObject? as? Int {
        if let name = json["name"] as AnyObject? as? String {
          if let email = json["email"] as AnyObject? as? String {
            let user = User(id: id, name: name, email: email)
            callback(.Value(user))
            return
          }
        }
      }
    }

    // if we couldn't parse all the properties then send back an error
    callback(.Error(NSError()))
  }
  task.resume()
}
getUser(request) { result in
  switch result {
  case let .Error(error):
    // display error message

  case let .Value(user):
    // do something with user
  }
}

Not a big change but let’s keep going.

Refactoring: Eliminate Type Checking Tree

Next, we will get rid of the ugly JSON parsing by creating separate JSON parsers for each type. We only have a String, Int, and Dictionary in our object so we need three functions to parse those types.

func JSONString(object: JSON?) -> String? {
  return object as? String
}

func JSONInt(object: JSON?) -> Int? {
  return object as? Int
}

func JSONObject(object: JSON?) -> JSONDictionary? {
  return object as? JSONDictionary
}

Now the JSON parsing will look like this:

if let json = JSONObject(jsonOptional) {
  if let id = JSONInt(json["id"]) {
    if let name = JSONString(json["name"]) {
      if let email = JSONString(json["email"]) {
        let user = User(id: id, name: name, email: email)
      }
    }
  }
}

Using these functions we’ll still need a bunch of if-let syntax. The functional programming concepts Monads, Applicative Functors, and Currying will help to condense this parsing. First, let’s look at the Maybe Monad which is similar to Swift optionals. Monads have a bind operator which, when used with optionals, allows us to bind an optional with a function that takes a non-optional and returns an optional. If the first optional is .None then it returns .None, otherwise it unwraps the first optional and applies the function to it.

infix operator >>> { associativity left precedence 150 }

func >>><A, B>(a: A?, f: A -> B?) -> B? {
  if let x = a {
    return f(x)
  } else {
    return .None
  }
}

In other functional languages, >>= is used for bind; however, in Swift that operator is used for bitshifting so we will use >>> instead. Applying this to the JSON parsing we get:

if let json = jsonOptional >>> JSONObject {
  if let id = json["id"] >>> JSONInt {
    if let name = json["name"] >>> JSONString {
      if let email = json["email"] >>> JSONString {
        let user = User(id: id, name: name, email: email)
      }
    }
  }
}

Then we can remove the optional parameters from our parsers:

func JSONString(object: JSON) -> String? {
  return object as? String
}

func JSONInt(object: JSON) -> Int? {
  return object as? Int
}

func JSONObject(object: JSON) -> JSONDictionary? {
  return object as? JSONDictionary
}

Functors have an fmap operator for applying functions to values wrapped in some context. Applicative Functors also have an apply operator for applying wrapped functions to values wrapped in some context. The context here is an Optional which wraps our value. This means that we can combine multiple optional values with a function that takes multiple non-optional values. If all values are present, .Some, then we get a result wrapped in an optional. If any of the values are .None, we get .None. We can define these operators in Swift like this:

infix operator <^> { associativity left } // Functor's fmap (usually <$>)
infix operator <*> { associativity left } // Applicative's apply

func <^><A, B>(f: A -> B?, a: A?) -> B? {
  if let x = a {
    return f(x)
  } else {
    return .None
  }
}

func <*><A, B>(f: (A -> B)?, a: A?) -> B? {
  if let x = a {
    if let fx = f {
      return fx(x)
    }
  }
  return .None
}

Before we put it all together, we will need to manually curry our User’s init since Swift doesn’t support auto-currying. Currying means that if we give a function fewer parameters than it takes, it will return a function that takes the remaining parameters. Our User model will now look like this:

struct User {
  let id: Int
  let name: String
  let email: String

  static func create(id: Int)(name: String)(email: String) -> User {
    return User(id: id, name: name, email: email)
  }
}

Putting it all together, our JSON parsing now looks like this:

if let json = jsonOptional >>> JSONObject {
  let user = User.create <^>
              json["id"]    >>> JSONInt    <*>
              json["name"]  >>> JSONString <*>
              json["email"] >>> JSONString
}

If any of our parser’s return .None then user will be .None. This looks much better, but we’re not done yet.

Now, our getUser function looks like this:

func getUser(request: NSURLRequest, callback: (Result<User>) -> ()) {
  let task = NSURLSession.sharedSession().dataTaskWithRequest(request) { data, urlResponse, error in
    // if the response returned an error send it to the callback
    if let err = error {
      callback(.Error(err))
      return
    }

    var jsonErrorOptional: NSError?
    let jsonOptional: JSON! = NSJSONSerialization.JSONObjectWithData(data, options: NSJSONReadingOptions(0), error: &jsonErrorOptional)

    // if there was an error parsing the JSON send it back
    if let err = jsonErrorOptional {
      callback(.Error(err))
      return
    }

    if let json = jsonOptional >>> JSONObject {
      let user = User.create <^>
                  json["id"]    >>> JSONInt    <*>
                  json["name"]  >>> JSONString <*>
                  json["email"] >>> JSONString
      if let u = user {
        callback(.Value(u))
        return
      }
    }

    // if we couldn't parse all the properties then send back an error
    callback(.Error(NSError()))
  }
  task.resume()
}

Refactoring: Remove Multiple Returns with Bind

Notice that we’re calling callback four times in the previous function. If we were to forget one of the return statements, we could introduce a bug. We can eliminate this potential bug and clean up this function further by first breaking up this function into 3 distinct parts: parse the response, parse the data into JSON, and parse the JSON into our User object. Each of these steps takes one input and returns the next step’s input or an error. This sounds like a perfect case for using bind with our Result type.

The parseResponse function will need a Result with data and the status code of the response. The iOS API only gives us NSURLResponse and keeps the data separate, so we will make a small struct to help out here:

struct Response {
  let data: NSData
  let statusCode: Int = 500

  init(data: NSData, urlResponse: NSURLResponse) {
    self.data = data
    if let httpResponse = urlResponse as? NSHTTPURLResponse {
      statusCode = httpResponse.statusCode
    }
  }
}

Now we can pass our parseResponse function a Response and check the response for errors before handing back the data.

func parseResponse(response: Response) -> Result<NSData> {
  let successRange = 200..<300
  if !contains(successRange, response.statusCode) {
    return .Error(NSError()) // customize the error message to your liking
  }
  return .Value(response.data)
}

The next functions will require us to transform an optional to a Result type so let’s make one quick abstraction before we move on.

func resultFromOptional<A>(optional: A?, error: NSError) -> Result<A> {
  if let a = optional {
    return .Value(a)
  } else {
    return .Error(error)
  }
}

Next up is our data to JSON function:

func decodeJSON(data: NSData) -> Result<JSON> {
  let jsonOptional: JSON! = NSJSONSerialization.JSONObjectWithData(data, options: NSJSONReadingOptions(0), error: &jsonErrorOptional)
  return resultFromOptional(jsonOptional, NSError()) // use the error from NSJSONSerialization or a custom error message
}

Then, we add our JSON to model decoding on the model itself:

struct User {
  let id: Int
  let name: String
  let email: String

  static func create(id: Int)(name: String)(email: String) -> User {
    return User(id: id, name: name, email: email)
  }

  static func decode(json: JSON) -> Result<User> {
    let user = JSONObject(json) >>> { dict in
      User.create <^>
          dict["id"]    >>> JSONInt    <*>
          dict["name"]  >>> JSONString <*>
          dict["email"] >>> JSONString
    }
    return resultFromOptional(user, NSError()) // custom error message
  }
}

Before we combine it all, let’s extend bind, >>>, to also work with the Result type:

func >>><A, B>(a: Result<A>, f: A -> Result<B>) -> Result<B> {
  switch a {
  case let .Value(x):     return f(x)
  case let .Error(error): return .Error(error)
  }
}

And add a custom initializer to Result:

enum Result<A> {
  case Error(NSError)
  case Value(A)

  init(_ error: NSError?, _ value: A) {
    if let err = error {
      self = .Error(err)
    } else {
      self = .Value(value)
    }
  }
}

Now, we combine all these functions with the bind operator.

func getUser(request: NSURLRequest, callback: (Result<User>) -> ()) {
  let task = NSURLSession.sharedSession().dataTaskWithRequest(request) { data, urlResponse, error in
    let responseResult = Result(error, Response(data: data, urlResponse: urlResponse))
    let result = responseResult >>> parseResponse
                                >>> decodeJSON
                                >>> User.decode
    callback(result)
  }
  task.resume()
}

Wow, even writing this again, I’m excited with this result. You might think, “This is really cool. Can’t wait to use it!”, but we’re not done yet!

Refactoring: Type Agnostic using Generics

This is great but we still have to write this for every model we want to get. We can use Generics to make this completely abstracted.

We introduce a Decodable protocol and tell our function that the type we want back must conform to that protocol. The protocol looks like this:

protocol Decodable {
  class func decode(json: JSON) -> Result<Self>
}

Now make User conform:

struct User: Decodable {
  let id: Int
  let name: String
  let email: String

  static func create(id: Int)(name: String)(email: String) -> User {
    return User(id: id, name: name, email: email)
  }

  static func decode(json: JSON) -> Result<User> {
    let user = User.create <^>
                json["id"]    >>> JSONInt    <*>
                json["name"]  >>> JSONString <*>
                json["email"] >>> JSONString
    return resultFromOptional(user, NSError()) // custom error message
  }
}

Our final performRequest function now looks like this:

func performRequest<A: Decodable>(request: NSURLRequest, callback: (Result<A>) -> ()) {
  let task = NSURLSession.sharedSession().dataTaskWithRequest(request) { data, urlResponse, error in
    let responseResult = Result(error, Response(data: data, urlResponse: urlResponse))
    let result = responseResult >>> parseResponse
                                >>> decodeJSON
                                >>> A.decode
    callback(result)
  }
  task.resume()
}

Further Learning

If you are curious about functional programming or any of the concepts discussed in this post, check out Haskell and specifically this post from the Learn You a Haskell book. Also, check out Pat Brisbin’s post about options parsing using the Applicative.

Episode #486 - August 5th, 2014

Posted about 1 month back at Ruby5

We React to some RubyGems Legal Stuff, Cover Unicorns and GitHub, and Serialize Matz's thoughts on the GIL in this episode of Ruby5.

Listen to this episode on Ruby5

Sponsored by CodeShip.io

Codeship is a hosted Continuous Delivery Service that just works.

Set up Continuous Integration in a few steps and automatically deploy when all your tests have passed. Integrate with GitHub and BitBucket and deploy to cloud services like Heroku and AWS, or your own servers.

Visit http://codeship.io/ruby5 and sign up for free. Use discount code RUBY5 for a 20% discount on any plan for 3 months.

Also check out the Codeship Blog!

CodeShip.io

ReactJS and Rails

If you want to take a look at React, the new JavaScript library released by Facebook, and how to use it with Rails Richard Nystrom wrote up a great tutorial which walks through the basics of writing your first app.
ReactJS and Rails

Rubygems.org - The Legal Stuff

Nick Quaranto started a thread over on the rubygems.org Google Group, highlighting the recent Privacy Policies and Code of Conduct posted by NPM (that’s Node’s package manager). And, overall, just putting a call out for help. Can you help him?
Rubygems.org - The Legal Stuff

Coverband

Coverband is a gem released at the end of last year by Dan Mayer who works at Living Social. It helps you generate “Production Ruby Code Coverage", and it's basically a middleware which you put on your production server to discover code which isn't being run that you can delete.
Coverband

Unicorn and GitHub

In the Unicorn mailing list, that’s the Rack HTTP server, maintainer Eric Wong didn't like the suggestion that Unicorn development be moved to GitHub.
Unicorn and GitHub

Oat - Another way to do JSON Serialization

Ismael Celis dropped me a line today about his API serialization library for Ruby called Oat. Oat uses serializer classes that kind of use the best of all the techniques for defining how JSON is serialized.
Oat - Another way to do JSON Serialization

Matz on the Ruby GIL

Matz put together his plans on the future of the Ruby GIL last week in 140 characters or less. Matz’s plan is to add actors, add a warning when developers use Threads directly, and then… finally… Remove the interpreter lock!
Matz on the Ruby GIL

Sponsored by Top Ruby Jobs

Keplar Agency is looking for a Rails developer in Amsterdam. Optoro is looking for a Ruby developer in Washington DC. SocialChorus is looking for a Chief Software Architect in San Francisco, CA. Smashing Boxes is looking for a Rails developer in Durham, NC or remote.
Top Ruby Jobs

Thank You for Listening to Ruby5

Ruby5 is released Tuesday and Friday mornings. To stay informed about and active with this podcast, we encourage you to do one of the following:

Thank You for Listening to Ruby5

Let's Talk About Dials

Posted about 1 month back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

There are often arguments about dials in dashboard design. Dials, occasionally called gauges, are a way to represent a single data point in a range. Think of a speedometer in a car.

It’s easy to glance down and get the current speed on a car dashboard while driving and navigating traffic. Dials work well in this context: we only care about the current speed and the direction the speed is going. When we need to slow down, we press down on the brake and can see our needle moving downwards. The speedometer tells us if we’re slowing down fast enough, and if we’re not, we adjust our braking so the needle moves at the right pace.

When we’re monitoring our applications, our dashboards tracking our application metrics should work the same way. We should be able to know what’s wrong with our app at a glance and respond quickly. Because people often want to replace a familiar analog interface on the computer screen, dials in dashboards are popular.

However, dials don’t work the same way on your computer screen. Let’s look at Google Chart’s “gauge” visualization.

These gauges are mostly grey with some red and orange to represent critical and warning ranges. If our metric goes into these ranges, we should start reacting.

But our eyes are attracted to colors of greatest contrast: the red and orange blocks. Every time we look at a dashboard with these dials, we see the areas of bright colors, even when nothing is critical. This may not be a big deal with one gauge, but dashboards display many other metrics. Things get messy when we have several visuazations on a page.

At one glance, do these dials:

Look much different than these?:

The second set of dials show some values that are just barely in the critical and warning zones. What if you had 10 other graphs on your dashboard? Would you be able to notice a critical metric in one glance?

Alternatives

A popular solution is to eliminate the extra “ink” around the dial and fill the current value up to its appropriate pixel value.

We get a sleeker dial that doesn’t need labels and dashes. If the designer chooses, they can change the color of the dial to reflect the state of the value. For example, if the current value is critical, the dial turns red.

This display is an improvement the analog dial design for our web dashboards. However, if we need to display several single values, we still lose a lot of screen real estate to the circular shape. For this, we can use a bullet chart, a dial alternative designed by Stephen Few.

We can display three metrics for a lot less space, and we can see where those current values land in their ranges.

Source: Bullet Graph Design Spec (original image edited)

Few avoids common “status” colors like red, green, and yellow to reduce the number of contrasting, attention-grabbing colors on the page and to account for color blind users. When a value hits the critical range, he suggests an external indicator.

The downside is that this type of visualization requires a lot of ink to be effective. In my experience, they often require explanations for the people who are viewing them. If you want to display a current value with context, sometimes a simple line graph with some annotations does the trick.

While this metric isn’t in a critical range yet, we can see that it’s rapidly approaching that critical value and should probably take action to resolve that.

Dials are a great way to display analog information, but don’t work the same in your dashboard application. Depending on the situation, some of these examples make a better alternatives.

If you’re interested in learning more about maximizing the effectiveness of your dashboards, I highly recommend the writings of Stephen Few. I will also be teaching a workshop on D3.js for eurucamp in Berlin, Germany at the beginning of August.

Running WeeChat on a Server for IRC Backlogs

Posted about 1 month back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

In a previous blog post we walked you through setting up and configuring the text based chat client WeeChat for use with the Slack IRC gateway.

The blog post also explained how to set up Slacklog - a WeeChat script for retrieving backlog from the Slack API. This works great, but if you want more backlog or want backlog for channels on other IRC servers a good solution is to run WeeChat on a server. By doing this you won’t miss anything that happens in the channels you are connected to, even when you’re not at your computer or connected to the Internet. When you reconnect you can go through messages that were meant for you or that you’re interested in.

This tutorial assumes that you have a server running Debian 7.0 (Wheezy) that you can access over SSH using an account with sudo privileges. If you’re using a different setup you may have to alter the instructions. If you don’t have a server but would like to set one up, I recently wrote a blog post explaining how to set up a basic Debian server.

Setting up WeeChat

Connect to your server using ssh:

ssh YOURIP

To get the latest version of WeeChat you have to install it from the Wheezy backports. If you haven’t already, add the Wheezy backports to your /etc/apt/sources.list file:

sudo sh -c 'echo deb http://http.debian.net/debian wheezy-backports main >> /etc/apt/sources.list'

Update your package index and install WeeChat using apt-get:

sudo apt-get update
sudo apt-get -t wheezy-backports install weechat-curses

You can now start WeeChat using the weechat-curses command:

weechat-curses

To exit type /quit and press Enter.

If you disconnect from your server by typing Ctrl+D, you can now reconnect to the server and run weechat-curses by running this command locally:

ssh YOURIP -t weechat-curses

When you pass a command to ssh it will run the command on the remote machine instead of running your normal shell. When it does so, it will also skip the allocation of a pseudo-tty. This works fine for one-off commands with no output, but since WeeChat is an interactive program, it needs a tty to connect to. The -t flag in the above command forces the allocation of a pseudo-tty.

When you exit out of WeeChat, ssh will automatically disconnect from the server.

Great! We can now connect to and run WeeChat on our server in one command. One problem with this approach is that WeeChat won’t continue to run after you disconnect. This means that you will miss anything that happens in the channels you were connected to since you’re simply no longer connected. Always being connected means that you will have access to the full backlog.

To fix this we will have to run WeeChat using a terminal multiplexer (like GNU Screen or tmux) that has support for running programs in background sessions that can be detached from and reattached to.

Setting up GNU Screen

We will use GNU Screen in this tutorial mainly because a lot of us here at thoughtbot use tmux locally and nesting tmux sessions can cause problems. If you’re not using tmux locally, using tmux on your server is perfectly fine.

Reconnect to your server and install GNU Screen using apt-get:

ssh YOURIP
sudo apt-get install screen

If you disconnect from your server you can now reconnect and run weechat-curses in a screen session using this command:

ssh YOURIP -t screen -D -RR weechat weechat-curses

This command will log in to your server and look for a screen session named “weechat”. If a session with that name exists, it will reattach to it. If not it will create a new session and run the weechat-curses command inside of it.

To disconnect from the server without quitting WeeChat press Ctrl+A and then D.

Awesome! We can now enjoy the upside of always being online even when we’re not.

There is still one problem though. When you’re done for the day, close your laptop and head home, your SSH connection will eventually time out. When you get home and open up your laptop, the terminal running the ssh command will be frozen and the only way to quit it will be by pressing ~ and .. To continue chatting you’ll then have to reconnect. This will also happen if you suddenly lose your Internet connection for a few minutes and it can be really annoying.

Wouldn’t it be simpler if you were reconnected automatically as soon as you were reconnected to the Internet?

Setting up Mosh

Mosh stands for “mobile shell” and is a remote terminal application that allows roaming and intermittent connectivity. In our case mosh will replace ssh. Under the hood Mosh still uses ssh to log in to the server to then start a mosh-server that you can communicate with over UDP.

Mosh connections will not freeze after losing the Internet connection but will instead show a timer that tells how long ago it last got a response from the server. As soon as you come online it will reconnect and you can continue chatting.

Install Mosh locally using your favorite package manager.

On Debian:

sudo apt-get install mosh

On Arch Linux:

sudo pacman -S mosh

On Mac OS X (using Homebrew):

brew install mobile-shell

Installation instructions for other systems can be found on the Mosh website.

Reconnect to your server and install Mosh using apt-get:

ssh YOURIP
sudo apt-get install mosh

By default, mosh-server binds to a UDP port between 60000 and 61000. If you’re running a firewall, you have to open these ports to be able to connect. If you are using UFW to manage your firewall you can run this command to open the ports:

sudo ufw allow 60000:61000/udp

If you disconnect from your server, you can now reconnect to and attach to your WeeChat screen session using mosh:

mosh YOURIP -- screen -D -RR weechat weechat-curses

There we go! All set up.

Connecting to servers and joining channels

You can now connect to an IRC server using WeeChat:

/connect chat.freenode.net

Once connected you can join your favorite channel:

/join #ruby

For more information on how to use WeeChat check out WeeChat for Slack’s IRC Gateway.

Ship You a Haskell

Posted 2 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

A few weeks ago, we quietly shipped a new feature here on the Giant Robots blog: comments. If you hover over a paragraph or code block, a small icon should appear to the right, allowing you to comment on that section of any article.

comments

With the release of this feature, we can now say something we’ve hoped to say for some time: we shipped Haskell to production! In this post, I’ll outline what we shipped, how it’s working out for us, and provide some solutions to the various hurdles we encountered.

Architecture

Comments are handled by a service “installed” on the blog by including a small snippet of JavaScript on article pages. The separate service is called Carnival and can be found on GitHub. It’s comprised of a RESTful API for adding, retrieving, and editing comments and a JavaScript front-end to handle the UX. The back-end portion is written in Haskell using the Yesod web framework.

Why Haskell?

The answer to this question depends on who you ask. A number of us really like Haskell as a language and look for any excuse to use it. This may stem from safety, quality of abstraction, joy of development, or any number of other positives we feel the language brings. Some of us are recently exposed to Haskell and would love to have something being actively developed that we could pair on from time to time and get more exposure to a language so unlike what we’re used to.

Ultimately, we want to know if Haskell is something we can build and scale for client projects. If a client comes along where Haskell may be a good fit, we need to be confident that, beyond writing the code, we can do everything else that’s needed to deploy it to production.

Development Process

During the development of this service, much of what is said about the benefits of type safety when it comes to rapidly producing correct code proved true. The bulk of the API was written in about a day and subsequent iterations and refactorings went smoothly using a combination of Type Driven Development (TyDD) and acceptance tests. For programmers used to interpreted languages, the long compiles were frustrating, and we did have some small battles with Cabal Hell. That said, the introduction of sandboxes and freezing are a definite improvement over my own previous experiences with dependency management in Haskell.

Writing an API-only services meant working with a lot of JSON. Doing this via the aeson library was concise, and provided safe (de)serialization with very limited validation logic required on our part. A large number of validations that we would typically write in a Rails API service are handled by virtue of the type system.

Libraries exist for most of the things we need like markdown, gravatar, and heroku support. One notable exception was authentication via OAuth 2.0, which we needed because we wanted to use our own Upcase as the provider. While Yesod has great support for authentication in general and there exists a plugin for OAuth 1.0, the only thing we could find for OAuth 2.0 was an out of date gist. Luckily, it wasn’t much trouble to move that gist to a proper package, ensure it worked, and publish it ourselves. Even though Yesod didn’t ship with this feature out of the box, the modular way in which authentication logic is handled allowed us to add it as a separate, isolated package.

Deployment

Part of this experiment was to develop in Haskell using as much of our normal process as possible. That meant deploying to Heroku. Because a clean compilation of a Haskell application (especially with libraries like Yesod or Pandoc) can take some time, the 15 minute build limit became an issue.

Before you mention it, yes this pain point could’ve been avoided with a binary deployment strategy. We could have compiled locally in a VM (to match Heroku’s architecture) then copied the resulting binary to the Heroku instance. But that’s not our normal process. Developers should be able to git push heroku master and have it Just Work.

And in theory, it could just work. Builds are largely cached so it’s only the first one that’s likely to go beyond 15 minutes. To mitigate this, the most popular Haskell buildpack supports a service called Anvil for running that first build in an environment with no time limit. After many attempts and vague error messages, we had to give up on these Anvil-based deployments. We were on our own.

In the end, we were never able to come in under 15 minutes, even after upgrading to a PX dyno. Our Heroku representative was able to increase our app’s time limit to 30 minutes and so far we’ve been able to make that. I wouldn’t consider this typical though: I suspect our dependency on pandoc (and its highlighting engine) is causing compilation to take longer than most Yesod applications. I recommend trying the standard build pack and hoping to come in under 15 minutes before attempting to subvert it.

Once successfully on staging, we noticed another issue. Users were getting logged out randomly. It turns out the default session backend in Yesod stores the key to a cookie-based session in a file. This has a number of downsides in a Heroku deployment: First of all, the file system is ephemeral. Any time a dyno restarts, all sessions would be invalidated. Secondly, we had two dynos running. This meant that if you logged in on one dyno, but a subsequent request got routed to the second, you’d be logged out. To support this scenario, we defined an alternative backend which read the key from an environment variable which we could set to the same value in each instance.

More Haskell?

We definitely consider this experiment a success. We solved a number of deployment problems which should make our next Haskell project (which is already in the works) go that much more smoothly. All in all, we found the language well-suited to solving the kinds of problems we solve, thanks in no small part to Yesod and the great ecosystem of available libraries.

Episode #485 - August 1st, 2014

Posted 2 months back at Ruby5

Learning to deploy with capistrano, memoization patterns, better APIs with mocaroni, middleman-presentation, and RubyConf 2014 all in this episode of the Ruby5!

Listen to this episode on Ruby5

Sponsored by New Relic

New Relic is _the_ all-in-one web performance analytics product. It lets you manage and monitor web application performance, from the browser down to the line of code. With Real User Monitoring, New Relic users can see browser response times by geographical location of the user, or by browser type.
This episode is sponsored by New Relic

Capistrano Tutorial

Need to get started with capistrano? This step-by-step tutorial will demystify the process and get you off on the right foot.
Capistrano Tutorial

Memoization Patterns

Curious about all the wonderful ways to memoize? This blog post from Justin Weiss will walk you through some patterns and a gem!
Memoization Patterns

Mocaroni

Mocaroni is a new service that lets you stub out and collaborate on an api!
Mocaroni

middleman-presentation

Middleman is a awesome for building static websites and now middleman-presentation lets you easily make HTML-based presentations, too!
middleman-presentation

RubyConf 2014

RubyConf is headed to San Diego this year and tickets are now on sale!
RubyConf 2014

WeeChat for Slack's IRC Gateway

Posted 2 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

WeeChat is a text based chat client that runs on many different platforms like Linux, Mac OS X, various BSD variants and Windows (through cygwin). WeeChat is mainly used for IRC, aims to be fast, light and extensible and is a more modern alternative to IRSSI.

After switching from Campfire to Slack for the thoughtbot company chat, more of us started using WeeChat together with the Slack IRC gateway.

WeeChat in action

Installing WeeChat

Binary packages for the latest version of WeeChat are available for a bunch of different distributions like Debian (Backports) and Arch Linux. Installation is as simple as using your favorite package manager.

WeeChat is also available through Homebrew as weechat. When installing using Homebrew be sure to pass these options to enable script support for Lua, Perl, Python and Ruby scripts:

brew install weechat --with-lua --with-perl --with-python --with-ruby

For instructions specific to your distribution check out the WeeChat user’s guide.

Connecting to Slack

The first thing you need to do after getting WeeChat up and running is to connect to Slack.

Add and connect to the Slack IRC gateway by issuing these commands inside of WeeChat:

/server add NAME HOST/6667 -autoconnect -ssl -ssl_dhkey_size=512 -password=PASSWORD -username=USERNAME -nicks=NICK
/connect

Where:

  • NAME is the name you want WeeChat to use when referring to this server.
  • HOST is the Host as provided on the Gateways page.
  • PASSWORD is the Pass as provided on the Gateways page.
  • USERNAME is the User as provided on the Gateways page.
  • NICK is your Slack username.

This command turns on the autoconnect feature, SSL encryption and sets the so called “Diffie-Hellman” key size to 512. This is required since the WeeChat default value (2048) will cause errors during the SSL handshake with the Slack IRC gateway.

When you connect, the Slack IRC gateway will automatically join all channels that you already belong to.

Joining channels

If you want to join an existing channel or want to create a new one, use the /join command:

/join #yourchannel

Switching between buffers

In WeeChat, channels, private messages and connections to servers are displayed in what’s called buffers. By default WeeChat only displays one buffer at a time.

You can cycle between all open buffers by pressing Alt+Up/Down.

You can also switch between buffers using the /buffer command:

/buffer #general

Partial buffer names can be tab-completed.

Buffers are also numbered. You can jump between buffers using these numbers by pressing Alt+NUMBER or Esc and then NUMBER. To get a better overview of open buffers and their numbers, check out the buffers.pl plugin.

Buffers that have unseen activity are listed in the so called “hotlist”. To jump to the next buffer on the hotlist, press Alt+A. The hotlist is ordered by importance.

Leaving channels

To leave a channel use the /part command. If you also want to close the current buffer use the /close command instead.

Sending direct messages

To send a direct message to another user, use the /query command:

/query calleerlandson Hello!

This will open a new buffer with your conversation.

The core buffer

The first buffer (buffer number 1) is the WeeChat core buffer. This buffer is used to interact with WeeChat and all servers you are connected to. When you issue commands like /help or /whois the output shows up in this buffer.

When you issue commands in the core buffer they get sent to its current target. The current target is either WeeChat itself or one of the servers you are connected to. You can see the current target on the line just above the buffer input field. To cycle between targets press Ctrl+X.

When issuing commands to WeeChat itself you don’t have to switch to the weechat target.

Configuring WeeChat

WeeChat is configured by issuing commands like /set, /unset, /server or /window. To get information about a command use the /help command like this:

/help COMMAND

Where COMMAND is the command you want information about.

If you issue the /help command without passing a command it will show you a list of all available commands.

WeeChat’s configuration is stored in files inside the ~/.weechat/ directory. To save the current configuration to disk use the /save command. To write the current layout configuration to disk use the /layout save command.

The current configuration is automatically saved to disk by WeeChat when you quit or certain other WeeChat commands. This prevents you from loosing your configuration but can also be a pain if you try to edit the files manually while WeeChat is running.

Configuring highlights

By default all messages containing your nickname will be highlighted. You might want other messages to be highlighted as well. For example, if you would like all messages that mention the word “fika” to be highlighted you can configure that by typing:

/set weechat.look.highlight fika

Tell WeeChat to highlight messages containing other words by separating them with commas:

/set weechat.look.highlight fika,pr,cats

Filtering out voice mode change messages

The Slack IRC gateway uses the IRC voice feature to indicate user activity. A user is automatically voiced when active and devoiced when inactive. Users that are using the Slack web interface are constantly voiced and devoiced and this creates a lot of noise in WeeChat buffers. To filter out these messages you can add this filter:

/filter add hidevoices irc.NAME.* irc_mode (\+|\-)v

Where NAME is the name of your server in WeeChat.

Add more nickname colors

By default WeeChat only uses a handful of colors to color the nicknames in buffers. In channels with a lot of people this can become confusing. This command lets WeeChat use a couple more colors:

/set weechat.color.chat_nick_colors red,green,brown,blue,magenta,cyan,white,lightred,lightgreen,yellow,lightblue,lightmagenta,lightcyan

To review what colors you are using you can use the /color command.

If you are running WeeChat in GNU Screen it is important that the TERM environment variable is set to screen-256color in order for WeeChat to be able to display all the above colors. You can force this by adding this line to your .screenrc:

term screen-256color

Scripts

WeeChat is extensible through scripts written in an array of different languages. Scripts are listed on the Script section of the WeeChat website.

You can also browse listed scripts inside WeeChat using the /script command without any arguments:

/script

This will open an interactive buffer where you can browse, install, remove and load scripts.

Below I walk you through installing some scripts that I find usable in general and some with Slack in particular.

Buffers

Buffers provides a sidebar where all open buffers are listed with their number. This gives me a much better overview of my buffers. The sidebar also shows which buffers has unread messages and highlights.

Install Buffers using the /script command:

/script install buffers.pl

By default Buffers will color the buffer names depending on activity in them. If you also want to know how many messages and highlights the buffers have you can tell Buffers to display a hotlist counter:

 /set buffers.look.hotlist_counter on

Interactive set

Navigating through all the different WeeChat configuration options can be hard. Interactive set lets you set configuration options interactively.

Install Interactive set using the /script command:

/script install iset.pl

When you use the /iset command a buffer with all options is opened. Navigate the list by using your arrow keys or PgUp or PgDn.

Highmon

Highmon is a highlight monitor. It creates a buffer containing all highlighted messages in other buffers. This script is great for quickly reviewing all my mentions to see if something important has come up.

Install Highmon using the /script command:

/script install highmon.pl

I keep a small Highmon buffer at the top of my WeeChat window at all times. This way, I just have to glance at the WeeChat window to see I have any new important highlights.

To do this, split your window into two horizontal splits; One small at the top for Highmon, and one large below for your chat buffers:

/window splith 20
/buffer highmon
/window 1

Save this layout to disk to persist it during restarts:

/layout save

Slacklog

Slacklog (written by fellow thoughtbotter Pat Brisbin) pulls chat history from the Slack API and prints it into the buffer as backlog whenever you join a new channel.

Since Slacklog is not yet published on the WeeChat website you have to install it from your shell:

curl https://raw.githubusercontent.com/thoughtbot/weechat-slacklog/master/slacklog.rb -o ~/.weechat/ruby/slacklog.rb

Load it in WeeChat:

/script load slacklog.rb

Create an API token on the Slack API website and configure Slacklog with your Slack server and API token:

/set plugins.var.ruby.slacklog.servers NAME
/set plugins.var.ruby.slacklog.NAME.api_token TOKEN

Where:

  • NAME is the name you gave your Slack IRC server.
  • TOKEN is the token as specified on the Slack API website.

By default, Slacklog pulls in the 100 latest messages. If you want to change this number you can set it to a value between 1 and 1000 using this option:

/set plugins.var.ruby.slacklog.count 500

Colorize Lines

Colorize Lines lets me color messages in the same color as the nicknames of the users that posted them. This helps me distinguish between messages more easily.

Install Colorize Lines using the /script command:

/script install colorize_lines.pl

Messages posted from now on will be colorized.

Auto loading scripts

Scripts can be auto loaded by WeeChat on startup. To tell WeeChat to auto load a script use the /script command:

/script autoload SCRIPT

Meditations on a Class Method

Posted 2 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

I keep a file of code I like. When looking for inspiration, I read through the file. It’s short; I often re-write the samples to be nothing but the minimally inspiring thought.

Here is the first snippet:

def self.run(user)
  new(user).run
end

Think on it for yourself before I explain what it means to me. I’d like to hear what it means to you — leave a long comment here before you keep reading.

To me, it’s a reminder of how to write a beautiful class method: instantiate the class then call a method on the instance.

Look at it from the perspective of the person calling the class method. When you call a class method you want one of two things: either you want to construct an instance of the class itself (.new, or perhaps .new_from_file, .new_for_widescreen, and .new_from_json), or you want convenience.

Think of the class methods you’ve seen, or have written. If they are not in the above style, they might look more like this:

class ImageUploader
  def self.run(xpm)
    @@dimensions = geometry_for(xpm)
    @@color_palette = colors_for(xpm)
    svg = generate_svg(xpm)
  end

  def self.geometry_for(xpm)
    # ...
  end

  def self.colors_for(xpm)
    # ...
  end

  def self.generate_svg(xpm)
    # ...
  end
end

What a mess of an object. An abuse of the singleton pattern, where it wasn’t even intended. Class variables being used in an especially not-thread-safe way, plus a jumble of code that is all exposed. It is daunting to extend because it is a tricky thought process to understand the full implications of even using it.

When dealing with an object you want a small interface. As a user you want the fewest number of options, and the one with the best abstraction; as a developer you want to hide as much of the implementation as possible, giving you full freedom to change the internals. The best object is one with no exposed methods at all.

The above pattern gives you that. You call .run and pass a user, and it takes care of the rest. If the default constructor changes its arity, the instance method (#run) changes its name, or the object is re-written in C and needs to do pointer arithmetic first: you are protected.

The snippet has explicit names for things: run and user. This brings to mind the command pattern, and especially a command pattern for dealing with users. Perhaps something to kick off the backend signup process. The command pattern is a quick way to start reducing a god class (PDF); pushing various bits of User into the SignUp class will help simplify both.

The simplicity of the snippet is a reminder to use abstractions on the same “level”. Create an instance and call a method on that; perhaps in the instance’s #run method, it will instantiate a few more objects and call a method on those; and so on. Short methods all the way down, explained with clear but concise names.

This snippet happens to be in Ruby, an inspiration unto itself. A part of the power behind the command pattern is in Ruby’s duck typing. Let’s say this is a class method on SignUp. I know that I can pass SignUp itself to something that expects to call run, passing a user. In doing so, I know that I can fake it in a test with any object that responds to run:

class FakeSignup
  def initialize(should_succeed = true)
    @should_succeed = should_succeed
  end

  def run(user)
    unless @should_succeed
      raise "I am supposed to fail"
    end
  end
end

The idea of passing SignUp around makes me think of queues: you can add the SignUp class to a background job runner to get an asynchronous workflow from within Rails. You could spawn a co-routine, passing SignUp and a user object. Once you’ve been inspired by the snippet, a world of concurrency opens up.

So that’s what I think of when I see my first snippet from my collection of inspirational code. What do you see?

Segment.io and Ruby

Posted 2 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

Segment.io is a tool that lets developers identify users and track their activity through an application. Armed with this information, developers can understand usage patterns through Segment.io’s integrations with Mixpanel, Klaviyo, Google Analytics, and many others.

Segment.io provides a Ruby gem, AnalyticsRuby, to make it easy to track custom events in Rails applications. In a project I worked on recently, we chose to encapsulate interaction with AnalyticsRuby within one class. Rather than always passing the user’s id, email, name, city, and state to AnalyticsRuby and conditionally sending the identifier from Google Analytics, our Analytics facade manages it for us.

# app/models/analytics.rb
class Analytics
  class_attribute :backend
  self.backend = AnalyticsRuby

  def initialize(user, client_id = nil)
    @user = user
    @client_id = client_id
  end

  def track_user_creation
    identify
    track(
      {
        user_id: user.id,
        event: 'Create User',
        properties: {
          city_state: user.zip.to_region
        }
      }
    )
  end

  def track_user_sign_in
    identify
    track(
      {
        user_id: user.id,
        event: 'Sign In User'
      }
    )
  end

  private

  def identify
    backend.identify(identify_params)
  end

  attr_reader :user, :client_id

  def identify_params
    {
      user_id: user.id,
      traits: user_traits
    }
  end

  def user_traits
    {
      email: user.email,
      first_name: user.first_name,
      last_name: user.last_name,
      city_state: user.city_state,
    }.reject { |key, value| value.blank? }
  end

  def track(options)
    if client_id.present?
      options.merge!(
        context: {
          'Google Analytics' => {
            clientId: client_id
          }
        }
      )
    end
    backend.track(options)
  end
end

Because the application had very little JavaScript and no client-side events we needed to track, we decided to use the Ruby library. Depending on the goals and types of events, oftentimes using both Ruby and JavaScript Segment.io libraries makes sense.

To use Analytics, we first defined a handful of methods on ApplicationController:

class ApplicationController < ActionController::Base
  include Clearance::Controller

  def current_user
    super || Guest.new
  end

  def analytics
    @analytics ||= Analytics.new(current_user, google_analytics_client_id)
  end

  def google_analytics_client_id
    google_analytics_cookie.gsub(/^GA\d\.\d\./, '')
  end

  def google_analytics_cookie
    cookies['_ga'] || ''
  end
end

With the analytics method available on ApplicationController, we were free to fire any events we wanted:

class UsersController < Clearance::UsersController
  private

  def url_after_create
    analytics.track_user_creation
    super
  end
end

class SessionsController < Clearance::SessionsController
  private

  def url_after_create
    analytics.track_user_sign_in
    super
  end
end

Tracking events for non-GET requests is much easier when using the Ruby library - imagine conditionally rendering JavaScript within your application layout using Rails' flash!

With a test fake, acceptance testing analytics events becomes a breeze:

# spec/features/user_signs_in_spec.rb
require 'spec_helper'

feature 'User signs in' do
  scenario 'successfully' do
    user = create :user, password: 'password'

    visit root_path
    click_on 'Sign in'
    fill_in 'Email', with: user.email
    fill_in 'Password', with: 'password'
    click_button 'Sign in'

    expect(analytics).to have_tracked('Sign In User').for_user(user)
    expect(analytics).to have_identified(user)
  end
end

This spec should look pretty familiar to those who’ve written features with RSpec and Capybara before. After exercising sign in functionality, we verify the analytics backend is tracking events correctly by ensuring we’ve tracked the appropriate event for the right person, as well as properly identified that person.

# spec/support/analytics.rb
RSpec.configure do |config|
  config.around :each do |example|
    cached_backend = Analytics.backend
    example.run
    Analytics.backend = cached_backend
  end
end

module Features
  def analytics
    Analytics.backend
  end
end

# spec/spec_helper.rb
RSpec.configure do |config|
  config.before :each, type: :feature do
    Analytics.backend = FakeAnalyticsRuby.new
  end
end

Here, we configure the analytics and ensure it’s using a fake, FakeAnalyticsRuby, which we define:

# lib/fake_analytics_ruby.rb
class FakeAnalyticsRuby
  def initialize
    @identified_users = []
    @tracked_events = EventsList.new([])
  end

  def identify(user)
    @identified_users << user
  end

  def track(options)
    @tracked_events << options
  end

  delegate :tracked_events_for, to: :tracked_events

  def has_identified?(user, traits = { email: user.email })
    @identified_users.any? do |user_hash|
      user_hash[:user_id] == user.id &&
        traits.all? do |key, value|
          user_hash[:traits][key] == value
        end
    end
  end

  private

  attr_reader :tracked_events

  class EventsList
    def initialize(events)
      @events = events
    end

    def <<(event)
      @events << event
    end

    def tracked_events_for(user)
      self.class.new(
        events.select do |event|
          event[:user_id] == user.id
        end
      )
    end

    def named(event_name)
      self.class.new(
        events.select do |event|
          event[:event] == event_name
        end
      )
    end

    def has_properties?(options)
      events.any? do |event|
        (options.to_a - event[:properties].to_a).empty?
      end
    end

    private
    attr_reader :events
  end
end

FakeAnalyticsRuby implements the part of the same interface as AnalyticsRuby (from the Ruby gem), namely #identify and #track, and maintains internal state for our RSpec matchers have_tracked and have_identified.

With the chainable with_properties, we can ensure additional information, like the user’s city and state, are passed along when firing the “Create User” event:

# spec/features/guest_signs_up_spec.rb
require 'spec_helper'

feature 'Guest signs up' do
  scenario 'successfully' do
    complete_registration city: 'Boston', state: 'MA'

    expect(analytics).to have_tracked('Create User').
      for_user(User.last).
      with_properties({
        city_state: 'Boston, MA'
      })
  end
end

To test Analytics at a unit level, we stub and spy:

# spec/models/analytics_spec.rb

describe Analytics do
  describe '#track_user_sign_in' do
    it 'notifies AnalyticsRuby of a user signing in' do
      AnalyticsRuby.stub(:track)
      user = build_stubbed(:user)

      analytics = Analytics.new(user)
      analytics.track_user_sign_in

      expect(AnalyticsRuby).to have_received(:track).with(
        {
          user_id: user.id,
          event: 'Sign In User'
        }
      )
    end
  end
end

In this app, we tracked seven different events and tested each appropriately. This solution worked well for a number of reasons:

  • with an injectible fake, we were able to easily test events were triggered with the correct properties
  • with a facade to simplify interaction with AnalyticsRuby, we were able to layer additional functionality like tracking with Google Analytics with relative ease
  • with two custom RSpec matchers, we were able to add expressive assertions against fired events
  • with no JavaScript event requirements, we were able to use Ruby and avoid patterns like using flash to conditionally render JavaScript triggering Segment.io events

Segment.io is a solid platform upon which we can develop robust event tracking, and with a bit of work, integrating this tracking into a Rails app is a great way to gain insight into your customers.

Episode #484 - July 29th, 2014

Posted 2 months back at Ruby5

In this episode we cover the new Rails 4.2 HTML sanitizer, speeding up tests with ActiveMocker, logging validation errors with validation_auditor, Understanding Timeouts in CRuby, parsing JSON API with Roar and RubyConf Portugal.

Listen to this episode on Ruby5

Sponsored by CodeShip.io

Codeship is a hosted Continuous Delivery Service that just works.

Set up Continuous Integration in a few steps and automatically deploy when all your tests have passed. Integrate with GitHub and BitBucket and deploy to cloud services like Heroku and AWS, or your own servers.

Visit http://codeship.io/ruby5 and sign up for free. Use discount code RUBY5 for a 20% discount on any plan for 3 months.

Also check out the Codeship Blog!

CodeShip.io

Rails 4.2 HTML sanitizer

Ruby Hero Rafael França posted on the PlataformaTec blog about the new HTML sanitizer coming up in Rails 4.2. He worked with Kasper Timm Hansen during last year's Google Summer of Code on refactoring the sanitize helper method to give Rails developers a more robust, faster and secure solution to sanitizing user input.
Rails 4.2 HTML sanitizer

Active Mocker

ActiveMocker is a gem by Dustin Zeisler which creates mocks from ActiveRecord models. It allows your test suite to run very fast by not loading Rails or hooking to a database.
Active Mocker

validation_auditor

Pablo Fernández wrote to us about his gem validation_auditor, which logs validation errors to the database for later inspection. Very useful for tracking bugs down.
validation_auditor

Understanding Timeouts in C Ruby

Reginald Tan wrote an article about understanding Timeouts in CRuby. It's a fairly complex process, but there’s some cool graphs on the post that make it easier to understand.
Understanding Timeouts in C Ruby

Roar

Nick Sutterer updated the Roar gem adding support for the JSON API format. This means you can start using Roar to format JSON API data for clients that expect that format - for example, EmberJS apps using the ActiveModelAdapter.
Roar

Sponsored by Top Ruby Jobs

Routehappy is looking for a Ruby Developer in New York City. thredUP is looking for a Lead Software Engineer in San Franciso, CA. GameGenetics is looking for a Ruby on Rails Front End Developer and a Ruby on Rails Lead Developer in Berlin, Germany.
Top Ruby Jobs

RubyConf Portugal

RubyConf Portugal is taking place on October 13th & 14th in the beautiful city of Braga. They are already on the second batch of tickets, so head over to http://rubyconf.pt/ to get yours!
RubyConf Portugal

Thank You for Listening to Ruby5

Ruby5 is released Tuesday and Friday mornings. To stay informed about and active with this podcast, we encourage you to do one of the following:

Thank You for Listening to Ruby5