WeeChat for Slack's IRC Gateway

Posted 28 days back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

WeeChat is a text based chat client that runs on many different platforms like Linux, Mac OS X, various BSD variants and Windows (through cygwin). WeeChat is mainly used for IRC, aims to be fast, light and extensible and is a more modern alternative to IRSSI.

After switching from Campfire to Slack for the thoughtbot company chat, more of us started using WeeChat together with the Slack IRC gateway.

WeeChat in action

Installing WeeChat

Binary packages for the latest version of WeeChat are available for a bunch of different distributions like Debian (Backports) and Arch Linux. Installation is as simple as using your favorite package manager.

WeeChat is also available through Homebrew as weechat. When installing using Homebrew be sure to pass these options to enable script support for Lua, Perl, Python and Ruby scripts:

brew install weechat --with-lua --with-perl --with-python --with-ruby

For instructions specific to your distribution check out the WeeChat user’s guide.

Connecting to Slack

The first thing you need to do after getting WeeChat up and running is to connect to Slack.

Add and connect to the Slack IRC gateway by issuing these commands inside of WeeChat:

/server add NAME HOST/6667 -autoconnect -ssl -ssl_dhkey_size=512 -password=PASSWORD -username=USERNAME -nicks=NICK
/connect

Where:

  • NAME is the name you want WeeChat to use when referring to this server.
  • HOST is the Host as provided on the Gateways page.
  • PASSWORD is the Pass as provided on the Gateways page.
  • USERNAME is the User as provided on the Gateways page.
  • NICK is your Slack username.

This command turns on the autoconnect feature, SSL encryption and sets the so called “Diffie-Hellman” key size to 512. This is required since the WeeChat default value (2048) will cause errors during the SSL handshake with the Slack IRC gateway.

When you connect, the Slack IRC gateway will automatically join all channels that you already belong to.

Joining channels

If you want to join an existing channel or want to create a new one, use the /join command:

/join #yourchannel

Switching between buffers

In WeeChat, channels, private messages and connections to servers are displayed in what’s called buffers. By default WeeChat only displays one buffer at a time.

You can cycle between all open buffers by pressing Alt+Up/Down.

You can also switch between buffers using the /buffer command:

/buffer #general

Partial buffer names can be tab-completed.

Buffers are also numbered. You can jump between buffers using these numbers by pressing Alt+NUMBER or Esc and then NUMBER. To get a better overview of open buffers and their numbers, check out the buffers.pl plugin.

Buffers that have unseen activity are listed in the so called “hotlist”. To jump to the next buffer on the hotlist, press Alt+A. The hotlist is ordered by importance.

Leaving channels

To leave a channel use the /part command. If you also want to close the current buffer use the /close command instead.

Sending direct messages

To send a direct message to another user, use the /query command:

/query calleerlandson Hello!

This will open a new buffer with your conversation.

The core buffer

The first buffer (buffer number 1) is the WeeChat core buffer. This buffer is used to interact with WeeChat and all servers you are connected to. When you issue commands like /help or /whois the output shows up in this buffer.

When you issue commands in the core buffer they get sent to its current target. The current target is either WeeChat itself or one of the servers you are connected to. You can see the current target on the line just above the buffer input field. To cycle between targets press Ctrl+X.

When issuing commands to WeeChat itself you don’t have to switch to the weechat target.

Configuring WeeChat

WeeChat is configured by issuing commands like /set, /unset, /server or /window. To get information about a command use the /help command like this:

/help COMMAND

Where COMMAND is the command you want information about.

If you issue the /help command without passing a command it will show you a list of all available commands.

WeeChat’s configuration is stored in files inside the ~/.weechat/ directory. To save the current configuration to disk use the /save command. To write the current layout configuration to disk use the /layout save command.

The current configuration is automatically saved to disk by WeeChat when you quit or certain other WeeChat commands. This prevents you from loosing your configuration but can also be a pain if you try to edit the files manually while WeeChat is running.

Configuring highlights

By default all messages containing your nickname will be highlighted. You might want other messages to be highlighted as well. For example, if you would like all messages that mention the word “fika” to be highlighted you can configure that by typing:

/set weechat.look.highlight fika

Tell WeeChat to highlight messages containing other words by separating them with commas:

/set weechat.look.highlight fika,pr,cats

Filtering out voice mode change messages

The Slack IRC gateway uses the IRC voice feature to indicate user activity. A user is automatically voiced when active and devoiced when inactive. Users that are using the Slack web interface are constantly voiced and devoiced and this creates a lot of noise in WeeChat buffers. To filter out these messages you can add this filter:

/filter add hidevoices irc.NAME.* irc_mode (\+|\-)v

Where NAME is the name of your server in WeeChat.

Add more nickname colors

By default WeeChat only uses a handful of colors to color the nicknames in buffers. In channels with a lot of people this can become confusing. This command lets WeeChat use a couple more colors:

/set weechat.color.chat_nick_colors red,green,brown,blue,magenta,cyan,white,lightred,lightgreen,yellow,lightblue,lightmagenta,lightcyan

To review what colors you are using you can use the /color command.

If you are running WeeChat in GNU Screen it is important that the TERM environment variable is set to screen-256color in order for WeeChat to be able to display all the above colors. You can force this by adding this line to your .screenrc:

term screen-256color

Scripts

WeeChat is extensible through scripts written in an array of different languages. Scripts are listed on the Script section of the WeeChat website.

You can also browse listed scripts inside WeeChat using the /script command without any arguments:

/script

This will open an interactive buffer where you can browse, install, remove and load scripts.

Below I walk you through installing some scripts that I find usable in general and some with Slack in particular.

Buffers

Buffers provides a sidebar where all open buffers are listed with their number. This gives me a much better overview of my buffers. The sidebar also shows which buffers has unread messages and highlights.

Install Buffers using the /script command:

/script install buffers.pl

By default Buffers will color the buffer names depending on activity in them. If you also want to know how many messages and highlights the buffers have you can tell Buffers to display a hotlist counter:

 /set buffers.look.hotlist_counter on

Interactive set

Navigating through all the different WeeChat configuration options can be hard. Interactive set lets you set configuration options interactively.

Install Interactive set using the /script command:

/script install iset.pl

When you use the /iset command a buffer with all options is opened. Navigate the list by using your arrow keys or PgUp or PgDn.

Highmon

Highmon is a highlight monitor. It creates a buffer containing all highlighted messages in other buffers. This script is great for quickly reviewing all my mentions to see if something important has come up.

Install Highmon using the /script command:

/script install highmon.pl

I keep a small Highmon buffer at the top of my WeeChat window at all times. This way, I just have to glance at the WeeChat window to see I have any new important highlights.

To do this, split your window into two horizontal splits; One small at the top for Highmon, and one large below for your chat buffers:

/window splith 20
/buffer highmon
/window 1

Save this layout to disk to persist it during restarts:

/layout save

Slacklog

Slacklog (written by fellow thoughtbotter Pat Brisbin) pulls chat history from the Slack API and prints it into the buffer as backlog whenever you join a new channel.

Since Slacklog is not yet published on the WeeChat website you have to install it from your shell:

curl https://raw.githubusercontent.com/thoughtbot/weechat-slacklog/master/slacklog.rb -o ~/.weechat/ruby/slacklog.rb

Load it in WeeChat:

/script load slacklog.rb

Create an API token on the Slack API website and configure Slacklog with your Slack server and API token:

/set plugins.var.ruby.slacklog.servers NAME
/set plugins.var.ruby.slacklog.NAME.api_token TOKEN

Where:

  • NAME is the name you gave your Slack IRC server.
  • TOKEN is the token as specified on the Slack API website.

By default, Slacklog pulls in the 100 latest messages. If you want to change this number you can set it to a value between 1 and 1000 using this option:

/set plugins.var.ruby.slacklog.count 500

Colorize Lines

Colorize Lines lets me color messages in the same color as the nicknames of the users that posted them. This helps me distinguish between messages more easily.

Install Colorize Lines using the /script command:

/script install colorize_lines.pl

Messages posted from now on will be colorized.

Auto loading scripts

Scripts can be auto loaded by WeeChat on startup. To tell WeeChat to auto load a script use the /script command:

/script autoload SCRIPT

Meditations on a Class Method

Posted 29 days back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

I keep a file of code I like. When looking for inspiration, I read through the file. It’s short; I often re-write the samples to be nothing but the minimally inspiring thought.

Here is the first snippet:

def self.run(user)
  new(user).run
end

Think on it for yourself before I explain what it means to me. I’d like to hear what it means to you — leave a long comment here before you keep reading.

To me, it’s a reminder of how to write a beautiful class method: instantiate the class then call a method on the instance.

Look at it from the perspective of the person calling the class method. When you call a class method you want one of two things: either you want to construct an instance of the class itself (.new, or perhaps .new_from_file, .new_for_widescreen, and .new_from_json), or you want convenience.

Think of the class methods you’ve seen, or have written. If they are not in the above style, they might look more like this:

class ImageUploader
  def self.run(xpm)
    @@dimensions = geometry_for(xpm)
    @@color_palette = colors_for(xpm)
    svg = generate_svg(xpm)
  end

  def self.geometry_for(xpm)
    # ...
  end

  def self.colors_for(xpm)
    # ...
  end

  def self.generate_svg(xpm)
    # ...
  end
end

What a mess of an object. An abuse of the singleton pattern, where it wasn’t even intended. Class variables being used in an especially not-thread-safe way, plus a jumble of code that is all exposed. It is daunting to extend because it is a tricky thought process to understand the full implications of even using it.

When dealing with an object you want a small interface. As a user you want the fewest number of options, and the one with the best abstraction; as a developer you want to hide as much of the implementation as possible, giving you full freedom to change the internals. The best object is one with no exposed methods at all.

The above pattern gives you that. You call .run and pass a user, and it takes care of the rest. If the default constructor changes its arity, the instance method (#run) changes its name, or the object is re-written in C and needs to do pointer arithmetic first: you are protected.

The snippet has explicit names for things: run and user. This brings to mind the command pattern, and especially a command pattern for dealing with users. Perhaps something to kick off the backend signup process. The command pattern is a quick way to start reducing a god class (PDF); pushing various bits of User into the SignUp class will help simplify both.

The simplicity of the snippet is a reminder to use abstractions on the same “level”. Create an instance and call a method on that; perhaps in the instance’s #run method, it will instantiate a few more objects and call a method on those; and so on. Short methods all the way down, explained with clear but concise names.

This snippet happens to be in Ruby, an inspiration unto itself. A part of the power behind the command pattern is in Ruby’s duck typing. Let’s say this is a class method on SignUp. I know that I can pass SignUp itself to something that expects to call run, passing a user. In doing so, I know that I can fake it in a test with any object that responds to run:

class FakeSignup
  def initialize(should_succeed = true)
    @should_succeed = should_succeed
  end

  def run(user)
    unless @should_succeed
      raise "I am supposed to fail"
    end
  end
end

The idea of passing SignUp around makes me think of queues: you can add the SignUp class to a background job runner to get an asynchronous workflow from within Rails. You could spawn a co-routine, passing SignUp and a user object. Once you’ve been inspired by the snippet, a world of concurrency opens up.

So that’s what I think of when I see my first snippet from my collection of inspirational code. What do you see?

Segment.io and Ruby

Posted 30 days back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

Segment.io is a tool that lets developers identify users and track their activity through an application. Armed with this information, developers can understand usage patterns through Segment.io’s integrations with Mixpanel, Klaviyo, Google Analytics, and many others.

Segment.io provides a Ruby gem, AnalyticsRuby, to make it easy to track custom events in Rails applications. In a project I worked on recently, we chose to encapsulate interaction with AnalyticsRuby within one class. Rather than always passing the user’s id, email, name, city, and state to AnalyticsRuby and conditionally sending the identifier from Google Analytics, our Analytics facade manages it for us.

# app/models/analytics.rb
class Analytics
  class_attribute :backend
  self.backend = AnalyticsRuby

  def initialize(user, client_id = nil)
    @user = user
    @client_id = client_id
  end

  def track_user_creation
    identify
    track(
      {
        user_id: user.id,
        event: 'Create User',
        properties: {
          city_state: user.zip.to_region
        }
      }
    )
  end

  def track_user_sign_in
    identify
    track(
      {
        user_id: user.id,
        event: 'Sign In User'
      }
    )
  end

  private

  def identify
    backend.identify(identify_params)
  end

  attr_reader :user, :client_id

  def identify_params
    {
      user_id: user.id,
      traits: user_traits
    }
  end

  def user_traits
    {
      email: user.email,
      first_name: user.first_name,
      last_name: user.last_name,
      city_state: user.city_state,
    }.reject { |key, value| value.blank? }
  end

  def track(options)
    if client_id.present?
      options.merge!(
        context: {
          'Google Analytics' => {
            clientId: client_id
          }
        }
      )
    end
    backend.track(options)
  end
end

Because the application had very little JavaScript and no client-side events we needed to track, we decided to use the Ruby library. Depending on the goals and types of events, oftentimes using both Ruby and JavaScript Segment.io libraries makes sense.

To use Analytics, we first defined a handful of methods on ApplicationController:

class ApplicationController < ActionController::Base
  include Clearance::Controller

  def current_user
    super || Guest.new
  end

  def analytics
    @analytics ||= Analytics.new(current_user, google_analytics_client_id)
  end

  def google_analytics_client_id
    google_analytics_cookie.gsub(/^GA\d\.\d\./, '')
  end

  def google_analytics_cookie
    cookies['_ga'] || ''
  end
end

With the analytics method available on ApplicationController, we were free to fire any events we wanted:

class UsersController < Clearance::UsersController
  private

  def url_after_create
    analytics.track_user_creation
    super
  end
end

class SessionsController < Clearance::SessionsController
  private

  def url_after_create
    analytics.track_user_sign_in
    super
  end
end

Tracking events for non-GET requests is much easier when using the Ruby library - imagine conditionally rendering JavaScript within your application layout using Rails' flash!

With a test fake, acceptance testing analytics events becomes a breeze:

# spec/features/user_signs_in_spec.rb
require 'spec_helper'

feature 'User signs in' do
  scenario 'successfully' do
    user = create :user, password: 'password'

    visit root_path
    click_on 'Sign in'
    fill_in 'Email', with: user.email
    fill_in 'Password', with: 'password'
    click_button 'Sign in'

    expect(analytics).to have_tracked('Sign In User').for_user(user)
    expect(analytics).to have_identified(user)
  end
end

This spec should look pretty familiar to those who’ve written features with RSpec and Capybara before. After exercising sign in functionality, we verify the analytics backend is tracking events correctly by ensuring we’ve tracked the appropriate event for the right person, as well as properly identified that person.

# spec/support/analytics.rb
RSpec.configure do |config|
  config.around :each do |example|
    cached_backend = Analytics.backend
    example.run
    Analytics.backend = cached_backend
  end
end

module Features
  def analytics
    Analytics.backend
  end
end

# spec/spec_helper.rb
RSpec.configure do |config|
  config.before :each, type: :feature do
    Analytics.backend = FakeAnalyticsRuby.new
  end
end

Here, we configure the analytics and ensure it’s using a fake, FakeAnalyticsRuby, which we define:

# lib/fake_analytics_ruby.rb
class FakeAnalyticsRuby
  def initialize
    @identified_users = []
    @tracked_events = EventsList.new([])
  end

  def identify(user)
    @identified_users << user
  end

  def track(options)
    @tracked_events << options
  end

  delegate :tracked_events_for, to: :tracked_events

  def has_identified?(user, traits = { email: user.email })
    @identified_users.any? do |user_hash|
      user_hash[:user_id] == user.id &&
        traits.all? do |key, value|
          user_hash[:traits][key] == value
        end
    end
  end

  private

  attr_reader :tracked_events

  class EventsList
    def initialize(events)
      @events = events
    end

    def <<(event)
      @events << event
    end

    def tracked_events_for(user)
      self.class.new(
        events.select do |event|
          event[:user_id] == user.id
        end
      )
    end

    def named(event_name)
      self.class.new(
        events.select do |event|
          event[:event] == event_name
        end
      )
    end

    def has_properties?(options)
      events.any? do |event|
        (options.to_a - event[:properties].to_a).empty?
      end
    end

    private
    attr_reader :events
  end
end

FakeAnalyticsRuby implements the part of the same interface as AnalyticsRuby (from the Ruby gem), namely #identify and #track, and maintains internal state for our RSpec matchers have_tracked and have_identified.

With the chainable with_properties, we can ensure additional information, like the user’s city and state, are passed along when firing the “Create User” event:

# spec/features/guest_signs_up_spec.rb
require 'spec_helper'

feature 'Guest signs up' do
  scenario 'successfully' do
    complete_registration city: 'Boston', state: 'MA'

    expect(analytics).to have_tracked('Create User').
      for_user(User.last).
      with_properties({
        city_state: 'Boston, MA'
      })
  end
end

To test Analytics at a unit level, we stub and spy:

# spec/models/analytics_spec.rb

describe Analytics do
  describe '#track_user_sign_in' do
    it 'notifies AnalyticsRuby of a user signing in' do
      AnalyticsRuby.stub(:track)
      user = build_stubbed(:user)

      analytics = Analytics.new(user)
      analytics.track_user_sign_in

      expect(AnalyticsRuby).to have_received(:track).with(
        {
          user_id: user.id,
          event: 'Sign In User'
        }
      )
    end
  end
end

In this app, we tracked seven different events and tested each appropriately. This solution worked well for a number of reasons:

  • with an injectible fake, we were able to easily test events were triggered with the correct properties
  • with a facade to simplify interaction with AnalyticsRuby, we were able to layer additional functionality like tracking with Google Analytics with relative ease
  • with two custom RSpec matchers, we were able to add expressive assertions against fired events
  • with no JavaScript event requirements, we were able to use Ruby and avoid patterns like using flash to conditionally render JavaScript triggering Segment.io events

Segment.io is a solid platform upon which we can develop robust event tracking, and with a bit of work, integrating this tracking into a Rails app is a great way to gain insight into your customers.

Episode #484 - July 29th, 2014

Posted 30 days back at Ruby5

In this episode we cover the new Rails 4.2 HTML sanitizer, speeding up tests with ActiveMocker, logging validation errors with validation_auditor, Understanding Timeouts in CRuby, parsing JSON API with Roar and RubyConf Portugal.

Listen to this episode on Ruby5

Sponsored by CodeShip.io

Codeship is a hosted Continuous Delivery Service that just works.

Set up Continuous Integration in a few steps and automatically deploy when all your tests have passed. Integrate with GitHub and BitBucket and deploy to cloud services like Heroku and AWS, or your own servers.

Visit http://codeship.io/ruby5 and sign up for free. Use discount code RUBY5 for a 20% discount on any plan for 3 months.

Also check out the Codeship Blog!

CodeShip.io

Rails 4.2 HTML sanitizer

Ruby Hero Rafael França posted on the PlataformaTec blog about the new HTML sanitizer coming up in Rails 4.2. He worked with Kasper Timm Hansen during last year's Google Summer of Code on refactoring the sanitize helper method to give Rails developers a more robust, faster and secure solution to sanitizing user input.
Rails 4.2 HTML sanitizer

Active Mocker

ActiveMocker is a gem by Dustin Zeisler which creates mocks from ActiveRecord models. It allows your test suite to run very fast by not loading Rails or hooking to a database.
Active Mocker

validation_auditor

Pablo Fernández wrote to us about his gem validation_auditor, which logs validation errors to the database for later inspection. Very useful for tracking bugs down.
validation_auditor

Understanding Timeouts in C Ruby

Reginald Tan wrote an article about understanding Timeouts in CRuby. It's a fairly complex process, but there’s some cool graphs on the post that make it easier to understand.
Understanding Timeouts in C Ruby

Roar

Nick Sutterer updated the Roar gem adding support for the JSON API format. This means you can start using Roar to format JSON API data for clients that expect that format - for example, EmberJS apps using the ActiveModelAdapter.
Roar

Sponsored by Top Ruby Jobs

Routehappy is looking for a Ruby Developer in New York City. thredUP is looking for a Lead Software Engineer in San Franciso, CA. GameGenetics is looking for a Ruby on Rails Front End Developer and a Ruby on Rails Lead Developer in Berlin, Germany.
Top Ruby Jobs

RubyConf Portugal

RubyConf Portugal is taking place on October 13th & 14th in the beautiful city of Braga. They are already on the second batch of tickets, so head over to http://rubyconf.pt/ to get yours!
RubyConf Portugal

Thank You for Listening to Ruby5

Ruby5 is released Tuesday and Friday mornings. To stay informed about and active with this podcast, we encourage you to do one of the following:

Thank You for Listening to Ruby5

Lovebirds

Posted about 1 month back at Mike Clark

Lovebirds

Two little lovebirds, just sittin' in a tree...

For You

Posted about 1 month back at Mike Clark

For You

I love the complementary colors of male and female bluebirds in the spring. He flew in to feed her a tasty grub.

Learn is now Upcase

Posted about 1 month back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

Big news: we’ve changed the name of our subscription service for building better developers. We’ve previously referred to ourselves as both Learn and Prime, but now we’re just Upcase.

You can find us at our new home: upcase.com.

What’s more, in the last few months we’ve added peer-reviewed coding exercises that have been a big hit. You can complete the exercises using your local development environment, submit it with a simple git push and get your solution reviewed by real humans (including thoughtbotters).

Here’s a look at the UI:

Upcase exercise UI

If you’ve been waiting to subscribe (or have subcribed in the past), now’s a great time to take another step toward getting hired full-time, launching your own app, or improving your skills despite being isolated.

If you’re interested in adopting the thoughtbot way of building Rails apps (including vim mastery, pragmatic TDD, and object-oriented design), check us out!

Episode #483 - July 25nd, 2014

Posted about 1 month back at Ruby5

Rails Rumble, Debug Anything and Speeding up Rails

Listen to this episode on Ruby5

Sponsored by NewRelic

You should be using New Relic by now but do you regularly checkout their blog?
NewRelic

Rails Rumble

Rails Rumble 2014 will take place on the weekend of October 18th & 19th
Rails Rumble

How to Debug Anything

James Golick gave a talk at GoRuCo on how to debug anything
How to Debug Anything

3 Ways to Create Classes in Ruby

THUVA THARMA wrote a classy blog post on the many different ways to create a class
3 Ways to Create Classes in Ruby

Clone and Dup

Aaron Lasseigne just wrote a blog post Clone and Dup
Clone and Dup

Speedup Your Code by Aaron Patterson

Aaron Patterson from RedDotRuby 2014 on how he's been Speeding up rails
Speedup Your Code by Aaron Patterson

Closing

Thank you for listening to Ruby5. Be sure to tune in every Tuesday and Friday for the latest news in the Ruby and Rails community.
Closing

Why Postgres Won't Always Use an Index

Posted about 1 month back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

We were analyzing slow responses on a project recently and found ourselves questioning the performance of some PostgreSQL queries. As is typical when digging into queries, we had some questions around what the query was doing. So, we used our handy EXPLAIN command to shed some light on the database’s behavior. Upon inspecting the query it turned out an index we had created was not being used. We wanted to know why.

EXPLAIN explained

Let’s first look at how the EXPLAIN command works.

The Postgres docs have a helpful article for learning about EXPLAIN. The basics are that in your psql console prepend the word EXPLAIN in front of your query and you’ll get a query plan.

In SQL

Run psql <name of your database>, or rails dbconsole if you are inside of a Rails project. Prepend EXPLAIN to your query to see an explanation of how Postgres is going to break down your problem in the most efficient way possible.

EXPLAIN SELECT * FROM posts;
                        QUERY PLAN
----------------------------------------------------------
 Seq Scan on posts  (cost=0.00..53.84 rows=281 width=704)
(1 row)

In Rails

If you don’t feel like dropping down to the database you can get explain statements from ActiveRecord::Relations by calling the method explain. Calling the following command will give you the same result you’ll see from the Postgres console:

Post.all.explain

How to interpret the output

Now we have the query plan. What is it telling us? Query plans are read bottom to top and display the following information for each step of the query:

  • estimated statement execution cost: an arbitrary unit of measuring cost
  • estimated number of rows: the number of rows the query will produce
  • estimated width of rows: the size of the rows in bytes

For a more thorough explanation of how to analyze these query plans' costs and number of rows check out the Postgres docs on using explain.

What’s important to our question about indexes is the type of query being performed and whether an index is being used or not. If we see a step containing the words Index Scan we’re using the index.

A side note about query costs

As noted in the Postgres docs, the query plans are based on a sample of statistics about the tables in the query. These statistics are kept up to date by Postgres when your database is VACUUMed. As a result, even when the underlying data is identical, you may see slightly different values for the same query run at different times.

An Example

It doesn’t take much digging to find interesting results in the query plans. Let’s use a database with a posts table containing a couple hundred post records. id is the primary key and by default has an index. Let’s see what the query looks like to get the first 10 posts.

EXPLAIN SELECT * FROM POSTS WHERE id < 10;

Our plan is:

 Bitmap Heap Scan on posts  (cost=2.07..16.39 rows=9 width=704)
   Recheck Cond: (id < 10)
   ->  Bitmap Index Scan on posts_pkey  (cost=0.00..2.07 rows=9 width=0)
         Index Cond: (id < 10)
(4 rows)

This query is using the index because we see Index Scan in the plan. The meaning of the Recheck Cond: statement is beyond the scope of this article, but the postgres-performance mailing list has a helpful explanation.

Now, let’s see what it looks like to get the rest of the posts in the table.

EXPLAIN SELECT * FROM POSTS WHERE id > 10;

Our plan is:

 Seq Scan on posts  (cost=0.00..53.98 rows=272 width=704)
   Filter: (id > 10)
(2 rows)

This query is going to perform a sequential scan (Seq Scan) and not use the index. A sequential scan? This plan means that Postgres is going to read the whole table to find what we’re looking for. That approach seems inefficient given the index we already have. What’s going on here? To answer this question it’s helpful to think about what an index is. An index is a specialized representation in memory of the contents of a particular column (or multiple columns).

How indexes are used

As we saw above, running a couple of queries on our posts table reveals that even given an index to use, Postgres will not always choose to use it. The reason why this is the case is that indexes have a cost to create and maintain (on writes) and use (on reads).

When an index is used in a SELECT query, first the position of the requested rows is fetched from the index (instead of from the table directly). Then, the requested rows (a subset of the total) are actually fetched from the table. It’s a two step process. Given a large enough number of requested rows (such as all the posts with an id greater than 10), it is actually more efficient to go fetch all those rows from the table directly rather than to go through the extra step of asking the index for the position of those rows and next fetch those specific rows from the table. If the percentage of the rows is smaller than 5-10% of all rows in the table the benefit of using the information stored in the index outweighs the additional intermediate step.

What’s Next:

Hear more about Postgres indexes, caching, and performance from Harold Giménez, head of Heroku Postgres, on A Beautiful Thing, an episode of the Giant Robots Smashing Into Other Giants podcast. Or, watch the Improving Rails Performance screencast from Joe Ferris, our CTO.

Phusion Passenger 4.0.48 released

Posted about 1 month back at Phusion Corporate Blog


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Phusion Passenger is under constant maintenance and development. Version 4.0.48 is a bugfix release.

Phusion Passenger also has an Enterprise version which comes with a wide array of additional features. By buying Phusion Passenger Enterprise you will directly sponsor the development of the open source version.

Recent changes

4.0.47 was a hotfix release for an Enterprise customer. The changes in 4.0.47 and 4.0.48 combined are as follows.

  • Fixed a race condition while determining what user an application should be executed as. This bug could lead to applications being run as the wrong user. Closes GH-1241.
  • [Standalone] Improved autodetection of Rails asset pipeline files. This prevents Standalone from incorrectly setting caching headers on non-asset pipeline files. Closes GH-1225.
  • Fixed compilation problems on CentOS 5. Thanks to J. Smith. Closes GH-1247.
  • Fixed compilation problems on OpenBSD.
  • Fixed compatibility with Ruby 1.8.5.

Installing or upgrading to 4.0.48

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Fork us on Github!Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

<iframe src="http://ghbtns.com/github-btn.html?user=phusion&amp;repo=passenger&amp;type=watch&amp;size=large&amp;count=true" allowtransparency="true" frameborder="0" scrolling="0" width="170" height="30"></iframe><iframe src="http://ghbtns.com/github-btn.html?user=phusion&amp;repo=passenger&amp;type=fork&amp;size=large&amp;count=true" allowtransparency="true" frameborder="0" scrolling="0" width="170" height="30"></iframe><iframe src="http://ghbtns.com/github-btn.html?user=phusion&amp;type=follow&amp;size=large&amp;count=true" allowtransparency="true" frameborder="0" scrolling="0" width="190" height="30"></iframe>

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



An Explained psqlrc

Posted about 1 month back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

Let’s walk through my short psqlrc(5) to see what I’ve set, and to inspire you to find your own configuration that fits into your workflow. Here is my complete psqlrc:

\set ON_ERROR_ROLLBACK interactive
\set COMP_KEYWORD_CASE upper
\set HISTFILE ~/.psql/history- :DBNAME

\pset pager off
\pset null '(null)'

PostgreSQL’s shell, psql(1), can be configured using \set and \pset. \pset is for changing the output format — HTML, pager, field separator, and so on — while \set is for everything else.

ON_ERROR_ROLLBACK

The ON_ERROR_ROLLBACK settings affects how psql(1) handles errors. The default value is off.

When this setting is on, errors are effectively ignored at all times. So if you have this script, slint.sql:

BEGIN;
CREATE TABLE members (id SERIAL, name TEXT);
INSERT INTO member (name) VALUES ('David Pajo');
INSERT INTO members (name) VALUES ('Brian McMahan');
COMMIT;

And run it from the command line:

psql -f slint.sql

You would end up with a members table with Brian McMahan but without David Pajo.

When it is set to off, the default, then you get nothing: no members table and no Brian McMahan. It either all works or it doesn’t, just like a transaction should.

There is a third value: interactive. Under interactive, the above command in which statements are piped into psql(1) non-interactively is treated like off, but if you type them into the interactive prompt it is treated like on. This gives you a chance to fix things without starting over:

$ psql
bands=# BEGIN;
BEGIN
bands=# CREATE TABLE members (id SERIAL, name TEXT);
CREATE TABLE
bands=# INSERT INTO member (name) VALUES ('David Pajo');
ERROR:  relation "member" does not exist
LINE 1: INSERT INTO member (name) VALUES ('David Pajo');
bands=# INSERT INTO members (name) VALUES ('David Pajo');
INSERT 0 1
bands=# INSERT INTO members (name) VALUES ('Brian McMahan');
INSERT 0 1
bands=# COMMIT;
COMMIT

COMP_KEYWORD_CASE

Some people format their SQL with uppercase keywords; others go downcase. Some mix and match depending on their mood. psql(1) handles that!

Possibly the greatest feature of any shell is tab completion, and psql(1) doesn’t disappoint. However, there’s a question of which case it should use to complete keywords.

The straight-forward thing to do is to set it to lower or upper.

sel tab completes to SELECT

But even fancier are preserve-lower and preserve-upper, with preserve-upper as the default. These preserve whatever case you were using, falling back to lower (or upper). For example:

preserve the case but default to upper

There, up was completed to update and S was completed to SET, preserving the case as the user typed it; n was completed to name, preserving the case of the object in the database; and the space after order was completed to BY, favoring uppercase when the user has typed nothing.

HISTFILE

Like any good shell, psql(1) will save the commands you have entered so you can run them again (it’s full Readline; try a ^R some time). By default it stores the history in ~/.psql_history, but we can do better than that.

To start, let’s introduce another psql(1) command: \echo

bands=# \echo hello
hello
bands=# \echo :DBNAME 
bands

The variable :DBNAME is automatically set to the name of the database and available to all psql(1) commands. There are other pre-set variables like :PORT, :HOST, :USER, :ENCODING, and so on, but we’re going to use :DBNAME to start.

It just so happens that psql(1) will concatenate strings for you, so if you want different history for each database (the queries against the desserts table won’t make sense in the zoology database, for example), you can set that up:

\set HISTFILE ~/.psql_history- :DBNAME

You can combine these as much as you please, such as:

\set HISTFILE ~/.psql_history- :USER - :HOST - :PORT - :DBNAME

pager

The pager is the program that paginates text. The classic is more(1), and the improvement is less(1). Puns.

The default value for the pager setting is on which — unlike the name suggests — only uses the pager sometimes. A few lines are shown without a pager, but 25 or more lines invoke pagination. (Specifically, if the text would scroll off the screen, it invokes the pager.)

To always have a pager, use the value always. To never use the pager — useful inside a terminal multiplexer or terminal emulator with good scrolling — use the value off.

You can also change the pager itself by setting the PAGER environment variable. For example:

export PAGER="/usr/local/bin/gvim -f -R -"

This will use gvim(1) as your pager.

null

By default NULL values show as blank spaces. Also by default the empty string shows as a blank space.

bands=# INSERT INTO members (name) VALUES ('');
INSERT 0 1
bands=# INSERT INTO members (name) VALUES (NULL);
INSERT 0 1
bands=# SELECT * FROM members;
 id |     name      
----+---------------
  1 | David Pajo
  2 | Brian McMahan
  3 | 
  4 | 
(4 rows)

To better distinguish NULL values from empty strings, you can have psql(1) show anything you want instead:

bands=# \pset null '(null)'
Null display is "(null)".
bands=# SELECT * FROM members;
 id |     name      
----+---------------
  1 | David Pajo
  2 | Brian McMahan
  3 | 
  4 | (null)
(4 rows)

And more

You can find all of this and more in the psql(1) manpage or in the official PostgreSQL Web documentation. We have also written previously on this topic.

As you read the documentation we’d love to see your favorite settings as pull requests against the .psqlrc in our dotfiles.

Episode #482 - July 22nd, 2014

Posted about 1 month back at Ruby5

Get your mind in the Gutter, agree that Programming is Not Math, be a RubyCritic, master Vim Plugins for Ruby, review 3 Ways to Create Classes in Ruby, and take a trip to RailsPacific.

Listen to this episode on Ruby5

Sponsored by CodeShip.io

Codeship is a hosted Continuous Delivery Service that just works.

Set up Continuous Integration in a few steps and automatically deploy when all your tests have passed. Integrate with GitHub and BitBucket and deploy to cloud services like Heroku and AWS, or your own servers.

Visit http://codeship.io/ruby5 and sign up for free. Use discount code RUBY5 for a 20% discount on any plan for 3 months.

Also check out the Codeship Blog!

CodeShip.io

Gutter

Rah-jeev Kannav Sharma wrote to us to let us know about a gem called Gutter, a low-overheard monitoring web dashboard for GNU and Linux machines. It features live, on-demand monitoring of RAM, load, uptime, disk allocation, users, and many more stats.
Gutter

Programming Is Not Math

Sarah Mei wrote a really interesting blog post last week called Programming Is Not Math. She talks about how most the time programming is in fact much more like a language skill. Yet, somehow most computer science degrees focus quite heavily on math while it’s very possible to not need it that much later on.
Programming Is Not Math

RubyCritic

Guilherme Simões sent us a note about RubyCritic, a gem he built for his Master's thesis. He describes it as an opinionated version of the MetricFu gem which does static code analysis.
RubyCritic

Vim Plugins for Ruby

Milos Dolobac sent us a note this week about a blog post he wrote called Vim Plugins for Ruby. According to him, these plugins are productivity boosters that every Ruby developer should know about.
Vim Plugins for Ruby

3 Ways to Create Classes in Ruby

If you’ve written almost anything in Ruby, you’ve probably come across a Ruby class, and as Thuva Tharma explains in his most recent blog post, there are actually three ways to create a class in Ruby. Each of these styles could come in handy if you know how to use them.
3 Ways to Create Classes in Ruby

RailsPacific

We recently found out about RailsPacific -- the first ever Ruby on Rails Conference in Asia. It runs September 26-27, one week after RubyKaigi, in Taipei, Taiwan. The conference is divided into one day of speaking and one day of workshops, including workshops on refactoring, performance tuning, object-oriented design, and TDD with RSpec.
RailsPacific

Sponsored by Top Ruby Jobs

Simon & Schuster is looking for a Ruby on Rails developer in New York, NY Adobe Systems is looking for a Senior Web Developer in San Jose & San Francisco, CA Underdog.io is looking for a Ruby on Rails Developer in New York, NY or remote and Cambridge Systematics is looking for a Ruby on Rails Engineer in Cambridge, MA
Top Ruby Jobs

Sponsored by Ruby5

Ruby5 is released Tuesday and Friday mornings. To stay informed about and active with this podcast, we encourage you to do one of the following:

Thank You for Listening to Ruby5

Spring Stroll

Posted about 1 month back at Mike Clark

Spring Stroll

Just out for a leisurely stroll through the balsam root...

Ruby 2 Keyword Arguments

Posted about 1 month back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

Ruby 2.0 introduced first-class support for keyword arguments:

def foo(bar: 'default')
  puts bar
end

foo # => 'default'
foo(bar: 'baz') # => 'baz'

In Ruby 1.9, we could do something similar using a single Hash parameter:

def foo(options = {})
  bar = options.fetch(:bar, 'default')
  puts bar
end

foo # => 'default'
foo(bar: 'baz') # => 'baz'

Ruby 2.0 blocks can also be defined with keyword arguments:

define_method(:foo) do |bar: 'default'|
  puts bar
end

foo # => 'default'
foo(bar: 'baz') # => 'baz'

Again, to achieve similar behavior in Ruby 1.9, the block would take an options hash, from which we would extract argument values.

Required keyword arguments

Unfortunately, Ruby 2.0 doesn’t have built-in support for required keyword arguments. Luckily, Ruby 2.1 introduced required keyword arguments, which are defined with a trailing colon:

def foo(bar:)
  puts bar
end

foo # => ArgumentError: missing keyword: bar
foo(bar: 'baz') # => 'baz'

If a required keyword argument is missing, Ruby will raise a useful ArgumentError that tells us which required argument we must include.

Keyword arguments vs options hash

With first-class keyword arguments in the language, we don’t have to write the boilerplate code to extract hash options. Unnecessary boilerplate code increases the opportunity for typos and bugs.

With keyword arguments defined in the method signature itself, we can immediately discover the names of the arguments without having to read the body of the method.

Note that the calling code is syntactically equal to calling a method with hash arguments, which makes for an easy transition from options hashes to keyword arguments.

Keyword arguments vs positional arguments

Assume we have a method with positional arguments:

def mysterious_total(subtotal, tax, discount)
  subtotal + tax - discount
end

mysterious_total(100, 10, 5) # => 105

This method does its job, but as a reader of the code using the mysterious_total method, I have no idea what those arguments mean without looking up the implementation of the method.

By using keyword arguments, we know what the arguments mean without looking up the implementation of the called method:

def obvious_total(subtotal:, tax:, discount:)
  subtotal + tax - discount
end

obvious_total(subtotal: 100, tax: 10, discount: 5) # => 105

Keyword arguments allow us to switch the order of the arguments, without affecting the behavior of the method:

obvious_total(subtotal: 100, discount: 5, tax: 10) # => 105

If we switch the order of the positional arguments, we are not going to get the same results, giving our customers more of a discount than they deserve:

mysterious_total(100, 5, 10) # => 95

Connascence and trade-offs

Connascence between two software components A and B means either 1) that you can postulate some change to A that would require B to be changed (or at least carefully checked) in order to preserve overall correctness, or 2) that you can postulate some change that would require both A and B to be changed together in order to preserve overall correctness. - Meilir Page-Jones, What Every Programmer Should Know About Object-Oriented Design

When one Ruby method has to know the correct order of another method’s positional arguments, we end up with connascence of position.

If we decide to change the order of the parameters to mysterious_total, we must change all callers of that method accordingly. Not only that, but our mental model of how to use this method must change as well, which isn’t as simple as a find/replace.

Like most things, keyword arguments have their trade-offs. Positional arguments offer a more succinct way to call a method. Usually, the code clarity and maintainability gained from keyword arguments outweigh the terseness offered by positional arguments. I would use positional arguments if I could easily guess their meanings based on the method’s name, but I find this rarely to be the case.

What’s Next?

Episode #481 - July 18th, 2014

Posted about 1 month back at Ruby5

Take a peek into your app, think about accessibility, write polyglot web apps, learn Rails, say goodbye to 1.8.7 and 1.9.2 support

Listen to this episode on Ruby5

Sponsored by New Relic

New Relic is _the_ all-in-one web performance analytics product. It lets you manage and monitor web application performance, from the browser down to the line of code. With Real User Monitoring, New Relic users can see browser response times by geographical location of the user, or by browser type.
This episode is sponsored by New Relic

Peek

Take a peek into your Rails application!
Peek

accessibility

Simple chrome extension to notify websites of your accessibility requirements.
accessibility

polyglot

Polyglot is a distributed web framework that allows programmers to create web applications in multiple programming languages.
polyglot

GoRails

GoRails is a series of screencasts and guides for all aspects of Ruby on Rails.
GoRails

EOL for 1.8.7 and 1.9.2

Extended maintenance of Ruby versions 1.8.7 and 1.9.2 will end on July 31, 2014.
EOL for 1.8.7 and 1.9.2