Site offline for maintenance

Posted 3 months back at entp hoth blog - Home

Hi all,

Tender will be offline for 5-10 minutes tonight, Thursday 1/9 at 9pm PST, while we perform a maintenance operation on one of our database servers. Emails coming in during this period will be slightly delayed.

We apologize for the inconvenience.

If you have any question or concern, drop us a line at support@tenderapp.com.

Thanks!

Arduino-Based Bathroom Occupancy Detector

Posted 4 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

At the thoughtbot Boston office, when nature calls, there is an arduous, uphill both ways and often snow covered trek to the bathroom area. The bathrooms are also out of sight from anywhere in the office. Many times someone will turn the corner to the bathrooms and see that they are all occupied. That person now must either wait around until a bathroom opens up or go back to their desk and try again later. We wanted an indicator, visible throughout our Boston office, that informs whether a bathroom is available. We used the power of hardware to hack together a solution.

Arduino-Based Bathroom Occupancy Detector

Overview

We decided to use two microcontrollers to monitor and report the bathroom status. One microcontroller would sit near the bathrooms and use door sensors to detect if they were closed or open. The other microcontroller would sit in an area visible to everyone in the office and use an LED to indicate the status of the bathroom doors. They will have to communicate wirelessly since their distance apart could be long. The bathroom door microcontroller has to run from batteries since there is not a nearby outlet. We also decided that besides using an LED to show if there was an available door, we would also post the info to a remote server so other applications could digest it.

Hardware

We'll be using Arduinos for the microcontrollers because there is a ton of support and information about how to use them available online. For similar reasons, we will use XBee V1 radios for the communication between the Arduinos. The door sensing Arduino will be an Arduino Fio because it comes with a LiPo battery charging circuit and an easy way to plug in an XBee radio. The reporting Arduino will be an Arduino Yún because of its WiFi capabilities. The door sensors are generic reed switches with magnets. We'll also use a couple of solderless breadboards to prototype extra circuitry needed to tie things together.

Door Sensors

These sensors are a combination of a magnet and a reed switch. The magnet is placed on the door and the reed switch is on the frame. When the door closes, the magnet closes the reed switch, closing the circuit. We connect one side of the switch to power and the other side goes into a GPIO port on the Arduino. We must also connect a pull down resistor from the GPIO port to ground so when the switch is open, the port reads a low signal. Now when the door is shut, the switch is closed and there is a high signal on the port. When the door is open, the switch is open and there is a low signal on the port.

Door Sensor

Arduino Fio (sensor module)

This module is responsible for sensing the bathroom door's state and sending that state to the reporter. First, we wire up the two door sensors to the Fio. We'll use digital pins 2 and 3 on the Fio so we can use the interrupts later for power savings. Then we connect a common 10K resistor as pull down to ground from each pin. The only thing left to do is plug the XBee into the dedicated connector on the Fio. Now that everything is connected, we need to write the code that reads the sensors and sends their state to the XBee.

Arduino Fio

void setup() {
  pinMode(2, INPUT);
  pinMode(3, INPUT);
  Serial1.begin(9600);
}

Here we initialize digital pins 2 and 3 as inputs and create a serial interface for communication with the XBee. The XBee is programmed at 9600 baud by default so we need to create the serial connection to match. Now in the main program loop, lets check the door sensors and send their state to the XBee.

void loop() {
  int leftDoor = digitalRead(2);
  int rightDoor = digitalRead(3);

  Serial1.write(0xF0);
  Serial1.write(leftDoor);
  Serial1.write(rightDoor);
  Serial1.write(0xF1);

  delay(1000);
}

Here we read the status of the doors and then send it to the XBee. Lets add a delay of 1000ms or 1 second so we are not constantly sending data, but also update often enough so the data isn't stale. We also add a prefix and postfix to our transmission so it is easier to receive on the other end.

This implementation isn't very efficient because there could be long periods of time where neither door has changed. Lets save the state of the doors and transmit only if one has changed.

int leftDoorState = 0;
int rightDoorState = 0;
boolean hasChanged = false;

void loop() {
  int leftDoor = digitalRead(2);
  int rightDoor = digitalRead(3);

  if (leftDoor != leftDoorState) {
    leftDoorState = leftDoor;
    hasChanged = true;
  }

  if (rightDoor != rightDoorState) {
    rightDoorState = rightDoor;
    hasChanged = true;
  }

  if (hasChanged) transmit();

  delay(1000);
}

void transmit() {
  Serial1.write(0xF0);
  Serial1.write(leftDoorState);
  Serial1.write(rightDoorState);
  Serial1.write(0xF1);
  hasChanged = false;
}

Here we create two global variables to hold the current door's state. When we read the doors' states, we check if it has changed since the last read. If either of the doors has changed state, we transmit the data with the XBee. This will save us from transmitting every second to maybe only 20 to 50 transmits per day depending on bathroom use. The XBee module is using the most power when transmitting so reducing the number of times it has to transmit will reduce the power consumption.

We can go one step further and use interrupts to notify the Arduino that a door state has changed.

void setup() {
  pinMode(2, INPUT);
  attachInterrupt(0, transmitDoorState, CHANGE);

  pinMode(3, INPUT);
  attachInterrupt(1, transmitDoorState, CHANGE);

  Serial1.begin(9600);
}

void transmitDoorState() {
  int leftDoor = digitalRead(2);
  int rightDoor = digitalRead(3);

  Serial1.write(0xF0);
  Serial1.write(leftDoorState);
  Serial1.write(rightDoorState);
  Serial1.write(0xF1);
}

void loop() {

}

We attach a change interrupt to each of our door sensors. Any time a door state changes, the interrupt will execute the function transmitDoorState. Since our interrupt is doing all the work now, we can remove the global variables and all functionality from the main program loop.

The sensor module is now waiting for a change on the interrupt pins before it does anything. While its waiting, the CPU is active and processing no-op commands. This is waisting power because we're keeping the CPU on when we're not doing anything. To save power lets sleep the CPU when we don't need it.

#include <avr/sleep.h>

void setup() {
  pinMode(2, INPUT);
  attachInterrupt(0, transmitDoorState, CHANGE);

  pinMode(3, INPUT);
  attachInterrupt(1, transmitDoorState, CHANGE);

  Serial1.begin(9600);
}

void transmitDoorState() {
  int leftDoor = digitalRead(2);
  int rightDoor = digitalRead(3);

  Serial1.write(0xF0);
  Serial1.write(leftDoorState);
  Serial1.write(rightDoorState);
  Serial1.write(0xF1);
  delay(100);
}

void loop() {
  set_sleep_mode(SLEEP_MODE_IDLE);
  sleep_enable();
  sleep_mode();
}

This will put the Arduino into an idle sleep while waiting for a change on the interrupts. When a change occurs, the Arduino will wake up, transmit the state, then go back to sleep. We need to add a delay after we send data to the XBee to ensure that all the data has been sent before we try to sleep the processor again.

Further Improvements

The transmitDoorState function is an Interrupt Service Routine or ISR. An ISR should be quick because you want to be out of it before another interrupt occurs. The transmitDoorState function is not very quick because it has to send data serially and delay for 100ms. We probably won't run into any issues since the time between interrupts (door opening and closing) will most likely be greater than the time it takes to transmit, but to be safe we could move this code into the program loop and execute it after the sleep function. We could also reduce the power consumption further by using the sleep mode SLEEP_MODE_PWR_DOWN. This sleep mode affords us the most power savings but doesn't allow us to use the CHANGE interrupt. Instead we would have to use level interrupts at LOW or HIGH and manage which one to interrupt on depending on the state of the door.

Arduino Yún (reporter module)

This module is responsible for receiving the door state data and reporting that state to the office. First, we need to wire up the XBee module. It would be easy to get a XBee shield for the Arduino and use that but we have an XBee explorer instead so we need to manually wire up the XBee. Connect XBee explorer power and ground to the Arduino, then the RX and TX to digital pins 8 and 9 respectively (pins chosen at random). We also need to add a 10K pull up resistor from the RX and TX lines to power. Now add an LED in series with a 330 ohm resistor (or similar value) to digital pin 4 (also chosen at random). The LED & resistor combo should be connected low side and we'll drive it high from the Arduino to turn it on. Time to code!

Arduino Yún

#include <Bridge.h>
#include <Process.h>
#include <SoftwareSerial.h>

SoftwareSerial xbee(8, 9); // RX, TX

void setup()  {
  Bridge.begin();
  pinMode(4, OUTPUT);
  digitalWrite(4, LOW);
  xbee.begin( 9600 );
}

Here we set up the Bridge to the Linux processor on the Yún, set the mode of our LED pin to an output, start the LED off, and initialize the software serial connection for the XBee. Next, we need to receive data from the XBee and report the door status.

int leftDoor, rightDoor;

enum state {
  waiting_for_prefix,
  get_left_door,
  get_right_door,
  waiting_for_postfix
};

state currentState = waiting_for_prefix;

void loop()  {
  if (xbee.available()) {
    int data = xbee.read();

    switch (currentState) {
      case waiting_for_prefix:
        if (data == 0xF0)
          currentState = get_left_door;
        break;

      case get_left_door:
        leftDoor = data;
        currentState = get_right_door;
        break;

      case get_right_door:
        rightDoor = data;
        currentState = waiting_for_postfix;
        break;

      case waiting_for_postfix:
        if (data == 0xF0) {
          currentState = waiting_for_prefix;
          reportState();
        }
        break;
    }
  }
}

We'll use a state machine to receive the data from the XBee. There are four states: one for each the prefix and postfix to see the start and end of our data and one for each door. The door states will record the data from the XBee and the prefix and postfix states are used for flow control. When we see the postfix data we report the door states with reportState(). This function should turn on an LED if both doors are closed.

void reportState() {
  digitalWrite(4, leftDoor & rightDoor);
}

When the doors are closed their state is HIGH or 1 so when they are both HIGH the bitwise & operator will evaluate that to HIGH and turn on the LED.

Lets also report the door status online. Connect the Arduino Yún to your WiFi network. The Arduino processor can send shell commands to the Linux processor on the board. We will use the curl command to post the door states to an external API.

void reportState() {
  digitalWrite(4, leftDoor & rightDoor);

  Process curl;
  curl.begin("curl");
  curl.addParameter("--data");

  String data = "leftDoor=";
  data += leftDoor;
  data += "&rightDoor=";
  data += rightDoor;

  curl.addParameter(data);
  curl.addParameter(YOUR_EXTERNAL_API_SERVICE);
  curl.run();
}

That's all! Now, you'll be able to see if there is an available bathroom via an LED in the room or with a cool application that digests the API web service.

Areas For Improvement

During the project we realized that the power consumption of the XBee module was very high and had no way of going into a low power state. We measured the current consumption from the Fio board by placing a 0.5 ohm resistor in series with the power from the battery. The voltage drop across the resistor divided by the resistance gives us the current which was around 50mA to 60mA. The battery we are using is a 400mAh battery, so we would get about 8 hours of use. Charging the battery everyday is not an option so more research needs to be done into a lower power wireless communication solution. Also, this solution was expensive and more research can be done to find a lower cost implementation. thoughtbot Boston has two other bathrooms also located in remote places. We eventually want door sensors on these as well and a low cost setup for the door sensor would make this easier.

Episode #430 - January 7th, 2014

Posted 4 months back at Ruby5

Test Driving a JSON API in Rails, Jubilee for Vert.x, Exception#cause, Hulse, Caching an API

Listen to this episode on Ruby5

This episode is sponsored by Top Ruby Jobs
If you're looking for a top Ruby job or for top Ruby talent, then you should check out Top Ruby Jobs. Top Ruby Jobs is a website dedicated to the best jobs available in the Ruby community.

Test Driving a JSON API in Rails
Eno Compton wrote a blog post on "Test Driving a JSON API in Rails" where he touches on some pretty relevant details that tend to be overlooked by most developers.

Jubilee for Vert.x
Jubilee is a rack server that uses the best features of Vert.x 2.0 such as Event bus, Shared data and Clustering.

Ruby 2.1 Exception#cause
The folks at Bugsnag wrote a blog post describing Ruby 2.1's Exception#cause feature which allows access to the root cause of an exception in cases where multiple exceptions are raised.

Hulse
Hulse, by Derek Willis, is a Ruby wrapper for House and Senate roll call votes. It parses Congressional vote data from the official House of Representatives and Senate websites.

Free Ruby Tapas on Caching an API
Avdi Grimm has just freed up another episode from the Ruby Tapas archives on Caching an API. He talks about building a caching layer over an HTTP API.

Playbook v2

Posted 4 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

We're pleased to announce the second major, 14,000 word version of our playbook. We're giving it away for free and licensing it as Creative Commons Attribution-NonCommercial.

The playbook is the most comprehensive description that we've ever gathered of how we run our company and how we make web and mobile products. We did not invent many of the techniques we describe, but we use them regularly in the course of doing real work.

Old

We released the first version in August of 2011.

Some of our tools and techniques are the same as then, such as Heroku for application hosting, New Relic for performance monitoring, and Airbrake for error tracking.

The developing section is strongly influenced by Extreme Programming rules, which continues to be best practice well into its second decade of popularity.

New

Some things are new.

Right up front are product design sprints, strongly influenced by Google Ventures' design team.

Another overhauled section is about using Trello to manage our planning (strongly influenced by UserVoice), sales, and hiring processes.

We mention for the first time our style guide and how to follow the guidelines pragmatically.

We include a production checklist to make sure we don't forget an important part of a production deployment before launch.

We mention Zapier quite a few times. It has become important to us as glue between systems like StackOverflow and Campfire, FreshBooks and Campfire, email to Trello, Trello to Campfire, Stripe to Campfire, Wufoo to Trello, and a few others.

PDF, MOBI, and EPUB

The HTML version is free at http://playbook.thoughtbot.com. You can get the PDF, MOBI, and EPUB versions for free by subscribing to our monthly-ish newsletter, The Bot Cave.

We hope you find it useful.

The Unix Shell's Humble If

Posted 4 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

The Unix shell is often overlooked by software developers more familiar with higher level languages. This is unfortunate because the shell can be one of the most important parts of a developer's toolkit. From the one-off grep search or sed replacement in your source tree to a more formal script for deploying or backing up an entire application complete with options and error handling, the shell can be a huge time saver.

To help shed light on the power and subtlety that is the Unix shell, I'd like to take a deep dive into just one of its many features: the humble if statement.

Syntax

The general syntax of an if statement in any POSIX shell is as follows:

if command ; then
  expressions

elif command ; then  # optionally
  expressions

else command         # optionally
  expressions

fi

The if statement executes command and determines if it exited successfully or not. If so, the "consequent" path is followed and the first set of expressions is executed. Otherwise, the "alternative" is followed. This may mean continuing similarly with an elif clause, executing the expressions under an else clause, or simply doing nothing.

if grep -Fq 'ERROR' development.log; then
  # there were problems at some point
elif grep -Fq 'WARN' development.log; then
  # there were minor problems at some point
else
  # all ok!
fi

The command can be a separate binary or shell script, a shell function or alias, or a variable referencing any of these. Success is determined by a zero exit-status or return value, anything else is failure. This makes sense: there may be many ways to fail but there should be exactly one way to succeed.

is_admin() {
  return 1
}

if is_admin; then
  # this will not run
fi

If your command is a pipeline, the exit status of the last command in the pipeline will be used:

# useless use of cat for educational purposes only!
if cat development.log | grep -Fq 'ERROR'; then
  # ...
fi

For the most part, this is intuitive and expected. In cases where it's not, some shells offer the pipefail option to change that behavior.

Negation, True, and False

The ! operator, when preceding a command, negates its exit status. Additionally, both true and false are normal commands on your system which do nothing but exit appropriately:

true; echo $?
# => 0

false; echo $?
# => 1

! true; echo $?
# => 1

The ! operator allows us to easily form an "if-not" statement:

if ! grep -Fq 'ERROR' development.log; then
  # All OK
fi

The availability of true and false is what makes statements like the following work:

if true; fi
  # ...
fi

var=false

if ! "$var"; then
  # ...
fi

However, you should avoid doing this. The idiomatic (and more efficient) way to represent booleans in shell scripts is with the values 1 (for true) and 0 (for false). This idiom is made more convenient if you have (( available, which we'll discuss later.

The test Command

The test command performs a test according to the options given, then exits successfully or not depending on the result of said test. Since this is a command like any other, it can be used with if:

if test -z "$variable"; then
  # $variable has (z)ero size
fi

if test -f ~/foo.txt; then
  # ~/foo.txt is a regular (f)ile
fi

test accepts a few symbolic options as well, to make for more readable statements:

if test "$foo" = 'bar'; then
  # $foo equals 'bar', as a string
fi

if test "$foo" != 'bar'; then
  # $foo does not equal bar, as a string
fi

The = and != options are only for string comparisons. To compare numerically, you must use -eq and -ne. See man 1 test for all available numeric comparisons.

Since commands can be chained together logically with && and ||, we can combine conditions intuitively:

if test "$foo" != 'bar' && test "$foo" != 'baz'; then
  # $foo is not bar or baz
fi

Be aware of precedence. If you need to enforce it, group your expressions with curly braces.

if test "$foo" != 'bar' && { test -z "$bar" || test "$foo" = "$bar"; }; then
  # $foo is not bar and ( $bar is empty or $foo is equal to it )
fi

Note the final semi-colon before the closing brace

If your expression is made up entirely of test commands, you can collapse them using -a or -o. This will be faster since it's only one program invocation:

if test "$foo" != 'bar' -a "$foo" != 'baz'; then
  # $foo is not bar or baz
fi

The [ Command

Surprisingly, [ is just another command. It's distributed alongside test and its usage is identical with one minor difference: a trailing ] is required. This bit of cleverness leads to an intuitive and familiar form when the [ command is paired with if:

if [ "string" != "other string" ]; then
  # same as if test "string" != "other string"; then
fi

Unfortunately, many users come across this usage first and assume the brackets are part of if itself. This can lead to some nonsensical statements.

Rule: Never use commands and brackets together

Case in point, this is incorrect:

if [ grep -q 'ERROR' log/development.log ]; then
  # ...
fi

And so is this:

if [ "$(grep -q 'ERROR' log/development.log)" ]; then
  # ...
fi

The former is passing a number of meaningless words as arguments to the [ command; the latter is passing the string output by the (quieted) grep invocation to the [ command.

There are cases where you might want to test the output of some command as a string. This would lead you use a command and brackets together. However, there is almost always a better way.

# this does work
if [ -n "$(grep -F 'ERROR' log/development.log)" ]; then
  # there were errors
fi

# but this is better
if grep -Fq 'ERROR' development.log; then
  # there were errors
fi

# this also works
if [ -n "$(diff file1 file2)" ]; then
  # files differ
fi

# but this is better
if ! diff file1 file2 >/dev/null; then
  # files differ
fi

As with most things, quoting is extremely important. Take the following example:

var="" # an empty string

if [ -z $var ]; then
  # string is empty
fi

You'll find if you run this code, it doesn't work. The [ command returns false even though we can clearly see that $var is in fact empty (a string of zero size).

Since [ OPTION is valid usage for [, what's actually being executed by the shell is this:

if [ -z ]; then
  # is the string "]" empty? No.
fi

The fix is to quote correctly:

if [ -z "$var" ]; then
  # is the string "" empty? Yes.
fi

When are quotes needed? Well, to paraphrase Bryan Liles…

Rule: Quote All the Freaking Time

Examples: "$var", "$(command)" "$(nested "$(command "$var")")"

It's common for script authors to use the following to "fix" this issue as well:

if [ x$var = x ]; then
  # $var is empty
fi

Don't do this. It will still fail if $var contains any whitespace. The only way to properly handle whitespace is to properly quote; and once the expression is properly quoted, the "x trick" is no longer needed.

Non-POSIX Concerns

In most modern shells like bash and zsh, two built-ins are available: [[ and ((. These perform faster, are more intuitive, and offer many additional features compared to the test command.

Best Practice: If you have no reason to target POSIX shell, use [[

Bracket-Bracket

[[ comes with the following features over the normal test command:

  • Use familiar ==, >=, and <= operators
  • Check a string against a regular expression with =~
  • Check a string against a glob with ==
  • Less strict about quoting and escaping

You can read more details about the difference here.

While the operators are familiar, it's important to remember that they are string (or file) comparisons only.

Rule: Never use [[ for numeric comparisons.

For that, we'll use (( which I'll explain shortly.

When dealing with globs and regular expressions, we immediately come to another rule:

Rule: Never quote a glob or regular expression

I know, I just said to quote everything, but the shell is an epic troll and these are the only cases where quotes can hurt you, so take note:

for x in "~/*"; do
  # This loop will run once with $x set to "~/*" rather than once 
  # for every file and directory under $HOME, as was intended
done

for x in ~/*; do
  # Correct
done

case "$var" of
  'this|that')
    # This will only hit if $var is exactly "this|that" 
    ;;

  '*')
    # This will only hit if $var is exactly "*"
    ;;
esac

# Correct
case "$var" of
  this|that) ;;
  *) ;;
esac

foo='foobarbaz'

if [[ "$foo" == '*bar*' ]]; then
  # True if $foo is exactly "*bar*"
fi

if [[ "$foo" == *bar* ]]; then
  # Correct
fi

if [[ "$foo" =~ '^foo' ]]; then
  # True if $foo is exactly "^foo", but leading or trailing 
  # whitespace may be ignored such that this is also true if $foo is 
  # (for example) "  ^foo  "
if

if [[ "$foo" =~ ^foo ]]; then
  # Correct
fi

If the glob or regular expression becomes unwieldy, you can place it in a variable and use the (unquoted) variable in the expression:

pattern='^Home sweet'

if [[ 'Home sweet home' =~ $pattern ]]; then
  # ...
fi

myfiles='~/*'

for file in $myfiles; do
  # ...
done

After regular expression matches, you can usually find any capture groups in a magic global. In bash, it's BASH_REMATCH.

if [[ 'foobarbaz' =~ ^foo(.*)baz$ ]]; then
  echo ${BASH_REMATCH[1]}
  # => "bar"
fi

And in zsh, it's match.

if [[ 'foobarbaz' =~ ^foo(.*)baz$ ]]; then
  echo $match[1]
  # => "bar"
fi

Note that in zsh, you don't need curly braces for array element access

Math and Numerical Comparisons

The built-in (( or Arithmetic Expression is concerned with anything numeric. It's an enhancement on the POSIX $(( )) expression which replaced the ancient expr program for doing integer math.

i=1

# old, don't use!
i=$(expr $i+1)

# better, POSIX
i=$((i+1))

# valid in shells like bash and ksh93
((i++))

# alternate syntax
let i++

The difference between $((expression)) and ((expression)) or let expression is whether you want the result or not. Also notice that in either form, we don't need to use the $ when referencing variables. This is true in most but not all cases ($# is one where it's still required).

When comparison operators are used within ((, it will perform the comparison and exit accordingly (just like test). This makes it a great companion to if:

if ((x == 42)); then
  # ...
if

if ((x < y)); then
  # ...
fi

Here's a more extended example showing that it can be useful to perform arithmetic and comparisons in the same expression:

retry() {
  local i=1 max=5

  while ((i++ <= max)); do
    if try_something; then
      print "Call succeeded.\n"
      return 0
    fi
  done

  printf "Maximum attempts reached!\n" >&2
  return 1
}

The (( form can also check numbers for "truthiness". Namely, the number 0 is false. This makes our boolean idiom a bit more convenient:

var=1

# POSIX
if [ "$var" -eq 1 ]; then
  # ...
fi

# bash, zsh, etc
if ((var)); then
  # ...
fi

# example use-case. $UID of the root user is 0.
if ((UID)); then
  error "You must be root"
fi

This will perform better than a fork-exec of /bin/true or /bin/false.

Conclusion

To recap, we've seen that if in the Unix shell is both simple and complex. It does nothing but execute a command and branch based on exit status. When combined with the test command, we can make powerful comparisons on strings, files, and numbers while upgrading to [ gives the same comparisons a more familiar syntax. Additionally, using non-POSIX enhancements like [[ and (( gives us globs, regular expressions, and better numeric comparisons.

You've also seen a number of rules and best practices to ensure your shell scripts act as you intend in as many shells as you choose to run them.

Episode #429 - January 3rd, 2014

Posted 4 months back at Ruby5

Writing a Ruby compiler, the Omega universe simulator, RubyGems 2.2.0, debugging with HTTP clients, detecting similarities in images, and the Lotus web framework all in this episode of the Ruby5!

Listen to this episode on Ruby5

This episode is sponsored by New Relic
New Relic is _the_ all-in-one web performance analytics product. It lets you manage and monitor web application performance, from the browser down to the line of code. With Real User Monitoring, New Relic users can see browser response times by geographical location of the user, or by browser type.

Writing a Compiler in Ruby from the Bottom Up
Ever wondered just what goes into writing a compiler? This long running series of blog posts from Vidar Hokstad will take you on a whirlwind tour!

The Omega Simulation Framework
The Omega universe simulation framework lets you create your very own game universe in the cloud!

RubyGems 2.2.0 Released
Happy Festivus, we got a new RubyGems!

Debugging an HTTP Client Library
Avdi gets to the bottom of an HTTP client.

Detect Similar Images
Check out part 1 of a series on detecting similar images.

Lotus
Lotus is a new full-stack web application framework for Ruby.

Phusion Passenger 4.0.33 released

Posted 4 months back at Phusion Corporate Blog


Phusion Passenger is a fast and robust web server and application server for Ruby, Python, Node.js and Meteor. Passenger takes a lot of complexity out of deploying web apps, and adds powerful enterprise-grade features that are useful in production. High-profile companies such as Apple, New York Times, AirBnB, Juniper, American Express, etc are already using it, as well as over 350.000 websites.

Phusion Passenger is under constant maintenance and development. Version 4.0.33 is a bugfix release.

Phusion Passenger also has an Enterprise version which comes with a wide array of additional features. By buying Phusion Passenger Enterprise you will directly sponsor the development of the open source version.

Recent changes

4.0.31 and 4.0.32 have been skipped because an incompatibility problem with very old Ruby versions was found by our build server shortly after tagging the releases. 4.0.32 fixes all those problems. The changes in 4.0.31, 4.0.32 and 4.0.33 combined are:

  • Introduced a new tool: passenger-config restart-app. With this command you can initiate an application restart without touching restart.txt. Unlike touching restart.txt, this tool initiates the restart immediately instead of on the next request.
  • Fixed some problems in process spawning and request handling.
  • Fixed some problems with the handling of HTTP chunked transfer encoding bodies. These problems only occurred in Ruby.
  • Fixed a compatibility problem in passenger-install-apache2-module with Ruby 1.8. The language selection menu didn’t work properly.
  • Fixed the HelperAgent, upon shutdown, not correctly waiting 5 seconds until all clients have disconnected. Fixes issue #884.
  • Fixed compilation problems on FreeBSD 10.
  • Fixed some C++ strict aliasing problems.
  • Fixed some problems with spawning applications that print messages without newline during startup. Fixes issue #1039.
  • Fixed potential hangs on JRuby when Ctrl-C is used to shutdown the server. Fixes issue #1035.
  • When Phusion Passenger is installed through the Debian package, passenger-install-apache2-module now checks whether the Apache module package (libapache2-mod-passenger) is properly installed, and installs it using apt-get if it’s not installed. Fixes issue #1031.
  • The passenger-status --show=xml command no longer prints the non-XML preamble, such as the version number and the time. Fixes issue #1037.
  • The Ruby native extension check whether it’s loaded against the right Ruby version, to prevent problems when people upgrade Ruby without recompiling their native extensions.
  • Various other minor Debian packaging improvements.

Installing or upgrading to 4.0.33

OS X OS X Debian Debian Ubuntu Ubuntu
Heroku Heroku Ruby gem Ruby gem Tarball Tarball

Final

Fork us on Github!

Phusion Passenger’s core is open source. Please fork or watch us on Github. :)

<iframe src="http://ghbtns.com/github-btn.html?user=phusion&amp;repo=passenger&amp;type=watch&amp;size=large&amp;count=true" allowtransparency="true" frameborder="0" scrolling="0" width="170" height="30"></iframe><iframe src="http://ghbtns.com/github-btn.html?user=phusion&amp;repo=passenger&amp;type=fork&amp;size=large&amp;count=true" allowtransparency="true" frameborder="0" scrolling="0" width="170" height="30"></iframe><iframe src="http://ghbtns.com/github-btn.html?user=phusion&amp;type=follow&amp;size=large&amp;count=true" allowtransparency="true" frameborder="0" scrolling="0" width="190" height="30"></iframe>

If you would like to stay up to date with Phusion news, please fill in your name and email address below and sign up for our newsletter. We won’t spam you, we promise.



Announcing Lotus

Posted 4 months back at Luca Guidi - Home

I’m pleased to announce Lotus: the Open Source project I’ve conceived, hacked and built during the last year.

Lotus is a full stack web framework for Ruby, built with lightness, performances and testability in mind. It aims to bring back Object Oriented Programming to web development, leveraging on stable APIs, a minimal DSL, and plain objects.

Standalone frameworks

It’s composed by standalone frameworks (controllers, views, etc..), each one is shipped as an independent gem, in order to remark the separation of concerns. They can be used with any Rack compatible application for a specific need: for instance, Lotus::Router can be used to dispatch HTTP requests for a pool of Sinatra applications.

Full stack application

The other way to use Lotus is to build a full stack application with it, like Rails does. The Lotus gem is designed to enhance those frameworks’ features with a few specific conventions.

Philosophy

Lotus is based on simplicity, less DSLs, few conventions, more objects, zero monkey-patching of the core language and standard lib, separation of concerns for MVC layers. It suggests patterns, rather than imposing. It leaves all the freedom to developers to build their own architecture, choose the inheritance structure. It simplifies testability, and encourages single, well defined responsibilities between classes.

Roadmap

Lotus is a complex software, it needs to be completed, and to get feedback in order to became production ready. Some of its frameworks already have reached a certain degree of maturity, other still needs to be crafted as a gem yet. A single release day would be hard to meet as expectation, so I would like to suggest an experiment: to open source a component on the 23rd of every month, starting from January with Lotus::Utils and Lotus::Router.

Happy new year!

Defense Wins Championships - Hope for the Trail Blazers

Posted 4 months back at Encytemedia - Home

"Defense Wins Championships." If you're a fan of the NBA (or most major sports), you've likely heard this phrase. I've always considered it fodder for punditry; something you say when you have nothing objective to add to a conversation. Sure, it’s cliché, but is it true?

--

December is almost over and we’re eight weeks into another NBA season. Some great story lines have emerged. Kobe makes a miraculous comeback after tearing his achilles only to promptly injure his knee; Derrick Rose made his return from a season ending injury and tore the meniscus in his right knee, knocking him out for yet another season; First year head coach Jeff Hornacek has the Phoenix Suns, a team people expected to be in the lottery, making a case for the playoffs in a stacked Western Conference. And then there’s the Portland Trail Blazers.

The Blazers, with a record 24-7, are the Cinderella team of the season. A few people expected them to improve, but no one expected them to be sitting near the top of the NBA rankings. The best-case preseason scenario was barely winning enough games to secure the 8th and final playoff spot in the west.

The Blazers area an interesting team. They have the NBAs best offense; One of the best offenses in the NBA in the last 17 years. Taking a look through stats back to the 1996-97 season, this year’s Blazer team ranks #10 in Offensive Rating, behind some dominating Nash-led Phoenix Suns teams.

Offensive Rating- The number of points scored per 100 possessions. By using possessions, you eliminate pace as a factor because some teams play faster, resulting in more possessions.


 rank | season  |       team_name        | wins | losses | advanced_off_rating
------+---------+------------------------+------+--------+---------------------
    1 | 2009-10 | Phoenix Suns           |   54 |     28 |               112.7
    2 | 2004-05 | Phoenix Suns           |   62 |     20 |               111.9
    3 | 2006-07 | Phoenix Suns           |   61 |     21 |               111.4
    4 | 2008-09 | Phoenix Suns           |   46 |     36 |               111.2
    5 | 2007-08 | Phoenix Suns           |   55 |     27 |               111.2
    6 | 1996-97 | Chicago Bulls          |   69 |     13 |               111.1
    7 | 2007-08 | Utah Jazz              |   54 |     28 |               110.8
    8 | 2008-09 | Portland Trail Blazers |   54 |     28 |               110.7
    9 | 1996-97 | Utah Jazz              |   64 |     18 |               110.5
   10 | 2013-14 | Portland Trail Blazers |   24 |      7 |               110.4

While the Blazers offense under Terry Stotts has been humming along, the defense is sitting near the bottom of the league at #23 allowing 105 points per 100 possessions.

Defensive Rating- The number of points allowed per 100 possessions. By using possessions, you eliminate pace as a factor because some teams play faster, resulting in more possessions.


 rank |       team_name        | wins | loses | advanced_def_rating
------+------------------------+------+-------+---------------------
    1 | Indiana Pacers         |   24 |     5 |                93.6
    2 | Oklahoma City Thunder  |   25 |     5 |                97.4
    3 | Charlotte Bobcats      |   14 |    18 |                98.3
    4 | Golden State Warriors  |   19 |    13 |                98.5
    5 | Chicago Bulls          |   12 |    17 |                98.6
    6 | San Antonio Spurs      |   24 |     7 |                98.8
    7 | Los Angeles Clippers   |   21 |    12 |               100.4
    8 | Miami Heat             |   24 |     7 |               100.7
    9 | Toronto Raptors        |   13 |    15 |               101.6
   10 | Phoenix Suns           |   19 |    11 |               101.8
   11 | Boston Celtics         |   13 |    17 |               102.1
   12 | Houston Rockets        |   21 |    12 |               102.4
   13 | Minnesota Timberwolves |   15 |    16 |               102.5
   14 | Orlando Magic          |   10 |    20 |               102.5
   15 | Atlanta Hawks          |   17 |    14 |               102.6
   16 | Denver Nuggets         |   14 |    16 |               102.8
   17 | Washington Wizards     |   14 |    14 |                 103
   18 | Cleveland Cavaliers    |   10 |    20 |               103.9
   19 | Los Angeles Lakers     |   13 |    18 |               103.9
   20 | Dallas Mavericks       |   18 |    13 |               104.4
   21 | Detroit Pistons        |   14 |    19 |               104.8
   22 | Milwaukee Bucks        |    6 |    24 |               104.8
   23 | Portland Trail Blazers |   24 |     7 |                 105
   24 | New Orleans Pelicans   |   14 |    15 |               105.3
   25 | Memphis Grizzlies      |   13 |    17 |               105.5
   26 | New York Knicks        |    9 |    21 |               105.8
   27 | Brooklyn Nets          |   10 |    20 |               106.2
   28 | Sacramento Kings       |    9 |    20 |               106.5
   29 | Philadelphia 76ers     |    9 |    21 |               106.8
   30 | Utah Jazz              |   10 |    24 |               107.7

Does this mean the Blazers can't compete for a title? Conventional wisdom suggests they can't, but there is a chance and that chance is the 2000-01 Lakers.

Take a look at the chart below showing the offensive and defensive ratings of the two teams who competed for the championship back to the 1996-97 season.

The numbers on the outside represent a teams respective regular season rating, while the numbers near the circles represent the teams regular season rank. Shorter bars are better.
Champion Runner-up

<script src="/assets/posts/nba-champions-705b515bd5fa28c7fe2e58fb203c6c6a.js"></script>

Here are some takeaways from the chart:

  • 16 of the last 17 champions had a top 10 defense (94%). The 2000-01 Lakers being the outlier.
  • 14 of the last 17 champions had a top 10 offense (82%).
  • 13 of the last 17 runner-ups had a top 10 defense (76%)
  • 10 of the last 17 runner-ups had a top 10 offense (59%).

In the last 17 years, only one time has a team won the championship without a top 10 defense. That probably isn't the news Blazer fans want to here (I include myself in that group), but the 2000-01 Lakers are the silver lining.

It's ironic that Blazer fans would need to look to the 2000-01 Shaq and Kobe Lakers for hope given how much Blazer fans loathe the Lakers. Especially a Lakers team that swept them in the first round. But if there's reason for hope, this Lakers team is it.

In the 2000-01 season the Lakers had the #19 ranked defense (#21 according to basketball-reference.com) and the #2 ranked offense (similar to this years' Blazer team), but went on to win the title. But, how did they do it?

The 2000-01 Lakers #19 ranked defense allowed 101.7 points per 100 possessions. That defense would be ranked #10 in the 2013-14 season. But, I believe this can almost certainly be explained away by rule changes that have favored offenses. The league PPG average has grown by almost 4 points since hand checking was eliminated in the 2004-05 season.

However, The answer becomes clear when you look at the stats for the 2001 playoffs. The Lakers were motivated in the playoffs, jumping from a regular season rank of #19 to a playoff rank of #1!

The table below shows just how much individual players improved during the playoffs:

      player      | Regular Season Defense | Playoff Defense
------------------+------------------------+-----------------
 Kobe Bryant      |                    105 |              99
 Shaquille O'Neal |                    101 |              96
 Derek Fisher     |                    105 |             101
 Rick Fox         |                    106 |              98
 Horace Grant     |                    106 |              98
 Robert Horry     |                    104 |              95
 Brian Shaw       |                    105 |              99
 Tyronn Lue       |                    108 |              96
 Mark Madsen      |                    107 |             103
 Ron Harper       |                    105 |              91
 Devean George    |                    105 |             106
 Greg Foster      |                    104 |              96

Everyone, save Devean George, who played a total of 27 minutes during the Lakers' playoff run, improved on defense—and pretty drastically.

So the 2013-14 Blazers can win, but they'll need to make drastic improvements on defense. How drastic? I'll leave you with one last table, comparing players who've played at least 300 minutes on the top 5 teams in the NBA.


  rank |      player       | Defense Rating
------+-------------------+----------------
    1 | Roy Hibbert       |             93
    2 | Ian Mahinmi       |             94
    3 | Paul George       |             94
    4 | David West        |             95
    5 | Luis Scola        |             96
    6 | C.J. Watson       |             97
    7 | Tim Duncan        |             97
    8 | George Hill       |             98
    9 | Kevin Durant      |             98
   10 | Kawhi Leonard     |             98
   11 | Steven Adams      |             98
   12 | Serge Ibaka       |             98
   13 | Lance Stephenson  |             98
   14 | Danny Green       |             99
   15 | Jeff Ayres        |            100
   16 | Russell Westbrook |            100
   17 | Orlando Johnson   |            100
   18 | Kendrick Perkins  |            101
   19 | Thabo Sefolosha   |            101
   20 | Tiago Splitter    |            101
   21 | Chris Andersen    |            101
   22 | Manu Ginobili     |            101
   23 | Nick Collison     |            102
   24 | Reggie Jackson    |            102
   25 | Jeremy Lamb       |            102
   26 | Dwyane Wade       |            102
   27 | Chris Bosh        |            102
   28 | Patrick Mills     |            102
   29 | Derek Fisher      |            103
   30 | Marco Belinelli   |            103
   31 | LeBron James      |            103
   32 | Mario Chalmers    |            103
   33 | Rashard Lewis     |            103
   34 | Michael Beasley   |            104
   35 | Boris Diaw        |            104
   36 | LaMarcus Aldridge |            105
   37 | Thomas Robinson   |            106
   38 | Ray Allen         |            106
   39 | Shane Battier     |            106
   40 | Tony Parker       |            106
   41 | Norris Cole       |            106
   42 | Joel Freeland     |            107
   43 | Nicolas Batum     |            107
   44 | Robin Lopez       |            107
   45 | Wesley Matthews   |            109
   46 | Dorell Wright     |            109
   47 | Damian Lillard    |            110
   48 | Mo Williams       |            110

Rendering Collections in Rails

Posted 4 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

Partials are a great way to break down complex view into more manageable chunks as well as keeping view code DRY. However, rendering a partial for each item in a collection looks ugly:

<% @user.comments.each do |comment| %>
  <%= render partial: 'comments/comment', locals: { comment: comment } %>
<% end %>

to_partial_path

Luckily, Rails gives us this beautiful shorthand syntax:

render @user.comments

How does this magic work? Under the hood, render calls to_partial_path on each our objects to determine which partial to render. Models that inherit from ActiveRecord::Base will return a partial name based on the model's name by default. For example:

User.new.to_partial_path
# => 'users/user'

You can override this:

class User < ActiveRecord::Base
  def to_partial_path
    'users/profile'
  end
end

POROs

This works with plain old Ruby objects too. Here's a Null Object Pattern example.

class Guest
  def name
    'Guest'
  end

  def to_partial_path
    'users/user'
  end
end

@dashboard.users_online is a mix of ActiveRecord User objects and non-persisted Guest objects

  <h1>Users online:</h1>
  <%= render @dashboard.users_online %>

The same partial is used for both guests and registered users.

  <%= user.name %>

Heterogeneous collections

It gets even better. The objects in the collection can be of different types, each with their own partial.

@user.favorites can contain any combination of Article, Comment, and Image objects

<h1><%= @user.name %>'s Favorites</h1>
<%= render @user.favorites %>

app/views/articles/_article.html.erb

<h1><%= article.name %></h1>
<p><%= article.content %></p>

app/views/comments/_comment.html.erb

  <div class="comment">
   <em>Last updated: <%= comment.updated_at %></em>
   <p><%= comment.content %></p>
  </div>

app/views/images/_image.html.rb

  <h1><%= image.title %></h1>
  <%= image_tag image.url %>

The beauty of this polymorphic approach is that we don't have to write any conditional code. As long as all the objects in the collection define to_partial_path, it all just works.

What's next?

If you found this useful, you might also enjoy:

Metis Info Session in Boston

Posted 4 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

We’ll be hosting a Metis and Ruby on Rails info session at our Boston office on January 9th from 7pm to 8pm. You can RSVP here.

The session will introduce you to Ruby on Rails and show you just how easy it is to quickly prototype websites. You'll also learn more about our upcoming 12 week Bootcamp Ruby on Rails, and how it can get you started in the world of web development.

The session will be led by Josh Steiner and Matt "Goose" Mongeau, the two instructors of the Metis bootcamp. It will be a great chance to meet Goose and Josh and ask them any questions you might have about the Metis program or Ruby on Rails in general. The session is free and food and beverages will be provided.

Currently the growth rate of web development jobs is one of the highest for any job in the United States. What better way to ring in the new year than to start learning web development!

If Metis isn’t right for you, but someone you know may be interested in Metis, we’d love for you to pass this event onto them.

If Metis is right for you, or you’re not sure it is yet, we’ll see you there!

Let it snow with Zepto—again!

Posted 4 months back at mir.aculo.us - Home

Updated with the latest and greatest Zepto.js… Let it snow!

<iframe src="http://zeptojs.com/let-it-snow/" width="100%" height="400"></iframe>

If you’re interested how this works, grab the source. Except for Zepto.js (built with detect & fx support) there’s no external dependencies (not counting the Twitter and analytics).

Happy Holidays!

P.S. Because it was so popular, I’ve started another flash sale of my ebook Retinafy your web sites and app—use coupon “hiresholidays” for 15% off, but only on December 24th and 25th!

Changelog 12/9 - 12/20

Posted 4 months back at entp hoth blog - Home

Happy Friday,

We have been a little quiet lately, mostly because we have been busy. We are in the process of upgrading a lot of our infrastructure: we bought shiny new servers, with lots of RAM, SSDs and myriad of other goodies, and we are installing them in our datacenter, configuring them, and slowly moving our different services to them. All of this takes time, as we are also upgrading our deployment process (we’re moving to Docker, for those interested). So there are a lot of balls in the air and this work will continue way into January. This slows down the development of new features and bug fixes, but it’s all for the best, I promise!

With that being said, we haven’t been idle on Tender proper either.

Elastic Search

We recently spent a few days optimizing our search cluster. There were a couple of low hanging fruits and we got a 100% improvement: not too shabby!

Graph of the performance improvement

Assignments

Some teams are really busy, with dozens of agents buzzing and humming at any time. If that’s your case, distributing workload can be challenging. So today, we are introducing a small change that should bring happiness and joy to those hard workers: in the assignment dropdown, next to the name, you will now see the number of discussions open assigned to this supporter. This way, if Tom is under water, but Marry ahead of schedule, you know who to give the next one too. It is a small improvement, but it should definitely help.

Screenshot of the new dropdown

Spam

We also deployed a couple of improvements to our spam engine that should reduce false positives.

While we are on the subject of spam, I have a couple of public announcements:

  • If you have issues with spam or false positives on your site, please let us know. We have full control over our spam engine and we can usually improve things for you. Sometimes it’s just bad configuration, but sometimes a rule needs to be tweaked, and we can do that too. Many people contact us after months of suffering, when we could have fixed their problem the day it happened.

  • If your spam folder has grown to unspeakable proportions, we can empty it for you. The best way to keep spam in check and customer happy is an empty spam folder. We recommend that you go through it regularly (every day if you can), restoring false positives and deleting the rest. By keeping it empty, you can make sure you never miss anything. You can also get a better idea as to whether the spam engine is doing its job correctly or not, which leads back to point 1. So if your spam folder is not empty, contact us: we’ll empty it for you and you can start fresh and renewed. You know 2014 is coming: let this be your new year resolution! (Well, one resolution at least. Enjoying life a little more should be on top of your list).

Happy Holidays.

Ho Ho Ho

Episode #428 – December 20th, 2013

Posted 4 months back at Ruby5

Today only! Some Rails 4.1 tidbits, dependency injection drama, a Rails engine for the Dashing framework, and some free tapas from Avdi.

Listen to this episode on Ruby5

This episode is sponsored by New Relic
New Relic is _the_ all-in-one web performance analytics product. It lets you manage and monitor web application performance, from the browser down to the line of code. With Real User Monitoring, New Relic users can see browser response times by geographical location of the user, or by browser type.

Rails 4.1 beta
Rails 4.1 beta is out! A nice summary here: http://coherence.io/blog/2013/12/17/whats-new-in-rails-4-1.html

secrets.yml
Rails 4.1 has a secret! It’s a new config file called secrets.yml intended to hold your sensitive bits out of the gits.

Another post about Dependency Injection
Piotr Solnica adds his own thoughts on DI, and uses a short example of how he practices DI in his own code.

Yield or Enumerate
Avdi Grimm is giving it away! He’s posted another of his brief Ruby Tapas episodes on his site for free. Yield or Enumerate will show you how many of Ruby’s iteration methods return an Enumerator object if not given a block and what you can do with it.

dashing-rails
Dash away, dash away, dash away all! Take a peek at dashing-rails, a Rails Engine for Shopify’s Dashing framework. So handsome.

Clearer Conditionals using De Morgan's Laws

Posted 4 months back at GIANT ROBOTS SMASHING INTO OTHER GIANT ROBOTS - Home

Ever seen a hard-to-parse conditional like this?

def allow_access_to_site?
  ! (signed_out? && untrusted_ip?)
end

Let's use De Morgan's Laws to clean it up and see who actually has access to our site.

De Morgan's Laws

Whoa, who's this De Morgan guy? Augustus De Morgan, in addition to looking like a 19th-century John C. Reilly, formulated two important rules of logical inference. You can check out the formal definition on the Wikipedia page, but here they are in Ruby code:

# One
!(a && b) == !a || !b
# Two
!(a || b) == !a && !b

Well hey, it looks like we can use these on our gnarly conditional above. Let's try it.

Law-abiding Ruby code

Recall that the original conditional was ! (signed_out? && untrusted_ip?). Let's use the second law and puzzle it out.

# Original
! (signed_out? && untrusted_ip?)
# Conversion using second law. I've added parentheses for clarity.
# a = signed_out?
# b = untrusted_ip?
(! signed_out?) || (! untrusted_ip?)

Here I notice that ! signed_out? and ! untrusted_ip? are double negatives: not signed out, not untrusted IP. Now what they're really trying to say is: signed in, trusted IP. Let's simplify further, using better method names.

# Simplify a: (! signed_out?) == signed_in?
(signed_in?) || (! untrusted_ip?)
# Simplify b: !(! untrusted_ip?) == trusted_ip?
(signed_in?) || (trusted_ip?)
# Remove parentheses
signed_in? || trusted_ip?

These methods, signed_in? and trusted_ip?, might exist and they might not. Creating them is part of this refactoring. You might even end up removing the signed_out? and untrusted_ip? methods in favor of these new, positively-named methods.

And that's it. We took a hard-to-parse conditional and made it clearer and easier-to-read using De Morgan's second Law.

Before:

def allow_access_to_site?
  ! (signed_out? && untrusted_ip?)
end

After:

def allow_access_to_site?
  signed_in? || trusted_ip?
end

What's next?

If you found this useful, you might also enjoy: