The daily miscellany

Posted about 7 years back at work.rowanhick.com

The Golden Rule Always isolate your speakers from whatever they're sitting on (be it floor, bookshelf whatever). It's just the golden rule in life. You must do it. Well, I built this new desk**, and completely forgot the rule, I was just so happy to have somewhere for these puppies to sit on. I happened upon some 8mm strips of rubbery packing foam - night and day difference, event at low volumes it makes a difference. I'm not sure the maple was really adding to music Just listening to some Ben Harper and it sounds GORGEOUS. No more annoying little bass resonances. Put on some Vivaldi and we get bone chilling clarity. Nice - that's what studio monitors are about, reproduction (this is a moment of triumph, I bought them 5-6 months ago and haven't really had a chance to *use* them until now). Of course a true audiophile might be a little horrified that sound waves are going to reflect off the desk, but you can't have it all. ** The other golden rule, never try to figure out how Ikea can sell furniture so cheaply. A long way from Paperboy.. Bioshock What a busy evening, picked up stuff for my significant other, went out for dinner, did some coding, managed to give the cats some attention, and last but not least sneaked in a few minutes of playing the Bioshock Demo. Holy sh-t. Games have come a very long way from Paperboy, and even further from "LOAD FROM A:\" "RUN" off a tape drive (yes I'm starting to feel old). What keeps me young is firing up the flat panel, powering up the receiver, turning on the XBox 360 and having a bit of good ole fun. All I've got to say with Bioshock, is I'll be very suprised if it doesn't come in a close 2nd, if not 1st with Gears of War for sales this year. It is absolutely spectacular (and this is just the demo). I was blown away from 15mins of play time, the water effects are very nice indeed. An eerie atmosphere pervades each section of gameplay. Well done. There goes my productivity for a few days...

Four Things: Ruby Hoedown

Posted about 7 years back at Alloy Code - Home

So I'm just now able to take a few minutes to reflect on my experiences at the past weekend's Ruby Hoedown in Raleigh, NC. Nathaniel Talbott and Jeremy McAnally did an outstanding job putting together the first in what I hope will be a long series of Ruby-focused events in the southeastern United States.

Rather than recap the entire conference (which has been done elsewhere by more prolific writers than yours truly), I would rather focus on four things I picked up over the 2-day conference.

Stubbing out Ruby's Time class

The pre-conference kicked off with a 2.5 hour workshop run by Bruce Tate, Marcel Molina Jr. and Chad Fowler. Chad, in particular, has been very vocal lately about the importance of developers (and particularly Ruby/Rails developers) giving back to the world. The workshop itself was a fundraising vehicle for the Food Bank of Eastern & Central North Carolina, with 100% of the workshop registration fee being donated. This is an interesting new model, and I'm curious to see where it's going to lead in the future.

So, one of the things I encounter frequently in my classes is needing to test time-specific behavior. Previously, I had been instantiating a plethora of instance variables to check edge cases for Date and Time processing in my tests. My first "a-ha!" moment of the weekend came during Bruce Tate's presentation, when he put up the following code snippet:


    Time.stubs(now).returns(groundhog_day_at_noon)

This is going to make a world of difference for me, because now I can create insular, repeatable tests regardless of the time of day at which the test is run.

VOIP has become accessible

Jay Phillips became a new hero of mine when he showed off his Ruby-based Dialplan using the Adhearsion library. After showing us a multi-thousand line Dialplan written in an obscure, telecom dialect, he switched back to his slides and showed us the equivalent code, written in near-human readable Ruby. VOIP has been one of those technologies that I wasn't really paying much attention to, but now my mind is spinning out a whole host of new telephony-enabled website applications faster than I can manage to write them down.

Driving return traffic to a website is a huge deal. Once they sign up for the account, what's the hook that keeps them coming back over and over again? What if Your Garage Online could CALL you to tell you your car was due for service, instead of simply sending you an email? Powerful stuff.

It's time to crack open those dusty C books again

Jared Richardson (who shares my agony of constantly being called "Jerry") gave an energetic talk on using Ruby's C extensions to speed up application performance. I had heard Jared speak back in January at the Rails Edge conference in Reston, VA, on Rails performance and how the search engine Cha-Cha was using Rails in an optimized fashion. This time, in between launching small boxes of candy at unsuspecting audience members, he gave us a great overview of the various methods to include C code to handle low-level, no-magic-required operations directly with our Ruby code. I actually coded along with his his presentation, and wrote more C code in 45 minutes than I'd written in the past 10 years combined.

In particular, RubyInline looks to be an incredibly useful library to write C code directly inside a Ruby class, and have it generated as needed. One caveat, which is particularly visible in the Rails spectrum, is that RubyInline's performance gains are only really visible AFTER the class has been run once and the C code compiled and cached. So, for Rails development, classes which use RubyInline should be housed in #{RAILS_ROOT}/lib, rather than directly in a model or controller, to avoid the code being recompiled on EVERY load.

The Germ of an Idea

This isn't directly something I learned at Ruby Hoedown, but more a general idea that I'd be curious to explore in the future. I've recently picked up Joe Armstrong's Programming Erlang from the Pragmatic Bookshelf. One of the most oft-repeated themes of the conference was concerns about Ruby's performance. If we eschew the idea that everything related to Ruby has to be written in Ruby (and I can thank Jared Richardson for this new mind-set), what if a version of the Ruby VM could be written in Erlang, to execute high performance Ruby in a multi-core environment?

I was sitting next to Brenton Leanhardt on Saturday, who happens to be another member of ErloungeRDU. We talked the idea over for a few minutes, and concluded that while neither of us had any solid idea how to write a Virtual Machine, the idea itself could be worth exploring. This may be a subject I return to in the future...

Four Things: Ruby Hoedown

Posted about 7 years back at Alloy Code - Home

So I'm just now able to take a few minutes to reflect on my experiences at the past weekend's Ruby Hoedown in Raleigh, NC. Nathaniel Talbott and Jeremy McAnally did an outstanding job putting together the first in what I hope will be a long series of Ruby-focused events in the southeastern United States.

Rather than recap the entire conference (which has been done elsewhere by more prolific writers than yours truly), I would rather focus on four things I picked up over the 2-day conference.

Stubbing out Ruby's Time class

The pre-conference kicked off with a 2.5 hour workshop run by Bruce Tate, Marcel Molina Jr. and Chad Fowler. Chad, in particular, has been very vocal lately about the importance of developers (and particularly Ruby/Rails developers) giving back to the world. The workshop itself was a fundraising vehicle for the Food Bank of Eastern & Central North Carolina, with 100% of the workshop registration fee being donated. This is an interesting new model, and I'm curious to see where it's going to lead in the future.

So, one of the things I encounter frequently in my classes is needing to test time-specific behavior. Previously, I had been instantiating a plethora of instance variables to check edge cases for Date and Time processing in my tests. My first "a-ha!" moment of the weekend came during Bruce Tate's presentation, when he put up the following code snippet:


    Time.stubs(now).returns(groundhog_day_at_noon)

This is going to make a world of difference for me, because now I can create insular, repeatable tests regardless of the time of day at which the test is run.

VOIP has become accessible

Jay Phillips became a new hero of mine when he showed off his Ruby-based Dialplan using the Adhearsion library. After showing us a multi-thousand line Dialplan written in an obscure, telecom dialect, he switched back to his slides and showed us the equivalent code, written in near-human readable Ruby. VOIP has been one of those technologies that I wasn't really paying much attention to, but now my mind is spinning out a whole host of new telephony-enabled website applications faster than I can manage to write them down.

Driving return traffic to a website is a huge deal. Once they sign up for the account, what's the hook that keeps them coming back over and over again? What if Your Garage Online could CALL you to tell you your car was due for service, instead of simply sending you an email? Powerful stuff.

It's time to crack open those dusty C books again

Jared Richardson (who shares my agony of constantly being called "Jerry") gave an energetic talk on using Ruby's C extensions to speed up application performance. I had heard Jared speak back in January at the Rails Edge conference in Reston, VA, on Rails performance and how the search engine Cha-Cha was using Rails in an optimized fashion. This time, in between launching small boxes of candy at unsuspecting audience members, he gave us a great overview of the various methods to include C code to handle low-level, no-magic-required operations directly with our Ruby code. I actually coded along with his his presentation, and wrote more C code in 45 minutes than I'd written in the past 10 years combined.

In particular, RubyInline looks to be an incredibly useful library to write C code directly inside a Ruby class, and have it generated as needed. One caveat, which is particularly visible in the Rails spectrum, is that RubyInline's performance gains are only really visible AFTER the class has been run once and the C code compiled and cached. So, for Rails development, classes which use RubyInline should be housed in #{RAILS_ROOT}/lib, rather than directly in a model or controller, to avoid the code being recompiled on EVERY load.

The Germ of an Idea

This isn't directly something I learned at Ruby Hoedown, but more a general idea that I'd be curious to explore in the future. I've recently picked up Joe Armstrong's Programming Erlang from the Pragmatic Bookshelf. One of the most oft-repeated themes of the conference was concerns about Ruby's performance. If we eschew the idea that everything related to Ruby has to be written in Ruby (and I can thank Jared Richardson for this new mind-set), what if a version of the Ruby VM could be written in Erlang, to execute high performance Ruby in a multi-core environment?

I was sitting next to Brenton Leanhardt on Saturday, who happens to be another member of ErloungeRDU. We talked the idea over for a few minutes, and concluded that while neither of us had any solid idea how to write a Virtual Machine, the idea itself could be worth exploring. This may be a subject I return to in the future...

XChain is now running on rSpec

Posted about 7 years back at work.rowanhick.com

Whilst I wait to sort out the UI issues I'm building up the specs' for XChain. I'm using RSpec, so it's a lot more 'self documenting' about what you can expect the application to do. Starting on a fresh project, it's immensely enjoyable to get into the spec -> code -> test -> repeat cycle. If you're at all hesitant about getting into them feel free to go browse the source of my specs. I only spent a little time on it last night but already starting to see progress - a lot of the schema is getting cleaned up as a result of the specs. Here's the doc out put so far (which is about a fraction of what the finished specs will look like !). Order when creating a new order - should be able to instantiate - should default to draft status - should be able to prefill an address - should have a default price type matching customer price type - should calculate return 0 dollars - should throw an error that it doesn't have line items - should be able to be edited - should be able to have an order line added Order when finding existing order - should have id = 1 - should have a purchase order number - should have 3 order lines - should have a billing address - should have a customer To get the specs running. 1. Grab a latest copy of xchain from subversion http://xchain.googlecode.com/svn/trunk/. 2. Dump all databases, then create, and run migrations. 3. Install rpsec using instructions from here 4. Run rake rspec 5. You should (at time of writing) get 13 examples with 1 failure (to work on!) By the end of the weekend there should be a healthy amount of specs in there and fair portion of the model structure of orders, pricing, customer management done. Heres a sample chunk of code require File.dirname(__FILE__) + '/../spec_helper' describe Order, "when creating a new order" do fixtures :orders, :order_lines, :customers, :addresses, :addressables, :products it "should be able to instantiate" do @order = Order.new end it "should default to draft status" do @order = Order.new @order.order_status_id.should == 10 end it "should be able to prefill an address" do @customer = Customer.find(1) @order = Order.new @order.should respond_to(:prefill_address) @order.prefill_address(Customer.find(1)) @order.billing_address.should eql(@customer.addresses.default_billing.address) @order.billing_city.should eql(@customer.addresses.default_billing.city) @order.billing_postcode.should eql(@customer.addresses.default_billing.postcode) @order.shipping_address.should eql(@customer.addresses.default_shipping.address) @order.shipping_city.should eql(@customer.addresses.default_shipping.city) @order.shipping_postcode.should eql(@customer.addresses.default_shipping.postcode) end end If you want an introduction on rspec follow your nose to either the rpsec docs (which are fairly self explanatory) or if you're the visual sort the head on over to http://peepcode.com/products/rspec-basics for a screencast (disclaimer, haven't watched it, but judging on Peepcode's previous efforts it should be good).

Get Ready To Rumble

Posted about 7 years back at zerosum dirt(nap) - Home

So like everyone else, you probably have a killer web app idea you’ve been sitting on. Maybe you’ve thought about building it, maybe you’ve even drawn out some diagrams on the back of a napkin, but you just haven’t found the time to execute it. The unfortunate reality is that sometimes you just need an excuse, or a little push, to get something awesome started.

Sound about right? Then this one’s for you. Enter the Rails Rumble, September 8-9, and consider yourself pushed.

Flex vs HTML part 2 .. how big ?

Posted about 7 years back at work.rowanhick.com

Following on from my post yesterday, I'm asking the question to the community, how big can Flex apps get. This is probably the great unknown which makes it hard (on this project) to commit to Flex. I'm not sure if I'm going to get halfway in and strike performance problems or such like (this is just conjecture!). HTML it's easy, it doesn't matter how big the app gets you're always just firing down a single page to the browser - you just have to manage page load times, this (for myself, and I guess anyone getting into Flex) is a little unknown. The questions are 1) Am I crazy trying to develop 1 person ~15hrs a week, given the apps functional list below, over ~6 months 2) Whats a good example app close to this size. 3) What kind of performance impact (if any) is there in adding each new functional area I've gone through the ever growing showcase and I've picked out the nearest one size wise Studio Cloud, but there doesn't seem to be a lot of them that get this 'big'. Where my definition is big is a large number of different functional areas. To give you an idea of what I'm building.. (this list is about 6mths long I guess, rebuilding an existing system which has approx 2/3 of these functional areas). Note that this is all backed by a Rails backend so all of the order calculations, report calculations will be coming from Rails, this just has to pick it up via AMF/HTTPService/WebORB. I'm a full time Rails web team lead so this is definitely cake for me to do the backend. [Public View] - Catalog - Product List - Product View - Application views - Cart - Checkout - Orders - Support [Backoffice view] - Dashboard (upcoming orders, tickets, shipments, forecasts) - Orders - List orders (search/archive/filtering) - Manufacturing sub view - View orders (new/edit/view) - Order Sub View (audit trail, shipping status, invoices, order calculation summary) - Customers - List customers (search/archive/filtering) - View customers (new/edit/view) - Customer Tracking Tickets (list/view/respond) - Agents/Distributors - List - View - View Customers - View Orders - Reports - Reports List - Individual report charts ~5-10 - Individual report list based ~5-10 - Admin - Catalog management - Categories - Products - Price List Management - List - Price matrix update (products vs price category) - System Options I'm sure it's doable, I'm not sure of the workload or performance of this. I'm not overly concerned of the swf file size, what I am concerned with is, will each additional functional area have an impact on overall performance and am I nuts for attempting this as my first large Flex project. I have knocked out a couple of small little Flex projects which weren't too bad, definitely didn't need frameworks for them. Any comments would be appreciated; again all of the source is going to be freely available to everyone by a Mozilla Public License, so anyone who's thinking of going down this path and wants to contribute please feel free to leave a comment (big *hint* if there's any Flex UI designers out there - I would absolutely love to have a clean skin like this one...http://coenraets.org/blog/2007/07/new-version-of-salesbuilder-flex-air-application/ world fame could be yours ;) )

Ruby, Rails, Micropayments, and Amazon FPS

Posted about 7 years back at zerosum dirt(nap) - Home

Last week Amazon unveiled a beta of their new Flexible Payments Service, a potential Paypal-killer and Google Checkout-killer, among other things. At first glance, FPS seems to be everything that Paypal isn’t: well designed, API-centric, and built with developers in mind.

While everyone is busy barking about how Amazon is going to go head to head with Google Checkout and Paypal for purchases, they seem to be missing the more interesting development here. I’m talking about something that neither Paypal or Google are even attempting to do, as far as I know: micropayments.

Ty, Steve, and I spent some serious time putting together a few different iterations of an idea/prototype earlier this year, that involves micropayments and the street performer protocol. And some other stuff, too :-). Unfortunately, when it came time to tie it into a payment gateway, it became painfully obvious that what we needed didn’t really exist, so we compromised. And that compromise led to a bunch of problems and eventually what we felt was an unworkable solution. So it’s been in mothballs since. This looks like it could be what we were maybe waiting for.

If you check out Amazon’s rate schedule you can see why I’m excited:

For Amazon Payments balance transfers < $0.05:
20% of the transaction amount, with a minimum fee of $0.0025

OK, OK, so 20% is a hefty fee but not when we’re talking about the alternative being somewhere in the neighborhood of $0.30 USD + 2-3%. So yeah, pfft. This is HUGE. And yes, it does look like these micropayments only work when you’re using Amazon payments (as opposed to using a credit card, etc) but so what? I’m not so sure that’s an obstacle.

Another reason it’s so interesting to me is that they’ve decided to support three languages/platforms in their initial beta rollout: Java, PHP, and yes, my friends, Ruby. Oh, and the sample app? Yep. Rails.

Why aren’t we more excited about this?

Flex vs HTML vs Time

Posted about 7 years back at work.rowanhick.com

The old simple triangle of time vs features vs quality has reared it's ugly head again. After one particularly unproductive weekend I've made the decision to place a hold on the Flex interface for XChain, in favour of a raw HTML interface, sprinkled with AJAX. I personally hate the decision, being enamored with Flex, I don't want to make it, but I look at the forward progress I've made in the past couple of weeks and it's been more or less stationary. I'm not trying to deliver the 'richest' interface possible, I'm going to deliver a working application. What impacts my client most, is that they can get their job done. Granted I am learning Flex, and grappling with the Cairngorm/No Cairngorm decision. I decided to go ahead with Cairngorm, particularly after reading the usual tales of woe 'i decided not to use it, now I've rebuild my app and am using it..'. However holy sh-t do you have to produce a lot of code to get things done, okay there are code generators out there, but still. For a pet project this blows out the window the time required to do the job. Sure if this was a 40hr work week project I'd be writing a completely different post right now. This makes me appreciate the elegant nature of Ruby and Rails™ so much more. It just gets out of your way and lets you get on with producing productive code. In just one hour I can do so much more. Firing up textmate I feel instantly at home. Is this a language familiarity thing, or a complexity problem, or am I using the right tool for the right job, within the given constraints, (probably not). I had a big discussion about it with a colleague at work, and it always boils down to trade-offs and compromises, there's no one magic silver bullet (unfortunately). Rails is fantastic at backend work, and helps some with front end work, Flex has some great time savers, rich interface components like datagrid which are cake to wire up and get working - but then you're spending a tonne of time on event and ui management. Where is the happy median ? Something like a super rich js framework ? What am I sorely missing from flex in going 'back' to HTML ? - Rich components - Consistent styling - Reduced network traffic A random thought floating through my mind is to publish it in AIR, that way I can still use an HTML interface hooked up to my back end, but a few extra features that might be very useful (drag n drop support, offline storage). More to investigate.. Grumpy but moving forward.

mms2r 1.1.4 Released

Posted about 7 years back at Mike Mondragon

mms2r version 1.1.4 has been released!

  1. DESCRIPTION:

MMS2R is a library that decodes the parts of an MMS message to disk while stripping out advertising injected by the cellphone carriers. MMS messages are multipart email and the carriers often inject branding into these messages. Use MMS2R if you want to get at the real user generated content from a MMS without having to deal with the cruft from the carriers.

If MMS2R is not aware of a particular carrier no extra processing is done to the MMS other than decoding and consolidating its media.

Contact the author to add additional carriers to be processed by the library. Suggestions and patches appreciated and welcomed!

Corpus of carriers currently processed by MMS2R:

  • AT&T => mms.att.net
  • AT&T/Cingular => mmode.com
  • Cingular => mms.mycingular.com
  • Cingular => cingularme.com
  • Dobson/Cellular One => mms.dobson.net
  • Helio => mms.myhelio.com
  • Nextel => messaging.nextel.com
  • Sprint => pm.sprint.com
  • Sprint => messaging.sprintpcs.com
  • T-Mobile => tmomail.net
  • Verizon => vzwpix.com
  • Verizon => vtext.com

Changes:

  1. 1.1.4 / 2007-08-07 (Dr. Rockso)
  • AT&T => mms.att.net support (thanks Mike Chen and Dave Myron)
  • get_body returns nil when there is not user text (sorry Will!)

mms2r 1.1.4 Released

Posted about 7 years back at Mike Mondragon

mms2r version 1.1.4 has been released!

  1. DESCRIPTION:

MMS2R is a library that decodes the parts of an MMS message to disk while
stripping out advertising injected by the cellphone carriers. MMS messages are
multipart email and the carriers often inject branding into these messages. Use
MMS2R if you want to get at the real user generated content from a MMS without
having to deal with the cruft from the carriers.

If MMS2R is not aware of a particular carrier no extra processing is done
to the MMS other than decoding and consolidating its media.

Contact the author to add additional carriers to be processed by the
library. Suggestions and patches appreciated and welcomed!

Corpus of carriers currently processed by MMS2R:

  • AT&T => mms.att.net
  • AT&T/Cingular => mmode.com
  • Cingular => mms.mycingular.com
  • Cingular => cingularme.com
  • Dobson/Cellular One => mms.dobson.net
  • Helio => mms.myhelio.com
  • Nextel => messaging.nextel.com
  • Sprint => pm.sprint.com
  • Sprint => messaging.sprintpcs.com
  • T-Mobile => tmomail.net
  • Verizon => vzwpix.com
  • Verizon => vtext.com

Changes:

  1. 1.1.4 / 2007-08-07 (Dr. Rockso)
  • AT&T => mms.att.net support (thanks Mike Chen and Dave Myron)
  • get_body returns nil when there is not user text (sorry Will!)

Introducing: Your Garage Online

Posted about 7 years back at Alloy Code - Home

Alloy Code is thrilled to announce the launch of a new web application. Your Garage Online can help you manage everything from your vehicle's routine maintenance to those major and minor repairs.

Features

Virtual garage to store your cars, including year, color, make, model, license and vehicle identification number data.
Searchable repair forms to look up any repair or maintenance record, regardless of when or where it was performed.
Comprehensive service history outlines work done by professional shops or in your own back yard.

YourGarageOnline.com was originally conceived as a Rails Day activity, but grew in scope beyond the confines of a single, 24 hour development project. The site was designed, coded and tested over a six week period, as an off-hours personal project and proof of concept application. The finished product serves as a demonstration of the quality and craftsmanship Alloy Code can deliver for our clients.

The site is coded in Ruby on Rails, using RESTful application design and leveraging cutting edge Web 2.0 technology.

Introducing: Your Garage Online

Posted about 7 years back at Alloy Code - Home

Alloy Code is thrilled to announce the launch of a new web application. Your Garage Online can help you manage everything from your vehicle's routine maintenance to those major and minor repairs.

Features

Virtual garage to store your cars, including year, color, make, model, license and vehicle identification number data.
Searchable repair forms to look up any repair or maintenance record, regardless of when or where it was performed.
Comprehensive service history outlines work done by professional shops or in your own back yard.

YourGarageOnline.com was originally conceived as a Rails Day activity, but grew in scope beyond the confines of a single, 24 hour development project. The site was designed, coded and tested over a six week period, as an off-hours personal project and proof of concept application. The finished product serves as a demonstration of the quality and craftsmanship Alloy Code can deliver for our clients.

The site is coded in Ruby on Rails, using RESTful application design and leveraging cutting edge Web 2.0 technology.

FragmentFu - Fun with Fragments

Posted about 7 years back at Revolution On Rails

First, please read the Advanced Rails Caching.. On the Edge for a primer on ESI, caching, and fragments.
To get started developing with ESI, I'm going to walk through a simple tutorial building an ESI enabled Rails application.
Get Mongrel-ESI
Find the latest mongrel ESI here: http://code.google.com/p/mongrel-esi/downloads/list


wget http://mongrel-esi.googlecode.com/files/mongrel-esi-0.0.5.gem
(Does anyone have a better solution of ruby projects hosted on Google Code? )

Install Mongrel ESI

sudo gem install mongrel-esi-0.0.5.gem

Create a new rails application (Currently using rails 1.2.3)

rails fragment_fu_demo

create
create app/controllers
create app/helpers
create app/models
.....

cd fragment_fu_demo


Create a home page for your application


Delete the public/index.html page

rm public/index.html

Edit the config/routes.rb and uncomment the following line:

map.connect '', :controller => "welcome"

Create a welcome controller

./script/generate controller welcome index


Install the FragmentFu plugin
   
./script/plugin install http://mongrel-esi.googlecode.com/svn/trunk/plugin/fragment_fu


Start your application

 
./script/server

visit http://localhost:3000 in your browser. You should see a "Welcome#index" page.

Start MongrelEsi
From the command line, run:

mongrel_esi start
** Starting Mongrel listening at 0.0.0.0:2000

visit http://localhost:2000 in your browser. You should see a "Welcome#index" page.

Create a fragment to cache
Edit the app/controllers/welcome_controller.rb
Add an action called now
 
def now
render :text => "#{Time.now} is #{Time.now.usec} nano-seconds"
end

Edit the app/views/welcome/index.html.erb and replace with the following:


<h1>Welcome</h1>
<p><%= render :esi => "/welcome/now" %></p>


Page without ESI parsing
visit http://localhost:3000 and you should see: "Time:" in the browser. If you view the source, you'll see


<h1>Welcome</h1>
<p><esi:include src="/welcome/now" max-age="0"/></p>


This is the simplest of esi tags, and the default max-age is 0, which means do not cache.

Page with ESI Parsing

visit http://localhost:2000 and you'll see:

Welcome
Date: Mon Aug 06 00:00:00 -0400 2007 which is 62972ns


Caching modules
Edit the index.html.erb and replace with the following:


<h1>Welcome</h1>
<p><%= render :esi => "/welcome/now" %></p>
<h2>What time is it again?</h2>
<p><%= render :esi => "/welcome/now"%></p>


If you refresh http://localhost:2000, you'll notice they nanoseconds are different

Lets add a ttl to first call


<h1>Welcome</h1>
<p><%= render :esi => "/welcome/now", :ttl => 45.seconds %></p>
<h2>What time is it again?</h2>
<p><%= render :esi => "/welcome/now"%></p>


If you refresh http://localhost:2000, you'll notice they are now the same. Refresh again in the next 45 seconds, and it will not change.


The next tutorial will cover a small TODO application, with inline invalidation and exception handling. Coming Soon!

FragmentFu - Fun with Fragments

Posted about 7 years back at Revolution On Rails

First, please read the Advanced Rails Caching.. On the Edge for a primer on ESI, caching, and fragments.
To get started developing with ESI, I'm going to walk through a simple tutorial building an ESI enabled Rails application.
Get Mongrel-ESI
Find the latest mongrel ESI here: http://code.google.com/p/mongrel-esi/downloads/list


wget http://mongrel-esi.googlecode.com/files/mongrel-esi-0.0.5.gem
(Does anyone have a better solution of ruby projects hosted on Google Code? )

Install Mongrel ESI

sudo gem install mongrel-esi-0.0.5.gem

Create a new rails application (Currently using rails 1.2.3)

rails fragment_fu_demo

create
create app/controllers
create app/helpers
create app/models
.....

cd fragment_fu_demo


Create a home page for your application


Delete the public/index.html page

rm public/index.html

Edit the config/routes.rb and uncomment the following line:

map.connect '', :controller => "welcome"

Create a welcome controller

./script/generate controller welcome index


Install the FragmentFu plugin
   
./script/plugin install http://mongrel-esi.googlecode.com/svn/trunk/plugin/fragment_fu


Start your application

 
./script/server

visit http://localhost:3000 in your browser. You should see a "Welcome#index" page.

Start MongrelEsi
From the command line, run:

mongrel_esi start
** Starting Mongrel listening at 0.0.0.0:2000

visit http://localhost:2000 in your browser. You should see a "Welcome#index" page.

Create a fragment to cache
Edit the app/controllers/welcome_controller.rb
Add an action called now
 
def now
render :text => "#{Time.now} is #{Time.now.usec} nano-seconds"
end

Edit the app/views/welcome/index.html.erb and replace with the following:


<h1>Welcome</h1>
<p><%= render :esi => "/welcome/now" %></p>


Page without ESI parsing
visit http://localhost:3000 and you should see: "Time:" in the browser. If you view the source, you'll see


<h1>Welcome</h1>
<p><esi:include src="/welcome/now" max-age="0"/></p>


This is the simplest of esi tags, and the default max-age is 0, which means do not cache.

Page with ESI Parsing

visit http://localhost:2000 and you'll see:

Welcome
Date: Mon Aug 06 00:00:00 -0400 2007 which is 62972ns


Caching modules
Edit the index.html.erb and replace with the following:


<h1>Welcome</h1>
<p><%= render :esi => "/welcome/now" %></p>
<h2>What time is it again?</h2>
<p><%= render :esi => "/welcome/now"%></p>


If you refresh http://localhost:2000, you'll notice they nanoseconds are different

Lets add a ttl to first call


<h1>Welcome</h1>
<p><%= render :esi => "/welcome/now", :ttl => 45.seconds %></p>
<h2>What time is it again?</h2>
<p><%= render :esi => "/welcome/now"%></p>


If you refresh http://localhost:2000, you'll notice they are now the same. Refresh again in the next 45 seconds, and it will not change.


The next tutorial will cover a small TODO application, with inline invalidation and exception handling. Coming Soon!

Advanced Rails Caching.. on the Edge

Posted about 7 years back at Revolution On Rails

When trying to scale a portal to millions of users with complex personalization, we had to rethink the application design. We needed partial page cacheability, and we wanted that close to the Edge... and recently the idea has been catching on. First with Components Are the New Black, and the idea of Nginx, SSI, and Memcache, what I'd like to briefly describe below is a partial page caching strategy that has worked well for the past year.

First, we use a technology called ESI. ESI stands for Edge Side Includes, which is a simple markup language for describing dynamic assembly of applications. At the core, its similar to SSI, but its a more versatile spec that has been accepted/implemented by a half dozen cache servers, both open source (Squid, Mongrel-ESI) and commercial (Oracle Web Cache, Akamai, among others). It also includes an invalidation protocol, exception handling on the edge, and a few other features.

The ESI W3C specification has been out for 6+ years, so these ideas are not new, but ESI is a diamond in the rough. No one seems to be using it.

A simple example of ESI in your rails application is including a common header. If the header is static, using a shared layout, or rendering a shared partial across applications, could be sufficient. But if you use the common idiom of a sign-in bar, like "Welcome Bob - Signout", on a page you want to fully cache, or a common header you want to share across multiple applications, then you may consider another approach.

If you leverage ESI, you can put the below in your application layout.

<div id="header">
<esi:include src="/header" max-age="300"/>
</div>


The <esi:include> tag will inject the response from /header into the page and cache that fragment for 300 seconds. You can also add a variety of options, request header parameters (which can be used to personalize the request) along with a dozen other optional parameters, some of which I will outline below. This is simple SSI (Server Side Includes) .

But if we take it a step further, lets look at how we can apply it to a site like Twitter.

Twitter


If you look at the image above, the majority of the page can be fully cached. I've outlined two green boxes, which appear to be dynamic content, which can have different TTLs. Using ESI, your markup could be:

<div id="latest">
<esi:include src="/latest" max-age="5"/>
</div>
...
<esi:include src="/featured" max-age="3600"/>


The ESI enabled cache server would parse this markup, make 2 separate HTTP requests (/latest, and /featured) and cache those with their corresponding TTLs. You can furthermore cache the wrapping template with a Surrogate-Control header, which tells the cache server to keep a cached copy of the template. A request to this page 4 seconds later would make 0 requests to your rails infrastructure. 8 seconds later would only hit the /latest, returning the rest of the page from cache. You can also envision a application pool of servers just to handle /latest, but I'll get into sharding applications via ESI in a later article.


Exception Handling

If you take this a step further, you can also define timeouts and exception behavior. As defined below, If the /latest request takes more than 1s, The cache server will give up, and retrieve the static snippet defined in the except block from your static web host.

<esi:try>
<esi:attempt>
<esi:include src="/latest" max-age="5" timeout="1"/>
</esi:attempt>
<esi:except>
<esi:include src="http://static.foo.com/latest" max-age="5" timeout="1"/>
</esi:except>
</esi:try>


We call these "Sorry" modules, because they often say "Sorry, this feature is having trouble", but the rest of the page may surface meaningful content. Even better, write out more meaningful content to disk once a day and serve that from Apache.


Invalidation


The other benefit of ESI is Invalidation support. There are various mechanisms to invalidate content, but my favorite is Inline Invalidation. Consider the common rails idiom of updating data. You post to a controller which redirects to a view of that data. Since the HTTP redirect (301) bubbles all the way back to the browser, any content in the body of the redirect can be parsed by the ESI server. Therefore you can put the invalidation xml is the redirect response body and not conditionally dirty your view logic.

<esi:invalidate>
<?xml version="1.0"?>
<!DOCTYPE INVALIDATION SYSTEM "internal:///WCSinvalidation.dtd">
<INVALIDATION VERSION="WCS-1.1">
<OBJECT>
<BASICSELECTOR URI="/foo/bar/baz"/>
<ACTION REMOVALTTL="0"/>
</OBJECT>
</INVALIDATION>
</esi:invalidate>


The above is a simple single URL example, but the specification supports regex, which would work well for restful resources (e.g. /people/#{person.id}/*), among other things.


FragmentFu


FragmentFu is a plugin that enables ESI support in rails applications. At this point it is an extraction from our internal plugin for modules, caching, and ESI. Its in active extraction/development, but I wanted to involve the community in gathering ideas, suggestions and comments.


Some sample snippets:
<% render :esi => widget_url(1) %>

<%= render :esi => latest_url, :ttl => 5.seconds, :timeout => 1.second %>

<%= render :esi => featured_url, :ttl => 10.hours, :except => '/some/static/path' %>


We've also toyed around with the idea of fragment respond_to's. Since ESI supports adding request header parameters, we can mimic Prototype's
X-Requested-With behavior to implement:

def latest
...
respond_to |wants| do
wants.html { do something }
wants.fragment { do something }
end
end



How to get started


Mongrel-ESI is a great starting place. It was built in-house and released a few months ago on Google Code. It supports a large subset of ESI, ESI Invalidation, and its a great tool for developing applications that utilize ESI. You can grab Mongrel-ESI, download the Oracle Web Cache Standalone or even Squid 3.0... and when you're site gets really big, its time to give Akamai a call. :)