The iNK blot

Simple Design

2011-12-04T00:00:00-08:00

Yesterday I attended the Global Day of Code Retreat, 2011 in Sydney. We were incredibly privileged to have the creator of Code Retreat, Corey Haines facilitating.

I found the experience to be really very profound and recommend it without reserve to developers of all levels.

Here are some of the things I took from the day:

My C++/RUP based thinking of OO really needs a makeover
I thought I was practising Test Driven Development (TDD) but I now see how narrow my understanding of TDD was. The evil coder or Mute with Find the Loophole session of the retreat was simultaneously hilarious, frustrating and enlightening on how well tests actually serve as a specification for code
The principles of Simple Design (attributed to the best of my knowledge to Kent Beck and Ron Jeffries in the Extreme Programming book)

I’ve come across each of the 4 principles of simple design but when I heard Corey talk through them in sequence, it really made a difference to the way I understand them. I know that this understanding will change as I practice with them and I will not remember them as well when I am in the thick of the work day, so I’m writing them down here as a note to self:

Runs all the tests
Expresses every idea that we need to express
Says everything once and only once
Has no superfluous parts

I look forward to practicing the principles I learned at code retreat. I have no doubt that I have had a step change in my perspective as a programmer - more power to Corey’s efforts at coderetreat.org I say!

A Top Down Operator Precedence parser for algebraic expressions

2011-10-07T00:00:00-07:00

In my previous post I wrote about a polar curve grapher and its use of Kevin Mehall’s tdop_math.js to parse a user input algebraic expression into a javascript eval(uable) string. I ended up spending some time with tdop_math.js - you can find the source here - and I fell into the magical rabbit hole of Top Down Operator Precedence (TDOP) parsers. This post describes how I ended up writing a TDOP parser for algebraic expressions in Ruby.

Some facts:

Vaughan Pratt first wrote about Top Down Operator Precedence parsers in 1973
Douglas Crockford wrote about them in Beautiful Code, and in an article that includes an implementation of a parser for a Simplified Javascript - he went on to use this in JSlint

Overview of TDOP parsers

In a nutshell, Top Down Operator Precedence parsers are similar to Recursive Descent parsers, but, as claimed by Vaughan Pratt, simpler to understand, implement, and providing of better performance. Borrowing from this excellent article by Eli Bendersky, the main features of a TDOP parser are:

A binding power mechanism to handle precedence levels - e.g. the binding power of multiplication '*' is higher than that of addition '+' in arithmetic
A means of implementing different functionality of tokens depending on their position relative to their neighbors: prefix or infix. For example, the minus '-' operator is behaving as a prefix operator in -2 but in its infix form, it is subtraction 4 - 2.
As opposed to classic Recursive Descent, where semantic actions are associated with grammar rules (BNF), TDOP associates them with tokens. Tokens have a binding power, and know how to handle themselves in infix and/or in prefix situations. Expressions are then evaluated recursively by evaluating them at each ‘precedence’ level and returning the result to its caller.

Compilers are difficult to understand at the best of times and most articles I have come across on TDOP parsers are by very bright people finishing (or pausing) on their journey to get to know Pratt’s work. At my current stage of comfort with TDOP parsers, I cannot come close to explaining the parser any better than Eli Bendersky.

Eli’s article, elaborates on the features described above and explains the parser by setting one up to handle simple arithmetic. He then works through a simple example that illustrates it in action as it parses the expression 3 + 1 * 2 * 4 + 5.

For further detail on TDOP parsers, I refer the interested reader to this and Douglas Crockford’s article referenced above.

TDOP and Ruby

As I was working on the polar curve grapher I got curious about implementing tdop_math.js in Ruby. I studied the source of tdop_math.js, and its tokeniser tokens.js, and felt that I had a decent understanding of its structure and operation (improving my Javascript skills along the way!). However, a like-for-like implementation in Ruby seemed out of the question. Well, it seemed like a hard proposition that would involve first developing new metaprogramming skills in Ruby!

And then I ran into Glen Vandenburg’s project radish, which is now called smithereen.

Smithereen

Smithereen is a result of Glen Vandenburg’s meditation on TDOP parsers, during which he decided to write an implementation in Ruby. He has implemented a library for building Top Down Operator Precedence parsers, using Ruby idioms that were completely unfamiliar to me, but that provided an excellent opportunity to learn. As Glen says, he has tried to provide a well-factored set of tools to help with the core TDOP algorithm and several related problems, and so I decided to fork Smithereen with the intention of helping with its development, and, using it to implement tdop_math in Ruby.

Smithereen comes with two samples, an arithmetic parser, and the Simplified Javascript parser. I found it quite difficult to get started with Smithereen and asked Glen a question or two. To my delight, he helped me get started and I found this to be so inspiring that I made rapid progress towards implementing a simple scientific calculator. This is available on my fork of Smithereen as equation_parser.rb

equation_parser.rb will, for instance, take strings such as sin(cos(2pi)) and return 0.84147

By way of explanation, it has hardcoded hashes that declare functions and constants, and these are tokenised as names and given a high binding power.

    fns = {
      'sin'  => Proc.new { |x| Math.sin(x) },
      'cos'  => Proc.new { |x| Math.cos(x) },
      'tan'  => Proc.new { |x| Math.tan(x) },
      'sqrt' => Proc.new { |x| Math.sqrt(x) },
      'exp'  => Proc.new { |x| Math.exp(x) }
      }

    CONSTANTS = {
      'pi' => Math::PI,
      'e'  => Math::E
      }

The infix method on the token ( checks for the presence of a function or a constant (essentially a key in the above hashes), and looks up the corresponding value for the definition of the function. Smithereen takes care of the rest. Well, that is, after tokens for the usual arithmetic operators are defined, with infix and prefix methods as described above.

tdop_math.rb

Completing my goal to create a Ruby equivalent to tdop_math.js, I’ve taken the work in the example equation_parser.rb and used it to create a Ruby gem called tdop_math. This takes an algebraic expression, parses it, and returns a string of Ruby that represents the expression.

Because the tdop_math gem depends on smithereen (which has yet to be released), it is not available as a built gem.

As far as I know, if you want to install this, you are going to need to pull down my repo and include '<path-to-tdop_math.rb>' manually. Once you have, you use it as follows:

    u = "sin(cos(2x))"  ## or some user input algebraic expression
    t = TDOPMath::Parser.new(u)
    f = t.parse   ## f == "Math.sin(Math.cos(2 * x))"

tdop_math reserves symbols x,y and t as variables. It also provides a basic list of functions, namely, sin, cos, tan, sqrt and exp. If you need to provide an alternate list of variables and functions, you can do so as follows:

    u = "baz(foo(2a))"  ## or some user input algrebraic expression
    t = TDOPMath::Parser.new(u,
                             :vars => ['a'],
                             :functions => {
                                'foo' => 'Foo::foo',
                                'baz' => 'Baz::baz'
                              })
    f = t.parse   ## f == "Baz::baz(Foo::foo(2 * a))"

You can use tdop_math wherever you need a user to input an algebraic expression, comprising of a known set of functions and variables. This could find use in apps in maths education or science/data analytics. I hope the community finds it useful, although my motivation was mainly to build it as an exercise, along my path to becoming a better programmer. I’ve made a gem!

A polar curve grapher

2011-10-02T00:00:00-07:00

Polar coordinates and polar functions are remarkably beautiful to people who get mathematics. I think that their graphs are beautiful to most people. Ever since I first met them, I’ve tried to draw them on different platforms and using different languages. Wanting to know more about canvas and javascript, I thought that it would be interesting to implement a polar curve grapher using these.

A quick reminder - the polar coordinates ( r, t ) of a point are expressed in terms of the distance r of the point from the origin and the angle t that the line connecting the point to the origin makes with the x-axis. Also, the way you get from a set of polar coordinates (r, t) to cartesian coordinates (x, y) is:

    x = r * cos(t), y = r * sin(t)

Background

Earlier on this year I had asked Gabe Hollombe for advice on things I could do to become a better programmer. Around the time of the conversation, I was particularly aware of my lack of knowledge of javascript and so, we chose my polar-curve-grapher as a project for Gabe to mentor me through.

I started off by checking out the landscape of javascript based graphing tools and apps and came across the following:

Kevin Mehall’s EquationExplorer
aanthony’s graph.tk

Given my relative lack of experience with javascript, I found both apps difficult to understand. So, I tried to figure out how to do it by myself. I looked for javascript libraries I could use to simply plot a bunch of points and found a lot of stuff for charts but relatively few that would just take an array of points and plot them. Here are some that fit the bill:

jQplot
Keith Lawler’s raphael.graphing.js

I played around with jqplot and a hard-coded function r = f(t) to get some quick results. Essentially, I created an array of points to be plotted using the following direct approach:

    points_to_be_plotted = []
    for t in (0 ... 6*PI)
    points_to_be_plotted.push [ f(t)*cos(t), f(t)*sin(t) ]

    jqPlot( point_to_be_plotted )

And this is what jQplot did for me:

This gave me a quick proof of concept and, the project had a life! I now needed to figure out how to get user input, process it, and then plot it.

Parsing user input

Getting user input turned out to be straightforward - use a form in the page with a submit button and respond to the button click by picking up the input in javascript as a string (say) "input_fn". Use eval("input_fn(t)") to turn this into a javascript function and follow the psuedocode above to generate an array of points to be plotted (see the source of the project for the detail - more on this below).

But what if you want the user to enter a function in plain, pedestrian maths (instead of the syntax required by javascript) - e.g. 2sin(x) instead of 2 * Math.sin(x)

Kevin’s EquationExplorer did this so I set out to understand how. And this lead me to the world of Top-Down-Operator-Precedence (TDOP) parsers.

I will write more about TDOP parsers in my next post but to complete the story on the polar curve grapher, these were the ingredients that went into the grapher that you can find here.

Parsing: Kevin Mehall’s tdop_math.js parser (that requires tokens.js, a lexer written by Douglas Crockford himself)
Graphing: Keith Lawler’s graphing library that is based on raphael.js and called, raphael.graphing.js

And thanks to GitHub pages, you can see it in action here: http://novemberkilo.com/polar-curve-grapher

At the time of posting this I still needed to apply a coat or two of polish - I’m not doing any graceful error handling (so try not to divide by zero!) and I’m only winding around the origin three times. You should be able to play around though and get some pretty graphs.

Try r = sin(3t) to get started with a propeller :)

Acknowledgements

I had a lot of fun putting this project together and owe a debt of gratitude to Gabe Hollombe for encouraging me to take this on, and for answering questions and providing guidance along the way. Also, thanks to Keith Lawler and Kevin Mehall for doing the heavy lifting - this project turned out to mainly involve stitching together their libraries.

A simple jquery scroller

2011-09-24T00:00:00-07:00

Recently we implemented a variation on a scroller. We needed to flash up a set of snippets, one by one, in the same text area of a page. Here’s a demo of it in action. You can view the source of the demo but here’s a quick, plain-English explanation of how it works.

The markup

In the page, the snippets are list elements li in a section. You can make them definition elements or whatever you like. The section has an id so we can find it from javascript - I’ve called it snippets in the example.

The CSS

Pretty much the only style elements that are important to the correct operation of the scroller is display: none on the list elements. This ensures that all list elements start out being invisible. Having overflow: hidden on the section makes it easier to deal with any stuttering in the javascript loop shown below.

The javascript

We’re using jQuery so make sure you’ve got that loaded. And here’s the javascript:

    $().ready(function(){
      var loopSnippets = function() {
        var snippets = $("li", $("#snippets"));

        snippets.each( function(index, element) {
          $(element).delay( index*5000 ).fadeIn(100).delay(4500).fadeOut(200);
          if (index == snippets.length-1) { setTimeout( loopSnippets, index*5000 + 5000 ) }
        });
      };
      loopSnippets();
    });

So this is where all the action is of course. The variable snippets collects all the list items and then we iterate over this collection, showing each list item and then hiding them again (using fadeIn and fadeOut).

What you have to wrap your head around is that all of the code gets executed in one shot, so to get items to display in sequence and to loop around again, you have to show-and-hide each item at different times.

The trick is to use delay and the index of the item in the array to calculate the amount of time we need to wait before kicking off the show-and-hide routine for the item.

Finally, to loop back and start over at the top, we use setTimeout to kick off the loopSnippets function again. Again, use the length of snippets and the amount of time used up in showing and hiding each element to figure out how long you have to wait before setting off loopSnippets again.

Keep in mind that the above code works irrespective of the length of the list of snippets - which is of course what you want if the snippets are dynamically loaded.

In summary

Use delay and the index of elements in an array if you want to operate on each element in sequence. Use setTimeout to repeat a function call at specific time intervals.

Thanks go to Josh Kunzmann and Damien Le Berrigaud for collaborating on this and showing me how to use delay and setTimeout creatively!

Selecting an editor

2011-02-05T00:00:00-08:00

The flame wars on the topic of editors have probably been around since, well, there was more than one editor to choose from! An IDE or an editor is such a fundamental tool for a developer that it is no wonder we end up having very strong views on our editor of choice. What I have never really understood is why we also tend to have strong negative opinions on editors that we choose not to use — live and let live people! This post is about my journey through the process of selecting an editor — I also present a few tips and resources along the way, particularly on Vim (the editor I settled on). I am not in the slightest bit interested in fuelling the aforementioned flame war; if I say something here that could be interpreted as an insult to your favourite editor, please keep my motivation in mind and resist the temptation for such interpretation. Oh and I work on OS X so some editors mentioned here may not apply to your environment, sorry.

Background

I learned to write code in 1987 on green screen terminals hooked up to a VAX. We were introduced to vi and I used it on a daily basis for a couple of years. No wonder I’ve got some of the basic vi commands baked into my brain stem! We moved on to using NeXT and Apollo workstations and I got to use editors that allowed you to click around with a mouse. Then came many years of using pencil, paper, chalk and blackboards, followed by the terrible years of only ever writing documents using MS Word and PowerPoint. So, until recently I hadn’t touched a developer’s editor in a very long time.

When I started writing code again I noticed that TextMate, Emacs and Vim were the main contenders in the Rails community. I decided fairly early in the piece to try and give each editor a good month of serious use so that I could, with some objectivity, find the right one for me. If you are starting out as a developer, I highly recommend this approach.

Foreground

And here is how it went. TextMate was the easiest to work with then because I could just use the OS X shortcuts that were in muscle memory and use my mouse to click around etc. When my TextMate trial ran out I looked around again and found AquaMacs – a version of Emacs for OS X that, well, allows you to edit at will and not use very many Emacs commands at all! Jokes aside, it mashes together Emacs with a Mac look-and-feel but this does get in the way of forcing any sort of discipline on oneself to learn Emacs. This opinion piece from Stack Overflow treats this topic quite well.

When I started my job I felt a lot of pressure to not be learning supporting tools like editors and instead, becoming immediately productive! This was laughable in hindsight really because I had nothing over an enthusiast’s understanding of Rails at this point and needed to really just learn for a while! My very awesome colleagues got this of course and encouraged the learning process, all the while nudging me ever so firmly towards getting a solid mind meld happening with my operating environment. I went back to TextMate for a short while but then followed the chosen platform of two of my co-workers and adopted the Carbon build for Emacs. I did this with the very awesome emacs-starter-kit and then things got really good! I spent a month with Emacs and watched my productivity increase.

Vim

And then the circle closed. Before I started, my dev team consisted of two Emacs users and one Vim devotee. Right about when I was finishing my month with Carbon Emacs, a new guy joined the team and brought the Vim users count up to two. At about this time I read the mighty Yehuda Katz on his journey from TextMate to Vim and decided to give MacVim a go. And that’s where I’ve settled! Why? Well, I really like the speed with which I can move around a Rails project with Tim Pope’s terrfic Rails.vim plugin (I am told that there are similar plugins for Emacs) and I like the ease with which plugins like NERDTree allows me to look at my filesystem. Most of all, Vim commands just fit better with my brain – I like that they are about movement around a buffer and that they seem to consistently build on this philosophy. It just seems to work for me I suppose, and I think it’s because I still carry some basic vi in my brain stem!

I’m very much a novice but I am comfortable with my environment and I am hungry to learn more. Here are some resources that I have found very useful thus far —

General reference and tutorials:

Efficient editing with Vim – Stellar article that is intended to help vi novices develop their skills so that they can use vi efficiently. It focuses on movements, the core of Vim commands
Use Vim like a Pro – comprehensive tutorial on some 38 different topics
Graphical vi-vim Cheat Sheet and Tutorial – indispensable cheat sheets with accompanying tutorials

Posts and screencasts that cover plugins and really make the connection with development come to life:

Steve Losh, Coming Home to Vim – well written and packed with pointers on plugins and vimrc customisations. Steve’s not a Rails guy (Python and Django I think) so nothing to do with Rails
Ben Orenstein’s screencast, Vim for Rails Developers. Recently I was looking around for a screencast that would show me how a seasoned Rails developer uses Vim and I came across Ben’s work and the Smash into Vim series by peepcode. I tried out the sample from peepcode and found the celebrity narrator’s style to be a bit off-putting (just a personal thing, I’m sure the ‘cast lives up to the high peepcode standard). Looked around at Ben’s stuff and found that he is a Rails developer, Vim obsessive and an efficiency junkie! Ben asks bloggers to contact him for a copy of his screencast in exchange for a review and this is what I did. I am happy to say that I was grinning like a Cheshire cat 10 minutes into the screencast and I only got happier as it went on. What about the first 10 minutes? Well, passionate as he is about efficiency, he starts off by spending (perhaps a tad too much) time on the topic of speed typing. It’s great advice, and it’s possibly the only part of the screencast that will apply to the complete Vim novice (if you don’t know the basics you won’t find them here). He really hit the sweet spot for me when he shows us how to effectively use Tim Pope’s Rails plugin. I use this plugin too but it became clear to me that I had only been using about 5% of its features! Boy is my productivity about to go up! Ben also takes us through the use of snippets, ctags and the surround plugins, spends some time on the wonders of ack and how to use it from within Vim, and closes with some general tips and tricks. Something to keep in mind is that Ben appears to be a Vim purist – he works within his terminal and does not use any of the bells and whistles that come with MacVim. No mice are touched at all during this screencast! All in all, I heartily recommend his screencast to the not-so-new-to-Vim Rails developer — well worth the $11.99 price tag I think!

Plugins – use them wisely

I’m not going to do the traditional thing of offering my vimrc and plugin list because I’m just using janus (give or take a plugin or two). Janus is a vim-starter-kit for Rails types that has been put together by that mythical Carlhuda! Modifications in my local gvirmc come mainly from things I have borrowed from Steve Losh’s article (referenced above).

I asked Ben for a comment on janus and this is what he had to say:

Actually, I’m not sure I’d recommend something like janus. The basic editor has so much built-in functionality that there’s already plenty to learn when you start out. Additionally, new users won’t know if certain behaviors are vanilla vim or accomplished through add-ons. Personally, I’d recommend new users go a couple weeks with no plugins. Start tweaking your vimrc to see learn what kind of customizations you can make. Then start adding plugins one at a time and mastering them slowly.

I’ll leave it up to you to figure out whether to pick up MacVim with janus and try out Vim … or whether you should do penance with the editor itself before adopting the gui version and plugins. I think it will depend a bit on whether you are the type that regards a solid understanding of a tool to be a pre-condition to using it, or, whether you are happy to get to the most productive environment as quickly as possible. I’ve chosen a middle ground – I’m mainly using Rails.vim and basic editor commands and gradually picking up what the other plugins in janus do.

In any case, that’s the story of how I ended up in smit with Vim. I hope you find some benefit from this post — good luck with the process of selecting an editor and making it your own!

Update (Oct 2011): Been using vim for a while now and am no longer using janus. My .vim is now on my github – forked from the remarkable Damien Le Berrigaud (vim superhero!), and updated with a few personal touches.

Front end stuff - HTML, CSS, JavaScript and jQuery

2011-01-15T00:00:00-08:00

When I started my job I had no CSS or JavaScript or jQuery to speak of. The first time I had to modify a page was when I realised that I had no foundation from which to learn CSS by example. Here are some of the resources that I found very helpful in my journey to attaining enough knowledge and skills to confidently assert that I am now a novice! In the second half of this post I present an example – constructing an “Add” button to a form to allow the user to add more rows of input fields.

References

If you are an absolute beginner, I suggest starting with the following goals:

Understand the box model. This is at the heart of CSS. All elements of a web page are boxes that have standard properties like height, width, border, padding and margin. Use Firebug to inspect anything on a webpage and use the Layout tab to view its box model representation. Incidentally, you can click into the layout view, change values and see the effect that this has on the element you are inspecting. Similarly, you can click into and edit anything you see in the Styles tab – a great way to experiment with what various CSS properties do.
There are so many tricks to the trade that you only really pick up with experience and I’ve got a ways to go before I am using CSS properties effectively – O’Reilly’s CSS cookbook and The Art and Science of CSS come in handy when I run into a CSS property or a pattern that I need to look up.
Get your head around IDs and classes. Simply put, IDs are unique while classes are not. Meanhwile, they are both hooks. See this article by Chris Coyier for more.
Understand how HTML documents are structured. For HTML 4 see this reference by the W3C and html-5.com looks like a good place to pick up resources for HTML 5. As I write this I realise that I am quite green on this particular topic – and the reason I know this is because I don’t know enough about the intention behind each structural element. However, and at the risk of sharp intakes of breath from those who can spell “semantic”(!), I will admit that I am no longer using <p> and <br> to create a list (use <ul> and <li> instead!).
JavaScript – learn the basics. Amy Hoy and Thomas Fuchs run a masterclass on the topic and so I asked them to suggest a book for the beginner. Amy suggested Eloquent Javascript by Marijn Haverbeke – it’s free, well written, and really very accessible. It even works as an introduction to programming.

At work we tend to use jQuery a lot. The documentation on jQuery.com is terrific but here are a few other references that I have found very useful – jQuery Fundamentals by Rebecca Murphey and jQuery: Novice to Ninja by Craig Sharkie and Earle Castledine. When starting out with jQuery, I suggest spending time getting to understand selectors. This is the hard part and gets clouded by the temptation to go shopping for jQuery actions immediately.

jQuery example – dynamically adding fields to a form

Here is an example of jQuery at work in a Rails app – something I learned from my colleague, Brad Wilson. We needed to create a form to collect names and email addresses and, wanted to make the form start out with a single row of input fields, giving the user the option to add more names and email addresses if they so wished (this was part of a referral process). Here’s what we came up with (if anyone out there has a better solution, please tell me about it through a comment or via email).

The partial for the three input fields looked like (this is in haml and uses the Rails form helper fields_for):

And the partial setting up the form along with a hidden row like so:

The resulting form (without the submit button) looks like this:

Finally, the jQuery that responds to a click on the “Add another” link:

Clicking on “Add another” results in another row of input fields being added to the form as shown in the image below:

So here is what is happening:

The form hides the simple three field _invitee_form in a div with class user_details_base. It also shows the first three field row in a div with id user_details.
The jQuery in form.js is invoked on clicking the link with id add_user (i.e. the link “Add another”). It selects the class user_details_base, clones it, appends it to the div with id user_details (i.e. to the visible part of the form). Finally, it invokes the action show(), thereby ensuring that the newly appended row is displayed.
What’s the return false(); about? Well, this prevents the browser from jumping back to the link anchor.

Hope this post has proved valuable – enjoy the wondrous world of HTML, CSS and JavaScript!

Notes on setting up a Rails development machine

2010-08-14T00:00:00-07:00

I got a job! I’m a Rails developer at Blake Education, a publishing company that specialises in textbooks, study aids and other material relevant to the education sector. We develop and maintain readingeggs.com – a Rails powered site that is cobranded with ABC Online. Its goal is to help children learn to read. I’m delighted to have discovered an opportunity for simultaneously feeding my passions for technology and education!

I’ve worked for high profile consulting firms and obscenely profitable banks but never have I had an employer give me such high quality kit – I was given a 24’’ display and a brand new MacBook Pro and spent the first day setting it up. Yay! Well, I learned a few things along the way that I will document here. Hopefully the tools mentioned here will be of benefit to other newbies looking for tips on setting up their (Rails) development machines/environments. Seasoned Ruby/Rails/dev types, please comment if you have suggestions for improvements! And yes, some of this is OS X specific.

XCode: Unless you get into ObjectiveC or MacRuby, you are unlikely to need the IDE but package managers for OS X depend on it. It’s on the DVD that came with your Mac but you may be better off just downloading the latest version from Apple.
Babushka: Described by its author, Ben Hoskings, as test-driven sysadmin, babushka is a great way to reliably negotiate the hitherto complex and messy installations of things like MySQL and ImageMagick. A great many useful deps (think recipes) have been written in babushka’s DSL and I was advised by my new coworkers to use them liberally! See Ben’s post on how to get started. Oh, babushka works on OS X and Ubuntu as well.
Homebrew: Max Howell’s homebrew is simply an awesome package manager for OS X. I’ve tried MacPorts and, well, I think Homebrew is just much more usable! I’m not the only one either – see this post for more. Use babushka to get homebrew with babushka homebrew (the dep will check for XCode and this is why it’s up there on this list).
The awesomeness of homebrew is demonstrated by the ease with which you can use it to install MySQL. brew install mysql will do all of the calisthenics involved! Installing the mysql gem requires some platform specific flags to be set – thanks to @defunkt’s tip something like this should work on Snow Leopard:
```
sudo env ARCHFLAGS="-arch x86_64" gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config
```
rvm: Wayne E. Seguin’s Ruby Version Manager, rvm is amongst the best things to happen to Ruby in 2010 – he won a very well deserved Ruby Hero award for rvm at RailsConf 2010. rvm is similar to Python’s virtualenv in that it allows you to set up separate environments for your Ruby projects, thereby enabling multiple versions of Ruby, Rails, and other gems to conveniently coexist on the same machine. You can associate gemsets to separate projects, and specify which version of Ruby and which gems belong to a gemset. Apart from the official documentation there are some excellent posts out there on getting started with rvm. See @ascarter’s post and also check out Michael Hartl’s updated Rails Tutorial for Rails 3 – it now includes the setup of a gemset for a Rails 3 environment in the first chapter. If you are working with a project that uses Bundler, check out Mikel Lindsaar’s excellent article, Bundle me some sanity for a quick overview of the magic that happens when rvm and Bundler get together. I was setting up a project that did not have a Bundler Gemfile associated with it so resorted instead to having one of my new colleagues export their rvm gemset (associated to the project) and then using rvm gemset import to simply set up all the gems I needed for my environment.

Django, Grappelli and Piston!

2010-07-01T00:00:00-07:00

After learning Ruby and Rails for a few months, my first commercial gig involved building the server side components of an iPhone application using Python and Django. In the words of Morpheus, “Fate, it seems, is not without a sense of irony!” Luckily for me, a lot of what I had learned in Ruby land ported across to this project and my relative inexperience at this point helped – I didn’t have a lot of Rails-muscle-memory to unlearn!

For context, I was required to build a website that would be used for basic CRUD (Create, Read, Update, Delete) management of the assets that the iPhone application would present. I also had to build an API that the iPhone application would call to update its assets.

The setup

I used virtualenv to sandbox the app. virtualenv allows fine-grained control of the Python libraries, versions, etc. used in constructing the app – and the move to a production environment is then trivial. Also, it allows for different environments to coexist on your development machine without contaminating each other. I’ve noticed that rvm is providing Rubyists with a similar experience.

Admin pages

Django generates admin pages by default and if all you need is basic CRUD maintenance of your database, these do a terrific job. So as to jazz up the presentation, I used grapelli. The main problem I ran into was general confusion with the set up of templates. Here’s what we ended up doing: In settings.py set ADMIN_MEDIA_PREFIX='/site-media/' and make sure that the media folder of grappelli is copied over to a folder site-media in the project. Then start runserver by:

       python manage.py runserver --adminmedia=`pwd`/site-media/

Tidy file paths

One of the more interesting problems with this project was that I needed to save uploaded media files in a directory corresponding to the id of the object to which they were associated. This would allow the local filesystem to be replicated on the iPhone and, knowing the structure of file paths would allow for efficient searching for media files. The issue here is that an object does not have an id until it has been saved.

Our first approach was to overload the save method by first calling the real save method, then redoing the file paths using the resulting id and then saving again. This was very messy and caused for duplicate objects to be created when importing from a CSV. However, this is a pattern that I found in a few places out there so here it is incase it is useful:

def save(self, *args, **kwargs):
	# Overload the save method so that we get the id of the parent object
	# move media files from tmp to category_thumbnails/category_id/
    super(Tag, self).save(*args, **kwargs) # Call the "real" save() method.
    file_name = os.path.basename(self.thumbnail.path)
    tmp_path = settings.MEDIA_ROOT + 'tmp/'
    # Transfer the contents of the tmp directory
    dest_path = "category_thumbnails/%d/" % (self.id,)
    # delete the dest_path if it exists
    shutil.rmtree(settings.MEDIA_ROOT + dest_path,ignore_errors=True)
    os.mkdir(settings.MEDIA_ROOT + dest_path)
    shutil.move(tmp_path + file_name, settings.MEDIA_ROOT + dest_path + file_name)
    self.thumbnail.name = dest_path + file_name
    # Now update the database entry with the new filename
    super(Tag, self).save(*args, **kwargs)

The solution that we ended up going with involved making a guid and then using something like this to set up the file path:

def category_thumbnail_path(instance, filename):
	return 'category/%s/%s' % (instance.guid, filename)
	thumbnail = models.FileField(upload_to=category_thumbnail_path)

Importing from a CSV

Import from CSV was frighteningly simple – the following two step process got me there:

Line up the fields and use get_or_create to either create the object or identify duplicates
Once the object has been created, use add to add in associated objects that are in a many-to-many relationship with the object

# First read in a line from the file and set fields of your object
for import_file in files:
    reader = csv.reader(open(import_file))
    # skip header line
    lines = list(reader)
    for row in lines[1:]:
      title = row[3]
      sub_title = row[4]
# etc. ...

# Next use get_or_create to create the object
# Note that duplicates detected when both title and address match

try:
   obj, created = Asset.objects.get_or_create(title=title, address=address,
		defaults={'sub_title':sub_title,
			'start_date': start_date,
			'end_date':end_date,
			'publish':publish
			# etc. ...
			})

# And here is how you add associated objects that are in many-to-many relationships with your object

print "Created=", created
if created:
    if row[19] == '0':
    	for i in range(5):
            obj.locations.add(Location.objects.get(pk=i+1))
    else:
        	obj.locations.add(Location.objects.get(pk=int(row[19])))

API and django-piston

Need a RESTful API that responds in JSON or XML? Django-piston to the rescue! It took me the better part of a day of looking at some examples and reading the documentation to get to understand it but once I got the simplicity of the django-piston framework, I had an API in no time flat! I won’t elaborate on this any further – I found that the framework comes with excellent supporting code snippets and the interested reader should simply work through these.

Tricks for young players

Here’s one that I think you only learn from experience! The system generated timestamp associated to an object need not be unique nor represent the order in which the object was entered. Implement an index that increments on every save to the database to achieve this

class DBstate (models.Model):
	# Counter that is incremented on every save to the database
	# Used when calling the API to specify the state of the DB on the iPhone
	save_count = models.IntegerField(default=0)

Of course you will need to overload each object’s save method to implement this.

def save(self, *args, **kwargs):
	# set db_count
	try:
		counter = DBstate.objects.get(pk=1)
		counter.save_count += 1
		self.db_count = counter.save_count
		counter.save()
	except SaveCounter.DoesNotExist:
		counter = DBstate()
		counter.save()
		self.db_count = counter.save_count

And this is definitely for fellow newbies – be very careful about the difference between using functions and return values when setting default values for fields. For instance,

guid = models.CharField(max_length=35, default = uuid.uuid1().hex, editable=False, blank=True)

will result in uuid.uuid1().hex being called once and its return value being used as the default value for the guid of all instances of the object. On the other hand, if you use the function only when specifying the default (note the missing parentheses):

guid = models.CharField(max_length=35, default = uuid.uuid1.hex, editable=False, blank=True)

then it will be called every time a default value is needed and each object will have a different guid.

The story of this blog

2010-06-01T00:00:00-07:00

Until recently I wasn’t much of a blogger. Not withstanding a couple of articles on trust ratings, I didn’t feel like I had anything to say that was profound enough to publish (still don’t but do read on!). I had set up something on blogger.com but was using it more as an OpenID server than anything else!

When I started my return to development, I began to appreciate how valuable it is to quickly document a trick, idea, or discovery. Googling for help when I was looking to get unstuck from a code related problem invariably lead me to posts of other developers. I also noticed that a lot of people in the Ruby and Python communities use their a blog as a mechanism for establishing themselves as professionals – a GitHub (or google code) account and a blog appear to be the business cards of the developer. So, I set out to resurrect the iNK blot with the intention of writing about my journey back to development, and, to document tips and tricks that other developers might find useful.

I looked around a bit and found that since I had mucked around with blogger, there were several options for creating quick and easy blogs. I checked out the usual suspects including Wordpress, Posterous and Tumblr, and I also saw several references in the Ruby community to the blog like a hacker approach. I guess it depends on what you are setting out to do but my simple take on things is that if you want to write prose, throw in images, videos, etc., then a service like Posterous or Tumblr is probably what you need. Editing a blog is easy on these platforms. You can concentrate on the content and never worry about hosting issues, but you also need to abdicate control of the minutiae of the style in which your pages are presented. If you want to present code blocks and get fancy with the features of HTML5 and CSS then you will likely find these services to be too constraining.

Anyway, I set myself a project to investigate further. I wanted to know more about what it would take to build a blog using Ruby tools and I was curious about whether this would then be too complicated and turn out to get in the way of writing – i.e. would it be a productivity blocker.

I started with the awesomely tiny toto which I discovered in an article by Peter Cooper on Ruby Inside. Toto will get you blogging in 10 seconds! It is minimalist in its approach, elegant, and it integrates neatly with heroku and git. I had a lot of fun playing with it and when I noticed that it is based on Scanty, I went off to see what this was about. Right about at this time I was also curious about Chris Wanstrath’s template language Mustache and lightweight alternatives to Rails like Sinatra and, found Scanty’s cousin Hackpad which uses both Sinatra and Mustache. Unfortunately, I found the scantiness of documentation on Hackpad to thwart my meager Ruby powers – I found it to be not so easy to set up after all.

The aforementioned Ruby Inside article also made reference to Jekyll, mentioning that it includes a blog-like system but at heart is designed to cope with full sites. Well, Jekyll is very well documented and owing to its connections with GitHub (Tom Preston-Werner, author of Jekyll, is also a co-founder of GitHub), comes with many examples of its use. It is used by GitHub to power its GitHub Pages feature that allows for publishing content in a GitHub repository to the web. Tom has written about his motivation and experience with creating Jekyll in the article, “Blogging like a hacker” and, presumably, is attributed with coining the phrase.

Eventually, I settled on Jekyll because I liked the fact that it is actively used and supported by so many of the GitHub community. I looked around the Sites list in the Jekyll wiki pages and settled on Alex Payne’s weblog as a starting point. Incidentally, I had no idea at the time that I was borrowing from such as strong member of the tech community – if you want to know more, check out his latest venture BankSimple. I had a prototype going in a few hours and found that rolling in google analytics, disqus comments and fonts from Typekit was ridiculously easy. By the end of the day I was up on GitHub pages and happily blogging like a hacker! The only hit to productivity appears to be that I have not yet settled on a text editor!

When I started writing the post on the flickr-notifier, I wanted to put up code snippets and so went looking around for examples of how to do this. Since then I have used CSS and HTML by Andrew Carter.

As I use Jekyll I am beginning to better understand Peter Cooper’s comment that Jekyll is designed to cope with full sites. I now see how to use Jekyll to create and easily maintain a static (i.e. not database driven) website. I am pleasantly surprised by this side-effect! See this post by Simon Pascal Klein for more on this topic.

Since learning about Jekyll I have heard of similar static-site generators such as nanoc and having spent some time in Python/Django land, run into Jekyll’s evil Python based twin, Hyde!

Things I would like to do on the topic of site generators:

Play around with Jekyll – look into replacing its Liquid template system with Mustache for instance
Learn more about nanoc and better understand what it does differently from Jekyll
Revisit Hackpad – see if it is suited to generating lightweight dynamic websites

A Flickr notifier app

2010-05-09T00:00:00-07:00

I am a daily user of Flickr, owing mainly to my participation in a Project 365 (post a picture for everyday of 2010). Not having attained enlightment, I check my photostream several times a day for comments or view stats etc. It’s obvious then that I set myself the task of using the Flickr API to make a nifty notifier of activity on a Flickr photostream. Here are some of the main things I learned from the experience.

The following were the requirements that I set myself for this project:

Should notify the user of new comments
Provide a summary of activity on their photostream
Use Growl as the notification mechanism
Use Ruby

Initially I thought I would need the Growl API in its full glory. I had recently discovered macruby and noticed a sample XCode project that showed macruby working with Growl. So, I started out aiming to build the app using macruby. Having seen macruby and Growl playing well together, I set about locating a ruby library that implemented Flickr API calls. The Flickr API documentation mentions flickr-ruby rflickr and flickr.rb and so I set about checking these out.

I spent most of my time with flickr.rb and got very stuck with getting the authentication workflow going. The Flickr API uses an OAuth styled approach that never requires a username and password but does require some clever shunting back and forth of API keys, shared secrets and frobs before a token is issued. Because my app needed to make authenticated API calls, I had to find a library whose authentication methods would be accessible to this newbie. And this is how I found flickraw.

Here are some of the main features of flickraw:

Small single file: flickraw.rb is less than 300 lines
Uses introspection so that changes to the API will not require an update of flickraw
Ruby syntax similar to the flickr api
Flickr authentication
Photo upload
Proxy support
Delayed library loading (for rails users)
Flickr URL helpers

Flickraw has excellent documentation and I found it to be simple and fun to use. It uses JSON internally and so allows for dot.lookup syntax to access attributes. I spent some time with flickraw in irb before understanding all this … but that was fun too! The following is a snippet from my code that illustrates the use of flickraw – keep in mind that the user has been authenticated at this point.

recent_activity = flickr.activity.userPhotos(:timeframe => timeframe)

 recent_activity.each do |x|
      message = ''
      x.activity.event.each do |z|
        case
        when z.type == "comment" then
        begin
            unless @displayed_comments.include? z.commentid
              @displayed_comments.push z.commentid
              message << "Comment: " << z._content << "\n"
            end
        end
        when z.type == "fave" then
          message << "Added as a favorite\n"
        end
        unless (message == '')
          message << "User: " << z.username << "\n"
          time_added = Time.at(z.dateadded.to_i)
          message << time_added.to_s << "\n"
        end
  end

Now unfortunately, macruby 0.5 did not play well with flickraw. Loading up the flickraw gem resulted in a message about not supporting native extensions yet. Not knowing how to fix this, I looked around some more and came across the far less complex, command line version of Growl. growlnotify ships with the Growl disk image (for mac os x) and allows for a very simple way of popping up Growl bubbles. More information on growlnotify can be found here. The following is a snippet from my code that demonstrates how to use it (string interpolation being used here of course).

 `growlnotify #{(sticky)? '-s':''} --image #{icon_filename} -m "#{message}" "#{title}"`

Thanks mainly to flickraw, I had a lot of fun with this, my first project with ruby. Screenshots available from the Flickr App Garden here and the app itself is on github here. I hope you check them out and give me your feedback. Time permitting I will try again with macruby 0.6 – I’d like to make the installation and setup a little more user-friendly.

In my next post I expect to write about the second project I set myself – constructing this blog using a Ruby static site generator.