Django, Grappelli and Piston!

After learning Ruby and Rails for a few months, my first commercial gig involved building the server side components of an iPhone application using Python and Django. In the words of Morpheus, “Fate, it seems, is not without a sense of irony!” Luckily for me, a lot of what I had learned in Ruby land ported across to this project and my relative inexperience at this point helped – I didn’t have a lot of Rails-muscle-memory to unlearn!

For context, I was required to build a website that would be used for basic CRUD (Create, Read, Update, Delete) management of the assets that the iPhone application would present. I also had to build an API that the iPhone application would call to update its assets.

The setup

I used virtualenv to sandbox the app. virtualenv allows fine-grained control of the Python libraries, versions, etc. used in constructing the app – and the move to a production environment is then trivial. Also, it allows for different environments to coexist on your development machine without contaminating each other. I’ve noticed that rvm is providing Rubyists with a similar experience.

Admin pages

Django generates admin pages by default and if all you need is basic CRUD maintenance of your database, these do a terrific job. So as to jazz up the presentation, I used grapelli. The main problem I ran into was general confusion with the set up of templates. Here’s what we ended up doing: In settings.py set ADMIN_MEDIA_PREFIX='/site-media/' and make sure that the media folder of grappelli is copied over to a folder site-media in the project. Then start runserver by:

       python manage.py runserver --adminmedia=`pwd`/site-media/

Tidy file paths

One of the more interesting problems with this project was that I needed to save uploaded media files in a directory corresponding to the id of the object to which they were associated. This would allow the local filesystem to be replicated on the iPhone and, knowing the structure of file paths would allow for efficient searching for media files. The issue here is that an object does not have an id until it has been saved.

Our first approach was to overload the save method by first calling the real save method, then redoing the file paths using the resulting id and then saving again. This was very messy and caused for duplicate objects to be created when importing from a CSV. However, this is a pattern that I found in a few places out there so here it is incase it is useful:

def save(self, *args, **kwargs):
	# Overload the save method so that we get the id of the parent object 
	# move media files from tmp to category_thumbnails/category_id/
    super(Tag, self).save(*args, **kwargs) # Call the "real" save() method.
    file_name = os.path.basename(self.thumbnail.path)
    tmp_path = settings.MEDIA_ROOT + 'tmp/'
    # Transfer the contents of the tmp directory
    dest_path = "category_thumbnails/%d/" % (self.id,)
    # delete the dest_path if it exists
    shutil.rmtree(settings.MEDIA_ROOT + dest_path,ignore_errors=True)
    os.mkdir(settings.MEDIA_ROOT + dest_path)
    shutil.move(tmp_path + file_name, settings.MEDIA_ROOT + dest_path + file_name)
    self.thumbnail.name = dest_path + file_name
    # Now update the database entry with the new filename
    super(Tag, self).save(*args, **kwargs)
  

The solution that we ended up going with involved making a guid and then using something like this to set up the file path:

def category_thumbnail_path(instance, filename):
	return 'category/%s/%s' % (instance.guid, filename)
	thumbnail = models.FileField(upload_to=category_thumbnail_path)
    

Importing from a CSV

Import from CSV was frighteningly simple – the following two step process got me there:

• Line up the fields and use get_or_create to either create the object or identify duplicates
• Once the object has been created, use add to add in associated objects that are in a many-to-many relationship with the object

		
# First read in a line from the file and set fields of your object 
for import_file in files:
    reader = csv.reader(open(import_file))
    # skip header line
    lines = list(reader)
    for row in lines[1:]:
      title = row[3]
      sub_title = row[4]
# etc. ...

# Next use get_or_create to create the object 
# Note that duplicates detected when both title and address match

try:
   obj, created = Asset.objects.get_or_create(title=title, address=address,
		defaults={'sub_title':sub_title,
			'start_date': start_date,
			'end_date':end_date,
			'publish':publish
			# etc. ...
			})
				
# And here is how you add associated objects that are in many-to-many relationships with your object

print "Created=", created
if created:
    if row[19] == '0':
    	for i in range(5):
            obj.locations.add(Location.objects.get(pk=i+1))
    else:
        	obj.locations.add(Location.objects.get(pk=int(row[19])))

API and django-piston

Need a RESTful API that responds in JSON or XML? Django-piston to the rescue! It took me the better part of a day of looking at some examples and reading the documentation to get to understand it but once I got the simplicity of the django-piston framework, I had an API in no time flat! I won’t elaborate on this any further – I found that the framework comes with excellent supporting code snippets and the interested reader should simply work through these.

Tricks for young players

Here’s one that I think you only learn from experience! The system generated timestamp associated to an object need not be unique nor represent the order in which the object was entered. Implement an index that increments on every save to the database to achieve this

class DBstate (models.Model):
	# Counter that is incremented on every save to the database
	# Used when calling the API to specify the state of the DB on the iPhone
	save_count = models.IntegerField(default=0)
	

Of course you will need to overload each object’s save method to implement this.

def save(self, *args, **kwargs):
	# set db_count
	try:
		counter = DBstate.objects.get(pk=1)
		counter.save_count += 1
		self.db_count = counter.save_count
		counter.save()
	except SaveCounter.DoesNotExist:
		counter = DBstate()
		counter.save()
		self.db_count = counter.save_count
			

And this is definitely for fellow newbies – be very careful about the difference between using functions and return values when setting default values for fields. For instance,

guid = models.CharField(max_length=35, default = uuid.uuid1().hex, editable=False, blank=True)  

will result in uuid.uuid1().hex being called once and its return value being used as the default value for the guid of all instances of the object. On the other hand, if you use the function only when specifying the default (note the missing parentheses):

guid = models.CharField(max_length=35, default = uuid.uuid1.hex, editable=False, blank=True)  

then it will be called every time a default value is needed and each object will have a different guid.

—01 Jul, 2010

The story of this blog

Until recently I wasn’t much of a blogger. Not withstanding a couple of articles on trust ratings, I didn’t feel like I had anything to say that was profound enough to publish (still don’t but do read on!). I had set up something on blogger.com but was using it more as an OpenID server than anything else!

When I started my return to development, I began to appreciate how valuable it is to quickly document a trick, idea, or discovery. Googling for help when I was looking to get unstuck from a code related problem invariably lead me to posts of other developers. I also noticed that a lot of people in the Ruby and Python communities use their a blog as a mechanism for establishing themselves as professionals – a GitHub (or google code) account and a blog appear to be the business cards of the developer. So, I set out to resurrect the iNK blot with the intention of writing about my journey back to development, and, to document tips and tricks that other developers might find useful.

I looked around a bit and found that since I had mucked around with blogger, there were several options for creating quick and easy blogs. I checked out the usual suspects including Wordpress, Posterous and Tumblr, and I also saw several references in the Ruby community to the blog like a hacker approach. I guess it depends on what you are setting out to do but my simple take on things is that if you want to write prose, throw in images, videos, etc., then a service like Posterous or Tumblr is probably what you need. Editing a blog is easy on these platforms. You can concentrate on the content and never worry about hosting issues, but you also need to abdicate control of the minutiae of the style in which your pages are presented. If you want to present code blocks and get fancy with the features of HTML5 and CSS then you will likely find these services to be too constraining.

Anyway, I set myself a project to investigate further. I wanted to know more about what it would take to build a blog using Ruby tools and I was curious about whether this would then be too complicated and turn out to get in the way of writing – i.e. would it be a productivity blocker.

I started with the awesomely tiny toto which I discovered in an article by Peter Cooper on Ruby Inside. Toto will get you blogging in 10 seconds! It is minimalist in its approach, elegant, and it integrates neatly with heroku and git. I had a lot of fun playing with it and when I noticed that it is based on Scanty, I went off to see what this was about. Right about at this time I was also curious about Chris Wanstrath’s template language Mustache and lightweight alternatives to Rails like Sinatra and, found Scanty’s cousin Hackpad which uses both Sinatra and Mustache. Unfortunately, I found the scantiness of documentation on Hackpad to thwart my meager Ruby powers – I found it to be not so easy to set up after all.

The aforementioned Ruby Inside article also made reference to Jekyll, mentioning that it includes a blog-like system but at heart is designed to cope with full sites. Well, Jekyll is very well documented and owing to its connections with GitHub (Tom Preston-Werner, author of Jekyll, is also a co-founder of GitHub), comes with many examples of its use. It is used by GitHub to power its GitHub Pages feature that allows for publishing content in a GitHub repository to the web. Tom has written about his motivation and experience with creating Jekyll in the article, “Blogging like a hacker” and, presumably, is attributed with coining the phrase.

Eventually, I settled on Jekyll because I liked the fact that it is actively used and supported by so many of the GitHub community. I looked around the Sites list in the Jekyll wiki pages and settled on Alex Payne’s weblog as a starting point. Incidentally, I had no idea at the time that I was borrowing from such as strong member of the tech community – if you want to know more, check out his latest venture BankSimple. I had a prototype going in a few hours and found that rolling in google analytics, disqus comments and fonts from Typekit was ridiculously easy. By the end of the day I was up on GitHub pages and happily blogging like a hacker! The only hit to productivity appears to be that I have not yet settled on a text editor!

When I started writing the post on the flickr-notifier, I wanted to put up code snippets and so went looking around for examples of how to do this. Since then I have used CSS and HTML by Andrew Carter.

As I use Jekyll I am beginning to better understand Peter Cooper’s comment that Jekyll is designed to cope with full sites. I now see how to use Jekyll to create and easily maintain a static (i.e. not database driven) website. I am pleasantly surprised by this side-effect! See this post by Simon Pascal Klein for more on this topic.

Since learning about Jekyll I have heard of similar static-site generators such as nanoc and having spent some time in Python/Django land, run into Jekyll’s evil Python based twin, Hyde!

Things I would like to do on the topic of site generators:

• Play around with Jekyll – look into replacing its Liquid template system with Mustache for instance
• Learn more about nanoc and better understand what it does differently from Jekyll
• Revisit Hackpad – see if it is suited to generating lightweight dynamic websites

—01 Jun, 2010

A Flickr notifier app

I am a daily user of Flickr, owing mainly to my participation in a Project 365 (post a picture for everyday of 2010). Not having attained enlightment, I check my photostream several times a day for comments or view stats etc. It’s obvious then that I set myself the task of using the Flickr API to make a nifty notifier of activity on a Flickr photostream. Here are some of the main things I learned from the experience.

The following were the requirements that I set myself for this project:

• Should notify the user of new comments
• Provide a summary of activity on their photostream
• Use Growl as the notification mechanism
• Use Ruby

Initially I thought I would need the Growl API in its full glory. I had recently discovered macruby and noticed a sample XCode project that showed macruby working with Growl. So, I started out aiming to build the app using macruby. Having seen macruby and Growl playing well together, I set about locating a ruby library that implemented Flickr API calls. The Flickr API documentation mentions flickr-ruby rflickr and flickr.rb and so I set about checking these out.

I spent most of my time with flickr.rb and got very stuck with getting the authentication workflow going. The Flickr API uses an OAuth styled approach that never requires a username and password but does require some clever shunting back and forth of API keys, shared secrets and frobs before a token is issued. Because my app needed to make authenticated API calls, I had to find a library whose authentication methods would be accessible to this newbie. And this is how I found flickraw.

Here are some of the main features of flickraw:

• Small single file: flickraw.rb is less than 300 lines
• Uses introspection so that changes to the API will not require an update of flickraw
• Ruby syntax similar to the flickr api
• Flickr authentication
• Photo upload
• Proxy support
• Delayed library loading (for rails users)
• Flickr URL helpers

Flickraw has excellent documentation and I found it to be simple and fun to use. It uses JSON internally and so allows for dot.lookup syntax to access attributes. I spent some time with flickraw in irb before understanding all this … but that was fun too! The following is a snippet from my code that illustrates the use of flickraw – keep in mind that the user has been authenticated at this point.

recent_activity = flickr.activity.userPhotos(:timeframe => timeframe)

 recent_activity.each do |x|
      message = ''      
      x.activity.event.each do |z|
        case
        when z.type == "comment" then
        begin
            unless @displayed_comments.include? z.commentid
              @displayed_comments.push z.commentid
              message << "Comment: " << z._content << "\n"
            end
        end
        when z.type == "fave" then
          message << "Added as a favorite\n"
        end
        unless (message == '')
          message << "User: " << z.username << "\n"
          time_added = Time.at(z.dateadded.to_i)
          message << time_added.to_s << "\n"
        end
  end

Now unfortunately, macruby 0.5 did not play well with flickraw. Loading up the flickraw gem resulted in a message about not supporting native extensions yet. Not knowing how to fix this, I looked around some more and came across the far less complex, command line version of Growl. growlnotify ships with the Growl disk image (for mac os x) and allows for a very simple way of popping up Growl bubbles. More information on growlnotify can be found here. The following is a snippet from my code that demonstrates how to use it (string interpolation being used here of course).

 `growlnotify #{(sticky)? '-s':''} --image #{icon_filename} -m "#{message}" "#{title}"`

Thanks mainly to flickraw, I had a lot of fun with this, my first project with ruby. Screenshots available from the Flickr App Garden here and the app itself is on github here. I hope you check them out and give me your feedback. Time permitting I will try again with macruby 0.6 – I’d like to make the installation and setup a little more user-friendly.

In my next post I expect to write about the second project I set myself – constructing this blog using a Ruby static site generator.

—09 May, 2010

Get to know the Ruby community

I like to watch and listen

I have benefited greatly from listening in on the chatter of Rubyists on twitter and from connecting with my local Rails community. It’s been a fantastic way of tuning into what’s new and interesting, getting connected with the community, and beginning to understand the culture of Ruby and Rails. I recommend it whole heartedly to anyone starting out.

I’m not a heavy user of social networks – I am not a Facebook user and am moderately active on twitter. I think the main reason I am unhappy with social networks is their poor signal to noise ratio and can say that for the most part, the Rubyists I follow provide entertaining and useful information. It feels a little weird to be eavesdropping without the (explicit) knowledge of the primary participants in the conversation but this is, after all, a central tenet of virtual social networks!

Here are a few things that I did to plug into the Ruby community:

• I built a list of Rubyists and news sources by starting with @peterc and @defunkt and checking out their lists to see who they follow. I picked @peterc because I read his book and enjoyed it very much (see my previous post) and I think I got to @defunkt after finding him on youtube and checking out his work on Mustache and Sinatra and Github (he is a co-founder). At this point I’ve got a fairly stable list of my own going and on most days I come across something cool or something new that’s fun and helps keep things fresh and interesting.
• I found out who features in searches for local Rails consultancies and using them as a starting point for discovering local Rubyists. Should be useful to identify what’s on peoples’ minds locally and when they might be hiring.
• I located my local Ruby and Rails meetup. It’s affectionately called rorosyd – it meets every month at a nice pub and features talks that occasionally include the speaker bursting into song (MJ’s Thriller in one very amusing instance)! A lot of experienced members of the community attend and you also invariably find more than the occasional recruiter and tech curioso. Newcomers are required to introduce themselves and after the talks there is an opportunity to match available work with available people. I’ve attended three of these so far and am looking forward to the next one. Oh and there is a bit of beer drinking that goes on as well!

Satish Talim (mentioned also in the previous post) has a blog post on Rubyists to follow – this might also be useful to those starting out.

—26 Apr, 2010

Ruby reading material and courses

Books and tutorials

Owing to the popularity of Rails, there are an incredible number of books out there that promise introductions, expertise and Ruby super powers. I haven’t read most of them and obviously, am not in a position to comment on any beyond the few that I have. Perhaps I missed a few gems (pun intended, sorry!) in my brief search and I make no claim that I found the best of the crop. Anyway, what follows is a short description of the way in which I went about deciding on which books were right for me.

I chose to start off with an introduction to Rails. It was pretty clear that I would also need an introduction to Ruby and so I looked for a Rails book that provided this as well. Having had some exposure to the excellent Pragmatic Programmer series, I was tempted to pick up Agile Web Development with Rails. After all, one of the authors, David Heinemeier Hansson, is the creator of the Rails framework. I did buy this book eventually but owing to some advice from @matthewsinclair, it wasn’t the book that I chose to get me started.

I spent some time on Amazon reading reviews of Rails books and based mainly on what I read there, chose to start off with Patrick Lenz’s, Simply Rails 2.

Patrick’s book is fun! It is written in a relaxed style and concentrates on getting the beginner to the point where they have quickly built a simple Rails app (modelled on Digg). It fits the ethos of Rails in that it provides near-real-time gratification and the reader is encouraged by the fact that they’re getting their hands dirty and have something to show for it. His here-hold-my-hand-while-we-connect-the-dots approach has the obvious problem of needing to gloss over the detail. Nevertheless, this is what this beginner needed so as to avoid getting bogged down early. Patrick’s chapter on Ruby basics is good, but basic. I read his book over a couple of sittings, working through the code that he presents and building the app that he uses to illustrate the main aspects of Rails.

@matthewsinclair also recommended Michael Hartl’s work-in-progress tutorial, “The Ruby on Rails Tutorial Book” and referred to here as Rails Tutorial. Michael’s bio mentions his accolades as a teacher at Caltech. Having taught for 8 years or so myself, I can spot a passion for good teaching from ten paces and Michael’s got it! He has successfully ported his teaching abilities to the written word and is developing a tutorial that is simply excellent. Rails Tutorial is written for the complete newbie while containing enough detail and secondary content to make it quite useful to the curious (non Rails) programmer as well. When I came across his tutorial, I had already started on Patrick’s book. I wanted to finish my first foray into Rails, so I put off tackling Michael’s tutorial until after I had finished Simply Rails 2. I don’t regret doing this but suggest to the newbie reader that they will likely benefit from using Michael’s tutorial as a starting point. Some of the highlights include:

• Using git from the start (so you learn about version control)
• Consistently pushing TDD (test driven development)
• Getting you started with an account on Github and Heroku before the end of Chapter 1
• Building a non-trivial mock up of a social micro-blogging site
• Integrating with Gravatar and other fun tidbits …

Michael had written 4 chapters (there are a total of 11) when I worked through his tutorial the first time. Chapter 3 is Rails-flavored Ruby, in which he starts off by asserting that:

Ruby is a big language, but fortunately the subset needed to be productive as a Rails developer is relatively small. Moreover, this subset is different from the usual approaches to learning Ruby, which is why, if your goal is making dynamic web applications, I recommend learning Rails first, picking up bits of Ruby along the way.

Well, instictively this clashed with my need to connect with first principles. I’m not suggesting that Michael is wrong – it’s just that at that point in my journey, I felt I needed to shore up my foundations and spend some time with Ruby. Michael makes some suggestions for further reading in the early part of his book and I just followed his recommendation and bought a copy of Peter Cooper’s, Beginning Ruby, Second Edition.

Michael recommends Peter’s book saying,

I recommend that newer programmers read Beginning Ruby by Peter Cooper, which shares the same basic instructional philosophy as Rails Tutorial. After finishing Beginning Ruby, you’ll be ready for Agile Web Development with Rails by Dave Thomas and David Heinemeier Hansson and The Rails Way by Obie Fernandez.

Peter’s writing style is simply delightful! He is funny, relaxed and conversational as he leads you through the basics of Ruby. Using some excellent and non-trivial examples and exercises he gets the reader beyond the reading and into the doing. His book is sub-titled, From Novice to Professional and he achieves this by splitting the book into two parts. The first half is focussed on foundational elements of the language and the second half covers topics like web application frameworks (Rails, Sinatra, etc.), desktop application development etc. This section also provides a whirlwind tour of some of ruby’s main libraries. I find that I use the first half as a reference and when I want to learn about something new (e.g. Shoes), the second half provides for an excellent starting point. There aren’t any formal exercises in the book but he closes the first half with a chapter that builds a large application that pulls together individual facets of Ruby. Peter is also the author of one of Ruby’s chief sources of information on the web, RubyInside and is quite active on twitter.

I’ve also acquired the pickaxe book or Agile Web Development with Rails. I expect that I will spend more time with it as I get my hands dirty with a Rails project or two. At this point in time the print version of this book covers Rails 2 and a beta version refactoring the book to Rails 3 is in progress. I suggest holding off until this version is complete and/or picking up a second hand copy of the Rails 2 version.

Courses

Learning a programming language by getting in there and writing some code is far more effective than (just) reading a book about its foundational elements. If you are an experienced programmer and have a project in mind, cracking open an IDE (or in the case of Ruby, an editor!) is an effective first step. I had a couple of projects in mind (more on this later) but wanted to find something a bit more bite sized and guided. There are some excellent resources targeted at providing exercises to the beginning and intermediate Rubyist and all of them appear to have some form of support or guidance so that you can have questions answered and a critique provided of your solution. My (limited) research uncovered the following:

• rubylearning.org
• rubyproblems.com
• Project Euler – Mathematical and computer programming problems – not Ruby specific

I signed up for the Core Ruby course at rubylearning.org and worked through 8 weeks of tutorials, exercises and quizzes. The format of the course is described here. In summary, it is free, based on notes and exercises written by Ruby Learning’s creator, Satish Talim and supported by an international cast of mentors. I really got a lot out of doing the exercises every week and getting feedback from the mentors. They suggested variations that I could explore and mistakes that I could try and fix! The interaction with the mentors provided something that books rarely do – I began to get learn more about the ruby way of doing things and (anti) patterns relating to efficiency. I learned some of Ruby’s idioms and my confidence in Ruby programming increased significantly. More than anything, I think Satish delivered on his mission to demonstrate that Ruby helps programmers have more fun! I really did have a lot of fun over the 8 weeks I spent with the course and thank Satish and his merry Rubyists for a great time! Check out Satish’s blog for more information on what he is up to (courses on Clojure, Git, etc.) and for some interesting interviews with top Rubyists exploring their views on effective ways of learning Ruby.

I am excited about exploring rubyproblems.com and the dormant mathematician in me is very curious about checking out the goings on at Project Euler. I expect that I will write about any adventures I have with these sites.

I hope that this article provides some useful points of reference for other newbies looking for books, tutorials and/or courses to help them on their way. In my next post I will write about the role that virtual and physical networks play in my continuing journey with Ruby and Rails.

—12 Apr, 2010