Django/PYTHONPATH gotcha

Just picking up django again after a while away and had had some problems with the following error when running 'django-admin.py runserver --settings=mysite.settings'

Error: Could not import settings 'mysite.settings' (Is it on sys.path? Does it have syntax errors?): No module named mysite.settings

Google was not my friend as all of the returned results were to do with apache config etc

Turned out I'd simply forgotten to put the current working directory (.) in the PYTHONPATH (e.g export PYTHONPATH=.:$APPENGINEPATH:$APPENGINEPATH/lib/yaml/lib:$APPENGINEPATH/lib/webob:$APPENGINEPATH/lib/ipaddr:$APPENGINEPATH/lib/fancy_urllib

) doh!

Google App Script: Experiences in Cloudland

I'm a bit of a fan of Software as a Service and Google Apps. It seems to solve a lot of problems allowing me to work with bigger building blocks and to focus on solving domain specific problems rather than writing another document storage solution or emailer service. So the announcement earlier this year of the availability of Google App Script (GAS) caught my eye. GAS is server side javascript plus a stack of libraries and which forms part of the suite of tools making up Googles Apps. My first thoughts on seeing the introductory webcast were:

• This looks powerful - it's the glue between SaaS
• How do you test drive it?
• Is it just Spreadsheets and VBA for the cloud?
• Is it a quick win with lots of pain down the line?

My experience with javascript in the past has always been with code which hacks about with the DOM in a browser or, even worse, implements business logic in the UI. I had not appreciated how elegant and powerful it can be. GAS comes with some standard libraries to do things with google apps like send email, update calenders, create documents and build sites plus some low level stuff like http fetches, parse xml, consume soap services and build UI's. All this makes it a very compelling offering and one worth a try to put it through it's paces. There is one notable omission from the libraries and that is access to the google data services which meant that there was no easily available persistence mechanism.

The project to try GAS out was a simple site inspection application where an inspector can be allocated an appointment to inspect a premises, complete an online inspection report and a reviewer can look at the completed report and make additional comments before generating a summary. This project would exercise a few of the available libraries.

There were a few false starts to do with persitence. Trying to use a spreadsheet as a relational database (doh!) and using the simple key-value store in Big Table every script gets for free were both discounted early on. Also I discovered a couple of useful SaaS applications:

• SurveyGizmo which seemed to do everything we wanted to do from a inspection report UI perspective. it does not have any workflow but does have a decent XML based API. This removed the need for us to create our own inspection report builder.
• Cloudant which offered an instance of CouchDB (a RESTful document database) which allowed us to simply use the GAS Http Fetch libraries for persistence.

The approach I used was to use a script to build and display a UI for scheduling an inspection and send an email to the inspector containing a link to the report to be completed. The UI was displayed within an iframe of a page within a dedicated google site. There were also other pages containing iframes: one which displayed a list of surveys and their status (pending, awaiting review & reviewed) and another which displayed an individual survey response.

Experiences

In a conversation with a colleague I was pointed at an excellent book by Douglas Crockford called 'Javascript: The Good Parts' which turned out to be invaluable and which completely changed my perspective on javascript. Coupled with this I had also recently attended Software Craftsmanship 2010 at Bletchley Park during which I had attended a session entitled Understand a language better – Test Drive your own Unit Testing Framework by Michael Hunger which was a real eye opener for me and answered one of my original questions (Q: How do I test drive this, A: By writing the gasUnit test framework!). Reading the book, writing the test framework and using the excellent jslint all together made me feel much more comfortable coding and resulted in tested code with clear separation of concerns using MVC.

Despite my enthusiasm for javascript as a language I had a number of problems with the GAS:

UI libraries

• These are a scripting layer on top of the GWT and allow you to programatically build UI's.
• It is marked as experimental and it is not a full implementation of all the features of GWT. For instance styling seemed a little clunky and there was no way of rendering pure html.
• Both IE and FireFox threw errors for some of my scripts which rendered OK in Chrome - this is very worrying since one of the purposes of using GWT is to remove concerns about cross browser compatability.
• Pages also took an age to render (presumably because of the rendering javascript download and execution) but once loaded the interaction with the page was very responsive
• The page source looked as ugly as sin but have you looked at the page source for the google home page recently!
• I think some of my problems were more to do with the change in mindset from pages and http post/get to one based around an ajax based web application.

General

• Each script needs to be stand alone so it is not easy to reuse code. For example, in order to reuse the test framework code in each script I placed the framework in a Google Code SVN repository and pulled in the file using a fetch + eval. This worked OK but is not ideal, probably hinders performance and is considered evil by jslint.
• The script editor was amazing considering it is a browser based application. It seemed to cope with intermittent networks OK, recovered code following disconnects, was very responsive on a good network and produced a very fast red-green-refactor cycle when doing TDD. Debugging required tracing using the Logger library which worked OK. Refactoring and navigation is very primitive. The revision history for a script is not a full history (~ last 20) so is of limited value when making frequent changes.
• There doesn't seem to be a very active community - I posted a couple of general questions (well phrased I hope!) which were never answered.

All in all GAS worked very well. The UI library is still very beta and we decided not to use it as a solution at this point but will return to it at some point. It was a great tool for quickly prototyping a workflow without having to worry about servers or deployment. I still have some concerns about integrating this into a managed source control, build and release process - I don't want to be part of a 'deploy and fix' hackfest.

There will be a separate post on my experiences of javascript and of developing gasUnit.

Entity-Attribute-Value models

Just listened to Ayendes podcast on RavenDB, a .NET document DB solution, at Herding Code (http://herdingcode.com/?p=255). A long podcast but worth the effort with some interesting stuff especially around dealing with model changes over time, event sourcing and the sweet spot for usage. Ayende mentioned Entity-Attribute-Value models as particularly appropriate for a document DB which caught my attention as it sounded very much like the approach we had taken at PAI. So I started digging around and found the following:

• Wikipedia entry: http://en.wikipedia.org/wiki/Entity-attribute-value_model
• Python/Django implementation: http://pypi.python.org/pypi/eav-django/1.3.3
• C# implementation: http://hanssens.org/post/Generic-Entity-Attribute-Value-Model-e28093-A-POCO-Implementation.aspx

The implementations are focused on persistence using a relation DB but with a document database this should all just happen. I think it is time for us to switch out our long thin relational model for a document database - probably RavenDB.

Edit: Just found this useful blog post which dissects an MVC/RavenDB app and comments on best practice: http://ayende.com/Blog/archive/2010/07/28/reviewing-communitycourses-a-ravendb-application.aspx

Reflections on tonight's agile testing uk meetup

Just completed a very interesting practical exercise with Gojko Adzic at the agile testing uk (http://agiletesting.org.uk/) event hosted by skillsmatter in London. Worked well as a group exercise and a very well attended meeting (~60 people).

Looked at a fit test and discussed why it was bad:

* Lack of focus
* Lack of clear definition of business rule(s) under test
* Included technical/software terms terms, db ids and references to fixture classes.
* Non locale specific terms (check vs cheque)
* Some irrelevant data

Key things I got out of it:

* Should describe behaviour for business rules - specification not a script
* OK to include non business rules (e.g. ordering of results) to make testing easier as long as it doesn't conflict with existing business rules. May also prompt discussion around rules
* Non-deterministic ordering is a particular problem which causes test failures and sometimes removal of tests by developers
* We missed an obvious business rule - do we send cheques for £0 (eg someone on the payroll who has not worked during the period in question)
* Feel much more comfortable with Given-When-Then (esp with tables) vs Fitness
* Most teams expressed acceptance tests using to Given-When-Then rather with tables - teams which did seemed to get into difficulty

Need to take another look at the slide stack to see if I have missed anything.

Some reflections on Martin Fowlers presentation at AgileYorkshire this week.

During Thursday's commute I read a Martin Fowler blog post - a great piece on the maturity of REST implementations (Richardson Maturity Model). It took about half an hour to read, re-read and digest. The information in it could have been a candidate for one of the presentations making up Martin's pick and mix selection from Wednesday. I could easily find 2 other blog posts of his which could be similarly described. So if I can read a few well written blog posts on an interesting subject why should we go through the hassle of organising a night, getting a speaker to travel 'oop North and having 100 people spend some of their valuable social life at a geek night?

For me the value of evenings such as Wednesday is only partly about the content. Equally important is the sense of event it generates and, in particular, the atmosphere of discussion and enquiry it fosters. It was a shame we had a tight deadline to keep to which meant that the time for questions was limited but even so the buzz both beforehand and afterwards was palpable. The presentations by Martin on Wednesday were great seed topics for the resulting discussions which occurred during the evening and will have continued for some days afterwards. As a case in point, I have just had a conversation about it with a non-technical friend who stopped by this afternoon.

I think it is true that developers - and by that I mean people who spend a large part of their working week turning user stories into delivered features - as a whole do not blog. They may document the solution to a problem they have found or throw out some observational/controversial stuff in the run up to redundancy or contract renewal, but the number who blog as reflection is small. For a profession which has a reputation as being "socially awkward" I have found that the preferred communication medium of good developers is a good ol' face to face discussion. This is odd in that some of the discussions around the topic of Martin's final segment (gender skewing in development teams) invoked the "autism" argument. This is that on an autistic spectrum there is evidence that males have a higher incidence than females (http://www.nas.org.uk/) and since there are more male developers than female developers then, ergo, autistic tendencies must be required to be a developer. I think this an example of a converse error).

The irony of observing the intense social interactions between developers arguing passionately and empathetically about the subject of where they lie on an autism scale was not lost on me. We were still discussing the subject matter the following day and I expect the themes will keep re-occurring.

So are social events like Agile Yorkshire events necessary or could we have achieved the same outcome by other means; comments on a blog post, VOIP meeting, a video link presentation? Asides from the importance of an influential figure like Martin acknowledging grass-roots community groups by presenting to them I think that a physical meet-up is important. This is because of the quality and effectiveness of the communication it encourages and in the end a developer who can not communicate effectively is not a good developer.