Friday, December 3, 2010

Django/PYTHONPATH gotcha

Just picking up django again after a while away and had had some problems with the following error when running ' runserver --settings=mysite.settings'

Error: Could not import settings 'mysite.settings' (Is it on sys.path? Does it have syntax errors?): No module named mysite.settings

Google was not my friend as all of the returned results were to do with apache config etc

Turned out I'd simply forgotten to put the current working directory (.) in the PYTHONPATH (e.g export PYTHONPATH=.:$APPENGINEPATH:$APPENGINEPATH/lib/yaml/lib:$APPENGINEPATH/lib/webob:$APPENGINEPATH/lib/ipaddr:$APPENGINEPATH/lib/fancy_urllib
) doh!

Thursday, December 2, 2010

Google App Script: Experiences in Cloudland

I'm a bit of a fan of Software as a Service and Google Apps. It seems to solve a lot of problems allowing me to work with bigger building blocks and to focus on solving domain specific problems rather than writing another document storage solution or emailer service. So the announcement earlier this year of the availability of Google App Script (GAS) caught my eye. GAS is server side javascript plus a stack of libraries and which forms part of the suite of tools making up Googles Apps. My first thoughts on seeing the introductory webcast were:
  • This looks powerful - it's the glue between SaaS
  • How do you test drive it?
  • Is it just Spreadsheets and VBA for the cloud?
  • Is it a quick win with lots of pain down the line?
My experience with javascript in the past has always been with code which hacks about with the DOM in a browser or, even worse, implements business logic in the UI. I had not appreciated how elegant and powerful it can be. GAS comes with some standard libraries to do things with google apps like send email, update calenders, create documents and build sites plus some low level stuff like http fetches, parse xml, consume soap services and build UI's. All this makes it a very compelling offering and one worth a try to put it through it's paces. There is one notable omission from the libraries and that is access to the google data services which meant that there was no easily available persistence mechanism.

The project to try GAS out was a simple site inspection application where an inspector can be allocated an appointment to inspect a premises, complete an online inspection report and a reviewer can look at the completed report and make additional comments before generating a summary. This project would exercise a few of the available libraries.

There were a few false starts to do with persitence. Trying to use a spreadsheet as a relational database (doh!) and using the simple key-value store in Big Table every script gets for free were both discounted early on. Also I discovered a couple of useful SaaS applications:
  • SurveyGizmo which seemed to do everything we wanted to do from a inspection report UI perspective. it does not have any workflow but does have a decent XML based API. This removed the need for us to create our own inspection report builder.
  • Cloudant which offered an instance of CouchDB (a RESTful document database) which allowed us to simply use the GAS Http Fetch libraries for persistence.
The approach I used was to use a script to build and display a UI for scheduling an inspection and send an email to the inspector containing a link to the report to be completed. The UI was displayed within an iframe of a page within a dedicated google site. There were also other pages containing iframes: one which displayed a list of surveys and their status (pending, awaiting review & reviewed) and another which displayed an individual survey response.


In a conversation with a colleague I was pointed at an excellent book by Douglas Crockford called 'Javascript: The Good Parts' which turned out to be invaluable and which completely changed my perspective on javascript. Coupled with this I had also recently attended Software Craftsmanship 2010 at Bletchley Park during which I had attended a session entitled Understand a language better – Test Drive your own Unit Testing Framework by Michael Hunger which was a real eye opener for me and answered one of my original questions (Q: How do I test drive this, A: By writing the gasUnit test framework!). Reading the book, writing the test framework and using the excellent jslint all together made me feel much more comfortable coding and resulted in tested code with clear separation of concerns using MVC.

Despite my enthusiasm for javascript as a language I had a number of problems with the GAS:

  • These are a scripting layer on top of the GWT and allow you to programatically build UI's.
  • It is marked as experimental and it is not a full implementation of all the features of GWT. For instance styling seemed a little clunky and there was no way of rendering pure html.
  • Both IE and FireFox threw errors for some of my scripts which rendered OK in Chrome - this is very worrying since one of the purposes of using GWT is to remove concerns about cross browser compatability.
  • Pages also took an age to render (presumably because of the rendering javascript download and execution) but once loaded the interaction with the page was very responsive
  • The page source looked as ugly as sin but have you looked at the page source for the google home page recently!
  • I think some of my problems were more to do with the change in mindset from pages and http post/get to one based around an ajax based web application.
  • Each script needs to be stand alone so it is not easy to reuse code. For example, in order to reuse the test framework code in each script I placed the framework in a Google Code SVN repository and pulled in the file using a fetch + eval. This worked OK but is not ideal, probably hinders performance and is considered evil by jslint.
  • The script editor was amazing considering it is a browser based application. It seemed to cope with intermittent networks OK, recovered code following disconnects, was very responsive on a good network and produced a very fast red-green-refactor cycle when doing TDD. Debugging required tracing using the Logger library which worked OK. Refactoring and navigation is very primitive. The revision history for a script is not a full history (~ last 20) so is of limited value when making frequent changes.
  • There doesn't seem to be a very active community - I posted a couple of general questions (well phrased I hope!) which were never answered.
All in all GAS worked very well. The UI library is still very beta and we decided not to use it as a solution at this point but will return to it at some point. It was a great tool for quickly prototyping a workflow without having to worry about servers or deployment. I still have some concerns about integrating this into a managed source control, build and release process - I don't want to be part of a 'deploy and fix' hackfest.

There will be a separate post on my experiences of javascript and of developing gasUnit.

Wednesday, September 22, 2010

Entity-Attribute-Value models

Just listened to Ayendes podcast on RavenDB, a .NET document DB solution, at Herding Code ( A long podcast but worth the effort with some interesting stuff especially around dealing with model changes over time, event sourcing and the sweet spot for usage. Ayende mentioned Entity-Attribute-Value models as particularly appropriate for a document DB which caught my attention as it sounded very much like the approach we had taken at PAI. So I started digging around and found the following:
The implementations are focused on persistence using a relation DB but with a document database this should all just happen. I think it is time for us to switch out our long thin relational model for a document database - probably RavenDB.

Edit: Just found this useful blog post which dissects an MVC/RavenDB app and comments on best practice:

Friday, June 11, 2010

Reflections on tonight's agile testing uk meetup

Just completed a very interesting practical exercise with Gojko Adzic at the agile testing uk ( event hosted by skillsmatter in London. Worked well as a group exercise and a very well attended meeting (~60 people).

Looked at a fit test and discussed why it was bad:

* Lack of focus
* Lack of clear definition of business rule(s) under test
* Included technical/software terms terms, db ids and references to fixture classes.
* Non locale specific terms (check vs cheque)
* Some irrelevant data

Key things I got out of it:

* Should describe behaviour for business rules - specification not a script
* OK to include non business rules (e.g. ordering of results) to make testing easier as long as it doesn't conflict with existing business rules. May also prompt discussion around rules
* Non-deterministic ordering is a particular problem which causes test failures and sometimes removal of tests by developers
* We missed an obvious business rule - do we send cheques for £0 (eg someone on the payroll who has not worked during the period in question)
* Feel much more comfortable with Given-When-Then (esp with tables) vs Fitness
* Most teams expressed acceptance tests using to Given-When-Then rather with tables - teams which did seemed to get into difficulty

Need to take another look at the slide stack to see if I have missed anything.

Saturday, March 20, 2010

Some reflections on Martin Fowlers presentation at AgileYorkshire this week.

During Thursday's commute I read a Martin Fowler blog post - a great piece on the maturity of REST implementations (Richardson Maturity Model). It took about half an hour to read, re-read and digest. The information in it could have been a candidate for one of the presentations making up Martin's pick and mix selection from Wednesday. I could easily find 2 other blog posts of his which could be similarly described. So if I can read a few well written blog posts on an interesting subject why should we go through the hassle of organising a night, getting a speaker to travel 'oop North and having 100 people spend some of their valuable social life at a geek night?

For me the value of evenings such as Wednesday is only partly about the content. Equally important is the sense of event it generates and, in particular, the atmosphere of discussion and enquiry it fosters. It was a shame we had a tight deadline to keep to which meant that the time for questions was limited but even so the buzz both beforehand and afterwards was palpable. The presentations by Martin on Wednesday were great seed topics for the resulting discussions which occurred during the evening and will have continued for some days afterwards. As a case in point, I have just had a conversation about it with a non-technical friend who stopped by this afternoon.

I think it is true that developers - and by that I mean people who spend a large part of their working week turning user stories into delivered features - as a whole do not blog. They may document the solution to a problem they have found or throw out some observational/controversial stuff in the run up to redundancy or contract renewal, but the number who blog as reflection is small. For a profession which has a reputation as being "socially awkward" I have found that the preferred communication medium of good developers is a good ol' face to face discussion. This is odd in that some of the discussions around the topic of Martin's final segment (gender skewing in development teams) invoked the "autism" argument. This is that on an autistic spectrum there is evidence that males have a higher incidence than females ( and since there are more male developers than female developers then, ergo, autistic tendencies must be required to be a developer. I think this an example of a converse error).

The irony of observing the intense social interactions between developers arguing passionately and empathetically about the subject of where they lie on an autism scale was not lost on me. We were still discussing the subject matter the following day and I expect the themes will keep re-occurring.

So are social events like Agile Yorkshire events necessary or could we have achieved the same outcome by other means; comments on a blog post, VOIP meeting, a video link presentation? Asides from the importance of an influential figure like Martin acknowledging grass-roots community groups by presenting to them I think that a physical meet-up is important. This is because of the quality and effectiveness of the communication it encourages and in the end a developer who can not communicate effectively is not a good developer.

Thursday, December 10, 2009

Kanban Presentation at Agile Yorkshire and NHS IC

The end of year event seemed to go well - it was good to be in a posh, well equipped venue where we didn't have to strain to hear the speaker. The food from Anands was great but my catering estimation technique leaves much to be desired. I thought about how much I usually eat and drink and then multiplied it by the number of expected attendees. Obvious to me now that I lie above the mean.

I was very impressed with David's Kanban Overview and Experience Report - a polished and compelling presentation. The combination of the two common sense ideas in Kanban; making the working state of a team clearly visible through the use of a Kanban board and reducing task switching by limiting the amount of work in process is very attractive to us. It was these two simple things coupled with the collection of simple metrics such as lead, development and engineering time across many tasks which has me hooked. The ability to use this data to quantify the effects, positive or negative, of any adjustments to the system (increased WIP limits, more team members). The ability to use these stats to the team's benefit is significant to me - the simplicity and volume of stats seems so relentless and very difficult to refute. In the past development teams have had gut feelings about whether process/environment changes are good or bad but this is difficult to sell to managers. A Kanban approach seems to give more bargaining power to the participants in the value stream by providing a ton of useful information without over-burdening them with meaningless time sheet style data collection. I've already sourced a mechanical date stamp (as found in libraries) to mark up the transition dates for tasks!

The following day David and Peter came to visit us at NHS Information Centre (NHS IC) where they participated in an introspective by the some of the development team. This was a useful exercise and revealed some interesting problems - none of which were particularly unusual for a development team. It was a good way to focus attention on the good and bad parts of our current process prior to Davids presentation to the NHS IT meeting later that day. The presentation was very well received across the board from senior managers to developers. By the following morning several teams had already made a start on creating a Kanban board and were questioning how their work was structured. Interestingly, they were keen to identify data to collect in order to generate statistics. This level of keenness for time-recording is not common for developers and hopefully reflects a desire for process improvement rather than sticks to beat others with!

Our team (an established team running along broadly xp/scum lines) very quickly had our kanban board up since we were using a scrum board in Mingle and the transition was relatively painless . What occurred next surprised me as it seems our iterations just vanished. The PM removed the current iteration and after a panicky moment we just shrugged and carried on with no fall as yet as we go with flow.

I have had an uneasy feeling about our sprints recently. For the last few as we had been operating in "release" mode where we had a list of stuff (mostly small "tweak" stories and defects) in priority order. Detailed story point estimation wasn't of much value, we were releasing to UAT very frequently (I'll try to blog about this separately). So when the iterations disappeared I felt like we had gained more than we had lost. Mostly we had a gained a more honest view of the way we were working in practice. We plan to release fortnightly but my guess is that this will converge to a release per feature - this is easier in a web environment.

Coincidentally, we also had a planning meeting on this day too. The planning around the Kanban board placed the focus on the only attendee from the team of 3 customer proxies. He clearly felt uncomfortable making decisions as to what should go into the input queue and the resulting prioritised queue may end up being adjusted after he has discussions with the others. Unlike our previous approach where we planned on a Monday for the following two weeks the flexibility inherent in the Kanban prioritisation process means that we have some leeway in the timing of planning. In some of the prioritisation discussions it became clear that for some features the discussion may need to involve more customers which raised the question as to who really owns the input queue. This is probably not the most shouty and may also change over time. Need to think more about this.

I was aware that there was no real estimation going on during the prioritisation process - specifically we didn't get to play planning poker. Kanban seems to encourage the use of T-Shirt sizing and the collection of data to give lead time values based on historical data values for those T-Shirt sizes. Story point estimation can be used but is often not felt to be of value. This is an area where we need to be clearer and will probably start with simple T-Shirt sizing.

This is as far as we have got will post further updates as we progress and refine.

I'll also post on Peters excellent "Why Do Coders Code" exercise separately.

Wednesday, September 30, 2009

Extreme Tuesday Club

Last night I attended my first ever Extreme Tuesday Club (XTC) event. XTC was one of the influences behind the setting up of AgileYorkshire for some of the original founders at Erudine. I was down in London for sys admin course so it was good to be able to go along and see XTC first hand. They have a good setup (although they are having to up sticks and find a new venue). The evening was combined with a presentation by David Anderson on Kanban which, for me, was fascinating. In my current job (Lead Dev) I am trying to direct things along in a traditional agile direction. As a team we already practice TDD and CI with some pair programming. I am trying to get us focused on iterations and the ability to deliver potentially releasable product (all agile buzzword bingo terms!) at the end of an iteration. The presentation made me feel a little uneasy. I am aware, at a high level, of the ideas behind Kanban and Work In Progress (WIP) limits but felt if we could get our iterations going smoothly then we would have a bit of a breather and then we could start moving towards a process based on WIP rather iterations once I understood Lean.

David seemed to be suggesting that implementing WIP is the first thing you should do. I need to read and discuss this a little more. I didn't manage to copy down the points but the essence of it was improve your existing process - don't bring in new processes wholesale. By simply implementing WIP and giving visibility through a physical Kanban board (augmented by an online system for reporting and organisational visibility) you will get process improvement which will lead to more organisational maturity which in turn will make it more likely for bigger process changes to be successful. The decoupling of the planning, development and release schedules seemed to be really important (they are all tightly coupled with agile iterations) as it is a dependancy which, if it existed between software components, would be removed during refactoring and which is something I need to think about.

Davids assertion was that if you implement simple changes to existing systems then you will get benefit earlier with minimal resistance and that given time the people you interact with will be asking you for ways that they can change too! It felt strange hearing the terms Software Engineering, IEEE, CMMI being mentioned in a positive way at an agile evening. I can guess that stuff like this might rub some in the agile community up the wrong way.

Other than the presentation I managed to talk to Steve Freeman who is a (?the) founder of XTC which was great and hopefully we can start talking to him about his experiences, some ideas and joint publicity. Also played a Kanban game to show the effects of batching work in a pipeline ( which was great fun and would go down well at AgileYorkshire. I might have persuaded a few people to make the trip up to t'frozen north to present/participate in AgileYorkshire but it might of just been Beer Bravado :)

All in all a great evening and, as usual, came away with more questions than answers but with some insights into Kanban and the feeling that I need to do more reading/talking and, if I do, it should move me forward (especially if beer fueled).