Asset Pipeline Organisation

MIQly (our soon-to-be flagship e-assessment product) is a Rails application, but could be classed as 3 large-scale Javascript applications (setting questions, marking assessments, providing student feedback). As it exists in the Rails 3.1+ ecosystem, the Javascript (written one-step removed as Coffeescript) lives inside the asset pipeline. I can already hear the shudders of dread [1] from many Rails people at this thought, but we managed to make the asset pipeline work for us. 

This is how.

The out-the-box organisation of assets within Rails provides very little control over the ordering and packaging of JavaScript and CSS assets. Problems that follow from the standard and rather painful approach of a default, single application.js manifest file become apparent as an application grows, especially with one default line in application.js:

//=require_tree . 

This one line causes major headaches with indeterminate code ordering. This is especially apparent if the assets are developed on Mac OS X and then precompiled on a Ubuntu machine, as these have different default ordering of files loaded from the filesystem with respect to capitalisation. The result is painful manual resolution of dependency ordering problems, and frequently ends up with Sprockets (the library that does a lot of the heavy lifting for the asset pipeline) being criticised for what is ultimately a code organisation issue.

We needed to impose a more suitable structure within the asset pipeline to avoid having to repeatedly perform manual resolutions as our code base grew.  We considered the use of a system like RequireJS, but the integrations were immature at the time of consideration, and the asset pipeline itself provided advantages in terms of an integrated build process that we didn't want to abandon. As such, we wanted a solution that worked well within the asset pipeline rather than attempting to force a different solution on top of the pipeline.

Our solution is mostly organisational and not tooled beyond the Sprockets tooling already available. The key part is in recognising and distinguishing two distinct types of sprocket require directives and two different types of files. These get paired together as follows.

The first type is the 'typical' rails usage, with a manifest that pulls in various files into a precompiled asset. These are something we term an 'assembly' and defines a compilation unit. The items that get pulled into such an assembly are components, not lower-level units of composition. The initial application.js file in a new Rails project is an example of this (albeit a poor one, as it uses require_tree rather than a more stable dependency declaration method). One of our manifest files looks like:

The second type of file is an implementation file. These also have sprocket directives that will pull in required dependencies. For example, a screen widget item will pull in lower-level widgets and components (such as autogrowing text fields, or in-line editor code). The lower level code will also pull in its own dependencies, eventually 'grounding out' in either dependency-less code, or base libraries like jQuery  (n.b. in miqly, jquery is pulled in with a separate sheet to avoid certain issues with inter-manifest conflicts). One of our implementation files looks somewhat like:

This approach lets us assemble various components into compiled files and be sure of the dependency loading and code ordering within these files with very little effort: We have the asset pipeline resolving all our dependency ordering issues for us across several thousand lines of coffeescript and CSS.

[1] These 3 links were just picked after skim reading some results from the query 'asset pipeline criticism'. Feel free to provide more criticisms (or praise) in the comments :)

Time for some paper prototyping.. Tooltips everywhere!!

Recently, as a team exercise, we sat around around a sketch book to decide where to place helpful tooltips on our interactive marker application. With a little cutting and writing this is what we came up with:

Conventionally, designing a set of tool tips like this might result in a long and boring email thread discussing specifications and showing edited screenshots, or a couple of hours of  design activities (using a browser inspector, whose contents could have been lost on accidentally pressing back or refresh).

Instead we did this as a fun  visual exercise that we finished in less than an hour with everyone understanding what was needed on the page. 

Avoiding developer interrupts

I'm going to start posting here about our lean and agile development processes. They will fit more here than on the main Hedtek blog. A fitting start is the elimination of waste and in particular waste caused by developers not being 'zoned in' and focusing totally on development processes. Often this is called flow (Wikipedia has a great background article). Being in the flow has two aspects, of how to get into the flow, and avoiding interrupts which jerk a developer out of the zone or flow.

Positive ways of getting into the flow is something we haven't touched on so far, although we are aware of the need for and strive for a good working environment with relatively good levels of quietness. What we have been actively working on (with a good level of success) is avoiding unnecessary interrupts.

What is the problem with interrupts?  Don't you just hate it when you are six levels deep in code and your manager walks in to say to the team: "Hey guys, don't you think we should transmogrify the hubbleity widget?" Instant loss of concentration and exit from the zone, only to struggle to get back to where you were.

One research study (blog post, research paper) examined developer behaviour of 414 developers in 10,000 programming sessions and found that for developers in the sample

• Devs took 10-15 minutes to resume editing code after an interruption.
• Devs were likely to get only one uninterrupted 2-hour period for development in a day.

Both of these are alarming statistics. Avoid interruptions at all costs. As a practical measure we identified what each of our devs was doing when zoned in or not zoned in, and now we just look for these behaviours  before asking something. For example:

• May be zoned in: Wearing headphones can be a sign of being zoned in.
• Definitely zoned in: Not looking at ancillary sources for information on the web, and only focusing on and flicking between vi buffers or IDE tabs.
• Not zoned in: Looking at a mobile (cell) phone.

Better still, we have adopted gmail chat and email for question requests, knowing that a developer will ignore these until having left the zone. Similarly, to remove more potential interrupts when I need to disappear from the office, theres a Where's Mark corner on one of our whiteboards, I simply write where I am and when I'm expected back there, and avoid the need to announce that to the dev team when I leave.

Of course pragmatism still rules, we can still interrupt, but we only use that for emergencies.

Additional sources

• 7 Tips for Programming in the Zone

• Keeping Developers in the Zone

• The Flow - Programming in Ecstasy

• Respecting the Zone

Testing Strategy part 3 - Acceptance Tests

After looking at code coverage and unit tests, I'd like to jump to the other end of the testing spectrum and examine what sort of Bayesian evidence an acceptance test provides. Again, this is skirting around the central question of "How can you demonstrate that your tests are correct?"

Helping others

It's an old cliche, but the saying 'if you want to learn a subject, teach it' holds a certain amount of truth. The act of ordering your thoughts and understanding about a subject allows you to spot the holes in your understanding, whether that subject is how to use a text editor or how nuclear fission works.

I've noticed this in many people, myself included. The primary case I would cite is the helpers in the IRC room #rubyonrails on freenode.net. The majority of newcomers will come into the room, ask a question, and leave (or remain silent) as soon as they have an answer. I've seen some of these people still asking the same sort of questions months or years after first appearing. The other newcomers are people that will ask some questions but will also try to answer other peoples questions straight away. These people have a tendency to develop all aspects of their development skills much quicker, as they are constantly exposing themselves to new situations that they would otherwise have never encountered.

Writing blog posts, doing podcasts, writing newsletters, helping out in a support channel. They're all ways to help teach other people and by extension help you to develop your own understanding further.

What are your preferences for helping develop the skills of team-mates, colleagues and your wider development community?

Testing Strategy part 2 - Unit tests

Following on from looking at code coverage and the central question of "How can you demonstrate that your tests are correct?" I'd like to consider another aspect of bayesian evidence - the tests themselves.

Taking the viewpoint of strong evidence and weak evidence again, I'd like to examine what level of evidence various tests provide. There are various types of test that are typically used in software development, each type provides a different level of granularity and a different balance of focus (developer-centric, performance-centric, business-centric, etc).

At the bottom of this stack is the most prevalent test type - unit tests. A unit test is a developer-centric test with a very fine granularity. A unit test is intended to be the most granular test, as each test should have an indivisible unit of functionality/behaviour as the subject.

Taking a bayesian perspective, I would characterise a single passing unit test as weak evidence of system correctness. Conversely, a single failing unit test is strong evidence of the system being incorrect. To understand this viewpoint, consider a typical case of hundreds of unit tests. Each test was written by a developer, usually with little or no direct input from other sources, with the aim of demonstrating that a particular unit of code works as expected.

These tests do not show that the system as a whole functions together, and they do not show that the system works as the customer expects it to. Instead, what is demonstrated by a full suite of passing unit tests is that each individual unit of behaviour works as the developer(s) expected, which I take as weak evidence of overall system correctness. A single failure in this suite of tests shows that some functionality isn't working as the developer(s) expected, a very strong piece of evidence that the system is not correct.

Where does this leave you? Taken in addition to code coverage, we now have two pieces of weak evidence for system correctness, but have yet to address the issue of test correctness. I'll attempt to move closer to this issue in future posts where I'll consider other types of test.

Further reading: If bayesian reasoning is a fairly new concept to you, I can recommend An Intuitive Explanation of Bayes' Theorem

Blogs

While newsletters and podcasts are both good ways of getting digests of weekly news, you're likely to want to find out news faster in some subjects. For this, having a good collection of blogs that you follow helps.

It's important to keep your collection well curated. If you follow too many blogs then you are likely to find yourself in the position of being unable to read and process the latest posts on them all. A good feed aggregator is useful for this purpose.

I use google reader to aggregate the blogs I follow. My collection is mostly curated for blogs relating to ruby, ruby on rails, and web programming in general. I also subscribe to the DZone feed, but this is less focussed on areas I am interested in and harder to keep up with as a lot of content gets pushed through DZone.

Good blog suggestions and curation tips are appreciated.

Testing Strategy Part 1 - Is 100% Coverage Enough?

When discussing testing, a question that sometimes comes up is "How can you demonstrate that your tests are correct?". One particular answer that I've encountered for this question is the use of code coverage tools.

Podcasts

Technical podcasts come in a variety of formats. I'm particularly a fan of the 'panel discussion' style used for Ruby Rogues and Javascript Jabber (amongst what I'm sure are many others).

These tend to provide about an hour of discussion, perfect for my morning commute. They also tend to meander through a particular topic, which allows for interesting side-discussions.

Ruby Rogues and Javascript Jabber also both have a 'picks' section, which lets the panelists plug anything they've found interesting recently, from code libraries, to books, even video games at times.

Any other suggestions for staying occupied on a long commute?

Keeping the Javascript Beast Contained, Part 2: Events

Last time, I talked about keeping containment within Javascript for DOM manipulation. Today, I want to talk about a different aspect of containment - Javascript events.

A small Javascript application can get away without much structure to their events. Events can 'float around' the application code and a developer can generally cope with understanding the interactions between separate events reasonably well.

Newsletters

It can be quite hard to keep up with the latest news in your field at times. We've been finding various technology newsletters useful to get a digest of some interesting weekly news.

Our particular specialties within web applications are nicely covered with a selection of newsletters from CooperPress:

1. Ruby Weekly
2. Javascript Weekly
3. HTML5 Weekly

What do you use to keep up to date?

Keeping the Javascript Beast Contained, Part 1: DOM Manipulation

Gradient, our eAssessment product, has within it several Javascript-heavy components. Each of these is structured and functions quite similarly to a document-presenter desktop UI, with a visual hierarchy, top-level messages, routing components and distinct widgets at the leaf-nodes of the hierarchy tree.

Having an organisation like this in Javascript leads to issues of containment, ensuring that each widget's DOM manipulation is encapsulated to just the screen area belonging to that widget. While in the future, the introduction of the shadow DOM in HTML5 will make this potentially easier, currently the encapsulation requires attention to detail and some HTML hooks.