Alec the Geek

The Open Source Developers Conference (Melbourne Australian this December — book your tickets now!) always has many excellent presentations from very knowledgeable experts. Sometimes however speakers could make better use of their speaking skills to enhance their excellent technical content.

So this year the committee is offering training to the presenters to help raise the standard even further — the training is being offered by Perl Training Australia and HeadWest Software. I am pleased to be representing HeadWest and I have started preparing my material on a public WiKi, please feel free to drop by and leave comments.

It feels a pretty exposed way of preparing material

In a previous post I talked about a Lightweight Ticket Process, but glossed over how we actually identify the steps we need in our process.

It’s important to note that I am talking about the process we apply to single set of changes, not the high level process we might apply to a complete project. I have referred to this in the past as a “Ticket Process”, other people have used the term change set. However it can be any sort of change or piece of work.

Remember the objective of the Lightweight Ticket Process is to have a process that is as simple as possible, but no simpler. Some of the issues to investigate:

1. What is the desired final outcome, purpose or product from the process. e.g.
   1. The creation of a review document notes
   2. The fixing of a bug in the current development
   3. The generation of revenue by implementing a custom feature
   4. Asking a question
2. Where in the Application lifecyle was it initiated and by whom e.g.
   1. Project Initiation
   2. Production support
   3. User Acceptance testing
3. What pieces of work need to happen to product the desired outcome? e.g.
   1. Code changes
   2. Rebuild of software
   3. Change in specification
   4. Change in budget or other piece of project documentation
   5. Deployment onto a production environment
   6. An update to a build system configuration
4. What role is responsible for performing the work. e.g. developer, project manager…
5. What reviews and approvals need to occur before and after the work is complete, e.g.
   1. Review of cost, benefit, impact and risk
   2. Approval by Change Control Board
   3. Code review
   4. Sign off by Quality Assurance team
6. What roles are responsible for performing reviews or authorisation.
7. Any process requirements from the customer or other external entity e.g. the FDA

Make sure that you cast around for the following sources of information

1. This projects current processes, documented and undocumented
2. Organisational processes and standards
3. Similar projects in the same organisation
4. External standards such as ITIL

Now comes the hard bit.

You have followed the activities above and come with a beautiful complete process. Unfortunately it is far too complex. So now we simplify, which can be difficult to do.

1. Cut out “information” steps and roles
2. Determine which of the outputs from each process step are useful. What are they used for? Nothing obvious? Then cut it out
3. Be brutal and cut out any thing that not essential and be severe with your definition of “essential”

It is better to start with too simple a process and improve it as the team understands more, than to start with something hard to use. Unfortunately it is very easy to give a team the the latter as the
siren call of “complete process” is highly seductive.

The Subversion developer community recently held a summit which is summarised on The Subversives blog. It looks some cool stuff could be coming down the pipe in the 2.x series — looking forward to it. Will this make svk obsolete?

powered by performancing firefox

I mentioned in my previous post the phrase “Lightweight Ticket process”. Let’s look in a bit more detail about what that might mean.

• Tickets are raised for a variety of reasons (e.g. feature request, bug report, request for clarification) throughout the Application Lifecycle (e.g. Project Initiation, UAT, production support…). No single process will answer all these needs and specific processes will need to change over time so be flexible
• Clearly identify what information you need to record in each of the processes. Broadly this can be grouped into two areas although there is a large overlap between them:
  • information required to support the currently active process (e.g. risk, benefit , cost, impacted subsystem, severity, schedule, review comments …)
  • Information required to support metrics (e.g. impacted subsystem, actuals vs. estimates, severity …)

  Because this is a lightweight process we gather the information we need, but no more

• Make the process as simple as possible, but no simpler. Consider the Magical Number 7 as a rule of thumb . Remember that we want a lightweight process, but we still have certain minimum requirements - the trick is working our what they are. In general processes will contain some variation on
  • Gather the initial request
  • Ensure that all required information is present
  • Review the information for benefit, costs, impact and risk, and possibly approve further work
  • Carry out real work i.e. change something
  • Review the work for correctness
  • Implement the change (e.g. roll out to production)
• Document the process in some fashion. A process that is not documented can be very hard to talk about, agree and improve. Brevity is the soul of wit, however the document should cover
  • Purpose and final outcomes
  • Rationale
  • Entry and exit gates
  • Entry and exit conditions on each stage of the process
  • The role responsible for each stage of the process
  • Brief description of the activity at each stage
  • Abnormal process stages or exit gates
• Look for some common gothas
  • No clear separation of roles, for example the people who request a change also approve it and implement it (this tends to make auditors jump up and down a lot)
  • The same role responsible for two consecutive parts of a process often indicates a poor process, but not always.
  • Beware of ticket changing identity half way through a process, for instance a bug report that morphs in a feature request. Depending on the features of your tool, change the ticket so that it becomes the correct type (on the correct process with the correct data capture fields) or close the ticket and open a new ticket of the correct type, ideally with a link back to the first ticket.

Recently I had an interesting conversation with a business partner about their software development process and areas for improvement. Whenever I have these conversations the quick wins are often the same:

• When doing changes ensure all code changes are booked against a ticket in your ticket system. Make sure this is enforced by the software if possible
• Make sure you have a consistent (but lightweight) ticket process that is applied to all changes. Try and automate the process flow with tools if possible
• Provide a coherent, automatic, build process. Make sure that all test candidates are build on environments that are controlled — not development environments. Tools such as CruiseControl can help. Consider placing built files under version control, providing adequete traceability information can be stored in the repository.
• When creating files for a release make sure that you can show:
  • Which source versions where used to build the software
  • How the software was build (on which machine, when, what 3rd party tools where part of the build process etc)
  • Which set of changes (tickets) are part of the release

So make sure your tickets record the correct information, that you install any automation points you need, have a documented process and spend the technical effort to create and maintain your build infrastructure (that can be non-trivial). I’ll try an post more about these topics in the next few days.

All decisions should be fact based and in software this presents a problem because facts are often elusive. The answer is of course to use metrics; however I learnt in college that information must be:

• Timely
• Useful
• Accurate

The Agile methodologies promote metrics as an important part of the planning and estimating process. However software creation and testing is only part of the process. We need metrics that address the whole of Application Lifecycle Management (ALM), which is a daunting task because of the potentially large amount of information and scope implied.

However by the simple process of:

1. Using a ticket management and capturing all issues, not matter where in our lifecycle they occur
2. Following a defined process, or processes (usually one process does not fit all needs), to address our issues and using our ticket system to track the process.

we can capture some valuable information which we can use in a variety of simple ways to obtain metrics.

So what are useful ALM metrics? Here are some example suggestions.

• Rework rate and costs — by documenting each issue via the ticket system we can get some idea of how much we are spending on rework. In particular we can capture information to tell us:
  • Where we are finding issues (in UAT, Regression or production). This may tell us that specific parts of our processes need attention (e.g. UAT or Unit Testing)
  • Which sub-systems require most rework. This might imply that some parts of the system under discussions are fragile and need to be re-architected
  • Classification of root causes (e.g. Poor requirements/user stories, poor unit testing, etc)
• Process instrumentation. By looking at the amount of time spent in each process we can find efficiencies and cost savings
• Trend analysis, which allows us to spot potential problems before they become too large and test our assumptions about where we should deploy our effort. It is possible to trend all sorts of data, provided that the information is initially captured in our system (e.g. Severity, cost to fix, time to fix, business feature impacted, time to test etc.)

However the core principal is that it should be simple and useful. Also note that as the lifecycle progresses, and remember we are also talking beyond the lifetime of a single development project, which metric is useful will change and it will be necessary to re-tool the metric reports. Make sure you capture that in a ticket as well!

And finally remember, “You get what you measure“

powered by performancing firefox

Like a lot of Geeks I have a Gmail account and love the searching power it has. On my laptop I have the excellent Thunderbird MUA with the GMailUI extension. However it’s not as good as GMail for searching all my email very quickly.

That would be not a problem if I only had one email account, but I have three in active use. So I have devised this little hack to have the power of Gmail search across all my accounts.

• Set up Thunderbird to BCC my Gmail account with a custom Gmail address, e.g. alecthegeek+othermail@gmail.com when I send email from my othermail@myotheraccount mail account.
• Then set up a Gmail filter so that anything from othermail@myotheraccount is archived and has a label applied so that I do not get my Gmail inbox cluttered up.
• Create a filter in Thunderbird that deletes all GMail where the Deliver-To field is alecthgeek+othermail@gmail.com. Thunderbird is still keeping a copy of my sent emails in my Sent email folder of course, but does not clutter up my local inbox with copies of emails I sent that got blind copied to Gmail.

The downside is that you are giving Google even more information about yourself and what you do

powered by performancing firefox

We went to see the movie An Inconvenient Truth this afternoon. It was a fascinating piece and I had better say up front that I have been a “greenie” for many years (although not a very good one) and fully support the ideas in the film.

I have not been active in the green movement for a long (moving to a new country, having kids etc) and so I am a little out of date on the current research, given that the mainstream press have been fudging the issue for so long.

It is certainly a very uncomfortable thought that we are so far down the ecological disaster path and the film throws into stark relief some unpleasent facts.

I would urge everyone to see the film, make whatever changes they feel able to in their personnel lives (or as the UK Lifestyle movement say Live simply that others may simply live) and use the politcal process to encourage changes over the things that they cannot control directly.

Twenty five years ago we marched against the bomb and apartheid, now it’s time to march again.

powered by performancing firefox

• If in doubt, cut it out. (If you are not sure or clear about something you have written, remove it)
• Assumption is the mother of all cock ups (says it all really)
• The devil is in the detail (You have to drill down to the lowest level as that is where the assumptions and problems lie — helicopter views are only useful for a short time)
• Bad news don’t get a better (Flag issues and problems as early as possible, otherwise it just gets worse)

Updated May 2008

As mentioned by Andrew below this is all too hard now that we have the chere utility. At the cygwin shell prompt try the following

chere -i -c -n -t rxvt -s bash -o -sl 2500 -fg lightblue -geometry -geometry 80×25 -bg midnightblue -sbt 10 -title bash” -e “Open Bash shell here”

If you want a shortcut to run bash for your home directory try

C:\bin\run.exe C:\bin\rxvt.exe -sl 2500 -fg lightblue -geometry 80×25 -bg midnightblue -sbt 10 -title bash -e /bin/xhere /bin/bash.exe %HOME%


——————————————————————————-
For users of Cygwin on Windows who like the Open Command Window Here power toy

I took this hack and this hack and combined them to make the following hack. It works on both directories and drives in Windows Explorer, and creates a visually attractive login shell (at least I think so).

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Directory\shell\bash]

@="Open Bash Shell Here"

[HKEY_CLASSES_ROOT\Directory\shell\bash\command]

@="c:\\\\bin\\\\bash.exe --login -i -c 'cd \"`cygpath \"$*\"`\";exec rxvt -sr -sl 2500 -fg lightblue --geometry 80x25 -bg midnightblue -sb -e bash' bash %L"

[HKEY_CLASSES_ROOT\Drive\shell\bash]

@="Open Bash Shell Here"

[HKEY_CLASSES_ROOT\Drive\shell\bash\command]

@="c:\\\\bin\\\\bash.exe --login -i -c 'cd \"`cygpath \"$*\"`\";exec rxvt -sr -sl 2500 -fg lightblue --geometry 80x25 -bg midnightblue -sb -e bash' bash %L"

The following may be useful for people looking for CM information:

• UCM
• CM Crossroads forums
• The CM Body of Knowledge
• CM WiKi
• CM Yellow Pages
• SCM Patterns for Agility
• SWEBOK
• SEI SCM area
• Software Program Managers Network

The other day someone aske me for some pointers on getting software development processes in place. I did it a bit of a hurry but I was able to come up with this list

• MS Build Master. Note that this does not address Change Management and is a little simplistic in places.
• SWEBOK
• Eclipse Process Framework
• CMMI models and perhaps more usefully a copy of the handbook
• ITIL is also worth a look for process suggestions – some can be adapted for Development.

Eclipse have recently released 1.0 of the EPF, which peeked my interest as a long time process consultant and documenter. I see it as potentially useful in two ways

• To help me work smarter and deliver better documentation
• Something we can provide in our ALM products to help our customers achieve better outcomes

EPF consists of two core “products”, a set of web pages containing ready to use process (e.g. Open Unified Process/Basic) and  EPF Composer that allows you to modify and create process documentation.

•  The OpenUP/Basic (portrayed as an Agile process) warrants study. For people used to methodologies such as XP there is sufficient difference and new material as to require reasonable analysis and evaluation. Given it’s ‘thud’ factor and IBM pedigree it may be a way to introduce more iterative process to skeptical management
• The Composer tool is a standalone Eclipse RCP client and, on Linux, requires a Mozilla browser with GTK2. The tool does not install as a plugin to your existing Eclipse.  Many thanks to Kelvin Low for the help

I will coming back to update this as I understand more about EPF

When I first became self-employed I very diligently (well, fairly diligently) researched the market for a laptop machine and was delighted to find a HP nx6125 for a neat price (A$1700), I also researched Linux compatibility and whilst there are issues, by and large it works. However there are some things that continue to really annoy me.

• I cannot hibernate or sleep. I suspect a problem with my video driver although I appear to have the ATI driver installed correctly and other people are reporting success with sleep mode.
• I have USB headphones that work perfectly with Skype and ALSA. However all other sound (e.g. Music CD’s) insists on coming out of the speakers. BTW, if you have audio jacks on your PC don’t buy USB headphones — they are bulky and take up a USB socket
• It took me 6 months to get Wireless working :-(, but at least it works now. Thanks to the Ndiswrapper team for saving my life
• I cannot get the sychronisation for my Pocket PC mobile phone working
• Nothing to do with Linux, but because of cost I compromised and accepted a machine with a trackerpad instead of a G spot. After 6 months I can say that the IBM G spot (also available on some HP machines) is far superior.

At the ThinkingLinux conference yesterday there were two presentations by the CEO and CIO of wotif.com. The messages I took away:

• Always test your assumptions. Make sure you validate what you are delivering to your users by using it yourself in the same environment
• What works today should work tomorrow, however it is unlikely to work as well next year and the year after. Review your processes and plans on a regular basis
• Work closely with the business. Use melded teams, give the business very early visibility into what you are building. Be Agile if appropriate
• Keep strategic development close, but outsource non-core functions if there is a benefit.
• Invest time and energy on recruiting and keeping the right people; and maintaining your business partnerships. It’s crucial
• Plan for the future, but only invest and implement what you need now
• Keeping your cost base low can allow you to keep competition away by making the market less attractive and harder for new new entrants
• Early cash flow and reduced borrowing are good things and make your business attractive
• Word of mouth is the best and lowest cost form of marketing, but the most difficult to get
• Make decisions based on data and test your assumptions (yes, I know I already said that, but it’s important)
• The user experience is key — make sure you (and everyone in the team) experience it directly for yourselfs (that’s worth repeating as wel). Make sure that the experience remains as good as possible under peak loads and during failures
• Provide a user community — e.g a market for for products and services, the exchange of ideas etc.

All pretty much common sense in many ways — but easy to forget in the daily trench warfare that is current business reality for many of us.

A few days ago I was a little sniffy about the ThinkingLinux conference, to which I had wangled a free ticket just by being in the first twenty to telephone a number in an newspaper ad. Having attended today I have to say it was very useful and well worth going to; it was admittedly a vendor event, as opposed to an industry conference, but still worth going to - if you booked before 2/Oct it was only A$99 anyway.

I shall be posting some more details over the next few days but here is my high level brain dump:

• Alfreso supports document management, embedded processes, version control etc. It can be used from Windows clients using CIFS.
• Pentaho has some neat data mining and dashboard technology. Compare to BusinessObjects.
• For software development process (a subject dear to my heart) consider ScrumWiki (I wonder if there is an Extreme Programming WiKi?)
• Eclipse have just released version 1.0 of their Process Framework, which includes the Open Unified Process (head for the hills!) and EPF composer so you can customise your own processes which sounds much more useful. Currently the composer creates web pages that document the process, there is no process automation engine, whatever you might guess that to be.
• TWiki might be highly configurable
• BackupPC is a neat way to backup Linux and Windows machines across a network. Minimises back up sets and it’s easy to restore (files copies from a web browser)
• Jabber can be used, not just to receive status messages, but also to send commands to software agents
• XEN is a sophisticated piece of server management technology for failover and server migration. I’m going to use it on my laptop for development.
• The award for the coolest acronym must be STONITH

powered by performancing firefox

I just notice that Mozilla Thunderbird’s new calendar extension, Lightning has been released (well 0.3 anyway). As a productivity pr0n additct I am looking forward to kicking the tyres. However first I will have to build it for my 64 bit platform, but it’s too far down my list of next actions

powered by performancing firefox

My son, Jack, will be moving from primary to secondary education in January (the academic year starts in January here). Today we got the first information pack from the new school, which was very exciting. The next thing I know he’ll be ready to leave home.

After Simon’s interesting talk on Thursday I got to thinking about using the correct tool for the job; is it better to use smaller specific tools, such as sed, or one general purpose tool like Perl.

Using sed, for example, to perform text transformation does have a couple of advantage

1. Performance — the specialist tool can often complete the job a lot faster

2. A more appropriate way of expressing a solution — specialist languages, such as sed, can provide more elegant way of solving a particular problem. As Simon pointed out, this may actually suggest a completely different and better way to write the program.

However there are two counter arguments.

1. The amount of mental effort required to learn and program in, possibly many, different mini languages. This can make maintenance in particular very expensive

2. Over time programs often grow and become more complex. Mini languages can have trouble scaling with this complexity and elegant solutions can become nightmarish very quickly. It was interesting to see some of the larger example scripts on Wednesday, they made the worst Perl script look elegant and expressive.

So what is the answer? Well in true consulting style the answer is ‘It depends’.

Small jobs with very fixed scope may well benefit from the use of specialist tools, in particular when these jobs are running frequently and performance improvements accumulate each time it is run.

However often several different small tools have to tied together into a script and very quickly it becomes apparent that a more powerful tool such as Perl (or Python, Ruby etc.) should have been used. Unfortunately if can also become an exercise in self indulgence to see how many ‘clever’ solutions can be packed into a single script.

powered by performancing firefox

I’ve been a little quiet on the Internet over the last couple of days as the recent wind blew the skylight off the roof and smashed it :-(. It is very surprising how hard it is to:

• Purchase a replacement skylight — there are no standard sizes
• Find someone who can obtain and fix a new skylight

Anyway — I have done a quick hardware hack and hope to get a better solution real soon.

powered by performancing firefox