Easy Introduction to Git

I’ve just had an article published by Sitepoint that aims to introduce Git to people who may not have programming experience and are more comfortable using a GUI rather the command Line. “Git: Your New Best Friend“

Top Tip: Upgrade Git on Cygwin

The current version of git installed with Cygwin (1.5.x) has at least one annoying bug so it’s worth building  the latest version (NB I have almost all of Cygwin installed – you may need to install additional development tools and libs):

1. Download and unpack the source code. Pointer here
2. Run ‘./configure –prefix=/usr/local’
3. Run ‘make test’
4. Ignore errors relating to not detecting unwritable repos
5. run ‘make install’
6. run ‘hash -r’
7. Make sure it’s installed with ‘type git’ and ‘git –version’

Update March/09 — I found more detailed instructions

Handy Hack: Comparing two directory trees on Solaris 8

After some information from jim mcnamara I developed a script to compare two directory trees. I needed something that worked on Solaris 8 so it uses basic UNIX tools (and dircmp was not very useful). If you are using a more modern system then try this

#!/bin/ksh

#  Compare 2 directories for differences (e.g. current release and new release)

CURRENT=$1 ; shift
NEW=$1; shift

CURRENTDIFFFILE=/tmp/$(echo $CURRENT | sed -e 's/[/.][/.]*/_/g')
NEWDIFFFILE=/tmp/$(echo $NEW | sed -e 's/[/.][/.]*/_/g')

# File pattens not to be included in compare
cat << 'EOF' > /tmp/patternFile
^.git/
^.*\.log
^logs/.*
^.*\.tmp
^.*\.err
EOF

for i in $CURRENT $NEW ; do
	cd $i
	# cut removed ./ that find puts at start of path name
	find . -type f -print |cut -b3- |egrep -v -f /tmp/patternFile | sort > /tmp/$(echo $i | sed -e 's/[/.][/.]*/_/g')
	cd -
done

echo New files
comm -13  $CURRENTDIFFFILE $NEWDIFFFILE

echo
echo Deleted files
comm -23  $CURRENTDIFFFILE $NEWDIFFFILE

echo
echo files that are different
# xargs has limitations so must pipe into a second shell
comm -12  $CURRENTDIFFFILE $NEWDIFFFILE | xargs -i echo cmp -s \"$CURRENT/{}\" \"$NEW/{}\" \|\|  echo {} changed | sh

It’s all about the process stupid!

BBC NEWS | Health | Surgical checklist ’saves lives’

Using a simple surgical checklist during major operations can cut deaths by more than 40% and complications by more than a third, research has shown.

Does ANYONE need persuading on the value of a process when performing technically complex work? Including software development and service management…

Application deployment and change auditing with git

Updated 6/Feb/09

I’m currently working on on a project to deploy some complex banking systems and we have some somewhere in the region of 40 environments (various testing, production and development instances)

I’ve written some Perl to automatically configure environments using Text::Template. Now I’d like to automate deployment and have some form of change auditing (c.f.AIDE or Tripwire).

So a new project (which I hope will also  have some useful ideas for others):

• Project Outline and User stories
• General Approach using git (other tools could probably be used)
  • Define a staging repo
  • Create a remote branch of each deployment environment
  • Create a repo on each environment
  • Define a custom merge driver that overwrites (old files with new releases)
  • For each release
    • Create a new branch (overkill?)
    • Add new release to new branch
    • tag
  • To deploy
    • Merge (using new driver) onto environment branch
    • tag
    • Push to remote
    • On remote perform git reset --hard HEAD
    • configure
    • add and commit configuration changes
    • on staging pull config changes from remote repo
  • Now use git status etc. on environments to track any changes
  • Staging should have historical record of changes

Top Tips: Using the EEEPC 1000H

I’ve has my EEEPC 1000H for over a week now and I’m so totally enamoured of the small device that it’s become my sole platform for all my computing. Obviously this has involved some adjustments.

1. The keyboard requires considerable patience — at least for someone with my large fingers
2. My fingers are having to learn new positions to try and stop keep brushing the trackpad and moving the cursor to somewhere completely different. This is currently the most frustrating aspect of using the machine.

Some things to try.

• Get a bluetooth mouse
• Try the LXDE desktop and the Xfce4 desktop. They are both tuned for low resource use
• Consider using DevilsPie to reduce the amount of screen space used by removing the title bar and forcing all screens to be full maxiumum, partucurly of your screen is smaller than 10″
• Configure Skype to use the hda sound devices
• Carry a short USB cable for thumb drives etc. Thy can’t always fit in space you have at the side of the netbook
• Get used to using keyboard shortcuts — it mush  easier than using the trackpad
• Try using Vim and it’s plugins rather than  a large IDE
• Try using an alternative email  and PIM program to Evolution. Thunderbird for instance works better on the smaller screen and is faster
• Use multiple desktops and flip between them using <ctrl><alt><arrow> (Gnome default), for instance when referring to other material in the middle of witing an email. It’s an easier way of switching back and forth between two applications that using something like <alt><tab>
• Use the <F11> in Firefox to move into full screen mode (many other applications have a similar feature)
• Do as much as possible by typing at the terminal

Thanks to git I’m releasing IP into the wild!

This week I at last did something I’d wanted to do for a long time and  release some of Voga’s intellectual property to the wider world.

I have already made material available either through this blog or via Slideshare. However for a geek it’s obviously better to publish complete projects in an Open Source manner so  they can be freely copied, modified and shared.

For historical reasons this had been inconvenient to do as I used to hold everything in a single Subversion repository. Not only did this contain my potentially open material, but also my customers private data and my business records (yes, I keep my accounts database in a version control repository!). A few months I migrated my Subversion  repo to git, but I still had a single large repo and no time or skill to fix it.

However today I bit the bullet

1. Removed all my possibly public data to another directory structure
2. Tidy up my current repo by removing all the old ‘trunk’ directories — as a solo consultant I was lucky and had never had to branch my projects. Commit my new streamlined repo

Now to work on the the public projects

1. Identify a complete project and make sure it had no proprietary data. This is potentially time consuming
2. Tidy up some of the names
3. Create a new git repo; add the project files; and commit
4. Create a new repo on Github for my project
5. Follow the simple instructions to upload my project to github

To Do

• Update existing references to my projects I can find and update to point to the repo
• Add the correct license files
• Start again for as many projects as possible, please be patient

I’ll still keep presentation copies on Slideshare because that is much more convient for most people.

A big thank you to the git developers and github for making this so easy

Handy Hack: Make failure optional

When programming it’s always a good idea to check return status to make that no error has occured. In Perl we often write

somethingWeWantToDo or die "we failed because .....";

However when running in a test environment we may not have all our infrastructure in place to make our operation pass when needed (ideally we should use mock objects or some other testing framework). A  hack is to pass a flag into the program telling it that we are running in a test jig and we don’t care if our operation fails. The solution is simple, but a pain to work out after lunch…

somethingWeWantToDo or $testJig or die "we failed because .....";

If we fail and we are not using our test jig then we will exit with the appropriate diagnostics. i.e. failure is optional in a test jig

This is much easier than cluttering the code with conditonal if statements.

What is Application Lifecycle Management?

The analysts and tools vendors have been talking about Application Lifecycle Management (ALM) for many years now and I think they have (of course) been somewhat self serving in their definitions. ALM seems to  include whatever product category they have in the current price list and then promising their integrations’ would be managements answer to cost and governance issues. I hope to say more about integrations in the future, but I’d like to look at the scope of ALM for now.

If you ask vendors and consultants (including me) about ALM scope you’ll get answers such:

• Requirements engineering
• Software Configuration management
• Dependency Tracking
• Change control and work tracking
• Test management
• Software development tools and process
• Software building and processes
• Deployment
• Process Governance

Depending on the vendor you might even get more traditional: Project and Program Management;  Project and Application Portfolio Management/Analysis; and Service Management activities thrown in as well.

The usual sales pitch is that everything is managed via a common tooling and provides a common repository. The major, and often single, selling point of this approach is improved governance and business oversight of the SDLC.

However I think it’s useful and important to look at the definition of ALM from a customers perspective rather than as analysts or vendors.

It seems that a more pragmatic approach is to describe ALM as the methods, processes and tools that support the management of change for software systems — with particular, but not exclusive, emphasise on the SDLC. In this context it does not matter if we are using Agile or Waterfall approaches, we chose the best tools and processes for our particular situation. This of course implies that in eighteen months time we may need to re-tool.

Furthermore the methods tools and processes of project management (PM) are much more mature and there is no real need to include them in this evolving segment — they have their own. Obviously PM will have an influence and impact on ALM and we need to design our ALM approach to dovetail with it. I realise that most vendors will not agree with this.

I have come to the conclusion in order to ’sell’ ALM into the teams that will use it we need to ensure, front and centre, that ALM provides a ‘better/faster/more’ improvement in daily productivity. That should be the primary focus of what is delivered.

Governance, audit etc should be secondary attributes and natural site affects of using ALM (important though they are of course and usually the primary sales message).

So at a practical level we need to be able to provide teams and their specialists with the best tools they need to do their job. Additionally we need to provide a supporting layer of data flows, events and data repositories to

1. Assist teams with traceability
2. Provide the tracking and management visibility to effectively steer the ship

So the point of this post is to plead with vendors to stop selling a one size fits all tool suite with deep integrations and start enabling shallow API and effective integrations to allow a truly best of breed approach for customers software purchases. It’s a shame that ALF project is shuting down.

What a young developer needs to know

(Updated 5/Oct/08 and 11/Nov/08)

I occasionally ponder on what skills and knowledge a novice who wants to become a skilled developer should acquire. In no particular order here is a suggested list of things to consider:

• Basic accounting: This is useful from both an IT perspective — a lot of IT systems are used for financial management; and from a general work perspective because it helps with your future in business
• Version control concepts and tools
• Waterfall Software Development Lifecycle (SDLC): Requirements, design, building, testing, release
• Agile SDLC such as XP and/or Scrum
• How to use a command line
• Basic compiler architecture — the 7 phases of compiling
• Data representation: Number bases (binary, Hex, BCD); character sets; XML markup
• Basic algorithms: Sorting, Linked lists etc, recursion,…
• Important design patterns
• Testing and quality as a substantial stand alone topic
• Programming in at least three languages: C (or Java, C++, Ada etc), Perl (or Python, Ruby etc) plus one other (e.g. Lisp, Haskell). At least one needs to be an OO language and OO programming should be a familiar habit. Perl does not count as OO for this discussion.
• How to use a word processor and spreadsheet
• Basic SQL, RDBMS architectures, 1st, 2nd & 3rd Normal forms and basic DB schema design
• TCP/IP networking and basics of http, email, ftp, etc. Needs to include conceptual understanding of DNS and related security issues
• Basic IT security concepts
• Basic IT System Management
• Personal task and work management (e.g. GTD or GSD, suggested by Sinewalker)
• Maintaining personal work records in a  log book

Further topics

• More advanced RDBMS use and schema design
• Embedded systems
• How to use LaTeX or Docbook
• Application engineering for quality (maintenance, extensibility, performance etc)
• OS design
• Requirements engineering
• UML

Further suggestions anyone?

Introduction to using Make to build software

I’ve placed another presentation online that I gave to Linux Users of Victoria. It’s a short and very basic introduction to make (as used by software developers).

If you are intrested in make then you maight also like my presentation on using make and subversion together. (Beware that it’s a lot more meaty than this short introduction).

Post-Agilism anyone?

Agile software development – Wikipedia, the free encyclopedia

Some commentators propose a model of post-Agilism that is effectively constructive anarchy, in that teams should be self-organising to the point where even the core values of the Agile movement are considered too presciptive, and that teams should simply “do whatever works for them”.

Does this seem a little scary to you? Gives me the heebie jeebies.

For those folks who are not familiar with Agile methods I have to say that I am firmly of the opinion that professional Agile developers are in fact some of the most process centric individuals on the planet. It’s one of the reasons I like the approach. This process (and we are talking about process in the small) leads to a) greater predictability and b) a mind more able to concentrate on the creative aspects of the job.

Post-agilsm seems to the antithiis of this. However a brief paragraph in WikiPedia does not a methodology make so I’m open to discussion.

Handy Hack:Convert a version controlled file to a different content type

This is not exactly rocket science, but does show up an important principle of Version Control (see below).

Let’s take a practical example that happened to me today:

I have a document under source code control (git in this case) and I am editing it using OpenOffice. Naturally it is an ISO standard Open Document format file and so it’s file name is something like GreatDocument.odt. However I now need to distribute this document to a wider audience for comment and feedback; and of course everyone else uses MS Word 2003. The practical solution is to change the document format to .doc. However I still want to keep the full history of the item — the fact that it changed from ODT to .DOC format is actually not significant in the grand scheme of things. For git the following commands need to be performed:

1. Using OpenOffice save the file GreatDocument.odt in MS Word format using the Save As function. This should create a second file called GreatDocument.doc
2. Rename GreatDocument.doc to something else, e.g. GreatDocument.word
3. issue a git-mv command to show that the file is changing. e.g. git-mv GreatDocument.odt GreatDocument.doc
4. Overwrite the .doc (which is actually in ODT format of course) with the true Word file. E.g. mv GreatDocument.word GreatDocument.doc
5. Now commit your changes to the repo. git-commit -a

Of course you will need to modify this to suite your platform and version control tool. But the principle should hold in the majarity of cases.

And what is the important Version Control principle involved?

The physical format of a controlled Item is less important that its history and logical purpose

Well deserved recognition for jarich

use Perl | White Camel Awards 2008

The first White Camel recipient for 2008 is Jacinta Richardson. She has been credited for saving more than one Perl Monger group from dying and has been heralded as the “gardener of the Australian Perl Community”, in the sense that she provides energy while subtly encouraging others to take responsibility. She is an active member in every Monger group in Australia, as well as many other FLOSS groups in the area. She has helped organize OSDC, Linux.conf.au, and SAGE conferences. She works with Perl Training Australia in supporting the Perl community with such ventures as perl.net.au and Perl Tips.

In in Australia we often punch above our weight in many areas, not just sport, and our communities for users of FLOSS are world class. It’s people like Jacinta (a.k.a jarich) who help make it happen and it’s good that we recognise the hard work she (and others) do to keep this infrastructure going.

And in further proof of Australian class from OSCON this year:
schwern’s status on Thursday, 24-Jul-08 00:29:13 UTC – Identi.ca

In Paul Fenwick’s “Illustrated History of Failure” and so far it’s brilliant. #OSCON

Something I have been whining about for years

Syncable tools for the offline web

SD is a peer to peer bug tracker which can sync to itself, RT or Hiveminder. You can extend SD to sync to other bug tracking tools by writing simple adaptors. If you’d like to help make SD work better with your bug tracker, drop us a line.

With all the cool kids (and me) moving to distributed version control tools it’s been obvious that that there is a gap because our bug databases are centralised. Why is that a problem? Because code changes should always be booked against either a user story or a bug report. The link needs to be hard! i.e. I should be at least warned when I commit code without a ticket and when I look at a completed ticket I should be able to find the corresponding code. When you are committing code changes at 11,000 meters above sea level that can be a bit hard.

The only thing I know about SD is what I’ve read above, but already I want it!

[update 25/July -- the SD website now works so go for it!]

Handy Hack: Enter the office lottery syndicate with Perl

If you are asked to chose numbers for a lottery here is a rather inelegant hack to create some entries. Assume you need x numbers from a range of a to b inclusive then run this program as

./lottery.pl  a b x.

#! /usr/bin/perl
#

our %dup;

our $lower = $ARGV[0];
our $upper = $ARGV[1];
our $needed = $ARGV[2];

for (1..$needed) {
    my $x = int(rand($upper-1)+$lower);
    if ($dup{$x}) { redo }
    print  "$x \n";
    $dup{$x}++;
}

Top Tip: Cygwin Perl has problems running with UNC path names

When running Cygwin Perl using libraries that live on UNC paths (e.g. when running prove with the -l option) Perl will croak because it is unable to load the library. An example error message will look something like:

t/00.load.............Can't open perl script ""-I//server/sharename/work/mymodule/lib"": No such file or directory

The fix is to map the UNC path to a drive letter and use the Cygwin mount point (e.g. ‘/cygdrive/z/work/mymodule’) as the path instead.

The command is

net use z: '\\server\sharename'

Handy Hack: Force cpan utiltiy re-run the configure questions

When you first the cpan command line utility it asks you a rather tedious list of questions about your setup. Occasionally you need to go through it all again. Use the command ‘o conf init‘ at the cpan prompt to force this behaviour.

An easy way to document process

What is a “Lightweight Ticket Process” ? « Alec the Geek

Document the process in some fashion. A process that is not documented can be very hard to talk about, agree and improve.

Following on from a brief twitter conversation with Kate this morning here is a simplistic example of how I currently document processes. The motivation for this approach is:

• Simple to edit and and maintain
• I hope simple to read and understand (even if you are not technical)
• Can be used as the basis for detailed work instructions

Take 2:What I add to Windows that helps make it better

(Updated 28/Aug/09 with Open Source Windows site)

Essentials to make Windows almost bareable « sinewalker

a list of what free software to install on top of Windows, and what adjustments to make, so that at least I don’t feel like I’m wearing a straight jacket

Some excellent ideas and suggestions from Sinewalker here and there are more at Open Source Windows. I posted my own list of Windows essentials back in 2002 and I’ve been thinking for some it’s time to tidy the list up a bit since the world has moved on, so here goes:

Software development related:

• Vim
• Eclipse
• VMWare Server for testing and building
• JDK (Java Development Kit)
• Strawberry Perl
• Cywgin (includes shell related stuff, X server, ssh et al, svn, git, etc). I use chere to add an “Open Bash Here” feature
• BOUML UML tool
• Apache James email server (for testing, not real email)
• WinSCP so I can access remote development systems

Desktop and Misc

• Pidgin IM Client
• Freemind mind mapping software
• GNU Cash for Windows (still testing this)
• CDBurnerXP
• Docbook
• 7-Zip
• Putty
• PDF Creator
• Firefox with the del.icio.us bookmarking and scribfire blogging extensions
• Launchy
• Filezilla (for ftp sites the login process is easier in Filezilla)
• OpenOffice
• Thunderbird email client with the Lightning plugin
• Open Workbench project planning — claims to be Open Source, but the important bits aren’t

Security

• Updated Nov/2008. Free AVG ClamAV anti virus
• Spybot Search&Destroy malware tool

All of these tools are free of charge and many are Free software, which helps with the budget. However please consider a donation if it’s asked for and you can afford it.

In addition I change the default XP theme and use Windows 2000 as it takes up less screen space (or least it feels like it does) but use the ‘modern’ start menu.