Application deployment and change auditing with git

Updated 6/Feb/09

I’m currently working on on a project to deploy some complex banking systems and we have some somewhere in the region of 40 environments (various testing, production and development instances)

I’ve written some Perl to automatically configure environments using Text::Template. Now I’d like to automate deployment and have some form of change auditing (c.f.AIDE or Tripwire).

So a new project (which I hope will also  have some useful ideas for others):

• Project Outline and User stories
• General Approach using git (other tools could probably be used)
  • Define a staging repo
  • Create a remote branch of each deployment environment
  • Create a repo on each environment
  • Define a custom merge driver that overwrites (old files with new releases)
  • For each release
    • Create a new branch (overkill?)
    • Add new release to new branch
    • tag
  • To deploy
    • Merge (using new driver) onto environment branch
    • tag
    • Push to remote
    • On remote perform git reset --hard HEAD
    • configure
    • add and commit configuration changes
    • on staging pull config changes from remote repo
  • Now use git status etc. on environments to track any changes
  • Staging should have historical record of changes

Handy Hack: Make failure optional

When programming it’s always a good idea to check return status to make that no error has occured. In Perl we often write

somethingWeWantToDo or die "we failed because .....";

However when running in a test environment we may not have all our infrastructure in place to make our operation pass when needed (ideally we should use mock objects or some other testing framework). A  hack is to pass a flag into the program telling it that we are running in a test jig and we don’t care if our operation fails. The solution is simple, but a pain to work out after lunch…

somethingWeWantToDo or $testJig or die "we failed because .....";

If we fail and we are not using our test jig then we will exit with the appropriate diagnostics. i.e. failure is optional in a test jig

This is much easier than cluttering the code with conditonal if statements.

Well deserved recognition for jarich

use Perl | White Camel Awards 2008

The first White Camel recipient for 2008 is Jacinta Richardson. She has been credited for saving more than one Perl Monger group from dying and has been heralded as the “gardener of the Australian Perl Community”, in the sense that she provides energy while subtly encouraging others to take responsibility. She is an active member in every Monger group in Australia, as well as many other FLOSS groups in the area. She has helped organize OSDC, Linux.conf.au, and SAGE conferences. She works with Perl Training Australia in supporting the Perl community with such ventures as perl.net.au and Perl Tips.

In in Australia we often punch above our weight in many areas, not just sport, and our communities for users of FLOSS are world class. It’s people like Jacinta (a.k.a jarich) who help make it happen and it’s good that we recognise the hard work she (and others) do to keep this infrastructure going.

And in further proof of Australian class from OSCON this year:
schwern’s status on Thursday, 24-Jul-08 00:29:13 UTC – Identi.ca

In Paul Fenwick’s “Illustrated History of Failure” and so far it’s brilliant. #OSCON

Handy Hack: Enter the office lottery syndicate with Perl

If you are asked to chose numbers for a lottery here is a rather inelegant hack to create some entries. Assume you need x numbers from a range of a to b inclusive then run this program as

./lottery.pl  a b x.

#! /usr/bin/perl
#

our %dup;

our $lower = $ARGV[0];
our $upper = $ARGV[1];
our $needed = $ARGV[2];

for (1..$needed) {
    my $x = int(rand($upper-1)+$lower);
    if ($dup{$x}) { redo }
    print  "$x \n";
    $dup{$x}++;
}

Top Tip: Cygwin Perl has problems running with UNC path names

When running Cygwin Perl using libraries that live on UNC paths (e.g. when running prove with the -l option) Perl will croak because it is unable to load the library. An example error message will look something like:

t/00.load.............Can't open perl script ""-I//server/sharename/work/mymodule/lib"": No such file or directory

The fix is to map the UNC path to a drive letter and use the Cygwin mount point (e.g. ‘/cygdrive/z/work/mymodule’) as the path instead.

The command is

net use z: '\\server\sharename'

Handy Hack: Force cpan utiltiy re-run the configure questions

When you first the cpan command line utility it asks you a rather tedious list of questions about your setup. Occasionally you need to go through it all again. Use the command ‘o conf init‘ at the cpan prompt to force this behaviour.

Handy Hack: How to configure the cpan utility to use Windows proxies

When using cpan.bat the following may be helpful. Please note that this is for a Windows corporate network infrastructure and was tested using both Strawberry and Cygwin Perl in the current network environment where I work. YMMV!

BEWARE THAT THIS APPROACH STORES YOUR LAN PASSWORD ON YOU LOCAL HARD DISK. THIS IS INSECURE!

Step 1: Information Gathering

1. Discover the address of your proxy: If using a PAC file (look in IE Tools->Internet Options-> Connections -> LAN Settings and identify the setting ‘Use automatic configuration script’) then download the PAC file and open in a text editor. The default proxy address (e.g. proxy.corp) and port (e.g. 80) will probably be at the bottom of the PAC script. If you are not using a PAC script then the proxy address will be in the LAN Settings dialog.
2. Know your domain name (e.g. corpdom) and lan ID (e.g. lanname)
3. You will also need your lan password (e.g. secret)

Step 2: Configure CPAN

1. From the command line ensure that you running the correct version of perl (perl –version)
2. run the command cpan
3. You may be asked some configuration questions — answer as best you can and take defaults if not sure
4. At the CPAN Prompt type the following commands:
   1. o conf proxy_user corpdom\\lanname (NB Double slash required)
   2. o conf proxy_pass secret
   3. o conf ftp_proxy http://proxy.corp:80 (NB No trailling slash on URL)
   4. o conf http_proxy http://proxy.corp:80 (NB No trailling slash on URL)
5. Now try and install something (e.g. i Bundle::Perl::Critic)

[Update: July 2008 -- tested on Strawberry and Cygwin Perl. See also]

Handy Hack: Can’t run perldoc on Cygwin

When running perldoc on certain Windows systems using Perl under Cygwin it is possible to get an error message

Error in tempfile() using ./XXXXXXXXXX: Parent directory (./) is not writable


This is more likely to happen if your system has been ‘hardned’ by a corporate IT department. The fix is as follows:

1. Make sure $TMPDIR is defined and points to somewhere sensible (e.g. $TMP)
2. Make sure you have read/write permissions (chmod 777 $TMPDIR)

Handy hack: Install CPAN modules by hand when there are multiple Perls

On Windows it’s very easy to end up with with multiple installtions of Perl (e.g. I have Strawberry Perl, Cygwin perl and Activestate Perl installed by Twiki).

If you download CPAN modules and then install them by hand things don’t always go where you think.

The (obvious) trick is to make sure you are running the correct Perl (and make tool) before you begin the install process. Let’s take Strawberry Perl as an example:

1. Make sure the correct perl bin is first in the PATH

   set PATH=c:\strawberry\perl\bin;%PATH%

2. Make sure the correct make program is first in the PATH

   set PATH=c:\strawberry\c\bin;%PATH%

You should then be able to run perl Makfile.PL, dmake, dmake test, dmake install. However if you have a problem then correct the environment and run dmake -u to force a rebuild. (If you are using something other than make program then you’ll need to find the correct option)

Agile in the Perl world

Or should that be Perl in an Agile World?

There was a discussion tonight at Melbourne Perl Mongers on using Agile practices whilst Perl programming. To my delight the answer from attendees was ‘Yes we do that every day’! Specifically there was a discussion around:

• TDD frameworks
• Refactoring
• Continuous integration and how important is is to have an SCM tool that makes developer branches (and merging) easy
• Perl Mock Objects for testing
• How to apply TDD to code that is not OO or even a proper module

There is a Yahoo Group on Perl Extreme Programming and an online book

Handy Hack:How to install Perl modules on Linux

An excellent series of posts on the Melbourne Perl Mongers mailing list on the correct way to install Perl modules on your RPM or Deb based distro.

Now I know what I was doing wrong and how to fix it. Thanks Mongers

Update: I thought the information was so good I slapped it into Perl Net.

Handy Hack:Perl DBI count rows returned from select statment

my $rowCount = $dbh->selectrow_array(

                                 qq{

                                      SELECT count(*)

                                         FROM my_table

                                       WHERE attr1 = ?
                                },undef,
                                $myValue);

In theory it’s a bad thing to assume a select will return a single scalar value instead of multiple attributes and rows, however with a count clause we should be pretty safe. N.B. using the $dbh->rows() function or similar is not correct on a select statement.

Modified after a suggestion from Randal. Thanks mate.

I want to attend this

Thoughtstream | Courses | The Productive Programmer

This course explores how Dr Damian Conway — one of the most prolific contributers to the Perl community and to the CPAN — manages to be so productive

Having seen Damian present some of his ideas about productivity, both in person and in books, I have a strong suspision this would be a very thought provoking and useful course.

Damian gave a presentation at OSDC 2005 that showed some of his techniques, including which CPAN modules he used and how to present the writing of Perl code by using IO::Prompt (which he wrote of course), I believe there was also a more substantial presentation at OSCON. I think he uses the Vim editor so I would even more out of it.

Generally Damian is very a) knowledgeable and b) amusing presenter so if you ever have the opportunity to hear him speak don’t miss it, even if you are not a Perl programmer.

powered by performancing firefox

Mason and YUI

I will be contributing some new work at soon at Unisolve which will revolve around HTML::Mason and the Yahoo! User Interface Library (YUI). I am really looking forward to digging further into these tools.

Mason is a templating engine for Perl programmers. That means that you can mix Perl code with your data (often HTML) so that additional content can be generated in the fly before the information is presented. For instance you can extract records from a database. There are a plethora of these tools, however Mason is one of the more established.

YUI is a AJAX library designed to allow web applications to present a rich user interface using Javascript (and other stuff) on the browser. This can make web applications provide features and functions that have only be possible previously with “thick” client software that have enough programmed intelligence to support the user interaction experience.

Perl inside-out objects

According to Perl Best Practices, a.k.a PBP, the “One True Way” to implement Objects is using a technique called inside-out. Unfortunately reading Daminan’s explanation, although as usual excellent in many ways, did my head in because I could not imagine how the storage was logically laid out (I’m kinda old fashioned in how I understand things you see).

Luckily a quick FWSEy and I was able to find Perl Australia’s cool explanation that lays out the way the attribute hashes store values in inside-out objects. I suggest you print out a copy, fold it up and place inside PBP (What do you mean, you don’t have a copy? I hope you’re not a Perl programmer). The content is Open so you can do that, Hurrah for Open Content I say.

There are still a number of questions in my mind about how this works but perhaps I can find someone at next Perl Mongers meetings, in a brewery this month, to fill in the gaps. I’m afraid the website is wrong about the meeting BTW — it’s on Friday 12/Jan.

powered by performancing firefox

More on testing Perl applications

A few weeks ago I said

As far as I can tell all, of the current CPAN testing frameworks are designed to test Perl Modules (.pm files). Because of this they are designed to be consumers of the services provided by the modules under test.

Testing Applications and Suites using Perl and TAP « Alec the Geek

It turns out of course that the test framework can be used with a Perl application. It is nessasry to imbed the tests directly in the code. The following simple, but complete, example is taken from Perl Testing: A Developers’s Notebook.

#!perl
use strict;
use warnings;

use Test::Simple tests =>1;

sub hello_world
{
return "Hello, world!";
}

ok( hello_world( ) eq "Hello, world!" );

Blogged with Flock

Testing Applications and Suites using Perl and TAP

Perl has excellent testing support with frameworks such as Test::More, these tools present test results in a standard text format call TAP.

As far as I can tell all, of the current CPAN testing frameworks are designed to test Perl Modules (.pm files). Because of this they are designed to be consumers of the services provided by the modules under test. (Update 10/Jan/07 — Correction)