Handy Hack for Git refugees in Subversion land

When you’re a Git user working on a project using Subversion (SVN) using the Git SVN client is a great option, I’ve been using it for the last twelve months and it works really well. However sometimes it’s not quite good enough when things like Ant scripts are written for SVN.

My colleagues use TortoiseSVN. which among other things provides support for standard commit message templates. This uses a SVN property called “tsvn:logtemplate“, is a specific feature of ToirtoiseSVN and it not standard in the SVN client. Git has a simliar feature using the config value commit.template and it’s standard on the git client.

Today I wrote an svn wrapper script to provide the standard template feature to the command line SVN client. You may find it useful. It’s written for OS X and assumes you have Git configured and SVN Installed via Mac Homebrew — however it will be trivial to modify it for your environment,

The script does does a number of things:

1. It passes all SVN sub-commands, except commit, to svn without change
2. It checks all the arguments to if the commit message has already been supplied (run the command and exit)
3. It checks to see if the ”tsvn:logtemplate” is set
4. If the TortoiseSVN template is not set then it attempts to find the Git template
5. If that fails then it falls back to an internal template
6. The template is placed in a file and another  message file
7. The user is prompted to edit the message file
8. The updated message file is checked against the template — if the user has not made a change the commit is aborted
9. The svn client is called with the message file.

Updated March/2013 – Somehow an stupid bug crept in. Now squashed.
Updated Jan/2013
I have added a new feature when using svn update:

1. Check if the working copy has uncommitted changes. NB Files that are not registered are ignored in this check
2. Abort the update if files are already modified

YMMV

Top Tip: “”Bad URL passed to RA layer: Unrecognized URL scheme for:” when running git svn

If you are getting the above problem try installing Subversion. If you are using homebrew for OS X then make sure you install package svn, not subversion.

Application or IT Environments Management

Note from Alec: I’ve copied this article here in case the original version disappears.

Author: Valentine

Application or IT Environments Management service will fall under Application Management as defined in ITIL2 (operational guidance) because it contributes to improving the overall quality of IT Software development and support through the lifecycle. Application Environments Management set encompasses a set of best practices proposed to provide an effective, end to end management service for test software platforms or development environments. The software test bed or development environment could consist of a client server application, Relational Database Management System (RDBMS), middleware, interfaces, daemons, customised processes (written in any software programming language), FTP utilities etc.

Functional test phases such as Unit, Integration, Acceptance, all manner of performance or non functional testing and development phases all require Application Environments.

The primary clients of an Application Environments Management service are Software Project and Test teams.

The service will cater for the following;

• Manage the creation, build, upgrade and support for all test and development Application Environments.
• Clearly defining auditable processes of allocating application environments, multiple bookings or shared usage, code upgrades, service level agreement, support, decommissioning and re-allocation.
• Manage data refreshes, collating test data and assist in the anonimising of production or other sensitive data if necessary.
• Supply, provision and manage all Application Environment Requirements from the Project and Test teams all through the software development cycle of a project.
• Assist the Project in establishing their application environment requirements, provide expert knowledge on the APPLICATION environment’s set up, connectivity and serve as a guide to the projects in using the application environment in the most efficient manner.
• Review and contribute to the Project Initiation Document (PID) ensuring that the IT Environments Management function and its deliverables are clearly defined and captured.
• Create and maintain project plans to assist in managing all activities required to successfully carry out major code upgrades to all application environments.
• Provide reports on usage/utilisation, availability, forward planning and schedules.

Application Environments Management is clearly a new and emerging area which has arisen due to the following reasons:

• The increased Application Environment requirements for many companies who have several software projects running at any one time.
• The increased levels of interfacing and connectivity between several systems in most organisations also known as spaghetti. For example in some companies more than thirty systems are interfaced or connected with each other exchanging files and data flows etc and has meant that any changes to one system most times could require a change to many others and then require large numbers of test and development application environments.
• Increased awareness and more commitment to carrying out rigorous software testing especially with more companies opting to use the Prince 2 methodology and ITIL Framework

A typical Application Environments Management tool should be able to provide the following services: environments bookings and allocation, manage multiple bookings and re-curring bookings. Provide reporting on usage, availability, interconnectivity or interfacing environments, utilisation and conflict reporting etc. It must also serve as a repository of all information on an Application Environment to include Host Server names, Hardware Type, Operating System, IP Address and Interfaces if any.

The ideal background for an Application Environments Management personnel could be Software Development, Application or Technical Support, Infrastructure Project Management, Configuration and Release Management etc but must be

exposed to at least the ITIL Framework, Client – Server development, System Architecture/Design, Networks, TCP/IP and Messaging systems etc.

Terminologies defined & explained:

Application Environment – A single test bed or development platform instance of a software application or system that can also be used for all manner of functional and non functional testing or could be the production instance (production environment). It could also be large, medium or small which normally refers to the size of data the RDBMS will be holding depending on the type of testing it is required for.

Integrated Application environments (also known as stripes): More than one application environment connected to each other also communicating with each other and exchanging files and data flows. Connections could be via Microsoft ODBC, via FTP, TCP/IP, daemons, middleware, defined interfaces and database links etc.

Anonimising of data

Anonimising of data refers to the manipulation or transformation of production data held in the RDBMS such as Oracle, SQL Server, Sybase, Microsoft Access, DB2 etc to be used in a test or development Application Environment ensuring that for example real names, addresses, date of birth, bank account details and other sensitive information or data is transformed to dummy data.

The data is transformed whilst still maintaining its defining characteristics in a Relational Database Management System table such as character length (Char 25 or Varchar 50) etc to ensure its usage in testing or development is not compromised and that the integrity is maintained. For example a valid name such as John Smith defined as Char 10 will now be updated in the table to become possibly a unique character string XXXXXXYYYY (comprising of ten characters including the space between John and Smith).

Pipe cleaning

Pipe cleaning caters for the all the activities required to be carried out before a test or development environment is handed over to the Test or Project team and includes disk clear down, archiving and purging logs, importing test data, killing off rogue processes, resetting passwords, changing environment settings, end to end connectivity or integration tests to make sure everything is working okay.

Depending on the complexity of the system a checklist of activities may be required and ticked off capturing all the checks and tests that have been completed on an environment or an integrated suite of environments prior to its hand over to a Project or Test team.

Smoke Test

A smoke test describes an initial end to end test of all the integrated or even stand alone environments very possibly using dummy data and carried out by the support teams who have created or built the environment or by the test team when the environment is handed over.

Article Source: http://www.articlesbase.com/information-technology-articles/application-or-it-environments-management-2884860.html

About the Author

• Freelance IT Environments Manager & Technical Project Manager

• Previously Global FX technology manager at Citibank International

• Worked in several sectors to include energy, utilities, Investment banking, railway transport, software development, healthcare, manufacturing, research, insurance and the gaming sectors.

OSDC AU 2010: Get your papers in!

Updated 12/Oct/2010 — This proposal has been accepted so I hope to see you at OSDC 2010

 

The request for papers is up at The Open Source Developers Conference AU so get your ideas in. Here is my proposal for this year

Using Git DVC in conjuntion with a legacy SCM tool

Often developers get a limited choice on the Version Control (VC) or Software Configuration management tool they have to use when working on a project. However even when the Git (the Distributed Version Control tool) does not interface with the project tool there is value in using Git for a personal workflow. This can work as follows.

1. Fetch a copy of the working code needed to make your change
2. Initialise a new Git repo in your working copy and add the code
3. Now checkout a new git branch for your work
4. hack hack hack test test….
5. Commit
6. Switch to master branch
7. Fetch from the project tool again.
8. Add changes and commit
9. Review differences, rebase or merge as required.
10. Verify project tool still at same state from step 7
11. Upload/ check in changes to project tool. You can use the output from git log as the basis of your commit message for the legacy tool

Rinse and repeat

Build Audit and Processes: A line of sight from DEV to PROD

I’m presenting at the Devops Down Under conference in May 2010

…a conference about advancing our processes & methodologies, building better tools, and improving communication between developers and sysadmins.

Here is the abstract for my presentation in case it tempts anyone to attend…

Both agile and waterfall approaches to software delivery are a way to translate business needs into production features. Being able to trace that path is often crucial to affective delivery (and sometimes passing critical compliance regulations). Frequently the activity of software building breaks these important traceability links because of poor process and technology.

This presentation briefly examines these problems and looks at various approaches to overcoming them. An emphasis is placed on using processes and version control tool to provide a) Useful information and b) Audit trails of individual builds. A simple, illustrative, example using the Git DVCS is demonstrated.

Say hello if you are coming along.

Handy Hack: Calculating Git SHA1 on files without Git installed

If you are not a Git geek this will be of limited interest and I suggest you go and surf whatever inappropriate material uses up your bandwidth.

Updated after tips from Randal Schwartz and Charles Bailey

Still here? OK. As you will know every file stored in a Git repo is identified by a SHA1 hexadecimal string (as explained in the Pro Git book, it’s the hash of the concatenation of the  file contents and a Git specific header). You can calculate the Git SHA1 of any file with the command

git hash-object $file

assuming you have Git installed

If you do not have access to Git, but do have access to a UNIX style shell or Perl you have a couple of options (the Pro Git book also shows a Ruby solution and Stackoverflow has examples in Python and #F))

On Cygwin or Linux

(printf  "blob %s\00" $(wc -c < $file);cat $file)|sha1sum -b | cut -d " " -f 1

On Solaris

(printf  "blob %s\00" $(wc -c < $file);cat $file)|digest -a sha1 | cut -d " " -f 1

In Perl

# See also Git::PurePerl at http://search.cpan.org/dist/Git-PurePerl/

use strict;
use warnings;
use Digest::SHA1;

my @input = <>;

my $content = join("", @input);

my $git_blob = 'blob' . ' ' . length($content) . "\0" . $content;

my $sha1 = Digest::SHA1->new();

$sha1->add($git_blob);

print $sha1->hexdigest();

Draft: Introduction to Simple Processes for Small Software Teams

N.B. This is an early draft of something I am currently writing. Please feel to comment and give feedback. The full ebook will be published in the next few weeks and I will follow with another blog so you can download it if interested.

How you can implement processes to improve quality, reduce stress and lower costs

Introduction

A lot of application development teams, and to a lesser extent operational IT teams, have either no or limited process. They depend on a combination of individual expertise, informal conventions and team habits (some good and some bad); and being prepared to put with all the problems of poor quality and unpredictability. It does not have to be like that and this ebook presents some ‘low hanging fruit’ you can pick to make your teams more productive – the approach is simple and practical.

This document is written for smaller teams who spend their time doing a combination of maintenance and new feature requests on an existing application. This actually covers the majority of software teams because, much as we all want to work on green-field projects and deliver Rel 1.0, most of our work is done on existing systems.

This ebook lists seven keys process areas (PA) that you need to think about: Metrics, Planning, Release, Testing, Issue¹ management, Version control and Communication. These represent broad headings across the Software Lifecycle than contain the low hanging fruit we can pick early and I chose them because we can (almost) focus in each PA individually. After some initial informal success teams can look to expand their improvement initiatives and use more formal and complex methods – which is beyond what can be covered in this short document. The resources section lists places that you should look for more information and different perspectives.

There is no magic to software process improvement – it’s a question of: common sense; team and personal discipline; patience; attention to detail; and a strong desire by everyone to improve the current situation. Also what works will require you to experiment and use judgement (obtained by experience) to decide what will work in your projects. There is no ‘one size fits all’.

Agile Processes : The tools and techniques presented here are written primarily for the benefit of more traditional teams. If you call yourself an agile team, but have no process then I recommend you engage a reputable agile coach; or take a step back into more traditional software lifecycle, improve your processes (using this ebook and other resources) and then step forward again.

1Issues in this context is a very broad term that covers requests for new features, bug reports, internal development tasks and questions

Top Tips and Handy Hacks for Git

Last updated 7/June/2012

I don’t think any of these ideas are originally mine and I apologise that I do no longer have the correct information to credit the appropriate people.

This is my place to put some of the tips and tricks I come across when using Git. There are of course lots of other places on the inter tubes that provide better tips. These are just mine and I’ll update this over time

1. Before making the first commit to a repo check the email address (git config --get user.email) and if needed set the email address to different value on the repo  by omitting the --global option to the command git config user.email myhandle@cool.com)
2. Set up a global .gitignore file to be used by all repos
   1. Create a file ~/.gitignore with some useful exclusions (e.g. the backup files for your editor or .DS_Store on OS/X)
   2. run the command git config --global core.excludesfile ~/.gitignore
3. Help Window users and yourself by running git config --global core.autocrlf input (see GitHub help), but beware of lots of apparent line ending changes after setting this , these are benign.
4. Keep a copy of this handy picture around as an aide-memoir
5. Get a public Git repo account. My favourite is GitHub, other options include Gitorious and BitBucket
6. Would you like the git help text to appear in a browser. Run git config --global help.format web. N.B. Needs the html version of the Git documents installed. Don’t like the default browser? Run git config --global web.browser chromium
7. Using https instead of git as your transport protocol? You can now use various helper programs to cache your https credentials. The GitHub setup help page has details for each platform

Using Git from Bash?

1. Add the following to your ~/.bashrc file NB file locations correct for Ubuntu Linux 9.10 and Cygwin

   # enable git programmable completion features
   if [ -f /etc/bash_completion.d/git ]; then
       . /etc/bash_completion.d/git
       PS1='\[\e]0;\u@\h: a\]${debian_chroot:+($debian_chroot)}\u@\h:\w \$(__git_ps1 \' (%s)\')$'
   fi
   

   On Cygwin try PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ '. If you cannot locate the bash completions file on your system then you can get a copy from the Git source tree

2. If you prefer to use gvim for your commit messages then add the following to .bashrc (NB You c set this in ~/.gitconfig if you prefer git config --global core.editor=....) export GIT_EDITOR="/usr/bin/gvim --nofork". See below for more Vim tips

Using Git on Ubuntu GNU/Linux? Ubuntu users should add the Git PPA to their sources so that they get Git updates and new releases.

KDiff3 is available for Windows (handy for diffs and merges)

Using Git from Vim? — see the Vim Git tips

Git and Openoffice.org

N.B. These are very rough notes. See also

Sometime ago I shouted my mouth off at a local Perl Mongers meeting that using custom merge drivers it would be possible to auto merge OpenOffice document from Git, which seemed to interest some people greatly at the time.

However I then backtracked because I failed to take a few things into  account. After some more investigation it turns out it should be possible (with some wrapper scripts and OpenOffice programming) and I  *believe* the following approach stands a chance of working (I’m  hedging my bets here)

1) Create a wrapper script that opens a document, switches on change recording and saves the document, called say changeRecordingOn

2) Create another wrapper script that takes two documents (a and b), runs the OpenOffice Edit->Changes->Merge Document function and saves the result as c, called say ooAutomerge(a,b,c) (NB Base revision is  useless)

3) Create and associate a git custom merge tool for .odt and .ods files that runs ooAutomerge(). Might also need a custom merge driver?

4) Create a pre-commit hook that runs changeRecordingOn for all .odt and .ods files

When running a git-merge and then git-mergetool the OpenOffice GUI should be presented with a list of changes to accept/reject, which is much easier

NB It seems that running the OO merge and saving file c removes the previous change history. This means that if the file is merged again then the user only sees the new history since the last merge (which is what we want)

I doubt I will need this feature for some time so I don’t intend to  take it further at the moment — however if you have a need, these  notes might be useful..

I hope this makes sense? Apologies if I just swallowed two minutes of your day that could have been spend doing something better.

Easy Introduction to Git

I’ve just had an article published by Sitepoint that aims to introduce Git to people who may not have programming experience and are more comfortable using a GUI rather the command Line. “Git: Your New Best Friend“

Top Tip: Upgrade Git on Cygwin

The current version of git installed with Cygwin (1.5.x) has at least one annoying bug so it’s worth building  the latest version (NB I have almost all of Cygwin installed – you may need to install additional development tools and libs):

1. Download and unpack the source code. Pointer here
2. Run ‘./configure –prefix=/usr/local’
3. Run ‘make test’
4. Ignore errors relating to not detecting unwritable repos
5. run ‘make install’
6. run ‘hash -r’
7. Make sure it’s installed with ‘type git’ and ‘git –version’

Update March/09 — I found more detailed instructions

Handy Hack: Comparing two directory trees on Solaris 8

After some information from jim mcnamara I developed a script to compare two directory trees. I needed something that worked on Solaris 8 so it uses basic UNIX tools (and dircmp was not very useful). If you are using a more modern system then try this

Application deployment and change auditing with git

Updated 6/Feb/09

I’m currently working on on a project to deploy some complex banking systems and we have some somewhere in the region of 40 environments (various testing, production and development instances)

I’ve written some Perl to automatically configure environments using Text::Template. Now I’d like to automate deployment and have some form of change auditing (c.f.AIDE or Tripwire).

So a new project (which I hope will also  have some useful ideas for others):

• Project Outline and User stories
• General Approach using git (other tools could probably be used)
  • Define a staging repo
  • Create a remote branch of each deployment environment
  • Create a repo on each environment
  • Define a custom merge driver that overwrites (old files with new releases)
  • For each release
    • Create a new branch (overkill?)
    • Add new release to new branch
    • tag
  • To deploy
    • Merge (using new driver) onto environment branch
    • tag
    • Push to remote
    • On remote perform git reset --hard HEAD
    • configure
    • add and commit configuration changes
    • on staging pull config changes from remote repo
  • Now use git status etc. on environments to track any changes
  • Staging should have historical record of changes

Thanks to git I’m releasing IP into the wild!

This week I at last did something I’d wanted to do for a long time and  release some of Voga‘s intellectual property to the wider world.

I have already made material available either through this blog or via Slideshare. However for a geek it’s obviously better to publish complete projects in an Open Source manner so  they can be freely copied, modified and shared.

For historical reasons this had been inconvenient to do as I used to hold everything in a single Subversion repository. Not only did this contain my potentially open material, but also my customers private data and my business records (yes, I keep my accounts database in a version control repository!). A few months I migrated my Subversion  repo to git, but I still had a single large repo and no time or skill to fix it.

However today I bit the bullet

1. Removed all my possibly public data to another directory structure
2. Tidy up my current repo by removing all the old ‘trunk’ directories — as a solo consultant I was lucky and had never had to branch my projects. Commit my new streamlined repo

Now to work on the the public projects

1. Identify a complete project and make sure it had no proprietary data. This is potentially time consuming
2. Tidy up some of the names
3. Create a new git repo; add the project files; and commit
4. Create a new repo on Github for my project
5. Follow the simple instructions to upload my project to github

To Do

• Update existing references to my projects I can find and update to point to the repo
• Add the correct license files
• Start again for as many projects as possible, please be patient

I’ll still keep presentation copies on Slideshare because that is much more convient for most people.

A big thank you to the git developers and github for making this so easy

Setting up Serena Dimensions Design Parts and Allocating Roles

Once thing that often confuses Dimensions CM administrators and users is how to set up and use an effective Design Part Structure (DPS). There is no ‘once size fits all’ answer but I hope the following comments will help guide people to a more comfortable structure.

What is Application Lifecycle Management?

The analysts and tools vendors have been talking about Application Lifecycle Management (ALM) for many years now and I think they have (of course) been somewhat self serving in their definitions. ALM seems to  include whatever product category they have in the current price list and then promising their integrations’ would be managements answer to cost and governance issues. I hope to say more about integrations in the future, but I’d like to look at the scope of ALM for now.

If you ask vendors and consultants (including me) about ALM scope you’ll get answers such:

• Requirements engineering
• Software Configuration management
• Dependency Tracking
• Change control and work tracking
• Test management
• Software development tools and process
• Software building and processes
• Deployment
• Process Governance

Depending on the vendor you might even get more traditional: Project and Program Management;  Project and Application Portfolio Management/Analysis; and Service Management activities thrown in as well.

The usual sales pitch is that everything is managed via a common tooling and provides a common repository. The major, and often single, selling point of this approach is improved governance and business oversight of the SDLC.

However I think it’s useful and important to look at the definition of ALM from a customers perspective rather than as analysts or vendors.

It seems that a more pragmatic approach is to describe ALM as the methods, processes and tools that support the management of change for software systems — with particular, but not exclusive, emphasise on the SDLC. In this context it does not matter if we are using Agile or Waterfall approaches, we chose the best tools and processes for our particular situation. This of course implies that in eighteen months time we may need to re-tool.

Furthermore the methods tools and processes of project management (PM) are much more mature and there is no real need to include them in this evolving segment — they have their own. Obviously PM will have an influence and impact on ALM and we need to design our ALM approach to dovetail with it. I realise that most vendors will not agree with this.

I have come to the conclusion in order to ‘sell’ ALM into the teams that will use it we need to ensure, front and centre, that ALM provides a ‘better/faster/more’ improvement in daily productivity. That should be the primary focus of what is delivered.

Governance, audit etc should be secondary attributes and natural site affects of using ALM (important though they are of course and usually the primary sales message).

So at a practical level we need to be able to provide teams and their specialists with the best tools they need to do their job. Additionally we need to provide a supporting layer of data flows, events and data repositories to

1. Assist teams with traceability
2. Provide the tracking and management visibility to effectively steer the ship

So the point of this post is to plead with vendors to stop selling a one size fits all tool suite with deep integrations and start enabling shallow API and effective integrations to allow a truly best of breed approach for customers software purchases. It’s a shame that ALF project is shuting down.

Software Configuration Management: An introduction

This is a presentation I gave a few years ago to the Linux Users of Victoria (Australia) on SCM. It’s brief and simplistic, but hopefully will server as a jumping off point if SCM is confusing you. As always feedback is welcome.

Top Tip: Restrict Dimensions Requests to Prime Only

If you are a Serena Dimensions administrator you might find this of some use, otherwise I’m afraid it’s ‘move along, nothing to see here…’

Handy Hack:Convert a version controlled file to a different content type

This is not exactly rocket science, but does show up an important principle of Version Control (see below).

Let’s take a practical example that happened to me today:

I have a document under source code control (git in this case) and I am editing it using OpenOffice. Naturally it is an ISO standard Open Document format file and so it’s file name is something like GreatDocument.odt. However I now need to distribute this document to a wider audience for comment and feedback; and of course everyone else uses MS Word 2003. The practical solution is to change the document format to .doc. However I still want to keep the full history of the item — the fact that it changed from ODT to .DOC format is actually not significant in the grand scheme of things. For git the following commands need to be performed:

1. Using OpenOffice save the file GreatDocument.odt in MS Word format using the Save As function. This should create a second file called GreatDocument.doc
2. Rename GreatDocument.doc to something else, e.g. GreatDocument.word
3. issue a git-mv command to show that the file is changing. e.g. git-mv GreatDocument.odt GreatDocument.doc
4. Overwrite the .doc (which is actually in ODT format of course) with the true Word file. E.g. mv GreatDocument.word GreatDocument.doc
5. Now commit your changes to the repo. git-commit -a

Of course you will need to modify this to suite your platform and version control tool. But the principle should hold in the majarity of cases.

And what is the important Version Control principle involved?

The physical format of a controlled Item is less important that its history and logical purpose