1. Introduction

As with most crafts that require a degree of technical ability, the principles, theories and processes that surround software development have seen its fair share of improvement and growth over the years. Venture onto the web, and you’ll be able to find courses and seminars on any computing-related topic ranging from design patterns to reflective programming to software design paradigms to different kinds of software processes, each with its own devoted practitioners swearing high and low that their method is best.

In a nutshell: The sheer amount of information available on how to write better software in a more efficient manner is mind-boggling, to say the least. Amongst this sea of information on how to develop better software, development teams often turn their attention spans to writing better and more modular code. While not a mistake in itself (products consist of compiled code, after all) teams often tend to neglect the supporting structures and processes involved with software development.

One such process is that of building and releasing software. On large software projects, uninformed decisions regarding automated build systems (or even no automated build systems at all) often return to haunt developers in the form of code integration difficulties and dependency issues. Keeping track of which module was built for which project with which version of the code becomes, in short, a logistical nightmare.

This article will hopefully give you a brief overview of what build systems are, how you can use them, how you can plan to implement them and which technologies are suited to the task.

2. What is a build/release system?

Build and release systems usually comprises of a set of software tools that allow software developers to compile and release their code. Build systems vary greatly in scope – the technologies they utilise, how dynamic they are, the artifacts they create and how easy they are to implement and/or use.

3. How do build systems relate to continuous integration?

Continuous integration requires software team members to integrate their newly written code on a continuous basis. Automated builds are one of the cornerstones of continuous integration. By employing a build and release system, development teams minimise the risk of developing code that does not integrate with existing code. You also avoid the situation of having code that lives on a single developer’s pc.

4. Why would we want to implement a build/release system?

A build release system allows a development team to have a way of standardising the way their software is build and packaged for release. Again, why is this necessary? Why not just build the code on your workstation and release it? The answer lies in the fact that, even in a tightly controlled corporate environment where IT policy dictates what developers should have installed on their workstations, code that builds or runs on one machine might not build or run on another. By having a machine designated as a build server, with a properly implemented build and release system, you eliminate many of the variables that may set your software up for failure in the deployment environment. Not one single machine has the exact same runtime environments and software set up in the exact same way with the exact same environmental variables.

The decision of setting up a pre-existing automated build system package versus developing a build system from scratch depends entirely on the needs of your development team. You might not want to implement a build system from scratch, but instead opt for a software package like CruiseControl, Team Foundation Server or Hudson, to name a few. A package like CruiseControl interacts with a number of scripts to check code out of repositories, compile the code, copy the artifacts to the correct folders etc. Keep in mind that learning to set up and maintaining some of these automated build systems is not a trivial task. CruiseControl, for example, requires a working knowledge of Ant scripting from the operator.

On the other hand, if you have specific needs for your build system and you are prepared to put in the time and effort, a custom system might be a better choice. If the complexity of the software you develop requires features that are not present in any other automated build system, or if your build system needs to interface in specialized ways with other sub-systems within your organisation, you might want to consider going the DIY route.

5. What tools can I use to implement a build/release system?

Many build systems are an amalgamation of interacting scripts written in a variety of languages, ranging from DOS batch files or Linux Shell scripts to scripting languages more suited to the task of build automation, such as Ant (Another Neat Tool) or Maven.

I recommend that you learn to use at least Ant or Maven – either of the two will do. Both are extremely powerful tools when it comes to implementing your build system, and both provide the developer with a set of tools that eases the task of interacting with repositories and compiling code. Of the two tools, Maven probably has the steeper learning curve.

When deciding on which one of these to use, keep in mind the language and IDE your developers work with. IDE’s such as NetBeans allow the developer to choose between Ant and Maven when creating new projects. If your project has been set up with Ant, you might want to consider implementing your build system in Ant. Unless you are one of the adventurous types, of course!

Other valuable tools for build systems include web-based scripting languages (such as PHP) and, if you’re implementing something more complex, databases such as mySQL.

6. How can I go about implementing my build/release system?

6.1 Questions to ask beforehand

i. How maintainable and dynamic should my build system be?
When implementing from scratch, this question should be first and foremost on your mind. If the software you develop rarely requires repository gymnastics from your side, if you develop relatively few software packages with dependencies that are easy to keep track of, or if adding new code to be compiled by the build system is a rare occurrence, a simple Ant script that compiles your code might very well do the trick.

On the other hand, large software projects with multiple developers (some in remote locations) working on complex plug-in frameworks could require a bit more thought and planning than simply hacking together a set of scripts. If your build system requires absolute flexibility, with a minimum amount of effort required for setting up new projects within the build system, you could do worse than managing the build system with a database.

ii. What kind of code do I need to compile and link?
In other words, how do I compile the code that I check out of my repository? Keep in mind that your build system will automatically attempt to compile any code you commit to your repository, so to avoid build failures you’ll need to ensure that your code compiles command-line. Also important is the mechanism you use for compilation. Do your projects use C++ makefiles? Ant scripts? Perhaps they are built through Maven scripts?

iii. What artifacts do I need from my build system?
Artifacts refer to any product that is generated from your code. By compiling your code, you usually create a binary artifact. By linking code (as with C++), an executable. Artifacts may also include metadata about the code, such as Javadoc or Xlint.

iv. How do I version my artifacts?
How will your developer know which version of the software they’ve just released? Before implementing a build system you will need to determine how the software and its associated artifacts are versioned. Where do you keep a version history? In a database perhaps?

6.2 Implementing your build/release system – a few guidelines

i. Interfacing with your repository
Hopefully, your software project (or projects) will grow as it reaches maturity. This implies additional features, implemented in additional classes, packages and/or plug-ins. Since all of your code resides in the repository it would be well worth your time to investigate how to create a generic script that interfaces with your repository.

ii. Slaying the copy/cut/paste dragon
This relates to the maintainability of your system. Hard-coding the repository url’s of your project in an Ant script might work flawlessly first time round, but what if you decide to add paths? How about adding additional projects? The fact of the matter is that unless your projects are extremely small and uncomplicated, you really do not want to hack and re-hack a set of scripts to keep the build system up to date. Apart from the time spent on such activities, you’re very likely to make mistakes.

iii. Decisions on modularity
Once again, having your entire build system contained in one gigantic build script might work for small and uncomplicated projects. However, consider the hypothetical situation of releasing multiple artifacts for a project. If you only want to execute certain sections of a build script, you need to be able to do so without any major time expenditure.

7. Example of a build system implemented from scratch: The KBR (Kaizen Build Release)

7.1 Requirements for the build system

As the most junior member of a development team, our software manager bestowed upon me the great honour of implementing our team’s build and release system.

A bit of background information: Our software development takes place in Java, more specifically on the NetBeans platform. NetBeans encourages programmers to develop using a plug-in framework, which we adopted for our latest software releases. The move to Java has been a recent one – prior to this our products were developed using C++ and Qt. Build automation was achieved by means of CruiseControl.

Many of our plug-ins are collectively re-used as a core (aptly named the KORE, acronym for Kaizen Core) for the products we release. Bearing this in mind, it would make sense to allow developers to quickly set up a new product in the build release system. In other words, it should take the minimum amount of effort to chop and change plug-ins to create a product. Previously, setting up projects with CruiseControl was not a trivial task, and involved manual editing of configuration files.

Once a product has been defined in the build system, the creation of artifacts should happen automatically when a developer triggers a build.

7.2 Implementation of the build system

After taking the aforementioned requirements into consideration, the final build system was implemented to work in the following manner:

• Developers may set up new projects from a web interface (Information regarding a project is contained in a stand-alone database).
• Developers may log into the build system via the intranet:
  

  Kaizen Build Release (KBR) login

• The web interface allows developers to add modules to the build system database by interfacing with the repository. The developer may select a module from the trunk or tag of the repository, along with a module version:
  
  

  Adding modules, suites and libraries from the repository.

• The developer specifies a KORE for the software, which consists of any number of KORE plug-ins. The KORE is versioned separately from other products.
  
  

  Building a version of the KORE

• A product is defined by adding a pre-defined KORE, along with a number of additional application-layer plug-ins to the project.
  
  

  Setting up a project

• Along with the plug-ins contained in a project, the developer also specifies details such as where code should be compiled, where the repository is located, which classes should be javadoc’d, etc.
• Once this is saved, the build system automatically creates a set of build scripts for the project. These scripts are accessible from the project management server, from where builds are triggered.
• While all of this sounds rather complicated, an entire project’s build and release scripts can be set up within mere minutes. Since the web interface for the build system consists mostly of sets of dropdown boxes, the problem of typing errors (common in copy/cut/paste/change build systems) disappears.
• Once a project is set up, the developer may trigger a build for the software from a web interface on the project management server. The project management server checks out a small console bash script which triggers a target in the auto-generated build system scripts.
  
  

  Triggering a build

• The build release system compiles the code and creates the necessary artifacts. Once a build has been successfully completed, the build system automatically moves the artifacts to a folder and creates a small web interface through which each of the interfaces can be downloaded.
  
  

  Artifacts

• The developer navigates to the project’s homepage on the project management server, where all artifacts generated for the build are available for download, along with a build history (and each of the previous builds’ artifacts.) The build artifacts, from a to g:
  a. Binary release (.zip)
  b. Windows installer (.exe)
  c. Linux installer (.jar)
  d. Javadoc (tar.gzip)
  e. Unit test results (.htm)
  f. Plug-in manifest for build (.htm)
  g. Developer checkout script (.xml)
  
  

  Unit test results

• The entire process repeats itself. As a particular product grows, the developers simply add the new plug-ins from the repository to the build release system’s database.

7.3 Technologies used

• Polarion (the project management server. All builds are triggered from here.)
• PHP, HTML, CSS and Javascript for the web front-end.
• PHP, for dynamically creating Ant build scripts from a database.
• Shell scripts for completing specific actions, such as build cleanup.
• MySQL, for containing product data – versions, plug-ins, etc.
• Ant, for checking out code out of the repository, compiling NetBeans suites and modules etc.

8. Summary

Build systems frequently turn out to be a set of graceless, inflexible spaghetti scripts. All too often we find that user-friendliness is kept to a minimum in a system that never quite works the way everyone wished it did. By anticipating the way the development team will use such a system, we can eliminate many usability issues. For example, it is easy to make mistakes when copy/cutting/pasting parts of a build script, especially when setting up repository paths. I managed to eliminate this problem by providing a web front-end that interfaces with the repository. It’s difficult to go wrong when you can only select items from a drop-down list!

Another problem highlighted in this article was that of maintaining the build system as the software under development matures, and once again, having to hack and re-hack a set of build-scripts to reflect any changes in the software structure. I circumvented this issue by not having anyone edit any build scripts at all. Instead, a list of modules, along with any other important details of a software product is stored in a stand-alone database. Once any aspect of the product changes (e.g. adding a new module or changing a directory where code should be compiled) the web interface creates a brand-new set of build scripts.

Finally, by publishing build results in a central place, accessible in a window unambiguously titled ‘Latest build’ no-one needs to wonder where they can find the latest build of a product, or which artifacts were created for a specific build.

Hopefully this article will help you to implement a user-friendly, modular and flexible build and release system.

Remember: every hour you spend on improving the flexibility and user-friendliness of your system will see you reap exponential rewards when the implementation phase of your project arrives.

---

Kepler Engelbrecht
December 17th, 2009