Java Build Structure

Philosophy
Have you ever downloaded a piece of software only to find that it missing a dependency? Has a piece of software ever failed for you because your environment is slightly different to that of the developers? These problems are both frustrating and easy to fix. The build structure and process outlined here is designed around the following principals:


 * All software should be modular and self-contained
 * IDE Neutral: It should run from the command line
 * Simplicity: Checkout and Run

As developers, we want to create the best possible experience for any users (other developers in this case).

Every external dependency an application has is a potential failure point. A user may not have the required library, or, they may have an older or incompatible version. It doesn’t matter if you document the exact requirements of your application carefully and clearly, people will still miss it.

When they try to run your software and fail, they get annoyed. When they come for support and have to wait until you have free time to answer their question, they get annoyed. These problems are easily fixed by shipping everything your application needs directly with it. Installing an application should be as simple as drag-and-drop (or unzip). An application should be entirely self contained and considered a single module.

IDE preference is a point of much contention in developer circles. Regardless of personal preference, entangling a build process with a specific IDE is untidy and created an unnecessary dependency. At a minimum, all applications should be capable of being built from the command line.

To ensure that no problems are hidden behind the environment of an IDE, the build structure used here provides everything a developer needs to build, test and package Portico. It is just as easy to do via ssh on a headless box in a data center halfway around the world as it is to do on your local workstation.

Building an application from code is not uncommon in the open source world. Whether a user wants the latest developments or they wish to rummage around the code and make changes, there is nothing worse when a project assumes the presence of anything but the basics. In the Java world, this means the JDK and appropriate version control tools (be is Subversion or CVS) are the only acceptable requirements.

The process for building an application from source should involve two steps: '''1. Check it out, 2. Build it.''' NOTHING MORE. No running around getting the appropriate version of Ant or JUnit. No fetching the required plug-ins. Check it out and build it.

Using the Build Process
The build process is powered by Apache Ant. The provided build file includes a number of targets for typical processes (run ) to see all the targets:


 * compile: Compiles all the source code and places it in the  directory
 * javadoc: Generates javadoc documentation for all the source code in the  directory and places it in the   directory
 * release: Generates release tar-balls and zip files (see below)
 * sandbox: Generates a sandbox (see below)
 * test: Compiles all code and runs any unit tests (see below)
 * clean: Removes all artefacts generated by the build process

Thus, if you were using Linux, you would compile all your source code with the following command:

./ant compile

Compiling the Source
When you run the  target, all the source code is compiled and placed in the   directory. By default, that folder doesn't exist and will be created by the process.

The Subversion repository only contains the primary build artefacts, that is, the things that cannot be automatically generated. As such, things like compiled code, or javadoc HTML files are created only as needed. If at any point you want to remove any generated items, issue the  target.

The code for all the unit tests will only be compiled right before they are run. Running the tests is discussed later.

The Sandbox
The Subversion repository contains a number of items that are of no interest to a user. The sandbox is a directory that contains everything a user would see if they extracted the tarball or zip file. This allows developers to do testing in much the same environment their users would have. To generate the sandbox, use the following command:

./ant sandbox

When this target is invoked, the  folder is created and the following actions occur:


 * 1) All souce code is compiled
 * 2) The jar file containing the source code is built and copied into the sandbox (at )
 * 3) Any files contained in the  directory (and sub-dirs) will be copied into the jar file
 * 4) The  directory is copied to  . This will not include any files whose names start with "devel."
 * 5) The  directory is copied to   (sub-folders included, the whole thing goes across)
 * 6) The  directory is copied to  . Notice that it changes names. As with the   directory, this will not include any files whose names start with "devel."
 * 7) The contents of the  directory are copied into

Once this process is complete, the  folder represents what a user will see when they unzip the distribution file they download.

Here, we see how the codebase looks as it was checked out of Subversion: [tim@moby codebase]$ ls -l total 44 -rwxr-xr-x  1 tim      tim           208 Mar 23 14:52 ant -rw-r--r--  1 tim      tim           655 Mar 21 18:19 ant.bat -rw-r--r--  1 tim      tim         17953 Jun  7 01:26 build.xml drwxr-xr-x  3 tim      tim           102 Aug 22 23:59 documentation drwxr-xr-x  6 tim      tim           204 Jun  5 22:03 etc drwxr-xr-x 10 tim      tim           340 Jun  5 22:07 examples -rwxr-xr-x  1 tim      tim           362 Mar 17 13:56 javacc drwxr-xr-x  7 tim      tim           238 Jun  7 15:10 lib -rw-r--r--  1 tim      tim          5494 Jun  7 01:28 project-common.xml -rw-r--r--  1 tim      tim          3545 Aug  6 18:56 project-specific.xml drwxr-xr-x  7 tim      tim           238 Jun  7 01:15 resources drwxr-xr-x  5 tim      tim           170 Jun  5 22:03 scripts drwxr-xr-x  5 tim      tim           170 Jun  5 22:03 src drwxr-xr-x  4 tim      tim           136 Jun  5 22:02 src-test drwxr-xr-x  6 tim      tim           204 Jun  5 22:03 support

The next listing shows what it looks like after we run the sandbox command. The new directories has been highlighted:

./ant sandbox [tim@moby codebase]$ ls -l total 44 -rwxr-xr-x  1 tim      tim           208 Mar 23 14:52 ant -rw-r--r--  1 tim      tim           655 Mar 21 18:19 ant.bat drwxr-xr-x  4 tim      tim           136 Aug 23 00:01 build -rw-r--r--  1 tim      tim         17953 Jun  7 01:26 build.xml drwxr-xr-x  3 tim      tim           102 Aug 23 00:01 dist drwxr-xr-x  3 tim      tim           102 Aug 22 23:59 documentation drwxr-xr-x  6 tim      tim           204 Jun  5 22:03 etc drwxr-xr-x 10 tim      tim           340 Jun  5 22:07 examples -rwxr-xr-x  1 tim      tim           362 Mar 17 13:56 javacc drwxr-xr-x  7 tim      tim           238 Jun  7 15:10 lib -rw-r--r--  1 tim      tim          5494 Jun  7 01:28 project-common.xml -rw-r--r--  1 tim      tim          3545 Aug  6 18:56 project-specific.xml drwxr-xr-x  7 tim      tim           238 Jun  7 01:15 resources drwxr-xr-x  5 tim      tim           170 Jun  5 22:03 scripts drwxr-xr-x  5 tim      tim           170 Jun  5 22:03 src drwxr-xr-x  4 tim      tim           136 Jun  5 22:02 src-test drwxr-xr-x  6 tim      tim           204 Jun  5 22:03 support

This final listing shows the contents of the  folder (annotated with the location each item came from)

[tim@moby portico-0.6]$ ls -l total 4 -rw-r--r--  1 tim   tim    1802 Aug 23 00:07 README  (from resources/dist) drwxr-xr-x  8 tim   tim     272 Aug 23 00:07 bin     (contents of scripts) drwxr-xr-x  6 tim   tim     204 Aug 23 00:07 etc     (contents of etc) drwxr-xr-x  7 tim   tim     238 Aug 23 00:07 lib     (contents of lib, generated jar files) drwxr-xr-x  3 tim   tim     102 Aug 23 00:07 plugins (portico plugins)

Creating a Release
While the primary purpose of the sandbox is for use as a staging and testing area, it is also used as a temporary location for building releases. As releases are what it shipped to a user, the process of building them is far more rigid than just building the sandbox. For example, unlike creating the sandbox, release builds involve running all unit tests, and if any fail, the release will not complete.

To build a release, use the following command:

./ant release

The process involves is as follows:


 * 1) The release will run  so as to ensure that all releases are starting from the scratch and no problems can sneak into the build due to the current state of the file system.
 * 2) The unit tests are run to ensure that there has not been a known regression. If any test fails, the build will fail
 * 3) The sandbox is built
 * 4) All javadoc is created and anything in the  directory is copied into
 * 5) The appropriate license file is copied into the  directory
 * 6) Zip and Tar-zip files of the  directory are created and put in

Once this process has finished, the  directory will contain zip and tar-zip file distributables that can be passed on to users.

Building and running the Tests
The unit tests ensure that Portico is behaving as expected, and if it is not, they will cause a build to fail. More information on the Portico unit test suite can be found here.

To build and run the tests, call the  target:

./ant test

Project Layout Reference
The main section of the Portico repository (as checked out from ) is as follows:

''The  directory is just a container. When I talk about the "root" of the project I'm talking about being inside that directory'' -codebase |- ant |- ant.bat |- build.xml |- project-common.xml |- project-specific.xml |- documentation |- etc |- lib |- testng |- 5.1             |- testng-5.1-jdk15.jar |- resources |- dist |- jar |- test |- scripts |- src |- src-test |- support |- ant-bin

Apache Ant

Each project contains a copy of Apache Ant. Thus, there is no need to have it previously installed. The Ant installation includes a few optional extras that are used by the labs build file. The  and   files will invoke the Ant installation for you. If you are on Linux or OS X, remember to call the file as  so that the local script is run (not any one that may exist on your path).

The  file represents the main logic for the build process. It imports two other files:


 * : This file contains property and classpath declarations that are common to all projects using this structure
 * : This file contains project-specific build information

Source Code

Generally speaking, all source code is contained within the  folder. All the source code for unit tests (and other such non-production code) is contained in the  folder. Some projects may include additional source folders. For example, the Portico project includes a  folder that contains all the source code for the console.

Libraries

All dependent libraries are contained in the  directory. The structure for this directory is simple. There is a folder for each dependent library and beneath it is a folder for the specific version of that library. In that folder is all the artefacts needed by the project (jar files, licenses, etc...). The entire  folder is copied over into the sandbox or any release.

Resources

The  folder contains project resources. The sub-folders are:


 * : Any files in this directory (and sub-folders) will be copied into the root directory of the sandbox and any release. For example, this folder may contain a README file.
 * : The labs build process is focused around building a single jar file. The contents of this folder will be copied into root of that jar file. For example, the  file would be copied into the jar file at the location
 * : This folder is where artefacts used for testing can be placed. The contents are not made part of the sandbox of releases.

Other Locations

The  directory is where project documentation can be placed. When javadoc is generated, the build process places it in here. When a release is generated, this folder will be copied into it. Note that this will not happen on a sandbox build.

The  directory contains any executable files that should go into a release of your project (shell scripts, batch files, whatever). All artefacts in this directory will be placed in the  directory of a generated release or sandbox. Any files whose names start with "devel." will be ignored and not copied over to the sandbox or into a release.

The  directory is where all configuration files for your project should go. This folder will be copied into any sandbox or release builds as it. Any files whose names start with "devel." will be ignored and not copied over to the sandbox or into a release.

The  directory contains the copy of Ant used by the build process.