Headless build problems with Grouped Configurations

PDE/Build has a property named groupConfigurations. If you set this property, build will group all the configurations (ie win32,win32,x86 & gtk,linux,x86) into one archive instead of creating a separate archive per configuration.

Kim & I, while working to get the performance baselines going, found a couple of gotchas/bugs around the use of this property.

Unexpected archive format for the group

The first thing we noticed was that the group archive was being created using ant's zip task instead of using the native zip as expected. Our builder's build.properties specified the following:

configs = *,*,*
archivesFormat = *,*,*-zip
groupConfigurations = true

If we had read the documentation, we would have seen that the archivesFormat is ignored for groups. However, this is not strictly true, and we can instead set a format for the group directly:

configs=*,*,*
archivesFormat = group,group,group-zip
groupConfigurations = true

This results in the zip format as desired.

The directory group.group.group does not exist

We also ran into the following error:

/builds/src/assemble.org.eclipse.sdk.tests.group.group.group.xml:257: The directory
/builds/src/tmp/eclipse/group.group.group does not exist

This turns out to be a rather old bug. The problem is that when features gather together the rootfiles they contribute, they copy them into platform specific folders. In this case, building the *,*,* configuration, the rootfiles were copied into tmp/ANY.ANY.ANY. However, because of the way the grouped configurations feature is implemented, the assembly scripts end up looking for the rootfiles in the group.group.group folder.

There are a few workarounds to this problem. One is to stick a mkdir in your customTarget/allElements assemble.<feature-id>.group.group.group target. This avoids the error, but you don't get the rootfiles in your group archive.

If you are using 3.4, then you can use the pre.archive target in customAssembly.xml to collect all the rootfiles into the correct folder:

   <target name="pre.archive">
      <!-- for each config being built -->
      <move file="${eclipse.base}/ANY.ANY.ANY/${collectingFolder}"
               todir="${rootFolder}" failonerror="false"/>
      <move file="${eclipse.base}/win32.win32.x86/${collectingFolder}"
               todir="${rootFolder}" failonerror="false" />
      <move file="${eclipse.base}/linux.gtk.x86/${collectingFolder}"
               todir="${rootFolder}" failonerror="false" />
   </target>

The ${rootFolder} property is defined by the caller of the pre.archive target, and in this case will be ${eclipse.base}/group.group.group/${collectingFolder}.

As usual, I have not actually tried running the above ant, the details may be different.

Sorting Bundles and Parallel Compilation in PDE/Build

In PDE/Build the compile order for bundles has always been based on the feature structure. Features are visited depth first, and for each feature the included bundles are sorted according to their dependencies. Dependencies outside the given feature are not considered, and must be included in a previously visited feature.

This can lead to some less than ideal feature structures as releng teams try to ensure that everything that a bundle depends on is included in a "deeper" feature.

This has been fixed for 3.5 M3. You can now define a property "flattenDependencies=true" in your build configuration build.properties file. This will result in bundles being sorted across feature boundaries.

Previously, bundles got compiled by delegation through the build.xml scripts for the containing features. When using the new flattenDependencies option, a new compilation xml script will be generated in the build directory. This only affects compilation, other build stages (ie gather.bin.parts) will still be delegated through the feature structure.

Parallel Compilation

With the above changes in compile order, it turns out to be a small step to get parallel compilation. Set both flattenDependencies and "parallelCompilation=true" in your build configuration. The result is that the compilation xml script will then group bundles using ant's parallel task. The result is something that looks something like this:

<target name="main">
  <parallel threadsPerProcessor="3">
     <ant antfile="build.xml" dir="plugins/org.eclipse.swt" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.swt.win32.win32.x86" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.osgi" target="build.jars"/>
  </parallel>
 
  <parallel threadsPerProcessor="3">
     <ant antfile="build.xml" dir="plugins/org.eclipse.osgi.util" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.equinox.transforms.xslt" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.equinox.supplement" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.equinox.simpleconfigurator" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.equinox.p2.jarprocessor" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.equinox.launcher" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.equinox.launcher.win32.win32.x86" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.equinox.common" target="build.jars"/>
  </parallel>
 
  <parallel threadsPerProcessor="3">
     <ant antfile="build.xml" dir="plugins/org.eclipse.update.configurator" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.equinox.frameworkadmin" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.cvs" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.core.runtime.compatibility.auth" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.core.jobs" target="build.jars"/>
  </parallel>
....

Each group depends only only bundles that appeared in a previous group. You can control the ant threading attributes by setting parallelThreadCount and parallelThreadsPerProcessor.

We tested this by using it to compile the Eclipse SDK. Compile time dropped from 6:53 to 4:54, while this is only a 2 minute savings, it is a 29% improvement.

Custom Compiler Arguments in PDE/Build

[Edit 2010/07/14: As of the Helios release, PDE/Build now supports per bundle custom compiler arguments using "compilerArg" in the bundle's build.propeties file.]

The other day there was a post to the eclipse.platform.pde newsgroup asking how to include debug information for classes compiled with PDE/Build. The answer is to use the compilerArg property.

This property sets compiler arguments to use when compiling all your bundles. This got me wondering: What if I wanted to set custom compiler arguments for just one bundle, or if different bundles needed different arguments?

There is no explicit support in PDE/Build to do this, but a little digging shows the way!

The Generated Compilation Target

First, take a look at the javac task in the generated build.xml for a bundle. For a typical jar shaped bundle, it looks something like this:

<javac destdir="${temp.folder}/@dot.bin" [...]>
   <compilerarg line="${compilerArg}" compiler="${build.compiler}"/>
   <classpath refid="@dot.classpath" />
   <src path="src/" />
   <compilerarg value="@${basedir}/javaCompiler...args"
                   compiler="org.eclipse.jdt.core.JDTCompilerAdapter"/>
   <compilerarg line="-log '${temp.folder}/@dot.bin${logExtension}'"
                   compiler="org.eclipse.jdt.core.JDTCompilerAdapter"/>
</javac>

Notice the first <compilerarg/>, it is passing the ${compilerArg} property that will affect all bundles. More importantly, notice the second <compilerarg/> whose value starts with '@'. A quick peek at the jdt docs reveals that this specifies a file where more command line arguments will be read.

javaCompiler.<library>.args

For each library specified in the bundle's build.properties file, PDE/Build generates a corresponding javaCompiler.<library>.args file. For most jar shaped bundles, this will be "javaCompiler...args".

We use this file to pass access rules (and custom file encodings) to the compiler. Access rules tell the compiler which packages you are allowed to see for each classpath entry; this is what allows PDE/Build to ensure that your compile-time classpath is as close as possible to the OSGi runtime classpath. The actual #ADAPTER#ACCESS# entries in this file are instructions to the JDT compiler adapter to modify the classpath with the given access rules.

The contents of this file are too ugly to bother pasting here, but the point is that this file is just additional compiler arguments. We can append to this file and insert whichever arguments we like.

Appending to the arguments file

We can use custom callbacks to append to the arguments file. For each library being compiled we can specify a target that will be called just before compilation. Here it is simple to append whatever we like to the arguments file:

<target name="pre.@dot">
   <concat append="true" destfile="${basedir}/javaCompiler...args">
-g
-preserveAllLocals
   </concat>
</target>

I haven't actually tried this for myself, so you may need to twiddle with the details a little. Note also that because the <compilerarg/> entries have

compiler="org.eclipse.jdt.core.JDTCompilerAdapter"

this will only work with the JDT compiler.

Example Headless build for a RCP product with p2

*UPDATE* For best results, I would suggest using the upcoming 3.4.1 release. Candidate builds are available from the download page under "3.4.1 Stream Builds", RC3 is available now.

I have prepared an example build setup that builds the RCP Mail Template and produces a p2-ized product. All the files needed for this are provided HERE.

The first step is to set up Eclipse with the RCP Delta pack which is required for the headless build of RCP products:

1. Get the Eclipse SDK and the RCP Delta pack and unzip into the same location on disk.
2. Start Eclipse and go to Window -> Preferences -> Plug-in Development -> Target Platform.
3. Uncheck the option "Build target platform based on the target's installed plug-ins."

Create a new plug-in "com.acme.rcp" based on the RCP Mail Template. Create a new Product Configuration file "acme.product" for this plug-in. On the overview tab of the product editor, I set the id "com.acme.rcp.product" and the application "com.acme.rcp.application". I also set the version to be "1.0.0.qualifier".

The p2 Installable Unit (IU) that will be created for this product will take its id and version from the product id and version set here. I am setting the version to end in "qualifier" so that I can replace the qualifier at build time with the build id.

The product configuration for this example is based on plug-ins. On the configuration tab remove all plug-ins, add the plug-ins listed below and click Add Required Plug-ins.

• com.acme.rcp
• org.eclipse.equinox.p2.exemplarysetup
• org.eclipse.equinox.p2.ui.sdk
• org.eclipse.equinox.p2.touchpoint.eclipse
• org.eclipse.equinox.p2.touchpoint.natives
• org.eclipse.ecf.filetransfer
• org.eclipse.ecf.provider.filetransfer
  

The product can now be run from overview tab of the product editor. If you do so you will notice the Help -> Software Updates... menu item. Selecting Software Updates at this time gets you a "This installation has not been configured properly for Software Updates" message.

Setting up the headless build

I like to run the headless build from inside Eclipse using a launch configuration. To do this, first import (Import... -> Plug-in Development -> Plug-ins and Fragments) "org.eclipse.pde.build" from the target platform into your workspace as a binary project.

Next, create a new general project named "Builder" and copy

org.eclipse.pde.build/templates/headless-build/build.properties"
                                  /customTargets.xml"

into the builder project.

I want my headless build to do 3 things differently from a normal build:

1. Set the version number in the product file to match the build id.
2. Generate p2 metadata
3. Use the generated metadata to perform p2 installs to get the final archives

1: Set the version number in the product file.

See the Build/customTargets.xml/preSetup target. We copy the "acme.product" file out of the com.acme.rcp bundle and into our buildDirectory. We then use the ant replace task to replace the version qualifier based on the time stamp. This uses a "acmePlugin" property which we will define separately. We also save the time stamp to a properties file that we read later when calling the p2 director.

2: Generate p2 metadata

PDE/Build provides integration with p2 to automatically generate p2 metadata. See Builder/build.properties where the following properties have been set:

    generate.p2.metadata = true
p2.metadata.repo=file:${buildDirectory}/repo
p2.artifact.repo=file:${buildDirectory}/repo
p2.flavor=tooling
p2.publish.artifacts=true

3: Perform p2 installs and archive the results

We do this in the Builder/customTargets.xml/postBuild target. We exec a new instance of eclipse to run the director application to install the product into a temporary directory and then we zip up the results.

We are running the same eclipse install again to do this. This means that any repositories that the eclipse install is set up with will also be considered during the director call (in addition to the repositories we pass on the command line).

The rest: build.properties

There are a few other properties to set in the build.properties file. We also set a number of properties on the command line in the launch configuration.

In the build.properties we set the product property to be the one that has the modified version qualifier. We also set the configuration we are building for.

 product=${buildDirectory}/acme.product
configs=win32,win32,x86

On the command line we are setting:

• pluginPath: allows us to not copy the com.acme.rcp plugin into the buildDirectory.
• buildDirectory: where all scripts will be generated and the location of our modified acme.product file.
• baseLocation: We use the eclipse install we are running as our base.
• acmePlugin: Define the location of the acme plugin for our custom task.

All these properties are set in the provided "Build Acme Product.launch" launch configuration and values are set at launch time using variables.

Results

Go to the Run Configurations... dialog and run the "Build Acme Product" launch config. The resulting zip should be under Builder/result. The project may need to be refreshed to see the results.

Running the build multiple times will result in the repository containing multiple versions of the product. You can then use that repository to update the product from one version to the next.

Problems Encountered

I did encounter some problems while creating this example.

• Bug 237662: The metadata generator currently does not properly handle the new p2 simpleconfigurator style of config.ini and loses some start level information. This causes the installed product to not start. Workaround for this example is to not include org.eclipse.equinox.simpleconfigurator in the product. I will provide another example later to work around this problem.


• Bug 222969: When including the org.eclipse.equinox.simpleconfigurator bundle in the product, the install is getting a bad relative path in the config.ini. Since we are excluding the simpleconfigurator because of the previous bug, this doesn't affect our example.


• Bug 237647: NullPointerException in the director. This only occured once, workaround was simply to remove the bad profile directory under my eclipse install (eclipse/p2/org.eclipse.equinox.p2.engine/profileRegistry/ACMEProfile.profile)

Doc Time (with a little help from p2)

It is that time of year again: documentation time. Irrespective of any personal feeling towards writing documentation, I do realize that it is quite important for something like PDE/Build.

I will start with the eclipse help itself and then perhaps write some blog posts with small examples and how-tos. I may even take a look at the venerable Build and Test Automation article. It was written for 3.0, now that we are hitting 3.4 I'm sure it could use an update.

And a little p2 plug:
I used to use mozilla to write the html help docs, but I decided I'd like to stay within Eclipse. I'm sure there has to be a decent html editor somewhere in Ganymede.

So, to the Software Updates dialog, I add the Ganymede update site (http://download.eclipse.org/releases/ganymede/). "Web and Java EE Development -> WTP Web Page Editor" looks promising. I click the checky box and install. p2 downloaded a bunch of stuff, and after a restart, voila! : A webpage editor, easy as pie.

Examples of using PDE/Build

It can be very useful to have an example when setting up your first headless build. It occurs to me that there is a growing number of relatively simple examples: the pde.build junit tests.

The org.eclipse.pde.build.tests project can be checked out of cvs from dev.eclipse.org/cvsroot/eclipse/org.eclipse.pde.build.tests. The main test suite is PDEBuildTestSuite.

If you run the tests with the vm argument pde.build.noCleanup=true then the tests will run without deleting any files and you can then go take a look at the setup for each test.

Not every test is a full build, but many are. Generally the tests consist of generating some features and plugins, then generating the build.properties and allElements.xml configuration files and then running the build.

Of particular interest would be the generation of the build.properties file. This is done by loading the template file and then setting just a handful of the properties. See BuildConfiguration.getBuilderProperties(IFolder). This should give a good idea of what properties you need to change for your build.

*Edit 1/28/08 : The "pde.build.noCleanup" argument was added on 1/23/08, you must use a build N20080124-0010 or newer. The first I-Build containing these changes will be I20080129.

On Building Cycles

One of the enhancements I am considering for pde.build in 3.4 is partial handling of cycles.

The latest idea is allowing a cycle that contains at least one binary bundle. The presence of the binary bundle could be enough to break the cycle and allow us to compile the remaining bundles.

Consider the following:

  A <- B <- C <- D <- E
       ----------^

Where we have a cycle between B, C, and D. If C is binary and does not need to be compiled, we may be able to compile the remaining bundles in the order A, D, B, E.

However, there are cases where this would not work. It could be that allowing this wouldn't actually help and would simply be giving people rope to hang themselves with.

If you have an opinion on this, please go comment on the bug and give the patches there a try.