Sorting Bundles and Parallel Compilation in PDE/Build

In PDE/Build the compile order for bundles has always been based on the feature structure. Features are visited depth first, and for each feature the included bundles are sorted according to their dependencies. Dependencies outside the given feature are not considered, and must be included in a previously visited feature.

This can lead to some less than ideal feature structures as releng teams try to ensure that everything that a bundle depends on is included in a "deeper" feature.

This has been fixed for 3.5 M3. You can now define a property "flattenDependencies=true" in your build configuration build.properties file. This will result in bundles being sorted across feature boundaries.

Previously, bundles got compiled by delegation through the build.xml scripts for the containing features. When using the new flattenDependencies option, a new compilation xml script will be generated in the build directory. This only affects compilation, other build stages (ie gather.bin.parts) will still be delegated through the feature structure.

Parallel Compilation

With the above changes in compile order, it turns out to be a small step to get parallel compilation. Set both flattenDependencies and "parallelCompilation=true" in your build configuration. The result is that the compilation xml script will then group bundles using ant's parallel task. The result is something that looks something like this:

<target name="main">
  <parallel threadsPerProcessor="3">
     <ant antfile="build.xml" dir="plugins/org.eclipse.swt" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.swt.win32.win32.x86" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.osgi" target="build.jars"/>
  </parallel>
 
  <parallel threadsPerProcessor="3">
     <ant antfile="build.xml" dir="plugins/org.eclipse.osgi.util" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.equinox.transforms.xslt" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.equinox.supplement" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.equinox.simpleconfigurator" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.equinox.p2.jarprocessor" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.equinox.launcher" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.equinox.launcher.win32.win32.x86" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.equinox.common" target="build.jars"/>
  </parallel>
 
  <parallel threadsPerProcessor="3">
     <ant antfile="build.xml" dir="plugins/org.eclipse.update.configurator" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.equinox.frameworkadmin" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.cvs" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.core.runtime.compatibility.auth" target="build.jars"/>
     <ant antfile="build.xml" dir="plugins/org.eclipse.core.jobs" target="build.jars"/>
  </parallel>
....

Each group depends only only bundles that appeared in a previous group. You can control the ant threading attributes by setting parallelThreadCount and parallelThreadsPerProcessor.

We tested this by using it to compile the Eclipse SDK. Compile time dropped from 6:53 to 4:54, while this is only a 2 minute savings, it is a 29% improvement.