Releng tricks from e4 and Orion

In the last couple of months I've found myself in charge of two releng builds: e4 and Orion. The e4 build is actually 2 pieces: building the Eclipse 4.1 SDK and building additional e4 bundles which are not part of the SDK by default.

Being the PDE/Build project lead gives me a unique perspective on this entire process so I thought I would share some tip and tricks for specific problems I encountered.

The first covers how we do signing when building the Eclipse 4.1 SDK.

Signing the Eclipse 4.1 SDK

We produce signed bundles in our builds. The specifics of how to do this have already been worked out by Kim. Essentially we provide a zip file that gets sent off to eclipse.org to be signed.

For the 4.1 SDK there is a slight twist to the problem. The 4.1 SDK is mostly composed of binary bundles we reconsume from 3.7 together with some new e4 bundles that we compile ourselves. We really only want to sign the bundles that we compiled ourselves and avoid resigning the binary bundles.

The trick for creating an archive containing only the bundles we compiled works best for p2 enabled builds (using p2.gathering=true).

Custom Assembly Targets

PDE/Build supports customization of your build using provided template files. In particular we are interested in the customAssembly.xml script. This provides targets that will be invoked by PDE/Build during the packaging and assembly phases of the build.

Specifically, there is a target gather.bin.parts which is invoked for every bundle that we are building immediately after the contents for that bundle are published into the p2 repository. There is another target post.gather.bin.parts which is called after we are finished with all the bundles.

The idea is that we use the gather.bin.parts target to record which bundles we compiled, and the post.gather.bin.parts to sign these bundles and update the p2 repository. At the time post.gather.bin.parts is called, the p2 repository will contain binary bundles as well as the compiled ones which is why we need a record of which ones to sign.

The script looks something like this:

<project name="CustomAssemble.overrides" default="noDefault">
<import file="${eclipse.pdebuild.templates}/headless-build/customAssembly.xml" />

<!-- every time gather.bin.parts is called, we will record the project being built -->
<target name="gather.bin.parts" >
  <echo append="true" file="${buildDirectory}/built.list"
                      message="**/${projectName}.jar${line.separator}" />
</target>

<target name="post.gather.bin.parts" >
  <property name="signingArchive" value="${buildDirectory}/${buildLabel}/sign-${buildId}.zip" />
  <zip zipfile="${signingArchive}" basedir="${p2.build.repo}"
          includesFile="${buildDirectory}/built.list" />

  <!-- sign! -->
  <ant antfile="${builder}/sign.xml" dir="${basedir}" target="signMasterFeature" >
     <property name="signingArchive" value="${signingArchive}" />
  </ant>
  
  <!--unzip signed archive over top of the repository -->
  <unzip dest="${p2.build.repo}" src="${signingArchive}" />

  <!--update repository with new checksums for signed bundles -->
  <p2.process.artifacts repositoryPath="file://${p2.build.repo}" />
</target>
</project>

Some notes:

• ${projectName} is a property set by PDE/Build and it contains the bundle symbolic name and the version of the bundle being built (ie org.eclipse.foo_1.0.0.v2011).
• The bundles are recorded in built.list in the form of an ant include pattern.
• The signing archive is created from the p2 repository using the generated built.list as an includes file.
• The sign.xml script being used is the one from the e4 build and is available here.
• The p2 artifact repository contains checksums for each artifact, so after extracting the signed archive over top of the repository, we need to update the repository to recalculate these checksums.
• I have not actually tested the above ant snippet, it may require some tweaks. The general strategy is based on what we do in the e4 build but some of the details have changed.