Releng tricks from e4 and Orion

In the last couple of months I've found myself in charge of two releng builds: e4 and Orion. The e4 build is actually 2 pieces: building the Eclipse 4.1 SDK and building additional e4 bundles which are not part of the SDK by default.

Being the PDE/Build project lead gives me a unique perspective on this entire process so I thought I would share some tip and tricks for specific problems I encountered.

The first covers how we do signing when building the Eclipse 4.1 SDK.

Signing the Eclipse 4.1 SDK

We produce signed bundles in our builds. The specifics of how to do this have already been worked out by Kim. Essentially we provide a zip file that gets sent off to eclipse.org to be signed.

For the 4.1 SDK there is a slight twist to the problem. The 4.1 SDK is mostly composed of binary bundles we reconsume from 3.7 together with some new e4 bundles that we compile ourselves. We really only want to sign the bundles that we compiled ourselves and avoid resigning the binary bundles.

The trick for creating an archive containing only the bundles we compiled works best for p2 enabled builds (using p2.gathering=true).

Custom Assembly Targets

PDE/Build supports customization of your build using provided template files. In particular we are interested in the customAssembly.xml script. This provides targets that will be invoked by PDE/Build during the packaging and assembly phases of the build.

Specifically, there is a target gather.bin.parts which is invoked for every bundle that we are building immediately after the contents for that bundle are published into the p2 repository. There is another target post.gather.bin.parts which is called after we are finished with all the bundles.

The idea is that we use the gather.bin.parts target to record which bundles we compiled, and the post.gather.bin.parts to sign these bundles and update the p2 repository. At the time post.gather.bin.parts is called, the p2 repository will contain binary bundles as well as the compiled ones which is why we need a record of which ones to sign.

The script looks something like this:

<project name="CustomAssemble.overrides" default="noDefault">
<import file="${eclipse.pdebuild.templates}/headless-build/customAssembly.xml" />

<!-- every time gather.bin.parts is called, we will record the project being built -->
<target name="gather.bin.parts" >
  <echo append="true" file="${buildDirectory}/built.list"
                      message="**/${projectName}.jar${line.separator}" />
</target>

<target name="post.gather.bin.parts" >
  <property name="signingArchive" value="${buildDirectory}/${buildLabel}/sign-${buildId}.zip" />
  <zip zipfile="${signingArchive}" basedir="${p2.build.repo}"
          includesFile="${buildDirectory}/built.list" />

  <!-- sign! -->
  <ant antfile="${builder}/sign.xml" dir="${basedir}" target="signMasterFeature" >
     <property name="signingArchive" value="${signingArchive}" />
  </ant>
  
  <!--unzip signed archive over top of the repository -->
  <unzip dest="${p2.build.repo}" src="${signingArchive}" />

  <!--update repository with new checksums for signed bundles -->
  <p2.process.artifacts repositoryPath="file://${p2.build.repo}" />
</target>
</project>

Some notes:

• ${projectName} is a property set by PDE/Build and it contains the bundle symbolic name and the version of the bundle being built (ie org.eclipse.foo_1.0.0.v2011).
• The bundles are recorded in built.list in the form of an ant include pattern.
• The signing archive is created from the p2 repository using the generated built.list as an includes file.
• The sign.xml script being used is the one from the e4 build and is available here.
• The p2 artifact repository contains checksums for each artifact, so after extracting the signed archive over top of the repository, we need to update the repository to recalculate these checksums.
• I have not actually tested the above ant snippet, it may require some tweaks. The general strategy is based on what we do in the e4 build but some of the details have changed.

Versioning & p2, slides from EclipseCon

This year at EclipseCon, I presented a 10 minute talk on the importance of versioning with p2. I have posted the slides here. Because this was only a 10 minute talk, there is not a lot of content in the slides, so this post is an attempt at an overview.

id + version == 1 set of bytes

Hopefully by now everyone has heard this before. In the world of p2, an id and a version represent a particular set of bytes. If multiple copies of a given artifact exist in multiple places, and they all have the same id and version, they are assumed to be the same (or equivalent) bytes.

In particular, this means that if your user already has a bundle org.foo_1.0.0 on his machine, and you are trying to deliver an update, then the user will not download your updated org.foo if the version number is still 1.0.0.

So make sure you increase your version numbers, it is hard to support a customer when you can't tell if his org.foo_1.0.0 is the broken version or the fixed version!

We tend to version sources

The process we follow in Eclipse tends to version sources. We tag our source code with the version qualifier which determines the version of the resulting binary. The idea is this leads to reproducible builds.

However, we must realize that there is more than just the source code that affects what the resulting binary looks like. These are things like what compiler was used, the build scripts, and most importantly the dependencies that were on our build-time classpath.

If some bundle B depends on bundle A and A changes in some way, then recompiling B can potentially result in different byte-code even though the source code did not change. In this case, B deserves to have its version number increased.

Mirroring with a baseline and comparator

This scenario of a bundle changing without having its version incremented has happened more than once in the Eclipse SDK releng builds. In order to detect this, we use a previous build as a baseline in a mirror operation. If the baseline already contains a bundle with the same version as the one that was just built, then we mirror the old bundle and discard the new one.

We install our product using the mirrored results which ensures that our new install contains the same old bundles that already exist out on user's machines. The unit tests run against these old bundles.

In order to detect when a bundle actually changes and needs its version number increased, the mirror operation supports a comparator. Olivier kindly contributed some code to p2 that can do a semantic compare on java class files to see if they are equivalent. Similarly, things like manifests and properties files can be compared for semantic instead of bitwise equivalence.

Example Build

I put together an example feature build that uses a comparator to detect if a bundle has changed when the version hasn't. The projects are in CVS under dev.eclipse.org:/cvsroot/eclipse/pde-build-home/examples/comparator. There are 4 projects to check out:

• example.Builder - the builder project, contains a runBuild.xml
• org.example.a - a project that we will change between builds
• org.example.b - depends on a, won't change source between builds
• org.example.feature - the feature to build

To run the example, just run runBuild.xml script as an Ant Build using the same JRE as the workspace. The build will mirror the results into example.Builder/composite/I<timestamp>. You can run the build multiple times, and each time the results will be added to the composite repository.

The mirror call is done in example.Builder/customTargets.xml/postBuild. It looks like this:

<target name="postBuild">
 <antcall target="gatherLogs" />

 <!-- mirror from build results, comparing against previous builds that are in the composite repo -->
 <p2.mirror>
     <source location="file:${assemblyTempDir}/${buildLabel}"/>
     <destination location="file:${builder}/composite/${buildLabel}"/>
     <comparator comparator="org.eclipse.equinox.p2.repository.tools.jar.comparator" comparatorLog="${buildDirectory}/comparator.log">
         <repository location="file:${builder}/composite" />
     </comparator>
 </p2.mirror>

 <!-- add the new build to the composite -->
 <p2.composite.repository destination="file:${builder}/composite">
     <add>
         <repository location="file:${builder}/composite/${buildLabel}"/>
     </add>
 </p2.composite.repository>
</target>

Each build uses the previous contents of the composite repository as a baseline, and then adds itself to the composite.

To see the comparator in action, edit org.example.a.Sub and uncomment the doSomething method:

public void doSomething(List o) {
 for (Iterator iterator = o.iterator(); iterator.hasNext();) {
     super.doSomething(iterator.next());
 }
}

This change to the source code of org.example.a will change which doSomething B ends up calling:

public class B {
 public void method() {
     ArrayList list = new ArrayList();
     list.add(this);
 
     Sub d = new Sub();
     d.doSomething(list);
 }
}

That is, recompiling B against the new A results in different byte-code for B. If we run the build again with the changed source, then it will now fail with an error message:

[p2.mirror] Messages while mirroring artifact descriptors.
[p2.mirror] Compare and download of canonical: osgi.bundle,org.example.b,1.0.0 from baseline.
[p2.mirror] Difference found for org/example/b/B.class within [canonical: osgi.bundle,org.example.b,1.0.0]
                     from file:/C:/workspace/example.Builder/composite/I20090806060019/

When the build fails in this manner, it means we need to increment the version for org.example.b

Build Notes

1. The builder's build.properties sets pluginPath and elementPath to enable the build to use the sources in the workspace without copying them to the buildDirectory. elementPath points to the top level feature that we are building.
2. The build is using p2.gathering=true, for feature builds, this groups the configurations, we set archivesFormat to leave the results as a folder.
3. runBuild.xml automatically refreshes the workspace when it finishes, however when the build fails after changing the source in org.example.a, this refresh doesn't happen and you need to refresh the workspace manually to see the new results.
4. In customTargets.xml/preGenerate we create an empty composite repository if there wasn't already one there.
5. We changed the default customTargets.xml/gatherLogs to use elementPath since the top level feature is not under buildDirectory.
6. org.example.a has version 1.0.0.qualifier where the qualifier gets replaced each build with the timestamp. org.example.b just has version 1.0.0. In a releng build, the CVS tag from a map file would be used instead of the timestamp.
   

ADT Part 2: More like the EPP

Yesterday, I posted an example of how to compose a product made up out of updatable sub-components. The first question there was about how this relates to the packages built by the EPP. The EPP packages are just products, much like my ADT. They use their own feature and plugin to brand the Eclipse package and do things like open a different default perspective.

The EPP packages are built slightly different from my first example. They use feature requirements to get version ranges for the sub-components, but they don't currently perform the extra director calls to make those sub-components into updatable roots.

Here is another example that is structured a bit more like the EPP packages.
This example is also available in cvs with the first (dev.eclipse.org:/cvsroot/eclipse/pde-build-home/examples/adt.feature.builder).

[Edit 2013/07/03: Eclipse CVS has migrated to git starting in 2010.  The examples are now available under the examples folder in http://git.eclipse.org/gitroot/pde/eclipse.pde.build.git. The example has not been updated to work with git and may require modifications.]

Features instead of p2.inf

In our first example, we used a p2.inf file to add requirements with version ranges to our product. I did it this way because I didn't have anything to add to the product, and didn't want to bother creating a feature.

Perhaps a more familiar way of doing things would have been to use a feature. So we create a feature "adt.feature", and instead of including our sub-components, we require them using a "compatible" match rule:



I also made my feature include the "org.eclipse.platfom" feature. The adt.product file then just includes adt.feature. We no longer need the p2.inf to add requirements to our product.

Adding Branding

Since we created our own feature for this product, we may as well take the next step and add our own branding plug-in. The first thing I did way create my own product extension in the plugin.xml:

    <extension id="product" point="org.eclipse.core.runtime.products">
      <product application="org.eclipse.ui.ide.workbench" name="ADT Product">
         <property name="aboutText" value="Andrew's Development Tools"/>
         <property name="windowImages" value="icons/icon.gif"/>
         <property name="aboutImage" value="product.gif"/>
      </product>
   </extension>

Note that the word 'product' here is slightly overloaded. There is a 'org.eclipse.core.runtime.products' extension point that defines branding for eclipse, and there is the product itself which is what we are building using the .product file.

This product is just running the normal workbench application, but it uses my own icons and images. We then change our adt.product file to use adt.plugin.product instead of the org.eclipse.platform.ide product extension we were using before.

In my adt.feature I set the branding plug-in to be my new adt.plugin. I also provide the normal about.* files so that my feature shows up in the about dialog.

As a final touch, I made my own splash screen.

Building the new Product

The build script is essentially unchanged from the first example. So I won't bother explaining the details again. The only difference is that I made some minor changes to the builder's build.properties file so that pde.build can find our feature and plugin.



Note that PDE/Build does not follow feature requirements when performing a build and publishing metadata. This means that for the director install to work, you need to have pre-existing metadata for the things that you are requiring. This same requirement exists with the first example where we used a p2.inf file.

Composing and updating custom Eclipse distros

I've recently seen a couple of different posts to the newsgroups dealing with problems updating RCP applications using p2. [edit 2009/10/21, update links to forums]

As an example, I've created my own Eclipse product. It is composed of the Eclipse Platform, CVS support, the CDT and Mylyn. I'm calling it the ADT (Andrew's Development Tools).

It's not hard to create a feature based product that includes these things, and do a product build to end up with something like this:



As explained in this newsgroup post, there are two kinds of things that are included in an Eclipse install:

1. Things that are explicitly installed
2. Things that are required by the things that are installed.

Here in my example, only my development tools "org.example.adt" is installed, the rest (CDT, CVS, Mylyn) are required by my product.

Only things that are explicitly installed will be searched for when you look for updates. Also, the installed things generally specify the versions of things they require, which makes it hard to install/update those required items independently of the root product. In both the newsgroup postings I referred to above, the problem was trying to install/update one of the required items without updating the root product.

So the question becomes how to allow updating sub-components of the product without updating the product itself.

Composing for Updatability

What we want to do is to update sub-components of the product without updating the root product itself. In this example we do not to allow updating the Eclipse Platform independently, to do that, the user will need to update the product itself.

I have created a example builder to do this. Get it from cvs (dev.eclipse.org:/cvsroot/eclipse/pde-build-home/examples/adt.builder).

[Edit 2013/07/03: Eclipse CVS has migrated to git starting in 2010.  The examples are now available under the examples folder in http://git.eclipse.org/gitroot/pde/eclipse.pde.build.git. The example has not been updated to work with git and may require modifications to run properly.]

We need to do two things:

1. Use version ranges to include sub-components in our product so that we allow upgrading those components.
2. Explicitly install those sub-components so they will be found when checking for updates. This is essentially a book-keeping step.

The ADT .product File

There is a adt.builder/product/adt.product file which we will use to run a product build. If we were to include the features for our sub-components in the .product file, then we would end up with requirements on specific versions of those components. Instead we only include the platform feature [1].

To get requirements to our sub-components, we use a p2.inf file to customize the metadata. We add requirements with entries that look like this:

requires.1.namespace = org.eclipse.equinox.p2.iu
requires.1.name = org.eclipse.cvs.feature.group
requires.1.range = [1.1.100, 1.2.0)

requires.2.namespace = org.eclipse.equinox.p2.iu
requires.2.name = org.eclipse.mylyn_feature.feature.group
requires.2.range = [3.2.0, 3.3.0)

...

The .feature.group suffix is the name of the p2 Installable Unit corresponding to the features we are interested in. We specify the version ranges in which we will allow those components to be updated.

The ADT Builder

The adt.builder project includes a buildADT.xml ant script which will run a headless product build for us. The first thing it does is download zips containing the things we need. This example illustrates three different ways of reconsuming metadata.

1. The CDT and CVS both come as zipped p2 repositories. Things that are not referenced directly by the .product file only need to be available as repositories. We can reuse these zips directly by specifying them as context repositories using jar: urls. See the p2.context.repos property in the adt.builder/build.properties file.
2. Mylyn is not a p2 repository, it is a zipped old style update site. For this, we use a publisher task to generate p2 metadata for it. [2]
3. The Eclipse Platform is a p2 repository just like the CDT and CVS. It is similar to the delta pack in that it contains the org.eclipse.equinox.executable feature that is need to get launchers in product builds. Because the platform feature is included directly in the product, we can't just specify the platform as a context repository, we need the bundles available to pde.build like in a normal headless build. To do this we transform the repository using the p2.repo2runnable task. See the transformedRepoLocation and repoBaseLocation properties in the build.properties file. The transformed repository automatically gets included along with the pluginPath property used by pde.build.

Adding additional director calls

In order for our sub-components to be independently updatable, they need to be explicitly installed in our resulting product. By default PDE/Build performs a director install for just the product being built. We can use a customAssembly.xml script to perform additional director[3] calls before the final archive is created.

It looks like this:

<target name="pre.archive">
<ant antfile="${genericTargets}" target="runDirector" inheritAll="true">
<property name="p2.repo" value="${p2.build.repo}"/>
<property name="p2.director.iu" value="org.eclipse.cvs.feature.group"/>
<property name="p2.director.installPath" value="${eclipse.base}"/>
</ant>
...
</target>

We make director calls for each of the sub components we allow to be updated. In the example we do CVS, Mylyn, CDT, and the CDT-Mylyn bridge.

The final result

Run the adt.builder by right-clicking on buildADT.xml and choosing Run As -> Ant Build... Be sure to run in the same JRE as the workspace. After running the build, the results are available under adt.builder/buildDirectory/I.<timstamp>.

Running the resulting product, we see that the CDT, Mylyn and CVS are all showing up as installed roots, and are therefore independently updatable.

Notes

1. PDE/Build will automatically generate start level configuration information, but only for things that are included in the .product file. If we didn't include the platform feature, or at least the bundles that need start level information, then this would not happen automatically and we would need to handle start levels ourselves. See the help page here for more information of configuring start levels.
2. We publish the p2 metadata for mylyn into ${p2.build.repo}. This property specifies the location of the p2 repository that will be used internally by the build. Publishing the mylyn metadata here instead of some location specified as a context repository saves the build from mirroring the required IUs into the build repository.
3. PDE/Build provides a "runDirector" target that can be used to invoke the director. This works by executing the director application in a new process. Normally, this requires setting the "equinoxLauncherJar" property specifying the location of the equinox launcher to use, but because we are calling the director from customAssembly.xml, we inherit this property from the generated assembly scripts.
4. Running this build produces a properly p2 enabled product. It does not produce a corresponding repository for that product other than the build time repository ${p2.build.repo}. To produce a final repository containing the final product, define the properties p2.metadata.repo and p2.artifact.repo in the build.properties. The product and its requirements will then be automatically mirrored into that repo.

Patching Features (Part 2)

In my last post I showed how to create a simple feature patch. However, this patch leaves us in a state where we can no longer upgrade the platform because the patch itself requires a specific version of the feature it is patching.

Making the patch Optional

If we make the inclusion of the patch in our org.example.feature optional, then we no longer block upgrading the platform underneath us.


When a new version of the platform is available, there is a conflict between the version of the p2.user.ui feature required by the patch and the one required by the new platform. By making the patch optional, p2 will not install the patch if there is a conflict.

This is fine if the new version of the platform includes the fix we want. But what if it doesn't? What if we need to have our patch apply to several different versions of the platform?

Relaxing version ranges

Unfortunately, currently the only way to do the following is to edit the metadata by hand. (Paul Webster uses XSLT transforms, I have also raised a bug to allow changing things using the p2.inf file.)

Looking at the metadata for the patch, we see a few interesting sections:

The Patch Scope

The scope section of the patch's metadata specifies which feature(s) the patch applies to:

<patchScope>
<scope>
  <requires size='1'>
    <required namespace='org.eclipse.equinox.p2.iu'
                 name='org.eclipse.equinox.p2.user.ui.feature.group'
                 range='[1.1.0.v20090605-1440-7u6Fb3FbPbJP5MiKiZgpdl,1.1.0.v20090605-1440-7u6Fb3FbPbJP5MiKiZgpdl]'/>
  </requires>
</scope>
</patchScope>

Here we see the patch applies to a specific version of the org.eclipse.equinox.p2.user.ui feature. By relaxing the version range here we can make the patch apply to other versions of the platform. For example, say we knew our bug was fixed on June 11, but there were several version between June 5th and the 10th that still contained the bug. We could instead use a version range like [1.1.0.v20090605-1440,1.1.0.v20090611) (the upper end of the range is open, it includes everything up to but not including v20090611).

Note that the patch Installable Unit also has normal requirements on the feature it is patching. When widening the scope range, the regular requirements should also be widened to match.

Change From / Change To

The 'changes' section of the patch's metadata specifies which plug-in we are patching, and the version to change it to.

<changes>
<change>
  <from>
    <required namespace='org.eclipse.equinox.p2.iu'
                 name='org.eclipse.equinox.p2.touchpoint.eclipse'
                 range='0.0.0'/>
  </from>
  <to>
    <required namespace='org.eclipse.equinox.p2.iu'
                 name='org.eclipse.equinox.p2.touchpoint.eclipse'
                 range='[1.0.101.v20090611,1.0.101.v20090611]'/>
  </to>
</change>
</changes>

In the above example, we are changing the org.eclipse.equinox.p2.touchpoint.eclipse bundle from any version ("0.0.0") to version 1.0.101.v20090611.
We could imagine changing these ranges in the following ways:

• Change the "scope" range to "0.0.0" to match any version of the p2.user.ui feature, and then change the "change from" range to [1.0.101, 1.0.101.v20090611). This would result in patching any version of the p2.user.ui feature that includes any 1.0.101 versioned bundle up to v20090611 (which is the version with the proper fix).
• Change the "change to" range to something like [1.0.101.v20090611, 1.0.102). We know v20090611 contains the fix we want, but the in the future there will be more bug fixes before the maintenance stream is released. By widening the range here, we allow future bug fixes to also be included in our patch.
  

Patching features with p2

The Galileo release of Eclipse is in its final days. The eclipse SDK itself is effectively done and the current build is under consideration to be the final release.

Imagine This...

Say I have a feature that I build, it runs on top of the Eclipse Platform. I know Eclipse Galileo is coming out soon, so I download it and try it out with my feature.

Oh No! It doesn't work! I've found a bug in the platform, and its a major blocker for me. (Oops, maybe I should have tried this a couple of months ago when there still would have been time to fix the bug.)

Meta-Comment: The Eclipse Platform Project has a reputation for shipping on time (even if that means there are unresolved bugs). There is a strict end game plan that is followed for the release, lock down started back at the beginning of May. The further we are along in the plan, the harder it is to get a fix approved for release. A lot of people don't seem to relealize this happens, and perhaps wonder why their important bugs are defered with the comment "Its too late".

Ok, I need to patch the platform. How do I do this?

The Example Feature

To work through the steps of patching the platform, I created my own example feature:

1. Create a New Plug-in Project "org.example.plugin". The plug-in is not a "Rich client Application". On the templates page of the wizard I chose "Plug-in with a view".
2. Create a New Feature "org.example.feature". Add org.example.plugin to the feature.
3. Export the feature, and on the Options tab in the export wizard check "Generate metadata repository".

For the purposes of the example, I'm starting from the Eclipse Platform Runtime 3.5RC4. Because I want to strictly control the upgrade path in the example, I removed the Galileo and Eclipse Project repositories from the available sites (Window -> Preferences -> Install/Update->Available Software Sites).

Unzip the platform, and installed the feature from the exported repository. I didn't create any categories for the feature, so "Group items by category" must be unchecked.

The Patch

For the example, I am inspired by bug 279542 which came in very late and almost was not fixed in 3.5. The bundle org.eclipse.equinox.p2.touchpoint.eclips has the bug and this plugin is included by the feature org.eclipse.equinox.p2.user.ui.

Say we have a fix with touchpoint.eclipse version 1.0.101.v20090611. (When following the eclipse versioning guidelines, the maintenance version of this bundle should be 1.0.101 and the version for the next year's release will probably be 1.0.200).

We can easily create a new Feature Patch with the wizard:
We specify the version of p2.user.ui that we are patching (here the one shipped in 3.5RC4). In the feature patch, include org.eclipse.equinox.p2.touchpoint.eclipse with the specific version containing the fix. Then, add the feature patch as an inclusion in our org.example.feature:

<feature id="org.example.feature" label="Feature" version="1.0.0.qualifier">
 <includes id="org.example.patch" version="0.0.0" /> 
 <plugin id="org.example.plugin" version="0.0.0" unpack="false"/>
</feature>

<feature id="org.example.patch" label="Patch" version="1.0.0">
 <requires>
    <import feature="org.eclipse.equinox.p2.user.ui" version="1.1.0.v20090605-1440-7u6Fb3FbPbJP5MiKiZgpdl" patch="true"/>
 </requires>
 <plugin id="org.eclipse.equinox.p2.touchpoint.eclipse" version="1.0.101.v20090611" unpack="false"/>
</feature>

If we have the bundle containing the fix in our target platform, we can now export a new version of our example feature that includes it. And we can update the platform where our feature is installed to get the new version that brings the patch with it.

The metadata for the Patch installable unit looks like this:

<unit id='org.example.patch.feature.group' version='1.0.0' singleton='false'>
 <patchScope>
   <scope>
     <requires size='1'>
       <required namespace='org.eclipse.equinox.p2.iu' name='org.eclipse.equinox.p2.user.ui.feature.group' range='[1.1.0.v20090605-1440-7u6Fb3FbPbJP5MiKiZgpdl,1.1.0.v20090605-1440-7u6Fb3FbPbJP5MiKiZgpdl]'/>
     </requires>
   </scope>
 </patchScope>
 <changes>
   <change>
     <from>
       <required namespace='org.eclipse.equinox.p2.iu' name='org.eclipse.equinox.p2.touchpoint.eclipse' range='0.0.0'/>
     </from>
     <to>
       <required namespace='org.eclipse.equinox.p2.iu' name='org.eclipse.equinox.p2.touchpoint.eclipse' range='[1.0.101.v20090611,1.0.101.v20090611]'/>
     </to>
   </change>
 </changes>
...
<requires size='2'>
   <required namespace='org.eclipse.equinox.p2.iu' name='org.eclipse.equinox.p2.user.ui.feature.group' range='[1.1.0.v20090605-1440-7u6Fb3FbPbJP5MiKiZgpdl,1.1.0.v20090605-1440-7u6Fb3FbPbJP5MiKiZgpdl]' greedy='false'/>
...  

This says, in org.eclipse.equinox.p2.user.ui.feature.group version 1.1.0.v20090605-1440-7u6Fb3FbPbJP5MiKiZgpdl, change whatever version of org.eclipse.equinox.p2.touchpoint.eclipse is there to version 1.0.101.v20090611.

It works! Except...

It works, except that we notice that the patch IU has hard requirements on the version of the p2.user.ui feature. This would prevent upgrading the underlying platform because the feature patch has locked down the version of the p2.user.ui feature.

p2 is much more flexible about patches than the old Update manager was, so there are a few different ways to address this, and I will write another post covering this soon.

(Right now its Friday after 5pm, and time to go home for the weekend :) )

Building p2 RCP products in Eclipse 3.5M6

We can also call this "Building for the Cloud" just to take advantage of the newest buzz word.

Susan Franklin McCourt put together a nice wiki page covering how to add p2 self-updating support to your RCP application. One of those examples was called "Updating from the Cloud".

As part of preparing for EclipseCon, I put together a simple releng product build that builds this cloud product for p2. The cloud bundle "org.eclipse.equinox.p2.examples.rcp.cloud" is available in CVS together with a releng project that I created.

[Edit 2013/07/03: Eclipse CVS has migrated to git starting in 2010.  The examples are now available under the examples folder in http://git.eclipse.org/gitroot/pde/eclipse.pde.build.git. The example has not been updated to work with git and has an obsolete CVS entry in the .map file.]

To run this example, check both projects out into your workspace, and read the rcp.cloud.releng/readme.txt.

Builder Setup

The releng project serves at the builder for a headless build. The build.properties file is a copy of the template provided by PDE/Build with some things we don't need removed. We modified the following properties:

• product : This defines the .product file which specifies how to create the cloud product. The first segment of this path is the plug-in id for the cloud project that contains the file.
• configs : Here we define the platforms we are building for. This is currently set to build windows, other platforms can be added. (The default here is *,*,* which is platform independent pieces, which doesn't make sense for a product).
• J2SE-1.5 : The cloud project has a Bundle-RequiredExecutionEnvironment of 1.5. We set this property to be the bootclasspath to use for compiling 1.5. See the eclipse help for more details on these properties. We also set CDC-1.1/Foundation-1.1 because we had the org.eclipse.osgi bundle in our workspace (and so were compiling it for the product) and osgi requires that EE defined in order to compile properly.
• p2.gathering : We set this to true to use the new support for publishing metadata directly from source. This will also automatically install the product using the p2 director. More details on this below.
• p2.metadata.repo, p2.artifact.repo : In addition to the installed product, we also want a p2 repository.

We also provide a simple ant script named "buildProduct.xml". This is a very simple script that does the following:

1. Set the baseLocation. This is the location of your target binaries against which you want to compile. In the script we set it to ${eclipse.home} which is a property that is automatically set to be the eclipse that is running the build.
2. Find the delta pack. The delta pack is required for headless product builds because it contains all the platform specific fragments, and org.eclipse.equinox.executable feature which contains launchers for all the platforms. Download the 3.5M6 deltapack from here. The deltapack location will be added to the pluginPath property.
3. builder : the location of the build configuration files (ie: build.properties). We set it to be the directory containing the buildProduct.xml script.
4. buildDirectory: the directory where everything happens, set it to a subfolder of our builder.
5. pluginPath: the location to find more plugins and features. Here we add the workspace (one level up from the builder), and the deltapack.
6. Run the build. The property ${eclipse.pdebuild.scripts} is automatically set to the location of the PDE/Build scripts directory. Here we call the product build.
7. Victory! : copy the resulting archive into the root of the releng product.

The results of this build is a fully provisioned, p2 enabled product with an accompanying repository.

p2.gathering

In 3.4, p2 metadata was generated by running the metadata generator on the binary jars that were the result of the build. At the end of the build you were left with a repository and you need to perform a director install for get a fully p2-enabled product.

In 3.5 we have this new property p2.gathering. Setting this to true does a couple of things:

• Generate metadata from source. PDE/Build extends the p2 publisher to publish metadata and artifacts directly from source (and the compiled .class files) into a repository. There are no intermediate steps. (This also has the side effect of increased performance).
• Feature Builds result in a repository that is a group of all the platforms that were built. (This is an implicit groupConfigurations=true).
• Product Builds result in a fully installed p2 enabled product. This removes the need for a manual director call after the build is complete. The product build can also create a repository containing the IUs required to install the product.

The 3.4 properties still work for doing the old style generation.

Example Headless build for a RCP product with p2

*UPDATE* For best results, I would suggest using the upcoming 3.4.1 release. Candidate builds are available from the download page under "3.4.1 Stream Builds", RC3 is available now.

I have prepared an example build setup that builds the RCP Mail Template and produces a p2-ized product. All the files needed for this are provided HERE.

The first step is to set up Eclipse with the RCP Delta pack which is required for the headless build of RCP products:

1. Get the Eclipse SDK and the RCP Delta pack and unzip into the same location on disk.
2. Start Eclipse and go to Window -> Preferences -> Plug-in Development -> Target Platform.
3. Uncheck the option "Build target platform based on the target's installed plug-ins."

Create a new plug-in "com.acme.rcp" based on the RCP Mail Template. Create a new Product Configuration file "acme.product" for this plug-in. On the overview tab of the product editor, I set the id "com.acme.rcp.product" and the application "com.acme.rcp.application". I also set the version to be "1.0.0.qualifier".

The p2 Installable Unit (IU) that will be created for this product will take its id and version from the product id and version set here. I am setting the version to end in "qualifier" so that I can replace the qualifier at build time with the build id.

The product configuration for this example is based on plug-ins. On the configuration tab remove all plug-ins, add the plug-ins listed below and click Add Required Plug-ins.

• com.acme.rcp
• org.eclipse.equinox.p2.exemplarysetup
• org.eclipse.equinox.p2.ui.sdk
• org.eclipse.equinox.p2.touchpoint.eclipse
• org.eclipse.equinox.p2.touchpoint.natives
• org.eclipse.ecf.filetransfer
• org.eclipse.ecf.provider.filetransfer
  

The product can now be run from overview tab of the product editor. If you do so you will notice the Help -> Software Updates... menu item. Selecting Software Updates at this time gets you a "This installation has not been configured properly for Software Updates" message.

Setting up the headless build

I like to run the headless build from inside Eclipse using a launch configuration. To do this, first import (Import... -> Plug-in Development -> Plug-ins and Fragments) "org.eclipse.pde.build" from the target platform into your workspace as a binary project.

Next, create a new general project named "Builder" and copy

org.eclipse.pde.build/templates/headless-build/build.properties"
                                  /customTargets.xml"

into the builder project.

I want my headless build to do 3 things differently from a normal build:

1. Set the version number in the product file to match the build id.
2. Generate p2 metadata
3. Use the generated metadata to perform p2 installs to get the final archives

1: Set the version number in the product file.

See the Build/customTargets.xml/preSetup target. We copy the "acme.product" file out of the com.acme.rcp bundle and into our buildDirectory. We then use the ant replace task to replace the version qualifier based on the time stamp. This uses a "acmePlugin" property which we will define separately. We also save the time stamp to a properties file that we read later when calling the p2 director.

2: Generate p2 metadata

PDE/Build provides integration with p2 to automatically generate p2 metadata. See Builder/build.properties where the following properties have been set:

    generate.p2.metadata = true
p2.metadata.repo=file:${buildDirectory}/repo
p2.artifact.repo=file:${buildDirectory}/repo
p2.flavor=tooling
p2.publish.artifacts=true

3: Perform p2 installs and archive the results

We do this in the Builder/customTargets.xml/postBuild target. We exec a new instance of eclipse to run the director application to install the product into a temporary directory and then we zip up the results.

We are running the same eclipse install again to do this. This means that any repositories that the eclipse install is set up with will also be considered during the director call (in addition to the repositories we pass on the command line).

The rest: build.properties

There are a few other properties to set in the build.properties file. We also set a number of properties on the command line in the launch configuration.

In the build.properties we set the product property to be the one that has the modified version qualifier. We also set the configuration we are building for.

 product=${buildDirectory}/acme.product
configs=win32,win32,x86

On the command line we are setting:

• pluginPath: allows us to not copy the com.acme.rcp plugin into the buildDirectory.
• buildDirectory: where all scripts will be generated and the location of our modified acme.product file.
• baseLocation: We use the eclipse install we are running as our base.
• acmePlugin: Define the location of the acme plugin for our custom task.

All these properties are set in the provided "Build Acme Product.launch" launch configuration and values are set at launch time using variables.

Results

Go to the Run Configurations... dialog and run the "Build Acme Product" launch config. The resulting zip should be under Builder/result. The project may need to be refreshed to see the results.

Running the build multiple times will result in the repository containing multiple versions of the product. You can then use that repository to update the product from one version to the next.

Problems Encountered

I did encounter some problems while creating this example.

• Bug 237662: The metadata generator currently does not properly handle the new p2 simpleconfigurator style of config.ini and loses some start level information. This causes the installed product to not start. Workaround for this example is to not include org.eclipse.equinox.simpleconfigurator in the product. I will provide another example later to work around this problem.


• Bug 222969: When including the org.eclipse.equinox.simpleconfigurator bundle in the product, the install is getting a bad relative path in the config.ini. Since we are excluding the simpleconfigurator because of the previous bug, this doesn't affect our example.


• Bug 237647: NullPointerException in the director. This only occured once, workaround was simply to remove the bad profile directory under my eclipse install (eclipse/p2/org.eclipse.equinox.p2.engine/profileRegistry/ACMEProfile.profile)

Doc Time (with a little help from p2)

It is that time of year again: documentation time. Irrespective of any personal feeling towards writing documentation, I do realize that it is quite important for something like PDE/Build.

I will start with the eclipse help itself and then perhaps write some blog posts with small examples and how-tos. I may even take a look at the venerable Build and Test Automation article. It was written for 3.0, now that we are hitting 3.4 I'm sure it could use an update.

And a little p2 plug:
I used to use mozilla to write the html help docs, but I decided I'd like to stay within Eclipse. I'm sure there has to be a decent html editor somewhere in Ganymede.

So, to the Software Updates dialog, I add the Ganymede update site (http://download.eclipse.org/releases/ganymede/). "Web and Java EE Development -> WTP Web Page Editor" looks promising. I click the checky box and install. p2 downloaded a bunch of stuff, and after a restart, voila! : A webpage editor, easy as pie.