Versioning & p2, slides from EclipseCon

This year at EclipseCon, I presented a 10 minute talk on the importance of versioning with p2. I have posted the slides here. Because this was only a 10 minute talk, there is not a lot of content in the slides, so this post is an attempt at an overview.

id + version == 1 set of bytes

Hopefully by now everyone has heard this before. In the world of p2, an id and a version represent a particular set of bytes. If multiple copies of a given artifact exist in multiple places, and they all have the same id and version, they are assumed to be the same (or equivalent) bytes.

In particular, this means that if your user already has a bundle org.foo_1.0.0 on his machine, and you are trying to deliver an update, then the user will not download your updated org.foo if the version number is still 1.0.0.

So make sure you increase your version numbers, it is hard to support a customer when you can't tell if his org.foo_1.0.0 is the broken version or the fixed version!

We tend to version sources

The process we follow in Eclipse tends to version sources. We tag our source code with the version qualifier which determines the version of the resulting binary. The idea is this leads to reproducible builds.

However, we must realize that there is more than just the source code that affects what the resulting binary looks like. These are things like what compiler was used, the build scripts, and most importantly the dependencies that were on our build-time classpath.

If some bundle B depends on bundle A and A changes in some way, then recompiling B can potentially result in different byte-code even though the source code did not change. In this case, B deserves to have its version number increased.

Mirroring with a baseline and comparator

This scenario of a bundle changing without having its version incremented has happened more than once in the Eclipse SDK releng builds. In order to detect this, we use a previous build as a baseline in a mirror operation. If the baseline already contains a bundle with the same version as the one that was just built, then we mirror the old bundle and discard the new one.

We install our product using the mirrored results which ensures that our new install contains the same old bundles that already exist out on user's machines. The unit tests run against these old bundles.

In order to detect when a bundle actually changes and needs its version number increased, the mirror operation supports a comparator. Olivier kindly contributed some code to p2 that can do a semantic compare on java class files to see if they are equivalent. Similarly, things like manifests and properties files can be compared for semantic instead of bitwise equivalence.

Example Build

I put together an example feature build that uses a comparator to detect if a bundle has changed when the version hasn't. The projects are in CVS under dev.eclipse.org:/cvsroot/eclipse/pde-build-home/examples/comparator. There are 4 projects to check out:

• example.Builder - the builder project, contains a runBuild.xml
• org.example.a - a project that we will change between builds
• org.example.b - depends on a, won't change source between builds
• org.example.feature - the feature to build

To run the example, just run runBuild.xml script as an Ant Build using the same JRE as the workspace. The build will mirror the results into example.Builder/composite/I<timestamp>. You can run the build multiple times, and each time the results will be added to the composite repository.

The mirror call is done in example.Builder/customTargets.xml/postBuild. It looks like this:

<target name="postBuild">
 <antcall target="gatherLogs" />

 <!-- mirror from build results, comparing against previous builds that are in the composite repo -->
 <p2.mirror>
     <source location="file:${assemblyTempDir}/${buildLabel}"/>
     <destination location="file:${builder}/composite/${buildLabel}"/>
     <comparator comparator="org.eclipse.equinox.p2.repository.tools.jar.comparator" comparatorLog="${buildDirectory}/comparator.log">
         <repository location="file:${builder}/composite" />
     </comparator>
 </p2.mirror>

 <!-- add the new build to the composite -->
 <p2.composite.repository destination="file:${builder}/composite">
     <add>
         <repository location="file:${builder}/composite/${buildLabel}"/>
     </add>
 </p2.composite.repository>
</target>

Each build uses the previous contents of the composite repository as a baseline, and then adds itself to the composite.

To see the comparator in action, edit org.example.a.Sub and uncomment the doSomething method:

public void doSomething(List o) {
 for (Iterator iterator = o.iterator(); iterator.hasNext();) {
     super.doSomething(iterator.next());
 }
}

This change to the source code of org.example.a will change which doSomething B ends up calling:

public class B {
 public void method() {
     ArrayList list = new ArrayList();
     list.add(this);
 
     Sub d = new Sub();
     d.doSomething(list);
 }
}

That is, recompiling B against the new A results in different byte-code for B. If we run the build again with the changed source, then it will now fail with an error message:

[p2.mirror] Messages while mirroring artifact descriptors.
[p2.mirror] Compare and download of canonical: osgi.bundle,org.example.b,1.0.0 from baseline.
[p2.mirror] Difference found for org/example/b/B.class within [canonical: osgi.bundle,org.example.b,1.0.0]
                     from file:/C:/workspace/example.Builder/composite/I20090806060019/

When the build fails in this manner, it means we need to increment the version for org.example.b

Build Notes

1. The builder's build.properties sets pluginPath and elementPath to enable the build to use the sources in the workspace without copying them to the buildDirectory. elementPath points to the top level feature that we are building.
2. The build is using p2.gathering=true, for feature builds, this groups the configurations, we set archivesFormat to leave the results as a folder.
3. runBuild.xml automatically refreshes the workspace when it finishes, however when the build fails after changing the source in org.example.a, this refresh doesn't happen and you need to refresh the workspace manually to see the new results.
4. In customTargets.xml/preGenerate we create an empty composite repository if there wasn't already one there.
5. We changed the default customTargets.xml/gatherLogs to use elementPath since the top level feature is not under buildDirectory.
6. org.example.a has version 1.0.0.qualifier where the qualifier gets replaced each build with the timestamp. org.example.b just has version 1.0.0. In a releng build, the CVS tag from a map file would be used instead of the timestamp.