Example Headless build for a RCP product with p2

*UPDATE* For best results, I would suggest using the upcoming 3.4.1 release. Candidate builds are available from the download page under "3.4.1 Stream Builds", RC3 is available now.

I have prepared an example build setup that builds the RCP Mail Template and produces a p2-ized product. All the files needed for this are provided HERE.

The first step is to set up Eclipse with the RCP Delta pack which is required for the headless build of RCP products:

1. Get the Eclipse SDK and the RCP Delta pack and unzip into the same location on disk.
2. Start Eclipse and go to Window -> Preferences -> Plug-in Development -> Target Platform.
3. Uncheck the option "Build target platform based on the target's installed plug-ins."

Create a new plug-in "com.acme.rcp" based on the RCP Mail Template. Create a new Product Configuration file "acme.product" for this plug-in. On the overview tab of the product editor, I set the id "com.acme.rcp.product" and the application "com.acme.rcp.application". I also set the version to be "1.0.0.qualifier".

The p2 Installable Unit (IU) that will be created for this product will take its id and version from the product id and version set here. I am setting the version to end in "qualifier" so that I can replace the qualifier at build time with the build id.

The product configuration for this example is based on plug-ins. On the configuration tab remove all plug-ins, add the plug-ins listed below and click Add Required Plug-ins.

• com.acme.rcp
• org.eclipse.equinox.p2.exemplarysetup
• org.eclipse.equinox.p2.ui.sdk
• org.eclipse.equinox.p2.touchpoint.eclipse
• org.eclipse.equinox.p2.touchpoint.natives
• org.eclipse.ecf.filetransfer
• org.eclipse.ecf.provider.filetransfer
  

The product can now be run from overview tab of the product editor. If you do so you will notice the Help -> Software Updates... menu item. Selecting Software Updates at this time gets you a "This installation has not been configured properly for Software Updates" message.

Setting up the headless build

I like to run the headless build from inside Eclipse using a launch configuration. To do this, first import (Import... -> Plug-in Development -> Plug-ins and Fragments) "org.eclipse.pde.build" from the target platform into your workspace as a binary project.

Next, create a new general project named "Builder" and copy

org.eclipse.pde.build/templates/headless-build/build.properties"
                                  /customTargets.xml"

into the builder project.

I want my headless build to do 3 things differently from a normal build:

1. Set the version number in the product file to match the build id.
2. Generate p2 metadata
3. Use the generated metadata to perform p2 installs to get the final archives

1: Set the version number in the product file.

See the Build/customTargets.xml/preSetup target. We copy the "acme.product" file out of the com.acme.rcp bundle and into our buildDirectory. We then use the ant replace task to replace the version qualifier based on the time stamp. This uses a "acmePlugin" property which we will define separately. We also save the time stamp to a properties file that we read later when calling the p2 director.

2: Generate p2 metadata

PDE/Build provides integration with p2 to automatically generate p2 metadata. See Builder/build.properties where the following properties have been set:

    generate.p2.metadata = true
p2.metadata.repo=file:${buildDirectory}/repo
p2.artifact.repo=file:${buildDirectory}/repo
p2.flavor=tooling
p2.publish.artifacts=true

3: Perform p2 installs and archive the results

We do this in the Builder/customTargets.xml/postBuild target. We exec a new instance of eclipse to run the director application to install the product into a temporary directory and then we zip up the results.

We are running the same eclipse install again to do this. This means that any repositories that the eclipse install is set up with will also be considered during the director call (in addition to the repositories we pass on the command line).

The rest: build.properties

There are a few other properties to set in the build.properties file. We also set a number of properties on the command line in the launch configuration.

In the build.properties we set the product property to be the one that has the modified version qualifier. We also set the configuration we are building for.

 product=${buildDirectory}/acme.product
configs=win32,win32,x86

On the command line we are setting:

• pluginPath: allows us to not copy the com.acme.rcp plugin into the buildDirectory.
• buildDirectory: where all scripts will be generated and the location of our modified acme.product file.
• baseLocation: We use the eclipse install we are running as our base.
• acmePlugin: Define the location of the acme plugin for our custom task.

All these properties are set in the provided "Build Acme Product.launch" launch configuration and values are set at launch time using variables.

Results

Go to the Run Configurations... dialog and run the "Build Acme Product" launch config. The resulting zip should be under Builder/result. The project may need to be refreshed to see the results.

Running the build multiple times will result in the repository containing multiple versions of the product. You can then use that repository to update the product from one version to the next.

Problems Encountered

I did encounter some problems while creating this example.

• Bug 237662: The metadata generator currently does not properly handle the new p2 simpleconfigurator style of config.ini and loses some start level information. This causes the installed product to not start. Workaround for this example is to not include org.eclipse.equinox.simpleconfigurator in the product. I will provide another example later to work around this problem.


• Bug 222969: When including the org.eclipse.equinox.simpleconfigurator bundle in the product, the install is getting a bad relative path in the config.ini. Since we are excluding the simpleconfigurator because of the previous bug, this doesn't affect our example.


• Bug 237647: NullPointerException in the director. This only occured once, workaround was simply to remove the bad profile directory under my eclipse install (eclipse/p2/org.eclipse.equinox.p2.engine/profileRegistry/ACMEProfile.profile)

Doc Time (with a little help from p2)

It is that time of year again: documentation time. Irrespective of any personal feeling towards writing documentation, I do realize that it is quite important for something like PDE/Build.

I will start with the eclipse help itself and then perhaps write some blog posts with small examples and how-tos. I may even take a look at the venerable Build and Test Automation article. It was written for 3.0, now that we are hitting 3.4 I'm sure it could use an update.

And a little p2 plug:
I used to use mozilla to write the html help docs, but I decided I'd like to stay within Eclipse. I'm sure there has to be a decent html editor somewhere in Ganymede.

So, to the Software Updates dialog, I add the Ganymede update site (http://download.eclipse.org/releases/ganymede/). "Web and Java EE Development -> WTP Web Page Editor" looks promising. I click the checky box and install. p2 downloaded a bunch of stuff, and after a restart, voila! : A webpage editor, easy as pie.

Examples of using PDE/Build

It can be very useful to have an example when setting up your first headless build. It occurs to me that there is a growing number of relatively simple examples: the pde.build junit tests.

The org.eclipse.pde.build.tests project can be checked out of cvs from dev.eclipse.org/cvsroot/eclipse/org.eclipse.pde.build.tests. The main test suite is PDEBuildTestSuite.

If you run the tests with the vm argument pde.build.noCleanup=true then the tests will run without deleting any files and you can then go take a look at the setup for each test.

Not every test is a full build, but many are. Generally the tests consist of generating some features and plugins, then generating the build.properties and allElements.xml configuration files and then running the build.

Of particular interest would be the generation of the build.properties file. This is done by loading the template file and then setting just a handful of the properties. See BuildConfiguration.getBuilderProperties(IFolder). This should give a good idea of what properties you need to change for your build.

*Edit 1/28/08 : The "pde.build.noCleanup" argument was added on 1/23/08, you must use a build N20080124-0010 or newer. The first I-Build containing these changes will be I20080129.

On Building Cycles

One of the enhancements I am considering for pde.build in 3.4 is partial handling of cycles.

The latest idea is allowing a cycle that contains at least one binary bundle. The presence of the binary bundle could be enough to break the cycle and allow us to compile the remaining bundles.

Consider the following:

  A <- B <- C <- D <- E
       ----------^

Where we have a cycle between B, C, and D. If C is binary and does not need to be compiled, we may be able to compile the remaining bundles in the order A, D, B, E.

However, there are cases where this would not work. It could be that allowing this wouldn't actually help and would simply be giving people rope to hang themselves with.

If you have an opinion on this, please go comment on the bug and give the patches there a try.

Selecting Plug-in Versions in a Headless Product Build

As Pascal likes to say: "PDE.Build is an onion." Don't be afraid to reach in and savor those inner layers. Here is an example:

We support building from a .product file. Generally, people base their product files on a list of plug-ins. There is currently no way to specify plug-in versions in the product file. If you have a requirement on a specific version of some bundle and for some reason there are multiple versions in your target, then you may be stuck with pde.build choosing the wrong version for you.

In a product build, a container feature is generated based on the contents of the .product file and the build is run using this generated feature. Features can specify the versions of the plug-ins they want to include, or they can specify "0.0.0" which means any version. They can also specify a version like "3.2.1.qualifier" which means any version starting with "3.2.1". (Though beware of this bug). The generated feature simply specifies "0.0.0" for each plug-in from the .product file.

PDE.Build provides an eclipse.idReplacer ant task. It uses this task to replace versions in the feature.xml with the actual versions that are built. It is easy to imagine using this task to modify the generated feature.xml to specify the plug-in versions you want before fetching or generating scripts.

In your customTarges.xml preFetch (or preGenerate) task, do something like this:

 <eclipse.idReplacer
    featureFilePath="${buildDirectory}/features/org.eclipse.pde.build.container.feature/feature.xml"
    selfVersion="1.0.0"
    featureIds=""
    pluginIds="org.eclipse.foo:0.0.0,1.3.0.qualifier,org.bar:0.0.0,3.2.0.qualifier," />

Beware of this bug in 3.3.0, and this bug in 3.3.1.

Eclipse launcher is losing its head

The Eclipse launcher has always required a graphics library (ie GTK, Motif) to show error messages and display the splash screen. It can't even start if the library is not found. This is a problem for people who have headless Eclipse applications that run in places without such luxuries.

I raised a bug to track this issue. Though I should point out that bug is incorrect in its description of the launcher's linking: the launcher is dynamically linked to the graphics libraries, (not statically linked as the bug says). The problem is that these dependencies are resolved at loadtime, not runtime. The fix is to load the graphics libraries at runtime using dlopen and lookup all the functions we need to call using dlsym.

I have created launchers for GTK and Motif that do this and attached them in bugzilla. These changes require more testing before I release them to CVS. People should try them out and give feedback on the bug.


One thing to note is that the Motif version is using a DT_RPATH tag to help find the libXm.so.2 library that ships on linux in the root of Eclipse. Most of what I've read suggests this is obsolete and not a good idea, so that will probably change.

PDE Build Slides from EclipseCon 2007

The slides we (Sonia Dimitrov, Pascal Rapicault, and myself) presented at EclipseCon 2007 are now posted online. We're only about 6 months late putting them up.

They are attached to the Eclipsezilla entry for the tutorial. If you want to run the examples, you will need the Eclipse SDK and the RCP delta-pack. The examples were tested against an old integration build which is no longer available, but using 3.3 should work just fine.

Greasemonkey: Making the internet slightly less annoying

One of the more annoying memes I've seen on some blogs is the use of "Canadia" instead of "Canada". The level of annoyance has finally risen high enough to overcome a certain level of laziness on my part. So I downloaded Greasemonkey and wrote my first JavaScript.

I'm sure it is not the best script ever written, but I now feel like I've done my part to make the internet a better place.

// ==UserScript==
// @name          Canadia
// @description   Change "Canadia" to "Canada"
// @include       *
// ==/UserScript==

(function() {
var expression, replacement, regex, textnodes, node, s;

expression  = "([Cc]anad)ia([^n]|$)";
replacement = "$1a$2";

regex = new RegExp(expression, 'g');

textnodes = document.evaluate( "//body//text()", document, null,
             XPathResult.UNORDERED_NODE_SNAPSHOT_TYPE, null);

for (var i = 0; i < textnodes.snapshotLength; i++) {
 node = textnodes.snapshotItem(i);
 s = node.data;
 s = s.replace(regex, replacement);
 node.data = s;
}
})();

Eclipse and the Console

Starting in the warmup to 3.3M7 there is a second launcher on Windows: eclipsec.exe.

The Eclipse launchers in 3.3 have had issues with various aspects of the console on Windows. See some bugs: 146807, 167310, 168726, and 173962.

All of our problems came from the fact that the eclipse.exe was linked as a UI application and not a console application. In 3.2, the console behaviour was tied to using java.exe instead of javaw.exe, but starting in 3.3 M5, the vm is loaded in the eclipse process using the JNI Invocation API.

So just like javaw/java, we now have eclipse/eclipsec. Eclipsec.exe is linked as a console application so that the OS handles allocating the console for us. Using eclipsec.exe does not imply any default arguments, so you still need to specify things like -console and -consoleLog.

Equinox launcher & Swing

The equinox launcher starts the Java VM using the JNI invocation API. Because we are such good friends with SWT (and also because it is easier) the VM was always started on the first thread. This turns out not to be so good when you want to run Swing on the Mac. There was a conversation about it over on EclipseZone.

So I fixed the bug (181698) today by adding a new option to the launcher "--launcher.secondThread". Use it if you want swing to work. This option can be compared to the -XstartOnFirstThread for the java launcher.

Source Feature generation with PDE/Build

One of the cool features in PDE/Build is the generation of source features and plug-ins. This is perhaps one of those features that leads towards the "black magic" reputation that we seem to have.

Recently the CDT SDK builds have been missing their source. The source features and plug-ins exist, but they just didn't contain any src.zips. After some investigation, it seems that a long standing bug in pde.build has risen to the surface. It has always been there, it was just hidden by bug 114150 which was fixed in M6.

The workaround for the CDT is simple: just make sure that in the SDK feature, the generated org.eclipse.cdt.source feature is included after the org.eclipse.cdt feature upon which it is based.

Unfortunately, I'm sure that the fix in pde.build will be a bit harder.

EclipseCon 2007

As everyone knows, EclipseCon 2007 is March 5-8. I will be presenting a long tutorial on PDE Build with Pascal Rapicault and Sonia Dimitrov. I will also be giving a short talk on the new launcher story.

I even got a pretty little graffic:

Feature Version Suffixes in PDE/Build

This week I released a patch from David Olsen which is a significant change to how pde/build calculates feature version suffixes. The details are in bug 157049, as well, the problems outlined in bug 162020 are fixed as a consequence.

The biggest thing to note is that the major, minor and service segments are now included in the calcuation. This covers the case when a plugin decrements its qualifier and increments, for example, the service number.

These changes will be in 3.3 M5. There may be cases where these changes could cause a one-time decrease in the version suffix, so builders picking up these changes are advised to increment their feature qualifiers.

Equinox Launchers and SWT splash screens

The Eclipse Integration build I20061219-1300 has the new Equinox launchers in it. Read all about them on the wiki, and in my email to the Eclipse-dev mailing list.

This will allow for some potentially cool things to happen in the splash screen.

I have been slowly going crazy getting this stuff working on all the different platforms. There are still problems on linux.x86_64, linux.ppc, and aix.ppc.

Everybody's doing it

All the cool people are doing it, and if I'm ever going to be famous I suppose I should start too.

So this is my blog, I wonder if anybody will actually read it.