Thursday, June 25, 2009

Using the deltapack in Eclipse 3.5

Eclipse Galileo was released yesterday and I have already seen a couple of questions on the newsgroups regarding how to install the delta pack.

What's a delta pack?

For those who don't know what it is, the delta pack is a zip file provided by the Eclipse Platform and it is used for developing RCP applications for multiple platforms. The delta pack archive contains all the platform specific fragments from the Eclipse SDK. It also includes a feature "org.eclipse.equinox.executable" which contains binary launchers for all the platforms. The archive is available for download through the Eclipse Classic "Other downloads" page.

The delta pack is required if you want to do headless product builds, or to export products from the UI:



If you don't have the delta pack installed, then the option to export your product for multiple platforms will not appear in the wizard.

How to install the delta pack

Ian Bull blogged about the improved Target Platform management system as #3 in his Galileo Feature top ten list. This is the recommended method of installing the delta pack.
  1. Extract the delta pack archive into its own directory on disk.
  2. Open the Target Platform preferences (Window -> Preferences -> Plug-in Development -> Target Platform).
  3. Add a new target, or edit the active target.
  4. Add the directory where you extracted the delta pack.



Monday, June 15, 2009

Patching Features (Part 2)

In my last post I showed how to create a simple feature patch. However, this patch leaves us in a state where we can no longer upgrade the platform because the patch itself requires a specific version of the feature it is patching.

Making the patch Optional


If we make the inclusion of the patch in our org.example.feature optional, then we no longer block upgrading the platform underneath us.


When a new version of the platform is available, there is a conflict between the version of the p2.user.ui feature required by the patch and the one required by the new platform. By making the patch optional, p2 will not install the patch if there is a conflict.

This is fine if the new version of the platform includes the fix we want. But what if it doesn't? What if we need to have our patch apply to several different versions of the platform?

Relaxing version ranges

Unfortunately, currently the only way to do the following is to edit the metadata by hand. (Paul Webster uses XSLT transforms, I have also raised a bug to allow changing things using the p2.inf file.)

Looking at the metadata for the patch, we see a few interesting sections:

The Patch Scope


The scope section of the patch's metadata specifies which feature(s) the patch applies to:

<patchScope>
<scope>
<requires size='1'>
<required namespace='org.eclipse.equinox.p2.iu'
name='org.eclipse.equinox.p2.user.ui.feature.group'
range='[1.1.0.v20090605-1440-7u6Fb3FbPbJP5MiKiZgpdl,1.1.0.v20090605-1440-7u6Fb3FbPbJP5MiKiZgpdl]'/>
</requires>
</scope>
</patchScope>


Here we see the patch applies to a specific version of the org.eclipse.equinox.p2.user.ui feature. By relaxing the version range here we can make the patch apply to other versions of the platform. For example, say we knew our bug was fixed on June 11, but there were several version between June 5th and the 10th that still contained the bug. We could instead use a version range like [1.1.0.v20090605-1440,1.1.0.v20090611) (the upper end of the range is open, it includes everything up to but not including v20090611).

Note that the patch Installable Unit also has normal requirements on the feature it is patching. When widening the scope range, the regular requirements should also be widened to match.

Change From / Change To

The 'changes' section of the patch's metadata specifies which plug-in we are patching, and the version to change it to.
<changes>
<change>
<from>
<required namespace='org.eclipse.equinox.p2.iu'
name='org.eclipse.equinox.p2.touchpoint.eclipse'
range='0.0.0'/>
</from>
<to>
<required namespace='org.eclipse.equinox.p2.iu'
name='org.eclipse.equinox.p2.touchpoint.eclipse'
range='[1.0.101.v20090611,1.0.101.v20090611]'/>
</to>
</change>
</changes>

In the above example, we are changing the org.eclipse.equinox.p2.touchpoint.eclipse bundle from any version ("0.0.0") to version 1.0.101.v20090611.
We could imagine changing these ranges in the following ways:
  • Change the "scope" range to "0.0.0" to match any version of the p2.user.ui feature, and then change the "change from" range to [1.0.101, 1.0.101.v20090611). This would result in patching any version of the p2.user.ui feature that includes any 1.0.101 versioned bundle up to v20090611 (which is the version with the proper fix).
  • Change the "change to" range to something like [1.0.101.v20090611, 1.0.102). We know v20090611 contains the fix we want, but the in the future there will be more bug fixes before the maintenance stream is released. By widening the range here, we allow future bug fixes to also be included in our patch.

Friday, June 12, 2009

Patching features with p2

The Galileo release of Eclipse is in its final days. The eclipse SDK itself is effectively done and the current build is under consideration to be the final release.

Imagine This...

Say I have a feature that I build, it runs on top of the Eclipse Platform. I know Eclipse Galileo is coming out soon, so I download it and try it out with my feature.

Oh No! It doesn't work! I've found a bug in the platform, and its a major blocker for me. (Oops, maybe I should have tried this a couple of months ago when there still would have been time to fix the bug.)

Meta-Comment: The Eclipse Platform Project has a reputation for shipping on time (even if that means there are unresolved bugs). There is a strict end game plan that is followed for the release, lock down started back at the beginning of May. The further we are along in the plan, the harder it is to get a fix approved for release. A lot of people don't seem to relealize this happens, and perhaps wonder why their important bugs are defered with the comment "Its too late".

Ok, I need to patch the platform. How do I do this?

The Example Feature

To work through the steps of patching the platform, I created my own example feature:

  1. Create a New Plug-in Project "org.example.plugin". The plug-in is not a "Rich client Application". On the templates page of the wizard I chose "Plug-in with a view".
  2. Create a New Feature "org.example.feature". Add org.example.plugin to the feature.
  3. Export the feature, and on the Options tab in the export wizard check "Generate metadata repository".
For the purposes of the example, I'm starting from the Eclipse Platform Runtime 3.5RC4. Because I want to strictly control the upgrade path in the example, I removed the Galileo and Eclipse Project repositories from the available sites (Window -> Preferences -> Install/Update->Available Software Sites).

Unzip the platform, and installed the feature from the exported repository. I didn't create any categories for the feature, so "Group items by category" must be unchecked.


The Patch

For the example, I am inspired by bug 279542 which came in very late and almost was not fixed in 3.5. The bundle org.eclipse.equinox.p2.touchpoint.eclips has the bug and this plugin is included by the feature org.eclipse.equinox.p2.user.ui.

Say we have a fix with touchpoint.eclipse version 1.0.101.v20090611. (When following the eclipse versioning guidelines, the maintenance version of this bundle should be 1.0.101 and the version for the next year's release will probably be 1.0.200).

We can easily create a new Feature Patch with the wizard:
We specify the version of p2.user.ui that we are patching (here the one shipped in 3.5RC4). In the feature patch, include org.eclipse.equinox.p2.touchpoint.eclipse with the specific version containing the fix. Then, add the feature patch as an inclusion in our org.example.feature:

<feature id="org.example.feature" label="Feature" version="1.0.0.qualifier">
<includes id="org.example.patch" version="0.0.0" />
<plugin id="org.example.plugin" version="0.0.0" unpack="false"/>
</feature>

<feature id="org.example.patch" label="Patch" version="1.0.0">
<requires>
<import feature="org.eclipse.equinox.p2.user.ui" version="1.1.0.v20090605-1440-7u6Fb3FbPbJP5MiKiZgpdl" patch="true"/>
</requires>
<plugin id="org.eclipse.equinox.p2.touchpoint.eclipse" version="1.0.101.v20090611" unpack="false"/>
</feature>


If we have the bundle containing the fix in our target platform, we can now export a new version of our example feature that includes it. And we can update the platform where our feature is installed to get the new version that brings the patch with it.

The metadata for the Patch installable unit looks like this:

<unit id='org.example.patch.feature.group' version='1.0.0' singleton='false'>
<patchScope>
<scope>
<requires size='1'>
<required namespace='org.eclipse.equinox.p2.iu' name='org.eclipse.equinox.p2.user.ui.feature.group' range='[1.1.0.v20090605-1440-7u6Fb3FbPbJP5MiKiZgpdl,1.1.0.v20090605-1440-7u6Fb3FbPbJP5MiKiZgpdl]'/>
</requires>
</scope>
</patchScope>
<changes>
<change>
<from>
<required namespace='org.eclipse.equinox.p2.iu' name='org.eclipse.equinox.p2.touchpoint.eclipse' range='0.0.0'/>
</from>
<to>
<required namespace='org.eclipse.equinox.p2.iu' name='org.eclipse.equinox.p2.touchpoint.eclipse' range='[1.0.101.v20090611,1.0.101.v20090611]'/>
</to>
</change>
</changes>
...
<requires size='2'>
<required namespace='org.eclipse.equinox.p2.iu' name='org.eclipse.equinox.p2.user.ui.feature.group' range='[1.1.0.v20090605-1440-7u6Fb3FbPbJP5MiKiZgpdl,1.1.0.v20090605-1440-7u6Fb3FbPbJP5MiKiZgpdl]' greedy='false'/>
...


This says, in org.eclipse.equinox.p2.user.ui.feature.group version 1.1.0.v20090605-1440-7u6Fb3FbPbJP5MiKiZgpdl, change whatever version of org.eclipse.equinox.p2.touchpoint.eclipse is there to version 1.0.101.v20090611.

It works! Except...

It works, except that we notice that the patch IU has hard requirements on the version of the p2.user.ui feature. This would prevent upgrading the underlying platform because the feature patch has locked down the version of the p2.user.ui feature.

p2 is much more flexible about patches than the old Update manager was, so there are a few different ways to address this, and I will write another post covering this soon.

(Right now its Friday after 5pm, and time to go home for the weekend :) )

Thursday, March 26, 2009

Building p2 RCP products in Eclipse 3.5M6

We can also call this "Building for the Cloud" just to take advantage of the newest buzz word.

Susan Franklin McCourt put together a nice wiki page covering how to add p2 self-updating support to your RCP application. One of those examples was called "Updating from the Cloud".

As part of preparing for EclipseCon, I put together a simple releng product build that builds this cloud product for p2. The cloud bundle "org.eclipse.equinox.p2.examples.rcp.cloud" is available in CVS together with a releng project that I created.

[Edit 2013/07/03: Eclipse CVS has migrated to git starting in 2010.  The examples are now available under the examples folder in http://git.eclipse.org/gitroot/pde/eclipse.pde.build.git. The example has not been updated to work with git and has an obsolete CVS entry in the .map file.]

To run this example, check both projects out into your workspace, and read the rcp.cloud.releng/readme.txt.

Builder Setup

The releng project serves at the builder for a headless build. The build.properties file is a copy of the template provided by PDE/Build with some things we don't need removed. We modified the following properties:
  • product : This defines the .product file which specifies how to create the cloud product. The first segment of this path is the plug-in id for the cloud project that contains the file.
  • configs : Here we define the platforms we are building for. This is currently set to build windows, other platforms can be added. (The default here is *,*,* which is platform independent pieces, which doesn't make sense for a product).
  • J2SE-1.5 : The cloud project has a Bundle-RequiredExecutionEnvironment of 1.5. We set this property to be the bootclasspath to use for compiling 1.5. See the eclipse help for more details on these properties. We also set CDC-1.1/Foundation-1.1 because we had the org.eclipse.osgi bundle in our workspace (and so were compiling it for the product) and osgi requires that EE defined in order to compile properly.
  • p2.gathering : We set this to true to use the new support for publishing metadata directly from source. This will also automatically install the product using the p2 director. More details on this below.
  • p2.metadata.repo, p2.artifact.repo : In addition to the installed product, we also want a p2 repository.

We also provide a simple ant script named "buildProduct.xml". This is a very simple script that does the following:
  1. Set the baseLocation. This is the location of your target binaries against which you want to compile. In the script we set it to ${eclipse.home} which is a property that is automatically set to be the eclipse that is running the build.
  2. Find the delta pack. The delta pack is required for headless product builds because it contains all the platform specific fragments, and org.eclipse.equinox.executable feature which contains launchers for all the platforms. Download the 3.5M6 deltapack from here. The deltapack location will be added to the pluginPath property.
  3. builder : the location of the build configuration files (ie: build.properties). We set it to be the directory containing the buildProduct.xml script.
  4. buildDirectory: the directory where everything happens, set it to a subfolder of our builder.
  5. pluginPath: the location to find more plugins and features. Here we add the workspace (one level up from the builder), and the deltapack.
  6. Run the build. The property ${eclipse.pdebuild.scripts} is automatically set to the location of the PDE/Build scripts directory. Here we call the product build.
  7. Victory! : copy the resulting archive into the root of the releng product.
The results of this build is a fully provisioned, p2 enabled product with an accompanying repository.


p2.gathering

In 3.4, p2 metadata was generated by running the metadata generator on the binary jars that were the result of the build. At the end of the build you were left with a repository and you need to perform a director install for get a fully p2-enabled product.

In 3.5 we have this new property p2.gathering. Setting this to true does a couple of things:
  • Generate metadata from source. PDE/Build extends the p2 publisher to publish metadata and artifacts directly from source (and the compiled .class files) into a repository. There are no intermediate steps. (This also has the side effect of increased performance).
  • Feature Builds result in a repository that is a group of all the platforms that were built. (This is an implicit groupConfigurations=true).
  • Product Builds result in a fully installed p2 enabled product. This removes the need for a manual director call after the build is complete. The product build can also create a repository containing the IUs required to install the product.
The 3.4 properties still work for doing the old style generation.

Wednesday, February 25, 2009

Installing and running plugins in Eclipse applications

A common question on the newsgroups is: "I copied new plug-ins into my RCP application but they are getting ignored, what's the problem?". The answer to this is really quite fundamental and is something I think every RCP developer should understand.

My answer here is about the Equinox runtime which is used in Eclipse and is the reference implementation of the OSGi spec.

Installing into the OSGi Runtime

Every plug-in (or bundle, I use the terms interchangeably), must be installed into the OSGi runtime before you can even think about it running. There are two main ways to install a bundle: programmatically through the BundleContext API, or declaratively through the osgi.bundles property in the config.ini file.

The osgi.bundles property is read at startup. It is a list of bundles that are automatically installed and optionally started once the framework is up and running. The format of each entry is
<URL | simple bundle location>[@ [<start-level>] [":start"]]

Starting your bundles

It is not enough to simply have your plug-in installed, it must also be started. When the OSGi framework starts up, it increments through the start levels starting bundles as it goes. A bundle won't be started until its start level is reached, and to get started it must be marked as such, or be marked as lazy start. A lazy start bundle will be started when something tries to load a class from that bundle (and once the start level has been reached for that bundle).

[Edit:] In most cases, marking a bundle as Lazy-Start is sufficient and one need not worry about starting bundles explicitly. See Neil's comment attached to this post.

The default start level for bundles is 4.

Managing installed bundles


There are 2 strategies here, one is to list all of your bundles on the osgi.bundles list, the other is to list only a few bundles that can bootstrap the rest of the application. I will call this kind of bootstrap bundle a "configurator".

If you list all your plug-ins on the osgi.bundles property, then it can be painful to maintain and add new plug-ins to your application. I won't try to make arguments for or against any particular way of managing your system, but it a large way it often comes back to the differences between the old update manager and the new p2 provisioning system.

Update Configurator


In Eclipse 3.3 and earlier, there was the update configurator (org.eclipse.update.configurator). It was installed on the osgi.bundles list and started at level 3. It would scan the plugins directory and automatically install everything it found there.

This is where the magic ability to just copy things into the plugins directory and have them work came from. However, update configurator is essentially forcing the found bundles into the framework with no regard to any conflicts or unresolved dependencies that may result.

Simple Configurator

In Eclipse 3.4, with the advent of p2, update configurator is replace with simple configurator (org.eclipse.equinox.simpleconfigurator). As the name suggests, it is quite simple: it installs everything it finds listed in its configuration file (specified by the org.eclipse.equinox.simpleconfigurator.configUrl property, usually bundles.info).

Editing the bundles.info file to add plug-ins to your product is not really any easier than editing the osgi.bundles list. The difference here is that p2 manages bundles.info for you. In fact, if your product is p2 enabled for updating and installing plug-ins, then any changes made manually are likely to be lost when p2 updates or installs new software.

Dropins

Whatever the other problems update has with respect to avoiding conflicts and ensuring that dependencies are satisfied, it is hard to ignore the simplicity of just copying new bundles into your install. p2 has support for dropins (provided by org.eclipse.equinox.p2.reconciler.dropins).

Like the update configurator, the dropins reconciler will automatically watch a directory and install things it finds there. However, unlike the update configurator, it will ensure that the bundles found will not conflict with others in the system and that all dependencies are satisfied.

Tuesday, October 28, 2008

Headless build problems with Grouped Configurations

PDE/Build has a property named groupConfigurations. If you set this property, build will group all the configurations (ie win32,win32,x86 & gtk,linux,x86) into one archive instead of creating a separate archive per configuration.

Kim & I, while working to get the performance baselines going, found a couple of gotchas/bugs around the use of this property.

Unexpected archive format for the group

The first thing we noticed was that the group archive was being created using ant's zip task instead of using the native zip as expected. Our builder's build.properties specified the following:
configs = *,*,*
archivesFormat = *,*,*-zip
groupConfigurations = true
If we had read the documentation, we would have seen that the archivesFormat is ignored for groups. However, this is not strictly true, and we can instead set a format for the group directly:
configs=*,*,*
archivesFormat = group,group,group-zip
groupConfigurations = true
This results in the zip format as desired.

The directory group.group.group does not exist

We also ran into the following error:
/builds/src/assemble.org.eclipse.sdk.tests.group.group.group.xml:257: The directory
/builds/src/tmp/eclipse/group.group.group does not exist
This turns out to be a rather old bug. The problem is that when features gather together the rootfiles they contribute, they copy them into platform specific folders. In this case, building the *,*,* configuration, the rootfiles were copied into tmp/ANY.ANY.ANY. However, because of the way the grouped configurations feature is implemented, the assembly scripts end up looking for the rootfiles in the group.group.group folder.

There are a few workarounds to this problem. One is to stick a mkdir in your customTarget/allElements assemble.<feature-id>.group.group.group target. This avoids the error, but you don't get the rootfiles in your group archive.

If you are using 3.4, then you can use the pre.archive target in customAssembly.xml to collect all the rootfiles into the correct folder:
   <target name="pre.archive">
<!-- for each config being built -->
<move file="${eclipse.base}/ANY.ANY.ANY/${collectingFolder}"
todir="${rootFolder}" failonerror="false"/>
<move file="${eclipse.base}/win32.win32.x86/${collectingFolder}"
todir="${rootFolder}" failonerror="false" />
<move file="${eclipse.base}/linux.gtk.x86/${collectingFolder}"
todir="${rootFolder}" failonerror="false" />
</target>

The ${rootFolder} property is defined by the caller of the pre.archive target, and in this case will be ${eclipse.base}/group.group.group/${collectingFolder}.

As usual, I have not actually tried running the above ant, the details may be different.

Wednesday, October 15, 2008

Sorting Bundles and Parallel Compilation in PDE/Build

In PDE/Build the compile order for bundles has always been based on the feature structure. Features are visited depth first, and for each feature the included bundles are sorted according to their dependencies. Dependencies outside the given feature are not considered, and must be included in a previously visited feature.

This can lead to some less than ideal feature structures as releng teams try to ensure that everything that a bundle depends on is included in a "deeper" feature.

This has been fixed for 3.5 M3. You can now define a property "flattenDependencies=true" in your build configuration build.properties file. This will result in bundles being sorted across feature boundaries.

Previously, bundles got compiled by delegation through the build.xml scripts for the containing features. When using the new flattenDependencies option, a new compilation xml script will be generated in the build directory. This only affects compilation, other build stages (ie gather.bin.parts) will still be delegated through the feature structure.

Parallel Compilation

With the above changes in compile order, it turns out to be a small step to get parallel compilation. Set both flattenDependencies and "parallelCompilation=true" in your build configuration. The result is that the compilation xml script will then group bundles using ant's parallel task. The result is something that looks something like this:

<target name="main">
<parallel threadsPerProcessor="3">
<ant antfile="build.xml" dir="plugins/org.eclipse.swt" target="build.jars"/>
<ant antfile="build.xml" dir="plugins/org.eclipse.swt.win32.win32.x86" target="build.jars"/>
<ant antfile="build.xml" dir="plugins/org.eclipse.osgi" target="build.jars"/>
</parallel>

<parallel threadsPerProcessor="3">
<ant antfile="build.xml" dir="plugins/org.eclipse.osgi.util" target="build.jars"/>
<ant antfile="build.xml" dir="plugins/org.eclipse.equinox.transforms.xslt" target="build.jars"/>
<ant antfile="build.xml" dir="plugins/org.eclipse.equinox.supplement" target="build.jars"/>
<ant antfile="build.xml" dir="plugins/org.eclipse.equinox.simpleconfigurator" target="build.jars"/>
<ant antfile="build.xml" dir="plugins/org.eclipse.equinox.p2.jarprocessor" target="build.jars"/>
<ant antfile="build.xml" dir="plugins/org.eclipse.equinox.launcher" target="build.jars"/>
<ant antfile="build.xml" dir="plugins/org.eclipse.equinox.launcher.win32.win32.x86" target="build.jars"/>
<ant antfile="build.xml" dir="plugins/org.eclipse.equinox.common" target="build.jars"/>
</parallel>

<parallel threadsPerProcessor="3">
<ant antfile="build.xml" dir="plugins/org.eclipse.update.configurator" target="build.jars"/>
<ant antfile="build.xml" dir="plugins/org.eclipse.equinox.frameworkadmin" target="build.jars"/>
<ant antfile="build.xml" dir="plugins/org.eclipse.cvs" target="build.jars"/>
<ant antfile="build.xml" dir="plugins/org.eclipse.core.runtime.compatibility.auth" target="build.jars"/>
<ant antfile="build.xml" dir="plugins/org.eclipse.core.jobs" target="build.jars"/>
</parallel>
....

Each group depends only only bundles that appeared in a previous group. You can control the ant threading attributes by setting parallelThreadCount and parallelThreadsPerProcessor.

We tested this by using it to compile the Eclipse SDK. Compile time dropped from 6:53 to 4:54, while this is only a 2 minute savings, it is a 29% improvement.

Friday, September 26, 2008

Custom Compiler Arguments in PDE/Build

[Edit 2010/07/14: As of the Helios release, PDE/Build now supports per bundle custom compiler arguments using "compilerArg" in the bundle's build.propeties file.]

The other day there was a post to the eclipse.platform.pde newsgroup asking how to include debug information for classes compiled with PDE/Build. The answer is to use the compilerArg property.

This property sets compiler arguments to use when compiling all your bundles. This got me wondering: What if I wanted to set custom compiler arguments for just one bundle, or if different bundles needed different arguments?

There is no explicit support in PDE/Build to do this, but a little digging shows the way!

The Generated Compilation Target

First, take a look at the javac task in the generated build.xml for a bundle. For a typical jar shaped bundle, it looks something like this:
<javac destdir="${temp.folder}/@dot.bin" [...]>
<compilerarg line="${compilerArg}" compiler="${build.compiler}"/>
<classpath refid="@dot.classpath" />
<src path="src/" />
<compilerarg value="@${basedir}/javaCompiler...args"
compiler="org.eclipse.jdt.core.JDTCompilerAdapter"/>
<compilerarg line="-log '${temp.folder}/@dot.bin${logExtension}'"
compiler="org.eclipse.jdt.core.JDTCompilerAdapter"/>
</javac>
Notice the first <compilerarg/>, it is passing the ${compilerArg} property that will affect all bundles. More importantly, notice the second <compilerarg/> whose value starts with '@'. A quick peek at the jdt docs reveals that this specifies a file where more command line arguments will be read.

javaCompiler.<library>.args

For each library specified in the bundle's build.properties file, PDE/Build generates a corresponding javaCompiler.<library>.args file. For most jar shaped bundles, this will be "javaCompiler...args".

We use this file to pass access rules (and custom file encodings) to the compiler. Access rules tell the compiler which packages you are allowed to see for each classpath entry; this is what allows PDE/Build to ensure that your compile-time classpath is as close as possible to the OSGi runtime classpath. The actual #ADAPTER#ACCESS# entries in this file are instructions to the JDT compiler adapter to modify the classpath with the given access rules.

The contents of this file are too ugly to bother pasting here, but the point is that this file is just additional compiler arguments. We can append to this file and insert whichever arguments we like.

Appending to the arguments file

We can use custom callbacks to append to the arguments file. For each library being compiled we can specify a target that will be called just before compilation. Here it is simple to append whatever we like to the arguments file:
<target name="pre.@dot">
<concat append="true" destfile="${basedir}/javaCompiler...args">
-g
-preserveAllLocals
</concat>
</target>



I haven't actually tried this for myself, so you may need to twiddle with the details a little. Note also that because the <compilerarg/> entries have
compiler="org.eclipse.jdt.core.JDTCompilerAdapter"
this will only work with the JDT compiler.

Wednesday, June 18, 2008

Example Headless build for a RCP product with p2

*UPDATE* For best results, I would suggest using the upcoming 3.4.1 release. Candidate builds are available from the download page under "3.4.1 Stream Builds", RC3 is available now.

I have prepared an example build setup that builds the RCP Mail Template and produces a p2-ized product. All the files needed for this are provided HERE.

The first step is to set up Eclipse with the RCP Delta pack which is required for the headless build of RCP products:
  1. Get the Eclipse SDK and the RCP Delta pack and unzip into the same location on disk.
  2. Start Eclipse and go to Window -> Preferences -> Plug-in Development -> Target Platform.
  3. Uncheck the option "Build target platform based on the target's installed plug-ins."
Create a new plug-in "com.acme.rcp" based on the RCP Mail Template. Create a new Product Configuration file "acme.product" for this plug-in. On the overview tab of the product editor, I set the id "com.acme.rcp.product" and the application "com.acme.rcp.application". I also set the version to be "1.0.0.qualifier".

The p2 Installable Unit (IU) that will be created for this product will take its id and version from the product id and version set here. I am setting the version to end in "qualifier" so that I can replace the qualifier at build time with the build id.

The product configuration for this example is based on plug-ins. On the configuration tab remove all plug-ins, add the plug-ins listed below and click Add Required Plug-ins.
  • com.acme.rcp
  • org.eclipse.equinox.p2.exemplarysetup
  • org.eclipse.equinox.p2.ui.sdk
  • org.eclipse.equinox.p2.touchpoint.eclipse
  • org.eclipse.equinox.p2.touchpoint.natives
  • org.eclipse.ecf.filetransfer
  • org.eclipse.ecf.provider.filetransfer
The product can now be run from overview tab of the product editor. If you do so you will notice the Help -> Software Updates... menu item. Selecting Software Updates at this time gets you a "This installation has not been configured properly for Software Updates" message.

Setting up the headless build

I like to run the headless build from inside Eclipse using a launch configuration. To do this, first import (Import... -> Plug-in Development -> Plug-ins and Fragments) "org.eclipse.pde.build" from the target platform into your workspace as a binary project.

Next, create a new general project named "Builder" and copy
org.eclipse.pde.build/templates/headless-build/build.properties"
/customTargets.xml"
into the builder project.

I want my headless build to do 3 things differently from a normal build:
  1. Set the version number in the product file to match the build id.
  2. Generate p2 metadata
  3. Use the generated metadata to perform p2 installs to get the final archives

1: Set the version number in the product file.

See the Build/customTargets.xml/preSetup target. We copy the "acme.product" file out of the com.acme.rcp bundle and into our buildDirectory. We then use the ant replace task to replace the version qualifier based on the time stamp. This uses a "acmePlugin" property which we will define separately. We also save the time stamp to a properties file that we read later when calling the p2 director.

2: Generate p2 metadata

PDE/Build provides integration with p2 to automatically generate p2 metadata. See Builder/build.properties where the following properties have been set:
    generate.p2.metadata = true
p2.metadata.repo=file:${buildDirectory}/repo
p2.artifact.repo=file:${buildDirectory}/repo
p2.flavor=tooling
p2.publish.artifacts=true

3: Perform p2 installs and archive the results

We do this in the Builder/customTargets.xml/postBuild target. We exec a new instance of eclipse to run the director application to install the product into a temporary directory and then we zip up the results.

We are running the same eclipse install again to do this. This means that any repositories that the eclipse install is set up with will also be considered during the director call (in addition to the repositories we pass on the command line).

The rest: build.properties

There are a few other properties to set in the build.properties file. We also set a number of properties on the command line in the launch configuration.

In the build.properties we set the product property to be the one that has the modified version qualifier. We also set the configuration we are building for.
 product=${buildDirectory}/acme.product
configs=win32,win32,x86
On the command line we are setting:
  • pluginPath: allows us to not copy the com.acme.rcp plugin into the buildDirectory.
  • buildDirectory: where all scripts will be generated and the location of our modified acme.product file.
  • baseLocation: We use the eclipse install we are running as our base.
  • acmePlugin: Define the location of the acme plugin for our custom task.
All these properties are set in the provided "Build Acme Product.launch" launch configuration and values are set at launch time using variables.

Results

Go to the Run Configurations... dialog and run the "Build Acme Product" launch config. The resulting zip should be under Builder/result. The project may need to be refreshed to see the results.

Running the build multiple times will result in the repository containing multiple versions of the product. You can then use that repository to update the product from one version to the next.

Problems Encountered

I did encounter some problems while creating this example.
  • Bug 237662: The metadata generator currently does not properly handle the new p2 simpleconfigurator style of config.ini and loses some start level information. This causes the installed product to not start. Workaround for this example is to not include org.eclipse.equinox.simpleconfigurator in the product. I will provide another example later to work around this problem.

  • Bug 222969: When including the org.eclipse.equinox.simpleconfigurator bundle in the product, the install is getting a bad relative path in the config.ini. Since we are excluding the simpleconfigurator because of the previous bug, this doesn't affect our example.

  • Bug 237647: NullPointerException in the director. This only occured once, workaround was simply to remove the bad profile directory under my eclipse install (eclipse/p2/org.eclipse.equinox.p2.engine/profileRegistry/ACMEProfile.profile)

Wednesday, May 28, 2008

Doc Time (with a little help from p2)

It is that time of year again: documentation time. Irrespective of any personal feeling towards writing documentation, I do realize that it is quite important for something like PDE/Build.

I will start with the eclipse help itself and then perhaps write some blog posts with small examples and how-tos. I may even take a look at the venerable Build and Test Automation article. It was written for 3.0, now that we are hitting 3.4 I'm sure it could use an update.

And a little p2 plug:
I used to use mozilla to write the html help docs, but I decided I'd like to stay within Eclipse. I'm sure there has to be a decent html editor somewhere in Ganymede.

So, to the Software Updates dialog, I add the Ganymede update site (http://download.eclipse.org/releases/ganymede/). "Web and Java EE Development -> WTP Web Page Editor" looks promising. I click the checky box and install. p2 downloaded a bunch of stuff, and after a restart, voila! : A webpage editor, easy as pie.

Wednesday, January 23, 2008

Examples of using PDE/Build

It can be very useful to have an example when setting up your first headless build. It occurs to me that there is a growing number of relatively simple examples: the pde.build junit tests.

The org.eclipse.pde.build.tests project can be checked out of cvs from dev.eclipse.org/cvsroot/eclipse/org.eclipse.pde.build.tests. The main test suite is PDEBuildTestSuite.

If you run the tests with the vm argument pde.build.noCleanup=true then the tests will run without deleting any files and you can then go take a look at the setup for each test.

Not every test is a full build, but many are. Generally the tests consist of generating some features and plugins, then generating the build.properties and allElements.xml configuration files and then running the build.

Of particular interest would be the generation of the build.properties file. This is done by loading the template file and then setting just a handful of the properties. See BuildConfiguration.getBuilderProperties(IFolder). This should give a good idea of what properties you need to change for your build.

*Edit 1/28/08 : The "pde.build.noCleanup" argument was added on 1/23/08, you must use a build N20080124-0010 or newer. The first I-Build containing these changes will be I20080129.

Friday, January 18, 2008

On Building Cycles

One of the enhancements I am considering for pde.build in 3.4 is partial handling of cycles.

The latest idea is allowing a cycle that contains at least one binary bundle. The presence of the binary bundle could be enough to break the cycle and allow us to compile the remaining bundles.

Consider the following:

A <- B <- C <- D <- E
----------^


Where we have a cycle between B, C, and D. If C is binary and does not need to be compiled, we may be able to compile the remaining bundles in the order A, D, B, E.

However, there are cases where this would not work. It could be that allowing this wouldn't actually help and would simply be giving people rope to hang themselves with.

If you have an opinion on this, please go comment on the bug and give the patches there a try.

Thursday, October 04, 2007

Selecting Plug-in Versions in a Headless Product Build

As Pascal likes to say: "PDE.Build is an onion." Don't be afraid to reach in and savor those inner layers. Here is an example:

We support building from a .product file. Generally, people base their product files on a list of plug-ins. There is currently no way to specify plug-in versions in the product file. If you have a requirement on a specific version of some bundle and for some reason there are multiple versions in your target, then you may be stuck with pde.build choosing the wrong version for you.

In a product build, a container feature is generated based on the contents of the .product file and the build is run using this generated feature. Features can specify the versions of the plug-ins they want to include, or they can specify "0.0.0" which means any version. They can also specify a version like "3.2.1.qualifier" which means any version starting with "3.2.1". (Though beware of this bug). The generated feature simply specifies "0.0.0" for each plug-in from the .product file.

PDE.Build provides an eclipse.idReplacer ant task. It uses this task to replace versions in the feature.xml with the actual versions that are built. It is easy to imagine using this task to modify the generated feature.xml to specify the plug-in versions you want before fetching or generating scripts.

In your customTarges.xml preFetch (or preGenerate) task, do something like this:

<eclipse.idReplacer
featureFilePath="${buildDirectory}/features/org.eclipse.pde.build.container.feature/feature.xml"
selfVersion="1.0.0"
featureIds=""
pluginIds="org.eclipse.foo:0.0.0,1.3.0.qualifier,org.bar:0.0.0,3.2.0.qualifier," />
Beware of this bug in 3.3.0, and this bug in 3.3.1.

Thursday, September 13, 2007

Eclipse launcher is losing its head

The Eclipse launcher has always required a graphics library (ie GTK, Motif) to show error messages and display the splash screen. It can't even start if the library is not found. This is a problem for people who have headless Eclipse applications that run in places without such luxuries.

I raised a bug to track this issue. Though I should point out that bug is incorrect in its description of the launcher's linking: the launcher is dynamically linked to the graphics libraries, (not statically linked as the bug says). The problem is that these dependencies are resolved at loadtime, not runtime. The fix is to load the graphics libraries at runtime using dlopen and lookup all the functions we need to call using dlsym.

I have created launchers for GTK and Motif that do this and attached them in bugzilla. These changes require more testing before I release them to CVS. People should try them out and give feedback on the bug.


One thing to note is that the Motif version is using a DT_RPATH tag to help find the libXm.so.2 library that ships on linux in the root of Eclipse. Most of what I've read suggests this is obsolete and not a good idea, so that will probably change.

Thursday, August 23, 2007

PDE Build Slides from EclipseCon 2007

The slides we (Sonia Dimitrov, Pascal Rapicault, and myself) presented at EclipseCon 2007 are now posted online. We're only about 6 months late putting them up.

They are attached to the Eclipsezilla entry for the tutorial. If you want to run the examples, you will need the Eclipse SDK and the RCP delta-pack. The examples were tested against an old integration build which is no longer available, but using 3.3 should work just fine.

Thursday, July 26, 2007

Greasemonkey: Making the internet slightly less annoying

One of the more annoying memes I've seen on some blogs is the use of "Canadia" instead of "Canada". The level of annoyance has finally risen high enough to overcome a certain level of laziness on my part. So I downloaded Greasemonkey and wrote my first JavaScript.

I'm sure it is not the best script ever written, but I now feel like I've done my part to make the internet a better place.

// ==UserScript==
// @name Canadia
// @description Change "Canadia" to "Canada"
// @include *
// ==/UserScript==

(function() {
var expression, replacement, regex, textnodes, node, s;

expression = "([Cc]anad)ia([^n]|$)";
replacement = "$1a$2";

regex = new RegExp(expression, 'g');

textnodes = document.evaluate( "//body//text()", document, null,
XPathResult.UNORDERED_NODE_SNAPSHOT_TYPE, null);

for (var i = 0; i < textnodes.snapshotLength; i++) {
node = textnodes.snapshotItem(i);
s = node.data;
s = s.replace(regex, replacement);
node.data = s;
}
})();

Wednesday, May 02, 2007

Eclipse and the Console

Starting in the warmup to 3.3M7 there is a second launcher on Windows: eclipsec.exe.

The Eclipse launchers in 3.3 have had issues with various aspects of the console on Windows. See some bugs: 146807, 167310, 168726, and 173962.

All of our problems came from the fact that the eclipse.exe was linked as a UI application and not a console application. In 3.2, the console behaviour was tied to using java.exe instead of javaw.exe, but starting in 3.3 M5, the vm is loaded in the eclipse process using the JNI Invocation API.

So just like javaw/java, we now have eclipse/eclipsec. Eclipsec.exe is linked as a console application so that the OS handles allocating the console for us. Using eclipsec.exe does not imply any default arguments, so you still need to specify things like -console and -consoleLog.

Monday, April 23, 2007

Equinox launcher & Swing

The equinox launcher starts the Java VM using the JNI invocation API. Because we are such good friends with SWT (and also because it is easier) the VM was always started on the first thread. This turns out not to be so good when you want to run Swing on the Mac. There was a conversation about it over on EclipseZone.

So I fixed the bug (181698) today by adding a new option to the launcher "--launcher.secondThread". Use it if you want swing to work. This option can be compared to the -XstartOnFirstThread for the java launcher.

Tuesday, March 27, 2007

Source Feature generation with PDE/Build

One of the cool features in PDE/Build is the generation of source features and plug-ins. This is perhaps one of those features that leads towards the "black magic" reputation that we seem to have.

Recently the CDT SDK builds have been missing their source. The source features and plug-ins exist, but they just didn't contain any src.zips. After some investigation, it seems that a long standing bug in pde.build has risen to the surface. It has always been there, it was just hidden by bug 114150 which was fixed in M6.

The workaround for the CDT is simple: just make sure that in the SDK feature, the generated org.eclipse.cdt.source feature is included after the org.eclipse.cdt feature upon which it is based.

Unfortunately, I'm sure that the fix in pde.build will be a bit harder.

Monday, February 05, 2007

EclipseCon 2007

As everyone knows, EclipseCon 2007 is March 5-8. I will be presenting a long tutorial on PDE Build with Pascal Rapicault and Sonia Dimitrov. I will also be giving a short talk on the new launcher story.

I even got a pretty little graffic:

I'm speaking at EclipseCon 2007