The problem of mixing library versions

Usually you wouldn’t mix versions. Build environments like Maven should handle this for you – although even maven can get things wrong.

Yesterday I was playing with Apache TomEE looking at migrating some webapp’s away from Apache Tomcat 7. The impetus for this was to get Rest services working and one of the TomEE profiles supports JAX-RS rest services. Tomcat only supports JAX-WS as thats in Java EE6 Web profile.

Anyhow, with the theory that the webapp’s should just run with minor config changes to the container (declaring datasources, that sort of thing) I installed TomEE and got Netbeans to deploy to it.

Bang!

None of the apps would deploy. They all complained about slf4j missing a method:

java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
    at org.apache.commons.logging.impl.SLF4JLocationAwareLog.debug(SLF4JLocationAwareLog.java:133)
    at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:221)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:401)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)

Now if you ever see this, it’s down to having mixed versions of slf4j in your classpath – and unlike Tomcat, TomEE uses the most recent version of slf4j.

The solution

Many hours later I find this article where someone had a similar problem last year: SLF4J Dependencies and LocationAwareLogger.log Method Not Found Exception.

Now my webapps also use the latest standalone tiles library for layout so after getting maven to show me the dependency tree (mvn dependency:tree) on the webapp it showed that I was pulling in an old slf4j version into the webapp.

So, with a minor change to the pom telling maven to exclude slf4j it finally worked.

Here’s the changes I ended up making to the pom:

<dependency>
  <groupId>org.apache.tiles</groupId>
  <artifactId>tiles-extras</artifactId>
  <!-- needed to exclude slf4j which causes incompatibilities -->
  <exclusions>
    <exclusion>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-nop</artifactId>
    </exclusion>
    <exclusion>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
    </exclusion>
    <exclusion>
      <groupId>org.slf4j</groupId>
      <artifactId>jcl-over-slf4j</artifactId>
    </exclusion>
  </exclusions>
</dependency>

<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-api</artifactId>
  <scope>provided</scope>
</dependency>

The first dependency is for tiles. The <version/> element is missing as I declare that in the project’s root pom and ensures all webapps use the same version.

The important bit is the <exclusions/> element, here we tell maven to exclude those artifacts from the war. There’s one or more <exclusion/> elements which contain just the <groupId/> and <artifactId/> elements – no <version/> here.

The second dependency is for slf4j. It’s here to allow any code provided by the war to have access to it. Again <version/> is missing as its in the root pom but we provide a <scope/> of provided. This tells maven it can use it for compiling but does not include it in the war.

Finally in the project’s other modules I just make sure that any pom with a reference to slf4j is also declared as provided so they don’t cause maven to include it either.

That’s about it. It only took me about 4 hours of hunting around to find the underlying cause. In the Windows world this is known as DLL Hell – well it can happen to any OS/Environment & it is hell when it strikes.

Notes:

  • If you use another logging library within your webapp then that shouldn’t be affected. log4j should just work, this problem appears just to be for slf4j.
  • If you use apache commons logging then you can declare that as provided if you wanted. This is because TomEE provides it at the container level as well as slf4j.

Manually downloading an artefact in Maven

Usually maven will download an artefact on it’s own however there are times when you need to do this manually – in this instance the local Nexus installation is down for maintenance so I had no choice but do it the hard way.

In this instance I needed the exec-maven-plugin which my local repository didn’t have but fortunately the maven-dependency-plugin allows you to download them:

 mvn org.apache.maven.plugins:maven-dependency-plugin:2.1:get \
     -Dartifact=org.codehaus.mojo:exec-maven-plugin:1.2 \
     -DrepoUrl=http://repo1.maven.org/maven2

All you need is to set artifact= to the required artifact & repoUrl to the remote repository – in this case Maven Central

Performing remote Maven 3 builds with NetBeans 7.0

This article covers how to get NetBeans to use a Maven installation on a remote server. This has been tested with Maven 3.0.2 but should work on any other version. It may also work for other IDE’s like Eclipse if you can get them to use an alternate maven installation other than the internal one.

Sometimes you may want to run maven on a remote machine from inside the NetBeans IDE. For me this is down to available memory on the local machine is limited enough on large projects that both NetBeans and Maven compete slowing the local machine down as it swaps the IDE in and out of main memory.

Another instance may be to perform builds on a standard OS (Linux in this case) rather than the OS running the IDE (OSX here).

Now this is pretty easy to implement. Here I have the IDE running on OSX and the builds on Linux (Ubuntu 10.10) but if you can get the prerequisites running in theory any OS should work.

Prerequisites

  • ssh access to both machines with public key authentication working
  • sshfs configured for mounting drives (you can use nfs or smb if you wish)
  • maven 3.0.2 installed on both machines – both machines need the same version

Setting up the environment

In this example:

  • sabrina – The local machine running NetBeans. It’s running OS X
  • kira – The remote machine which will run the builds. It’s running Ubuntu 10.10

Shared filesystems

The remote needs to have access to the local project directories for maven to work. As I have all development under a dev directory under home, on the remote machine I simply create a dev directory and then mount the local host from it using sshfs

<br />
peter@kira:~$ mkdir dev<br />
peter@kira:~$ sshfs peter@sabrina:dev dev<br />

If you have a custom settings.xml file, make sure it’s also present on the remote’s .m2 directory, i.e.

<br />
sabrina:~ peter$ scp .m2/settings.xml peter@kira:.m2<br />

Finally NetBeans needs access to the built artifacts, so to do this we mount the remotes .m2/repository locally. I use .m2/remoterepo for this

<br />
sabrina:~ peter$ mkdir .m2/remoterepo<br />
sabrina:~ peter$ sshfs peter@kira:.m2/repository .m2/remoterepo<br />

Maven

Next install your maven installation on both machines. I keep mine in the same place, but thats optional. For the purposes of this article I’m going to use /usr/local/apache/apache-maven-3.0.2

Next on the machine running NetBeans you need to create a duplicate installation /usr/local/apache/apache-maven-3.0.2-remote We need to have a copy for NetBeans to recognise it as a valid environment later on.

Now in this new installation we need to replace the /usr/local/apache/apache-maven-3.0.2-remote/bin/mvn script with the following:

<br />
#!/bin/bash<br />
#<br />
# Enable NetBeans 7.0 beta to run maven on a remote host</p>
<p># Pass the maven args as is<br />
MVN_ARGS=&quot;$@&quot;</p>
<p># The current directory, NetBeans does a cd so this will be the project<br />
# directory.<br />
#<br />
# If you need to translate the path between local and remote machines then<br />
# pass the output of pwd through sed first<br />
REMOTE_PROJECT_DIR=$(pwd)</p>
<p># Execute maven on remote host<br />
REMOTE_USER=peter<br />
REMOTE_HOST=kira</p>
<p># Location of remote maven installation<br />
REMOTE_MAVEN_HOME=/usr/local/apache/apache-maven-3.0.2</p>
<p># JAVA_HOME on remote machine<br />
REMOTE_JAVA_HOME=/usr/lib/jvm/java-6-openjdk/</p>
<p># ======================================<br />
# DO NOT EDIT ANYTHING BEYOND THIS POINT<br />
# ======================================</p>
<p># Remote mvn command<br />
REMOTE_MVN=$REMOTE_MAVEN_HOME/bin/mvn</p>
<p># Build the commands to run remotely<br />
CMD=&quot;cd ${REMOTE_PROJECT_DIR};&quot;<br />
CMD=&quot;${CMD}export JAVA_HOME=${REMOTE_JAVA_HOME};&quot;<br />
CMD=&quot;${CMD}$REMOTE_MVN ${MVN_ARGS}&quot;</p>
<p># Now run the remote build<br />
echo<br />
echo ------------------------------------------------------------------------<br />
echo Executing remote build on $REMOTE_HOST as $REMOTE_USER<br />
echo Remote command: $CMD<br />
echo ------------------------------------------------------------------------<br />
echo<br />
exec ssh ${REMOTE_USER}@${REMOTE_HOST} &quot;${CMD}&quot;<br />

In this script you need to ensure that the settings are correct:

  • REMOTE_USER and REMOTE_HOST point to your remote host,
  • REMOTE_MAVEN_HOME is the location of where maven is installed on the remote host,
  • REMOTE_JAVA_HOME is the location of the JDK on the remote host.

Now you may need to alter the line defining REMOTE_PROJECT_DIR. Shown above it gets set to the directory containing the maven project on the local host. If this differs on the remote then you need to either change to translate this to the location on the remote.

For example I have my NetBeans projects under a directory called dev in my home directory. As this is OS X that becomes /Users/peter/dev however on Linux the path is /home/peter/dev

I get around this by on the remote simply creating a symlink for /Users pointing to /home but you could modify the script to do that for you.

Configuring NetBeans

The last step is getting NetBeans to use the new remote.

Start NetBeans and enter preferences Then the Miscellaneous tab and finally Maven:

  • Under Maven Home select Browse and select the dummy remote installation, in my case /usr/local/apache/apache-maven-3.0.2-remote
  • Under Local Repository select Browse and point it at .m2/remoterepo
  • Select OK

Now if you try to run any build you’ll now see it run on the remote machine and as far as NetBeans is concerned it’s running locally. For example in the NetBeans output window you should see something like the following:

<br />
cd /Users/peter/dev/misc/example; JAVA_HOME=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home /usr/local/apache/apache-maven-3.0.2-remote/bin/mvn --batch-mode --lax-checksums --fail-fast --no-plugin-updates --batch-mode clean</p>
<p>------------------------------------------------------------------------<br />
Executing remote build on 192.168.30.108 as peter<br />
Remote command: cd /Users/peter/dev/misc/example;export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/;/usr/local/apache/apache-maven-3.0.2/bin/mvn --batch-mode --lax-checksums --fail-fast --no-plugin-updates --batch-mode clean<br />
------------------------------------------------------------------------</p>
<p>Command line option -npu is deprecated and will be removed in future Maven versions.<br />
Scanning for projects...<br />
------------------------------------------------------------------------<br />
Reactor Build Order:<br />

Future invocations

Once setup, whenever you restart either machines you simply need to mount each others drives before starting NetBeans as all the rest should then be setup.

<br />
sabrina:~ peter$ sshfs peter@kira:.m2/repository .m2/remoterepo<br />
peter@kira:~$ sshfs peter@sabrina:dev dev<br />

Getting NetBeans 6.9 RC1 to insert the correct copyright text in maven projects

Netbeans has supported for quite some time the ability to set the license for a project so that when a new file is created, the template automatically inserts the correct one. The problem is that in the past this has not been that well supported when it comes to maven based projects.

Now in some of the recent releases this now works, with a little bit of configuration – I’ve tested this only with 6.9 RC1, but this should (in theory) work with 6.8 and possibly 6.5.

First you need to choose your license. Open the Tools menu and select Templates. This should then open the Template Manager. Once open, expand the Licenses entry to show the available licenses.

Netbeans 6.9 RC1 Template Manager

Now the ones that can be used in maven are the ones named: license-*.txt. If the one you want is there, then make a note if it’s name.

Creating a new License Template

If the license you want is not there then you’ll need to create a template for it. For example the above screenshot shows bsd and gpl3cp which are the two I use for my Open Source projects but they are not provided by Netbeans so I had to create them.

For the BSD template, create the following file in the root directory of your project (so others can reuse the same license), in this case license-bsd.txt:

<#if licenseFirst??>
${licenseFirst}
</#if>
${licensePrefix} Copyright (c) 1998-${date?date?string("yyyy")}, Peter T Mount
${licensePrefix} All rights reserved.
${licensePrefix}
${licensePrefix} Redistribution and use in source and binary forms, with or without
${licensePrefix} modification, are permitted provided that the following conditions
${licensePrefix} are met:
${licensePrefix}
${licensePrefix} * Redistributions of source code must retain the above copyright
${licensePrefix}   notice, this list of conditions and the following disclaimer.
${licensePrefix}
${licensePrefix} * Redistributions in binary form must reproduce the above copyright
${licensePrefix}   notice, this list of conditions and the following disclaimer in the
${licensePrefix}   documentation and/or other materials provided with the distribution.
${licensePrefix}
${licensePrefix} * Neither the name of the retep.org.uk nor the names of its contributors
${licensePrefix}   may be used to endorse or promote products derived from this software
${licensePrefix}   without specific prior written permission.
${licensePrefix}
${licensePrefix} THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
${licensePrefix} "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
${licensePrefix} LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
${licensePrefix} A PARTICULAR PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER
${licensePrefix} OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
${licensePrefix} EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
${licensePrefix} PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
${licensePrefix} PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
${licensePrefix} LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
${licensePrefix} NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
${licensePrefix} SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
<#if licenseLast??>
${licenseLast}
</#if>

Next select the Licenses heading and press Add. Select your new file then add. The template’s now been added. Later if you need to tweek it you can now select it and press Open in editor.

Enabling the template for your Maven project

The next step is to tell Netbeans to use this template. To do this we need to add another file, profiles.xml, which goes into the same directory as your project’s root pom.

<profilesXml>
    <profiles>
        <profile>
            <id>netbeans-private</id>
            <activation>
                <property>
                    <name>netbeans.execution</name>
                    <value>true</value>
                </property>
            </activation>
            <properties>
                <netbeans.hint.license>bsd</netbeans.hint.license>
            </properties>
        </profile>
    </profiles>
</profilesXml>

You’ll notice the netbeans.hint.license property. It’s value is the middle part of your template’s name.

Once set, reload the project within netbeans and every time you create a new file who’s template includes the license it will include the correct one for the project.

Netbeans Template Tips

When creating your own templates, you can include the license at any time using the following snippet:

<#assign licenseFirst = "/*">
<#assign licensePrefix = " * ">
<#assign licenseLast = " */">
<#include "../Licenses/license-${project.license}.txt">

Here you’ll notice we have to assign three values then include the license – the above example is for .java files but you can change it to any values for any type of file.

The other useful tip is the following line from the license template above:

${licensePrefix} Copyright (c) 1998-${date?date?string("yyyy")}, Peter T Mount

Here the template will always insert the current year into the file, rather than having to remember to change the templates every year – although you’ll still have to amend existing files.

It’s been a busy couple of months

It’s been a couple of busy months with most of my time being taken up with my day job.

Most of my time has been spent with either tracking down issues with our live environment, or trying to finish off a couple major projects (both related to XMPP) interspersed with the usual major partner getting in the way.

Any how, over the last couple of weeks I’ve been finishing off some new features which cover most of my public projects and this post will hopefully cover some of the details.

Hopefully these will be released this weekend, time allowing.

The new features are:

RetepTools
* the builder api within retepTools has been updated
* the jaxb plugin library has been cleaned up with common generation code split out to enable reuse
* retepTools as a project is almost ready for deployment to maven central
* a new pligun has been added to jaxb which generates builders for jaxb objects

RetepMicroKernel
* spring has been updated to the latest version 3 (it was on 2.5)
* the core module has been broken up into individual & independent modules
* a new groovy module which enables groovy scripts to be run from the command line
* a major bug fix where exceptions thrown during application startup causes the process to hang has been fixed
* web applications can now be deployed as a war with either jetty or tomcat (they are both supported with their own modules)
* you can now embed Apache Derby within the environment

I’m leaving out the retepXMPP changes out of this list as they need their own article. Suffice it to say, I’ve got a lot waiting for release, just need the time.

Finally, this post is also a test of submitting a blog post from a BlackBerry using the WordPress app so the formatting may be off a tad – won’t know how it goes until I see it in a real browser.

Using groovy to generate java sources in maven

In this article I’ll briefly cover how to use Groovy to generate java sources as part of a maven build.

There are times when you need to generate Java source which is either repetitive, based on external data or both. Now there are tools out there for some tasks like this, but for most simple tasks running a groovy script is more than enough.

Now for the purposes of this article, I’m going to presume we need an enum which represents the cards in a Standard deck of cards, one entry per card.

First we need to create our groovy script. Now the best place to put this is in the src/main/script directory of your project. Here’s the script for this (writing groovy is not in scope for this article) which goes into the file src/main/script/CardDeckGen.groovy:

class CardDeckGen {

    public generate( project, String packageName, String className ) {

        // Where to write the classes
        File targetDirectory = new File( project.build.directory + '/generated-sources/groovy' )

        // The directory to write the source to
        File packageDir = new File( targetDirectory, packageName.replace( '.', '/' ) )

        // Now to create our enum
        def out = []
        out<<'package '+packageName+';\n'
        out<<'public enum '+className+' {\n'

        // We have four suits
        def suit = [ 'D', 'S', 'H', 'C']

        // Each suit has A, 2-9, T, J, Q & K
        def rank = [ 'A', '2', '3', '4', '5', '6', '7', '8', '9', 'T', 'J', 'Q', 'K' ]

        // For each suit, write an entry for each card
        suit.eachWithIndex {
            s, i -> def indent = i==0 ? '    ' : ',\n    '
            rank.each {
                r -> out<< indent + s + r
                indent = ','
            }
        }

        // Mark the end of the enum list
        out<<';\n'

        // Finish the enum class
        out<<'}\n'

        // Convert the array into a string
        StringBuilder sb = new StringBuilder()
        out.each { sb.append(it) }

        // Now write the source, ensuring the directory exists first
        packageDir.mkdirs()
        new File( packageDir, className + ".java" ).write( sb.toString() );
    }

}

Next we need to get maven to run this script, so in pom.xml we first need to make sure we are using the correct version of the jdk – 1.5 or 1.6 would be enough:

<plugin>
    <artifactId>maven-compiler-plugin</artifactId>
    <configuration>
        <source>1.6</source>
        <target>1.6</target>
    </configuration>
</plugin>

Next groovy, here we add the gmaven-plugin which runs a small code snippet calling our class:

<plugin>
    <groupId>org.codehaus.groovy.maven</groupId>
    <artifactId>gmaven-plugin</artifactId>
    <executions>
        <execution>
            <phase>generate-resources</phase>
            <goals>
                <goal>execute</goal>
            </goals>
            <configuration>
                <scriptpath>
                    <element>${basedir}/src/main/script</element>
                </scriptpath>
                <source>
                    CardDeckGen cdg = new CardDeckGen()
                    cdg.generate( project, 'uk.org.retep.example', 'CardDeck' )
                </source>
            </configuration>
        </execution>
    </executions>
</plugin>

Now that’s enough to generate our enum, however maven doesn’t know about it so it wont compile. To do that we need to add it so the compiler picks it up:

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>build-helper-maven-plugin</artifactId>
    <executions>
        <execution>
            <id>add-groovy-sources</id>
            <phase>generate-sources</phase>
            <goals>
                <goal>add-source</goal>
            </goals>
            <configuration>
                <sources>
                    <source>${project.build.directory}/generated-sources/groovy</source>
                </sources>
            </configuration>
        </execution>
    </executions>

Now you can run maven.mvn clean install
If all goes well you’ll get a maven artifact containing an enum:

package uk.org.retep.example;
public enum CardDeck {
    DA,D2,D3,D4,D5,D6,D7,D8,D9,DT,DJ,DQ,DK,
    SA,S2,S3,S4,S5,S6,S7,S8,S9,ST,SJ,SQ,SK,
    HA,H2,H3,H4,H5,H6,H7,H8,H9,HT,HJ,HQ,HK,
    CA,C2,C3,C4,C5,C6,C7,C8,C9,CT,CJ,CQ,CK;
}

This was a simple example but quite a powerful one. I’ve used this technique for generating more complex versions of Card’s to classes based on data contained within RFC’s (STRINGPREP etc).

For help on writing Groovy stripts take a look at the Groovy Getting Started Guide.

Releasing to Kenai via Maven

Ever since the website feature appeared on Kenai I’ve been trying to get releases to work using it to host the maven repository. Although it works, it’s always had the odd issue where whilst uploading artifacts to the repository, the connection would timeout causing the release build to fail.

For small projects this isn’t really a problem, but for some of mine this is an issue to either the number of artifacts present (retepXMPP currently has over a hundred) we can’t have the release build to fail part way through leaving both the mercurial and maven repositories in an inconsistent state.

The retepMicroKernel project is further complicated by the fact that it has native code – so a release has to be performed in parts with the native code built on the respective platforms.

Fortunately there is a solution to this – perform the release build into a staging repository then once it’s complete merge that repository with the public one. This also means we can perform a release and if something horrible goes wrong we just redo the release without the public seeing the failed release.

Fabrizio Giudici also found this solution with an additional benefit. The maven release process makes several commits into the repository, but as mercurial is a distributed scm we can use that to our advantage – if the release build fails it doesn’t cause issues remotely with having to remove tags etc (which I’ve done before).

So here’s what I did when releasing version 9.11 of retepTools this week, first I followed Fabrizio’s suggestions for configuring the root pom.xml file.

Then I created a temporary directory and inside that clone the public mercurial repo then make a clone of the clone:

# The maven staging repository
mkdir -p ~/tmp/stagingrepo

# The staging mercurial repository - this is a clone of the public repository
cd ~/tmp
hg clone https://hg.kenai.com/hg/reteptools~hg reteptools-stage

# The release mercurial repository - this is the one we will run maven against
# it is a clone of the staging repository
hg clone reteptools-stage reteptools

Now we run the release process:

cd ~/tmp/reteptools
mvn --batch-mode \
       -Dstaging.hg.repo.url=~/tmp/reteptools \
       -Dstaging.mvn.repo.url=file:///Users/peter/tmp/stagingrepo \
       -DreleaseVersion=9.11 \
       -DdevelopmentVersion=9.12-SNAPSHOT \
       -Dtag=retepTools-9.11 \
       release:prepare release:perform

Maven will now perform the release, making commits into the ~/tmp/reteptools repository and pushing them to ~/tmp/reteptools-stage.

Once we are happy that the release is clean we make it public. First mercurial – we need to push the changes back to kenai:

cd ~/tmp/reteptools-stage
hg push

Next the artifacts:

# A temporary directory for maven - it will not run without this directory
mkdir -p tmpdir
mvn org.codehaus.mojo:wagon-maven-plugin:1.0-beta-2:merge-maven-repos \
  -Dwagon.source=file:///Users/peter/tmp/stagingrepo/ \
  -Dwagon.target=dav:https://kenai.com/website/reteptools/maven/releases/ \
  -Dwagon.targetId=retep.releases \
  -Djava.io.tmpdir=target

[Edited to fix a couple of errors in the maven commands]