The problem of mixing library versions

Usually you wouldn’t mix versions. Build environments like Maven should handle this for you – although even maven can get things wrong.

Yesterday I was playing with Apache TomEE looking at migrating some webapp’s away from Apache Tomcat 7. The impetus for this was to get Rest services working and one of the TomEE profiles supports JAX-RS rest services. Tomcat only supports JAX-WS as thats in Java EE6 Web profile.

Anyhow, with the theory that the webapp’s should just run with minor config changes to the container (declaring datasources, that sort of thing) I installed TomEE and got Netbeans to deploy to it.

Bang!

None of the apps would deploy. They all complained about slf4j missing a method:

java.lang.NoSuchMethodError: org.slf4j.spi.LocationAwareLogger.log(Lorg/slf4j/Marker;Ljava/lang/String;ILjava/lang/String;[Ljava/lang/Object;Ljava/lang/Throwable;)V
    at org.apache.commons.logging.impl.SLF4JLocationAwareLog.debug(SLF4JLocationAwareLog.java:133)
    at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:221)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:401)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)

Now if you ever see this, it’s down to having mixed versions of slf4j in your classpath – and unlike Tomcat, TomEE uses the most recent version of slf4j.

The solution

Many hours later I find this article where someone had a similar problem last year: SLF4J Dependencies and LocationAwareLogger.log Method Not Found Exception.

Now my webapps also use the latest standalone tiles library for layout so after getting maven to show me the dependency tree (mvn dependency:tree) on the webapp it showed that I was pulling in an old slf4j version into the webapp.

So, with a minor change to the pom telling maven to exclude slf4j it finally worked.

Here’s the changes I ended up making to the pom:

<dependency>
  <groupId>org.apache.tiles</groupId>
  <artifactId>tiles-extras</artifactId>
  <!-- needed to exclude slf4j which causes incompatibilities -->
  <exclusions>
    <exclusion>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-nop</artifactId>
    </exclusion>
    <exclusion>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-api</artifactId>
    </exclusion>
    <exclusion>
      <groupId>org.slf4j</groupId>
      <artifactId>jcl-over-slf4j</artifactId>
    </exclusion>
  </exclusions>
</dependency>

<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>slf4j-api</artifactId>
  <scope>provided</scope>
</dependency>

The first dependency is for tiles. The <version/> element is missing as I declare that in the project’s root pom and ensures all webapps use the same version.

The important bit is the <exclusions/> element, here we tell maven to exclude those artifacts from the war. There’s one or more <exclusion/> elements which contain just the <groupId/> and <artifactId/> elements – no <version/> here.

The second dependency is for slf4j. It’s here to allow any code provided by the war to have access to it. Again <version/> is missing as its in the root pom but we provide a <scope/> of provided. This tells maven it can use it for compiling but does not include it in the war.

Finally in the project’s other modules I just make sure that any pom with a reference to slf4j is also declared as provided so they don’t cause maven to include it either.

That’s about it. It only took me about 4 hours of hunting around to find the underlying cause. In the Windows world this is known as DLL Hell – well it can happen to any OS/Environment & it is hell when it strikes.

Notes:

  • If you use another logging library within your webapp then that shouldn’t be affected. log4j should just work, this problem appears just to be for slf4j.
  • If you use apache commons logging then you can declare that as provided if you wanted. This is because TomEE provides it at the container level as well as slf4j.

Formatting xml in emacs

I had this problem of debugging some xml but when reading the output of some log4j it was almost impossible to read so I needed some way of prettifying the xml quickly.

For this example I have the following xml:

<?xml version="1.0"?><xml><iq xmlns="jabber:component:accept" from="test1@temp.retep.org/client" id="iq_257" to="service.retep.org" type="get"><query xmlns="some:namespace"/></iq></xml>

So how do we pretify this in emacs?

Well the first thing to do is to write an extension function & place it into your ~/.emacs file. Placing it here means that when you open emacs the extension is available:

(defun xml-format ()
  (interactive)
  (save-excursion
    (shell-command-on-region (mark) (point) "xmllint --format -" (buffer-name) t)
  )
)

Now this works by passing the buffer to the xmllint utility and replaces it with the output – in this case nicely formatted xml.

Now we need to install xmllint:

pi@lindesfarne: ~$ sudo apt-get install libxml2-utils

Ok so now open emacs and open the xml. To format first select the xml you want to format then Press Escape then x followed by xml-format & press return. You should then get the xml nicely formatted:

<?xml version="1.0"?>
<xml>
  <iq xmlns="jabber:component:accept" from="test1@temp.retep.org/client" id="iq_257" to="service.retep.org" type="get">
    <query xmlns="some:namespace"/>
  </iq>
</xml>

NioSax – Sax style xml parser for Java NIO

NioSax (pronounced ‘Neo-Sax’) provides a Java NIO friendly XML push parser similar in operation to SAX. Unlike SAX, with NioSax it is possible for the xml source to contain partial content (i.e. only part of the XML stream has been received over the network). When this occurs, instead of failing with an error, NioSax simply stops. As soon as your application receives more data you simply call the same instance of the parser again and it will resume parsing where it left off.

NioSax (pronounced ‘Neo-Sax’) provides a Java NIO friendly XML push parser similar in operation to SAX. Unlike SAX, with NioSax it is possible for the xml source to contain partial content (i.e. only part of the XML stream has been received over the network). When this occurs, instead of failing with an error, NioSax simply stops. As soon as your application receives more data you simply call the same instance of the parser again and it will resume parsing where it left off.

The public API consists of the classes within this package, although the bare minimum required for use are the NioSaxParser, NioSaxParserHandler and NioSaxSource classes.

To use NioSax you simply use NioSaxParserFactory to create a NioSaxParser, implement a SAX ContentHandler and finally create a NioSaxSource which references the content.

Then you can parse one or more ByteBuffer’s by updating the NioSaxSource with each buffer and pass it to the NioSaxParser.parse(NioSaxSource) method.

The only other two things you must to do with the parser is to ensure that you call NioSaxParser.startDocument() prior to any parsing, and call NioSaxParser.endDocument() once you are done with the parser so any resources used can be cleaned up.

Example

First in maven we need to add a dependency to NioSax. For details of the repository click on the ‘reteptools’ menu above. However you’ll need to add the following to your pom:

<dependency>
    <groupId>uk.org.retep</groupId>
    <artifactId>niosax</artifactId>
    <version>10.6</version>
</dependency>

Now we’ll create a parser:

import java.nio.ByteBuffer;
import uk.org.retep.niosax.NioSaxParser;
import uk.org.retep.niosax.NioSaxParserFactory;
import uk.org.retep.niosax.NioSaxParserHandler;
import uk.org.retep.niosax.NioSaxSource;

public class MyParser
{
    private NioSaxParser parser;
    private NioSaxParserHandler handler;
    private NioSaxSource source;

    public void start()
    {
        NioSaxParserFactory factory = NioSaxParserFactory.getInstance();

        parser = factory.newInstance();
        parser.setHandler( handler );
        source = new NioSaxSource();

        parser.startDocument();
    }
}

Next, when you receive data from some nio source and have the data in a ByteBuffer you need to pass it to the parser:

    public void parse( ByteBuffer buffer )
    {
        // flip the buffer so the parser starts at the beginning
        buffer.flip();

        // update the source (presuming the buffer has changed)
        source.setByteBuffer( buffer );

        // Parse the available content then compact
        parser.parse( source );
        source.compact();
    }

Finally we must call endDocument() to release any resources:

    public void close()
    {
        // releases any resources and notifies the handler the docment has completed
        parser.endDocument();
    }

Now all we need to is when we receive some data from an external source like a Socket, we pass the ByteBuffer to the parse method. This then passes it to the NioSax parser which in turn calls the ContentHandler as the parse progresses.

When it gets to the end of the available content, it compacts the buffer so that it can be reused.

Usually the buffer will now be empty, however if there was partial content (like only part of a Unicode character was present) then the parser would stop prior to that character and that character would remain in the buffer. The next packet received via nio would have the rest of that character and the parser would then continue where it left off.

This was originally posted early in 2009 but the post seemed to have vanished so this article is loosely based on the documentation for NioSax.

Implementing Builders with JAXB generated objects

While JAXB is a good method of mapping XML into Java objects, if your schema’s are large or complex, generating the required Object graph prior to marshaling into XML can be both tedious and error prone due to the large amount of ‘boiler plate’ code of creating an object, calling accessor methods to set the values etc.

A common pattern for easing this sort of thing is known as the Builder pattern where you either have a common object holding configuration and then it builds the objects for you, or a set of objects each configured with the information and when required then build the final Object graph.

Previously however, this involved writing these builder classes manually once you have generated the model from a schema, however retepTools provides a JAXB plugin which, with a little configuration within the bindings will automatically generate these builders for you.

First an example of a simple pair of objects to be generated by JAXB – this is an abridged version of the jabber:client namespace in XMPP:

<?xml version='1.0' encoding='UTF-8'?>

<xs:schema
    xmlns:xs='http://www.w3.org/2001/XMLSchema'
    targetNamespace='jabber:client'
    xmlns='jabber:client'
    elementFormDefault='qualified'>

  <xs:element name='message'>
     <xs:complexType>
        <xs:sequence>
          <xs:choice minOccurs='0' maxOccurs='unbounded'>
            <xs:element ref='subject'/>
            <xs:element ref='body'/>
          </xs:choice>
        </xs:sequence>
        <xs:attribute name='from' type='xs:string' use='optional'/>
        <xs:attribute name='from' type='xs:string' use='optional'/>
     </xs:complexType>
  </xs:element>

  <xs:element name='body'>
    <xs:complexType>
      <xs:simpleContent>
        <xs:extension base='nonEmptyString'>
          <xs:attribute ref='xml:lang' use='optional'/>
        </xs:extension>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>

  <xs:element name='subject'>
    <xs:complexType>
      <xs:simpleContent>
        <xs:extension base='nonEmptyString'>
          <xs:attribute ref='xml:lang' use='optional'/>
        </xs:extension>
      </xs:simpleContent>
    </xs:complexType>
  </xs:element>

  <xs:simpleType name='nonEmptyString'>
    <xs:restriction base='xs:string'>
      <xs:minLength value='1'/>
    </xs:restriction>
  </xs:simpleType>

</xs:schema>

Now if run through JAXB this would generate three classes: Message, Body and Subject. With this default model you would have to do something like the following to create a valid Message:

Message message = new Message();
message.setFrom( "juliet@example.com/balcony" );
message.setTo( "romeo@example.net" );

Body body = new Body();
body.getBodyOrSubject().add( "Wherefore art thou, Romeo?" );
message.setBody( body );

Now with this simple example thats fine, but imagine something where your model is far more complex, or worse can contain arbitary namespaces as children such as in XMPP where Message can contain objects within any namespace?

This is where Builders become useful. Imagine the above being converted into this:

Message message = new MessageBuilder().
	setFrom( "juliet@example.com/balcony" ).
	setTo( "romeo@example.net" ).
	addBody( new BodyBuilder().
		 setValue( "Wherefore art thou, Romeo?") ).
	build();

Although at first this may look similar to the standard way, the builders break the code into more manageable chunks. The builders themselves can be reused, so if you are sending multiple messages then you could change the above code to:

MessageBuilder builder = new MessageBuilder().
	setFrom( "juliet@example.com/balcony" ).
	addBody( new BodyBuilder().
		 setValue( "Wherefore art thou, Romeo?") );

Message message1 = builder.setTo( "romeo@example.net" ).build();

Message message2 = builder.setTo( "tybalt@example.com/cousin" ).build();

As you can see, the use of builders becomes apparent, we created two Message objects with a single change between them in a lot less code.

On the next page we’ll configure maven to build these builders.

Using JAXB to marshall XMPP

Back in late 2008, one of the original design goals for the rewrite of my retepXMPP project was to use JAXB for handling the marshalling of XMPP Stanzas into POJO’s. The main ideas behind this was to standardise against the XMPP schemas available online and to make the addition of further protocols easier – mainly not to break existing code.

Now this was fine until I started testing against a couple of client libraries, certain messages were being ignored. It turned out that the XML generated by JAXB does not follow the rules defined in RFC 3920bis. The problem here was that JAXB was placing all of the namespaces together in the root.

For example here’s one of the examples from RFC3921bis as generated by JAXB:

  <iq from='juliet@example.com/balcony'
       id='rg1'
       type='get'  xmlns='jabber:client'  xmlns:ns1='jabber:iq:roster'>
    <ns1:query/>
  </iq>

Here’s what it should look like:

  <iq from='juliet@example.com/balcony'
       id='rg1'
       type='get'>
    <query xmlns='jabber:iq:roster'/>
  </iq>

There’s two differences here, but the main issue is where ‘jabber:iq:roster’ is declared in the root iq element and not against the query element it’s supposed to be – the other is the declaration of the ‘jabber:client’ namespace but that one is not a problem here.

Although both are strictly correct XML wise, it isn’t for XMPP and most parsers expect the stream to follow the rules.

So why is JAXB doing this? Well apparently its by design and, if you use the JAXB-RI like I do, there’s not much we can do to change this.

In JAXB 1 it actually declared the namespaces at the appropriate places so we would get the ‘jabber:iq:roster’ namespace against the query element as expected and the XMPP stanza would then conform correctly.

However in JAXB 2.0 they changed this to the current behaviour for performance reasons. Apparently with large documents looking ahead for what namespaces are present uses a lot of resources, so they declare them first.

Now I can see their point but for one small issue – this presumes that, when a document is being parsed then unmarshalled by JAXB it was originally produced by JAXB – what if it wasn’t generated by JAXB? In that case JAXB would still do the lookup anyhow.

Unfortunately when I took a look at the source for JAXB 2.2 this declaring of namespaces is done pretty close to the start of the marshalling process, so unless they add an option to disable it we have to live with this.

So how to fix this so we can still use JAXB but generate conforming XML?

Well I’ve got a solution but it’s not pretty. The solution simply involves breaking the marshalling/unmarshalling process up into separate units of work, one per namespace. We marshal the first object, then – if it has children, marshal them.

Now this involves additional work during the marshalling/unmarshalling process but it works. The downside is that we have to modify the schemas to do this.

Fortunately the XSF’s schemas are not concrete. As Peter Saint-Andre’s put it a few weeks ago on the muc mailing list, those schemas are ‘descriptive, not normative’. This means they are representative of what the XEP’s define, but they can be changed. In fact for some (like XEP-0045/MUC) the schemas don’t define the extension points used by other XEP’s. This means we can make some simple modifications to them to get this ‘hack’ to work.

So now I have this working pretty well – marshalling to xml runs smoothly. Unmarshalling still has a few problems but I’m being hit by a nice issue with Java’s generics but that’s going to be a later post.

Binding custom objects to attributes in JAXB

When generating Java sources from an XML Schema with JAXB, the type of field used to represent an attribute is determined by it’s type in the schema.

For example, a snippet from jabber-client.xsd:

<xs:element name="message">
  <xs:complextype>
    <xs:attribute name="from" type="xs:string" use="optional">
      <xs:attribute name="to" type="xs:string" use="optional">
    </xs:complextype>
  </xs:sequence>
</xs:element>

By default, JAXB would create the object (Message in this case) with two String properties. To change this to another object (say JID) we would add a simple binding changing these to use JID and define two static methods who’s job it is to convert between a String and a JID.

So the bindings would be something like:

<bindings schemalocation="jabber-client.xsd">
  <bindings node="/xsd:schema/xsd:element[@name='message']/xsd:complexType">
    <bindings node="xsd:attribute[@name='from']">
      <property>
        <baseType>
          <javaType name="uk.org.retep.xmpp.JID"
            parseMethod="uk.org.retep.xmpp.jaxb.adaptor.XMPPDatatypeConverter.parseJID"
            printMethod="uk.org.retep.xmpp.jaxb.adaptor.XMPPDatatypeConverter.printJID"
          />
        </baseType>
      </property>
    </bindings>
  </bindings>
</bindings>

Now for mose usecases this is fine and works pretty well. The problem is when you have a large number of custom bindings.

What JAXB does for each binding is that it generates an anonymous Adapter class. In that adaper class are two methods which call the two methods declared in the binding – in the case of binding to a Double you would get the following:

public class Adapter1
    extends XmlAdapter<String, Double>
{

    public Double unmarshal(String value) {
        return (javax.xml.bind.DatatypeConverter.parseDouble(value));
    }

    public String marshal(Double value) {
        if (value == null) {
            return null;
        }
        return (javax.xml.bind.DatatypeConverter.printDouble(value));
    }

}

The problem here is that JAXB generates one of these for every instance, so you can end up having multiple Adapter classes present, all doing the same thing. In one instance I had a single schema contain 16 duplicates of the above code. In retepXMPP I had almost 60 spread over multiple schemas just for handling JID. This is a lot of wasted, duplicated code and if you use a large number of schemas then this is going to cause both your jar sizes and permgen use to increase for no real reason.

So it would be nice to reduce this down to a single adapter.

Fortunately there is a way when using the JAXB RI. There is an alternate javaType binding which takes the full class name of an adapter class instead – using this causes JAXB to use that single class instead of generating these duplicate classes. The main difference is replacing the parseMethod and printMethod attributes with a single adapter attribute, and qualifying the javaType with the http://java.sun.com/xml/ns/jaxb/xjc namespace.

So the bindings would be something like:

<bindings
  xmlns="http://java.sun.com/xml/ns/jaxb"
  xmlns:xsd="http://www.w3.org/2001/XMLSchema"
  xmlns:xjc="http://java.sun.com/xml/ns/jaxb/xjc"
  xmlns:retep="http://retep.org/xml/ns/retepTools"
  version="2.0">
  <bindings schemalocation="jabber-client.xsd">
    <bindings node="/xsd:schema/xsd:element[@name='message']/xsd:complexType">
      <bindings node="xsd:attribute[@name='from']">
        <property>
          <baseType>
            <xjc:javaType adapter="uk.org.retep.xmpp.jaxb.adaptor.JIDAdapter"
                          name="uk.org.retep.xmpp.JID" />
          </baseType>
        </property>
      </bindings>
    </bindings>
  </bindings>
</bindings>

Then just create a single adapter implementation. in retepXMPP thats some 60 classes reduced to one, if you generate any javadocs based on the generated sources there’s no more “Adapter” spam – large numbers of apparently duplicated classes in the javadocs etc.

Tip: Just make sure you keep an eye on that namespace – miss it out and XJC will spurt out some unusual error messages.

Update: I’ve disabled comments on this article simply because spammers are the only ones posting comments to it. Peter 27 Feb 2010

Implementing custom types within JAXB

With JAXB it does a pretty good job in mapping between XML and Java Objects. In normal use this is fine but what happens when you need to use a custom data type for an attribute? Well JAXB supports this in binding so this article will describe this.

The originating issue that’s caused this article to be written cropped up at work earlier today where we had an issue with currency values. Over time the various programmers had naturally used a Java double to represent them but problems started to appear when some testers started to use amounts equivalent to €1.2 Billion and things started to fall apart.

Now obviously it’s bad practice to use double for monetary values (we’ve all done it at some point) due to the fact that specific values cannot be represented accurately (eg 39.995 is represented as 39.9949989[1]), it gets worse when dealing with large values. With large amounts you start to loose precision at the milli and penny levels, so adding 1¢ to the amount will fail because the double cannot handle it because of rounding errors.

So the solution is to use BigDecimal as it has arbitrary precision. With JAXB this is not a problem as it supports BigDecimal when defining an attribute with type xsd:decimal – as long as you own the xsd then it’s simply a case of changing it from xsd:double. The problem is when you don’t own the xsd, i.e. it’s an external specification. In this case you cannot change it as you would break other implementations. This is exactly where JAXB bindings[2] come into play.

Changing the object used in an attribute

In this first example taken from my xmpp project we have a series of xsd files and a single xjb file containing the bindings.

From client.xsd which defines the “jabber:client” namespace[3]:

<xs:element name=’message’>

     <xs:complexType>

        <xs:sequence>

          <xs:choice minOccurs=’0′ maxOccurs=’unbounded’>

            <xs:element ref=’subject’/>

            <xs:element ref=’body’/>

            <xs:element ref=’thread’/>

          </xs:choice>

          <xs:any namespace=’##other’ minOccurs=’0′ maxOccurs=’unbounded’/>

          <xs:element ref=’error’ minOccurs=’0’/>

        </xs:sequence>

        <xs:attribute name=’from’ type=’xs:string’ use=’optional’/>

        <xs:attribute name=’id’ type=’xs:NMTOKEN’ use=’optional’/>

        <xs:attribute name=’to’ type=’xs:string’ use=’optional’/>

       <xs:attribute ref=’xml:lang’ use=’optional’/>

     </xs:complexType>

  </xs:element>

Now here the from and to attributes represent not just a string but a Jabber IDentifier JID which within the project is represented by the class uk.org.retep.xmpp.JID. This class ensures it’s both syntactically correct and provides access to the various components within the JID.

For JAXB to use this JID class instead of String we can either add the bindings into the original xsd or into a separate xjb file. I prefer the latter as it keeps the originals untouched so that if someone updates them then the bindings are not lost:

So to add the bindings for JID into the above message element we have the following bindings.xjb file:

<?xml version=’1.0′ encoding=’UTF-8′?>

<bindings

    xmlns=”http://java.sun.com/xml/ns/jaxb”

    xmlns:xsd=”http://www.w3.org/2001/XMLSchema”

    xmlns:retep=”http://retep.org/xml/ns/retepTools”

    version=”2.0”>

   <bindings schemaLocation=”client.xsd” node=”/xsd:schema/xsd:element[@name=’message’]/xsd:complexType/xsd:attribute[@name=’from’]”>

        <property>

            <baseType>

                <javaType name=”uk.org.retep.xmpp.JID” parseMethod=”uk.org.retep.xmpp.JID.parse” printMethod=”uk.org.retep.xmpp.JID.print” />

            </baseType>

        </property>

    </bindings>

    <bindings schemaLocation=”client.xsd” node=”/xsd:schema/xsd:element[@name=’message’]/xsd:complexType/xsd:attribute[@name=’to’]”>

        <property>

            <baseType>

                <javaType name=”uk.org.retep.xmpp.JID” parseMethod=”uk.org.retep.xmpp.JID.parse” printMethod=”uk.org.retep.xmpp.JID.print” />

            </baseType>

        </property>

    </bindings>

</bindings>

Here we define the attributes we want to change using an XPath expression and then the object we want to bind to that attribute. We also provide two methods to parse the attribute from a String to the object and to print it (from the object to a String). In this case those two methods are provided by the JID class itself but they don’t have to be.

Implementing the parse and print methods

The main things to watch out with the parse and print methods is that they have to be public static and they must handle null values. If the attribute is not present or should be a default value then they will be passed nulls:

From JID.java (abridged):

    public static JID parse( final String jid )

    {

        return jid == null ? null : new JID( jid );

    }

    public static String print( final JID jid )

    {

        return jid == null ? null : jid.toString();

    }

Back to the currency issue

Now with the currency issue which initially started off this rambling, there’s one more complication. In the specific DB the currency values are not actually stored as floats but in a custom SQL Type which I’ll call Money to protect the innocent. On the Java side there’s a similar object which uses a BigDecimal for the storage but also handles things like null’s etc. Anyhow in this case all that is required is to add a parse and print method into that class and add the equivalent bindings to the xjb file.

References

  1. Round and Round on the Daily WTF 
  2. Customising JAXB Bindings on sun.com
  3. jabber-client.xsd on xmpp.org