Java 8 running out of file handles on Linux

When I checked one of my websites this morning I discovered it had stopped responding. All it did was sit there then time out with a standard Tomcat error page. So I logged in to check the logs and found that tomcat was complaining about “Too many open files”. Now a few days earlier I had released a new version of the site which moved the serving of static content from Apache to Tomcat so that was my initial though of where the problem was.

This turns out to be true. Where I was reading a text file I was using the new Java 8 Files.line() method which returns a Stream consisting of each line in a file. Now this was nice as instead of writing a block of code to read a file into a string we could reduce it down to:

String page = Files.line( f.getPath() ).
    collect( Collectors.joining( "\n" ) ) );

The problem here is that although nice and concise, it never closes the file. So after a while the Operating System complains that too many files are open and tomcat grinds to a halt.

Now hidden away in the javadocs you’ll find tha Stream actually implements AutoClosable & the reason why is that you can close the stream once you have done with it. Now in 99% of all Streams you don’t need to do anything but if a Stream is operating against some external resource like a File then you are supposed to be closing the stream afterwards.

Now this is where I went wrong, and I suspect a lot of others would also fall foul of this as every example of using Java 8 streams do not show closing them – so who would know that they need to?

So, the correct way of reading a file using streams is this:

try( Stream lines = Files.line( path ) ) {
    // do something with the stream
}

For example, in my case it ended up looking something like this:

String page;
try( Stream lines = Files.line( f.getPath() ) ) {
    page = lines.collect( Collectors.joining( "\n" ) ) );
}

So the lesson here is make certain when using an external resource within a Stream then ensure you close it afterwards.

Installing Java 7 on Debian Squeeze

For all of my servers I use Debian, however that distribution has a few problems, mainly the packages can be a bit behind the cutting edge.

Now this is usually a good thing if you are looking for stability – cutting edge software can have issues, especially from new features etc, so for a live environment you want something thats stable.

However, there does come a time when this can bite back. You either need a feature thats not in the standard repositories or in this case the version is now unsupported.

In Debian Squeeze it has Java 6 – but that was EOL’d a couple of months ago so is no longer supported by Oracle. The current version is Java 7 update 17.

So how do we get Java 7 installed?

Well it’s pretty easy to do, we just need to add another repository into apt and install it.

First the repository:

sudo su -
echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886
apt-get update
exit

What that does is to install the ubuntu ppa repository into apt, setup the public keys and then load the package lists.

Next we need to install it:

sudo apt-get install oracle-java7-installer

This will now download Oracle Java 7 and present you with a couple of screens about licensing. Just ok and accept it and it will now install.

That’s it. You now have Java 7 installed – but it’s not the default JDK (if you already had Java 6 installed). If you want it to be the default then there’s just one more thing to do:

sudo apt-get install oracle-java7-set-default

That’s a dummy package but it will make Java 7 the default on that machine. If you want to check then you can check:

peter@titan ~ $ java -version
java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

Performing remote Maven 3 builds with NetBeans 7.0

This article covers how to get NetBeans to use a Maven installation on a remote server. This has been tested with Maven 3.0.2 but should work on any other version. It may also work for other IDE’s like Eclipse if you can get them to use an alternate maven installation other than the internal one.

Sometimes you may want to run maven on a remote machine from inside the NetBeans IDE. For me this is down to available memory on the local machine is limited enough on large projects that both NetBeans and Maven compete slowing the local machine down as it swaps the IDE in and out of main memory.

Another instance may be to perform builds on a standard OS (Linux in this case) rather than the OS running the IDE (OSX here).

Now this is pretty easy to implement. Here I have the IDE running on OSX and the builds on Linux (Ubuntu 10.10) but if you can get the prerequisites running in theory any OS should work.

Prerequisites

  • ssh access to both machines with public key authentication working
  • sshfs configured for mounting drives (you can use nfs or smb if you wish)
  • maven 3.0.2 installed on both machines – both machines need the same version

Setting up the environment

In this example:

  • sabrina – The local machine running NetBeans. It’s running OS X
  • kira – The remote machine which will run the builds. It’s running Ubuntu 10.10

Shared filesystems

The remote needs to have access to the local project directories for maven to work. As I have all development under a dev directory under home, on the remote machine I simply create a dev directory and then mount the local host from it using sshfs

<br />
peter@kira:~$ mkdir dev<br />
peter@kira:~$ sshfs peter@sabrina:dev dev<br />

If you have a custom settings.xml file, make sure it’s also present on the remote’s .m2 directory, i.e.

<br />
sabrina:~ peter$ scp .m2/settings.xml peter@kira:.m2<br />

Finally NetBeans needs access to the built artifacts, so to do this we mount the remotes .m2/repository locally. I use .m2/remoterepo for this

<br />
sabrina:~ peter$ mkdir .m2/remoterepo<br />
sabrina:~ peter$ sshfs peter@kira:.m2/repository .m2/remoterepo<br />

Maven

Next install your maven installation on both machines. I keep mine in the same place, but thats optional. For the purposes of this article I’m going to use /usr/local/apache/apache-maven-3.0.2

Next on the machine running NetBeans you need to create a duplicate installation /usr/local/apache/apache-maven-3.0.2-remote We need to have a copy for NetBeans to recognise it as a valid environment later on.

Now in this new installation we need to replace the /usr/local/apache/apache-maven-3.0.2-remote/bin/mvn script with the following:

<br />
#!/bin/bash<br />
#<br />
# Enable NetBeans 7.0 beta to run maven on a remote host</p>
<p># Pass the maven args as is<br />
MVN_ARGS=&quot;$@&quot;</p>
<p># The current directory, NetBeans does a cd so this will be the project<br />
# directory.<br />
#<br />
# If you need to translate the path between local and remote machines then<br />
# pass the output of pwd through sed first<br />
REMOTE_PROJECT_DIR=$(pwd)</p>
<p># Execute maven on remote host<br />
REMOTE_USER=peter<br />
REMOTE_HOST=kira</p>
<p># Location of remote maven installation<br />
REMOTE_MAVEN_HOME=/usr/local/apache/apache-maven-3.0.2</p>
<p># JAVA_HOME on remote machine<br />
REMOTE_JAVA_HOME=/usr/lib/jvm/java-6-openjdk/</p>
<p># ======================================<br />
# DO NOT EDIT ANYTHING BEYOND THIS POINT<br />
# ======================================</p>
<p># Remote mvn command<br />
REMOTE_MVN=$REMOTE_MAVEN_HOME/bin/mvn</p>
<p># Build the commands to run remotely<br />
CMD=&quot;cd ${REMOTE_PROJECT_DIR};&quot;<br />
CMD=&quot;${CMD}export JAVA_HOME=${REMOTE_JAVA_HOME};&quot;<br />
CMD=&quot;${CMD}$REMOTE_MVN ${MVN_ARGS}&quot;</p>
<p># Now run the remote build<br />
echo<br />
echo ------------------------------------------------------------------------<br />
echo Executing remote build on $REMOTE_HOST as $REMOTE_USER<br />
echo Remote command: $CMD<br />
echo ------------------------------------------------------------------------<br />
echo<br />
exec ssh ${REMOTE_USER}@${REMOTE_HOST} &quot;${CMD}&quot;<br />

In this script you need to ensure that the settings are correct:

  • REMOTE_USER and REMOTE_HOST point to your remote host,
  • REMOTE_MAVEN_HOME is the location of where maven is installed on the remote host,
  • REMOTE_JAVA_HOME is the location of the JDK on the remote host.

Now you may need to alter the line defining REMOTE_PROJECT_DIR. Shown above it gets set to the directory containing the maven project on the local host. If this differs on the remote then you need to either change to translate this to the location on the remote.

For example I have my NetBeans projects under a directory called dev in my home directory. As this is OS X that becomes /Users/peter/dev however on Linux the path is /home/peter/dev

I get around this by on the remote simply creating a symlink for /Users pointing to /home but you could modify the script to do that for you.

Configuring NetBeans

The last step is getting NetBeans to use the new remote.

Start NetBeans and enter preferences Then the Miscellaneous tab and finally Maven:

  • Under Maven Home select Browse and select the dummy remote installation, in my case /usr/local/apache/apache-maven-3.0.2-remote
  • Under Local Repository select Browse and point it at .m2/remoterepo
  • Select OK

Now if you try to run any build you’ll now see it run on the remote machine and as far as NetBeans is concerned it’s running locally. For example in the NetBeans output window you should see something like the following:

<br />
cd /Users/peter/dev/misc/example; JAVA_HOME=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home /usr/local/apache/apache-maven-3.0.2-remote/bin/mvn --batch-mode --lax-checksums --fail-fast --no-plugin-updates --batch-mode clean</p>
<p>------------------------------------------------------------------------<br />
Executing remote build on 192.168.30.108 as peter<br />
Remote command: cd /Users/peter/dev/misc/example;export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/;/usr/local/apache/apache-maven-3.0.2/bin/mvn --batch-mode --lax-checksums --fail-fast --no-plugin-updates --batch-mode clean<br />
------------------------------------------------------------------------</p>
<p>Command line option -npu is deprecated and will be removed in future Maven versions.<br />
Scanning for projects...<br />
------------------------------------------------------------------------<br />
Reactor Build Order:<br />

Future invocations

Once setup, whenever you restart either machines you simply need to mount each others drives before starting NetBeans as all the rest should then be setup.

<br />
sabrina:~ peter$ sshfs peter@kira:.m2/repository .m2/remoterepo<br />
peter@kira:~$ sshfs peter@sabrina:dev dev<br />

Compiling Java 7.0 on OS X

Since the announcement by Apple last week about deprecating Java on OS X, there’s been a few people wanting to know how to compile OpenJDK on the Mac.

Although I’ve not done this for JDK 6, this article cover’s how to compile and use the current development version of JDK 7.0 on OS X.

First a few notes:

  1. This only enables Java 7 within an X environment, native UI’s are not supported – one of the main parts of the Apple JVM
  2. When I tested this by running Netbeans 7.0 M2 within X the menus were a bit screwey – try it you’ll see what I mean
  3. These instructions are for 10.5.8 but should work for 10.6.x
  4. This is for Intel processors only

So as a word of warning: Don’t expect this to either work, or work well – and don’t use this in production – JDK7 isn’t due out for another 12 months or so…

Ok, first we need a bootstrap JDK6 environment. This is needed to do some of the initial java compilation during the build. For this the apple JVM can’t be used so we need to download and install the i386 Soylate binaries JDK – don’t get the amd64 version, get the i386 one…

Once you have it downloaded, copy it to /usr/local/soylatte16-i386-1.0.3 and test it:

sabrina:~ peter$ /usr/local/soylatte16-i386-1.0.3/bin/java -version
java version "1.6.0_03-p3"
Java(TM) SE Runtime Environment (build 1.6.0_03-p3-landonf_19_aug_2008_14_55-b00)
Java HotSpot(TM) Server VM (build 1.6.0_03-p3-landonf_19_aug_2008_14_55-b00, mixed mode)

Next create a blank directory under which we will build everything. I’m using ~/dev/ojdk but you could use any directory. Under this we need to create a couple of directories and some symlinks:

sabrina:~ peter$ mkdir -p dev/ojdk
sabrina:~ peter$ cd dev/ojdk
sabrina:~ peter$ mkdir -p bin ALT_COMPILER_PATH
sabrina:~ peter$ cd ALT_COMPILER_PATH
sabrina:~ peter$ ln -s /usr/bin .SOURCE
sabrina:~ peter$ ln -s .SOURCE/g++-4.0 g++
sabrina:~ peter$ ln -s .SOURCE/gcc-4.0 gcc
sabrina:~ peter$ cd ../bin

Now in the bin directory you need to create two scripts. Fortunately these are readily available from http://gist.github.com/617451 – specifically update.sh and update-usr-local.sh. Copy these two files into the bin directory and ensure they are executable.

Now open update.sh in your favourite editor and find the line with ALT_COMPILER_PATH in it. Change it to hold the full path to the ALT_COMPILER_PATH directory defined above. In my case this looks like:

ALT_COMPILER_PATH=/Users/peter/dev/ojdk/ALT_COMPILER_PATH/ \

Next we need to checkout a copy of the source:

sabrina:~ peter$ cd ~/dev/ojdk
sabrina:~ peter$ hg fclone http://hg.openjdk.java.net/bsd-port/bsd-port bsd

We should now be setup. The last step is to run a build. This can be done at any time. It will check for any updates, clear down and then run a full build:

sabrina:~ peter$ cd ~/dev/ojdk/bsd
sabrina:~ peter$ source ../bin/update.sh

If all goes well, after about 20 minutes you should see something like the following at the end of the build:

testing build: ./build/bsd-amd64/j2sdk-image/bin/java -version

openjdk version "1.7.0-internal"
OpenJDK Runtime Environment (build 1.7.0-internal-peter_2010_10_25_11_19-b00)
OpenJDK 64-Bit Server VM (build 19.0-b05, mixed mode)

If the build succedes the last step is to install it under /usr/local/java-1.7.0:

sabrina:~ peter$ cd ~/dev/ojdk/bsd
sabrina:~ peter$ source ../bin/update-usr-local.sh

To use Java 7, you need to simply point the app to the installed application, usually by setting JAVA_HOME=/usr/local/java-1.7.0 and running $JAVA_HOME/bin/java…

sabrina:bin peter$ /usr/local/java-1.7.0/bin/java -version
openjdk version "1.7.0-internal"
OpenJDK Runtime Environment (build 1.7.0-internal-peter_2010_10_25_11_19-b00)
OpenJDK 64-Bit Server VM (build 19.0-b05, mixed mode)