Tag: java

Java 8 running out of file handles on Linux

When I checked one of my websites this morning I discovered it had stopped responding. All it did was sit there then time out with a standard Tomcat error page. So I logged in to check the logs and found that tomcat was complaining about “Too many open files”. Now a few days earlier I had released a new version of the site which moved the serving of static content from Apache to Tomcat so that was my initial though of where the problem was.

This turns out to be true. Where I was reading a text file I was using the new Java 8 Files.line() method which returns a Stream consisting of each line in a file. Now this was nice as instead of writing a block of code to read a file into a string we could reduce it down to:

String page = Files.line( f.getPath() ).
    collect( Collectors.joining( "\n" ) ) );

The problem here is that although nice and concise, it never closes the file. So after a while the Operating System complains that too many files are open and tomcat grinds to a halt.

Now hidden away in the javadocs you’ll find tha Stream actually implements AutoClosable & the reason why is that you can close the stream once you have done with it. Now in 99% of all Streams you don’t need to do anything but if a Stream is operating against some external resource like a File then you are supposed to be closing the stream afterwards.

Now this is where I went wrong, and I suspect a lot of others would also fall foul of this as every example of using Java 8 streams do not show closing them – so who would know that they need to?

So, the correct way of reading a file using streams is this:

try( Stream lines = Files.line( path ) ) {
    // do something with the stream
}

For example, in my case it ended up looking something like this:

String page;
try( Stream lines = Files.line( f.getPath() ) ) {
    page = lines.collect( Collectors.joining( "\n" ) ) );
}

So the lesson here is make certain when using an external resource within a Stream then ensure you close it afterwards.

Installing Java 7 on Debian Squeeze

For all of my servers I use Debian, however that distribution has a few problems, mainly the packages can be a bit behind the cutting edge.

Now this is usually a good thing if you are looking for stability – cutting edge software can have issues, especially from new features etc, so for a live environment you want something thats stable.

However, there does come a time when this can bite back. You either need a feature thats not in the standard repositories or in this case the version is now unsupported.

In Debian Squeeze it has Java 6 – but that was EOL’d a couple of months ago so is no longer supported by Oracle. The current version is Java 7 update 17.

So how do we get Java 7 installed?

Well it’s pretty easy to do, we just need to add another repository into apt and install it.

First the repository:

sudo su -
echo "deb http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list
echo "deb-src http://ppa.launchpad.net/webupd8team/java/ubuntu precise main" | tee -a /etc/apt/sources.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys EEA14886
apt-get update
exit

What that does is to install the ubuntu ppa repository into apt, setup the public keys and then load the package lists.

Next we need to install it:

sudo apt-get install oracle-java7-installer

This will now download Oracle Java 7 and present you with a couple of screens about licensing. Just ok and accept it and it will now install.

That’s it. You now have Java 7 installed – but it’s not the default JDK (if you already had Java 6 installed). If you want it to be the default then there’s just one more thing to do:

sudo apt-get install oracle-java7-set-default

That’s a dummy package but it will make Java 7 the default on that machine. If you want to check then you can check:

peter@titan ~ $ java -version
java version "1.7.0_17"
Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)

Performing remote Maven 3 builds with NetBeans 7.0

This article covers how to get NetBeans to use a Maven installation on a remote server. This has been tested with Maven 3.0.2 but should work on any other version. It may also work for other IDE’s like Eclipse if you can get them to use an alternate maven installation other than the internal one.

Sometimes you may want to run maven on a remote machine from inside the NetBeans IDE. For me this is down to available memory on the local machine is limited enough on large projects that both NetBeans and Maven compete slowing the local machine down as it swaps the IDE in and out of main memory.

Another instance may be to perform builds on a standard OS (Linux in this case) rather than the OS running the IDE (OSX here).

Now this is pretty easy to implement. Here I have the IDE running on OSX and the builds on Linux (Ubuntu 10.10) but if you can get the prerequisites running in theory any OS should work.

Prerequisites

  • ssh access to both machines with public key authentication working
  • sshfs configured for mounting drives (you can use nfs or smb if you wish)
  • maven 3.0.2 installed on both machines – both machines need the same version

Setting up the environment

In this example:

  • sabrina – The local machine running NetBeans. It’s running OS X
  • kira – The remote machine which will run the builds. It’s running Ubuntu 10.10

Shared filesystems

The remote needs to have access to the local project directories for maven to work. As I have all development under a dev directory under home, on the remote machine I simply create a dev directory and then mount the local host from it using sshfs

<br />
peter@kira:~$ mkdir dev<br />
peter@kira:~$ sshfs peter@sabrina:dev dev<br />

If you have a custom settings.xml file, make sure it’s also present on the remote’s .m2 directory, i.e.

<br />
sabrina:~ peter$ scp .m2/settings.xml peter@kira:.m2<br />

Finally NetBeans needs access to the built artifacts, so to do this we mount the remotes .m2/repository locally. I use .m2/remoterepo for this

<br />
sabrina:~ peter$ mkdir .m2/remoterepo<br />
sabrina:~ peter$ sshfs peter@kira:.m2/repository .m2/remoterepo<br />

Maven

Next install your maven installation on both machines. I keep mine in the same place, but thats optional. For the purposes of this article I’m going to use /usr/local/apache/apache-maven-3.0.2

Next on the machine running NetBeans you need to create a duplicate installation /usr/local/apache/apache-maven-3.0.2-remote We need to have a copy for NetBeans to recognise it as a valid environment later on.

Now in this new installation we need to replace the /usr/local/apache/apache-maven-3.0.2-remote/bin/mvn script with the following:

<br />
#!/bin/bash<br />
#<br />
# Enable NetBeans 7.0 beta to run maven on a remote host</p>
<p># Pass the maven args as is<br />
MVN_ARGS=&quot;$@&quot;</p>
<p># The current directory, NetBeans does a cd so this will be the project<br />
# directory.<br />
#<br />
# If you need to translate the path between local and remote machines then<br />
# pass the output of pwd through sed first<br />
REMOTE_PROJECT_DIR=$(pwd)</p>
<p># Execute maven on remote host<br />
REMOTE_USER=peter<br />
REMOTE_HOST=kira</p>
<p># Location of remote maven installation<br />
REMOTE_MAVEN_HOME=/usr/local/apache/apache-maven-3.0.2</p>
<p># JAVA_HOME on remote machine<br />
REMOTE_JAVA_HOME=/usr/lib/jvm/java-6-openjdk/</p>
<p># ======================================<br />
# DO NOT EDIT ANYTHING BEYOND THIS POINT<br />
# ======================================</p>
<p># Remote mvn command<br />
REMOTE_MVN=$REMOTE_MAVEN_HOME/bin/mvn</p>
<p># Build the commands to run remotely<br />
CMD=&quot;cd ${REMOTE_PROJECT_DIR};&quot;<br />
CMD=&quot;${CMD}export JAVA_HOME=${REMOTE_JAVA_HOME};&quot;<br />
CMD=&quot;${CMD}$REMOTE_MVN ${MVN_ARGS}&quot;</p>
<p># Now run the remote build<br />
echo<br />
echo ------------------------------------------------------------------------<br />
echo Executing remote build on $REMOTE_HOST as $REMOTE_USER<br />
echo Remote command: $CMD<br />
echo ------------------------------------------------------------------------<br />
echo<br />
exec ssh ${REMOTE_USER}@${REMOTE_HOST} &quot;${CMD}&quot;<br />

In this script you need to ensure that the settings are correct:

  • REMOTE_USER and REMOTE_HOST point to your remote host,
  • REMOTE_MAVEN_HOME is the location of where maven is installed on the remote host,
  • REMOTE_JAVA_HOME is the location of the JDK on the remote host.

Now you may need to alter the line defining REMOTE_PROJECT_DIR. Shown above it gets set to the directory containing the maven project on the local host. If this differs on the remote then you need to either change to translate this to the location on the remote.

For example I have my NetBeans projects under a directory called dev in my home directory. As this is OS X that becomes /Users/peter/dev however on Linux the path is /home/peter/dev

I get around this by on the remote simply creating a symlink for /Users pointing to /home but you could modify the script to do that for you.

Configuring NetBeans

The last step is getting NetBeans to use the new remote.

Start NetBeans and enter preferences Then the Miscellaneous tab and finally Maven:

  • Under Maven Home select Browse and select the dummy remote installation, in my case /usr/local/apache/apache-maven-3.0.2-remote
  • Under Local Repository select Browse and point it at .m2/remoterepo
  • Select OK

Now if you try to run any build you’ll now see it run on the remote machine and as far as NetBeans is concerned it’s running locally. For example in the NetBeans output window you should see something like the following:

<br />
cd /Users/peter/dev/misc/example; JAVA_HOME=/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home /usr/local/apache/apache-maven-3.0.2-remote/bin/mvn --batch-mode --lax-checksums --fail-fast --no-plugin-updates --batch-mode clean</p>
<p>------------------------------------------------------------------------<br />
Executing remote build on 192.168.30.108 as peter<br />
Remote command: cd /Users/peter/dev/misc/example;export JAVA_HOME=/usr/lib/jvm/java-6-openjdk/;/usr/local/apache/apache-maven-3.0.2/bin/mvn --batch-mode --lax-checksums --fail-fast --no-plugin-updates --batch-mode clean<br />
------------------------------------------------------------------------</p>
<p>Command line option -npu is deprecated and will be removed in future Maven versions.<br />
Scanning for projects...<br />
------------------------------------------------------------------------<br />
Reactor Build Order:<br />

Future invocations

Once setup, whenever you restart either machines you simply need to mount each others drives before starting NetBeans as all the rest should then be setup.

<br />
sabrina:~ peter$ sshfs peter@kira:.m2/repository .m2/remoterepo<br />
peter@kira:~$ sshfs peter@sabrina:dev dev<br />

%d bloggers like this: