One of the systems that can be used for monitoring trends on a system is munin. It turns out that adding a graph to munin is reasonably simple.
There is a good guide on the munin site.
So - I need a plugin that will return the number of open files.
lsof -u glassfish | wc -l
This is a rough guide - it includes files, pipes, tcp connections etc etc - so it may give a slightly larger count - but the aim here is to see trends - are we stable or do we have a file leak?
So - here's the plugin code I ended up using:
#!/bin/sh case $1 in config) cat <<'EOM' graph_title Open file count for glassfish graph_vlabel filecount filecount.label filecount graph_args --base 1000 -l 0 graph_category Processes EOM exit 0;; esac echo -n "filecount.value " lsof -u glassfish | wc -l</code></pre>
This got added as /etc/munin/plugins/glassfish_filecount. I also had to specify in /etc/munin/plugin-conf.d/munin-node (paths are from a debian install):
[glassfish_filecount] user root
to avoid errors from lsof.
The output shows that the trend is stable, that the limit increase should be enough and that there isn't a file leak.
It also shows that the lsof count includes more than the ulimit enforces (since lsof contains many different types) - since we see the graph going higher than 2048 which is the ulimit for open files. However - since it is showing the trend I needed to see I decided that it would do.