Editing Xymon custom graph config

===Prerequisites===
Make sure the client has xymon-client installed:
<pre>
root@merkli:/var/lib/xymon# dpkg -l "*xymon*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name             Version          Description
+++-================-================-================================================
un  xymon            <none>           (no description available)
ii  xymon-client     4.3.0~beta2.dfsg client for the Xymon network monitor
un  xymon-plugins    <none>           (no description available)
</pre>

I found that it was necessary to run 
<pre>
dpkg-reconfigure xymon-client
</pre>
in order to tell the client how to contact the server. (/etc/default/xymon-client)

Another important configuration point:  CLIENTHOSTNAME in /etc/default/xymon-client on the client must match the name for the client in /etc/xymon/hosts.cfg on the server.

===collect the data with a script===
hobbit client:/usr/lib/xymon/client/ext/cputemp:
<source lang="bash">
#!/bin/sh

#/usr/bin/sensors -f | grep "CPU Temp" | awk '{print $1 $2 $3}' |
#	awk {'sub("\+", ""); sub("°F", ""); print }' > /tmp/cputemp.txt
#/usr/bin/sensors -f | grep -i temp | grep -v k8temp| 
# awk '{ sub("°F", ""); sub("\+", ""); sub("/", ""); sub(" +", ""); sub("\(.*$", ""); print}' 
# > /tmp/cputemp.txt

RAWTEMP=`nvidia-smi -a | grep Temperature | sed -e "s/.*.://g" -e "s/C//g"`
FGPUTEMP=$((${RAWTEMP:-0}*9/5+32))
echo "temp1:$FGPUTEMP" > /tmp/cputemp.txt

/usr/bin/sensors -f | grep Temp | grep -v k8temp | awk '{sub(".F.*$",
""); sub(" +",""); sub("+",""); print}' >> /tmp/cputemp.txt

RESULT=`grep Core0Temp /tmp/cputemp.txt | awk '{ FS=":";  print int($2) }'`;
COLOR=green
if test "$RESULT" -gt 120
then
	COLOR=red
fi
if test "$RESULT" -lt 70
then
	COLOR=red
fi

$BB $BBDISP "status $MACHINE.cputemp $COLOR `date`

`cat /tmp/cputemp.txt`
"

exit 0
</source>
* you can run the script from the command line to test it
This is the expected output:
<pre>
root@merkli:/usr/lib/xymon/client/ext# ./cputemp 
./cputemp: 24: status .cputemp green Thu Jul 21 10:23:23 PDT 2011

temp1:95.0
Core0Temp: 
Core1Temp: 
: not found
</pre>
If you run the script as root, make sure you leave the permissions on /tmp/cputemp.txt such that it is writable by user hobbit.



===launch the script===
running the script on the xymon client:/etc/xymon/tasks.d/cputemp:
<pre>
[cputemp]
    ENVFILE /usr/lib/xymon/client/etc/xymonclient.cfg
    CMD /usr/lib/xymon/client/ext/cputemp
    INTERVAL 5m
</pre>
<strike>running the script on the hobbit client:/usr/lib/xymon/client/etc/clientlaunch.cfg
or better, put into client:/var/run/xymon/clientlaunch-include.cfg</strike>

* wait for five minutes and then you should see the data in the web interface, but not the graph

This particular script creates a temp file, so you can look at the timestamp on that to see if it is running:
<source lang=bash>
root@merkli:/usr/lib/xymon/client/ext# ls -l /tmp/cputemp.txt 
-rw-r--r-- 1 hobbit hobbit 35 2011-07-21 10:26 /tmp/cputemp.txt
</source>
<pre>
root@merkli:/usr/lib/xymon/client/ext# cat /tmp/cputemp.txt 
temp1:95.0
Core0Temp: 
Core1Temp: 
</pre>

===collect the data in an RRD===
From here on out, the configuration should all be on the server and not the client, since the RRD is stored on the server and the graphs are generated from that.

But what gets transferred between the client and server?  The script output or the rrd file?  My guess is that just the script output is transferred and the rrd is first constructed on the server.

server:/etc/xymon/xymonserver.cfg
<pre>
TEST2RRD="cpu=la,disk,inode,qtree,memory,$PINGCOLUMN=tcp,http=tcp,dns=tcp,dig=tcp,time=ntpstat,
vmstat,iostat,netstat,temperature,apache,bind,sendmail,mailq,nmailq=mailq,socks,bea,iishealth,
citrix,bbgen,bbtest,bbproxy,hobbitd,files,procs=processes,ports,clock,lines,ops,stats,cifs,
JVM,JMS,HitCache,Session,JDBCConn,ExecQueue,JTA,TblSpace,RollBack,MemReq,InvObj,snapmirr,
snaplist,snapshot,if_load=devmon,temp=devmon,paging,mdc,mdchitpct,cics,dsa,getvis,maxuser,
nparts,cputemp=ncv,heater=ncv"
</pre>

<pre>
NCV_cputemp="temp1:GAUGE,Core0Temp:GAUGE,Core1Temp:GAUGE"
</pre>

*restart xymon after making these changes

* you can verify the RRD:
server:/var/lib/xymon/rrd/localhost/cputemp.rrd
<pre>
root@weasel:/var/lib/xymon/rrd/localhost# rrdtool info ./cputemp.rrd 
filename = "./cputemp.rrd"
rrd_version = "0003"
step = 300
last_update = 1311266629
header_size = 2320
ds[temp1].index = 0
ds[temp1].type = "GAUGE"
ds[temp1].minimal_heartbeat = 600
ds[temp1].min = NaN
ds[temp1].max = NaN
ds[temp1].last_ds = "131"
ds[temp1].value = 2.9999000000e+04
ds[temp1].unknown_sec = 0

root@weasel:/var/lib/xymon/rrd/localhost# ls -lt cputemp.rrd 
-rw-r--r-- 1 hobbit hobbit 57616 2011-07-21 09:43 cputemp.rrd
</pre>

The timestamp on the file should be less than 5 minutes old and the data inside should correspond to the output of the script.

===create a graph definition===
server:/etc/xymon/graphs.cfg:
<syntaxhighlight lang=ini>
[cputemp]
    TITLE CPU Temperature
    YAXIS Degrees Fahrenheit
    DEF:temp1=cputemp.rrd:temp1:AVERAGE
    DEF:Core0Temp=cputemp.rrd:Core0Temp:AVERAGE
    DEF:Core1Temp=cputemp.rrd:Core1Temp:AVERAGE
    LINE2:temp1#@COLOR@:temp1
    LINE2:Core0Temp#@COLOR@:Core0Temp
    LINE2:Core1Temp#@COLOR@:Core1Temp\n
</syntaxhighlight>

=== troubleshooting===
If a graph is not appearing, you can view source on the page with the missing graph and click on the IMG link that it is trying to display.  But often that won't reveal anything useful.

[[Image:hobbit-graph-error.png]]


It is difficult to manually run /usr/lib/xymon/cgi-bin/hobbitgraph.sh with the parameters from the web page to see what the error is. hobbitgraph.sh is a shell wrapper around a binary called /usr/lib/xymon/server/bin/hobbitgraph.cgi.

<pre>
#!/bin/sh

# This is the Hobbit CGI script interface to hobbitgraph.cgi
#
# Install this script in your webservers' cgi-bin directory

. /usr/lib/xymon/server/etc/hobbitcgi.cfg
 exec /usr/lib/xymon/server/bin/hobbitgraph.cgi $CGI_HOBBITGRAPH_OPTS
</pre>

Failing URL:
 "/hobbit-cgi/hobbitgraph.sh?host=merkli&amp;service=ncv:cputemp&amp;graph_width=576&amp;graph_height=120&amp;disp=merkli&amp;nostale&amp;color=green&amp;graph_start=1311105181&amp;graph_end=1311277981&amp;graph=hourly&amp;action=view"

Looking at the environment variables as hobbitgraph.sh runs, I see this on a working query:

 REQUEST_URI=/hobbit-cgi/hobbitgraph.sh?host=localhost&service=ncv:cputemp&graph_width=576&graph_height=120&disp=localhost&nostale& 
 color=green&graph_start=1311116740&graph_end=1311289540&graph=hourly&action=view

 QUERY_STRING=host=localhost&service=ncv:cputemp&graph_width=576&graph_height=120&disp=localhost&nostale&color=green&
 graph_start=1311116862&graph_end=1311289662&graph=hourly&action=view

And this for a failing query:

 REQUEST_URI=/hobbit-cgi/hobbitgraph.sh?host=merkli&service=ncv:cputemp&graph_width=576&graph_height=120&disp=merkli&nostale&color=green&
 graph_start=1311116888&graph_end=1311289688&graph=hourly&action=view

 QUERY_STRING=host=merkli&service=ncv:cputemp&graph_width=576&graph_height=120&disp=merkli&nostale&color=green&graph_start=1311117131&
 graph_end=1311289931&graph=hourly&action=view

====manually generating a graph====
In order to force a graphing error into view, you can manually attempt to generate the graph outside of hobbit, but using the same configuration and rrd that hobbit is using.

* locate the rrd
 /var/lib/xymon/rrd/merkli/cputemp.rrd

* locate the existing graph config
 /usr/lib/xymon/server/etc/hobbitgraph.cfg
<pre>
[cputemp]
    TITLE CPU Temperature
    YAXIS Degrees Fahrenheit
    DEF:temp1=cputemp.rrd:temp1:AVERAGE
    DEF:Core0Temp=cputemp.rrd:Core0Temp:AVERAGE
    DEF:Core1Temp=cputemp.rrd:Core1Temp:AVERAGE
    LINE2:temp1#@COLOR@:temp1
    LINE2:Core0Temp#@COLOR@:Core0Temp
    LINE2:Core1Temp#@COLOR@:Core1Temp\n
</pre>
 
* cook up a corresponding graph command

<pre>
 $period = `date --date="7 days ago" +%s`;
 cp /var/lib/xymon/rrd/merkli/cputemp.rrd /tmp/cputemp.rrd
 rrdtool graph /tmp/output.png --width 300 --start $period -v "Degrees Fahrenheit"  \
 DEF:temp1=/tmp/cputemp.rrd:temp1:AVERAGE \
 DEF:Core0Temp=/tmp/cputemp.rrd:Core0Temp:AVERAGE \
 DEF:Core1Temp=/tmp/cputemp.rrd:Core1Temp:AVERAGE \
 LINE2:temp1#FF0000:temp1 \
 LINE2:Core0Temp#00FF00:Core0Temp \
 LINE2:Core1Temp#0000FF:Core1Temp 
</pre>

* the errors should be much more helpful now
<pre>
root@weasel:/var/www/temp# rrdtool graph /tmp/output.png --width 300 --start 1311615323 -v "Degrees Fahrenheit"   DEF:temp1=/tmp
/cputemp.rrd:temp1:AVERAGE  DEF:Core0Temp=/tmp/cputemp.rrd:Core0Temp:AVERAGE  DEF:Core1Temp=/tmp/cputemp.rrd:Core1Temp:AVERAGE  
LINE2:temp1#FF0000:temp1  LINE2:Core0Temp#00FF00:Core0Temp
ERROR: No DS called 'Core0Temp' in '/tmp/cputemp.rrd'
</pre>

====a graph appears, but there is no data====
The script seems to be generating correct output, the rrd file is accumulating data, and the graph is appears on the custom graph page, but there is no data on the graph.

server:/var/log/xymon/rrd-status.log contains this:
<pre>
2011-08-01 11:09:59 RRD error updating /var/lib/xymon/rrd/merkli/cputemp.rrd from 10.0.0.14: 
/var/lib/xymon/rrd/merkli/cputemp.rrd: found extra data on update argument: 89:89
</pre>

rrdtool info tells me that the data from the script is unparseable:
<pre>
root@weasel:/var/lib/xymon/rrd/merkli# pwd
/var/lib/hobbit/rrd/merkli
root@weasel:/var/lib/xymon/rrd/merkli# rrdtool info cputemp.rrd
filename = "cputemp.rrd"
rrd_version = "0003"
step = 300
last_update = 1312221286
header_size = 2320
ds[temp1].index = 0
ds[temp1].type = "GAUGE"
ds[temp1].minimal_heartbeat = 600
ds[temp1].min = NaN
ds[temp1].max = NaN
ds[temp1].last_ds = "U"
ds[temp1].value = 0.0000000000e+00
ds[temp1].unknown_sec = 286
</pre>
The entry of interest is ds[temp1].last_ds = "U"
That should be a Fahrenheit temperature instead of U.

The script generates a floating point temperature.  In case that is the problem, I rewrote the script to generate an integer temperature.

Sure enough, after making sure that the script was generating an integer and then restarting hobbit, it fixed the problem.

Another way the number could be unparseable is if there is a leading or trailing space.  This shows up in the RRD as NaN (not a number).  Fix the script to remove all spaces.

====removing enormous spikes from graphs====

By the way, this problem is really a symptom that the rrd config is missing a min/max value defined for a counter datasource.  So when the counter resets, rrd is fooled into thinking that there has been a counter wrap-around.

It is fairly easy to confuse some counters by wrapping around.  This can be caused by rebooting or otherwise reseting the network counters.  This can leave artifacts in the graphs that will cause auto-scaling to flatten all the data that you want to see.

* dump the rrd to xml
** identify the rrd file.  Mine is in /var/lib/xymon/rrd/localhost/bandwidth.rrd.  I have a false 30 petabyte spike around Oct 29.
 root@weasel:/var/lib/xymon/rrd/localhost# rrdtool dump bandwidth.rrd > bandwidth.xml

* edit the xml
I look for lines that are more than 10^4 or so.  It is handy to search for "e+1".  Replace the exponential notation of e+15 with e+04.  If you delete the large data points it will mess up the resolution of my graph.  Deleting entries makes the graph blocky and low-res.

* restore the xml to rrd
 root@weasel:/var/lib/xymon/rrd/localhost# rrdtool restore -f bandwidth.xml bandwidth.rrd

You'll lose the data that was captured between the first and last steps, so try not to dilly dally.

==== preventing spikes in graphs ====

This post seems to be a good summary of what is going on:
https://lists.oetiker.ch/pipermail/rrd-users/2000-December/001871.html
Also this:
https://collectd.org/wiki/index.php/Data_source

* determine what kind of data source is being used:

<pre>
root@weasel:/var/lib/xymon/rrd/localhost# rrdtool info bweth0.rrd
filename = "bweth0.rrd"
rrd_version = "0003"
step = 300
last_update = 1435941912
header_size = 1760
ds[bytesin].index = 0
ds[bytesin].type = "COUNTER"
...
</pre>

So this is a COUNTER and I would like it to be DERIVE instead.
How does xymon set the datasource type? /etc/xymon/xymonserver.cfg

 root@weasel:/etc/xymon# grep bweth0 xymonserver.cfg
 ...
 NCV_bweth0="bytesin:DERIVE,bytesout:DERIVE"

Look at that, I've already started this process but haven't finished it.
I guess I was hoping that xymon would take care of the conversion for me but it doesn't.

* dump with rrdtool
* edit the type in the xml, and set min to 0
* restore with rrdtool restore
* verify with rrdtool info
<pre>
root@weasel:/var/lib/xymon/rrd/localhost# rrdtool info bweth0.rrd
filename = "bweth0.rrd"
rrd_version = "0003"
step = 300
last_update = 1435941912
header_size = 1760
ds[bytesin].index = 0
ds[bytesin].type = "DERIVE"
ds[bytesin].minimal_heartbeat = 600
ds[bytesin].min = 0.0000000000e+00
...
</pre>