Changes between Initial Version and Version 1 of SystemMonitorConfiguration

Timestamp:

11/24/09 15:38:04 (16 years ago)

Author:

martin

Comment:

--

SystemMonitorConfiguration

@@ +1 @@
+= System Monitoring =
+Our system monitoring is based on configuring the monitor tool. The current draft of this configuration is below.
+
+The initial configuration is based on just a few states:
+
+START::
+  the standard initial state, used to perform priming reads of status and to setup operating defaults
+UNKNOWN::
+  the standard state used when no other state fits with the current conditions
+startingNetwork::
+  we do not have an ethernet address
+running::
+  all is ok, our process pid file exists and contains a process id and the ethernet has an IP number in the range we expect
+FXOhang::
+  we have detected a hung state from our FXO modules.  In this state, we attempt to reset the devices.
+dead::
+  used when we are in the UNKNOWN state for too long
+
+{{{
+#   Private and confidential.
+#
+#   Copyright Jazmin Communications Pty Ltd, 2009
+#   All rights reserved.
+#
+#   Not for external release.
+#
+# A monitor configuration for the ip04 system
+#
+# 
+
+ENTER START {
+        LOG "starting monitor"
+        SET CYCLE = 2  # monitor aggressively while booting
+        SET enetStatus = RUN "/sbin/ifconfig en1"
+}
+
+STATE startingNetwork {
+        enetStatus NOT ~ /inet[ ]+192.168/
+}
+
+ENTER startingNetwork {
+        SET CYCLE = 2  # monitor aggressively while waiting for the network to come back
+}
+
+POLL startingNetwork {
+        SET enetStatus = RUN "/sbin/ifconfig en1"
+}
+
+# we define operational states based on various conditional tests
+# if all conditions pass, the monitor enters the given state and runs 
+# our 'ENTER' method.
+
+# the following reads a .pid file and verifies that it contains a number. 
+# note that we demonstrate the 'COLLECT' verb here. COLLECT can be used
+# to collect data into a variable for later tests. This is largely for
+# optimisation.
+#       The other way to specify the condition for this state is simply:
+#
+#   FILE 'myproc.pid' ~ /[0-9]+/
+#
+STATE running {
+        enetStatus ~ /inet[ ]+192.168/;
+        COLLECT myprocpid FROM FILE '"/tmp/myproc.pid"';
+        myprocpid ~ /[0-9]+/ # note: current bug, the file name cannot contain a path
+}
+
+# if a pid file was found, we log the fact.  Every 'CYCLE' seconds, we will test that the
+# system is still running.
+ENTER running {
+        LOG "running ok"
+        SET CYCLE = 5  # less frequent monitoring while things are running nicely
+}
+
+POLL running {
+        SET fxostatus = RUN "'/Users/martin/Desktop/current/Jazmin Communications/check_installed_FXO_status'"
+        SET enetStatus = RUN "/sbin/ifconfig en1"
+}
+
+# if no states match, the monitor automatically enters the state 'UNKNOWN' 
+# we can catch this by setting up an enter method:
+
+ENTER UNKNOWN {
+        LOG "unknown system state"
+}
+
+STATE FXOhang {
+        fxostatus ~ /0xff/
+}
+
+ENTER FXOhang {
+        LOG "detected hang in FXO module"
+        SET fxostatus = RUN "'/Users/martin/Desktop/current/Jazmin Communications/reset_FXO'"
+}
+
+# if we do not enter the 'running' state, our monitor will enter the UNKNOWN
+# state because we have not setup any conditions for any other states.
+# If we have been in the UNKNOWN state for 10 seconds or more, we give up and 
+# decide the system is dead.
+
+STATE dead {
+        CURRENT ~ /UNKNOWN/;
+        TIMER >= 4
+}
+
+# in this sample, if the system is dead, we simply log the fact and exit.
+
+ENTER dead {
+        LOG "program is not running, restarting monitor";
+        SPAWN "/bin/date >>/tmp/dates"
+}
+
+}}}