Caucho maker of Resin Server | Application Server (Java EE Certified) and Web Server


home / admin/configuration / health / meters
 

Resin Documentation

Aug 2012: Resin outscales C-based web server nginx in AutoBench benchmark
Feb 2012: NetCraft survey says Resin experiencing strong growth in last year and used in number of the Million Busiest Sites.
home company blog wiki docs 
app server web server 
health cloud java ee pro 		 Resin Server | Application Server (Java EE Certified) and Web Server
resin 4.0
admin/configuration
starting
migrating
administration
clustering
command line
rest interface
configuration
database
deployment
health
health checking
meters
report
watchdog
installation
logging
scheduled tasks
security
url rewrite
web server
details
development
changes
reference
 

health meters

health checking
report

Health meters are a simple way to create visually pleasing graphs in /resin-admin.

  1. Configuration
    1. health.xml
    2. Meter names
  2. JMX meters
    1. <health:JmxMeter>
    2. <health:JmxDeltaMeter>
  3. Statistical Analysis
    1. Detecting Anomalies
      1. <health:AnomalyAnalyzer>
    2. Reacting to Anomalies
      1. <health:IfHealthEvent>

Configuration

health.xml

Meters are configured as part of health.xml using CanDI to create and update Java objects. Refer to health checking configuration for a full description of health.xml. Resin 4.0.17 and later includes a full compliemnt of pre-configured JMX meters in health.xml.

Example: importing health.xml into resin.xml
<resin xmlns="http://caucho.com/ns/resin"
       xmlns:resin="urn:java:com.caucho.resin">
  <cluster-default>  
    ...
    <!--
       - Admin services
      -->
    <resin:DeployService/>
    
    <resin:if test="${resin.professional}">
      <resin:AdminServices/>
    </resin:if>

    <!--
       - Configuration for the health monitoring system
      -->
    <resin:if test="${resin.professional}">
      <resin:import path="${__DIR__}/health.xml" optional="true"/>
    </resin:if>
    ...
  </cluster-default>
</resin>

Note: <resin:AdminServices/> (or more precisely just <resin:StatsService/>) is required to support health meters and graphing.

Meter names

Health meters are named using a concatenation of keys separated by pipe (|) characters, loosely organized from least specific to most specific. Since meter statistics are shared between each member in a Resin cluster, Resin will automatically prefix each meter name with the cluster node index to insure the name is unique between cluster members.

The pipe character in the name provides a secondary benefit of helping to enhance the /resin-admin UI by categorizing meters into drill downs. Consider the following example.

Example: meter naming
<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:JmxDeltaMeter>
    <name>JVM|Compilation|Compilation Time</name>
    <object-name>java.lang:type=Compilation</object-name>
    <attribute>TotalCompilationTime</attribute>
  </health:JmxDeltaMeter>

</cluster>

In this example, JVM|Compilation|Compilation Time provides the base of the name. For cluster node index 0, Resin prefixes the name with 00|. /resin-admin will then use the cluster index and first two keys to create drill downs to logically organized meters for display.

Graphs: 00|JVM|Compilation|Compilation Time, Time:1 Hour

JMX meters

Virtually any local numeric JMX MBean attribute can be graphed using JMX meters.

<health:JmxMeter>

child of cluster
javadoc <health:JmxMeter>

Creates a meter that graphs the current value of a numeric JMX MBean attribute.

<health:JmxMeter> Attributes
ATTRIBUTEDESCRIPTIONTYPEDEFAULT
nameThe name of the meter to display in /resin-admin (see meter names)StringN/A
objectNameThe JMX MBean nameStringN/A
attributeThe MBean attribute to sampleStringN/A
Example: <health:JmxMeter> in health.xml
<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:JmxMeter>
    <name>OS|Memory|Physical Memory Free</name>
    <object-name>java.lang:type=OperatingSystem</object-name>
    <attribute>FreePhysicalMemorySize</attribute>
  </health:JmxMeter>

</cluster>

<health:JmxDeltaMeter>

child of cluster
javadoc <health:JmxDeltaMeter>

Creates a meter that graphs the difference between the current and previous values of a numeric JMX MBean attribute.

<health:JmxDeltaMeter> Attributes
ATTRIBUTEDESCRIPTIONTYPEDEFAULT
nameThe name of the meter to display in /resin-admin (see meter names)StringN/A
objectNameThe JMX MBean nameStringN/A
attributeThe MBean attribute to sampleStringN/A
Example: <health:JmxDeltaMeter> in health.xml
<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">

  <health:JmxDeltaMeter>
    <name>JVM|Compilation|Compilation Time</name>
    <object-name>java.lang:type=Compilation</object-name>
    <attribute>TotalCompilationTime</attribute>
  </health:JmxDeltaMeter>

</cluster>

Statistical Analysis

Detecting Anomalies

Meters alone are useful for manual inspection in resin-admin since every meter can be graphed. However Resin provides an extremely useful automatic analysis tool called AnomalyAnalyzer. AnomalyAnalyzer looks at the current meter value, checking for deviations from the average value. So unusual changes like a spike in blocked threads can be detected.

<health:AnomalyAnalyzer>

child of cluster
javadoc <health:AnomalyAnalyzer>

AnomalyAnalyzer examines a meter value, checking for deviations from the average value. So unusual changes like a spike in blocked threads can be detected, logged, and trigger health actions.

<health:AnomalyAnalyzer> Attributes
ATTRIBUTEDESCRIPTIONTYPEDEFAULT
meterName of the meter to analyze (ie. from <health:JmxMeter>)Stringrequired
health-eventA string to use to match using <health:IfHealthEvent>StringNone: when absent no health event will fire
min-samplesMinimum number of samples required to calculate an averageint60 (typically 1 hour of data)
sigma-thresholdThe number of standard deviations for a sample to be considered an anomaly.int5
Example: <health:AnomalyAnalyzer> in health.xml
<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">
         
  <health:JmxMeter>
    <name>JVM|Thread|JVM Blocked Count</name>
    <objectName>resin:type=JvmThreads</objectName>
    <attribute>BlockedCount</attribute>
  </health:JmxMeter>

  <health:AnomalyAnalyzer>
    <meter>JVM|Thread|JVM Blocked Count</meter>
    <health-event>caucho.thread.anomaly.jvm-blocked</health-event>
  </health:AnomalyAnalyzer>

  <health:DumpThreads>
    <health:IfHealthEvent regexp="caucho.thread"/>
    <health:IfNotRecent time="15m"/>
  </health:DumpThreads>
  
</cluster>

Reacting to Anomalies

The <health-event> attribute of AnomalyAnalyzer allows us to tie health actions to a detected anomaly by using the <health:IfHealthEvent> condition.

<health:IfHealthEvent>

child of cluster
javadoc <health:IfHealthEvent>

Causes an action to fire in response to a matching health event. This is usually used in combination with AnomalyAnalyzer with a <health-event> attribute.

<health:IfHealthEvent> Attributes
ATTRIBUTEDESCRIPTIONTYPEDEFAULT
regexpA regular expression the event must match.java.util.regex.Patternrequired
Example: <health:IfHealthEvent> in health.xml
<cluster xmlns="http://caucho.com/ns/resin"
         xmlns:resin="urn:java:com.caucho.resin"
         xmlns:health="urn:java:com.caucho.health"
         xmlns:ee="urn:java:ee">
         
  <health:JmxMeter>
    <name>JVM|Thread|JVM Blocked Count</name>
    <objectName>resin:type=JvmThreads</objectName>
    <attribute>BlockedCount</attribute>
  </health:JmxMeter>

  <health:AnomalyAnalyzer>
    <meter>JVM|Thread|JVM Blocked Count</meter>
    <health-event>caucho.thread.anomaly.jvm-blocked</health-event>
  </health:AnomalyAnalyzer>

  <health:DumpThreads>
    <health:IfHealthEvent regexp="caucho.thread"/>
    <health:IfNotRecent time="15m"/>
  </health:DumpThreads>
  
</cluster>
health checking
health
report
Copyright © 1998-2012 Caucho Technology, Inc. All rights reserved. Resin ® is a registered trademark. Quercustm, and Hessiantm are trademarks of Caucho Technology.