SCOM, SNMP and TRAPS or The Good, the Bad and the Ugly : Part 1

Recently I was encouraged to find a way to monitor many of our appliances with our shiny new deployment of  Microsoft System Center Operations Manager 2007 R2. Up until this point we had not used SCOM for SNMP monitoring little did I know the adventure I was about to embark on.

Requirements:

Log (SCOM Rule) various values from different SNMP appliances

Alert (SCOM Monitor) on various values from different SNMP appliances

Basic availability checks for all appliances

Receive and alert on SNMP TRAPS from all appliances

The Process : (no point in trying to skip to the solution)

The first thing I found was a lot of people in various blogs with a warning “Abandon hope all ye who enter here”. Usually I would save my ranting comments to the end but perhaps it’s best to give you a quick glimpse now. SCOM 2007 R2 is not exactly the pinacle of SNMP monitoring, but I will get into that more later 😉

In part 1 we will find a way to discover and differentiate between the different kinds of SNMP appliances so that we can target various monitors and rules at specific devices. No point is weighing down SCOM with monitoring all sorts of thinsg that don’t exist on a given device right?

Step 1: Install required tools

Here I have to start with a major and well deserved tip of the hat to 2 individuals without whom my adventures into SCOM and SNMP would have been long and likely fatal… or very short and almost painless (Hmmmm) either way, Thanks to Raphael Burry and his SNMP Discovery Provider for OpsMgr 2007 and Scott Vintinner with his Example SNMP Management Pack for SCOM 2007. Without these 2 pieces I would likely have left SCOM as an SNMP monitoring too altogether.

First download the sealed managed pack from Raphael rename it to a zip, break out the management pack and install it into you SCOM installation. This gives us the extensions to start writing out own custom management packs to start discovering different types of SNMP devices.

Second do your self a favor and download and install the iReasoning MIB browser and WireShark (formerly knows as Ethereal for those that missed the name change a while back) these tools will be invaluable shortly, trust me.

Step 2:  Figure out how to identify devices

First we need to figure out what makes this device different from any other device in our network, this is where the iReasoning MIB Browser comes in handy. After you have configured your device with an SNMP community name and allowed traffic from the machine you are running the MIB Browser from it’s time to pick an OID. Start the MIB Browser. You can load manufacturer MIB’s if you want but we don’t need them here. Enter the IP address of the device, clear anything that is in the IOD field and use the operations dropdown to pick walk. Then click GO!

Edit : Daniel Morrison makes a good comment – you may also need to hit “Advanced” and enter the SNMP community you configured on teh device for the walk operation to work. The default value is Public.  Just in caseyou missed it, you can see it below just between Address and OID. 

Now you can go down the list of OID’s that you see and find something specific that will be unique to this type of device. For this example we are looking at an SG series GE UPS.  In this case I am going to go with 1.3.6.1.2.1.1.2.0 that provides an answer of  1.3.6.1.4.1.818.1.100.1.1 with these noted it’s time to start editing our management pack.

Step 3: Your first custom SNMP discovering management pack!

Here again you will want a couple of tools to make life much easier.  First XML Notepad 2007 a Microsoft tool for editing XML, handy for when the next tool does not expose what you want. Second SCOM Authoring Console 2007 R2, from the AuthoringConsole directory in the root of your download or SCOM 2007 R2 CD. This is probably on the website somewhere but what isn’t anymore?

Now you are ready to download Scott Vintinner’s Example Management pack, (EDIT: or the updated version here) this is not installed just used as a framework to build our own custom pack for the device in question.  Make a copy and open it with notepad.  The first thing to notice is right near the top, <ID>RBH.Ecosaire.AC.Management.Pack</ID> we need to change this to match whatever we are doing like <ID>GEUPS.Example.Management.Pack</ID> so I do a find and replace on the whole XML from “RBH.Ecosaire.AC.” to “GEUPS.Example.” then I save and close the file. At this point you have to rename the file  to match the ID in this case GEUPS.Example.Management.Pack.xml

Now for the discovery bit. Open the XML with your preferred editor and look for <Discoveries> this section is all we plan to edit at this stage.  Within <Discoveries> you will find

          <SnmpVarBinds>
<SnmpVarBind>
<OID>1.3.6.1.2.1.1.2.0</OID>
<Syntax>0</Syntax>
<Value VariantType=”8″ />
</SnmpVarBind>
</SnmpVarBinds>

The key to this section is the OID this is what is queried to determine if this device is a GE UPS as defined by our management pack. A few lines further down you have another important section, we will start with a simple expression first.

<Expression>
<SimpleExpression>
<ValueExpression>
<XPathQuery>/DataItem/SnmpVarBinds/SnmpVarBind[OID=’1.3.6.1.2.1.1.2.0′][1]/Value</XPathQuery>
</ValueExpression>
<Operator>Equal</Operator>
<ValueExpression>
<Value Type=”String”>1.3.6.1.4.1.8072.3.2.10</Value>
</ValueExpression>
</SimpleExpression>
</Expression>

A few important things to notice in this section. First on the line with XPathQuery you see an OID, if you changed the OID in the <SnmpVarBind> section above you need to change it here as well as this is the pointer to the variable that was read and it needs to match for the compare or who knows what will happen. I of course know one thing that will happen and that is that you will never discover anything. So I guess with further thought I do know what will happen.  The second thing to note is <Operator>Equal</Operator> so this is a simple X = Y expression, we will look at another option in a second. Third <Value Type=”String”>1.3.6.1.4.1.8072.3.2.10</Value> this is telling us that the data type of this variable is a string, so you can’t try and match based on > or anything like that, and the value we are looking for is 1.3.6.1.4.1.8072.3.2.10, of course this is the example value and not the value we determined above so I will replace it with 1.3.6.1.4.1.818.1.100.1.1 and save the XML.      Optionally You can look for <Interval>3600</Interval> this is the number of seconds between discoveries. This can also be altered once the MP has been imported but unless you like to wait (in which case you will get along great with SCOM) you may want to reduce this now for testing. Try not to forget to change it back later 😉

Now if a simple expression is not good enough you may need a regular expression here is an example for another device I recently used. To be honest I am still looking for a good source of documentation on all the options for e regular expression if anyone knows a good one. (EDIT – Thanks Steve for pointing out the document Regular expression support in SCOM 2007.docx from the OpsManJam website. )

Here is the sample

<Expression>
<RegExExpression>
<ValueExpression>
<XPathQuery>/DataItem/SnmpVarBinds/SnmpVarBind[OID=’1.3.6.1.2.1.1.1.0′][1]/Value</XPathQuery>
</ValueExpression>
<Operator>MatchesRegularExpression</Operator>
<Pattern>^.*SensorHawk .*$</Pattern>
</RegExExpression>
</Expression>

With the above section of XML if the txt SensorHawk appears in the result then it’s considered a match.

Edit: Ben needed a discovery that would detect 2 different kinds of devices. Together we found the following appears to work best.

<Expression>
<Or>

  <Expression>
<RegExExpression>
<ValueExpression>
<XPathQuery>/DataItem/SnmpVarBinds/SnmpVarBind[OID=’1.3.6.1.2.1.1.1.0′][1]/Value</XPathQuery>
</ValueExpression>
<Operator>MatchesRegularExpression</Operator>

              <Pattern>^.*RICOH .*$</Pattern>
</RegExExpression>
</Expression>
<Expression>
<RegExExpression>
<ValueExpression>
<XPathQuery>/DataItem/SnmpVarBinds/SnmpVarBind[OID=’1.3.6.1.2.1.1.1.0′][1]/Value</XPathQuery>
</ValueExpression>
<Operator>MatchesRegularExpression</Operator>

              <Pattern>^.*Canon .*$</Pattern>
</RegExExpression>
</Expression>
</Or>
</Expression>

As awesome as Scott Vintinner’s example one thing I believe it lacks is a view in the console so you can see whats going on.  We need to add just a bit of XML to the existing pack, to make this easy I have added it to base pack using the RBH.Ecosaire.AC naming so you can just edit it all at once if you prefer.  Download updated management pack example Here we have added a section just after the end of  </Monitoring>

<Presentation>
<Views>
<View ID=”RBH.Ecosaire.AC.Management.Pack.AlertView” Accessibility=”Internal” Enabled=”true” Target=”RBH.Ecosaire.AC.Management.Pack.SNMPDevice” TypeID=”SC!Microsoft.SystemCenter.AlertViewType” Visible=”true”>
<Category>Custom</Category>
<Criteria />
</View>
<View ID=”RBH.Ecosaire.AC.Management.Pack.EventView” Accessibility=”Internal” Enabled=”true” Target=”RBH.Ecosaire.AC.Management.Pack.SNMPDevice” TypeID=”SC!Microsoft.SystemCenter.EventViewType” Visible=”true”>
<Category>Custom</Category>
<Criteria />
</View>
<View ID=”RBH.Ecosaire.AC.Management.Pack.PerformanceView” Accessibility=”Internal” Enabled=”true” Target=”RBH.Ecosaire.AC.Management.Pack.SNMPDevice” TypeID=”SC!Microsoft.SystemCenter.PerformanceViewType” Visible=”true”>
<Category>Custom</Category>
<Criteria />
</View>
<View ID=”RBH.Ecosaire.AC.Management.Pack.StateView” Accessibility=”Internal” Enabled=”true” Target=”RBH.Ecosaire.AC.Management.Pack.SNMPDevice” TypeID=”SC!Microsoft.SystemCenter.StateViewType” Visible=”true”>
<Category>Custom</Category>
<Criteria />
</View>
</Views>
<Folders>
<Folder ID=”RBH.Ecosaire.AC.Management.Pack.ViewFolder” Accessibility=”Internal” ParentFolder=”NetLib!Microsoft.SystemCenter.NetworkDevice.AllDevices.ViewFolder.Root” />
</Folders>
<FolderItems>
<FolderItem ElementID=”RBH.Ecosaire.AC.Management.Pack.AlertView” Folder=”RBH.Ecosaire.AC.Management.Pack.ViewFolder” />
<FolderItem ElementID=”RBH.Ecosaire.AC.Management.Pack.EventView” Folder=”RBH.Ecosaire.AC.Management.Pack.ViewFolder” />
<FolderItem ElementID=”RBH.Ecosaire.AC.Management.Pack.PerformanceView” Folder=”RBH.Ecosaire.AC.Management.Pack.ViewFolder” />
<FolderItem ElementID=”RBH.Ecosaire.AC.Management.Pack.StateView” Folder=”RBH.Ecosaire.AC.Management.Pack.ViewFolder” />
</FolderItems>
</Presentation>

And a few display strings that will form our text labels for the above entries, these go in the <LanguagePacks> section just before </DisplayStrings>

        <DisplayString ElementID=”RBH.Ecosaire.AC.Management.Pack.AlertView”>
<Name>Alerts</Name>
</DisplayString>
<DisplayString ElementID=”RBH.Ecosaire.AC.Management.Pack.EventView”>
<Name>Events</Name>
</DisplayString>
<DisplayString ElementID=”RBH.Ecosaire.AC.Management.Pack.PerformanceView”>
<Name>Performance View</Name>
</DisplayString>
<DisplayString ElementID=”RBH.Ecosaire.AC.Management.Pack.StateView”>
<Name>State View</Name>
</DisplayString>
<DisplayString ElementID=”RBH.Ecosaire.AC.Management.Pack.ViewFolder”>
<Name>Ecosaire AC</Name>
</DisplayString>

Now all you have to do is install your new MP and you should see in the SCOM monitoring console

With luck now you can see your base device management pack and may have even have enough good karma to see some devices.  Of course if all you have done is followed my instructions you still won’t have anything because there is one last step.

Step 4: You need to discover the device in SNMP. This is done via the operations console, in teh Administration tab.  Right click on “Device Management” and run the “Discovery Wizard”

Simple enough, select network devices and select next.

Here enter the IP or range of IP’s that your devices user, make sure to enter the community name you configured on the device, and drop the SNMP version down to 1, unless of course you know your device is V2. Select the management server that you want to handle traps, monitors and rules and click discover.

If all goes well a couple of minutes later you will get a screen showing the devices that have been doscovered, check the box(es) of the ones you want to be managed ckick finish and you are done. Then wait a while and they should start showing up in the management console.  If things didn’t go well there is likely a problem with either the community name or the SNMP configuration on the device allowing your root management server to contact the device using SNMP. Best to configure the device to send SNMP traps to all you management servers and allow SNMP read-only from all of your management servers.

Here are the completed XML files for a few of the MP’s I created if they are of use to you. Download and rename to .XML

GEUPS.GreaterThen.Five.Management.Pack

GEUPS.Single.Phase.Management.Pack

RBH.Ecosaire.AC.Management.Pack

Stay tuned for Part 2 where we will look into createing rules and monitors for the discovered devices.

Part 1

Part 2

Part 3