The Gun Counter

About 10 years ago I got an SNMP question, worked on it for a day, answered it, and then never got to work with it again. Now I get to work with it again and a question has arisen.

We have a site with an HP ProCurve 2610 switch. The customer wants to either monitor a particular port's bandwidth usage, or accept traps when usage exceeds a limit; either way to alert 'someone' when usage peaks and if it stays peaked. As per usual the budget is minimal so we're not going to be purchasing OpenView or other commercial net management package and we don't have a ProCurve Manager license either.

I downloaded the HP MIB set and a MIB browser. Loaded all the MIBS and I can browse quite a lot of info from the switch. But I can't actually find the "port utilization" options (if any), by guess, by searching the MIBs for assumed terms, or by searching all the procurve info I can get my hands on. I can see the generic ones like port up/down and the various port error counters.

If I can find the names, I can find the OIDs, then the actual monitoring/polling or setup for catching traps (and knowing that this is the trap I want to deal with) is easy.

So is there a trick to this or is HP just being obtuse and I'm taking the hit?

Thanks...

If they're worried about who is using all the bandwidth and what they're doing, they'll get MUCH better information from the router than trying to do switch port monitoring.

Thanks both of you.

Chris,
I'm not the one who doesn't want the proper tools; thats up to the customer. We don't have resources onsite that we could run our own tools on, and we don't have the tools anyway. To be fair this is the first time a customer has run into what appears to be port congestion on a switch and this kind of network management is not our forte.

We can try to take a look at an openNMS or Nagios box as you mentioned but we're already over-committed on work; I have zero time for a new project, as interesting as that might be (and compared to taking care of sick microsoft crap, it would be wonderful...). It won't happen in time for this customer's desires.

The lack of a utilization stat certainly explains why its so hard to find

The Procurve summary page has reported max utilization of 100% on two ports, but doesn't provide a way (that I've found, somewhat new to that too) to say when or for how long that happened. It doesn't appear that the default SNMP traps cover high port utilization either. So we're just trying to find out if we can come up with a way to localize the usage times and hopefully tie them to specific site or user events.

If they don't see something immediately that triggers their suspicions, don't discount a chattering network card or a compromised machine being used for various purposes and trying to scan for other vulnerable machines on the network and outside too. ONE MACHINE can bring a network to a crawl.

I wouldn't be surprised if what they're seeing is an adware infection on one or two machines on that switch (one employee trying to share something with another employee and both machines becoming compromised).

CByrneIV wrote:Also, take a look at this and make sure it's not relevant:

http://anarchangel.blogspot.com/2008/04 ... rking.html

Thanks. Two of the three wintels are not using Broadcom adapters; the third is and I'll check it for patches; it is not on one of the ports that is logging high peak utilization. The proprietary linux box I'll have to throw at the support vendor. Its an IBM server and it is running on the gigabit port but the MAC lookups point to IBM, not Broadcom (understandably). That particular port barely registers usage.

Just checked the switch and its showing 85% utilization solid on two ports right now so since one of them is a server I have access to I'm off to do some packet captures...

308Mike wrote:If they don't see something immediately that triggers their suspicions, don't discount a chattering network card or a compromised machine being used for various purposes and trying to scan for other vulnerable machines on the network and outside too. ONE MACHINE can bring a network to a crawl.

I wouldn't be surprised if what they're seeing is an adware infection on one or two machines on that switch (one employee trying to share something with another employee and both machines becoming compromised).

The firewall should alert on most things going outbound; its running a full suite of gateway security protocols. So far we're not seeing excess use of the WAN bandwidth either. But yes scans are being done, and a more complete set are probably going to happen Sunday when they can take the servers down for a while.

Systems are clean per preliminary scans. The one broadcom equipped system does have old firmware (2006) on the card and an old driver but it is running standalone; no settings for the arp issues listed in the driver management screen and the registry entry I found elsewhere is also not present. We're searching HP now for info on any updates just the same. The Proliant pack listed for this server did not include (per release notes) anything by way of firmware for the broadcom board.

And we found part of the problem; the customer is doing Acronis image backups from the server on the separate switch to the DC (which has a USB drive attached for the purpose). That accounts for some of the peaks, but not all since I saw them this morning well after backups completed, but wasn't able to get a sniffer installed and running in time to catch.

Proliant Support Packs (PSP) are drivers and utilities... Look for the firmware maintenance CD.

ETA: A lot of the older low end Broadcoms don't have firmware upgrades so don't be surprised.. If the Broadcom is PCI and it's not HP branded, HP won't have firmware for it on the maintenance CD..

Ben, thanks.

Two of the servers came from my company and are box stock from HP. The third appears to be stock HP. We'll find out.

FWIW I have a q&d monitor running; it turns out the switch can turn on traps at 'levels' but not allow control of individual traps. I setup a trap receiver (a utility that comes with one of our proprietary systems) and its happily tucking away everything sent to it and sending email 'alerts' on receipt of the two possible relevant traps we found so far. Bit of a kludge but it was 'free' and working fine.

And its not running on windows so I had fun working on a proper system for a while; a rare treat any more.