On Tue, 13 Sep 2005 21:34:02 -0500, ibuprofin@painkiller.example.tld
(Moe Trin) wrote:

>In the Usenet newsgroup alt.internet.wireless, in article
><gjnci15ei80kbrlanvk0c5lc78974etjt9@4ax.com>, Jeff Liebermann wrote:
>
>>Been there. HP LaserJet 4 with Jetdirect J2552 card. If I run out of
>>paper, it floods the networks with garbage that was impossible to
>>decode with Ethereal. That took me 6 months to find.

>Don't know what your network looks like, but HP only has a handful of
>OUI blocks:
>
>[compton ~]$ zgrep -i Hewlett MACaddresses.gz | grep base | cut -d' '
>-f1 | column
>0001E6 000883 000E7F 00110A 001321 001560 0060B0
>0001E7 000A57 000F20 001185 001438 00306E 0080A0
>0004EA 000D9D 001083 001279 0014C2 0030C1 080009
>[compton ~]$
>
>That's straight out of the IEEE file. I'm at an R&D facility, and we're
>super paranoid, so every host is 'registered' meaning we know MAC, IP,
>user, location, which drop from which switch, serial and decal numbers,
>and the date of last tetanus shot for everything that connects to our net.
>If something starts squittering, I can ID the box in seconds. If the box
>is unknown, I can ID the drop, and it's 50/50 if the security goons get
>there before me or not.
>
> Old guy

Well, if the 802.3 Ethernet packets were well formed and contained MAC
addresses, tracing the problem back to the source would have been
trivial. Instead, what I was seeing was bursts of garbage that I
couldn't decode. I tired Ethereal, a Network General Sniffer, NT
Netmon, and a bunch of demo sniffers I downloaded just to see if they
could make sense of the traffic. I could see the garbage very lightly
flashing the lights on the hubs, but could not decode anything. I
spend two days with a logic analyzer trying to capture useful data and
decode the contents manually, but even that didn't produce anything
useful.

Just to make it interesting, I made a rather stupid series of
mistakes. This was in the days when hubs were in fashion and switches
were expensive and scarce (approx 1997). They had about 50 boxes, in
3 locations, connected with Cisco 340 series wireless bridges, all
interconnected with hubs. There were three identical HP LaserJet 4
printers involved. Nobody every deduced that the network running slow
was caused by running out of paper because there was always someone
around to replace the paper that was not directly involved in using
the computahs. Running out of paper was a very uncommon experience,
so the time of slow downs were not easy to predict.

I had wrongly decided that the various 16 port Linksys 10baseT hubs
were the likely culprits and convinced management to go for an HP
Procurve 4000 switch, mostly on the basis of speeding things up to
100baseTX-FDX. The switch arrived before I could finish some
necessary re-wiring so one of the four hubs remained. The nice thing
about switches is that garbaged and trashed packets do not go through
a store and forward switch. Everything that was plugged into the
Procurve switch worked without a slowdown. Everything that was still
on the hub slowed to a crawl whenever the HP LJ4 ran out of paper.

Again, I wrongly interpreted the problem as being the hub and
performed an overnight panic rewiring job to move everything to the
switch. The slowing stopped. I thought it was fixed.

The nice thing about managed switches is that you can use SNMP and the
internal diagnostics to detect problems. The three HP4 printers in
question were on the last hub. When connected to the switch, the
stats started showing large numbers of corrupted packets. Of course,
I didn't bother labeling the cables so I didn't have an immediate clue
as to where the junk was coming from. This time, I incorrectly blamed
the wiring. After wasting some time with a borrowed cable certifier,
I eventually figured out the corrupted packets were associated with
the printers.

Upgrading the flash on an HP J2552 is somewhat of a challenge. The HP
software sucks. One mistake and the $300 card is a paperweight. It
took a while but I eventually got all three cards upgraded and
configured. The problem hasn't surfaced since.

If there had been anything decoded by a sniffer, I would have found
the source almost immediately. Instead, it was a painful 6 month
ordeal, with lots of bad guesswork, and a substantial amount of luck
in finding the problem. What I consider the most important lesson
from the aformentioned exercise was that I could not have figured it
out without the statistics and diagnostics from the managed switch.


--
# Jeff Liebermann 150 Felker St #D Santa Cruz CA 95060
# 831.336.2558 voice Skype: JeffLiebermann
# http://www.LearnByDestroying.com AE6KS
# http://802.11junk.com
# jeffl@comix.santa-cruz.ca.us
# jeffl@cruzio.com