NETWORK MANAGEMENT
NETWORK MANAGEMENT
REQUIRES THE RIGHT
COMBINATION OF SKILLS,
TOOLS, AND INTUITION
TO TRACK DOWN AND
SOLVE PROBLEMS.
WE ALL HAVE STORIES ABOUT OUR
worst network-management nightmares.
I’m going to share a few choice ones to
show you that, sometimes, you should
expect the unexpected.
BY DAVID STROM
THE CASE OF THE BAD MEDIA FILTER
Remember when fiber networks were
first being implemented and required
the use of media filters (converters that
connected the fiber drops to a copper
connector)? Well, Eric Kimminau,
enterprise architect at a major automotive firm, remembers—and he has this
tale to tell:
“As a fourth-level field support
engineer, I was called onsite to a major
petroleum company in Texas to investigate an NFS [Network File System]
server performance-related issue. The
interesting bit was that there were
two groups of workstations: one that
could communicate with the main file
server in another building and one
that couldn’t.
“We discovered that high-speed
copper interconnects were used to
connect floor-distribution switches
to the building core, and media converters were used to go back to fiber
for the campus WAN connections to
other buildings. These specific media
converters had a tendency to maintain
ARP [Address Resolution Protocol]
cache in the core switches.
“After a lot of troubleshooting
with multiple vendors and installers,
and spending about a hundred man-hours, we figured out that half the
path of a copper-to-fiber media converter was bad. Simply disconnecting
the media converter, taking down the
link and forcing an ARP cache reset
cleared the problem. All performance
issues were instantly eliminated.
“The solution required a combination of load-generation tools, traffic
and protocol analyzers, switch OS
diagnostics, TCP stack-level kernel
diagnostics, and multiple hardware
vendors and support personnel. All
this was needed to solve a problem
caused by a 100 media converter!
“Of course, we then did a thorough
quality check of the 40 or so media
converters around the building and
identified about 20 percent that either
were failing to some degree or were
contributing to significant degradation
of network quality. A decision was made
to implement high-quality fiber patch
panels and to implement fiber runs
from core to patch panels end to end
for WAN links, eliminating the media
converters from the environment.”
