TT60009: Merit Network Problems 08/16/07
If you are experiencing network problems connecting to or from sites located in Michigan, Merit's TT60009 has some details.
(This is of course not an official announcement of anything, etc etc; "the information in this post is historical and may have changed since the date of the posting")
For more information:
- ITCS Service Status outages page: "Service Degradation"
This also affects some voice traffic from the University of Michigan Ann Arbor to the Dearborn campuses, Wayne State University, library and school systems statewide including the Ann Arbor District Library, and transit systems like the Ann Arbor Transportation Authority that share connections to Merit.
UPDATE August 21 2007 from the si.all list:
I’m sorry to be the harbinger of bad news, but there appears to be a repeat of last week’s network problems, caused by a switch failure at Merit – you may find it difficult or impossible to reach sites outside of UM.
Merit is working on it, but there is no time estimate of repair.
UPDATE Thu Aug 16 21:46:31 2007
As of 8:20 PM, EST, Thursday, August 16,2007, the Merit backbone network alerts have cleared and network services have started to return to normal operations. Merit's Network Engineers are performing additional diagnostic testing to ensure the stability of the backbone. We expect the Merit backbone will returned to normal operations by 10:00 PM, EST.
A detail report and further communication will be provided to the Merit Community, tomorrow, Friday, August 17, 2007.
If you are currently experiencing any network performance problems please contact Merit's Network Operations Center (NOC) immediately.
On behalf of all Merit staff, we apologize for the inconvenience this service issue has caused your organization. We at Merit continuously strive to provide the highest level of service to our Membership and regret this service issue.
The network is experiencing a continuing problem within the 10gig core routers. This does not appear as a total outage, but as a general slowdown as traffic seeks alternate routing around the switch that has the problem at any point in time.
The first event occurred at about 11:00 PM Wednesday Aug 15. At that time an Extreme Networks router processor card shutdown at the Ann Arbor site. The router was restarted and appeared to be functioning properly. Since then at least three other Extreme networks routers have exhibited the same symptom. Two of the routers also shutdown their 10gig interface cards. The issue has moved from router to router (not always the same router) and sometimes requires more than one restart for recovery.
At this point engineers from Extreme Networks are one the phone with Merit reviewing the console and log dumps from the routers. This does not appear to be a known problem with the routers.
Thank you for your patience and understanding.
some excerpts:
SCOPE: Merit Regional Network.
STATUS: The hardware failure previously experienced has
resurfaced.
The problem has been escalated with our vendor and Merit
lead engineers are working with them to resolve the issue.
Sites may lose connectivity or experience difficulty
reaching internet sites.
There is currently no ETR. Updates will be provided as
available. Thank you for you patience while
we resolve this matter.
SUBJECT: Merit Network hardware failure.
SCOPE: Merit Regional Network.
STATUS: A hardware failure in the Ann Arbor area resulted in
Merit member and affiliate sites being unable to
reach some destination within and external to Merit
Network. The problem was compounded because it also
caused some routes to be rejected by external network
providers. The hardware problem has been corrected and
most routes are again being accepted.
START TIME: Thursday, 15 August 2007, 22:32 EDT
Comments