Service Outage, Nun & Moor Monkton – 21/11/2012

A power cut at the school has knocked out the NyNet router. We are awaiting further details from NyNet, and will be failing over to 3G if required. This message will be updated with further details.

DETAILED REPORT AT THE BOTTOM OF THIS POST

UPDATE 3:30PM

NyNet are aware of the problem, and have sent an engineer to investigate. We have switched over to our 3G backup link in the mean time. Unfortunately outgoing SMTP may not work while using 3G. Further details will be added as they become available.

UPDATE 4:30PM

A fault has been logged with BT to attend on site as the Nynet engineer hasn’t been able to rectify the fault. We will be attending on site to give BT access to the school. The 3G service is running power permitting.

UPDATE 7:45PM

Unfortunately our backup 3G connection has now failed. BT have assigned the engineers to the support ticket, we are just awaiting an ETA of arrival. BT operate 24 hours, and we will be attending on site just as soon as they are able. Service will be restored ASAP, with a full breakdown of incidents published once things are back to normal.

UPDATE 9:30PM

Service has been restored following a visit from BT. Full details will be posted when they become clear.

INCIDENT REPORT

  • The NyNet UPS failed, meaning power to the router and BT hardware was lost during a short power outage. As the batteries had failed, the device didn’t repower the attached equipment once power restored and required intervention from NyNet to bring the router back online. We reported the UPS failure to NyNet the morning of the failure, but unfortunately they didn’t have time to respond before another power cut.
  • During this time, we moved customers to a backup 3G connection
  • Once NyNet attended onsite and repowered equipment, it became clear the power failure had caused a fault on the fibre connection. A fault was then logged with BT to be resolved in the 4 hour support contract.
  • Unfortunately our 3G connection expired during the evening, which meant from approx 7:30 until BT fixed the fault there was a total service failure.
  • BT attended both at Nun Monkton School and the exchange, and fixed the fault.
  • Full service was restored at 9:30PM.

OUTCOMES

  • NyNet are to source and install a new UPS
  • We have restored our 3G connection, and added a system to warn of nearing expiry of accounts.
  • The Nun & Moor Monkton network is under enhanced monitoring until NyNet replace the UPS, so we can respond quickly if needed.

Leave a Reply


External Server: GoDaddy
WP Login