UPS-B - Critical Event (Resolved)
  • Priority - Low
  • Affecting Other - POWER
  • Please call our NOC if you have any problems remaining +64 9 280 4135


    26/05/2020 - 5pm - Power to UPS B has been restored and our B feed is back with inline UPS power. Event is now resolved.


    Affected Services: Customers drawing power from B UPS Feed.  We have 3 UPS's providing power to 3 feeds called A, B, and C UPS. This event only impacts customers drawing power from the B feed.
    Impact: Customers running on single UPS-B power feeds are now subject to the stability of the power grid until this window ends, at this stage engineers are making a plan to bring back online protected B UPS Power on Tuesday 26th.


    6:59AM - UPS B is in bypass mode - B Power Restored - PDU's running from UPS B are now running from non-protected mains power.

    6:00AM - Systems Team are assessing servers & services which are affected and moving their plugs to new PDU's on other UPS feeds and restoring hard down services.


    Called UPS/Power engineer (2020-05-24 05:44)

    Engineer onsite, investigating. (2020-05-24 05:15)

    4:50 AM - Critical Event - UPS B Faulted and is Hard Down - This has affected Servers/network devices which only have single power connected to B feed PDU's in some racks.

     

  • Date - 24/05/2020 04:50 - 26/05/2020 17:18
  • Last Updated - 26/05/2020 17:20
Scheduled Network Change 28/04/2020 4am (Resolved)
  • Priority - Medium
  • Affecting Other - Network
  • In an ongoing effort to provide the best possible service to our customers, we are increasing our bandwidth and moving national and international transit providers at the above time.

     

    Whilst we are not anticipating any noticeable network outage or degradation when we switch the routing across please contact us if you experience any issues you feel relate to the above infrastructure change.

     

    Kind regards,

    HD Networking Team

  • Date - 28/04/2020 04:00 - 30/04/2020 16:59
  • Last Updated - 22/04/2020 15:45
UPS-B - Scheduled Remedial Actions (Resolved)
  • Priority - High
  • Affecting Other - Datacentre
  • Event B - Planned UPS-B Remedial Actions - Scheduled 09:00 on 29/02/2020 

    Update 1 - 27/02/2020 12:30 : Repair work & testing of UPS-B has been completed by it's authorised supplier, we have scheduled to replace the temporary loan unit currently in service between 09:00 and 12:30 on Sat 29/02/2020. This procedure has been performed before without incident however there will be 3 risk stages below for clients running on single UPS-B fed power supplies, further updates will be provided as we progress through these stages. 

    • Stage 1 - Moving from a protected state to hard bypass - minimal risk that clients on UPS-B power feed will be impacted for a few milliseconds.
    • Stage 2 - During the replacement - Clients running on single power feeds dependent on UPS-B are subject to the stability of the vector power grid.
    • Stage 3 - Moving from hard bypass to a protected state - minimal risk that clients on UPS-B power feed will be impacted for a few milliseconds.

     


    Event A - Original UPS-B Incident 18/12/2019

    Analysis Description: Routine preventative maintenance detected failed components in UPS-B
    Systems Impacted: Clients running on single power feeds dependent on UPS-B are subject to the stability of the vector power grid until this window ends. UPS-A and UPS-C are operational.
    Next Steps: UPS repair and or replacement underway.
    Window Duration: Dependant on repair or replacement timeframe estimated 24-72 hours.

    Update 1 - 18/12/2019 12:10 : Faulty output fuse identified, planned to place UPS-B into hard bypass and determine if replaceable. Contingency plans are also being expedited to organise a complete replacement loan UPS unit.

    Update 2 - 18/12/2019 12:46 : at 13:40 UPS-B will be placed into hard bypass,  this procedure has been routinely performed in the past without incident however there is a minimal risk that clients on B power feed will be impacted for a few milliseconds.

    Update 3 - 18/12/2019 13:40 : First transition from normal mode into hard bypass for UPS-B has occurred without incident, there will be a second transition which will occur when we bring this UPS back into service, more information to follow.

    Update 4 - 18/12/2019 14:56 : Further diagnostics and testing have been unable to isolate the fault on UPS-B, as a result HD is expediting its plans to replace UPS-B in entirety with a loan unit whilst we work towards a permanent replacement or fix, more information to follow.

    Update 5 - 20/12/2019 10:07 : Replacement loan UPS arriving 12:00 today, we plan between 13:00 and 17:00 to bring UPS-B power back into a protected state. This procedure has been performed before without incident however when we move out of hard bypass back to a protected state there is a minimal risk that clients on B power feed will be impacted for a few milliseconds. Further information on exact times to follow.

    Update 6 - 20/12/2019 13:12 : Replacement loan UPS currently being installed, planned move from hard bypass to protected state at 13:45

    Update 7 - 20/12/2019 14:19 : Final checks on load UPS in progress, switchover to protected power now scheduled for 14:35

    Update 8- 20/12/2019 14:53 : Task completed, UPS-B power now protected with loan unit, scheduled for mid Jan as to full replacement or repair of our existing unit.

  • Date - 29/02/2020 09:00 - 29/02/2020 13:05
  • Last Updated - 29/02/2020 13:05
Network equipment maintenance - Tauranga (Resolved)
  • Priority - Low
  • Affecting Other - Broadband
  • Regional backhaul for Tauranga area is scheduled for network equipment maintenance, all broadband customers will experience a short outage during the maintenance window in that area.

  • Date - 11/12/2019 07:45 - 11/12/2019 08:15
  • Last Updated - 10/12/2019 15:44
UFB Fibre Services - UFF Hamilton Region (Resolved)
  • Priority - Medium
  • Affecting System - Broadband Internet
  • 13:10 HD is investigating an issue impacting customers with UFB fibre services from UFF in Hamilton.

    13:30 Confirmed issue resolved

  • Date - 21/11/2019 13:10
  • Last Updated - 21/11/2019 13:37