Network & Service Outages
December 14, 2024
No outages scheduled today
Sunday | Monday | Tuesday | Wednesday | Thursday | Friday | Saturday |
1 | 2 | 3 | 4 | |||
5 | 6 | 7 | 8 | 9 | 10 | 11 |
12 | 13 | 14 | 15 | 16 | 17 | 18 |
19 | 20 | 21 | 22 | 23 | 24 | 25 |
26 | 27 | 28 | 29 | 30 | 31 |
- Panguitch Power Outage: 2019-05-02 @ 9:05 AM
Description: Rocky Mountain Power has reported power outage impacting the Panguitch area. Repair crews have been notified. They estimate power will be restored by 1PM. Update: This outage was resolved on time.
Impact: Garfield School District UTN Site Panguitch Dental - Century Link: 2019-05-05 @ 9:30 AM
Description: Century Link is currently having issues with a WU line card in the Parowan area. They are currently working with tech support to resolve the issue. There is currently on ETR. Update: This outage was resolved at 12:36 PM.
Impact: TDC --> DDC Backbone SUU --> RDC Backbone UTN - Beaver Valley Hospital Parowan SD - Strata Networks Maintenance: 2019-05-07 @ 11:55 PM
Description: Equipment Maintenance Update: This maintenance completed on time.
Impact: Neola Elementary Whiterocks Community Center Whiterocks Headstart - WSU Life Safety testing: 2019-05-08 @ 5:00 AM
Description: Facilities Management will be testing the life safety systems in the Hurst center at WSU. Update: This maintenance completed on time.
Impact: None expected, but site may lose connection briefly due to unforeseen issues. - Route Reflector Code Upgrade: 2019-05-09 @ 11:00 PM
Description: Upgrading the last backbone route reflector. Device will need to reboot one time. Update: This maintenance will begin in 5 minutes. Update: This maintenance completed at 11:25 PM.
Impact: No noticeable impact, we peer with three total route reflectors, only one will reboot. - Emergency Fiber testing UofU: 2019-05-09 @ 11:00 PM
Description: Fiber between EBC and DDC needs to be tested due to error conditions. This is a Ciena circuit that has multiple connections, Primarily 2 40 GB circuits for the U. Update: This outage was resolved at 1:38 AM.
Impact: During testing, these circuits will not be available and will show as down. - UVW to UBATC Maintenance: 2019-05-14 @ 12:00 AM
Description: CIRCUITS AFFECTED: 10 gig circuit from UVW to UBATC DESCRIPTION OF ACTIVITY: Equipment Maintenance Update: This maintenance completed on time.
Impact: This circuit may be unavailable during this time frame. - Outage On Backbone links: 2019-05-15 @ 9:25 AM
Description: We have experienced an outage on our equipment. We are dispatching now. Update: This outage was resolved at 10:55 AM.
Impact: Granite School District Tooele School District Dixie and SCC backbone links - Century Link Maintenance: 2019-05-16 @ 11:00 PM
Description: Centurylink has seen errors on their device and will need to swap out an SMTMU and components associated with that. They will perform this maintenance at 11:00pm. UEN ticket:108787 Update: This maintenance will begin in 5 minutes. Update: This maintenance completed on time.
Impact: 5 Circuits at Canyons Do-In This will not affect services for these sites, all of them are dual homed as well 69.KRFS.000539 Copperview El 69.KRFS.000549 Hillcrest Hs 69.KRFS.000553 Jordan Valley 69.KRFS.000545 Eastmidvale 69.KRFS.000531 cab west - New Ten Gig circuit between SUU and Panguitch CO: 2019-05-16 @ 11:00 PM
Description: UEN will be moving traffic between SUU and the Panguitch CO to a new Ten Gig circuit. Update: This maintenance will begin in 5 minutes. Update: This maintenance completed at 11:43 PM.
Impact: The Kane, Piute, Garfield, and Wayne districts may experience a brief service interruption. - ASR Code upgrade: 2019-05-16 @ 11:00 PM
Description: Upgrading code on ASR9K. Will require 2 reloads. Update: Will be at the DATC pop site. Update: This maintenance will begin in 5 minutes. Update: This maintenance completed at 11:48 PM.
Impact: Redundant backbone link will go down durning reloads, traffic will reroute. No customers will have any down time. - Century Link Maintenance: 2019-05-16 @ 11:00 PM
Description: Centurylink has seen errors on their device and will need to swap out an SMTMU and components associated with that. They will perform this maintenance at 11:00pm. UEN ticket:108787 Update: This maintenance will begin in 5 minutes. Update: This maintenance completed on time.
Impact: 5 Circuits at Canyons Do-In This will not affect services for these sites, all of them are dual homed as well 69.KRFS.000539 Copperview El 69.KRFS.000549 Hillcrest Hs 69.KRFS.000553 Jordan Valley 69.KRFS.000545 Eastmidvale 69.KRFS.000531 cab west - Network Outage: 2019-05-16 @ 12:45 PM
Description: UEN is experience a large scale outage on the network. Our NOC engineers are looking into the issue. We will provide updates as they come in. Update: This appears to be a provider issue with Century Link, they are investigating the issue Update: UPDATE: Century Link is reporting a possible fiber cut or card issue between Draper and Riverton. Their technicians are en route for root cause discovery. Update: UPDATE: Root cause is still unknown, Century Link technicians have arrived on site Update: UPDATE: Large scale fiber cut is confirmed in the Draper area. Repair crews will be on site at 4:15pm. Repair time unknown but is expected to go into the night. Update: UPDATE: The damage to the fiber is significant and will require extensive repairs. Estimated time to resolution could be as late as Noon tomorrow. Update: General Update, Here is the latest. SUMMARY At a little before 1p, one of the main conduit paths for CenturyLink fiber in the southern part of Salt Lake County was damaged by utility construction in the area. CenturyLink dispatched personnel to locate the cut, and they identified the area and assessed damage. The damage is extensive and will require that new conduit and fiber be installed to complete the repair. This will require splicing at multiple points along the path and considerable effort. In addition, splicing and conduit repair are delayed due to the need to close part of a major roadway in order to have access to the damaged infrastructure. TIMEFRAMES Although the timeframes are still only projections, CenturyLink is now projecting the ETR for this repair to extend into the late morning/early afternoon of Friday May 17th. As a result, we are asking impacted sites to plan on disruptions to network access in the morning hours. MITIGATION UETN is working with impacted organizations to re-route traffic through alternate links where possible to restore service. These efforts are ongoing this evening. MORE TO COME We expect to get regular updates from CenturyLink throughout the evening and overnight, and will continue to update this outage thread. HOW TO CONTACT US UETN Network Operations is staffed 24 hours a day M-F. Please don't hesitate to contact us with any questions. Direct dial: 801-xxx-xxxx --TROY Update: We have received an update from CenturyLink. They have proceeded to bore a new path for the fiber because the existing conduit was beyond repair. Boring a new path will take several hours with splicing currently planned to begin mid-morning if all goes as planned. ETR remains early to mid afternoon Friday. We will give more specific updates once we get word on when splicing begins and have better timeframes. Efforts to move organizations with alternate path options has begun and is in various stages of implementation. All sites with alternative paths have been contacted and options have been discussed. Sites still impacted include: Jordan School District (Partial Outage), Canyons School District (Complete Outage), Murray School District (Minor Outage), and multiple Salt Lake Valley Charter Schools (each with Complete Outage). Our next update is planned for ~7:00a MDT unless we have significant details to communicate prior to that. Update: CenturyLink has been working diligently through the night, and has made progress. They are still working to restore service. We will update further with more information as it becomes available. Please contact the UEN NOC with any questions. 801-xxx-xxxx Update: This outage was resolved on time.
Impact: Multiple sites. - Draper Area Fiber Cut (Network Outage): 2019-05-17 @ 12:05 AM
Description: UETN is continuing to track the Network Outage of multiple sites in the Salt Lake valley area. The following is a summary of the current situation: --- SUMMARY Yesterday just before 1p local time, two major bundles of CenturyLink (CL) fiber optic cable near 12300 S and 88 W were damaged by horizontal boring in the area by another company. CL dispatched crews to the area and discovered extensive damage to their fiber and conduit. Restoration required installation of about 2000' of new conduit, and those efforts proceeded overnight. Splicing of the fiber will proceed as soon as possible this morning. --- TIMEFRAMES Current timeframes for restoration of service are speculative still, but could come online at various times throughout the late-morning through late-afternoon based on multiple factors including when individual fiber splices occur. The more official ETR has been published as 6p for the entire repair, but we could see service for many sites much earlier than that. --- IMPACT Impact from the outage has been mitigated by moving entities to alternate paths wherever possible. Canyons School District remains completely offline, Jordan School District is impacted with partial outages, and Murray School District is impacted with only a few services offline. In addition to that, about 13 local area Charter Schools are offline. -- CONTACT Please contact UEN Network Operations if you have additional questions. 801-xxx-xxxx Update: --- UPDATE 10:00a MDT We have received reports from CL that they have finished placing fiber in the new conduit and have started the process of splicing fiber. As a result we expect to see some sites returning to service over the next few hours. Update: --- UPDATE 1:02p We are beginning to see sites returning to service at this time. We will be doing testing and sending out a more specific update before 2p. Update: We continue to see sites returning to service. UETN is currently aware of six sites with ongoing outages. We expect these to return to service over the next few hours. Update: This outage was resolved at 9:35 PM.
Impact: xx - West Haven Syringa: 2019-05-17 @ 1:00 AM
Description: Description: Syringa Networks will be splicing in an existing enclosure with live fibers in the West Haven area. 19/LXFS/100009//SYG 89/KRFS/100007//SYG Update: This maintenance completed on time.
Impact: Anticipated Impact: Low Potential Impact: Loss of Service - Strata Maintenance: 2019-05-21 @ 11:55 PM
Description: Strata Networks will be performing maintenance on their equipment. CIRCUITS AFFECTED: Tabiona High School Tabiona Clinic Update: This maintenance completed on time.
Impact: Sites listed above may lose connectivity during this maintenance period. - Utah Valley area outage: 2019-05-21 @ 1:30 PM
Description: UETN is investigating the cause of an outage of approximately 15-20 sites in the Utah Valley area. Sites are currently recovering, but we are not yet sure what the root cause is yet. Update: This outage was resolved at 3:28 PM.
Impact: Multiple sites were offline. - Emery Telcom Blanding Fiber repair: 2019-05-21 @ 11:00 PM
Description: Emery Telcom will be splicing fiber and replacing a fiber multi service terminal (MST) in Blanding which was damaged by a tall vehicle during the winter. Emergency repairs were made after the original damage but further repair is needed at this time. ustomer Internet access and point to point circuits will be unavailable during the fiber splicing which can be up to two hours. Update: This maintenance completed on time.
Impact: San Juan Health – Blanding Clinic to Monticello San Juan Health – 20 Mbps to Telehealth San Juan Counseling - UEN Outage Base out of SUU: 2019-05-21 @ 8:20 AM
Description: UEN lost services to several sites in western Utah. We are still investigating. We'll provide more information as we gather more details. Update: A power event in the SUU POP (Point of Presence) Caused the UEN devices to reboot. Crews Are working to restore house power. Some cards have not recovered gracefully from the power loss. We are working with field techs to troubleshoot. IMPACT: Iron School District and sites based out of SUU POP are still impacted. Update: All sites at districts affected by this outage were recovered at 9:16AM. We are investigating possible issues with our UPS backup system and will provide another update today. Update: UETN will be switching our equipment back to Utility power today at 4:00p Local Time. We anticipate some sites in the area experiencing a brief outage as we Bypass a faulty UPS system as part of the cut back. Update: We have made progress and are back on stable utility power, however the SUU POP is in a degraded, non-redundant power state pending a permanent fix to our UPS system. Though we do not anticipate it, In the event of a house utility power failure, customers connected at SUU will experience a 10 minute outage while backup power comes online. UETN is working to repair the UPS as soon as possible to restore full uninterrupted power redundancy. Update: This outage was resolved on time.
Impact: Kane, Piute, Wayne, Garfield, SUU, Beaver, UTN sites in the area. - Emergency Power maintenance - SUU POP: 2019-05-22 @ 4:40 PM
Description: At 4:40pm today, UETN needs to perform emergency power maintenance on a switch at our SUU POP to ensure stable power redundancy for tomorrow's UPS maintenance. We anticipate 5-10 minutes of downtime while the switch power cycles. Update: This outage was resolved at 4:54 PM.
Impact: Cedar City Library Iron School District Beaver Valley Hospital Enoch Head Start Fiddlers Head Start St George Head Start Parowan Head Start Hurricaine Valley Head Start Young Sunset Head Start Southwest Applied Technical College Southwest Educational Development Center - Ciena: Replace Card: 2019-05-23 @ 11:00 PM
Description: Ciena Case Number: 79169 DTDC-B-S-2 100GOCLD-2-13 Card was bouncing at irregular intervals about 20 minutes apart. The shelf was re-booting and Ciena noc has determined the card needs to be replaced. An RMA has been set up. UEN-DTDC-B-S-2-SHELF-2-ETH100G-2-12-1, 140.197.209.23 Update: This maintenance completed on time.
Impact: Will work with Kevin Quire for circuit info - Strata Networks Maintenance: 2019-05-28 @ 11:55 PM
Description: Equipment Maintenance Update: This maintenance completed on time.
Impact: Myton Elementary Myton Community Center - UETN Maintanance: 2019-05-28 @ 11:30 PM
Description: UETN Operations will be reloading a primary backbone device for maintenance purposes's. Update: This maintenance will begin in 5 minutes. Update: This maintenance will begin in 5 minutes Update: This maintenance completed at 11:47 PM.
Impact: 100 Gig connection to UofU Uofu To Dixi PAC 12 Network EBC DMARC Devices UEN Building UEN VRF UEN VOIP UEN VPN1 EBC Level 3 USOE board of regents Private/Charter/School Districts Impacted: Our Lady of Lourdes J.E Cosgriff Murray Reading Center Weilenmann School Radio link Ascent Murray Winter Sports School Salt Lake School of Performing Arts Salt Lake City Library Canyons School District Westminster Campus chpc perfsonar server UTN utn-rt-dc-ebc-internet DC_utn-rt-ebc - Brief outage at EBC: 2019-05-29 @ 1:35 PM
Description: UETN experienced a brief outage on one of our nodes at EBC from 1:32PM to 1:39PM. The issue has been resolved. Update: This outage was resolved at 1:51 PM.
Impact: Various sites off of this node were unreachable during this time. - ASR Code upgrade: 2019-05-30 @ 11:00 PM
Description: Upgrading code on the EBC-PEP-B Device. Update: This maintenance will begin in 5 minutes. Update: This maintenance completed at 1:13 AM.
Impact: This will take down several links and connections for the upgrade. Most major links are redundant but any single homed customer will be down for the duration of the outage. - CenturyLink OC192 Maintenance: 2019-05-30 @ 11:00 PM
Description: Centurylink will be performing maintenance on an OC-192. Update: This maintenance completed on time.
Impact: It will interrupt service to the following locations: Canyons SD Administration Copperview Elementary East Midvale Elementary East Sandy Elementary Hillcrest HS Jordan HS Canyons Tech Ed Center - Replacing SSL cert on dns.uen.net: 2019-05-31 @ 3:00 PM
Description: The SSL cert on dns.uen.net expires next week. I will be renewing the certificate on Friday, May 31 at 3:00 PM. Update: This maintenance will begin in 5 minutes. Update: This maintenance completed at 3:05 PM.
Impact: Any active sessions at the time of the replacement will be terminated. The outage should last less than one minute.
Outage Types:
Outage: An observed outage, not scheduled.
Normal: A planned outage, 24 hours in advance.
Urgent: A planned outage, 12 hours in advance
Emergency: Immediate outage required
Third Party Maintenance: A planned outage performed by an organization other than UEN
Additional Information: Normal maintenance outages, if necessary, are scheduled on Tuesdays and Thursdays starting at 11pm. If any additional information about outages is required, please contact the UEN Operations Center at noc@uen.org, 801-585-7440 or 800-836-4396.
Outage Notifications:
- By Email: If you would like to be added to our outage notification email list please click here.