Azure Network Availability Issues - 2025-10-29

Incident Report for ITS System

Resolved

Microsoft has issued that the issues have been resolved.
Official Microsoft Update :
"Impact Statement:
Starting at approximately 16:00 UTC on 29 October 2025, customers and Microsoft services leveraging Azure Front Door (AFD) may have experienced latencies, timeouts, and errors. We have confirmed that an inadvertent configuration change was the trigger event for this issue.

Affected Azure services may have included, but were not limited to:

App Service, Azure Active Directory B2C, Azure Communication Services, Azure Databricks, Azure Healthcare APIs, Azure Maps, Azure Portal, Azure SQL Database, Azure Virtual Desktop, Container Registry, Media Services, Microsoft Defender External Attack Surface Management, Microsoft Entra ID (Mobility Management Policy Service, Identity & Access Management, and User Management UX), Microsoft Purview, Microsoft Sentinel (Threat Intelligence), and Video Indexer.

Current status:
We have completed deployment of our ‘last known good’ configuration, and recovery is progressing steadily. We are currently recovering nodes and re-routing traffic through healthy nodes across the global fleet. As recovery continues, some requests may still land on unhealthy nodes, resulting in intermittent failures or reduced availability for a subset of customers.

This recovery effort involves reloading configurations and rebalancing traffic across a large number of nodes to restore full operational scale. The process is gradual by design, ensuring stability and preventing overload as dependent services recover.

Mitigation has been implemented for the AFD service for most customers, and services across the affected regions have largely recovered. We are continuing to work on residual impact and will closely monitor the situation in the coming hours.

Customer configuration changes remain temporarily blocked to prevent new deployments that could interfere with recovery. We will notify customers once this block has been lifted."

Posted Oct 29, 2025 - 22:37 EDT

Monitoring

Microsoft has communicated that were issues with Azure connectivity and infrastructure.
At this time, several UofT services may still be affected. The team will continue to provide updates on Microsoft’s progress.

Here are some of the services affected:
• Accommodated Testing Services (Student and Admin)
• ACORN Launchpad
• Calendar Launchpad
• Degree Confirmation
• OASIS
• Timetable Builder
• U of T Directory
• Transfer Explorer
• StarRez

From Microsoft:
" Current Status:
We initiated the deployment of our ‘last known good’ configuration, which has now successfully been completed. Customers may have begun to see initial signs of recovery. We are currently recovering nodes and routing traffic through healthy nodes, and as we make progress in this workstream, customers will continue to see improvement.

Customer configuration changes will remain temporarily blocked while we continue mitigation efforts. We will notify customers once this block has been lifted.

Some customers may also have experienced issues accessing the Azure management portal. We have failed the portal away from AFD to mitigate these access issues. Customers should now be able to access the Azure portal directly, and while most portal extensions are functioning as expected, a small number of endpoints (e.g., Marketplace) may still experience intermittent loading problems.

At this stage, we anticipate full mitigation within the next four hours as we continue to recover nodes. This means we expect recovery to happen by 23:20 UTC on 29 October 2025. We will provide another update on our progress within two hours, or sooner if warranted.

Although we are seeing signs of recovery and have an estimated timeline, customers may also consider implementing failover strategies using Azure Traffic Manager to redirect traffic from Azure Front Door to their origin servers as an interim measure.

Learn more about Azure Front Door failover strategies for AFD: https://learn.microsoft.com/en-us/azure/architecture/guide/networking/global-web-applications/overview

This message was last updated at 19:22 UTC on 29 October 2025"

Posted Oct 29, 2025 - 15:25 EDT

Update

Microsoft has communicated that are issues with Azure connectivity and infrastructure.
At this time, several UofT services are affected. The team will continue to provide updates on Microsoft’s progress.

Here are some of the services affected:
• Accommodated Testing Services (Student and Admin)
• ACORN Launchpad
• Calendar Launchpad
• Degree Confirmation
• OASIS
• Timetable Builder
• U of T Directory
• Transfer Explorer
• StarRez
• Course Evaluations
including:
Forms & Workflows (https://forms.provost.utoronto.ca)
Graduate Portal (https://graduateportal.sgs.utoronto.ca)

From Microsoft:
"Current status:
We have pushed our ‘last known good’ configuration, and customers may begin to see initial signs of recovery. We are currently recovering nodes and routing traffic through healthy nodes, and as we make progress in this workstream, customers will continue to see improvement.

Customer configuration changes will remain temporarily blocked while we continue mitigation efforts. We will notify customers once this block has been lifted.

Some customers may also have experienced issues accessing the Azure management portal. We have failed the portal away from AFD to mitigate these access issues. Customers should now be able to access the Azure portal directly, and while most portal extensions are functioning as expected, a small number of endpoints (e.g., Marketplace) may still experience intermittent loading problems.

We are continuing to monitor progress closely and will provide an ETA for full mitigation within the next 20 minutes as we assess recovery across the AFD service.

Although we are seeing signs of recovery, customers may also consider implementing failover strategies using Azure Traffic Manager to redirect traffic from Azure Front Door to their origin servers as an interim measure. https://learn.microsoft.com/en-us/azure/architecture/guide/networking/global-web-applications/overview

This message was last updated at 19:01 UTC on 29 October 2025"

Posted Oct 29, 2025 - 15:14 EDT

Identified

Microsoft has communicated that are issues with Azure connectivity and infrastructure.
At this time, several UofT services are affected. The team will continue to provide updates on Microsoft’s progress.

Here are some of the services affected:
• Accommodated Testing Services (Student and Admin)
• ACORN Launchpad
• Calendar Launchpad
• Degree Confirmation
• OASIS
• Timetable Builder
• U of T Directory
• Transfer Explorer
• StarRez

From Microsoft:
"Current status:
We have initiated the deployment of our last known good configuration, which is expected to complete within 30 minutes. As this deployment progresses, customers should begin to see initial signs of recovery. Once completed, we will begin recovering nodes and routing traffic through these healthy nodes.

Customer configuration changes will remain temporarily blocked while we continue mitigation efforts. We will notify customers once this block has been lifted.

Some customers may also have experienced issues accessing the Azure management portal. We have failed the portal away from AFD to mitigate these access issues. Customers should now be able to access the Azure portal directly, and while most portal extensions are functioning as expected, a small number of endpoints (e.g., Marketplace) may still experience intermittent loading problems.

We do not yet have an ETA for full mitigation, but we will provide another update within 30 minutes, once the deployment has completed.

Customers may also consider implementing failover strategies using Azure Traffic Manager to redirect traffic from Azure Front Door to their origin servers as an interim measure.

This message was last updated at 18:24 UTC on 29 October 2025"

Posted Oct 29, 2025 - 14:42 EDT

Update

Update from Microsoft:
"We are taking several concurrent actions: Firstly where we are blocking all changes to the AFD services, this includes customer configuration changes as well. At the same time, we are rolling back our AFD configuration to our last known good state. As we rollback we want to ensure that the problematic configuration doesn't re-initiate upon recovery.

Customers may have experienced problems accessing the Azure management portal. We have failed the portal away from AFD to mitigate the portal access issues. Customers should be able to access the Azure management portal directly, while all portal extensions are working correctly there may be a small number of endpoints that might have a problem loading (i.e. Marketplace).

We do not have an ETA for when the rollback will be completed, but we will update this communication within 30 minutes or when we have an update.

While we dont have an ETA yet. customers can consider implementing failover strategies with Azure Traffic Manager, to fail over from Azure Front Door to your origins: https://learn.microsoft.com/azure/architecture/guide/networking/global-web-applications/overview

This message was last updated at 17:50 UTC on 29 October 2025"

Posted Oct 29, 2025 - 13:54 EDT

Update

We are continuing to investigate this issue.
DNS Services have been affected as well

Posted Oct 29, 2025 - 13:44 EDT

Investigating

We have received reports of issues with Azure services. Microsoft has officially released a notice (https://azure.status.microsoft/en-us/status)

"Starting at approximately 16:00 UTC, we began experiencing Azure Front Door (AFD) issues resulting in a loss of availability of some services. We suspect that an inadvertent configuration change as the trigger event for this issue. We are taking two concurrent actions where we are blocking all changes to the AFD services and at the same time rolling back to our last known good state.
We have failed the portal away from AFD to mitigate the portal access issues. Customers should be able to access the Azure management portal directly.
We do not have an ETA for when the rollback will be completed, but we will update this communication within 30 minutes or when we have an update.
This message was last updated at 17:18 UTC on 29 October 2025"

We will post regular updates as we receive them from Microsoft.

EDIT: DNS services related to Azure are also affected.

** 16:00 UTC = 12:00 pm EDT **

Posted Oct 29, 2025 - 13:41 EDT

This incident affected: Student Web Services (ACORN, Accommodated Testing Services (Student & Admin), calendar.utoronto.ca, Course Evaluations, Degree Confirmation, Online Administrative Student Info System (OASIS), Timetable Builder, Transfer Explorer), Information Security (Azure Cloud Services), Enterprise Networks (DNS), and Telecommunications (U of T Online Directory).