UPDATE as of 11:45 a.m. 9/29/20
Microsoft has identified the preliminary root cause and the extended impact as a combination of three separate and unrelated issues.
- A code defect in a service update.
- A tooling error in the Azure AD safe deployment system that impacted regional scoping.
- A code defect in Azure AD’s rollback mechanism, resulting in a delay in reverting the service update.
Microsoft’s monitoring automatically detected the issue within a minute of initial impact, and their engineering teams engaged immediately to initiate troubleshooting. Impact was variable based on regional load patterns and they immediately scaled out the services to help process the increased volume as a result of authentication retries due to the issue. Upon the successful rollback, full recovery for most customers was confirmed at 00:23 UTC on September 29. Microsoft’s engineers are engaged and monitoring the system to help ensure it continues to operate within normal parameters.
UPDATE as of 5:35 p.m.
All Microsoft 365 services have been restored and are operational.
Please contact the Service Desk (714) 997-6600 or Servicedesk@chapman.edu if you are continuing to experience technical difficulties accessing your Microsoft 365 services.
(Original message from 3:00 p.m.)
Microsoft is currently investigating an issue affecting access to multiple Office 365 services. IS&T is working with Micosoft to identify the full impact and will provide more information shortly.
The following services have been impacted:
- Outlook email
- G Suite
- Office 365