Incident Report

Summary

On 19 November 2024 at 10:44 US/Pacific, Google Calendar experienced service disruptions globally for web users due to a misconfigured denial-of-service rule. This resulted in some users being unable to access their calendars for a duration of 1 hour and 40 minutes. Google Calendar’s mobile application users were not impacted.

To our Google Calendar users impacted during this disruption, we sincerely apologize. This is not the level of quality and reliability we strive to offer you, and we are taking immediate steps to improve the platform’s performance and availability.

Root Cause

The incident was triggered by the implementation of a denial-of-service (DoS) rule, intended to mitigate malicious traffic. However, a misconfiguration led to the rule being applied to all traffic instead of traffic to a specific endpoint. Consequently, some users who triggered the rule may have had all of their web traffic to Google Calendar blocked, resulting in a denial of service for those users.

The root cause was an incorrect scope for a rule within the DoS configuration. The rule was applied too broadly with the purpose of blocking malicious traffic, and led to unintended blocking of legitimate user traffic.

Internal monitoring shows a traffic anomaly during the incident, but no alerts were triggered.

Remediation and Prevention

The issue was escalated by the support team to the engineering team at 12:12 US/Pacific. The on-call Engineer identified the misconfiguration and rolled back the faulty change to mitigate the issue at 12:24 US/Pacific.

Google is committed preventing a repeat of this issue in the future and is completing the following actions:

• Ensuring oncall engineers are alerted when unusual amounts of traffic are blocked.
• Updating processes and documentation around denial-of-service configurations to prevent recurrence.
• Creating internal trainings on best practices for denial-of-service configurations to prevent recurrence.
• Investigating validation mechanisms to catch bad configurations.

Detailed Description of Impact

Google Calendar:

On 19 November 2024 at 10:44 US/Pacific,

• Google Calendar web users globally were impacted.
• The primary impact was the inability to access calendar functionalities after transferring an event.
• The issue was mitigated at 12:24 US/Pacific.

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Workspace Support using help article https://support.google.com/a/answer/1047213.

(All Times US/Pacific)

Incident Start: 19 November 2024 10:44

Incident End: 19 November 2024 12:24

Duration: 1 hour, 40 minutes

Affected Services and Features:

Google Calendar

Regions/Zones: Global

Description:

Google Calendar experienced service issues, preventing some web users from accessing their calendars for a duration of 1 hour, 40 minutes. From preliminary analysis, the root cause of the issue was a misconfigured traffic management rule that incorrectly blocked legitimate traffic.

Google engineers mitigated impact by rolling back the misconfigured traffic management rule that was blocking legitimate traffic.

Google will complete a full IR in the following days that will provide a full root cause.

Customer Impact:

The issue affected Google Calendar Web users and was triggered when users attempted to transfer calendar event ownership. Once this happened, one or more users were unable to access any part of their calendar through the website.

Calendar mobile application users were not affected.

Summary: Google Calendar experienced service issues, preventing some web users from accessing their calendars.

Description:

On November 19, 2024, between 10:44 AM and 12:24 PM US/Pacific, a subset of Google Calendar users encountered problems accessing the Calendar web UI.

We sincerely apologize for any inconvenience this may have caused our valued customers.

Diagnosis:

The issue affected Google Calendar Web users and was triggered when users attempted to transfer a calendar event ownership. Once this happened, one or more users were unable to access any part of their calendar through the website.

The issue did not affect the Calendar mobile application

Workaround: There were no workarounds at the time of the issue