Monday.com Investigating Platform Latency Issues

postmortem Jul 7, 9:28 PM

**What Happened:** Today, following a gradual rollout of a new feature, our platform experienced widespread cross-system performance degradation. While most features continued to work, a heavy load on our database impacted overall service availability, causing automations to run slowly and the automations store and agents to occasionally fail to load. **Please note that at no point was any data lost or compromised.** **Timeline:** * **US Region:** 15:30 – 17:10 UTC \(1 hour and 40 minutes of degradation\) ‌ **What caused it:** A new Dynamic Client Registration \(DCR\) feature for OAuth \(supporting MCP applications\) caused an unexpected chain reaction. Even with the limited rollout of this feature, frequent automated DCR requests triggered a continuous, global cache invalidation storm across our core caching infrastructure. With this shared cache repeatedly cleared, all traffic was directed straight to the database, causing a severe CPU spike. ‌ **How we responded:** Our engineering team identified the source and immediately performed a significant system scale-up to handle the extreme load. We then disabled the DCR endpoint in production. This stopped the cache invalidation loop, restoring database stability and returning system performance to normal. **What we're doing to prevent recurrence:** Moving forward, we plan to use a setup that will take the load off core caching systems. Because this was a limited gradual rollout, turning it off has zero impact on your current workflows or integrations. ‌ We sincerely apologize for any disruption this caused to your workflow. Thank you for your understanding as we work to continuously improve our platform's reliability. Your team at [**monday.com**](http://monday.com)

postmortem Jul 7, 9:28 PM

**What Happened:** Today, following a gradual rollout of a new feature, our platform experienced widespread cross-system performance degradation. While most features continued to work, a heavy load on our database impacted overall service availability, causing automations to run slowly and the automations store and agents to occasionally fail to load. **Please note that at no point was any data lost or compromised.** **Timeline:** * **US Region:** 15:30 – 17:10 UTC \(1 hour and 40 minutes of degradation\) ‌ **What caused it:** A new Dynamic Client Registration \(DCR\) feature for OAuth \(supporting MCP applications\) caused an unexpected chain reaction. Even with the limited rollout of this feature, frequent automated DCR requests triggered a continuous, global cache invalidation storm across our core caching infrastructure. With this shared cache repeatedly cleared, all traffic was directed straight to the database, causing a severe CPU spike. ‌ **How we responded:** Our engineering team identified the source and immediately performed a significant system scale-up to handle the extreme load. We then disabled the DCR endpoint in production. This stopped the cache invalidation loop, restoring database stability and returning system performance to normal. **What we're doing to prevent recurrence:** Moving forward, we plan to use a setup that will take the load off core caching systems. Because this was a limited gradual rollout, turning it off has zero impact on your current workflows or integrations. ‌ We sincerely apologize for any disruption this caused to your workflow. Thank you for your understanding as we work to continuously improve our platform's reliability. Your team at [**monday.com**](http://monday.com)

resolved Jul 7, 5:30 PM

This incident has been resolved.

resolved Jul 7, 5:30 PM

This incident has been resolved.

monitoring Jul 7, 5:14 PM

A fix has been implemented and we are monitoring the results.

monitoring Jul 7, 5:14 PM

A fix has been implemented and we are monitoring the results.

monitoring Jul 7, 5:14 PM

A fix has been implemented and we are monitoring the results.

investigating Jul 7, 4:40 PM

We are continuing investigating the issues related to platform not loading. This is also affecting automations, boards, board views, dashboards, AI features, my work, item creation.

investigating Jul 7, 4:40 PM

We are continuing investigating the issues related to platform not loading/working. This is also affecting automations, boards, board views, dashboards, AI features, my work, item creation, workspaces.

investigating Jul 7, 4:40 PM

We are continuing investigating the issues related to platform not loading/working. This is also affecting automations, boards, board views, dashboards, AI features, my work, item creation, workspaces.

investigating Jul 7, 4:40 PM

We are continuing investigating the issues related to platform not loading/working. This is also affecting automations, boards, board views, dashboards, AI features, my work, item creation, workspaces.

investigating Jul 7, 4:02 PM