Grafana Cloud reports multiple service issues

Phone call notifications to numbers in Russia and Kazakhstan are still blocked. We are working to resolve the issue. As SMS support is limited in these countries, we recommend using mobile push notifications as an alternative. We do not currently have an estimated resolution time. We will provide further updates as more information becomes available.

This incident has been resolved.

Things are starting to recover, and we continue to monitor the issue.

This incident has been resolved.

We are investigating Intermittent write failures in tempo-prod (prod-us-central-0) due to Google Cloud Storage internal errors.

Things are starting to stabilize as we continue to work on a mitigation.

We are continuing to work on a fix for this issue. Next update by 16:30 UTC

We observed some spikes in latency starting at approximately 12:50 UTC. We are working on a resolution. Next update by 15:30 UTC

This incident has been resolved.

The fix has been implemented, and we are monitoring the results.

We have noticed improvements across services. We will now be applying a more permanent fix. Impacted users may experience a disruption in the auth-api service of up to 15min while we apply a DB patch that will improve performance.

The issue has been identified, and we have applied mitigations on our end. We are currently observing the progress.

We are investigating increased authentication errors affecting Grafana Cloud in the prod-us-west-0 region. Customers may experience intermittent failures for requests that require authentication. Our engineering team has identified database performance degradation affecting the authentication service and is investigating the underlying cause. We will provide another update as more information becomes available.

Phone call notifications to numbers in Russia and Kazakhstan are currently blocked. We are working to resolve the issue. As SMS support is limited in these countries, we recommend using mobile push notifications as an alternative. We do not currently have an estimated resolution time. We will provide further updates as more information becomes available.

This incident has been resolved. Thank you for your patience.

We have put some mitigations in place which seem to have helped with the errors. We continue to monitor the results.

We are continuing to investigate this issue.

A small number of customers with stacks in prod-us-central-4 are receiving "forbidden" errors when querying Prometheus data sources. We are investigating and will provide an update soon.

Phone call notifications to numbers in Russia and Kazakhstan are currently blocked. We are working to resolve the issue. As SMS support is limited in these countries, we recommend using mobile push notifications as an alternative. We do not currently have an estimated resolution time. We will provide further updates as more information becomes available.