resolved PDC Degraded performance Jul 9, 7:54 PM
This incident has been resolved. Thank you for your patience.
monitoring PDC Degraded performance Jul 9, 5:52 PM
We are now seeing recovery of PDC traffic to the affected customers. Our team will continue to monitor across shifts.
identified PDC Degraded performance Jul 9, 4:08 PM
We are continuing to see disruptions across multiple deployments, ranging from degraded performance to full outages for some customers.
Additional resources have been engaged to mitigate this issue. We will post updates as they become available.
monitoring PDC Degraded performance Jul 9, 2:44 PM
We are continuing to monitor for any further issues.
monitoring PDC Degraded performance Jul 9, 2:10 PM
We are seeing recovery in some deployments, while others not just yet. We are continuing to monitor this incident.
resolved Delayed ingestion and recording rule eva... Jul 9, 1:07 PM
This incident has been resolved.
monitoring PDC Degraded performance Jul 9, 1:04 PM
A fix has been applied and we are seeing recovery. We will continue to monitor this.
investigating PDC Degraded performance Jul 9, 12:36 PM
We are still currently investigating the issue.
investigating PDC Degraded performance Jul 9, 11:37 AM
We are currently facing performance degradation on PDC service hosted on Multiple clusters. Our Engineering Team is currently working on fixing the issue, we do apologize for any inconvenience.
monitoring Delayed ingestion and recording rule eva... Jul 9, 11:18 AM
A fix has been implemented. We are currently monitoring the results.
investigating Delayed ingestion and recording rule eva... Jul 9, 10:28 AM
We are observing delayed ingestion and recording rule evaluation failures for Mimir in prod-ap-south-1. As of yet we have not noticed any customer impact however we are currently observing the cell.
resolved Grafana rulers crash-looping on promethe... Jul 8, 5:23 PM
This incident has been resolved.
resolved Loki Billing Data Lost Jul 8, 3:33 PM
Between 13:57 and 14:20 UTC (23 minutes), a data gap occurred affecting billing usage data for Loki. Data for this window was not recorded and cannot be recovered.
monitoring Grafana rulers crash-looping on promethe... Jul 8, 2:23 PM
We are in the process of rolling out the fix. Our engineers are monitoring the progress.
investigating Grafana rulers crash-looping on promethe... Jul 8, 12:17 PM
We are investigating issues with the grafana-ruler service on prod-us-east-2 and prod-us-west-0 which are causing periodic crash conditions. A code fix is currently being deployed to mitigate this
resolved Cannot access dashboard set up as a home... Jul 8, 11:00 AM
We have found that the issues are related to the grafana.unifiedHomepage feature rollout active 16:50 UTC yesterday, to 10:20 UTC today. We have rolled this back and systems are now working as expected.
investigating Cannot access dashboard set up as a home... Jul 8, 10:28 AM
We are currently investigating an issue affecting some customers who are unable to load or access their dashboards when configured as their home page. We will update once we have more information on this.