resolved Partial outage in AMS Oct 3, 8:57 AM
All impacted services are operational now.
monitoring Partial outage in AMS Oct 3, 8:35 AM
All MPG clusters have recovered. We are monitoring for additional impact.
monitoring Partial outage in AMS Oct 3, 8:19 AM
All hosts are back online. MPGv2 control plane is fully operational now. Some MPGv1 clusters in AMS might be degraded while MPGv1's control plane recovers.
identified Partial outage in AMS Oct 3, 8:08 AM
Those hosts are coming back online after a brief interruption. We are assessing the impact on MPG.
identified Partial outage in AMS Oct 3, 8:01 AM
Some of our hosts are offline in AMS. Control plane for Managed Postgres v1 and v2 in AMS is unavailable as a result of this. MPG clusters should remain online but any control plane ops will fail. Some machines in AMS are also unavailable.
resolved Delayed ingestion of app logs Oct 1, 2:49 AM
This incident has been resolved.
monitoring Delayed ingestion of app logs Oct 1, 2:35 AM
The backlog of app logs has been processed, and we're monitoring the health of the log cluster. No logs were lost.
identified Delayed ingestion of app logs Oct 1, 2:27 AM
We've resolved the ingestion issue and our systems are processing the backlog of app logs. No logs have been lost.
identified Delayed ingestion of app logs Oct 1, 1:39 AM
We've identified the cause of the slowed log ingestion and are working on a fix.
investigating Delayed ingestion of app logs Oct 1, 1:11 AM
We're investigating delayed ingestion of app logs across all regions. New logs from machines may not be visible, but past logs are still available.
resolved Depot builder failures Sep 15, 6:09 AM
This incident has been resolved.
monitoring Depot builder failures Sep 15, 5:07 AM
A fix has been implemented and we are monitoring builds. Standard flyctl builds should be working again for customers in all regions.
identified Depot builder failures Sep 15, 5:07 AM
We have identified an issue with deploying via Depot for users connecting through our SYD and JNB regions. Affected customers in these regions can deploy successfully using the --depot=false or --buildkit arguments to flyctl.
investigating Depot builder failures Sep 15, 5:07 AM
We are investigating reports of Depot builds failing for some customers. Affected customers can deploy successfully using the --depot=false or --buildkit arguments to flyctl.
resolved Network issues in US West Coast Sep 12, 11:09 PM
This incident has been resolved.
monitoring Network issues in US West Coast Sep 12, 10:07 PM
Private networking between Fly Machines is resolved, and most outbound connections are healthy. We're continuing to monitor the network, and some issues will still be expected from clients physically located in US West until upstream transit issues are resolved.
investigating Network issues in US West Coast Sep 12, 10:07 PM
We are investigating upstream network issues from US West Coast (SJC, LAX). Apps hosted in US West regions may experience higher latency or packet loss, and requests from clients physically located in US West may experience higher latency.
resolved Upstream network issues Sep 2, 4:11 PM
This incident has been resolved.
monitoring Upstream network issues Sep 2, 4:11 PM
A fix has been implemented and we are monitoring the results.
identified Upstream network issues Sep 2, 3:13 PM
We're updating the affected region list to also include SJC since this seems to be a wider upstream issue in US West Coast.
identified Upstream network issues Sep 2, 3:13 PM
We have put in some temporary mitigations along with our providers. However since the root cause of this issue lies within a bigger upstream transit provider, you may continue to see some elevated latency and connection issues in/around affected regions. We're still working closely with them to resolve the root cause.
identified Upstream network issues Sep 2, 3:13 PM
We have observed an upstream network issue in LAX. Connections to some destinations may see elevated latency and packet loss. We're working with our upstream to resolve this issue.
resolved Packet loss in ORD Aug 31, 9:10 AM
This incident has been resolved.
monitoring Packet loss in ORD Aug 31, 8:09 AM
Packet loss in ORD is improving and impacted services are recovering; we’re continuing to monitor for intermittent issues
investigating Packet loss in ORD Aug 31, 7:10 AM
Due to an upstream provider, we are seeing ~50% packet loss on a subset of hosts in ORD. Some MPG clusters in ORD are slow to replicate as a result.
resolved Networking Issues in GRU Aug 28, 11:11 PM
This incident has been resolved.
identified Networking Issues in GRU Aug 28, 11:11 PM
Our upstream provider has implemented a fix. Network performance in GRU has normalized.
identified Networking Issues in GRU Aug 28, 11:11 PM
We are seeing a recurrance in networking issues in GRU. Some apps in the region may experience increased latency or packet loss. We are working with our upstream networking provider to resolve.
monitoring Networking Issues in GRU Aug 28, 10:10 PM
Networking performance in GRU has normalized and we are no longer seeing issues. We are continuing to monitor to ensure a full recovery.
investigating Networking Issues in GRU Aug 28, 10:10 PM
We are investigating networking issues impacting some hosts in GRU (São Paulo, Brazil) region. Some apps in GRU may experience increased latency or packet loss.
resolved Increased packet loss Aug 28, 11:12 AM
This incident has been resolved.
investigating Increased packet loss Aug 28, 9:09 AM
We are currently investigating this issue.
resolved Metrics in some regions are lagging behi... Aug 24, 2:10 PM
This is now resolved
monitoring Metrics in some regions are lagging behi... Aug 24, 1:10 PM
All hosts have caught up with metrics and we're monitoring the situation
investigating Metrics in some regions are lagging behi... Aug 24, 11:09 AM
We are currently experiencing some metrics lag on servers in some regions. We are provisioning more metric processing instances to accommodate the backlog and catch up.
