Our upstream provider has implemented a fix and we are monitoring the results. Customers can resume publishing of their sites.
Posted Nov 10, 2021 - 04:19 UTC
Update
Our provider has applied mitigations for their issues. Our monitors indicate that errors have significantly decreased although some requests are in degraded performance and may take longer to complete. Our provider expects full recovery within the next 30 minutes.
Current Scope of impact includes some hosted sites, asset loading, new publishes, and asset upload.
The next update will be posted as new information becomes available.
Posted Nov 10, 2021 - 03:47 UTC
Update
We are continuing to receive updates from our upstream provider but still have no ETA.
Current Scope of impact includes some hosted sites, asset loading, new publishes, and asset upload.
Our current monitors indicate this is affecting less than 10% of all requests
We recommend that no customers attempt to publish sites as of right now so our caching layer can continue to serve currently published sites.
The next update will be posted as new information becomes available.
Posted Nov 10, 2021 - 01:24 UTC
Update
We are continuing to receive updates from our upstream provider but still have no ETA. They've communicated that error rates have stabilized and they are continuing to work to mitigate the problem.
Current Scope of impact includes some hosted sites, asset loading, new publishes, and asset upload.
Our current monitors indicate this is affecting less than 10% of all requests
We recommend that no customers attempt to publish sites as of right now so our caching layer can continue to serve currently published sites.
Next update will be posted in 1 hour or as new information becomes available.
Posted Nov 10, 2021 - 00:26 UTC
Update
Our team has restored site preview images on the dashboard and site searches and we will continue to monitor them. As of right now, they are fully operational again. This was caused by an internal configuration issue that coincided with the timeframe of our upstream provider and was unrelated to site published and assets as originally thought.
We continue to receive to receive updates from our upstream provider but still have no ETA.
Current Scope of impact includes some hosted sites, asset loading, new publishes, and asset upload.
Our monitors indicate this is affecting ~10% of all requests
We recommend that no customers attempt to publish sites as of right now so our caching layer can continue to serve currently published sites.
Next update will be posted as new information becomes available.
Posted Nov 09, 2021 - 23:29 UTC
Update
We are continuing to receive to receive updates from our upstream provider but still have no ETA. Their initial attempt to mitigate the issue have failed and they are now exploring additional options to resolve the issue.
Current Scope of impact includes some hosted sites, internal site searches, asset loading, new publishes, asset upload, and site screenshots used for preview images in the dashboard.
Our current monitors indicate this is affecting ~10% of all requests
We recommend that no customers attempt to publish sites as of right now so our caching layer can continue to serve currently published sites.
Next update will be posted in 1 hour or as new information becomes available.
Posted Nov 09, 2021 - 23:09 UTC
Update
We are continuing to receive to receive updates from our upstream provider but still have no ETA.
Current Scope of impact includes some hosted sites, internal site searches, asset loading, new publishes, asset upload, and site screenshots used for preview images in the dashboard.
Our current monitors indicate this is affecting ~10% of all requests
We recommend that no customers attempt to publish sites as of right now so our caching layer can continue to serve currently published sites.
Next update will be posted in 30 mins or as new information becomes available.
Posted Nov 09, 2021 - 22:33 UTC
Update
We continue to wait for updates from our upstream provider with no ETA.
Current Scope of impact includes some hosted sites, internal site searches, asset loading, new publishes, asset upload, and site screenshots used for preview images in the dashboard.
Our current monitors indicate this is affecting ~10% of all requests
We recommend that no customers attempt to publish sites as of right now so our caching layer can continue to serve currently published sites.
Next update will be posted in 30 mins or as new information becomes available.
Posted Nov 09, 2021 - 22:04 UTC
Identified
Our upstream provider has confirmed elevated error rates for file upload and retrievals. We are actively working with them and awaiting updates.
In the meantime customers may experience issues with hosted site availability, asset loading, new publishes, and asset upload. We do not yet have an ETA but will send another update in 20 mins.
Posted Nov 09, 2021 - 21:40 UTC
Update
We have identified an issue with an upstream provider and are working with them to determine the nature of the problem
Posted Nov 09, 2021 - 21:17 UTC
Investigating
We are seeing reports that some sites are failing to publish. All currently published sites remain available. Our team is currently investigating the issue and will continue to post updates as we determine the root cause.
Posted Nov 09, 2021 - 21:11 UTC
This incident affected: Hosted Websites (Static pages, Ecommerce) and Dashboard, Publishing, API Status.