Salesforce fell over so hard today, it took out its own server status page • The Register
Salesforce fell over so hard today, it took out its own server status page
It’s not DNS. There is no way it’s DNS. It was DNS
Chris Williams, Editor in Chief Wed 12 May 2021 // 02:10 UTC SHARE
UPDATED Salesforce is digging itself out of a multi-hour outage right now that it has blamed on a DNS issue.
At one point today, the IT breakdown was so severe that its status page was pretty much inaccessible for netizens, and staff resorted to posting updates on their help and training sub-site.
"Salesforce is experiencing a major disruption due to what we believe is a DNS issue causing our service to be inaccessible," CTO Parker Harris said in a statement. "We recognize the significant impact on our customers and are actively working on resolution.
"We believe we have isolated the issue and are in the process of bringing affected services back online. Customers may continue to experience issues as we work through remediation. This remains a top priority for Salesforce."
The downtime started shortly before 2200 UTC (1500 PT), knocked pretty much the entire software-as-a-service giant offline, and right now, everything is not yet back to normal. Though products have been restored, we're told, customers can't login if they are using multi-factor authentication.
Microsoft says Outlook hit by 'email visibility issues' – as in, they're blank
Trend Micro hosted email service is down, inboxes still stuck in cloudy limbo
British bank TSB says it will fix days-long transaction troubles tonight
Imagine your data center backup generator kicks in during power outage ... and catches fire. Well, it happened
"At 2146 UTC on May 11, 2021, the Salesforce technology team became aware of an issue impacting multiple Salesforce services," the CRM goliath noted on its status page.
"Customers will experience issues while navigating the Core application, Marketing Cloud, Commerce Cloud, and Experience Cloud (formerly known as Communities).
"The issue is also impacting the Salesforce Trust site as the status.salesforce.com page is only intermittently accessible."
About half an hour ago, the biz added:
The technology team continues to manually restore service. We currently don’t have an estimated time on when full recovery is expected. The team is confident that this incident was the result of an internal DNS change.
The Government Cloud environment is now out of impact. Marketing Cloud, Commerce Cloud, and Heroku services have been restored, however, customers using multifactor authentication will still be unable to log in.
Just as we were going to press, Parker tweeted: "Services are significantly restored though we are still not out of impact." ®
Updated to add on May 12
The outage is now over, and Salesforce has given some details on the cause of the IT breakdown:
The technology team has resolved the issue and called an all-clear as of 02:20 UTC on May 12, 2021.
The team determined the root cause was related to the implementation of an emergency fix that triggered a software issue and caused a DNS network incident. To resolve the issue, the team manually restored DNS service on a data center by data center basis until normal service levels were restored.
It’s not DNS. There is no way it’s DNS. It was DNS
Chris Williams, Editor in Chief Wed 12 May 2021 // 02:10 UTC SHARE
UPDATED Salesforce is digging itself out of a multi-hour outage right now that it has blamed on a DNS issue.
At one point today, the IT breakdown was so severe that its status page was pretty much inaccessible for netizens, and staff resorted to posting updates on their help and training sub-site.
"Salesforce is experiencing a major disruption due to what we believe is a DNS issue causing our service to be inaccessible," CTO Parker Harris said in a statement. "We recognize the significant impact on our customers and are actively working on resolution.
"We believe we have isolated the issue and are in the process of bringing affected services back online. Customers may continue to experience issues as we work through remediation. This remains a top priority for Salesforce."
The downtime started shortly before 2200 UTC (1500 PT), knocked pretty much the entire software-as-a-service giant offline, and right now, everything is not yet back to normal. Though products have been restored, we're told, customers can't login if they are using multi-factor authentication.
Microsoft says Outlook hit by 'email visibility issues' – as in, they're blank
Trend Micro hosted email service is down, inboxes still stuck in cloudy limbo
British bank TSB says it will fix days-long transaction troubles tonight
Imagine your data center backup generator kicks in during power outage ... and catches fire. Well, it happened
"At 2146 UTC on May 11, 2021, the Salesforce technology team became aware of an issue impacting multiple Salesforce services," the CRM goliath noted on its status page.
"Customers will experience issues while navigating the Core application, Marketing Cloud, Commerce Cloud, and Experience Cloud (formerly known as Communities).
"The issue is also impacting the Salesforce Trust site as the status.salesforce.com page is only intermittently accessible."
About half an hour ago, the biz added:
The technology team continues to manually restore service. We currently don’t have an estimated time on when full recovery is expected. The team is confident that this incident was the result of an internal DNS change.
The Government Cloud environment is now out of impact. Marketing Cloud, Commerce Cloud, and Heroku services have been restored, however, customers using multifactor authentication will still be unable to log in.
Just as we were going to press, Parker tweeted: "Services are significantly restored though we are still not out of impact." ®
Updated to add on May 12
The outage is now over, and Salesforce has given some details on the cause of the IT breakdown:
The technology team has resolved the issue and called an all-clear as of 02:20 UTC on May 12, 2021.
The team determined the root cause was related to the implementation of an emergency fix that triggered a software issue and caused a DNS network incident. To resolve the issue, the team manually restored DNS service on a data center by data center basis until normal service levels were restored.
