Rackspace Email & Apps System Status Page
Support Home | System Status
Our award-winning Fanatical Support® team is always available! Call or chat with your dedicated support team at anytime by logging into your control panel.
< Back to DashboardHosted Exchange Issues
Thursday Jan 12, 2023
Hosted Exchange Disruption
05:41 PM EST
01/05/23
Our Racker team has been hard at work over the holidays and into the New Year to support recovery efforts. We have recently completed our forensic investigation and are now in a position to share more information about the root cause and full scope of the incident.
While there has been widespread speculation that the root cause of this incident was the result of the ProxyNotShell exploit, we can now definitively state that is not accurate. We have been diligent about this forensic investigation and prioritizing accuracy and precision in everything we say and do, because our credibility is important to us at Rackspace.
The forensic investigation determined that the threat actor, known as PLAY, used a previously unknown security exploit to gain initial access to the Rackspace Hosted Exchange email environment. This zero-day exploit is associated with CVE-2022-41080. Microsoft disclosed CVE-2022-41080 as a privilege escalation vulnerability and did not include notes for being part of a Remote Code Execution chain that was exploitable.
We urge all organizations and security teams to read the blog CrowdStrike recently published about this exploit and learn how to take action to protect your own organization, available at https://www.crowdstrike.com/blog/owassrf-exploit-analysis-and-recommendations/. We will be sharing more detailed information with our customers and peers in the security community so that, collectively, we can all better defend against these types of exploits in the future.
As a reminder, no other Rackspace products, platforms, solutions, or businesses were affected or experienced downtime due to this incident.
Of the nearly 30,000 customers on the Hosted Exchange email environment at the time of the attack, the forensic investigation determined the threat actor accessed a Personal Storage Table ("PST") of 27 Hosted Exchange customers. We have already communicated our findings to these customers proactively, and importantly, according to CrowdStrike, there is no evidence that the threat actor actually viewed, obtained, misused, or disseminated emails or data in the PSTs for any of the 27 Hosted Exchange customers in any way. Customers who were not contacted directly by the Rackspace team can be assured that their PST data was not accessed by the threat actor.
Notably, this information does not impact the ongoing process of recovering historical email data for our Hosted Exchange email customers. As of today, more than half of impacted customers have some or all of their data available to them for download. However, less than 5% of those customers have actually downloaded the mailboxes we have made available. This indicates to us that many of our customers have data backed up locally, archived, or otherwise do not need the historical data. We will continue working to recover all data possible as planned, however, in parallel, we are developing an on-demand solution for those customers who do still wish to download their data. We expect that the on-demand solution will be available within two weeks.
As a reminder, we are proactively notifying customers for whom we have recovered greater than 50% of their mailboxes. Those PST files are now available through the customer portal. To check if your historical email data is available, please follow Step 2 on our Data Recovery Resources page (https://www.rackspace.com/hosted-exchange-incident-data-recovery-resources) and see if your mailbox is ready to download. As a reminder, we have prepared additional support resources that are available on our landing page (https://www.rackspace.com/hosted-exchange-incident), but if your data is available and you are having trouble downloading it, please contact our support channels by either joining us in chat or by calling +1 (855) 348-9064 (INTL: +44 (0) 203 917 4743). and we will be happy to assist you.
Finally, the Hosted Exchange email environment will not be rebuilt as a go-forward service offering. Even prior to the recent security incident, the Hosted Exchange email environment had already been planned for migration to Microsoft 365, which has a more flexible pricing model, as well as more modern features and functionality. There will be no price increase for our Hosted Exchange customers if they choose to move to Microsoft 365 and select a plan with the same capabilities as they currently have. Every Hosted Exchange customer has the option to migrate and pay exactly what they are paying today or even slightly lower costs and have the same capabilities. Also, Rackspace Email continues to be unaffected and is an alternative option for customers who do not wish to migrate to Microsoft 365. Rackspace will continue to assist customers with choosing the best plan to meet their needs depending on the capabilities required for their businesses.
As the forensic investigation has now concluded, we will no longer be posting updates to this status page. Our customer support teams will continue to work directly with customers to make their data available for download and remain on standby for any additional customer questions. Data recovery for our customers remains the top priority for Rackspace, and we will continue to work around the clock on this effort.
While the Hosted Exchange email environment was a small part of our business, it represents thousands of long-time and loyal customers whom we deeply value. We sincerely thank all of our Hosted Exchange customers for their continued patience and trust in us throughout this process and will continue to work hard to maintain the relationships we have built with them over the years.
10:15 AM EST
12/27/22
Our data recovery process for our Rackspace Hosted Exchange email customers is currently progressing as expected.
As the process remains underway, we want to remind customers that due to the nature of the incident, certain elements of email and other data may remain unavailable to our customers. The recovery process is limited to historic, pre-December 2, 2022, Hosted Exchange email data. Email data received after December 2, 2022 is available for those customers who chose to migrate to a new email service (either the offered Microsoft 365 service provided by Rackspace, or a third party alternative), those who implemented email forwarding as suggested, or who purchased archiving services.
Additionally, for customers who set up email forwarding, historical email data that has been forwarded will not be available through the data recovery process, but can be found in the archives of the forwarding address.
Our internal and external cybersecurity experts have and continue to work diligently to streamline the data recovery process through the dedicated data recovery workstream. These experts have been working meticulously through an extensive and systematic process to recover data on the Hosted Exchange email environment servers so that we can extract email data and deliver the recoverable data to our customers in an organized and secure manner.
As a reminder, you can find additional support resources on our landing page linked here: https://www.rackspace.com/hosted-exchange-incident-data-recovery-resources. Our customer support Rackers are available to answer your questions about the email data transfer process.
Thank you for your patience. 07:14 PM EST
12/22/22
Rackspace is pleased to share that we have seen success with the email data recovery process for our Hosted Exchange email customers. We are proactively notifying customers for whom we have recovered greater than 50% of their mailboxes. Those PST files are now available through the customer portal.
We are still working meticulously to upload the remaining data into the portal. Once available for download, the PST files will be available through the customer portal for 30 days.
To check if your historical email data is available, please follow Step 2 on our Data Recovery Resources page (https://www.rackspace.com/hosted-exchange-incident-data-recovery-resources) and see if your mailbox is ready to download.
In order for Rackspace to contact you, please ensure your contact information is updated with a working email address. Instructions are available here: (https://docs.rackspace.com/support/how-to/manage-cloud-office-administrator-information-in-the-control-panel/).
As a reminder, we have prepared additional support resources that are available on our landing page (https://www.rackspace.com/hosted-exchange-incident).
Rackers are standing by to answer your questions about this process should you need assistance transferring your email data to your new environment.
We will continue to provide updates as appropriate. 06:29 PM EST
12/21/22
Thank you for your patience as we have worked expeditiously to secure, clean, and recover historical email data for our Hosted Exchange email environment customers.
Rackspace is pleased to announce that we are now in the position to begin handing PST files over to customers. We appreciate your patience, and we want to provide you with information on what Rackspace has been doing behind the scenes to prepare for the customer data recovery process.
Rackspace has devoted an entire workstream to this effort, with numerous internal and external experts contributing to the data recovery team. These experts have been working meticulously through an extensive and systematic process to recover data on the Hosted Exchange email environment servers so that we can extract email data and deliver the recovered data to our customers in an organized and secure manner.
We are pleased to share that our diligent preparation for the customer data recovery process is complete. We now look forward to initiating the customer email data recovery process to each of our Hosted Exchange email customers. We will be making historical email data available on PST files through the portal for each customer. Please note that while not all PST files are available at this time, we will update them in due course. If you are a Hosted Exchange email customer, you will receive additional information via email about this important update and how you will be able to download and import your historical emails.
Rackers are standing by to answer your questions about this process should you need assistance transferring your email data to your new environment. We have also prepared additional support resources that are available on our landing page found here: https://www.rackspace.com/hosted-exchange-incident-data-recovery-resources.
Please note, this does not mean that every piece of historical email data has been recovered at this time. Data included in public folders will not be available through this process. Additionally, once available for download, data will be available through your customer portal for 30 days. 09:29 PM EST
12/20/22
Rackspace is now in a position to begin testing our procedures to facilitate a successful handover of email data to Hosted Exchange email customers. Our engineers have been working through a rigorous process to recover, test, and secure this data, and have implemented a number of security controls and processes to protect the security of the data and the associated transfer process. We look forward to providing data access instructions to our Hosted Exchange email environment customers at scale soon.
07:06 PM EST
12/19/22
We continue to make progress on our data recovery efforts and look forward to beginning the process of transferring data to our Hosted Exchange email environment customers in the coming days. Rackspace engineers are diligently preparing for this process, and our customer support team will be available to customers as they begin to recover their data. Additionally, support resources for customer use will be available on our landing page once this process begins.
As a reminder, this process is being performed with great care and will likely take some additional time to complete. We understand the urgency of this situation - getting our customers their data securely and efficiently is our priority. Thank you for your patience. 08:55 PM EST
12/18/22
We understand how important data recovery is to our customers, and we want to provide you with information on what Rackspace has been doing behind the scenes to prepare for the customer data recovery process, which we expect to start soon. At this time, we can share the following about our preparation for the customer data recovery process:
Secure & Validate
> Our systems were restored in an isolated environment. Once they were restored, our experts installed Falcon, CrowdStrike's endpoint detection and monitoring tool, onto every impacted server for enhanced visibility across the environment.
> We worked carefully to put proper security protocols in place and, while it took time, we were extraordinarily diligent throughout this process. Due to the complexity of this work, each impacted device required significant attention to examine and process it. Again, this effort was time consuming, but it was a necessary step for us to take to validate security controls.
> Following the manual removal of malicious files and additional scans to validate that each server was clean, we then released the servers with Falcon deployed on them into a clean environment and tagged them as ready for the next phase of the process.
Recovering Servers
> After re-enabling, securing, and validating each server, our team of internal experts, including some of our most experienced engineers at Rackspace, worked around the clock to recover the data on those servers. This process is now complete.
Preparing the New Environment
> Once systems are recovered and have CrowdStrike's endpoint monitoring running on them, we place them into a clean environment that is separate and apart from the rest of the Rackspace network.
> Upon validation from our cybersecurity experts, we initiate the process of releasing servers in the new clean environment to prepare for data extraction.
Extraction
> After the servers are cleared for extraction, Rackspace has created automation that opens the exchange database files and reviews the details of each individual PST file, then correlates it to a customer account. The correlated files are then routed to a staging environment, from which data will be extracted and released to customers by account.
> We have been exploring every possible avenue to maximize speed while also prioritizing security. This includes writing code for the extraction process which was reviewed, evaluated and optimized multiple times by our internal and external experts.
> Our experts are also testing the restoration of data to ensure that PST files can be successfully transferred and validated.
> This does not mean that every PST will be recoverable as corruption can occur, particularly in larger files. However, we maintain secure copies of data for added redundancy and in the hopeful non-frequent case of PST corruption, will leverage these backups to potentially provide missing email data to customers where possible.
As this process continues to progress, we look forward to initiating data recovery for each customer by distributing PST files to them through their secure customer portal. Our Rackspace support team will be available as customers begin to recover their data, and support resources will be available on our landing page soon. We will be notifying customers in the coming days as their PST files become available for download in the secure control panel. 06:14 PM EST
12/17/22
We continue to make significant progress in our email data recovery efforts and are planning to begin transferring email data to our Hosted Exchange customers in the next few days. At this time, Rackspace engineers are continuing to extract data off of impacted servers, and move the data to a clean environment, where it is continually tested for security and availability.
Our engineers are also testing the processes that we have developed for the email data transfer to ensure the transfer can be completed securely and efficiently. Based on the process of extracting data to date, we have high visibility into the available email data, and are confident that the majority of customers will receive their email data back.
We look forward to providing our customers with more information on the email data recovery process, which will include resources on how to complete the transfer, in the coming days. 07:35 PM EST
12/14/22
Rackspace Customers,
As we continue working to bring our remaining customers back online, we want to reiterate some additional information we shared at our recent webinar around the measures Rackspace and its partners have taken to secure the Hosted Exchange email environment, as well as the status of our investigation.
As soon as we detected suspicious activity within our systems, we followed our incident response plans and acted immediately to contain the threat. Our containment efforts included pulling all of our servers in the Hosted Exchange environment offline out of an abundance of caution and engaging cybersecurity experts, including industry leading global cybersecurity firm CrowdStrike and other cybersecurity experts to assist us with the investigation.
Thanks to work by our external and internal experts, we have increased visibility throughout the Hosted Exchange environment. Importantly, there have been no signs of attacker activity in the environment since December 2, 2022 and there is no evidence that the attackers were able to move laterally beyond the Hosted Exchange environment.
Regarding our investigation into how this incident occurred and who was behind it, we can confirm that this was a financially motivated threat actor. Rackspace is working with CrowdStrike, other cybersecurity experts, and federal authorities to finalize the investigation as expeditiously as possible. This investigation is in the process of concluding, and we look forward to sharing more information with our customers. We appreciate your patience as we conclude these important investigations. It is critical that we do verify and present confirmed facts, rather than speculation.
We continue to make all of our internal and external resources available to provide support to the remaining Hosted Exchange customers, including additional surge staff and a Microsoft Fast Track team who has deployed to supplement our Rackspace work force. If you have not yet transitioned to Microsoft Office 365 or have not fully completed the transition, please leverage our support channels by either joining us in chat or by calling +1 (855) 348-9064 (INTL: +44 (0) 203 917 4743). Wait times continue to average less than 30 minutes.
For resources, frequently asked question, and updates about this incident, please visit our landing page here: https://www.rackspace.com/hosted-exchange-incident
Thank you to our valued customers for their patience during this time.
08:44 AM EST
12/14/22
Rackspace is continuing to make all of our internal and external resources available to provide support to the remaining Hosted Exchange customers, including additional surge staff and a Microsoft Fast Track team deployed to supplement our Rackspace work force. At this time we have transitioned more than two thirds of our customers to the Microsoft 365 environment.
As a reminder, if you have not yet transitioned to Microsoft 365 or have not fully completed the transition, please leverage our support channels by either joining us in chat or by calling +1 (855) 348-9064 (INTL: +44 (0) 203 917 4743). Wait times continue to average less than 30 minutes.
Please know that we are also continuing to work alongside external resources on our data recovery efforts. We understand how important data recovery is to our customers. In ransomware attacks, data recovery efforts do necessarily take significant time, both due to the nature of the attack and need to follow additional security protocols. We will continue to keep you updated on these efforts.
As an update on the ransomware attack itself, CrowdStrike has confirmed that they have obtained very good visibility throughout the entire Rackspace environment enabling CrowdStrike to confirm that the attack was limited to the Hosted Exchange environment. CrowdStrike has also confirmed that there have been no signs of attacker activity in the Hosted Exchange environment since the ransomware attack on December 2, 2022. We are also continuing to support the FBI's investigation into the attack.
We want to remind you that in situations like these, it's common for scammers and cybercriminals to try to take advantage of the public nature of this incident. Please be assured that while Rackers will continue to reach out to you to provide support in transitioning to Microsoft 365 and get your email back up and running, there are important ways that you can distinguish legitimate Racker outreach from unauthorized individuals claiming to be Rackers:
> Emails from Rackspace will only have the domain name @rackspace.com or @rackspace.co.uk without any special characters or numbers.
> Phone interactions with Rackspace support will not include requests for login credentials, or personal information. Please log in to your control panel to review pre-established security procedures.
> As a general reminder:
>> Do not open suspicious email attachments;
>> Do not click on suspicious links;
>> Ensure you recognize senders and email domains. Scammers will often try to mask emails to look legitimate.
Lastly, we wanted to draw your attention to our new landing page, launched late on December 13, 2022 - it features resources, frequently asked questions, and updates about the incident for your use. You can access the landing page here: https://www.rackspace.com/hosted-exchange-incident
Thank you again for your continued patience as we work through this incident. We greatly appreciate your support and will continue to communicate with you any time we have new information to share. 07:13 PM EST
12/12/22
We are continuing to diligently work on recovering data for our Hosted Exchange customers, which we know is very important to you. In ransomware attacks, data recovery efforts do necessarily take significant time both due to the nature of the attack and need to follow additional security protocols.
However, we want to remind our customers who have purchased the Barracuda archiving service that this incident has had no impact on that service, which continues to operate as usual due to Rackspace leveraging the best-in-class technology from our partner, Barracuda. If you have archived emails, they remain accessible. If you are not sure whether you have archived your emails or need additional support to access them, please contact [email protected].
Additionally, if Hosted Exchange customers have been using a desktop application, for example Microsoft Outlook, as their email client, a local copy of those customers' data may be available on your desktop computer.
Finally, if neither of these options are applicable to you, please know that we are engaging our internal technical experts, together with external data recovery resources, to diligently work on our data recovery efforts. We have made progress and will continue to keep you updated on these efforts.
If you need assistance with your Hosted Exchange account, please reach out via our support channels. Join us in chat or call us at +1 (855) 348-9064. (INTL: +44 (0) 203 917 4743). We are eager to support and assist you.
For your reference, we have produced a video tutorial that explains how to complete the transition that addresses some commonly asked questions regarding the Microsoft 365 transition process at the following link: https://www.rackspace.com/resources/hosted-exchange-assistance-video.
We appreciate your continued patience and apologize for the inconvenience this situation has had on our valued customers. 05:13 PM EST
12/11/22
While Rackspace continues to work diligently and expeditiously in getting all customers back up and running, our surged staff of Rackers and Microsoft Fast Track team is standing by to support our remaining Hosted Exchange customers in the transition to the more modern Microsoft 365. Wait times are currently averaging less than 30 minutes, so please leverage our support channels by either joining us in chat or by calling +1 (855) 348-9064. (INTL: +44 (0) 203 917 4743). We are eager to support and assist you.
For your reference, we have produced a video tutorial that explains how to complete the transition that addresses some commonly asked questions regarding the Microsoft 365 transition process at the following link: https://www.rackspace.com/resources/hosted-exchange-assistance-video.
As a reminder, there is a temporary solution available for customers while setting up Microsoft 365. It is possible to implement a forwarding option that will allow mail destined for a Hosted Exchange user to be routed to an external email address. Please log in to your customer account for a ticket with instructions to request this option. Notably, the forwarding rule will not apply retroactively to mail sent before the rule is put into place. This option can be used as a temporary solution while the customer sets up Microsoft 365. Once the customer has fully set up Microsoft 365 and updated their DNS MX records, this forwarding rule will no longer be needed.
This incident has had no impact on the email archiving service, which continues to operate as usual due to Rackspace leveraging the best-in-class technology from our partner, Barracuda. If you have archived emails they remain accessible. If you are not sure whether you have archived your emails or need additional support to access them, please contact [email protected].
Rackspace understands the importance of addressing this incident and has prioritized communication with customers, exploring every potential avenue to reach them, share the information that is known, and most importantly, get them access to email. As such, we continue to make significant progress in our recovery efforts and will provide additional updates as we have more information to share.
Thank you again to our valued customers, Rackers appreciate your continued patience and apologize for the inconvenience that this situation may have caused. 07:30 PM EST
12/10/22
While our team continues to work diligently in supporting our customers throughout the migration process, we want to take this opportunity to express our deep appreciation for you all. We understand the difficulties many of you have experienced during this process. Please know our dedicated team is working diligently and expeditiously with the goal of getting all customers back up and running.
As of yesterday, more than two-thirds of our customers on the Hosted Exchange environment are back on email. Every customer who has reached us has been offered support to transition to Microsoft 365.
If you still need support to get email, please leverage our support channels by either joining us in chat or by calling +1 (855) 348-9064. (INTL: +44 (0) 203 917 4743). Wait times are currently averaging less than 30 minutes. Our surged team of Rackers and Microsoft Fast Track partners are standing by to support our customers in transitioning to the more modern Microsoft 365.
For your reference, we have put together a video tutorial that explains how to complete the transition that addresses some commonly asked questions regarding the Microsoft 365 transition process at the following link: https://www.rackspace.com/resources/hosted-exchange-assistance-video.
Please know if you have archived emails, this incident has had no impact on the email archiving service. Those emails remain accessible and the service continues to operate as usual. To deliver this solution, Rackspace leverages best-in-class technology from our partner, Barracuda. If you are unsure whether you have archived your emails or need additional support to access them, please contact [email protected].
We pride ourselves in being a customer-first organization and will do everything we can to assist our customers throughout this process. We appreciate your patience and understanding during this incident and look forward to continuing to share more information. 06:27 PM EST
12/09/22
As of today, more than two-thirds of our customers on the Hosted Exchange environment are back on email. Every customer who has reached us has been offered support to transition to Microsoft 365. If you still need support to get email, please leverage our support channels by either joining us in chat or by calling +1 (855) 348-9064. (INTL: +44 (0) 203 917 4743). Wait times are currently averaging less than 3 minutes.
We have a surge team, and have partnered with Microsoft Fast Track, to assist customers through the transition process to Microsoft 365. We are proactively reaching out to customers who have not yet transitioned to Microsoft 365 to assist them in this process. To address some commonly asked questions regarding the Microsoft 365 migration process, a video tutorial that explains how to complete the transition is available at the following link: https://www.rackspace.com/resources/hosted-exchange-assistance-video.
Importantly, this incident has had no impact on the email archiving service, which continues to operate as usual. To deliver this solution, Rackspace leverages best in class technology from our partner, Barracuda. If you have archived emails they remain accessible. If you are not sure whether you have archived your emails or need additional support to access them, please contact [email protected].
We are continuing to make significant progress in our recovery efforts. We have engaged industry-leading global cybersecurity firm CrowdStrike to help investigate and remediate. Due to swift action on the Company's part in disconnecting its network and following its incident response plans, CrowdStrike has confirmed the incident was quickly contained and limited solely to the Hosted Exchange Email business.
We remain focused on understanding the root cause of the incident, and implementing additional security measures to defend against future cyber threats. We will continue to share additional updates on these measures as appropriate.
We appreciate your continued patience and apologize for the inconvenience this situation has had on our valued customers. 01:59 PM EST
12/08/22
Rackers are continuing to work diligently to get everyone back up and running and making all resources available to provide support to customers, including by deploying a Microsoft Fast Track team to supplement our workforce. Please be assured we will continue to work around the clock to limit the impact our customers are currently experiencing.
In situations like these, it's common for scammers and cybercriminals to try to take advantage. Please be assured that while Rackers will continue to reach out to you to provide support in transitioning to Microsoft 365 and get your email back up and running, there are important ways that you can distinguish legitimate Racker outreach from unauthorized individuals claiming to be Rackers:
> Emails from Rackspace will only have the domain name @rackspace.com without any special characters or numbers
> Phone interactions with Rackspace support will not include requests for login credentials, or personal information such as social security number or driver's license. Please log in to your control panel to review pre-established security procedures.
We want to take this opportunity to remind all customers of best practices for keeping your account safe. Please stay vigilant, change your passwords frequently, utilize different passwords across your personal and professional accounts, and monitor your banking account statements and credit report for suspicious activity.
As a reminder:
> Do not open any suspicious email attachments;
> Do not click on any suspicious links;
> Ensure you recognize the sender and the email domain. Scammers will often try to mask emails to look legitimate. Be wary of suspicious emails, including those with typos or claims of "urgent request"
If you do receive a message from an individual you do not recognize, do not reply. Please login to your control panel and create a ticket, including details about the message you received.
We understand that contact such as this may be alarming, but we currently have no evidence to suggest that you are at increased risk as a result of this direct contact.
Thank you. 09:11 PM EST
12/07/22
We are working diligently to meet our customers' needs regarding access to email and email forwarding. As noted in our recent updates, we have arranged for all Hosted Exchange customers to have access to Microsoft 365, which can be activated by using these instructions at the following link: https://docs.rackspace.com/support/how-to/how-to-set-up-O365-via-your-cloud-office-control-panel. To address some commonly asked questions regarding the Microsoft 365 migration process, a video tutorial that explains how to complete the transition is available at the following link: https://www.rackspace.com/resources/hosted-exchange-assistance-video.
We are also partnering with Microsoft's Fast Track team to add resources to our extended team to better assist customers with troubleshooting and any technical questions. This means we can reduce wait times and expedite service.
You can access our support channels by joining us in chat or by calling +1 (855) 348-9064. (INTL: +44 (0) 203 917 4743). 12:50 PM EST
12/07/22
Was this ransomware?
Yes.
How did the ransomware attack happen?
The investigation is ongoing and will take time to complete. Alongside our internal security team, we have engaged a leading cyber defense firm to investigate. The investigation is ongoing and is in its early stages.
Who was responsible for this incident?
Our investigation into the incident is ongoing and will take time to complete. To ensure the integrity of the ongoing investigation, we do not have additional details to share at this time.
What information might have been impacted?
The investigation is in its early stages. We will continue to investigate and share additional information with customers regarding the impact of this incident on customer data as needed.
Did you pay a ransom?
Our investigation into the incident is ongoing and will take time to complete. To ensure the integrity of the ongoing investigation, we do not have additional details to share at this time.
When will the investigation be complete?
Our investigation into the incident is ongoing and will take time to complete. We will continue to provide relevant updates as we have more information to share.
Are your systems secure?
Based on the investigation to date, we believe that this incident was isolated to our Hosted Exchange business. The Company's other products and services are fully operational, and we have not experienced an impact to our Rackspace Email product line and platform. Out of an abundance of caution, we have put additional security measures in place and will continue to actively monitor for any suspicious activity.
What further containment measures are you taking?
While the investigation is ongoing and in its early stages, out of an abundance of caution, our team of experts has put additional security measures in place and will continue to actively monitor for any suspicious activity.
What is your remediation and return to normal operations plan and schedule?
While the investigation is ongoing and in its early stages, at this time, we are unable to provide any timeline or expectations for restoration to the Hosted Exchange environment. We are working to provide customers with archives of inboxes where available.
What if I'm a customer who is currently finding access to email or support challenging?
We have assisted thousands of customers to date, and we want to assure any customers who still need to access their email that Rackers are working hard to get them access to email as soon as possible. We know that there have been long hold times, and in response, we've added surge capacity to our support staff to help walk customers through the migration process. We are also partnering with Microsoft's Fast Track team to add resources to our extended team to better assist customers with troubleshooting and any technical questions.
As a temporary option while customers set up Microsoft 365, it is possible to also implement a temporary forwarding rule that will allow mail destined for a Hosted Exchange user to be routed to an external email address. Please log in to your customer account for a ticket with instructions to request this option. Notably, the forwarding rule will not apply retroactively to mail sent before the rule is put into place. Once the customer has fully set up Microsoft 365 and updated their DNS MX records, this forwarding rule will no longer be needed.
While we are unable to provide any timeline or expectations for restoration to the Hosted Exchange environment at this time, we are working to provide customers with archives of inboxes where available. We have surged our support staff and will be taking additional steps to help guide our customers through this process in order to limit the impact to their own operations.
Have there been any disruptions outside of the Hosted Exchange environment?
Based on the investigation to date, we believe that this incident was isolated to our Hosted Exchange business. Our other products and services are fully operational, and we have not experienced an impact to our Rackspace Email product line and platform. Rackspace is making available resources so that customers can migrate their users and domains to Microsoft 365. 08:26 AM EST
12/06/22
We appreciate your patience as we continue to work through the security issues that have affected our Hosted Exchange environment. As you know, on Friday, December 2nd, 2022, we became aware of suspicious activity and immediately took proactive measures to isolate the Hosted Exchange environment to contain the incident. We have since determined this suspicious activity was the result of a ransomware incident.
Alongside our internal security team, we have engaged a leading cyber defense firm to investigate. Our investigation is still in its early stages, and it is too early to say what, if any, data was affected. If we determine sensitive information was affected, we will notify customers as appropriate.
Based on the investigation to date, we believe that this incident was isolated to our Hosted Exchange business. The Company's other products and services are fully operational, and we have not experienced any impact to our Rackspace Email product line and platform. Out of an abundance of caution, we have put additional security measures in place and will continue to actively monitor for any suspicious activity.
Rackspace is making available resources so that customers can migrate their users and domains to Microsoft 365. At this time, we are unable to provide a timeline for restoration of the Hosted Exchange environment. We are working to provide customers with archives of inboxes where available, to eventually import over to Microsoft 365.
As a temporary solution while you set up Microsoft 365, it is possible to also implement a forwarding option that will allow mail destined for a Hosted Exchange user to be routed to an external email address. Please log in to your customer account for a ticket with instructions to request this option. Customers should reply to the ticket to request the forwarding rule be put into place for each of their users.
If you do not see this ticket in your account and would like to take advantage of this option, please open a support ticket with the title: REQUESTING FORWARDING FOR HOSTED EXCHANGE. We will work with you to get this set up. NEW mail that is sent after the forwarding rule is put in place will be forwarded to the external address specified.
Notably, the forwarding rule will not apply retroactively to mail sent before the rule is put into place. This option can be used as a temporary solution while you set up Microsoft 365. Once you have fully set up Microsoft 365 and updated your DNS MX records, this forwarding rule will no longer be needed.
We understand the frustration this situation has caused for our customers and are doing everything we can to support them in migrating to Microsoft 365. We have surged our support staff and will be taking additional steps to help guide our customers through this process in order to limit the impact to their own operations.
We appreciate your continued patience and apologize for the inconvenience this situation has had on our valued customers. We are doing everything we can to make this right and will continue to provide updates and resources as available. 10:59 PM EST
12/05/22
We continue to help customers execute the immediate resolution path of leveraging Microsoft 365 and have helped thousands of customers move tens of thousands of users to this platform. We recognize that setting up and configuring Microsoft 365 can be challenging and we have added all available resources to help support customers. If you require assistance, please leverage our support channels by either joining us in chat or by calling +1 (855) 348-9064. (INTL: +44 (0) 203 917 4743). Call queue hold times are currently 2-3 hours, so we encourage you to take advantage of our callback feature to request a call. Your place in the queue will be held and you will receive a call when a Racker becomes available.
Many of our customers had previously subscribed to our Archive service for their mailbox and users. The archive service remains functional and any customer who had previously subscribed to Archive can follow the following steps to recover their archived mail as a .pst and import into their new M365 profile.
Login to Archive Manager (https://docs.rackspace.com/support/how-to/log-in-to-the-archive-manager/)
Create an Archive Search (https://docs.rackspace.com/support/how-to/create-an-archive-search)
Export the .pst file (https://docs.rackspace.com/support/how-to/export-archive-search-results-in-cloud-office)
Import the .pst file to your new M365 account
In assisting thousands of customers with moving to M365, one of the most challenging steps our customers have faced is reconfiguring their DNS for their domain. DNS can be complicated and for the vast majority of our customers, the authoritative nameserver (where you will need to make changes) is NOT Rackspace. Oftentimes, your authoritative nameserver will be with your registrar (where you originally purchased your domain).
If you are not sure who your authoritative nameserver is, you can use an online dig tool such as https://toolbox.googleapps.com/apps/dig/#NS/ to find it.
Visit the dig tool above, enter your domain name (without using www. or any other subdomain), and ensure you are searching for NS records. The results under TARGET is the name of your authoritative nameserver. If your authoritative nameserver looks unfamiliar, try searching for the results listed under TARGET, as it will often be your registrars default nameserver.
You will need access to modify your DNS records at your authoritative nameserver to complete the migration to M365 as is described in the instructions at https://docs.rackspace.com/support/how-to/how-to-set-up-O365-via-your-cloud-office-control-panel under the section Configuring DNS records for your Office 365 services steps 8-11.
We will update this status page as additional information becomes available.
01:28 AM EST
12/05/22
We have successfully restored email services to thousands of customers on Microsoft 365 and continue to make progress on restoring email service to every affected customer. At this time, moving to Microsoft 365 is the best solution for customers who can now also implement temporary forwarding. Following is additional information on the solution:
Set up and configure accounts on Microsoft 365 to immediately begin sending and receiving mail. You can achieve this by following the instructions here: https://docs.rackspace.com/support/how-to/how-to-set-up-O365-via-your-cloud-office-control-panel.
Your account administrator will need to manually set up each individual user on your account. Once your users have been set up and all appropriate DNS records are configured, their email access will be reactivated, and they will start receiving emails and can send emails. Please note, that DNS changes take approximately 30 minutes to provision and in rare cases can take up to 24 hours.
IMPORTANT: If you utilize a hybrid Hosted environment (Rackspace Email and Exchange on a single domain) then you will be required to move all mailboxes (Rackspace Email and Exchange) to M365 for mail flow to work properly. To preserve your data, it is critical that you do not delete your original mailboxes when making this change.
Self-migrating can be challenging so if you need assistance, please leverage our support channels by either joining us in chat or by calling +1 (855) 348-9064. (INTL: +44 (0) 203 917 4743).
You can also implement a temporary forwarding that will allow mail destined for a Hosted Exchange user to be routed to an external email address. Please log in to your customer account for a ticket with instructions to request this option. Customers should reply to the ticket to request the forwarding rule be put into place for each of their users.
If you do not see this ticket in your account and would like to take advantage of this option, please open a support ticket with the title: REQUESTING FORWARDING FOR HOSTED EXCHANGE. We will work with you to get this setup.
NEW mail that is sent after the forwarding rule is put in place will be forwarded to the external address specified.
The forwarding rule will not apply retroactively to mail sent before the rule is put into place. This option can be used as a temporary solution while you set up Microsoft 365.
Once you have fully set up Microsoft 365 and updated your DNS MX records, this forwarding rule will no longer be needed.
To assist customers through options, Rackers are contacting every Hosted Exchange customer by phone. Customers will also be contacted via alternate email addresses. This outreach is being performed in addition to chat, phone, and ticketing. Our support channels can be reached via chat or by calling (855) 348-9064 (INTL: +44 (0) 203 917 4743).
We will update this status page as additional information becomes available. 02:05 PM EST
12/04/22
Our teams continue to work diligently to restore email service to our customers and we are continuing to add resources to reduce wait times and increase response to tickets.
In addition to moving to Microsoft 365, customers can implement a temporary solution that will allow mail destined for a Hosted Exchange user to be routed to an external email address. Please log in to your customer account for a ticket with instructions to request this option. Customers should reply to the ticket to request the forwarding rule be put into place for each of their users. If you do not see this ticket in your account and would like to take advantage of this option, please open a support ticket with the title REQUESTING FORWARDING FOR HOSTED EXCHANGE and we will work with you to get this setup. NEW mail that is sent after the forwarding rule is put in place will be forwarded to the external address specified. The forwarding rule will not apply retroactively to mail sent before the rule is put into place. This option can be used as a temporary solution while you set up Microsoft 365. Once you have fully set up Microsoft 365 and updated your DNS MX records, this forwarding rule will no longer be needed.
To assist customers through options, Rackers are contacting every Hosted Exchange customer by phone. Customers will also be contacted via alternate email addresses. This outreach is being performed in addition to chat, phone, and ticketing. Our support channels can be reached via chat or by calling (855) 348-9064.
Please continue to check this status page for further updates.
Thank you for your continued patience and understanding. 12:37 AM EST
12/04/22
We appreciate your patience and understanding as we work diligently to seek to restore email services to every affected customer. The full extent of our support resources are dedicated to this issue.
We continue to make progress in addressing the incident. The availability of your service and security of your data is of high importance. We have committed extensive internal resources and engaged world-class external expertise in our efforts to minimize negative impacts to customers. We will continue to report our progress and update you as we have more information that we can share.
In order to best protect the environment, this will continue to be an extended outage of Hosted Exchange. At this time, moving to Microsoft 365 is the best solution for customers, and we highly encourage affected customers to move to this platform. Since our last update, we have been able to successfully restore email services to thousands of customers on Microsoft 365. As a reminder, instructions for how to migrate can be found here:
https://docs.rackspace.com/support/how-to/how-to-set-up-O365-via-your-cloud-office-control-panel
We are listening carefully to you throughout this incident. We have heard and understand that self-migration may not be simple and can be challenging to implement. Our customer support teams are working on a 24/7 basis to assist impacted customers. Our support channels can be reached via chat or by calling (855) 348-9064.
Current wait times for customer support are much longer than usual. For those who are finding the process challenging and are awaiting support, we ask for your patience as we increase staff to help every customer. Since our last update, we have mobilized roughly 1000 support Rackers to reduce wait times and address ticket queues. We will continue to accelerate and deploy even more resources to further help customers.
We thank you for your continued patience and understanding as we work to resolve the impacts of this incident. 02:31 PM EST
12/03/22
Our security and operations teams continue to work both internally and closely with outside experts to determine the full scope and impact of the issue involving our Hosted Exchange environment.
Since our last update, we have assisted numerous customers to open replacement Microsoft 365 accounts so they can resume sending and receiving emails. This remains our topmost priority. Our support teams across the company continue working to assist customers in all hands-on deck effort during this time. We are working diligently to source additional resources to help our customers over the weekend. If you need assistance, please contact our support team via our usual support channels.
Please continue to monitor our status page for the latest updates and FAQs: https://status.apps.rackspace.com/index/viewincidents?group=2.
Again, thank you for your patience. 01:57 AM EST
12/03/22
What happened?
On Friday, Dec 2, 2022, we became aware of an issue impacting our Hosted Exchange environment. We proactively powered down and disconnected the Hosted Exchange environment while we triaged to understand the extent and the severity of the impact. After further analysis, we have determined that this is a security incident.
The known impact is isolated to a portion of our Hosted Exchange platform. We are taking necessary actions to evaluate and protect our environments.
Has my account been affected?
We are working through the environment with our security teams and partners to determine the full scope and impact. We will keep customers updated as more information becomes available.
Has there been an impact to the Rackspace Email platform?
We have not experienced an impact to our Rackspace Email product line and platform. At this time, Hosted Exchange accounts are impacted, and not Rackspace Email.
When will I be able to access my Hosted Exchange account?
We currently do not have an ETA for resolution. We are actively working with our support teams and anticipate our work may take several days. We will provide updates as more information becomes available.
As a result, we are encouraging admins to configure and set up their users accounts on Microsoft 365 so they can begin sending and receiving mail immediately. If you need assistance, please contact our support team. We are available to help you set it up.
Is there an alternative solution?
At no cost to you, we will be providing access to Microsoft Exchange Plan 1 licenses on Microsoft 365 until further notice.
To activate, please use the below link for instructions on how to set up your account and users.
https://docs.rackspace.com/support/how-to/how-to-set-up-O365-via-your-cloud-office-control-panel
Please note that your account administrator will need to manually set up each individual user on your account. Once your users have been set up and all appropriate DNS records are configured, their email access will be reactivated, and they will start receiving emails and can send emails. Please note, that DNS changes take approximately 30 minutes to provision and in rare cases can take up to 24 hours.
IMPORTANT: If you utilize a hybrid Hosted environment (Rackspace Email and Exchange on a single domain) then you will be required to move all mailboxes (Rackspace Email and Exchange) to M365 for mail flow to work properly. To preserve your data, it is critical that you do not delete your original mailboxes when making this change.
I don't know how to setup Microsoft 365. How can I get help?
Please leverage our support channels by either joining us in chat or by calling +1 (855) 348-9064. (INTL: +44 (0) 203 917 4743).
Can I access my Hosted Exchange inbox from before the service was brought offline?
If you access your Hosted Exchange inbox via a local client application on your laptop or phone (like Outlook or Mail), your local device is likely configured to store your messages. However, while the Hosted Exchange environment is down, you will be unable to connect to the Hosted Exchange service to sync new mail or send mail using Hosted Exchange.
If you regularly access your inbox via Outlook Web Access (OWA), you will not have access to Hosted Exchange via OWA while the platform is offline.
As a result, we are encouraging admins to configure and set up their user's accounts on Microsoft 365 so they can begin sending and receiving mail immediately. If you need assistance, please contact our support team. We are available to help you set it up.
Will I receive mail in Hosted Exchange sent to me during the time the service has been shut down?
Possibly. We intend to update further as we get more information.
As a result, we are encouraging admins to configure and set up their user's accounts on Microsoft 365 so they can begin sending and receiving mail immediately. If you need assistance, please contact our support team. We are available to help you set it up.
08:19 PM EST
12/02/22
To our valued customers,
First and foremost, we appreciate your patience as we are working through the issue with your Hosted Exchange account, which we know impacted you greatly today. We experienced a significant failure in our Hosted Exchange environment. We proactively shut down the environment to avoid any further issues while we continue work to restore service. As we continue to work through the root cause of the issue, we have an alternate solution that will re-activate your ability to send and receive emails.
At no cost to you, we will be providing you access to Microsoft Exchange Plan 1 licenses on Microsoft 365 until further notice.
To activate, please use the below link for instructions on how to set up your account and users.
https://docs.rackspace.com/support/how-to/how-to-set-up-O365-via-your-cloud-office-control-panel
Please note that your account administrator will need to manually set up each individual user on your account. Once your users have been set up and all appropriate DNS records are configured, their email access will be reactivated, and they will start receiving emails and can send emails. Please note, that DNS changes take approximately 30 minutes to provision and in rare cases can take up to 24 hours.
IMPORTANT: If you utilize a hybrid Hosted environment (Rackspace Email and Exchange on a single domain) then you will be required to move all of your mailboxes (Rackspace Email and Exchange) to M365 in order for mail flow to work properly. To preserve your data, it is critical that you do not delete your original mailboxes when making this change.
Again, we apologize that this has been a major disruption to you, but we hope this will allow you to resume regular business as soon as possible.
Our support team is available to assist you via our usual support channels. Please reach out and continue to monitor our status page for further updates. Link to incident: https://status.apps.rackspace.com/index/viewincidents?group=2
Thanks again for your patience in this matter, we appreciate your business as a valued customer. 04:51 PM EST
12/02/22
To all of our valued customers, we understand the connectivity and login issues in our Cloud Office environments are greatly impacting you. We are working diligently to resolve the issue and it is currently our highest priority. Please continue to monitor our status page for the latest updates. Again, thank you for your patience, as we work to provide you a resolution soon. 04:01 PM EST
12/02/22
We are aware of an issue impacting our Hosted Exchange environments. Our Engineering teams continue to work diligently to come to a resolution. At this time we are still in the investigation phase of this incident and will update our status page as more information becomes available. 01:54 PM EST
12/02/22
We are aware of an issue impacting our Hosted Exchange environments. Our Engineering teams continue to work diligently to come to a resolution. At this time we are still in the investigation phase of this incident and will update our status page as more information becomes available.
09:38 AM EST
12/02/22
All hands are on the deck & right resources have been engaged and are actively working on the issue. All new updates will be posted here as they become available. 06:36 AM EST
12/02/22
We continue to investigate the connectivity and login issues to our Exchange environments. Users may experience an error upon attempting to access OWA (Webmail) & sync mail to their email client, or a prompt to re-enter their password.
We will provide further information as this becomes available. 04:39 AM EST
12/02/22
We continue to investigate the connectivity issues to our Exchange environments. We will provide further updates as they become available. 04:32 AM EST
12/02/22
We continue to investigate the connectivity issues to our Exchange environments. We will provide further updates as they become available. 03:02 AM EST
12/02/22
We are investigating reports of connectivity issues to our Exchange environments. Users may experience an error upon accessing the Outlook Web App (Webmail) and syncing their email client(s).
We will provide further updates as they become available. 02:49 AM EST
12/02/22
We are investigating an issue that is affecting our Hosted Exchange environments. More details will be posted as they become available.
