Hello everyone,
The majority of our Atlanta servers are now back online after the power failure due to the UPS vendor’s mistake during routine maintenance.
We are closely monitoring everything to ensure everyone is back online as soon as possible.
Please note: Windows hardware nodes will take slightly longer to return online as containers start. The majority have already started but quite a few are still in the process.
Thanks,
Server Complete
Hello everyone,
We have received word from the datacenter technicians of what has caused the downtime.
During routine maintenance and stress testing on the UPS systems by the vendor, the vendor mistakenly caused the UPS to drop all loads. This caused a complete power outage for our entire 55 Marrietta location.
The vendor has corrected the problem and technicians are currently in the process of powering all hardware back on.
Due to the abrupt shutdown it is a possibility that some hardware will require a File System Check (FSCK) upon booting. We are monitoring all hardware and will bring everyone back online as soon as possible.
Thanks,
Server Complete
Hello everyone,
I regret to inform you that our Atlanta network is currently down… Technicians are on site and working to diagnose and correct the problem as soon as possible.
We apologize for the trouble this has caused.
Thank you,
Server Complete
Earlier this morning at approximately 10AM EST, the null route on all IP addresses, with the exception of the 4, was lifted.
We have been working to have the null lifted on the remaining IP addresses, however, the datacenter is not cooperating due to the potential of a future DDoS.
We are working with those clients to develop a solution for this.
Thank you,
Server Complete
Hello,
The attack on the einstein hardware node is still ongoing and is still trying to be mitigated. We are sincerely sorry for this inconvenience and are working with the DC to try and bring the node back online soon.
Thanks,
Server Complete
Hello everyone,
As of right now we are still facing a null route on all of our IP space on the einstein.nodesdirect.com hardware node due to the rather that DDoS attack which is still ongoing.
Unfortunately tracking the attack to a specific set of IPs is not proving to be a viable solution.
We are waiting on the datacenter to let us know more as to when the null route can be removed.
Thanks,
Server Complete
Hello everyone,
It has been discovered that the DDoS on the NJ network was actually bound towards the einstein hardware node…
We are in the midst of determining which VPS was the culprit for the attack. At this time all IPs for the Einstein node are null routed.
Thanks,
Server Complete
Hello everyone,
We have just received a response from the datacenter in regards to the current status of the procedures being done (replacing a switch). Upon further investigation it was also found that the network is being hit with a DDoS which is several Gbps in size — This was not discovered until the switch was replaced.
The attack is being mitigated as we speak and hopefully will return to service soon.
Thanks,
Server Complete
Hello everyone,
Datacenter technicians are currently in the process of replacing the switch which has caused the downtime.
Thank you for your patience.
—–
Server Complete
Hello everyone,
We have been trying to get in touch with the datacenter regarding the problem with the einstein.nodesdirect.com hardware node not having network connectivity. We suspect this issue is due to a problem with the switch or network cabling. We have done everything possible over KVM on the server that we can and simply cannot gain network connectivity.
Unfortunately we have been having unusually high response times from the datacenter with this matter and we are doing everything in our power to have this problem taken care of.
Please bear with us — We are working to have this taken care of.
All the best,
Daniel Stephens
Server Complete, LLC