** ezboard Attack Update - 06/06/05 **
Dear ezboard Family,
As promised earlier today and to some of you who I've been e-mailing today here is a comprehensive update of what we know at this point and what we don't know.
I would like to offer some clarification before we get into the update. The information we use for decision-making and reporting to you is constantly changing. I realize that earlier global announcements have contradicted later ones and confusion has resulted from that. We have been giving you as much information as possible as early as possible and at times that information has changed between global announcements. Please accept my apologies for any confusion the different reports have caused and understand that each global announcement stands on its own at the time it is posted.
Now, on to answering the questions you all have on your minds:
How many servers have been restored?
At this point there is no such thing as a completely restored server.
Let me give you some background to put this in perspective. We have gone back into terabytes of non-back-up data looking for any board data we can find. Where possible we've put this data in the restore queue. When we learned of the hack we also dumped the contents of RAM from each server onto a hard drive. We have been restoring that data as well. With the main mode of data restoration gone (the regular back-ups) we have had to create new restoration processes from scratch. That means writing software, testing it and then putting it into production - all on no sleep. In some cases working slower IS safer - but more frustrating for you and for us.
At this time we are restoring from five different data sources. Each source requires a separate step or process to restore data. Roughly half the servers have finished the first step of restoration and are in the second step of what are ultimately five steps of restoration. No severs will be fully restored until they have passed through all five steps. We cannot give you specifics of what servers/boards are where in the process and we cannot have your board opt out of the restore processes.
Why am I still losing threads that were on my board after the attack?
We have isolated the problem and fixed three issues that were leading to thread loss. These fixes have been applied to all servers and you should not be seeing additional losses after today.
Thank you to the over 500 people who submitted information on this problem. We do not need additional information at this time, but if we get new reports of new data loss after tonight we may ask you for more information.
Why am I getting request errors on my board?
We have identified twelve variations of this error. We believe it will be resolved as part of the restoration process. Thanks to the hundreds of people who submitted bug reports on this error already. Should you still be experiencing this bug after the restorals are complete, we will ask for more information at that time.
What is happening to Gold Community subscriptions that have expired since the attack, or will expire soon?
We are currently identifying every Gold Community whose subscription has expired on 5/31 or later and will be reinstating or extending subscription services through at least the next two weeks. This will take a few days to complete. In the meantime, you do not need to let us know your board is expiring.
Why is it taking so long?
Believe me, we are not dragging this out for fun. We are working carefully to ensure that we don't lose more data. That means double and triple checking what we are doing. For example, we don't want to overwrite new posts in a forum when we restore that forum. So, we need to perform tests to ensure that does not happen. This caution takes time, but reduces the chance that we lose data that can be restored.
How successful are the restorations?
We don't know yet. The data that is restored will not be evenly distributed across all boards. Some boards will get all their data restored and some will not get any data back. We will not know exactly which boards will get all, some or none of their data back until the five restoration phases are completed on all servers/boards. We would love to send you a list of which boards are getting what data restored, but it is not information that can be generated at this time.
What don't we know at this point?
We don't know how long the five phased recovery process will take. Different servers restore at different rates and there is no pattern to the time it takes.
We don't know if we will find more data that can be restored. But we are looking for it.
What are you doing to make sure this does not happen again?
We are changing your back-up processes and we are improving security. For security reasons what specifically we are doing is not something we will share in a public forum.
Why are board web pages loading slowly?
The restore process is taking computing power from serving web pages. Web page delivery speeds will return to normal when the restoration process is completed.
Can I e-mail you with questions?
Yes, I have been responding to every e-mail I have received. Sometime it takes a few hours, but I am responding as fast as possible. Please understand that the global announcements have the most up to date information. We cannot provide any information or special restores for specific boards or servers.
Thank you again for your patience and understanding.
Sincerely,
Rob
Robert Labatt
CEO
ezboard, Inc.
robertlabatt@ezboard.com
Tweet This Topic
