The cause of the issue was a wide-scale malicious attack performed against DynDNS, a DNS service provider that handles name resolution for FormAssembly and many other websites. DynDNS has resolved the issue and posted more information on their website.

While DynDNS was down, many FormAssembly users were unable to reach our service because they could not obtain the correct IP address for our servers. Users who were using FormAssembly at the time of the incident were less likely to be affected as they would have relied on the IP address cached by their web browser or operating system.

Our own servers were operating normally and no data was lost during this incident.

At this time, we expect FormAssembly.com to be fully operational.  You can check the status of our service at status.formassembly.com, and see up to the minute updates from twitter at @FormAssembly.

While today’s incident was out of our control, we will still review our hosting strategy and see what actions we can take to prevent or mitigate such incidents in the future.

We appreciate your patience and understanding. Please let us now if you have any questions or concerns.

The FormAssembly Team.

At this time we believe that only 17 customers were affected. We will be contacting them directly to apologize for the incident and help them obtain the complete responses from their respondents.

The cause of the incident was a system configuration error on a backup server. We put this server online today in order to perform an unrelated routine maintenance on a different machine. The issue was compounded by an application design issue which helped the problem go unnoticed  until one of the affected user reported it to us.

The integrity of your data is our responsibility and we take today’s incident very seriously. We hope that we’ll be able to work with the affected users to get their missing files and we’ll work tomorrow to make sure we won’t run into this problem again.

Please let us know if you have any questions or comments.

While we were able to respond to the incident as soon as it was detected by our monitoring service, we did not immediately understood the cause of the failure. We proceeded to play by the book and ran through our usual procedures. Unfortunately, our actions turned out to be ineffective. Better monitoring and diagnostic capabilities would have been critical in helping us understand and address the issue properly, at the time of the incident, and later on during the post-mortem analysis. We’re now incorporating more metrics into a system dashboard to help us monitor the health of our service and pinpoint failures more quickly.

2. Underpowered Backup Database Server.

We had a number of subsequent very short downtimes in the following hours due to our switch to our backup database server. Some of the improvements we had made recently to our database had not yet been ported over to the backup server (we intentionally staged the upgrade), and as we started relying on it, old and known deficiencies started to surface again. While not critical, it did keep us on our toes much more that we would have wanted to. We’re now moving ahead with the upgrade.

3. Inadequate Downtime Notification.

During the incident, users would often get a blank page after a long time out, while only a few would get the correct “service unavailable” error message. While we work to minimize downtime altogether, we realize it’s important to properly inform users when an incident happens. A time-out is simply not acceptable. We’re going to address this by adding more ways to serve the downtime notice, upstream in our infrastructure.

Conclusion

Regarding the root cause of the incident, everything points to a saturation of our servers, due to a high load and several high-traffic forms uploading large files (>10Mb). There might have been another, still unidentified factor, as we’ve handled similar loads before and load testing indicates that we should have been fine. We’re counting on our improved monitoring to help us address any re-occurence properly, as well as guide our ongoing capacity planning.

This was our worst incident in 2 years. We apologize for the downtime and the inconvenience caused, and are immensely grateful for the patience shown and the support we received. Please let us know if you have any feedback or comments regarding this issue.

You can stay up-to-date with our service status at http://status.formassembly.com or by following us on Twitter: @FormAssembly.

To complete our recovery from yesterday’s incident, we are scheduling approximately one hour of downtime for maintenance at 1am EDT October 7, 2010 (5am GMT). During this time, form processing will be suspended and the application will be inaccessible. We do not expect the maintenance to take the whole hour.

If this time frame is inconvenient for you or your company, please let us know. We may be able to adjust the downtime by a few hours.

We continue to investigate the causes of yesterday’s outage and will update you when we know more.

At this time we believe all forms hosted by FormAssembly to be functional.

If you experience any further service disruptions, please let us know at: support@formassembly.com.

We will post another update once we have identified the full cause of the downtime and determined a plan of action for preventing it in the future.

Thank you for your understanding and patience.
The FormAssembly.com Team

If you do not use the Google Spreadsheet connector, you may disregard this notification. Affected users who requested email notifications of downtime and incidents have been contacted by email.

To renew our authorization, follow these steps (est. time < 5 min.):

1. If you installed FormAssembly through the Google Apps Marketplace:
Please visit your Google Apps’ Domain Management console and update the permissions for the FormAssembly application.

This will update the security token held by FormAssembly. This is the only step you need take.

2. If you did not install FormAssembly through the Google Apps Marketplace:
Please visit your FormAssembly form’s Google Spreadsheet connector page. Within the connector, ‘revoke’ the current authorization,

then click ‘begin’

to start the process of issuing FormAssembly a new security token.

This process must be repeated for each form that connects to Google Spreadsheet.

We apologize for the inconvience, and if you experience any difficulty, please contact us.

We’re very sorry for the trouble caused by this interruption of service. We are still investigating the issue and will update this report when more information is available.

Thank you for your patience and understanding.

Our uptime report is available here: http://status.formassembly.com

You can follow us on twitter for the most current status reports.

UPDATE 5/12/2010: Although form processing was restored at 4:46CST on 5/11/2010, some users remained impacted by lingering issues.

1. Salesforce users who rely on a login IP range instead of a security token to authorize FormAssembly need to add our backup IP address to the list.
For more information visit our Salesforce documentation.

If you have responses that didn’t get into Salesforce because our backup server wasn’t authorized, you can resend the data from the connector log, or open a support request if you want us to take care of it. No data is lost, but your Salesforce org. will not be updated until you authorize our backup server.

2. A configuration error on our backup caused PayPal-integrated forms that uses the “encrypted cart” option to fail. The problem started at 4:46pm CST yesterday and was resolved at 12:20 CST today. Responses submitted during that time were received properly, but customers could not proceed with the PayPal payment part. In FormAssembly, these responses are visible and shown as complete, but there’s no payment receipt attached. We’re contacting affected users directly to help them recover their payments.

We have been notified today by our data center operator that an emergency maintenance is scheduled for this Monday, November 30th between 02:00 and 02:30 CST This maintenance is required to perform a vendor recommended upgrade on their network infrastructure.

Our service will be completely unavailable during this time, including form processing and access to your account. We are sorry for the inconvenience.

Due to the short notice and the short downtime expected, we decided to not switch the traffic to our alternate data center. We will, however, be monitoring the progress of the maintenance and we will keep this option open should there be any issue.

Maintenance Date: Monday, November 30, 2009 (11/30/2009)
Expected Downtime Start: 2:00 CST (08:00 UTC)
Expected Downtime End: 02:30 CST (08:30 UTC)

Update 1:  Maintenance window changed to 08:00 – 08:30 GMT

Update 2: Maintenance completed. Total downtime was only 16mn.

http://app.formassembly.com/forms/view/1234

and is now

http://www.tfaforms.com/1234

FormAssembly will continue to function as it always had, and any forms you published, linked to, or gave out with the original address will continue to work. Again, we made this change to make form URLs easier to type and share, as well as to comply with the recommendations of our annual Salesforce security audit.

We’ll continue rolling out this change gradually over the next few days, but again, you do not need to do anything. Your forms will continue to be accessible at either address.

Thank you for your patience and understanding.

The FormAssembly Team