The forums were out of service from approx. 3 AM to 12:05 PM (~9 hours) today.
I noticed while completing today's
Botany Photo of the Day entry that the data was not being saved to its database. After fiddling around for a couple hours, I determined that the server hosting the databases simply ran out of room.
Unfortunately, the solution is not as easy as deleting some files on the server to make space. The database system tries its best to protect the data stored within it, and "locked up" to prevent data loss. To unlock it, the database needed to be repaired, and this is only possible when a significant amount of space is available on the server - so you can see where the two problems make a bigger problem. Lack of space causes a problem that can't be repaired because of lack of space...
While I could repair the database for the Botany Photo of the Day and get it working, I didn't have access to be able to clean enough space to repair the database for the forums, which is significantly larger in size. I had to wait for the server administrator of that computer to help out.
Fortunately, he's been in the process of testing a new server dedicated to databases, so that meant we had a machine to which we could copy the database and repair it. Once this was done and everything seemed to be working, we had to decide what to do next.
The upshot is that the database that runs the forums is now running off of the new server. We're confident that the server is safe to store data, so that won't be a problem. The only thing that hadn't been fully tested is the encrypted connection between machines - so we're going to see how that works with this database over the next two weeks.
Again, the data on the machine is as safe as can be now (with plenty of room to grow). If something fails, it will likely be due to the not fully-tested connection - no data will be lost because of it, but access to the forums might go down for a few hours from time to time until we work out any bugs that might exist.
I apologize if this has inconvenienced anyone, and I hope that you continue to use and enjoy the forums.
