The new day dawned. So far, this catastrophe seemed like it would go on forever, a cascading series of failures which, honestly, were starting to get on my nerves. I take alot of pride in certain numbers, particularly the number “0”. That’s how many times I have lost a customer’s data. This catastrophe was not going to change that number.
Maps
With a fresh outlook nurtured by a bit of sleep, I was able to quickly troubleshoot this latest problem. Installing this program requires a folder to be created and mapped as a network drive, which I had done each time. When reinstalling Windows, the original desktop, with all of its shortcuts, was retained. The old shortcut referred to the old mapped drive, which was eliminated when I reinstalled Windows. I remapped the drive, and the program worked fine.
I spent a little time making sure a few other things were correct, and went to test the program on the users’ machines. Everything connected and ran perfectly. Back in the server room, I restored the backup files to bring the program up to date. Then another round to make sure the remotes could see that. All the users were happy, and ready to catch up on the work that had been steadily accumulating. I asked them all to stay out of the program until I had ran the rebuild routine before I headed back to the server.
Now that the shortcuts all pointed to the right mapped drive, the rebuild ran fine. It takes awhile to run it. There’s decades of information that has to be sorted through. The best thing to do is just sit back and watch the file names change.
With about 2 minutes left, the routine halted. When the error appeared I knew immediately what must have happened. The program has 2 main functions, event co-ordination and accounting. By far, the event coordination side is the largest, and the most used. But the accounting program is there, and someone had logged into it while the files were rebuilding.
Sigh… So after a short talk to the users about staying out of the program while I’m rebuilding it, I reload the backups, check the remotes, remind the users, and rebuild the database for the, well, I forget how many times actually. This all took about another hour, and by now everybody was gone to lunch. I decided to skip a final test until they returned and moved on to the last program.
Data Loss!
The cd installed fine and this time, the backup copied right into the correct folder. But when I ran the program, the data was not there. I compared the backup file size to the “last loaded” file size on a remote machine, and the file sizes were different. My backup was considerably smaller than it should have been. It was my worse fear. Data was lost.
I checked the backup file, on the external drive. It showed the decreased size. I looked at the properties, and there I found an indication of what might have happened. The last time the file was changed was the day before, at about the time the server crashed. Somehow, and I do not even pretend to know how, when the server crashed, the file became corrupted.
I sat back and stared at the screen. In my mind I was going over the amount of information contained in that database. There were about 600 files. Each file would take about 30 minutes to key in from paper. So that was about 300 hours of work. Working 5 hours a day, 6 days a week, I could get it done in a few months, maybe.
Sigh… I surely was not looking forward to that. But even more, I was not looking forward to telling my customers that their data was gone. That data was my job. Though I keep my prices below industry standards, they still pay me well to protect that data, and I had failed. The data was gone, and soon I would be too, most likely.
That thought didn’t appeal to me either. These people were more than just customers to me. They were my friends. These guys were a source of great motivation and friendship when I was first getting started. They were one of the first to sign on for contractual services. They had stood by me since the very beginning.
I decided not to roll over. The thing that has always set me apart is my insanely steadfast determination not to fail. There was still one more thing I could do. The desperate act of nothing to lose. Though it had slightly less than a 50-50 chance of working, I had to go for it.
Clickback next time for the exciting conclusion of ….. Catastrophe!