Coordinating Multi-Host Backups Using Perl Sockets

A. Clay Stephenson

Systems administrators are often tasked with cleaning up someone else's mess. I encountered such a situation when I started working with my current employer. The company's newly installed Enterprise Resource Planning (ERP) system needed a backup, and the system integrator had configured an Oracle "hot" backup as the perfect solution. After looking at the situation, I found a serious problem with this "perfect" solution -- the backups were essentially useless. The problem was that this ERP system utilized a split architecture -- meaning that the application servers hosted metadata that described the actual database data hosted by additional servers. Unless the metadata exactly matched the described data, the backups, no matter how reliable, were useless.

A method was needed to halt or freeze the application servers and to then halt the databases. Then, the backups would be taken and, finally, databases and applications would be restarted in reverse order. At this point, the system integrator's answer was to shut down everything and do a traditional "cold" backup -- not very practical for a production environment.

An alternative to these long downtimes was quite simple -- snapshot mounts of the filesystems. The backups could then be made at leisure using these snapshots. The entire system would only need to be down briefly (in practice less than two minutes per day, which was deemed acceptable). However, I still had to address the problem of coordinating the shutdown and restart of this group of processes on multiple hosts in the correct order.

In an ideal world, nothing more complicated than the following would be required of a simplified system consisting of two ERP application servers connecting to a database on another dedicated server:

1.