Store to Disk preserves backups' attributes
December 18, 2013
By Brian Edminster
Second of two parts
Yesterday I outlined some of the powers of the Posix program pax, as well as tar, to move MPE/iX backup files offsite. Here’s a warning. There are some file types that cannot be backed up by tar/pax while also storing their attributes: ;CIR (circular) and ;MSG (message) files (and possibly others. I haven’t tested all possible file types yet. Also, there is an issue with tar that is a fairly well known and has been discussed on the 3000 newsgroup. Occasionally it does not un-tar correctly. It is unclear if and when this was fixed, but I’d love to hear from anybody that might be in the know, or which specific situations to avoid.
Regardless of these limitations, I’ve found a simple way around this. Use store-to-disk to make your backup, then tar to wrap it, so as to preserve the store-to-disk files’ characteristics, before shipping the files off-system. Later, when you retrieve your tar backups and un-tar them, you’ll get your original store-to-disk files back without having to specify the proper ‘;REC= , CODE= , and DISC=’ options on an FTP ‘GET’. I’ve been doing this for several months now on several systems, and I have not had any failures.
If you have a version of STORE that has compression, use it to reduce the size of backup. If not, use the ‘z’ option in the tar/pax archive you create from your store-to-disk backup. Do not use both. They don’t play well together, and you may end up with a larger tar file.
But what about the tar archive size limit of 2GB? There’s an easy way around this as well, as this limit is common on early Unix and Linux systems. Just pipe the output through ‘split’ to create chunks of whatever size you want. Below, there's simple examples for both directions.
Figure 1, just below, is an example of the ‘cksum’ file produced.
Below, Figure 2 is an example of a ‘cksum’ created of the files as they’re stored on the NAS.
As both the hashes and #bytes shown in each file are the same as on the MPE/iX server — we know the backups are transferred correctly. The same technique can be used ‘in reverse’ to verify that when FTP’d back to the FTP server, they’re still intact.
When un-taring this backup, ‘cat’ the pieces together and pipe it through tar. At least, that’s the way it’s supposed to work. Yes, there is a known issue with the MPE/iX Posix shell’s built-in cat command. But I’ve so far been unable to successfully use the external cat command to successfully cat either. Here’s how this should work for a 2-chunk tar backup:
sh>/bin/cat ./CS1STD1.ustar.aa ./CS1STD1.ustar.ab | tar -xfv - *
Unfortunately, for me at least, it always throws an error indicating bad format for the tar files.There is a work-around, however. Note that while ‘cat’ing the tar ‘chunks’ didn’t work using the internal or external cat command, untar with multi-file option does work. Even though it gives a minor error messages, files were returned to proper store-to-disk format, and the recovered store-to-disk backup is intact and has been used to recover the desired files. To do this, use tar like this:
sh>tar -xfv ./CS1STD1.ustar.aa *
Also note that when using tar in this way, it will ask for the name of the 2nd-nth component tar files, as it finishes reading each prior piece. You must give the filename and press return to continue for each. I believe that it should be possible to script this so that it’s fed the filenames, but I haven’t gotten around to doing that yet.
Brian Edminster is president of Applied Technologies.