An MPE Power Tool: Byte Stream Files
July 17, 2013
Although he left HP's MPE labs long ago, system architect Craig Fairchild still subscribes to the Newswire to keep an eye on with 3000 matters. Four years ago, after he'd ended his official 3000 work, he educated the community on the use of byte stream files in the OS. The knowledge is still useful for any homesteader today.
These fundamental files are a lot like those used in Windows and Linux and Unix, Fairchild explained. HP engineered "emulation type managers" into MPE/iX, an addition that became important once the 3000 gained an understanding of Posix. In 1994, MPE XL became MPE/iX when HP added this Unix-like namespace.
It's a rare gift to see a primer on 3000 file types emerge from HP. Understanding the 3000 at this level is important to the customer who wants 3000 third party companies to take on the tasks HP has dropped. Although migration advocates point out that HP support is long gone for MPE, the knowledge remains intact among independent providers. Fairchild explained the basics of this basic file type.
Byte stream files are the most basic of all file types. They are simply a collection of bytes of data without any structure placed on them by the file system. This is the standard file model that is used in every Unix, Linux and even Windows systems. MPE's file system has always been a structured file system, which means that the file system maintains a certain organization to the data stored in a file. The MPE file system understands things like logical records, and depending on the file type, performs interesting actions on the data (for example, circular files, message files, KSAM files and so on).
Fairchild detailed how HP gave MPE's byte stream files the knowledge of "organization of data" for applications.
To bridge the gap between standard byte stream file behavior (only the application knows the organization of data) and traditional MPE file type behavior (the file system knows what data belongs to what records), emulation type managers were created. To an MPE application, a byte stream file looks and behaves like a variable record file, even though the data is stored in a way that would allow any Posix application to also read the same data. (Posix applications also have emulator type managers that allow them to read fixed, variable and spool files in addition to plain byte stream files.) The way that the byte stream emulator detects record boundaries is through the use of the newline (\n) character, which is used, by convention, to separate data in ASCII text files on Unix-based systems.
The underlying properties of a byte stream file is that each byte is considered its own record. In MPE file system terms, a record is the smallest unit of IO that can be performed on a file. (You can write a partial record fixed length record, but the file system will pad it to a full record.) Since the smallest unit of IO that can be performed on a byte stream file is a single byte, that becomes its MPE record size. In the MPE file system, the EOF tracks the number of records that are in a file. Since the record size of a byte stream file is one byte, the EOF of a byte stream file is also equal to the number of bytes in the file. This is why one 4-byte variable sized record is equal to 5 byte stream records (4 bytes of data + 1 \n character).
It's also worth noting that any file can be in any directory location and will behave the same way. (Well, almost. CM KSAM files are restricted to the MPE namespace. And of course the special files (that you don't normally see) that make up the file system root, accounts and groups are also restricted... one root, accounts as children of the root, groups as children of accounts. And lockwords aren't allowed outside the MPE namespace. But other than that the first sentence is true.)
The general model that we had in architecting the whole Posix addition was that behavior of a file does not change regardless of where it is located. This was summed up in the saying, "A file is a file." So there are no such things as "MPE files" and "Posix files." There's just files.
What does change is the way you name that file. Files in the MPE namespace can be named either through the MPE syntax (FILE.GROUP.ACCOUNT), or through the HFS syntax (/ACCOUNT/GROUP/FILE). You can also use symbolic links to create alternate names to the same file. This was summed up as a corrallary to the first saying, "But a name is not a name."
The underlying properties of a byte stream file is that each byte is considered its own record. In MPE file system terms, a record is the smallest unit of IO that can be performed on a file. (You can write a partial record fixed length record, but the file system will pad it to a full record.) Since the smallest unit of IO that can be performed on a byte stream file is a single byte, that becomes its MPE record size. In the MPE file system, the EOF tracks the number of records that are in a file. Since the record size of a byte stream file is one byte, the EOF of a byte stream file is also equal to the number of bytes in the file. This is why one 4-byte variable sized record is equal to 5 byte stream records (4 bytes of data + 1 \n character).
It's also worth noting that any file can be in any directory location and will behave the same way. (Well, almost. CM KSAM files are restricted to the MPE namespace. And of course the special files (that you don't normally see) that make up the file system root, accounts and groups are also restricted... one root, accounts as children of the root, groups as children of accounts. And lockwords aren't allowed outside the MPE namespace. But other than that the first sentence is true.)
The general model that we had in architecting the whole Posix addition was that behavior of a file does not change regardless of where it is located. This was summed up in the saying, "A file is a file." So there are no such things as "MPE files" and "Posix files." There's just files.
What does change is the way you name that file. Files in the MPE namespace can be named either through the MPE syntax (FILE.GROUP.ACCOUNT), or through the HFS syntax (/ACCOUNT/GROUP/FILE). You can also use symbolic links to create alternate names to the same file. This was summed up as a corrallary to the first saying, "But a name is not a name."