Previous month:
May 2021
Next month:
September 2021

August 2021

Cooking with Python on MPE

Whip up tasty dishes with ample applications using this scripting language

By Curtis Larsen

Something I’m sure I share with other HP 3000 folks is a love for technology that solves problems in some straightforward yet elegant manner. After all, this is why we all continue to use our HP 3000 systems (and probably why we chose them in the first place).

Especially exciting, though, are the new technologies that further extend our systems and solve even more problems. I can remember eagerly reading each new “Communicator” for the enhancements list, and my personal favorites – upgrades to the Command Interpreter. I’d be excited to try out each new function and new “FINFO” parm. (When the JOBINFO function hit the streets I was practically singing.)

As much fun as I have with the CI though, I recognize that every programming language has its limitations. So HP 3000 programmers have used various different languages available such as COBOL, Business Basic, or VESoft scripting – or a combination of them — to solve larger tasks. Each language brings its own flavor and abilities to the programming table, and when POSIX scripting was added to the ingredients, a wonderful curry resulted. Like a great curry, you can still see, smell and taste each individual ingredient, but they all contribute to a sum greater than the parts.

Some of the new flavors the POSIX world gives us are the Bourne, BASH, C, and Korn, shells, as well as public-domain C, FORTRAN, Assembly, and Basic compilers. It also gives us the newer interpreted scripting languages such as Perl, Java, and Python.

Now chances are good you’ve heard about Perl and Java, but how much do you know about Python? Ahhh… let’s start opening some of those spice jars, shall we?

Python is a fast, extensible, object-oriented scripting language created by Guido Van Rossum and named after the ’70’s British comedy troupe and television show “Monty Python’s Flying Circus”. (It’s too good to make up folks. If that doesn’t begin to spell “curry” to you, I don’t know what does.)

Python and Perl are very similar in capabilities —what can be done in one, can also be done in the other – it’s just that their approaches to those capabilities differ. Perl leans towards terse syntax, and many small “add-in” modules, while Python (if not verbose) leans more towards self-describing syntax and fewer, larger, “add-in” modules. Both can be written in Ye Olde Top-Downe style, or by using objects and methods in the newer Object-Oriented style of programming — pick the sauce that suits you. Both languages have been ported to almost every OS imaginable (and a few that weren’t). Python also uses OOP syntax similar to Java with it’s “Object<dot>Method”. This similarity to Java comes in very handy, but more on that later.

One quirk that Python people have to get used to though is the indenting. The beginning of a text line is significant in Python, and indicates dependence. Now, now – I heard that agonized cry, like you had just eaten too much pepper. Trust me on this one – like a pepper you may find it hard to swallow at first (especially in one large bite), but if you come to it slowly, you’ll learn to appreciate the subtleties and grow just as addicted to it. And let’s be reasonable – if you can start a significant line of COBOL at a certain column for twenty-odd years, you can handle a little reasonable indenting.

Rather than explain Python’s syntax in detail though, I’ll point you to the Python web site where you can really chew on it. The main plate is at www.python.org, and many other links can be found from there for general consumption.

Ok, so let’s say you buy into it for now – you’re at the MPE table checking out the various Python dishes for appearance and aroma before biting into one. A wise course of action, but while some of the offerings can be a bit exotic, I’ve found nothing unattractive. You can find an incredible list of Python applications at “The Vaults of Parnassus” (www.vex.net/parnassus) — here are a few that might be of interest to you in combination with an HP 3000:

• “ZOPE”: An extremely powerful Web-application creation utility — and then some!

• “Gadfly”: A powerful Python-based SQL engine – see its Web page for details.

• “Peercat”: Web-based aggregate data collector. (Usually news items, but…)

• “Scanerrlog”: A script to meaningfully parse Apache’s “error_log” file to HTML, etc.

• “txt2html.py”: Converts various ASCII formats to HTML.

• “inpim.py”: Calendar/To-Do app w/cross-system sync. Budgeting subsystem, too.

• “PyBook”: Searches, downloads, and displays Project Gutenberg books.

• “Roundup”: Issue tracking system with web,CLI, and e-mail interfaces.

• “RoutePlanner”: A highway trip planner.

• “Pyrite”: Allows Palm Pilot synchronization.

• “PyPsion”: Similar application for Psion PDAs.

• “web2ldap.py”: Full-featured web-based LDAPv2+ client.

• “flad”: Create, read, and write INI-like config files.

• “ODBC Socket Server”: Access Windows ODBC sources via XML TCP/IP interface.

• “Python-DSV”: Parses CSV, TSV, etc. files, guesses string encoding, etc.

And more CGI scripts than you can shake a kebob stick at.

This list just scratches the surface, since there are far more HP-useful Python applications than can be shown here, and due to its ease of use and power, more people are sitting down to the “Python picnic” each day. There are many Python applications that could have a strong impact on your regular MPE recipes, and more are continually arriving. Look over the list above – ever want to directly use your HP data with a Palm Pilot? Send and receive data from an ODBC source directly from a batch job? Map the shortest routes to clients? Check on the weather before shipping a product there? Many more possibilities exist!

Python includes very good support for XML (including XML-RPC and SOAP), LDAP, and offers an interesting native object persistence (or “object pickling”) to a file, so that you can save an object and its properties, freezing it for cooking another day.

Although the “original” Python was written in C, a newer version of Python named “Jython” has been written using Java. In addition to running native Python scripts, this version also permits use of the native Java objects and features from within Python. Looking (and reading) very similar to Java, you can actually code something very quickly using Python for later export to Java, etc., treating Python as a sort of RAD language. So whether you think of Python as a “C food” entree, or as a side dish to chew on with your morning “cup of Java,” you’ll find it both digestible and flavorful. You can use a little or a lot, spread it thinly or plaster it on thickly – whatever suits your taste and mood.

I’ve only used the C-based Python on my HP system, but you could give Jython a whirl. The C version is a bit dated but runs quickly, while the Java version is very current, but starts a tad slower. It just depends on what you want to use Python for, and whether or not you’ll need some of the newer functions and modules.

Probably the thing that really excites me personally about Python is its “dictionaries,” or what Perl calls “hashes.” Folks who use these types of variables almost always get hooked on them. For example, you can create a dictionary named “D”, and store a value in it using a string key named “info” which might look like this: D[“info”] = “555-1212”. Now whenever you say ‘print D[“info”]’ you’ll get “555-1212”. It’s like using an array, but without the limitations of an array — you don’t need to preset a dictionary, worry about internal structure, use only numbers, or know how many “elements” it will have. You can even dynamically store a dictionary within another dictionary (and so on) to get some incredible depths of sophistication. Suddenly a “simple scripting language” allows fast use of tables and three-dimensional databases. (And it pulls up those values just as quickly!)

For an HP 3000-specific example of using Python dictionaries, I combined a VESoft Security/3000 report with a quick Python hack – er – script to show me the last date any terminal in a given date range was used, then print those results sorted by terminal. Sure, I could have coded it all some other way, but using the report as input to a filtering script was simplest for me, and gave me the results I wanted very quickly. Figure 1 below shows the script (be gentle).

Figure 1

import fileinput, string # These modules provide some
# nice extra methods for file
# and string handling, etc.

x = " " # Inits: Not needed, but. . .
hist = {}
totals = {}

for line in fileinput.input(): # Loop thru each report file
r = string.strip(line) # line (given as the parm)
if r[0:5] == "Logon": # Detail record? Process it.
rdate = "0" + string.strip(r[07:14]) # Load our work variables
rdate = rdate[-7:]
rdev = int(string.strip(r[23:27]))
rlogon = string.strip(r[27:62])
rname = string.strip(r[63:94])
hist[rdev] = (rdate, rlogon, rname) # Load the ‘history’ dictionary
# using the LDEV # as the key
if rdate <> x: # Print each new date as we go
x = rdate
print rdate
# Initialize the ‘totals’ dictionary logon count at first use
if not totals.has_key(rdev): totals[rdev] = 0
totals[rdev] = totals[rdev] + 1 # Add one to the logon count

keylist = hist.keys() # ‘keylist’ contains seen LDEVs
keylist.sort() # BAM! Now it’s a sorted list

for ldev in keylist: # for each LDEV we saw. . .
rdate, rlogon, rname = hist[ldev] # load some work vars, and:
print "%04u %5u %7s %s" % (ldev, totals[ldev], rdate, rlogon)

This script assumes that a chronologically-ordered list of LDEV logons will show the most recent logon for every terminal (with any luck, hey?), and records every logon for each LDEV. Each successive assignment overwrites the previous value used by that LDEV key, so the last logon for each LDEV key is the last value kept. Additionally, the script counts each time that LDEV key was seen, so that you have an idea as to how often the terminal is used. This little script could easily be adapted to record other information, like the first logon, or whatever else you might need.

I hope this helps whet your appetite for the Python language, and that you will give it a try – again, there’s far more to the language than can be described here. Many thanks to Joseph Koshy at HP Bangalore for bringing Python to MPE/iX, and to the HP3000-L list for assistance. You can  see what Koshy is doing in his spare time to port Python v2.x by visiting the Python/iX web site on Sourceforge: pythonix.sourceforge.net. For a copy of Jython, head on over to www.jython.org and download a copy.

Go ahead — join the Python/iX groundswell now, and get your piece of the Python.

Curtis Larsen has been working with HP 3000s for 11 years, and believes that, given enough time, any application can be written using the CI. He currently works for Covance Laboratories, in Madison, Wis.


The Spectrum Project, Part II

By Bob Green

Last month I presented the first half of our history of the PA-RISC 3000 development, using excerpts from our old customer newsletters, supplemented with new comments (My comments are shown below prefaced by “In Retrospect”). By 1986 we reached the point where Robelle was allowed to experiment with a prototype MPE system at the migration center and were aghast at how slow and unreliable it was. And since the Unix versions of Spectrum seemed to be humming along nicely, the problem seemed to be software, not hardware.

September 11, 1987 Newsletter:

First Spectrum Shipments: Rumor has it that HP shipped the first four 930 machines on Friday, August 21st, with more to follow every week thereafter. As of press time, we have been unable to find out whether ordinary mortals are allowed to touch these machines (as opposed to those who have signed non-disclosure agreements).

In Retrospect: Due to the NDA, over a year passed with no Spectrum news in our newsletter. The project was now 18 months past the original promised delivery date, but was still far from completion. Many people wrote articles, about the Spectrum, mostly based on marketing hype from HP, but no one broke the embargo on real information. We were all terrified. The MPE group had dug themselves into a very deep hole, and no one wanted to be the one who caught the eventual backlash.

October 19, 1987 Newsletter: The Spectrum Song

Orly Larson and his database singers performed again at the Interex show, including their hit, “The Spectrum Song:”

If it takes forever, we will wait for you
For a thousand summers, we will wait for you
‘Til you’re truly ready, ‘til we’re using you
‘Til we see you here, out in our shops!

From the HP Management Roundtable: Schedule for Shipping Spectrums — “We are shipping equally around the world. Our first shipments went to both North America and Europe. We are acknowledging availability for 930s and 950s through December at this time … We expect by the end of November to be able to have acknowledged the entire backlog.”

In Retrospect: HP continued to spin the “shipments” of Spectrums, without mentioning that these were not finished products. The models were 930 and 950 and the operating system was called MPE/XL, changed in later years to MPE/iX when POSIX was integrated into MPE. By this time, HP was worried about their stock price also and did not want any negative news in the financial press, no matter how accurate. As shown by the next Q&A at the roundtable…

Early 930/950 “Shipments”

Question: “Are the 930s and 950s being shipped or not? In public you tell me they are shipped. In private, however, I hear from both users and HP that these machines are still covered by non-disclosure agreements and that access to these machines is very restricted, even when in customer sites. What is the story?”

Answer: “MPE/XL architecture is very, very new. There’s a million new lines [of code] that go into MPE/XL, and a lot of software sub-systems as well. And so we are being extremely cautious in how we proceed at this point. We’re going through what we call a slow ramp-up through the remainder of this year and going into large shipments in 1988. The reason for that is that we want to fully test out the system capability in a large number of customer environments and we want to make sure that the information on what’s going on in there and the people involved are protected from outside folks who either benevolently or not benevolently would like to find out what’s going on.

I’m sure we’re going to run into some problems along the way that haven’t been encountered in our earlier phases of testing. We haven’t really hit these machines with full production pressure yet. We know from experience that when you do that, you uncover things that you could never uncover in testing, even though extremely rigorous. [Rumor has it that the customers receiving Spectrums now are not allowed to put them into production until 1988.]”

In Retrospect: Early Spectrum customers called us to ask which version of Suprtool and Qedit they needed for their new systems, and whether there were any problems that they should be aware of. But legally, we could not even admit that we knew of the existence of the new servers. So we came up with the following wording: “If you had a new 3000, and we are not admitting that we know anything about a new 3000, you should be using Suprtool version 3.0 and Qedit version 3.6. On this hypothetical system, it might not be a good idea to hit Control-Y while copying a file from any other HP 3000. We can’t tell you what will happen, but you won’t like it.”

February 12, 1988 Newsletter

Spectrum Finally Leaves the Nest: Hewlett-Packard has officially released the 930 and 950 Spectrum computers, finally abandoning the protection of non-disclosure agreements. We have heard from several sources that the 930 and 950 attained Manufacturing Release during the month of January. This means that people who received “Control Shipment” Spectrums can now put them into production and let outsiders use them. You no longer need to sign any restrictive agreements to get a 930/950. It also means that we users can now compare notes on what the MPE/XL systems are good for.

Interestingly, we didn’t hear about the Manufacturing Release (MR) of the Spectrum from Hewlett-Packard itself. As far as we can determine, HP kept this event very quiet — no press conferences or splashes of publicity. Even some HP people in Cupertino were not aware that MR had occurred. Just because the 930 and 950 are released does not automatically guarantee that you can get one. Given the huge backlog of orders that HP has, it will make “controlled shipments” for a while, picking sites whose expectations match the state of the machines.

In Retrospect: Users had been following Spectrum for almost four years and you could see that we were eager for the release of the product. The MPE lab had grown to hundreds of engineers and technicians and hundreds of Spectrum servers. The amount of money being plowed into the project was awesome. Anyone with any kind of skills was being hired as a consultant, in an attempt to get the situation under control and begin shipping revenue-generating servers. But we were premature in our February proclamation of Manufacturing Release, an HP corporate milestone that requires signed confirmation that the product passes the performance tests set for it in the design specifications.

March 31, 1988 Newsletter

Spectrum Is Released but Not Released: In our last news memo, we reported that MPE/XL users were now removed from non-disclosure restrictions and allowed to put their Spectrum machines into production. In the last month, that news has been confirmed by many sources.

We also concluded, and reported, that MPE/XL had obtained MR (Manufacturing Release). That is untrue. MPE/XL has apparently obtained SR (System Release), but not MR. “System Release” seems to be a new category of release, created just for MPE/XL. We have heard from some new 950 customers who did not need to sign a non-disclosure agreement. However, one customer reported that before HP would allow him to order, he had to sign a document stating that he had no specific performance expectations. On the other hand, we heard from a school that recently went live with 35 student sessions and had great response times (“the machine is coasting along at 10 percent PCU utilization”).

In Retrospect: In order to stem the rising tide of bad expectations, HP released the MPE systems even though they could not pass the testing department. And the performance was still poor in many cases, less than the non-RISC 3000s being replaced, although excellent in a few other cases.

Non-disclosure restrictions are not lifted for everyone. Sites that are beta-testing subsystems which were not released with the initial MPE/XL system are still restricted. Also, third-party FastStart companies such as ourselves are still restricted from passing on any performance or reliability information that we obtain from HP. We face no restrictions regarding performance information received from our customers, so please call with your experiences.

Non-disclosure continues – HP is picking their initial customers carefully and coaching them to only pass on the good news about their new systems. We are still frustrated to not be able to pass on our ideas about how users can improve the performance of the Spectrum.

October 12, 1988 Newsletter

Gary Porto at Childcraft reports that with MPE/XL 1.1 the problem of a serial task in a batch job hogging the system is not so bad as it was with 1.0. This problem can occur with SUPRTOOL, QUERY, or any long serial task. The batch job still hogs the system, but at least people can get a minimum of work done. With 1.0, they couldn’t even get a colon! Gary reports that he has 65 on-line users on his 64-megabyte Series 950 and that the performance is pretty good — as good as his Series 70.

In Retrospect: On the 4 year anniversary of the project, HP released version 1.1 of MPE/XL, which made the systems much more useful, but still not up to the original promised performance of 1984. However, the promise of the “Precision Architecture” (HPPA) was there, as certain tasks were amazingly fast.

By this time, HP salesmen were getting irritated with us for not giving our customers any kind of endorsement for the switch to the 930/950. But our NDA was not cancelled until Manufacturing Release. Finally, the sales force convinced HP Cupertino to send us a signed release from our NDA. I don’t know when MR eventually happened.

From the UK’s HP World magazine: Early MPE/XL Migration Results. London Business School is not a typical installation. Much of their software is written using double precision floating point Fortran which benefits considerably from the Precision Architecture. MIS Director Gordon Miller says “Our straight line performance is up considerably — one program runs 40 times faster — but the performance gains are very application dependent and cannot be accurately forecast beforehand.”

Keith Howard of Collier-Jackson in Tampa, Florida participated in the Spectrum beta testing and upgraded from a Series 58 to a Series 950 — quite a leap. One application was found to be 6% slower due to constant switching between compatibility and native modes, but in most circumstances the machine was five to ten times faster than the Series 52 and one batch job ran 53 times faster!

Glaxo Export has temporarily deferred delivery on its second and third 950 systems due to implementation problems on the initial machine.

HP promises performance improvement for Precision Architecture over the next five years of 40-50 percent per year. Some of this will be achieved by further tuning of MPE/XL — version 1.1 is said to be at least 20 percent faster overall.

In Retrospect: As with the original 3000 project, the birth of the Spectrum was traumatic, expensive and embarrassing. But it paid off. HP was able to roll out better servers for the 3000 line on a regular basis for the next 10 years.

Despite the numerous expansions and revisions to the HP 3000 hardware and software, upgrades have been painless. Even the conversion to the PA-RISC design was backward-compatible and reasonably painless (if you ignore the slipped schedules). Often the user just rolled the new system in on a Sunday, plugged it into the power, reloaded the files, and resumed production. The original 1974 MPE Pocket Reference Card is still useful; everything on it works under MPE/iX version 7.5, except the Cold Load instructions. I have programs that I wrote in 1972, for which the source code was lost years ago, and they still run in Compatibility Mode.

When asked for an eulogy for the 3000, my reply was, “A great IT platform: reliable, affordable, flexible, easy to operate, and easy to program. And every release compatible with the previous for over 30 years. Perhaps some future OS team will adopt these same goals.”


The Spectrum Project, Part I

By Bob Green

Commemorating the Oct 31, 2003 “wake” for the HP 3000, Robelle are devoting our NewsWire column to some history. Our story of the original 16-bit HP 3000 (1972-1976) is told on our Web site.

After initial development, the HP 3000 grew and prospered. From 1974 to 1984, HP continued to produce more powerful 3000 hardware running more capable software. Each new model was compatible with the previous version and a joy to install.

But the pressure was on to switch to a 32-bit architecture, as other manufacturers were doing. So HP announced a radical change: a new 32-bit hardware for the 3000. The project was code-named Spectrum. As a 3000 consumer and 3000 vendor, Robelle was excited and concerned about the prospect of a new hardware architecture. Certainly it would be wonderful to have more powerful processors, but what about the disruption to our steady incremental, risk-less progress?

The first notice we took of the Spectrum appeared in our December 1984 customer newsletter, with continuing news to follow for the next four years (my retrospective comments are included as “In Retrospect”).

December 12, 1984

The first Spectrum machine will be an upgrade for the Series 68. Other size models will follow soon after, since HP is working on different Spectrum CPUs in three divisions at once (in the past, all 3000 CPUs came out of one division). This first Spectrum can be expected in the first half of 1986.

In Retrospect: Please make a note of that 1986 promised delivery date, and remember that HP faced serious competition from DEC and others. Customers who loved the 3000, but had outgrown the power of the system, were demanding more capable models.

Spectrum is based on the RISC concept, modified by HP Labs. RISC stands for Reduced Instruction Set Computing. Such a computer has no micro code, only a small number of key instructions implemented in very fast logic. The original Berkeley RISC machine had only 16 instructions. Spectrum has more than 16, but not many more. HP selected the instructions for the fast base set by studying typical application mixes on the existing HP machines. Other functions will be done via subroutines or co-processors (e.g., a floating-point processor, an array processor, or a database processor).

In Retrospect: The actual number of instructions in the Spectrum turned out to be about 130, not 16, but they were all simple enough to run in a single clock cycle. HP was the first computer company to go with the RISC philosophy and the only major one to risk the firm by converting all their computer models, both technical and commercial, to a single RISC design.

June 11, 1985

HP’s new Spectrum machine will have both Native-Mode software and 3000 software. The first Spectrum machine to be released will have 3-10 times more computing power than a 68, about 8-10 MIPS in Native Mode. Programs copied straight across will run about twice as fast as on a 68, and those that can be recompiled in Native Mode should run 6-8 times faster. Much of MPE, including the disk portion of the file system, has been recoded in Native Mode. Since most programs spend most of their time within MPE, even programs running in emulation mode should show good performance (unless they are compute-bound).

In Retrospect: The expectations were building in our minds: these machines would be much faster than our current models!

Spectrum will use much of the new operating system software that had been written for Vision, which saves a great deal of development time. Spectrum will use 32-bit data paths and will have a 64-bit address space. Forty Spectrum machines have been built and delivered for internal programming, but product announcement is not likely before 1986.

In Retrospect: Vision was an alternative 32-bit computer project at HP, using traditional technology, which was cancelled to make way for the RISC design from HP Labs. Invoking Vision re-assured us that this project is possible, that progress is being made. It was now six months after the first announcement of the project.

August 16, 1985

According to an HP Roundtable reported in the MARUG newsletter, “Most of what is printed about Spectrum is not to be trusted. Spectrum will be introduced at the end of 1985 and delivered in Spring 1986. There are 40-50 prototypes running in the lab and the project team consists of 700-800 engineers. HP will co-introduce a commercial version and a technical version with the commercial version fine-tuned to handle many interactive users, transaction processing, IMAGE access, and the technical version will be structured for computational programs, engineering applications, and factory automation. HP will eventually offer a choice of MPE and Unix. Most software will be available on Spectrum at introduction time and over time all software will be available.”

In Retrospect: HP tried to dispel rumors, but still predicted 1986 for delivery. HP would produce two Spectrum lines: the Unix line for technical users and the MPE line for commercial users, using the exact same hardware.

“The following describes what will be required to convert – Least: restore files and IMAGE databases as they are and run. Next: recompile programs in native mode. Next: change over to new IMAGE database system. Next: change source code to take advantage of RISC.” Robelle Prediction: Spring 1986 for a Spectrum that will reliably run existing MPE applications is not an attainable release date.

In Retrospect: The new relational HPIMAGE database mentioned here was cancelled much later in the project, after a brief encounter with end-users. I don’t remember much about HPIMAGE, except that a lot of work went into it and it didn’t succeed as hoped. TurboIMAGE ended up as the database of choice on the Spectrum. Without any inside information, but based just on past experience and common sense, Robelle tried to inject some caution about the 1986 release date. During the original traumatic HP 3000 project, Dave Packard “sent a memo to the HP 3000 team,” according to Chris Edler. “It was only two lines long and said, essentially, that they would never again announce a product that did not then currently meet specifications.” The division listened for over 10 years, but eventually, people forget….

September 20, 1985

From a Spring 1985 UK conference: Most existing peripherals will be supported and it will be possible to use networking software to link existing model HP 3000s to Spectrum, with the exception of Series II/III and 30/33. These would need a Series 37 or other current range machine to act as a gateway to Spectrum.

From an HP press release: “100 prototype models were already being used internally for system development as of April 1985.”
HPE, the new operating system for the commercial Spectrum is a superset of MPE. It will have two modes of operation: Execute mode (HP 3000) and Native Mode. The switch between the two will be made on a procedure call, but there will be some programming work needed to translate parameters when switching.

In Retrospect: Execute mode was eventually called Compatibility Mode and switching between modes turned out to be major CPU bottleneck in the new system, albeit one that would be removed over time.

The Spectrum is rumored at this time to provide 32 general-purpose registers to the user program and a virtual data space of 2 billion bytes.

December 30, 1985

From Gerry Wade of HP: The name of the Spectrum machine, when it comes out, will not be Spectrum. Another company already has that name. Spectrum will use the IEEE standard for floating-point arithmetic and will also support the HP 3000 floating point. Each data file will have a flag attached to it that tells which type of floating-point data it contains (the formats are not the same).

In Retrospect: The file flag idea never happened, although the TurboIMAGE database did introduce a new data type to distinguish IEEE floating point. Information on implementation details is starting to flow, which helps us believe that the project is on schedule and likely to deliver the more powerful servers we desire.

June 16, 1986

In reporting on Joel Birnbaum’s Spectrum presentation, the HP Chronicle had these observations: “Comparisons with Amdahl and DEC mainframes in slides showed areas where the Spectrum computers topped the larger machines’ benchmarks. ‘Even with un-tuned operating systems software, it’s significantly superior to the VAX 8600,’ Birnbaum said.”

In Retrospect: Joel was the HP Labs leader who was the sparkplug of the RISC project, building on research that he had done previously at IBM. In retrospect, we can see that Joel was talking about the performance and delivery of the UNIX Spectrum, not the MPE version, but customers took this as a promise of vast performance improvements in the very near future. It was now past Spring 1986 and the promised new 3000 machines were nowhere in sight. In fact, HP has not yet announced the new models and pricing. This was the first slippage in the project, barely noticed at the time.

July 20, 1986

Many people have been asking, “What is Robelle doing about Spectrum?” HP has invited us to join its Fast Start program for third parties and we have agreed. This program gives us pre-release access to Spectrum information and actual systems. We have visited Cupertino and run our software on the new machines. We are confident that all of our products will operate properly at the time that Spectrum is officially released.”

In Retrospect: Since Suprtool and Qedit were essential to the large 3000 customers that HP was targeting, HP asked Robelle to start porting and testing our products on the new systems. But to do that, we had to sign a Non-Disclosure Agreement, the most draconian one we had ever seen. We used careful wording in our announcement above. From this date on, until years later, we could not tell our customers anything useful about the new machines. HP was especially sensitive about their reliability and performance.

When we arrived in Cupertino to do our first testing, we found the prototype Spectrum systems crashing every few minutes and running slower than our tiny system 37. We were appalled. Nothing in HP’s public statements had prepared us for the state of the project. I had personally gone through a similar situation with the original 3000 in 1972-74, and I wondered if upper management at HP knew how terrible things were. I thought about talking to them, but our NDA also prohibited us from talking to anyone at HP.

The Unix versions of Spectrum, on the other hand, seemed to be humming along nicely, showing that it was not a hardware problem.


Your Guide to Image Logging

Pexels-kaboompics-com-6076
By Bob Green

The system is down – the hard drive is toast – and you may have to restore your IMAGE database from yesterday’s backup. In the past, this is the scenario that typically got HP 3000 system managers interested in the transaction-logging feature of the TurboIMAGE database.

But now, as a result of the Sarbanes-Oxley law (SOX), IMAGE Logging is also being used to create audits for data changes. Managers who have never used transaction logging before are now enabling it to create an evidence trail for their SOX auditors.

Here is an example from Judy Zilka, posting to the 3000-L newsgroup:

“As a requirement of Sarbanes-Oxley we are in need of an HP 3000 MPE system program that will automatically log changes to IMAGE data sets, KSAM and MPE files with a user ID and time/date stamp. We often use QUERY to change values when a processing error occurs and the user is unable to correct the problem on their own. The external auditors want a log file to be able to print who is changing what and when.

George Willis and Art Bahrs suggested IMAGE Transaction Logging:

Judy, we have enabled Transaction Logging for our TurboIMAGE databases coupled with a reporting tool known as DBAUDIT offered by Bradmark. For your other files, consider enabling a System Level logging #105 and #160. The LISTLOG utility that comes with the system can extract these records and provide you with detail or summary level reporting.

Hi George & Judy:

Yep, Transaction Logging will meet the requirements for Sarbanes-Oxley and HIPAA for requirements relating to tracking “touching” data.

Also, remember you must have a corporate policy relating to this tracking and either a SOP or a formal procedure for reviewing the logs. The SOP or procedure needs to address what constitutes normal and abnormal activity with regards to reviewing the logs and what action to take when abnormal activity is noted.

— Art “Putting on the InfoSec Hat “ Bahrs

P.S. The fines for not being able to show who did what and who has access to what can be very, very eye-opening! Of course these comments only apply to the US and businesses linked into the US.

So what is IMAGE logging?

First of all, it is not the same as “system logging” or system “logfiles.” These record MPE system activities such as logon and file open, and have their own set of commands to control them. You can see in George’s answer above that he suggests system logging to track KSAM and file changes.

IMAGE logging is a variety of “user logging” and is a part of the TurboIMAGE database application. Once enabled, it writes a log record for each change to a database. There are three programs that can be used to report on those database log records:

LOGLIST (a contributed program written by Dennis Heidner; I am not certain what the current status of this program is).

DBAUDIT (a product of Bradmark; in the spirit of SOX disclosure, I must admit that I wrote this program and it was a Robelle product before we sold it to Bradmark!)

AuditTool 3000, from Summit Solutions (www.sumsystems.com), created for ERP system logging and expanded to work with any 3000 application.

Setting Up IMAGE Logging

A number of MPE Commands are used to manage IMAGE logging; see the MPE manual at docs.hp.com/en/32650-90877/

index.html

:altacct green; cap=lg,am,al,gl,nd,sf,ia,ba

:comment altacct/altuser add the needed LG capability

:altuser mgr.green; cap=lg,am,al,gl,nd,sf,ia,ba

:build testlog; disc=999999; code=log

:getlog SOX; log=testlog,disc ;password=bob

:comment Getlog creates a new logid

:run dbutil.pub.sys

>>set dbname logid=SOX

>>enable dbname for logging

>>exit

:log SOX, start

:log SOX, stop

You can use the same Logid for several databases. For a more detailed description, see Chapter 7 of the TurboIMAGE manual, under the topic “Logging Preparation.”

IMAGE Logging Gotchas

Although the basics of user logging are pretty straightforward, there are still plenty of small gotchas. For example, Tracy Johnson asks about backup on 3000-L

“If when backing up IMAGE Databases that have logging turned on and you’re not using PARTIALDB, shouldn’t the log file get stored also if you store the root file? This question also applies to third-party products that have a DBSTORE option.”

He continued, “One problem I’ve been having is that since a log file’s modify date doesn’t change until it is stopped, restarted, or switched over, one might as well abort any current users anyway, so any log files will get picked up on a @[email protected] “Partial” backup, because DBSTORE and “online” (working together) features won’t do the trick. Because even though a root file’s modify date gets picked up on a Partial backup, the associated log file’s isn’t.

Then Bruce Hobbs pointed out that there is the Changelog command to close the current logfile before backup (which ensures that its mod-date is current and that it will be included on the backup) and start a new logfile.

Later Tracy ran into another interesting gotcha regarding logging and the CSLT tape

“If you use IMAGE logging, always make your CSLT the same day you need to use it! (Or make sure no CHANGELOG occurred since the CSLT was made. Thanks be to SOX...which forced IMAGE logging.)

“We added so many log files identifiers for each of our production databases it reached the ULog limit in sysgen of 64 logging identifiers. So, per recommendations of this listserv (and elsewhere,) I had to update the tables in sysgen and do a CONFIG UPDATE this weekend to bring it to the maximum HP ULog limit of 128. Not a problem. Stop the logging identifiers with “LOG logid,STOP” Shut down the system and BOOT ALT from tape. System came up just fine — UNTIL it was time to restart logging! Every logging identifier reappeared with old log file numbers a few days old. (We do a CHANGELOG every night and move the old log file to a different group.) I scratched my head on this one for half of Sunday.

<Epiphany Begin> Then it occurred to me, the Log file numbers the system wanted were from the day the CSLT was created. I had made it before the weekend, thinking it would save me some time before the shutdown! </Epiphany End>

Therefore:

a. Logging Identifiers retain the copy number on the CLST tape in the case of an UPDATE or UPDATE CONFIG.

b. Logging Identifiers on the system retain the NEXT log file they need to CHANGELOG to.

So if one needs to use a CLST to load and you’re using Image Logging, remember to use it just after you create it, or make sure no CHANGELOGs occurred since it was made.

This may effect some sites as they may believe their CPU is a static configuration and only do a CSLT once a month or once a week. In the case of an emergency tape load, to save some heartache rebuilding image log files, they may need to do a CSLT every day.