Tape backup: Set DLT to beat DDS

Backing up enterprise-grade 3000s presents more interesting choices today than 5-10 years ago. Back then DDS had only two generations, neither of which were reliable for certain. A DDS tape used to be the common coin for OS updates and software upgrades. The media has advanced to a DDS-5 generation, but Digital Linear Tape (DLT) has a higher capacity and more reliability than DDS.

When a DDS tape backup runs slower than a DLT, however, something is amiss. DLT is supposed to supply a native transfer rate of 15 MBps in the SureStore line of tape libraries. You can look over at an HP PDF datasheet on the SureStore, even certified by HP for MPE/iX, at this link.

HP 3000 community partners such as Genisys and Bay Pointe and Pivital Solutions offer these DLTs, and Orbit Software has an "order with our backup software" option, too. But at an estimated cost of about $1,300 or more per DLT device, you'll expect them to beat the DDS-4 transfers of 5 MBps.

HP 3000 customer Ray Shahan didn't see the speed he expected after moving to DLT and asked the 3000 newsgroup community what might be wrong. Advice ranged from TurboStore commands, to channels where the drives are installed, to the 3000's bandwidth and CPU power to deliver data to the DLT. HP's MPE/iX IO expert Jim Hawkins weighed in among the answers, while users and third-party support providers gave advice on how to get the speed you pay extra for in DLT.

Dave Gale wrote in an answer that device configuration and CPU are potential problems:

If you are using a DLT it likes to get data in a timely manner. Otherwise it will do the old 'shoe shine'. This means that other devices on the line can affect the bandwidth on the channel and starve the DLT. If you are using something like RoadRunner, then the CPU can be a real factor in this equation (especially single-CPU machines). So, you may not only want to check the statistics portion of the report, but monitor your machine during backup with Glance or SOS.

Gilles Schipper of support company GSA said that a TurboStore command is essential. "If HP TurboStore, are you using MAXTAPEBUF option on STORE command?"

HP's Hawkins said channel configurations of backup devices are key to ensuring that DLT tops the DDS speed:

Generally this shouldn’t happen. It might happen if the DLT and disc are on the same channel while the DAT/DDS was on a separate one. Might also happen with large numbers of small files on semi-busy system as some DAT are better at start/stop than DLT. If you are running STORE the STATISTICS option can give a broad indication of throughput for A/B comparison.

EchoTech's Craig Lalley, who's made a business out of upgrading HP 3000 storage devices, said that even when a DLT is moved to a different channel than the disk drive, you can do more. "The easiest thing to do is run the backup in the C-queue. Also, try turning software compression off."

Allegro Consultants' Stan Sieler offered a basic remedy. "I'd try a new DLT tape. I've found that helps at times."

3000 user Jack Connor testified to how much faster a DLT backup becomes with the best software parameters for backup commands. "MAXTAPEBUF and INTER can make a major difference," he said. "I recently had a backup to DLT cut from 7 hours to under 2 by just adding these parms."

We recently ran an article about ScreenJet's advice about large backups which skip the tapes altogether. STORE-to-disk (STD) counts on the reliability of a second disk mechanism, but DLT tapes have moving parts and magnetic properties, too. They just seem to cost a good deal more than disks which hold 40 times more than a DLT tape.

Tape backup: Set DLT to beat DDS

Backing up enterprise-grade 3000s presents more interesting choices today than 5-10 years ago. Back then DDS had only two generations, neither of which were reliable for certain. A DDS tape used to be the common coin for OS updates and software upgrades. The media has advanced to a DDS-5 generation, but Digital Linear Tape (DLT) has a higher capacity and more reliability than DDS.

When a DDS tape backup runs slower than a DLT, however, something is amiss. DLT is supposed to supply a native transfer rate of 15 MBps in the SureStore line of tape libraries. You can look over at an HP PDF datasheet on the SureStore, even certified by HP for MPE/iX, at this link.

HP 3000 community partners such as Genisys and Bay Pointe and Pivital Solutions offer these DLTs, and Orbit Software has an "order with our backup software" option, too. But at an estimated cost of about $1,300 or more per DLT device, you'll expect them to beat the DDS-4 transfers of 5 MBps.

HP 3000 customer Ray Shahan didn't see the speed he expected after moving to DLT and asked the 3000 newsgroup community what might be wrong. Advice ranged from TurboStore commands, to channels where the drives are installed, to the 3000's bandwidth and CPU power to deliver data to the DLT. HP's MPE/iX IO expert Jim Hawkins weighed in among the answers, while users and third-party support providers gave advice on how to get the speed you pay extra for in DLT.

Dave Gale wrote in an answer that device configuration and CPU are potential problems:

If you are using a DLT it likes to get data in a timely manner. Otherwise it will do the old 'shoe shine'. This means that other devices on the line can affect the bandwidth on the channel and starve the DLT. If you are using something like RoadRunner, then the CPU can be a real factor in this equation (especially single-CPU machines). So, you may not only want to check the statistics portion of the report, but monitor your machine during backup with Glance or SOS.

Gilles Schipper of support company GSA said that a TurboStore command is essential. "If HP TurboStore, are you using MAXTAPEBUF option on STORE command?"

HP's Hawkins said channel configurations of backup devices are key to ensuring that DLT tops the DDS speed:

Generally this shouldn’t happen. It might happen if the DLT and disc are on the same channel while the DAT/DDS was on a separate one. Might also happen with large numbers of small files on semi-busy system as some DAT are better at start/stop than DLT. If you are running STORE the STATISTICS option can give a broad indication of throughput for A/B comparison.

EchoTech's Craig Lalley, who's made a business out of upgrading HP 3000 storage devices, said that even when a DLT is moved to a different channel than the disk drive, you can do more. "The easiest thing to do is run the backup in the C-queue. Also, try turning software compression off."

Allegro Consultants' Stan Sieler offered a basic remedy. "I'd try a new DLT tape. I've found that helps at times."

3000 user Jack Connor testified to how much faster a DLT backup becomes with the best software parameters for backup commands. "MAXTAPEBUF and INTER can make a major difference," he said. "I recently had a backup to DLT cut from 7 hours to under 2 by just adding these parms."

We recently ran an article about ScreenJet's advice about large backups which skip the tapes altogether. STORE-to-disk (STD) counts on the reliability of a second disk mechanism, but DLT tapes have moving parts and magnetic properties, too. They just seem to cost a good deal more than disks which hold 40 times more than a DLT tape.

Making NAS work with the 3000

Network Attached Storage (NAS) is a powerful enterprise resource, full of value now that disk prices have plummeted. At the GHRUG International Technology Conference, Alan Yeo of ScreenJet shared his secrets for making NAS an HP 3000 tool.

"Like most HP 3000 shops we were looking for a cheap way to [store many gigabytes of data] — and there was no way we could afford a DLT," he said. Digital Linear Tape boasts massive capacities, but most storage these days is going straight to another disk.

Yeo said that fundamentally, the method to include NAS as an option is to create STORE to Disk files, "and then FTP those STORE files up to the NAS device. A simple half-terabyte (500 GB) RAID-1 NAS device is the equivalent of 40 12-GB DDS tape drives."

It's a little unsettling to learn how much HP 3000 backups still go onto DDS tapes. Even the DLT tapes are a pain to handle, Yeo added.

You need enough free disk space on your HP 3000 to do the STORE to Disk files, Yeo explained. "If you haven't got 50 percent free disk space and you're doing a complete backup in one hit, you're going to have a problem," he said.

STORE to Disk speeds are not significantly slower than STORE to tapes. One way to speed up the process is to have a few separate volume sets for these STOREs, sets that are two or more high-speed spindles. HP's got disks today which spin up to 15,000 RPM. Third party disks work with HP 3000s, too, in case HP hasn't got a certified product for your MPE/iX server.

FTP bandwidth can be a bottleneck for some older HP 3000s, sometimes as slow as 10 megabits per second. "You may have a protracted FTP process to your NAS device," Yeo said.

Using NAS is not a substitute for having a good SLT tape for your system in case of disaster. Yeo added that doing an @.@.SYS backup onto the same SLT tape, "so you'll have everything you need when you bring the box back up to get the networking started."

Devices available for HP 3000 NAS use? The Buffalo Terastation Pro worked in one of Yeo's projects for a client, and the device starts at $650 and goes up for 1-4 TB. Another choice is the Infrant ReadyNAS at the same price point. Shop online.

It may seem crazy to be ordering HP 3000 storage devices from Amazon.com. But so much has changed for the HP 3000 customer, and some of the change opens up new opportunities to save money and make your server even more efficient.

Homesteaders remain in place, sound off

The HP 3000 homesteaders remain where they expected to be working when we polled them in 2004, meeting challenges of support but counting on few changes. The community checked in on the OpenMPE mailing list in recent weeks. Terry Simpkins, IT Director of Measurement Specialties, said his firm is using HP 3000 systems for “general ledger, accounts payable, inventory control, purchasing, production scheduling, order entry, and invoicing. With 11 locations around the world, we have a substantial investment in its continued operation.”

   Simpkins, who has established manufacturing IT operations in China over the past five years, was a customer spokesman in ads for HP 3000s in the years just prior to HP’s exit plans.

   Zelik Schwartzman of Estee Lauder Companies said “We are actively installing SAP; however as far as the HP 3000 is concerned we anticipate this system will be around for many many years to come as we use it as our MRP engine.”

   Catherine Litten of Valley Presbyterian Hospital said even through another information system has replaced its 3000, “it doesn’t look like the HP 3000 will be going away, as it has become our data repository for historical reporting.”

   Even a successful migration won’t turn off a large 3000 installation. Mark Ranft of Pro 3k said that “My largest client has over 30 HP 3000 systems. Most of these are N-Classes with a few large K-Class systems tossed in. They are hard at work trying to complete their migration. The time and effort required to migrate will continue until at least 2011. After that, the systems will remain for historical purposes.”

    Several customers who sounded off on the OpenMPE list hailed from large installations such as Ranft’s client, and they pointed to the 3000’s extraordinary lifespan.

    Peter Martin, the IT Operations Manager for Chubb Electronic Security Systems, said his company runs three HP 3000s “for the foreseeable future.“ Although all systems have HP support through 2010, Martin plans to use the independent support market to keep the 3000s running, in spite of HP’s strategy.

   “I think the problem is the 3000 was too good, with no built in obsolescence,” he said. “That’s why HP killed it — no future revenues outside of support.”

Migrations remain in play; 2006 targets slip for some

    In the shifting world of the 3000 community, the status quo has been easier to predict than the wave of the future. Among our survey of companies which planned to leave the platform during 2006, two out of three have hit their target.

   Migrating off the platform has been an effort that has fallen behind, but not for lack of trying. Meanwhile, the 3000 owner who’s homesteading until forced off the system spoke up over the Internet in recent weeks, identifying companies which are sized both large and small.

    The migration companies reported to The 3000 NewsWire in 2004 that they planned to be off the system before the end of 2006. The companies we surveyed identified success with migration from the 3000 to Unix platforms and Windows. But some have been hung up waiting on replacement software.

    These customers haven’t been caught unaware of how a migration deadline could slip. More than three years ago, Diana Wilson of Roanoke County, Va., said “The only reasons we would continue the 3000’s use beyond 2006 would be that we cannot find a vendor that can offer replacement applications or we are behind schedule in implementing the new apps.”

    Those are the classic reasons for a migration delay, and both turned out to be slowdowns for the county, Wilson said in an update. The county is following a replacement application strategy.

   “We lost about a  year of time when one of the new software vendors went bankrupt while we were in the middle of implementation,” she said. “This caused us to have to start all over with the bid and award process for those applications.  We’ve now selected vendors for all of the remaining applications and have implementation and deployment schedules for each.  Our schedule is aggressive, but so far it looks like we will be able to meet the 2008 deadline.”

    Several of the companies have migrated all but one or two 3000 applications, systems they are keeping online for archival purposes or just running with no migration date.

    Greg Barnes of Media General reported that “We’ve migrated all apps to HP-UX and Solaris, but we’re keeping two around for archival info lookups. One for lookup, one for disaster recovery at a different location. I’m the last HP 3000 system administrator in the company, and possibly in the city — so I’m always on call.”

    While about a third of the companies were running late, few companies reported that they had changed their decision to migrate. “Our plans fell through in 2006,” said Pedro Gonzalez of the health plan Dr. Leonard’s Healthcare. “We decided to homestead with no plans for conversion at this time.”

    At another site, the migration has stretched from a 2006 deadline into 2009. “Our Web Wise and Data Warehouse software modules are functional on Linux, while our batch reports and back office screens are still functioning on the HP 3000,” said John Wardenski, president and CTO of National College Management Systems. “Batch reports will go live this fall on Linux, with our back office screens the following summer.”

    Summit Racing Systems, whose migration has been publicized by several migration suppliers, remains behind schedule. “We are still in the very slow process on migrating off the HP 3000,” said Ron Pizor in IT Operations. “The current migration date is July 4, 2008. Not sure if we will make it. Later in the year would be more realistic.”

    For every delayed migration, however, two more said they just about hit the end of ’06 plans offered to us during our poll in 2004.

    “We did make it off the 3000, but later than originally planned,” said Lane Rollins of Boyd Coffee Company. The organization migrated in July, 2007 rather than by the end of 2006.

    “Some of the delay was due to also replacing our handheld computer system that our route trucks use,” Rollins said. “We had health-related issues pop up on the core team, and that slowed us down.”

    Rollins’s company originally planned to have a go-live date of Fall, 2005 for its replacement applications. He told us during his 2004 reply that surveying business processes was invaluable in making a successful migration.

    “I can’t say enough about value of doing the process mapping. We are a 104-year-old company and there is a lot of baggage. I knew we had some broken processes, but until we got into it I didn’t realize how bad some were. We’ve done a little future state mapping, but for the most part that is waiting until we have selected the vendor.”

    Jennie Rethman of Mac Equipment reported that Oracle’s E-Business Suites replaced the company’s 3000, “and Oracle is working out great for us.” MANMAN was running the manufacturer’s data operations until 2006. IT manager Will Bauman of Kato Engineering checked in to say that “we have migrated all the applications except for one. That migration should be done by the end of the first quarter of 2008.”

    Some delays revolve around end-users in a company’s base. “We did not stick to the 2006 date, but the primary application was converted to HP 9000 in July, 2007,” said Paula Brinson of Hampton Roads Sanitation District. “We’re still suffering conversion pains and the users have chained themselves to the legacy system to prevent my decommissioning. They are using it for various data validation activities.”

    In one case, a migration has been delayed because the customer is waiting on a Linux version of their current HP 3000 application. Sutter County, California Schools plans to migrate once its vendor’s Linux version is ready for them.

    Even Unix can be an environment left behind in the process, however. Ken Williams reported that the Azusa Pacific University already moved its 3000 applications to Sun’s Unix — but it’s leaving Sun’s platform by 2009, “and we have buyers for the Unix boxes, also.”

    The IT managers of some companies expressed the usual regrets about switching off 3000s. “The HP 3000 application was replaced with a Windows-based package, said Byron Youngstrom of Weyerheuser. “I miss the HP.”

Mine those reports and data for value

When it comes to experts on ERP, the customers who use MANMAN are hard to beat. You take an class of users running an application that's been powering manufacturing sites for more than three decades, then carry the solution into the 21st Century, and you get a critical mass of expertise. Data mining is one skill these customers are practicing, an art that makes an HP 3000 work harder at a company, no matter how long the system's expected lifespan might be.

The heart of MANMAN savvy resides in CAMUS, a user group that began in the 1970s and is still holding conferences and putting out newsletters. The latest CAMUS news came to us over the holiday, an issue that includes an article on data mining of MANMAN information. Mining for data is not much different than mining for high-grade ore or gems; what you net is of greater value to your company.

Inside the latest issue of the CAMUS newsletter, Eric Estes of Titan Tool outlined how he used Monarch, a PC-based tool from Datawatch, to reformat reports from MANMAN. The GUI revival never did make it to MANMAN, at least not without third party solutions. Estes showed how a standard MANMAN report that looks like this (click to get a larger version):

Can have key information extracted by an administrator and imported into Monarch, to end up looking like this

Monarch may be very well known, but it is far from the only tool to be able to transform MANMAN's information into a more powerful corporate asset. Estes's example illustrates how the mission-critical data reports can be refined with third party solutions. The MBF-UDALink from MB Foster performs this kind of magic, and a lot more, by creating a data mart out of MANMAN information. Monarch works all over, but it has limits to what it can do.

A full data mart, as opposed to the relatively simple reformatting of a report, gives a company a wide range of solutions. A UDALink data mart can offer views such as vendor performance reporting, lead time analysis, reports on where components are used, tools to plan and forecast sales, and detailed operational analysis, just as examples.

Both of these products have the power to pump up a 3000's data, which often has untold value because it covers a decade or two. The Monarch solution sells for less than $1,000, but as a data mart tool it simply doesn't show up. Monarch makes existing MANMAN reports look less complex, something that regular end users can make some sense of. As Estes reported in his article

I’ve got the fields highlighted that I want to extract from the report, using what’s called a template in Monarch, and once I execute the import I wind up with all of the fields I require in what Monarch refers to as a table. I can then name the fields any way I want, or simply use the default names.

It looks much like an Excel spreadsheet with the field names across the top and the part numbers and related data in rows from top to bottom. A few more clicks of the mouse and the results are exported to an Excel work sheet. I primarily use this file as a source to lookup data for other exports.

A data mart supplies a more complete solution, considering what it delivers. At MB Foster, in addition to UDALink, the company delivers

• A set of procedures for extracting and loading the data into the data mart structure
• File Definitions (FDs) modified to suit your specific requirements
• PSO personnel working with your team throughout the entire process to ensure a successful implementation
• Deploying into your environment
• Supporting the initial load

Questions good, no matter how late they're asked

Even while HP 3000 customer Korry Electronics is searching for HP 3000 expertise to fill a position, the company is figuring out its strategy for the rest of this decade. On the MANMAN user mailing list, Deborah Lester asked good, fundamental questions about the Transition Era.

Fundamental questions are often the best kind, even if they seem to be asked long after other people have heard the answers. I liked Lester's list of questions so much I'd like to share some of my answers, as well as ask for yours on few. Experience makes us all smarter.

1. Does any vendor refurbish HP 3000s legally without HP? What does this entail and how do we know those vendors from other vendors? Will others emerge?

If refurbish means upgrade, there are upgrade kits for sale on the third party market, as well as from Client Systems, which was the last authorized North American HP 3000 distributor. (With no official hardware resellers anymore, no distribution has taken place through HP since 2003.) The "legally without HP" part of the question gets more complicated. HP insists that every customer has a Right to Use License now for their HP 3000s. So an upgrade can involve a license transfer, if you're taking on a system from another customer.

This third party market in this community has the hardware which customers need to keep running. For a customer who recognizes the authority of a Right To Use (RTU) license on a 3000, HP's License Transfer process can make it simple to "know those vendors from other vendors" while doing a "refurbish."

2.  Has anyone had a business case for HP to convert an HP 9000 into an HP 3000?  What does it entail and how long does it take? Does anybody have a firm commitment from HP to create a HP 3000 from an HP 9000?

Many customers have had business cases for converting 9000s to 3000s, but HP has never recognized one. This "personality" of the PA-RISC server is set in Processor Dependent Code, which HP claims cannot be modified by anyone except Hewlett-Packard personnel. (Note: HP 3000 owners have done this kind of modification, aided by vendors, without regard to HP's wishes.)

3. When buying an HP 3000, how do you transfer SUSAN numbers legally?

Now here's an easy one. Or at least the answers are easy to understand, since HP has documented the transfer process in great detail at it Web site. The transfer costs $400 to precess, and you must provide proof of purchase from prior owner. The sticky part turns out to be The Proof, as HP calls it, that the 3000 had a legal MPE license to begin with. HP has several forms and elaborate instructions for this.

4. How should parts and entire HP 3000 systems be stored if stockpiled?  What ration of parts will be functional after being stored for 6-24 months?  What will be the level of workability for parts and systems after 2008?

We know of people who are shrink-wrapping old disk drives to stock parts. But nearly everything can be found on the used hardware market. Some of it really cheap. You could buy a similar spare system for parts alone. Keep in mind that these systems were built to last (more on that next week). Hardware breakdown, aside from power supplies and disk drives, is not the risky end of staying on an HP 3000.

I do wish others would report on how long a part might be useful when stockpiled. HP 3000 components don't have "sell by" date stamped on them

5. Will MPE emulators emerge? Will Infor allow emulators or have any control over what machine we run MANMAN on?

Emulators of HP 3000 hardware are being considered, even designed. But their marketability will be many years off. Strobe Data, if anyone, will be the company to offer that solution, really an emulator of PA-RISC servers. But the used HP 3000 hardware supply will stand in the way of any real market for an emulator.

As for Infor, the owners of MANMAN, the company hasn't stopped the application from running on any particular kind of HP system, unless that system runs Windows or Unix. A year or so earlier, there was talk of porting MANMAN to HP-UX, but it's only been talk. The emulator availability for HP 3000 hardware is so far off that Infor probably won't even own MANMAN by then.

6. Will new 3000 drivers be available or emerge?

HP has probably built its last device driver for the HP 3000. It's called SCSI Pass-Through, and it ws released earlier this month.

7. What will be the HP 3000 inventory after 2008?

You might as well ask, "where does the sun first start to rise on Earth?" The question puzzles all of us in the community, because HP has kept no inventory of how many systems were made, sold or resold. Nobody can even guess about 3000 inventory after 2008. Nobody knows how many are running now. Estimates run from 2,000 to 5,000 systems, but you can get a different number from anybody.

The real question might be, "Can I get enough inventory in 2009 and beyond to stay on my existing MPE/iX application?" The answer is "yes, if budget is not an issue. We haven't heard of any company who cannot find an HP 3000 this year, once they'd committed to buying one.

Lester also wanted to know how to safely move HP 3000s to a new facility. That's an answer for an expert we would like to hear from. If you have a "moved our 3000 recently" story, share it with us. It's a fundamental question about a complex process — something like being a 3000 community member during this Transition Era.

Rebooting can be optional on new patches

While we call down the creative thunder of the promised podcast for this week, we decided to update the process to apply the latest HP patches, software the vendor has labelled critical for all HP 3000 customers.

OpenMPE's director Donna Garverick-Hofmeister clarified the need for reboots to apply those Large File fixing patches HP released on Oct. 31. One patch won't need a reboot, she says, while the other needs a reboot but can be staged.

Oh, and the no-reboot patch, to avoid data corruption in very big files, cannot be staged, according to Donna.

The HP 3000 millicode patch (MILNX10A), the first in 16 years, is needed to repair access to any in-house applications that have used Large Files, or do a sort with a temporary file that can exceed 4GB. If your app has not been modified since March 30, 2000, it's safe. That's when HP introduced the Large File feature.

Large Files has been engineering which HP has been trying to remove from customers' 3000s. A November 2006 patch was designed to turn off Large Files and get those files on the system converted to Jumbo files, much better engineered.

Donna says that HP's advice that both patches will require a reboot, and that both can be staged, is not what she understands as a 3000 administrator of several decades.

MILNX10A is NOT stageable because it requires a installation job. in my opinion, this patch is best/fastest/most easily installed by using ‘autopat’.  autopat, at its conclusion, will say ‘stream this.job’....which you do....and a couple of blinks later....milli.lib.sys (and friends) is joyfully updated.

On the other hand, MPENX11A is stageable and is the patch that requires a reboot. this time, use patch/ix to get this patch staged and schedule a reboot.

dig out your notes about how to download patches. pay attention when you run ‘unpackp’ and the rest is all pretty easy.

Critical bug patches raise concern among experts

Article revised Nov. 2, based on HP responses

Yesterday's report on a critical patch for HP 3000s sparked immediate response from the user community, especially those experts in the 3000's internals. These experts believe the nature of the problem might require more than binary repair patches, to serve the long term needs of 3000 customers.

[HP disagrees, and its response is available in a Nov. 2 report]

HP intends to create these binary patches through 2010, as it said yesterday. OpenMPE advocates say they are concerned that these fixes will present a challenge to application developers who will need to integrate them into MPE/iX in the future. OpenMPE wants to do this work.

"We've done as much testing as we could get done," said HP's community liaison Craig Fairchild. "There has been some field testing, and a lot of in-house testing." He added that HP scanned its internal 3000 applications to test the abilities of FILECHEK, which finds Large Files.

Our report of yesterday may have been too cheery about the chance of hitting this corruption bug. Our Oct. 31 story estimated that in one case, users risked just an 800 million to 1 chance of hitting one of five bytes at the end of a sort of a 4-billion-byte file that could corrupt data. Stan Sieler at Allegro Consultants, a Resource 3000 partner, said it's not that rare.

We found it fairly easily. 133 file sizes from 2 to 32,766 bytes per record can directly encounter the problem ... but only for Large Files. (As it happens, 256 bytes isn’t one of them, nor is any power of 2.) We stumbled over one while doing a large sort.

Files of about 2 GB or more, and of any record size, can encounter the problem while being sorted, because HPSORT creates a scratch file whose record size isn’t identical to that of the input file — if it happens to create a file with one of those 133 file sizes, and that scratch file is 4GB or more, then it can run into the problem.

Most people don’t have Large Files ... they could be affected only if they happen to sort files that are bigger than about 2 to 3 GBs.  (And, even then, there are only a about 152 file sizes out of 32,766 that might trigger the problem.)

Systems without Large Files are safe. Adager's Alfredo Rego posted to the HP 3000 newsgroup about an hour after HP announced the critical patches. Rego said his lab found the problem in August, after an earlier HPSORT patch introduced a new problem while not solving the initial bug.

Rego's exploration also raised the question of a more serious issue in the nature of HP's repairs, one echoed by MPE/iX veteran and MPE-education.com co-founder Paul Edwards.

"There's some strong issues here," Edwards said. "I'm concerned that these binary patches were not done in the source code. So HP didn't generate a General Release patch, which means it may or may not be tested on all three of the supported versions of MPE/iX. There was no beta patch, or anything."

[HP said in its Nov. 2 reply that the patches have both been beta-tested and General Released.]

While Rego congratulated HP on the speed of its 'tremendous efforts" in creating the patches, he also took note of the lack of notice about third parties' assistance in isolating the problems.

It appears that HP continues its policy of not acknowledging any of the help it has obtained from its hard-working partners. That’s okay, because I have reason to believe that this is forced upon our vCSY friends by HP’s corporate legal department.  Boy, how I miss the days when I used to walk the floors of HP’s buildings while chatting with Bill Hewlett and Dave Packard in the early 1970s.

The Adager team described the problem with move_fast_64 to HP on August 8, 2007, after we discovered that the first HPSORT patch not only did not solve the problem but introduced a new one by using move_fast_64.

Craig Fairchild implies that only non-HP applications might be affected.  What about HP applications? I believe that the “move_fast_64” procedure is used extensively by HP within MPE/iX.  Has HP taken care of all the possible problems within MPE/iX?

Adager calls HPSORT programmatically.  We detected, analyzed, documented, and reported the initial problems to HP during tests on our own HP 3000 systems as well as on some large customers’ systems.

The problems encountered by Adager customers were not caused by Adager code, but by HP’s code.  Once the “latest” HPSORT patch was applied, the problems went away.  The patch ID we tested is MPENX06, which is different from the patches mentioned by Craig.  It seems that patch MPENX11 includes the fix for HPSORT.

Craig does not mention that the problem can exist while using simple calls using FREADDIR on files with bug-exposing characteristics.  He does provide a nice technical description under the section on “MPE/iX OS millicode handling of long pointer access to large files.”

It’s not clear (to me, at least) whether the patches identified by Craig are already available as General Release or still in beta.  We have not received either one of them, even though we generated the original bug report.  Is HP confident that the problem has been completely resolved without any user testing?

Adager users are safe.  Even when they may experience the problems that Adager exposes when calling HPSORT without the patch, Adager will leave the databases unchanged after reporting the problem.

Comparing next solutions for 3000s

When a 3000 utility goes dark — because its creator has dropped MPE/iX operations, or the trail to the support business for the tool has grown faint — the 3000 community can serve up alternatives quickly. A mature operating system and experienced users offer options that are hard to beat.

Such was the case last week when Walter Murray, a former HP development engineer now with the California Dept. of Corrections IT staff, wondered about an alternative for Aldon Computing's SCOMPARE. That development tool has compared source files for more than 15 years in the HP 3000 world. There was no record of a valid license on the Murray's server for How now to compare, Murray wondered.

Not for long. Within 24 hours the experts on the HP 3000 mailing list offered six alternatives to the now-defunct SCOMPARE. Resource 3000 partner Allegro Consultants offers a free MPE/iX solution in SCOM, as verified by Allegro's Steve Cooper:

And, it's free, too!

www.allegro.com/software/hp3000/allegro.html

and scroll down to "SCOM."

Other candidates included a compare UDC from Robelle, GNU Diff, diff in the HP 3000's Posix environment, DiffDaff on Windows, and more.

Bruce Collins of Softvoyage offered details on using diff in Posix:

run diff.hpbin.sys;info="FILE1 FILE2"

The file names use HFS syntax so they should be entered in upper case. If the files aren't in the current account or group they should be entered as /ACCOUNT/GROUP/FILE

Donna Garverick-Hofmeister, after verifying that Aldon is still in business, but not the MPE/iX business, offered a tip on using Robelle's compare UDC:

Regarding Robelle's compare.  Being a scripting advocate, I strongly recommend adapting their UDC into a script.... and take a few seconds to add a wee bit of help text to the script, to make life more enjoyable for all (which *is* the reason for scripting, yes?)

In Microsoft's Visual Studio lies a tool called windiff, reported Larry Simonsen. Another former HP engineer, Lars Appel, brought up a Linux option in the KDE development environment:

On Linux, if you are using KDE, you might also find Kompare handy...

http://en.wikipedia.org/wiki/Kompare (see screenshot)

On MPE, as others mentioned, there is still the Posix diff in two flavours: the HP supplied in /bin and the GNU version that lives in /usr/local/bin. The former allows two output formats (diff and diff -c), the latter also allows “diff -u” in addition.

Oh, regarding /bin/diff on MPE... I sometimes got “strange” errors (like “file too big”) from it when trying to compare MPE record oriented files. A workaround was to use tobyte (with -at options) to created bytestream files for diff’ing.

Appel was even able to address a concern of Murray's: "Then there’s the problem of comparing numbered files, like COBOL source files, when one or both files have been renumbered."

With Posix tools, one might use cut(1) with -c option to “peel off” the line number columns before using diff(1) for comparing the “meat”. Something in the line of ... /bin/cut -c7-72 SourceFile1 > BodyText1.

Murrary reported back at the end of the 24 hours to say that Aldon knew of HP 3000s and the licensing mechanisms, "and it sounds as though they are still willing to sell SCOMPARE and support contract for it." Which says something about the vigor of the 3000's ecosystem, six years after HP predicted its demise.

Ubuntu option opens for Linux

Pete Eggers, an MPE/iX veteran looking toward the next best operating environment, describes himself as a Linux bigot. Not long ago he echoed another 3000 expert in recommending Ubuntu, a distro of Linux as a newly-polished tech tool.

Vista got you down?  Slow on last generation hardware?  Can’t load it on old hardware?  Just need the basics including a full featured office suite?  Or, a bunch more applications (18,000), but don’t have money to burn?

Give Ubuntu 7.10 a try!  Comes in a LiveCD version (runs completely from a bootable CD without need of even 1 disk).

Earlier this year we interviewed Matt Perdue, Hill Country Technologies support specialist and an OpenMPE board member. Ubuntu was on his list of tools to assist in 3000 administration. He'd even gotten the Linux distro to boot up on PA-RISC hardware.

Eggers offered an outside look at the Linux distro, along with a note about the 7.10 release.

Here is an O’Reilly review link:

Yes, I know CentOS distribution is great, ultra-reliable (just repackaged RHEL), and easy to setup as a server.  Not bad as a programmer’s workstation too.  But it is really dull, uses old reliable software versions, missing features as desktop as delivered, and time consuming to setup and configure as a full blown jazzy end-user desktop.  If you are a disgruntled Vista user, Ubuntu is the Linux distribution to try first.

What 2008 means to migration

HP told customers last month that it will extend its product support for the HP 3000 through 2010, although the two years beyond 2008 will not include any new MPE/iX patches. One reason for the extension: Customers need more time to migrate. Sometimes those delays are completely unavoidable.

The County of Roanoke in Virginia is moving its governmental applications off their HP 3000. Just about a month before HP extended its support date, 2008 was looking like a very rigorous deadline for IT director Diana Wilson. She had reported back in 2004 the county expected to be all migrated by 2006. We checked back in with her and several other migration-bound customers.

Our current schedule puts us out to 2008. We lost about a  year of time when one of the new software vendors went bankrupt while we were in the middle of implementation (we are purchasing vendor applications to replace all of the HP 3000 apps). This caused us to have to start all over with the bid and  award process for those applications.

Yesterday's story about delays due to vendor changes offers counsel on what to do when a new app provider leaves your radar screen — or has its flight pattern changed by an acquisition.

Wilson reported the county's plans to us a month before HP's most recent extension of support, making reference to the 2008 deadline the 3000 community understood at the time.

We've now selected vendors for all of the remaining applications and have implementation and deployment schedules for each. Our schedule is aggressive, but so far it looks like we will be able to meet the 2008 deadline.

There are other customers who see HP's extension of support as a reason to keep the 3000 on the support price list indefinitely. There are no firm numbers to examine about HP's 3000 support revenues or profits. HP doesn't release these figures, so the rumors of $100 million annually in service contracts and more than $20 million in profits will just have to remain rumors, unverified. HP Services is one of several HP operating groups who have a say in how long HP will stay in the 3000 business.

But one customer who replied to our story about the extension of HP support made his case without referencing what might be in it for HP's bottom line.

"I think they should do it indefinitely," said Michael Caplin of Aero Corporation, manufacturers of safety products. "HP can freeze software updates and software support, but what's the harm in supporting the hardware?  It's the least they can do after abandoning the very customers who put them where they are today."

Java: Useful business tool that needs updating

Customers were chatting this week about Java in the business environment: whether this language has a place in creating applications and clients for enterprises. The consensus on the HP 3000 newsgroup is that the language once touted as "write once, run anywhere" has already earned its stripes across the world.

That's the biggest reason, perhaps, that Java is in sore need of updating on the HP 3000. In yet another project where HP's forthcoming open source white paper can help, Java needs to be rejuvenated from a 2000 version last updated by Mike Yawn, the 3000 division lab expert who was the Java go-to guy for years.

Yawn even made it a point to report on the Java One conference for several years. HP laid him off, more than once, until finally this superior technical resource landed at Quicken, expanding his reach beyond your venerated HP 3000 system.

Even though Yawn is probably out of reach now, Java improvements are another mission that an independent lab effort could tackle if interest and income could be tied to the technology. It's easy to see how this language that HP announced with great gusto for the 3000 in 1997 can make development easier a decade later. Mark Wonsil, a sharp developer with XML and Web savvy reported on Java's bounty

I have written Java programs for database access across multiple platforms. Type 4 JDCB drivers require no licensing on the machine that you’re on, so I was able to access SQL Server and Oracle at the same time - without using ODBC - very cool.

Using Java opens the door to many of the more recent technologies. Charles Finley of Transformix, a migration consultancy working in the HP 3000 community, says Java is pervasive.

The simple answer is yes, Java is widely used as a business language.  What makes the real question complicated is the number of different ways it is used and how it expands the definition of business language.  In order to really understand how pervasive it is you need to delve into such variations as J2EE, the various languages that run in the JVM (Java Virtual Machine) such as Jython and Groovy, Rich Client Architecture, etc. as well as the same kind of conventional applications that a language such as COBOL is used for.

Here is one resource that talks about the popularity of JAVA:

http://www.welton.it/articles/language_popularity.html

Older tech has big advantage: it works

You may catch a bit of razzing about using an HP 3000 in 2007. Slurs like dinosaur and ancient get tossed at you or your company. But in Britain the oldest tech in networking is still a mandate for some British Telecom (BT) products. What's more, key parts of the government use ISDN.

And you might have guessed, an HP 3000 is powering some of the ISDN in the country, according to today's article in The Register. Nobody wants to unplug ISDN — other than the providers who want to sell something newer. (Sound familiar to some of you?)

Why? As the article says, it works.

Guy Kewney writes, "Even if you could provide the signalling and interface "presentation" of ISDN to customers today, you'd have trouble replacing what ISDN is famous for: working."

As one veteran of the business told me: "It's like those old HP 3000 minicomputers. People installed them way, way back and they haven't touched them since. I know of ISDN2e installations that went in before 1980, which did a simple once-a-day dial to the ISDN link at head office, transferred a batch of data, and hung up; and they're still doing it 30 years later."

And the problem is, if you change a thing, the software behind it might stop working, and nobody knows what it does or how to adjust it if it stops.

"I went into one of our clients," said a sysadmin at a large software company which handles vehicle tracking applications. "We asked where the gear was, and nobody there knew. We had to track the cables and, eventually, under a load of old rags - literally - we found this HP 3000 connected to the ISDN socket, working. It's been doing that for decades!"

3000 community veterans and partners know this kind of story is common among 3000 customers. The ISDN tale points out the opportunity and reliability cost of making a migration. Not to be overlooked. Also in this category: Testing of the new platform. Every presentation during the last three years to promote a migration includes this line, or words to the effect: you can't test enough.

How to make IDE drives serve your 3000

Whether you are managing a migration shop, and keeping 3000 costs down until shutdown, or homesteading independently and indefinitely, HP 3000 guru Stan Sieler has discovered a process to make cheap IDE drives work with HP 3000 servers.

These drives come as cheap as $99 new, a tiny fraction of the price of HP-certified storage devices for the HP 3000. But as the community has debated in the past, HP's standards ensure greater reliability, but at a increased cost per gigabyte that can be hard to justify in a Transition world.

The key to Sieler's connection between SCSI ports on the 3000 and IDE is the Acard AEC-7720U Ultra SCSI-to-IDE Bridge, a device that Sieler bought on eBay for $30 plus shipping. The Acard has been on the market for at least four years; more information is available at the company's Web site.

Sieler reported, "I installed a Maxtor 120 GB IDE disk drive (probably 3 to 5 years old) on an HP 3000/918."

I took the Maxtor drive, plugged in the SCSI/IDE adapter,  selected a SCSI ID (I probably could have made it cable select, but I didn’t have the time), put the combo into a spare external  drive box, plugged in the SCSI cable and a power cable (it was a special power cable that splits the normal plower into two connectors, the normal (crappy) connector that I hate, and a smaller connector to power the adapter.

"I added the box onto a 918, sysgen'ed it in, rebooted, and voila!" He filled us in on the details of mapping the drive into the HP 3000's registry of peripherals:

SYSGEN is used to build (and display) an I/O configuration, and makes no requirement that it matches reality.  The only oddity I’ve encountered so far is that the initial “DOIONOW” didn’t notice that the drive was present ... but I wouldn’t be surprised to find that that has happened on standard SCSI drives from time to time, given the amount of user testing DOIONOW probably doesn’t get.

SYSGEN followed by DOIONOW failed to see the drive.  I shut down the 3000, power cycled it, brought it up to below the  ISL prompt, and did a "search" command, and it saw the drive!

Chronologically:

   - I ran SYSGEN and added the drive in as LDEV 2 (small system :) powered it on, and then did a “DOIONOW” (after confirming that the commands in LOG4ONLN.PUB.SYS were correct, of course ... I’ve often seen stale date in that file).  The I/O configuration change worked (in that LDEV 2 was now configured), but MPE didn’t realize that the drive was spinning.

     (I didn’t think to power cycle the drive at this point ... that may have worked and let me avoid the reboot)

   - I shutdown the system

   - I started the bootup, interrupted it so I could interact with the PDC (“below the ISL”) ... (that’s where you’d normally type the “BOOT PRI” command to boot into the ISL)

   - I typed “SEA” (for “search”), and it found the drive and reported it as a Maxtor drive with firmware revsion “YAR4”.

   - I did a “BOOT PRI”, and said “yes, I want to interact” (that got me to the ISL prompt)

   - At the ISL> prompt, I entered: ODE MAPPER at the MAPPER prompt, I entered: RUN
    

That displayed a map of the I/O devices (and memory/CPU) on the system:

     
Type  HW    SW    Revisions
Path       Component Name                 ID    Mod   Mod   Hdwr  Firm
---------- ----------------------------- ----- ----- ----- ----- -----
...
56/52      HP-PB SCSI                     4H    14H   39H   0     0
56/52.0.0  HPC1504(X)/HPC1521B DDS tape   -     -     -     -     T503
56/52.5.0  Maxtor 6 Y120P0                -     -     -     -     YAR4
56/52.6.0  HPC2490AM disk drive           -     -     -     -     5193

      At this point, I could have chosen to use some other ODE utilities
      to explore the disk drive, but I felt it wouldn’t be worth the time.
      (Normally, if the drive appears in a SEA command, it’s cabled
      correctly and working.)

   - I exited MAPPER, and got back to the ISL> prompt.

   - I did a “START NORECOVERY”

   - When the system came up, it saw LDEV 2 (and gave the usual misleading message (bug :) about it being a duplicate of some other drive ... this almost always happens when MPE tries to do volume recognition on a non-MPE disk, and can be ignored in this case).

   - I used DEBUG to confirm that I could read the drive:

       (logon as MANAGER.SYS, or some user with PM capability)

       :debug
       dsec 2.0, 40, b
       c

   - I used VOLUTIL to initialize the disk and create a volume set:

       :volutil
       newset maxtor, maxtor1, 2
          (this results in a drive with permanent and transient limits set
           to 100%)
       exit

   - I created a group on the new disk:

       :newacct test, mgr
       :newacct test, mgr; onvs=maxtor
       :altgroup pub.test; homevs=maxtor

   - I restored a database onto it:

       (on a different machine)

       :store @.pub.test

       (on the machine with the Maxtor IDE drive)

       :restore ;;show;olddate;keep

Voila!  I now had my 5 GB database on my 120 GB Maxtor drive!

It’s available as a normal retail item, but is somewhat hard to find (not a lot of call for SCSI compared to standard PC and even Mac items.

This adapter has been on the market since at least 2003 (based on the date on the manual).

When should you encrypt?

In yesterday's entry, we tracked the options available for HP 3000 data encryption. None looked simpler than the Orbit Software product, Backup+/iX, now engineered for 256-bit encryption of data during backups only. The backup-triggered encryption minimizes performance drain, a potential pitfall of encrypting data.

But the question of when to encrypt surfaced just a few hours after the discussion of product and freeware solutions. Tracy Johnson observed

Encryption of data on the host itself is really a waste of time. Why? Unless there is no access control at the host? Encryption during transmission between two computers is usually how it is done because that is when data is vulnerable.

Pete Eggers, whose name has been mentioned as a potential OpenMPE director, replied that the moment of encryption was not clear from the customer's question: how to encrypt a dataset in a TurboIMAGE database.

There is nowhere enough information presented to say that host data encryption is a waste of time — nor enough information to say that any form of transmission of the data warrants
encryption.

Johnson delivered an allegory to explain why host-based encryption appears redundant to him.

I just find it funny that all of a sudden after 60-odd years of computers there is a sudden need for encrypting data where it resides. It still begs the question of lack of access control. If the hypothetical HR department has its data on a host, and the hypothetical Shipping department has access to HR's data, what kind of access control is that?

I recall upon receipt of my set of rainbow books in the early 1980s and a discussion of the (then theoretical) "Class A1" trusted information system holding the highest levels of classified data:

"A blackboard with something written on it can be a Class A1 trusted information system.  All you need to do is put it in a locked room and have users sign in and out at the door where the armed guard is."

Taking away the armed guard and lowering the Trusted Criteria a bit, what I understand is being wanted here, is to require users to decode gibberish written on the blackboard after they have already been let in!

If you see my point, it is far more practical, (if not as efficient) to encrypt data as it is being transmitted, to and from a host and decrypted upon receipt. If a key is lost, you may always transmit again using a new key.

There is also additional risk if the data is encrypted on the host.  If you've lost the key, you've lost everything.

Encrypting data at the host does have its uses. On a PC where there is no access control and the hard drive can be compromised easily, such as at home or in airline baggage, host encryption makes sense and the user counts on it.  But that user also runs the same risk if he forgets the key.

I think the key here are differences between multiuser hosts and PCs. The line became blurred when they starting using PCs as multiuser servers and basic concepts of security became lost.

Eggers replied with another point, about the changes in computing — and how the new world demands a different standard, one that assumes the worst and demonstrates "due care."

Encrypting data is a tool.  Misapplying the tool falls under "due care," and not having proper and/or approved procedures in place to safely use it falls under "due diligence’.

The courts seem to be becoming the school of hard knocks for IT and executives alike as most consider security in a heavily networked society: Annoyingly time-consuming and complex; and/or are blissfully ignorant of consequences; and/or will cross that bridge when they come to it.

The world has changed a lot in 60 years, and at an increasing rate. Adaptability is essential to survive. The days of multiuser text interface database servers connected by simple serial lines to dumb terminals are gone.  Even when client/server systems begin to look like the old dinosaurs, the invisible underlying functionality and complexity is increasing dramatically. This hidden complexity needs to be secured.

With worldwide high-speed broadband interconnectivity increasing at a rapid rate, the bad guys are probing for holes in this ever-increasing complexity of our interconnected systems, making our systems harder and harder to secure.

Most of the government systems with highly-sensitive information or functions are disconnected from networks, or connected only to internal highly secured and controlled networks. There is no way to guarantee their safety if connected to public networks.  But business depends more and more on interconnectivity every year. Therefore, risk analysis is essential — even if it boils down to just purchasing insurance policy to cover a vulnerability.

But it's entirely likely that purchasing a commercial encryption solution could be less expensive, and preventative to boot. That's due diligence, if implemented with proper procedures.

Encryption: where and when and how

Regulations drive today's encryption needs, much of the time, for HP 3000 customers. Security and SOX, COBIT or PCI compliance go hand in hand. And even though HP has offered little in the way of MPE/iX encryption tools in recent years, the marketplace and clever administrators and developers know how to keep the mission-critical bits under wraps.

While OpenSSL routines offer encryption potential in the Apache WebWise free offering, these routines are now nearly seven years old. Even the latest OpenSSL versions date to 2005. Implementation is not for the faint of heart.

One commercial solution and some techniques emerged recently on the HP 3000 newsgroup and mailing list. A 3000 manager asked how to encrypt one field in a TurboIMAGE database. An easy-to-implement reply came from Orbit Software. Developer Mark Klein plugged Backup+/iX "that will do encryption only at backup time."

Sure enough, the 256-bit encryption key in the Orbit product stands as the strongest protection in the 3000 community. More bits in the key means a tougher challenge to crack it through a brute force method. Documents classified as Top Secret by the US Government require encryption keys of 192 or 256 bits. Orbit's software, available for MPE/iX, was driven by the needs of customers in the banking industry.

Klein, working as an independent developer for the Orbit labs, pointed out that HP 3000 databases are privileged, a step that offers reasonable security. But not crack-proof, unless good procedures to protect that privilege are in place. Without high-powered solutions, encrypting with open source software can have stiff penalties, he said.

Note that software encryption has large performance consequences, so you really need to be clear as to whether or not the data must be encrypted in the database or encrypted for other means. If your need for encryption is to secure your data transmissions, consider using a network link that itself is encrypted. These can be hardware-accelerated, which will mean the performance will be better.

Keeping up with changes and options became a lively discussion on the newsgroup.

SOX in particular forces 3000 managers to look at encryption options. Off site storage is an issue in this compliance with the Sarbanes-Oxley law, a ruling that applies to many hundreds of HP 3000 customers (publicly traded firms, large ones using security or bond-based funding, those which desire to look secure to customers.)

But things have moved on since the DES offerings of the late 1990s which HP engineered for the 3000. "DES has been replaced by AES," Klein said when a customer asked about the OpenSSL software HP offered. A common question might follow the one this customer posed while investigating how to get started with encryption:

We came across the option of installing PERL on the HP 3000, as it has some encryption functions/routines readily available; for example, 3-DES is present as a module. The question we have is whether we could write a program / executable in PERL and call it from a COBOL program on the HP 3000.

Another option we were thinking is of going the OpenSSL route, but there too, we have little knowledge on how to use the functions present in those libraries.

Klein replied,

OpenSSL has been ported to MPE by others and it may also provide you with a solution, if encryption of the data is the only way you can satisfy your need. Your real issue may be in calling these from COBOL.

In an even more interesting development, some 3000 developers and managers wonder if encryption on the 3000 host level is even necessary. We'll recap the strategy of guarded classrooms and blackboard gibberish tomorrow.

Spread out to max storage upgrades

Ron Keiper of the Nielsen Company asked how to maximize storage performance in his upgrade to HP’s XP series of disk arrays, linked to an HP 3000. This kind of upgrade is common in a site which is either homesteading or buying more time during a migration project.

We are in the process of migrating our HVD10 storage to an XP256 via FW-SCSI and IO Expansion (1828A) cabinets. Our performance testing thus far shows a large increase in CPU pause and disk queuing on the XP256 system. Job run time is similar, within 5-10 minutes on a one-hour job. I have seen many reports about the XP being a great deal faster, but we are not seeing that and have concerns about moving forward with our storage migration. I am wondering if it is set up properly for best performance.

The XP is set up Raid 1, I believe, (OPEN-9 in DSTAT) and are set up as HPDARRAY in SYSGEN. Is that the correct sysgen ID for the XP? There are a similar number of drives/LUNs on both of our test volume sets. The HVD is setup as Mirror/iX.

Craig Lalley of Echo Tech responds:

The XP256 is a decent box, but only half the speed of the XP512. The LDEV assignment within the XP is very important when it comes to performance.  Did you spread your LDEVs across ACPs and raidgroups?

You didn’t mention how much Array Cache you have in the XP256. Keep in mind the XP storage array is cache centric, meaning everything goes through cache. With the XP256, I would max out the array cache.

Lalley, who specializes in this kind of configuration upgrade, adds 

Remember, always spread your IO...  across XP ports, XP cards, XP array groups, XP ACPs, host HBAs, Host System Bus Adapters and switch ports.

Third parties: Fact of life for Windows support

HP's sold a lot of systems, and prodded many customers to migrate, on the benefit of its support. Even while a steady stream of 3000 sites say they get spotty HP support, it's apparently still miles ahead of Microsoft's help.

Our Google trawling software dug up a user report from Karl Palachuk, a consultant and author, which compared HP 3000 support back in the day with the current Microsoft disappointments. Palachuck supported 3000s in the early 1990s. On a blog which which covers his consulting practice, Palachuk says you'd better get a North American superstar at Microsoft on the support call, or look elsewhere:

Microsoft's support of Small Business Server is deplorable. Until you can escalate your call to one of the superstars in North America, you are just as likely to have them burn your server to the ground as to get a solution.

How does he solve his problem? The same way many 3000 sites have today: get a third party. Just like the 3000 community's economics, the third party's support is inexpensive. Like $37 a month per Small Business Server.

We mention this because Windows is the leading choice for 3000 sites making a migration. They often go to the Microsoft solution on price, plus the choice of a replacement application. These sites might not know a third party is often essential to uptime.

Zenith Infotech provides outsourced helpdesk and managed services for Palachuk's client base now. It's not a matter of dismay about overseas tech reps, either.

So when a partner works on a problem in-house for the allotted period of time, they don't escalate to Microsoft's incompetent tech support and hope to talk their way up to the competent technicians. They call Zenith. And the very competent technicians in India "Just solve the problem."

In your community, you have the same experience in many third party support companies. HP no longer takes a key spot in the problem resolution process. That's why the end of support date has been such a weak motivation for making a migration. The companies who need vendor-based support must move.

By all reports, many 3000 support suppliers also do Windows, along with the floors of the MPE/iX service. You should check to see how much of the total support package you can get. The price might differ from Zenith's. But then, they don't know anything about MPE/iX.

So choosing Windows as the new environment doesn't demand vendor-based support, for most companies migrating. Ever wonder why things are so different for this new platform than the old 3000? It's not as if the Windows Server is going to less mission critical than that 3000, after all. Palachuk's report illustrates another cost of migration: third-party support.

Ideas for version control

Whether your 3000 environment is in transition, or you're staying in the familiar fields to homestead, you will have versions. Any large  entity  like an application goes through changes. You'll want to track yours with a Version Control System (VCS).

Several are available today for the 3000 community, including one from HP you might still be able to purchase. (After all, software subsystem sales were supposed to go on through the end of HP's support.)

Walter J. Murray, formerly of HP's labs and now working in the IT group at the California Department of Corrections and Rehabilitation, says, "I heartily recommend HP SRC (Software Revision Controller). We have been using it extensively for about three years.  We are a COBOL and PowerHouse 4GL shop, but we use HP SRC for just about every kind of file except TurboIMAGE databases. Contact me if you'd like more information."

Non-HP VCS solutions include the open source CVS (Concurrent Version System) suite, which HP and Interex used as a method of HP's Shared Source project to put parts of MPE/iX source into the user community.

Pete Eggers added, "If your developers use PCs to edit and write COBOL, then set up a Linux server with either Subversion (sort of a next-generation CVS), or Git, if you are definitely more geek-oriented.  Both are being actively used and developed all over the world, and have more capabilities and features than you are ever likely to use. There are also a variety of add-ons to both, including GUI front-ends, including the old warhorse CVS."

Encompass goes Golden tomorrow in LUG meet

The Greater Houston Regional User Group (GHRUG) is the last RUG standing, but Encompass revs up an alternative tomorrow with its Golden State Local User Group. LUGs are the RUGs of Encompass, and this year's Golden State meeting happens in the home court of the former Interex Bay Area Regional User Group (BARUG). At least a couple of HP 3000 veterans will be at the April 25 meeting.

Duane Percox, one of the co-founders of K-12 education software supplier QSS, has told other 3000 gurus he plans to attend the meeting. Also on hand will be Steve Davidek, the newest board member of Encompass and a former advocacy chair for Interex, the HP user group that went bankrupt in 2005. (Not that the bust had anything to do with Davidek, who was nowhere near a fiscal management post for the group.)

The two-hour LUG meeting will take place on HP's campus starting at 1:30; for those of you in the Bay Area who like to Google Map your directions, and have not visited HP's Building 46 Lower office, the exact address is 19091 Pruneridge Ave, Cupertino, CA 95014. The Encompass LUG page promises

This meeting will focus on the discussion of Blade Servers on Integrity, presented by Markus Berber of HP. Snacks and beverages will be provided. For more information contact Mike Stewart at Mike@reve-enterprises.com.

Davidek, who leads the Encompass board efforts on Advocacy (the annual HP worldwide customer satisfaction survey that Interex used to manage, and other advocacy issues for HP enterprise customers), reports that his employer has one hit and one miss on its migration scorecard. But the City of Sparks, Nevada hasn't given up on getting away from all its 3000 applications.

Davidek, who also sits on the Encompass volunteer development committee, said the city's police have made a getaway from their 3000, but Finance will have to take another stab at escape this fall.

Our city Finance system move from the HP 3000 almost two years ago was a bust, as the software vendor picked by our Finance department was unable to work the way they expected it to — we had been live on the system for almost 11 months when the decision was made. Payroll was never able to migrate. Last June we "migrated" back to the 3000.

The city is in the process of finalizing our selection again with a planned migration for late fall of this year. Our Police department successfully moved from their HP 3000 in January of this year.

The bumps in the road away from the 3000 at Davidek's shop follow a common path. A replacement application gets selected, based on referrals and IT's close comparison and examination of features. Perhaps some test runs take place, but sometimes the live 3000 data on information as crucial as payroll is just not available. Or the app just doesn't perform as well as its data sheets did. When the application gets the 3000 data, the roadblocks rise or spring up, depending on how serious and how sudden they appear.

In a worst case, a 3000 that was scheduled for a scrap heap trip gets a reprieve. This case is why the lifespan of the HP 3000 remains hard to define  — for the customer, as well as for HP's support business. 2008, HP reminds us, may not mean the end of life for the 3000 at HP.

Arguments abound on which getaway path is easier: picking commercial off the shelf software packages to replace aging apps, or moving existing code in a migration. The truth is that it depends — on the replacement package chosen and its ability to be customized, versus the money and manpower a customer needs to move existing code in its apps to another environment.

What has changed in two years

All through the course of the HP 3000 Transition, we've been asked for advice on what to do. HP's extended its timeline for its 3000 presence. The market for third party services has grown up immensely. Meanwhile, the critical mass of HP 3000 users has not fallen off as sharply as HP and others predicted.

Those developments might change your decision, or your proposed schedule, about migration. We got a question from one user, reporting on what's gone on at Raytheon in the UK — and asking us what we think might happen.

We had some communications some time ago on HP 3000 MPE/iX platforms. I was trying to make a strong case for our company migrating, despite the uncertainty of what HP were going to do, and be in charge of our own destiny.

Despite this, the project was parked, and obviously HP made their decision to extend to December, 2008. However, this date is fast approaching, and I was wondering what your latest position was as regards this.

Hope you can provide me with an update,

Alex Purves

HP's changes to its timeline for leaving this market reflect the reality of the migration pace. It seems clear to me: the vendor wants to be of service to its 3000 customers as long as possible. The level of service offered has dropped. The cost has not dropped. What's more interesting is that the end of HP support doesn't seem to be as much of a motivator for migrations. Not for the customers, and probably not to HP anymore.

Make no mistake: HP will exit the market, at the end of 2008. Or perhaps later. The question to be answered is, "How much will this exit matter?"

In practical terms, HP's absence from the 3000 market shuts off development of 3000 software. That's MPE/iX, and IMAGE, a couple of keys to successful HP 3000 mission-critical use. No more HP patches in 2009 (or maybe 2010, if HP follows its "or later" clause in its December 2005 communique.)

HP support won't be available for purchase at that time, either. This is not the same kind of thing as losing the efforts of an MPE lab. Today, support from third parties is the predominant channel for keeping 3000s and the MPE/iX operating system up and running. Companies who use resources other than HP report consistent success in the support market.

In our view, migration and its driver doesn't really involve HP, for most customers. There are many sites who simply cannot allow a mission-critical application to run on a system the vendor no longer supports. These are the current HP Support customers. See the statement above; they are in the minority, by our estimates.

No, the migrating shops — and there are many — proceed because their applications need something the 3000 community cannot provide anymore. It may be application enhancements from a vendor. Or perhaps a technology that HP supports elsewhere, but never fleshed out or ever offered for the 3000. (Java comes to mind here right away. There are others.)

All along HP has told the customer that HP won't know what is the right time to migrate. (Later on, HP would add, "If at all," but that advice seemed to be aimed at smaller customers where HP sees the environment as static, with few changes.) The customer's business plan would dictate that, HP has said.

Our position, as Alex requested, is to look at the resources your company can assemble to migrate, as well as the upside of the migration. If it is simply to control your own fate, as Alex mentions, there are many places to do this. But the best may still be the platform that continues to run for you, provided you can assign support to a third party. Second on our list, if a free destiny is your main goal, is embracing open source solutions using Linux, MySQL for the database and a bounty of tools. No vendor will ever cut off those resources.

Finally, the migration decision needs to be taken with some external advice in hand, in my opinion. Kind of like a second opinion, really, if your company can agree on a first one. At Raytheon the management decided to "park" the migration plan a few years ago. That's a first opinion. Inviting a second from a consultant, migration services supplier, or even another application or systems provider, is a good start at that second opinion. (Although I'd bet that final group of vendors and app providers recommends a migration. And they could well be right.)

HP 3000 customers were always a more independent group than the IBM batch users. Now the prospect of independence appears at a time when Windows bolstering comes not from Microsoft, but third parties, with two to four letters in their names: HP, IBM, Dell. Third parties are a way of life in 2007 and beyond. Unless the embrace of HP's Unix is a key to your plan, your company will survive through the efforts of any party beyond HP. Coming to terms with that lifeline is the first step to freedom.

Leave trip wires to track modules

In his spare time, Mike Anderson of the Greater Houston 3000 user group helps a few local companies get ready for migration. Yesterday he shared a little trick with us. Leave bits of code, routines that act as trip wires, inside your forest of application modules. Go ahead, do it today, no matter how soon you will pull the trigger on your actual migration.

First, this counts as migration work. Creating code to identify when a module gets used, and report back to Routine Counting Central, can be the prelude to a savvy migration. Because like our At-Large Editor Birket Foster has said so often in the many talks he's given, you want to know how much of your code you will need to migrate.

Second, this kind of development can be a positive thing to report to a company boardroom. "Our migration project has begun with detailed research on the size of the task. Specialized code is being placed in our company applications. This software will deliver metrics we need to create budgets and schedules."

That's a report which may be good enough to buy some time to do the work correctly and efficiently. It's also true, too — this code must be specialized, because you will need knowledge and access to your source code to leave this trail of trip wires.

This tactical work won't help you nearly as much if your strategy is to replace your apps with what Foster likes to call Commercial On The Shelf (COTS) software. Yes, the use of these checking routines might help your search process for the best replacement package. But you're more likely to be ticking off company practices and business rules while you shop to replace.

It might be difficult to job out this "routine" task to an outsource supplier. A developer who's unfamiliar with your company's code could bill for quite a few hours figuring out the best spot to leave a crumb of this counting code.

HP has a tool to help inventory HP 3000 systems, the SIU, offered for free at the HP Jazz Web site. But the SIU won't report how often your data sits in each of the many trees in your forest of modules and applications. The SIU only counts and identifies files on your 3000.

This kind of bread crumb routine would be most useful if it could garner information over time. You'd think that six month-end closings would be enough to check if a module is being used. On the other hand, you might want to make sure one of those months is the end of a calendar year.

But every month in which your code does its checking might save many hours, or dollars, during the migration itself. Several migration suppliers charge by the line of code. Modules that you need not change would incur no charges, we figure.

PowerHouse customers eye new alliance

Migration business picked up during 2006 for several segments of the 3000 community. One group of customers who made plans and began projects: PowerHouse sites, especially those who want to stick with the Cognos language while they make the shift to other platforms.

Speedware's Chris Koppe, director of marketing a frequent presenter at 3000 events, has told customers one of the fastest migrations is a "stick-to-your-fourth generation language." Speedware-to-Speedware success stories quote fast turnarounds. Recently Speedware's migration unit started to migrate Cognos customers with the same strategy.

This Speedware-Cognos alliance announcement prompted concerns from a portion of the PowerHouse customer community. On the PowerHouse mailing list, a few customers said they feared the deal will accelerate Cognos’ detachment from PowerHouse futures.

Robert Edis, an independent PowerHouse consultant, took note of Cognos’ Automated Development Tools (ADT) income dropping out of Cognos’ shareholder reports, along with eliminated or reduced mentions of these ADT tools in Cognos conference presentations, sales force pitches and product lists. He sketched a forecast of PowerHouse futures.

Cognos took the concerns in stride and pledged loyalty — point by point — to a product still earning revenues for the company.

“Business is business,” PowerHouse product manager Bob Deskin said to customers on the mailing list. “Now that [Speedware] are primarily in the migration business, it makes sense since we have many common customers. It's a good partnership."

Edis speculated the Speedware alliance would mean less Cognos effort toward the Cognos Automated Development Tool (ADT) products.

Edis noted the following trends:

1. ADT income no longer included in shareholder reports
2. ADT tools disappear from product list on Cognos Web site
3. ADT tools no longer mentioned by sales people
4. ADT tools no longer included in conference presentations
5. Dramatic slowdown in development/enhancements/new features
6. Deal made with Speedware —once only mentioned by Cognos in disparaging terms — to support ADT tools. Speedware advertising in past put down PowerHouse and encouraged Cognos customers to port PowerHouse apps to "modern" tools (i.e. Java).

My prediction:

7. Development/enhancement work on PowerHouse stopped completely by Cognos
8. Cognos and Speedware announce deal where Cognos sells PowerHouse to Speedware.
9. Speedware takes over support (taking a few of the youngest and most promising ADT developers from Cognos to boot) and license income.
10. Speedware induces PH customers to "modenise" their PowerHouse apps by offering special deals and perhaps increasing annual license fees on PowerHouse.
11. PowerHouse RIP

Edis wondered if his forecast was pessimistic. “Maybe the Cognos-Speedware deal is just a case of ‘if you can't beat them, join them.’ “

Not RIP, but a reply

PowerHouse product manager Deskin noted in reply to Edis concerns thatPowerHouse is healthy product still earning for Cognos:

#1 - True, but then we don't break out any specific product unless we want to highlight major sales (like a new product)
#2 - We have our own Web site @ powerhouse.cognos.com
#3 - We have our own sales people
#4 - A business decision based on attendance
#5 - Not true as can be seen by what's in E. See some of my Supportlink articles.
#6 - Business is business. Now that they are primarily in the migration business, it makes sense since we have many common customers. It's a good partnership.
#7 - Eventually everything comes to an end. But we have a while to go yet.
#8 - #10 - Are you trying to start rumours?
#11 - Not for a long time yet. Too many customers and too much income regardless of who runs the show.

Ticking time zone clocks

Steve Cooper, one of the founders of Resource 3000 partner Allegro Consultants, updated the 3000 community on important details about the upcoming time zone changes. These are the adjustments managers must make to keep up with the new start and end of Daylight Saving Time.

Just when you thought you had the latest Time Zone Table file on your 3000, the world has gone and changed the rules again.

A new patch has just been released on HP-UX, with the following updates:

PHCO_35991:
    ( SR:8606475843 CR:JAGag30156 )
    tztab(4) needs to be modified for Mexico Daylight
    Saving (DST) changes.

    ( SR:8606475841 CR:JAGag30154 )
    tztab(4)needs to be modified to support Western Australia
    DST changes.

    ( SR:8606475842 CR:JAGag30155 )
    tztab(4) needs to be modified for Canada DST changes.

    PHCO_34673:
    1. Indiana will support Daylight Savings Time (DST) from
    April, 2006 onwards.

So, if your system or any of its users care about Mexico, Western Australia, Canada or Indiana, then you may need a newer TZTAB than you previously thought.  And, if you care about the rest of the United States and haven’t updated your TZTAB file yet, you might as well get the latest version when you do the update.  The clock is ticking...

What a certification is worth

A computer with a dodgy future shouldn't have this much customer interest in certification for its skills. But the HP 3000 is unlike a lot of systems abandoned by their makers. People still want to prove their 3000 skills are up-to-date. The difference? The HP community, still providing training and cert service long after HP's lost interest. When people will pay for this training isn't as interesting as the fact that it's offered at all.

Inside the folders passed out at this month's GHRUG HP 3000 conference (there were no bags, another cost-saving measure other conferences ought to observe), I saw a color flyer from Jon Diercks, who said that he was "sorry I couldn't be there to participate, but wanted to send my regards." Under Diercks' smiling face was an e-mail and Web link to garner contacts from your community. Like a lot of 3000 pros from the modern era, Diercks is able to take work from either a homesteading or migrating customer.

But he's important to the certification process of the 3000, the way professionals will try to demonstrate they're able enough to administer, program for or size up a 3000 for either homesteading or migration. Diercks wrote The HP MPE/iX System Administration Handbook a few years ago. You literally could use the book to study for a certification test. As a 3000 resource, the book remains the only title close to up to date with the operating system.

Why in the world would you want to pass a certification exam for a computer being abandoned by its vendor? The question sparked a lively debate not so long ago on the 3000 newsgroup. Good points were being made by both sides, even though a peristent gravedigger for the system said long ago all is lost for the 3000 customer.

"If I had to pick the biggest story of 2005 it would be HP’s killing OpenMPE, the  HP 3000 and its support “ecosystem” of vendors that were looking to take over  support at the end of this year," said an advocate who's now embracing Windows. "By extending HP’s support of the HP 3000 for two  more years, they’ve not only burnt the HP 3000 and its community to the ground,  they’ve eaten the ashes as well."

"If you didn’t think that the HP3000 was dead before, you’ve got to believe it now."

Like a lot of predictions delivered in advance of the data, this doesn't sound like the 2006 that followed those comments. Third parties like Paul Edwards and Associates, Pivital Solutions and independents like Diercks still want to help you homestead, even it's only for awhile until you move away. Dead? Depends on your definition of alive. How's the health of OpenVMS, currently trailing in all of the HP virtualization offerings behind Windows, Linux and HP-UX.

Then there's Jim Chance's "how can anybody see any value in this, without exception" point of view:

I a 22-year IT guy with 16 years HP 3000 and personally I don’t know why anyone would want to pursue or care about this particular cert. Just my opinion. As much as I hate it, the HP 3000 market, shops, places with it fully running in production, conferences/workshops, new sales, new ERP on that platform, and technical knowledge is DEAD or quickly DYING. When will folks realize this?

Turnover to other platforms and skills is at an all time high. Universities and corporations are de-commissioning there boxes at an astounding rate. How do I know this, by being a contractor since 1997. I could go on and on about this trend.

I asked......who was it, I forget, but some vendor who puts on training to become MPE cert isn’t getting any paying students, none. So why does the board or anyone else lobby for something that frankly  — and realistically, just plain out isn’t in demand? I appreciate efforts, but really? I can think of a dozen former and/or dwindling HP3k guys who are now pursuing MS cert’s; there is no way they’d want to spend $ or time on certifying as MPE.

Donna Garverick-Hofmeister, an OpenMPE board director, made a case :

There is value in MPE certification.  For migrations are taking longer than expected, for companies who plan to stay on MPE long term -- they’re going to need people who know MPE  They’re going to need people who know how to straighten out problems left behind or situations that have cropped up due to infrastructure changes.

MB Foster founder Birket Foster calls this the "flight attendants flying the plane" situation, commonplace in 3000 shops. The trained folks have been downsized or moved on of their own volition. Who ya gonna call? Maybe someone certified.

Tracy Pierce wants to know what any certificate is really worth, MPE or Unix or whatever:

What’s the certificate’s real value? 

As to an assurance of qualification, I bet someone holding said certificate would probably get consideration equal to that of a non-holder if they claimed competence in all the requisites mentioned. 

While the certificate is ostensibly an attempt to certify competence, its existence can be quite misleading.  I bet (I’d say I know but then you’d want examples) that there are people who hold the certificate who don’t know half as much about the machine, much less the project at hand, as do some other people who’ve never set foot inside an HP training class much less taken an HP certification test.

The certificate indicates that its holder passed a test, not that they can program their way out of a paper bag or can apply common sense to a business problem, much less unravel code written by possibly competent programmers who didn’t have time for documentation.  Does the certificate imply knowledge of every gotcha to be found in VPlus?

Said tests include questions about a lot of pretty arcane stuff, just the sort of knowledge that leaves a person pretty quickly if not exercised.  How long has it been since the certificate became unavailable?  Are all those certified actively working with the platform and using all the skills for which they’re certified?  Even if so, does that make them More competent than a non-holder?  Not in my book.

If I had one, I’d probably mention “HP Certified” on my resume.  But I don’t think I’d really want to work on the project if the employer’s more impressed by a certificate than by a demonstration of the real skills actually needed.

On the other hand, a certificate will make the bearer’s suit and shoes look much shinier to an HP 3000-ignorant entity offering a conversion contract; it’s also a probable indicator that the holder knows enough to hire competent staff for a project involving the HP 3000.

Hofmeister said that a more up-to-date test would prove more — a project Edwards and his colleague Frank Alden Smith are ready to take up.

Many of us have heard the complaints about the existing MPE test.  Personally, I'd love for the test to be revamped.  Rewrite it for 7.5, make it good evaluation of a person’s MPE knowledge, make it so the test means something.  If wishes were horses, I’d like for there to be a programmer’s test in addition to an administrators.

Having said that — nothing is going to replace an interview for hiring.  Piece of paper or not, the only way to know if someone is going to fit into your organization is to talk to them.

But getting on as a 3000 expert, after you've stepped away from the system for awhile, is going to take some retraining.

It’s hard to tell what is worthless and what is not.  I was a system operator for an HP 3000 Series 70 with MPE V back in 1990 when major restructuring meant job losses to my entire department and beyond.  It’s rather eerie going back to that building because the office/warehouse is gone and now is subdivided between several different companies. 

I came here because I once again am seeking employment and am wondering — is my past experience obsolete?  Can I build on my knowledge or would today’s MPE/iX 7.5 systems ( be too far removed, and I’d have to start over?

How could it hurt to have a copy of Diercks' book? At the moment, Amazon has an $11 used copy.

Keeping watch on 3000 hosts

HP 3000 servers may not need the babysitting that other environments demand. But when a server is mission-critical, as so many 3000s are, monitoring that resource's availability makes good IT sense.

New solutions for this kind of watching are rare on the 3000 market these days, but the existing product selection works just fine for HP 3000s — as well as HP-UX servers, Linux boxes, and even Digital VAX systems.

The software comes from ASP Technologies, based in Windsor, Colo. and it's been available for years now. Vantage is a console management package designed differently than many network watchers. This solution does not require installation of agent software on any managed resource like a 3000 or a 9000. ASP says this design "allows control over a much wider variety of systems and devices."

As a result of the Vantage design, no processing overhead is imposed on resources under Vantage control. All event detection and automation is performed 'outboard' on the management workstation.

Donna Hofmeister testified to the advantages of Vantage in a recent report to the user community over the 3000 newsgroup. In typical lower-case Donna-speak, she posted

vantage will watch your console traffic and will react according to what you've instructed it to do. the company owner (allen) knows mpe and is great to work with. the software itself is well written and well behaved. i think it's reasonably priced as well.

Veteran 3000 manager Greg Stigers added than many an SNMP-driven solution could fill the bill of requirements requested by Wesley Setree, who needs "a tool that will monitor HP 3000 and OpenVMS for certain conditions and send an alert via e-mail or pager and/or possibly a console or command center ... I would not only want to monitor down conditions but other items like disc space thresholds and job aborts." Stigers noted

While I’ve read that SNMP is becoming less important for system health and notification, it’s better than nothing. There are a number of tools that will query systems and handle SNMP responses. My last 3000 shop used What’s Up, although it never did much work to have it talk to the 3000, unfortunately.

I like to ask, who watches the watchers? Can you imagine a reasonable scenario, such as losing power or network connectivity, that would mean that the monitoring either was down or could not report the problem? Sure, it’s unlikely, but it’s not impossible.

At the other end of the problem, I will admit to writing a job that was a wrapper for a set of command files that checked the system for certain states, and reported their exceptions. Things like background jobs not running, or the number of jobs waiting increasing over three samples. That’s fine for OS & application states, local to the system.

Tony Summers adds that for the 3000 site with DIY skills, tracking solutions may be more flexible and engage more reporting devices:

For job tracking, we have written our own COBOL program which is invoked by the magic of system logon UDCs whenever a job is streamed, logs on, reports an error or logs off.  By carefully managing the status during each of these events, we are able to report to the operators any jobs that have failed or didn’t reach their normal !EOJ job card.

A separate monitor program (another COBOL program) simply keeps watch on the job tracking files sending out alerts via various means — to the console, to e-mail and by SMS to the operator’s (and my) mobile phone (cell-phone).

We use a product called Scrambler from UK-based Gainsborough Software to forward the SMS messages from a Windows server to our mobiles. A simple FTP script transfers the SMS files created on the HP 3000 to the Windows server running the scrambler software.

As for monitoring disk usage — we currently have a job (run once per business day) that essentially inspects the results of a DISCFREE and send a similar email/SMS when the disc falls below a certain percentage.

I think the Scrambler product has built in functions to monitor the HP 3000 on your behalf, but our operators have never bothered to implement them.

Our monitor program also checks for other unusual conditions — for example, reporting when the global job limit has been set to zero.

I’m not suggesting you write the whole solution yourself, but given the limited market for the HP 3000 you might find you need to consider a bit of Do It Yourself.

The other HP charged with crime

Today the California Attorney General announced he will seek indictments for former HP chairman Patricia Dunn, and HP's ethics attorney Kevin Hunsaker, based on their conduct in the hoax to quell leaks from the HP board.

This is the first time an HP board member — well, resigned board member — has faced indictment, let alone a chairman. Dunn wanted to keep her seat on this board before all allegations surfaced. Now at least HP is spared the embarrassment of having someone active in its executive level under indictment.

But the general news media are reporting headlines in shorthand. "HP executives face indictment," said NPR this afternoon, overlooking the fact that neither Hunsaker or Dunn are associated with HP anymore. That's a level of distinction some customers are applying to the whole pretexting scam out of the boardroom. That's the other HP, some say, not the one associated with the HP 3000. Other customers make no distinction, according to comments offered in our spot poll.

To be clear, the five indictments expected today are aimed at people not linked to HP anymore. But the three investigators and HP's folks each had a recent hand in the company's conduct. The New York Times reported that "The five will be indicted on four felony charges: using false or fraudulent pretenses to obtain confidential information from a public utility; unauthorized access to computer data; identity theft; and conspiracy to commit each of those crimes."

HP 3000 community members are of several minds. One OpenMPE board member said today that the indictments were "the other shoe dropping" in the scandal. A reseller and supplier of 3000 services, John Lee, said HP's conduct is

Just another example of the end justifying the means.  It has become OK to cheat in America if you get rich by doing so.  "Sometimes it's easier to apologize than to ask permission"  My question to you and the rest of the business community is this... where did these corporate employees learn to cheat?  Do they teach this in business school?  Most of them (I'm guessing) have advanced degrees from American universities.  Is this how they teach their students to get ahead?

Some are less incensed. Tracy Pierce replied to our use of the word hoax by commenting

"Hoax?  are you suggesting this is just a publicity stunt to get HP onto the front page for free?  You must be right about there being two HPs though: the folks at 633-3600
always want to know if the toner's fresh in my HP 9000."

The notion of two HPs was addressed most eloquently by Alfredo Rego, CEO of database experts Adager. Rego doesn't confuse the misguided ethics of HP board members and executives which have resulted in indictments with the dedication of the 3000-related staff at the company:

HP’s privacy blunder does not affect my relationship with Hewlett-Packard (please notice that I explicitly use “Hewlett-Packard” and not “HP”).

Through the decades, I have gotten used to the sad fact that HP’s “upper” level managers (please notice that I don’t use “leaders”) seem to exhibit a consistent lack of understanding regarding the technical treasures that Hewlett-Packard has developed.  Fortunately, I have always elected to associate myself with the technical “worker bees” at Hewlett-Packard.

HP’s “upper” level blunders do not affect my friendships with the many wonderful engineers with whom I have always enjoyed discussing the deep bits and bytes of MPE’s file system and TurboIMAGE databases.

In the early 1970s, I had the pleasure of meeting Bill Hewlett and Dave Packard in the assembly line for the HP 2100 computer in Cupertino (yes, there was an assembly line on Wolfe Road).  We ta