November 26, 2014
Something to give thanks for, and envy, too
On the eve of a holiday invented to promote thanks as well as outsized eating, Thanksgiving reminds us of what a 3000 user can thank the gods for -- and something to envy, too.
Prolific commenter Tim O'Neill asked, "Can you write about the current futures of other no-longer-supported systems such as HP 1000, Alpha, and old HP 9000s such as Series 300/400/700?" We can write that the HP 1000, a product line which HP turned off just after Y2K, still has third parties who will maintain and support RTE operating system applications. The HP 1000 got a proper emulator from Strobe Data, engineered in time to capture the business of companies who couldn't part with RTE apps.
A similar story is true of the AlphaServer line from HP. Killed off in the last decade, Alpha is a third-party supported product. No other Alpha computers were built after HP shunted its users to the Integrity line, a migration path of dubious future by now. Alpha has a good emulator in the AXP version of Charon from Stromasys, the company providing a future for long-serving MPE/iX apps, too. The presence of Charon prompts thanks from companies who can't support the concept of decade-old HP hardware running MPE/iX.
But while the Alpha and the 3000 will live on in the virtualization of Stromasys, they can be envious of the deal another retiring environment received this year. OpenVMS will live on in an exclusive license to VMS Software Inc. (VSI). The company got the arrangement to carry OpenVMS forward with new versions using the HP source code for the operating system.
The details released haven't yielded much more than a third-party road map for the OS, up to now. But that's a future with some tantalizing what-if's, both for the OS and for the 3000 user who wanted more MPE/iX future back in 2002. OpenMPE campaigned for use of HP's source code for MPE and got an arrangement that was announced six years ago this week. That source was limited to a technical support resource, however.
If, as happened with OpenVMS, that source had been promised to a single third party, six years before HP would drop support, there could be more to be thankful for this week. Extended third party applications. Support for newer technologies. A replacement vendor, blessed by HP, to mention in boardroom meetings about the 3000's future.
Perhaps OpenVMS customers should be thankful for something else, too: The lessons HP faced about ending the life of a business operating environment, an OS that brought HP to the computing game. Third parties that love and care for a legacy computer were on hand for the 3000. They fell short of convincing Hewlett-Packard to turn over a marketplace. Maybe HP learned that leaving customers with no better choice than replacing a system with Windows wasn't great business.
We'll give thanks for a few days off to celebrate this holiday with family in the Great Lakes -- regardless of frigid weather. We'll be back on Monday.
November 25, 2014
Open source SED manages 3000 streams
Open source resources make it possible to use SED, a stream editor built in the open source community. Since 2001 SED has worked on the HP 3000, thanks to Lars Appel, a former HP support engineer who ported Samba to the platform in the 1990s.
SED's main MPE page is on a page of Appel's. SED is an at your own risk download, but support is available through the 3000 community.
Dan Barnes, working on a problem he had to solve in his 3000 environment, asked:
The issue is incoming data from another platform that is being fed into MM 3000. This data occasionally has some unprintable characters, which of course wrecks havoc on the MM application when it is encountered. To address this, the user, using a cygwin (Unix-like) environment on their Windows PC, developed a SED script. When they test the script in the cgywin environment it works just fine. But when done on the target HP 3000 it gets an undesirable result.
Barnes added that "The user thought that because MPE/iX is Posix-compliant, that this should work." He explained his user created the expression
sed -e 's/[\x7F-\xFE]/*/g' < COMSHD > COMSHD1
But Appel noted that hex 7F thru hex FE portion of the expression isn't supported on the MPE/iX version of SED. It's a limitation of MPE/iX, but there's a workaround.
Not sure if the regular expression usage here matches Posix or GNU specs, but my guess is the "\xNN" format, that seems to indicate a char by hex code, doesn't work.
How about something like using the command sed -e 's/[^ -~]/*/g' instead, i.e. map the characters outside the range space through tilde?
November 24, 2014
Building a Portfolio Away From Retirement
Using analogies of moving out of a house and filling out scorecards, an hour of last week unreeled off a Webinar that showed how portfolios offer a plan for migrating and sustaining applications. Birket Foster and his crew at MB Foster showed how continuous and thorough software management helps available budget meet the most crucial needs.
A classic four-quadrant chart outlined the scoring of applications. One axis showed a business fit, the other a technical fit. Like all of the four-box charts, nobody wanted software in the bottom left, low in both aspects. But it's a business decision that drives most of the changes in IT these days. Scorecard the business fit of applications in a portfolio first, Foster said. If it scores well in that fit, go on to the technical fit.
The portfolio is the tool of governance, he added. Governing is a classic process to ensure the most needy get resources as required. Application Portfolio Management has been a favorite topic at MB Foster. It's only possible if a company knows its applications very well — and very well means with documentation that can be shared over time. The assets in a portfolio can be judged to be worthy of migration based on their risk-benefit-value. What helps a company most, and what could you least afford to let fall into that dreaded lower-left box?
Only about 5 percent of the community's applications can fall off that chart completely, ready for retirement. The largest group are suited for a same-capability migration, when they creep down. That 70 percent of the apps can get a lift-and-shift of their functionality, usually through replacement. It takes hundreds of hours per application.
What makes it less painful and swifter is cleaning in advance. As Foster said, "It's like moving out of a house. If you go through your closets regularly, you'll be moving less that you don't need." In this analogy, the closets are your data, which "has to be made available to the new app. It's not automatic."One tool that helps to automate moving out is UDA Central, software that until recently wasn't sold as a standalone utility. The program MB Foster created came onto the job as part of a services engagement. But after winning an innovation award from the Canadian government, UDA Central is being productized. It will be offered to systems integrators outside of the MB Foster labs.
But the advice shared last week was more than a pointer to useful software. When deciding whether to re-host (lifting code to another computer) or replace, the full range of software assets gets inventoried. The real answer about what needs to be moved, and in what priority, comes from asking about the whole portfolio. While that study's going on, there are those closets to be cleaned. Few people do such cleaning, VEsoft's Vladimir Volokh says.
"They see a list of 100,000 files and do not want to scrap any of them," he said. "So they move everything to the new system." A tool like MPEX can assist by looking at the data that drives companies. That's work that can take place even before a transition is underway. There's no such thing as Data Portfolio Management, but governance of data is one way to practice for the informed choices of application governance.
When to set up an APM?
- When there's no accurate accounting of apps used by a business
- When there's a lack of understanding of how apps support business processes
- When assets degrade through declining business fit, agility and maintainability
Deciding where to put new money for IT to maximize returns, mergers and acquisitions, and synchonizing IT and business priorities all lead to APM, too.
November 20, 2014
TBT: When Joy of Tech Was Necessary
The cover above of the SuperGroup Association magazine from January, 1985 came to mind here on ThrowBack Thursday. Fred White passed away this week, and it's been a delightful trek down the lane of memories to recall his gusto about the art of technology.
The cover above shows some of that gusto which is not easy to describe. SuperGroup understood the MPE and IMAGE technology of the '80s as well or better than any magazine of the day. But that 3000 publication edited by D. David Brown had a sense of humor and whimsy about it no other publication has been able to eclipse. (Even on my best day as HP Chronicle editor I was only cooking up editorial cartoons about PA-RISC that somebody else would illustrate, and there have been those Ken-Do strips from the NewsWire. But nothing as savvy as what was staged above.)
The players in the little romp were, from left, White, Adager's Alfredo Rego, and Robelle's Bob Green. The photo was a teaser into a great technical paper about a perceived need to acknowledge that databases needed "uncomfortable Procrustean designs... [using] methodologies associated wth normalizing and relating."
Like the paper that Eugene Volokh wrote in the following year, the technical report put relational databases in their place -- capable of permitting multiple views of data, but with a steep performance price to pay compared to IMAGE/3000. The article was on the vanguard of unmasking the shortcomings of relational databases of that era, as I read it. Also clever and playful, two words not often associated with technical writing. The paper was authored by more than the three in the picture; Allegro's Stan Sieler and Steve Cooper got credits, as did Leslie Keffer de Rego for editing.Two of the actors in that photo represented a database that had to be filled out for length, and one that needed to be chopped short. Procrustes would kill travelers by placing them in "beds of various sizes, and when he lighted on a traveler who was tall, he consigned him to one of his short beds, lopping off so much of him as exceeded the length of the stead; but if his guest were short, a long bed was provided him, and his limbs, by help of a machine, were stretched out to its length."
This kind of super-wizard comedy was essential to the period when White was spreading his wings. He was a consultant to Adager at the time and sometimes graced the speaker lists on that day's then-crowded user group meeting calendar. At one show in Southern California, held in the halls of the converted Queen Mary, I watched White expound on the exactitude of writing files to tape, an amazing talk that ran more than a quarter-hour over its 90 minutes allotted. White had more to say, too, even as the organizers had to turn over the room.
The 1980s of the HP 3000 were a time when the Joy of Tech was necessary to overcome the growing pains of the 3000's success. Users were outstripping the processing power of the CISC-based systems, and the competing databases of the era needed serious integration skills to maintain their value to their owners. That integration had been wired into the 3000 by the IMAGE work of White and others. Experts like him, Rego, and Green not only wielded the know-how, they made complex topics entertaining. In SuperGroup they found a wry editorial staff which knew how to showcase gusto.
November 19, 2014
Fred White, 1924-2014
Courtesy of his long-time collaborator and partner Alfredo Rego, this picture of Fred White was taken in 2004, when Fred was 80 and several years into retirement. The legendary co-creator of IMAGE and the SPL expert in Adager's Labs, White was a Marine Corps veteran. Rego said while offering this portrait, "I took this photo with my Olympus E-1 on October 26, 2004 (just a bit over 10 years ago!) in Cedar City, Utah, where he and Judy lived for a while. Fred invited Judy and me to lunch, and I snapped this image across the table. I loved everything there: The warm light, the delicious food, the stimulating conversation, the young college students rushing about..."
The creator of the heartbeat of the HP 3000, Fred White, passed away on November 18, 2014 at the age of 90. White died peacefully in the presence of his wife Judy and family members, of natural causes. He had relocated to Arizona after retiring from Adager in the year after Y2K. His work in building the essential database for MPE, alongside Jon Bale, was the keystone of the 3000 experience. Rego took note of a key identifier inside the IMAGE internals, one that signified a database was sound and accurate. The flag was FW, or as Rego said in a short tribute to his partner, "%043127, the octal representation of “FW” — the flag for a normal IMAGE/3000 database (and TurboIMAGE, and IMAGE/SQL)."
White's work for the 3000 community came in two stages. The first was his innovations while working for HP, building a network database which won awards until HP stopped selling IMAGE and included it with the HP 3000. (Bundled software would not be considered for prizes like the Datamation award bestowed on IMAGE in 1976.) IMAGE, integrated at a foolproof level with the MPE intrinsics and filesystem, delivered a ready field for a small army of developers to plant applications and tools. Without White's work, the 3000 would have been just a footnote in HP's attempts to enter the computer business.
The second stage of White's gifts to the community began when HP had infuriated him for the last time. Never a fan of large organizations, he left Hewlett-Packard when it became clear the vendor had no interest in enhancing IMAGE. But before he departed HP, White met with Rego when the latter was visiting HP in an effort to learn more about IMAGE from the vendor, in preparation for a forthcoming database manager he'd create. As the legend is told, White decided he'd try to help Rego just to ensure that the creation to be called Adager could emerge a little easier.
"He hoped we would answer his questions," White said in a post-retirement interview. His partner Jon Bale "said that kind of help would be contrary to HP company policy. I said to him, 'Jon, this guy’s going to get this done whether we help him or not. All we’re doing is helping a fellow human. Whatever it takes, Alfredo’s going to do it anyway.' "
"At that point, Jon said it was up to me, but he couldn’t do it because it wasn’t HP company policy. He wished Alfredo the best of luck and left. So I answered his questions, and even told him things he couldn’t possibly have thought of, such as privileged mode intrinsic calling and negative DBOPEN modes, things peculiar to the software rather than the database. We chatted for an hour and a half or so."
The exchange in 1977 pointed toward the door to the Adager segment of White's career. The years between 1980 and 2001 allowed Fred to make up for his reticence inside corporations by becoming the conscience of accuracy and fairness. Innovations for IMAGE finally arrived in the middle 1990s. But White's most saucy moment of advocacy came in Boston when HP was trying to make IMAGE a separate product once again.The battle raged in a conference hall on the scene of the Interex user group meeting in 1990. Unbundling was an HP strategy designed to make it easier to buy an Oracle database for the 3000, reducing the price of the hardware modestly while making room for an add-on product. HP's database would be on a footing with all other offerings, but White and others knew that a 3000 without IMAGE was not the product the community trusted with its loyalties.
It was an era when users offered advocacy in a tone of angst. This sometimes was not the exchange that HP desired to air in public. But it was good for the capability of the system. HP had to watch the international computer press listen to a rumbling roar of revolution from 3000 users. A meeting of the IMAGE Special Interest Group came to be known as your community's Boston Tea Party. Rego recalled the moment of highest revolt.
Fred White (co-author of IMAGE and at the time Senior Scientist at Adager Labs) addressed Bill Murphy (HP’s Director of Marketing) from the floor and complimented Bill on his tie. Fred then explained how stupid it was for HP to unbundle IMAGE. Fred continued by describing the negative effects in products that depended on having IMAGE on the HP 3000. Fred also provided some historic background by relating how Ed McCracken (a previous 3000 General Manager) had made a success of the HP 3000 by bundling IMAGE in the mid '70s. Fred was firm but courteous. No tomatoes (err, tea bags) were thrown. Perhaps the whole “Boston Tea Party” legend started because Fred used the word “stupid” in public, applying it to HP’s management, with no apologies.
The crucial work needed to support a dizzy array of date types was near the apex of White's work at Adager, details scrutinized and attended to during the advent of Y2K. After his retirement, White remained visible in both online communities and at gatherings of the 3000 community's most formidable minds.
His computer career crossed five decades, starting in 1957 when programmer degrees didn’t exist and math experts did the heavy lifting to create file systems, operating environments and applications. In the beginning of his work for HP, he was creating the first file system for the 3000. He was then transferred to another project that grew into the creation of IMAGE.
He came to his HP work from 12 years of positions at Sylvania Electronic Defense Lab, United Technology Center and IBM. White had prepared for his more than 43 years of programming by work and study in forestry, engineering, Japanese, criminology and math. He joined Sylvania two months before Sputnik was launched by the Russians. By 1969 he’d responded to HP’s entreaties and followed some UTC colleagues to HP Cupertino, where he headed up the File System Project for the Omega System, which evolved to MPE.
Never a fan of large organizations, White eventually left HP in 1981 after he had been moved away from IMAGE and onto other projects. He first met Rego when the latter traveled to HP Cupertino to meet the IMAGE creators and learn more about IMAGE and its data structures. White took a post which Rego offered as a consultant to Adager in 1981, and became a senior research engineer for that company in 1989.
During the 1980s and 1990s, the tall, silver-haired programmer cut a notable swath through the HP 3000 community, especially at the annual Interex user group meetings. Always ready to level with HP’s management about what the HP 3000 needed, White’s comments and criticisms in those meetings represented the same unflinching focus required for his SPL programming on the 3000’s internals.
White always wanted to stay busy at his work. In 1946 he worked on Okinawa as a Japanese interpreter for a construction company and applied for a decrease in pay when he thought the company hadn’t given him enough to do. His 19-plus years with Adager made up the biggest single stay in a career in which he said “I quit a lot of jobs. That’s what I’m prone to do when management screws up.”
In his retirement White was active with family members, traveling, hiking and bird watching. The subject of the watches was mostly raptors, he added. "We like our place in Clarkdale (desert plants and critters) with great views of Mingus Mountain and the red rock area of Sedona," he said in 2003. "I like keeping in touch with many of my old friends and enemies on the Internet and mailing lists."
When we asked him about the single biggest mistake HP made with the HP 3000, White was ready with
"at least five I can think of. 1. Not having the development teams being the support teams. 2. Getting in bed with Oracle. 3. Not being aware that there are no relational databases, just relational access to databases. 4. Following the Unix pied piper. 5. Not marketing the HP 3000. For example, they never bothered to tell the world that the computers they used at corporate headquarters were HP 3000s."
As to what kept him so productive for so long, he mentioned his single-language focus on SPL, as well as still being interested in his work. But he also said, "Having a boss who was more interested in quality than quantity." The community poured out good wishes to a special email address, firstname.lastname@example.org, in his final days. One developer who heard of his passing said, "Let's hope that when Fred gets upstairs, his entry permit to Heaven is stamped 'Automatic, Master'."
November 18, 2014
Replacing rises as migrator's primary choice
It's the end of 2014, just about. Plenty of IT shops have closed down changes for the calendar year. Many 2015 development budgets have been wrapped up, too. Among those HP 3000 operations which are still considering a strategy for transition, there's only one assured choice for most of who's left. They'll need to replace their application. Not many can rehost it.
We've heard this advice from both migration services partners as well as the providers of tools for making a migration. An HP 3000 is pretty likely to be running an application with extensive customization by this year. We've just now edged into the 14th year since HP announced a wrap-up of its interest in all things MPE/iX. Year One began in mid-November of 2011. After completing 13 years on watch during the Transition Era, there's a lot of migration best practices to report. More success has been posted, at a better price and on schedule, when a replacement app can be integrated along with a new server and computing environment.
Of course, massive applications have been moved. One of the largest was in the IT operations of the State of Washington Community College Computing Consortium. It was a project so large it was begun twice, over enough elapsed time that the organization changed its name. The second attempt better understood the nuances of VPlus user interface behaviors. There were 40 staffers and at least four vendor services groups working on the task.
One of the issues that's emerged for rehosting organizations is a reduction in MPE expertise. Companies can still engage some of the world's best developers, project managers, and rewriting wizards for MPE/iX. It's harder to assign enough expert human resources who know your company's business processes. That's why a top-down study of what your apps are doing is the sort of job that's been going out-of-house. By this year, it would be better to engage an outside company to replace what's been reliable. This hired expertise ensures a company doesn't lose any computing capability while it makes a transition.
You'll need the use of tools to manage data in a replacement, though. Everything else is likely to change, even in a replacement, except for the data. "Replacement requires reorganizing data," Birket Foster of MB Foster told us this summer. "You could start cleaning your data now." Foster is presenting a Webinar on the subject of the Three Rs -- Rehosting, Replacing, or Retiring -- tomorrow (Wednesday) at 2PM Eastern Time.A company making a transition to a replacement app needs to understand what data will be needed, at what detail level, and in what timeframe. The best answers to those questions might come from outside of the IT group. In face, Foster says they often do. A solid team of transition stakeholders always includes an important seat for a member from the business group.
Replacement of a 15- or 20-year MPE/iX app suite also might not be a favored choice in the IT group. That group includes the experts who know the programs best. Nothing seems like it will be a clean, quick fit for what's been running the company -- not at first. Replacing with a non-MPE version of the app sometimes leaves key integrated surround code at the curb, too. Replacing surround code is a good project for outside expertise. Companies which consult on that task have field experience on success to share.
The good news: replacing a business suite is not as dangerous as replacing a joint. You get to shop and specify and test for replacement software, even while the worn-down hip of the business suite continues to bear the weight of the company's enterprise. Backing out of a replacement -- replacing the replacement -- is just as extensive in software as it is in medicine. It's like doing it all over again. But replacing after an attempt at rehosting? That's the least effective strategy of all.
November 17, 2014
HP's 3000 power supply persists in failure
Amid a migration project, Michael Anderson was facing a failure. Not of his project, but a failure of his HP 3000 to start up on a bad morning. HP's original hardware is in line for replacement at customers using the 3000 for a server. Some of these computers are more than 15 years old. But the HP grade of components and engineering is still exemplary.
"I was working with a HP 3000 Series 969, and one morning it was down," he reported. "All power was on, but the system was not running; I got no response from the console. So I power-cycled it, and the display panel (above the key switch) reported the following."
Proceeding to turn DC on
Anderson explained that Charles Johnson of Surety Systems replaced the power supply for the system.
On the console it displayed garbage when the power was turned on, but the message on the display remained. I wasn’t sure what to replace. I was thinking the power supply — but all of the power was on. As it turned out, even in the middle of a power supply failure the 3000 was working to get out a message. The back side, the core I/O, FW SCSI, and so on, all appeared to have power. That is why I found it hard to believe that the power supply was the problem.
He explained that (back in the day) HP engineered some of the best power supplies in world, lots of checks and verifications. Even though the power supply had actually supplied DC power to the various components, it was not able to verify it.
So the message "Proceeding to turn on DC power" remained on the front panel display, meanwhile the boot process on the console would hang, and if you do a <cntrl-B>, RS it would time-out with a msg:
"FATAL ERROR: System held in reset. POW_ON never came back (APERR 21)"
"Waiting until it's reasserted......"
Bill & Dave's Excellent Machine — even with a power supply failure, it still manages to get a message out (in plain English) attempting to explain the failure.
November 14, 2014
Our World's Greatest Cartoon, Ever
Because it's so crucial, and because Alan Yeo was brilliant in commissioning it. Mark your calendars. (Click it for detail)
November 13, 2014
Thursday Throwback: IMAGE vs. Relational
As a precocious 18-year-old, Eugene Volokh wrote deep technical papers for HP 3000 users who were two or three times his age. While we pointed to the distinctions between IMAGE master and automatic datasets recently, Eugene's dad Vladimir reminded us about a Eugene paper. It was published in the fall of 1986, a time when debate was raging over the genuine value of relational databases.
While the relational database is as certain in our current firmament as the position of any planet, the concept was pushing aside proven technology 28 years ago. IMAGE, created by Fred White and Jon Bale at HP, was not relational. Or was it? Eugene offered the paper below to explore what all the relative fuss was about. Vladimir pointed us to the page on the fine Adager website where the paper lives in its original formatting.
The relationships between master and automatic and detail datasets pointed the way to how IMAGE would remain viable even during the onslaught of relational databases. Soon enough, even Structured Query Language would enter the toolbox of IMAGE. But even in the year this paper emerged, while the 3000 still didn't have a PA-RISC model or MPE/XL to drive it, there was a correlation between relational DBs and IMAGE. Relational databases rely on indexes, "which is what most relational systems use in the same way that IMAGE uses automatic masters," Eugene wrote in his paper presented at COBO Hall in Detroit (above). QUERY/3000 was a relational query language, he added, albeit one less easy to use.
Vladimir admits that very few IT professionals are building IMAGE/SQL databases anymore. "But they do look at them, and they should know what they're looking at," he explained.
Relational Databases Vs. IMAGE:
What The Fuss Is All About
By Eugene Volokh, VESOFT
What are "relational databases" anyway? Are they more powerful than IMAGE? Less powerful? Faster? Slower? Slogans abound, but facts are hard to come by. It seems like HP will finally have its own relational system out for Spectrum (or whatever they call it these days). I hope that this paper will clear up some of the confusion that surrounds relational databases, and will point out the substantive advantages and disadvantages that relational databases have over network systems like IMAGE.
What is a relational database? Let's think for a while about a database design problem.
We want to build a parts requisition system. We have many possible suppliers, and many different parts. Each supplier can sell us several kinds of parts, and each part can be bought from one of several suppliers.
Easy, right? We just have a supplier master, a parts master, and a supplier/parts cross-reference detail:
Now, why did we set things up this way? We could have, for instance, made the SUPPLIER-XREF dataset a master, with a key of SUPPLIERS#+PART#. Or, we could have made all three datasets stand-alone details, with no masters at all. The point is that the proof of a database is in the using. The design we showed -- two masters and a detail -- allows us to very efficiently do the following things:
- Look up supplier information by the unique supplier #.
- Look up parts information by the unique part #.
- For each part, look up all its suppliers (by using the cross-reference detail dataset).
- For each supplier, look up all the parts it sells (by using the cross-reference detail dataset).
This is what IMAGE is good at -- allowing quick retrieval from a master using the master's unique key and allowing quick retrieval from a detail chain using one of the detail's search items.However, let’s take a closer look at the parts dataset. It actually looks kind of like this:
PART# <-- unique key item
What if we want to find all the suppliers that can sell us a "framastat"? A "framastat", you see, is not a part number -- it's a part description. We want to be able to look up parts not only by their part number, but also by their descriptions. The functions supported by our design are:
- Look up PART by PART#.
- Look up SUPPLIERS by SUPPLIERS#.
- Look up PARTs by SUPPLIERS#.
- Look up SUPPLIERs by PART#.
What we want is the ability to
- Look up PART by DESCRIPTION.
The sad thing is that the PARTS dataset is a master, and a master dataset supports lookup by ONLY ONE FIELD (the key). We can't make DESCRIPTION the key item, since we want PART# to be the key item; we can't make DESCRIPTION a search item, since PARTS isn't a detail. By making PARTS a master, we got fast lookup by PART# (on the order of 1 or 2 I/Os to do the DBGET), but we forfeited any power to look things up quickly by any other item.
And so, dispirited and dejected, we get drunk and go to bed. And, deep in the night, a dream comes. "Make it a detail!" the voice shouts. "Make it a detail, and then you can have as many paths as you want to."
We awaken elated! This is it! Make PARTS a detail dataset, and then have two search items, PART# and DESCRIPTION. Each search item can have an automatic master dataset hanging off of it, to wit:
What's more, if we ever, say, want to find all the parts of a certain color or shape, we can easily add a new search item to the PARTS dataset. Sure, it may be a bit slower (to get a part we need to first find it in PART#S and then follow the chain to PARTS, two IOs instead of one), and also the uniqueness of part numbers isn't enforced; still, the flexibility advantages are pretty nice.
So, now we can put any number of search items in PARTS. What about SUPPLIERS? What if we want to find a supplier by his name, or city, or any other field? Again, if we use master datasets, we're locked into having only one key item per dataset. Just like we restructured PARTS, we can restructure SUPPLIES, and come up with:
Believe it or not, this is a relational database.
If this is a relational database, I'm a Hottentot
Surely, you say, there is more to a relational database than just an IMAGE database without any master datasets. Isn't there? Of course, there is. But all the wonderful things you've been hearing about relational databases may have more to do with the features of a specific system that happens to be relational than with the virtues of relational as a whole.
Consider for a moment network databases. IMAGE is one example, in fact an example of a rather restricted kind of network database (having only two levels, master and detail). Let's look at some of the major features of IMAGE:
- IMAGE supports unique-key MASTERS and non-unique-key DETAILS.
- IMAGE does HASHING on master dataset records.
- IMAGE has QUERY, an interactive query language.
Which of these features are actually network database features? In other words, which features would be present in any network database, and which are specific to the IMAGE implementation? Of the three listed above, only the first -- masters and details -- must actually be present in all databases that want to call themselves "network." On the other hand, a network database might very well use B-trees or ISAM as its access method instead of hashing; or, it might not provide an interactive query language. It would still be a network database -- it just wouldn't be IMAGE.
Why is all this relevant? Well, let's say that somebody said "Network databases are bad because they use hashing instead of B-trees." This statement is wrong because the network database model is silent on the question of B-trees vs. hashing. It is incorrect to generalize from the fact that IMAGE happens to use hashing to the theory that all network databases use hashing. If we get into the habit of making such generalizations, we are liable to get very inaccurate ideas about network databases in general or other network implementations in particular.
The same goes for relational databases. The reason that so many people are so keen on relational databases isn't because they have any particularly novel form of data representation (actually, it's much like a bunch of old-fashioned KSAM/ISAM-like files with the possibility of multiple keys); nor is it because of some fancy new access methods (hashing, B-trees, and ISAM are all that relational databases support). Rather, it's because the designers of many of the modern relational databases did a good job in providing people with lots of useful features (ones that might have been just as handy in network databases).
What are relational databases: functionality
The major reason for many of the differences between relational databases and network databases is simple: age. Remember the good old days when people hacked FORTRAN code, spending days or weeks on optimizing out an instruction or two, or saving 1000 bytes of memory (they had only 8K back then) ? Well, those are the days in which many of today's network databases were first designed; maximum effort was placed on making slow hardware run as fast as possible and getting the most out of every byte of disk.
Relational databases, children of the late '70s and early '80s had the benefit of perspective. Their designers saw that much desirable functionality and flexibility was missing in the older systems, and they were willing to include it in relational databases even if it meant some wasted storage and performance slow-down. The bad part of this is that, to some extent, modern relational databases are still hurting from slightly decreased performance; however, this seems to be at most a temporary problem, and the functionality and flexibility advantages are quite great.
For even more IMAGE education, like the advantages of IMAGE over relational databases, and a tour of the flexibility that automatic masters provide, see the remainder of the paper on the Adager website.
November 12, 2014
Public server with a mission: CHARON cloud
About a month ago, Stromasys announced that the CHARON HP 3000 emulator was going to be finding a home in the cloud. The company says it's talking to prospective partners to put such a 3000 capability onto a standard cloud provider, such as Amazon Web Services. That would establish a working environment where the 3000 has already performed a mission. More than a decade ago, Hewlett-Packard believed that a 3000 for public use would help the 3000 community.
The server was dubbed Invent3K, because its mission was to further the 3000's lifespan through the invention of software. HP stocked it with subsystems, offered accounts for free, and let development commence. Some useful products came out of Invent3K. The first that comes to mind is a version of perl ready for MPE/iX. That's a version that continues to work.
Now CHARON might do similar work to help extend the MPE/iX lifespan. Plenty of people want to experience the emulator's powers. Tapping a free server, including free accounts where homesteaders' applications and test databases could reside, would offer the world a useful test bed for transition onto Intel hardware and away from HP-branded boxes.
Last fall the MPE expertise inside Stromasys suggested if there would be interest, from a volunteer, in putting a public-access CHARON-fueled machine on the Internet." It's the sort of mission OpenMPE might have done in its heyday. A lot has changed since HP's labs shut down and OpenMPE was left without much mission. A public access server for the world's only HP 3000 emulator would get ample traffic. It could also be a vital proof of concept for using a 3000 based in the cloud.
November 11, 2014
Veterans get volunteered for transition's day
Here on Veteran's Day — I'm a vet of the '70's-era military — I'm remembering there are IT pros with another kind of veteran status. They are people who count more than a couple of decades of experience with the HP 3000, managing their servers since before the time that Windows was the default computing strategy. They've been through a different kind of conflict.
I've learned that the most embattled managers employ a surprising tool. It's a sense of humor, reflected in the tone of their descriptions of mothballing the likes of 25-year-old independent apps during migrations. They have to laugh and get to do so, because their attempts to advance their positions might seem like folly at first look, or even in a second attempt.
Really, an assignment like putting Transact code into an HP-UX environment? Or take the case of working around a financial app software from Bi-Tech -- an indie vendor that "really stopped developing it for the 3000 years ago," according to City of Sparks, Nevada Operations & Systems Administrator Steve Davidek. There's been some really old stuff doing everyday duty in HP 3000 shops. The age of the applications was often in line with the tenure of the project's management.
These pros typify the definition of veterans, a term we'll use liberally in the US today to celebrate their sacrifices and courage. Facing battle and bullets is not on par with understanding aged code and logic. But two groups of people do have something similar at heart. Both kinds of veterans have been tested and know how to improve the odds of success in a conflict. Youthful passion is important to bring fresh energy to any engagement, military or technological. What earns the peace is experience, however grey-haired it looks next to Windows warriors.
With each mission accomplished -- from what looks like the Y2K effort of 14 years ago to embracing a roll-your-own Unix that replaced MPE's integrated toolset -- these veterans moved forward in their careers. "Our knowledge base is renewed with this work," one said after migrating apps that served 34 Washington state colleges. "We're on the latest products."
Recruiting IT talent into small towns — and the 3000 runs in many small cities where manufacturing labor is less costly — meant hiring for Windows experience. Adopting Windows into an organization means leaving proprietary environments even more popular than MPE/iX. Like HP-UX.
Leaving a familiar environment means enduring risks. But a tone of "yeah, that'll happen, but we'll manage through it" is what I hear from the 3000 pros marching into the dark of 2015 and beyond. And if a migration is happening next year or the years beyond, you may want to thank a colleague -- anyone whose IT battles have promoted the knowledge that creates veterans, marching in the ranks of both managers and vendors alike.
Les Vejada worked with HP 3000s for more than 20 years at HP, then moved on to HP's other enterprise platforms until the vendor cut his job. It was as if he'd mustered out of a unit. "I don't work with the 3000 anymore," he told us when he joined the Linked In HP 3000 Community. "I know the 3000 is dying, but it brings back a lot of great memories. I probably would still take something on a MPE machine if offered." Whether it's in-house, or in-community, one motto remains as valid today as it did after any conflict: Hire a Vet.
November 10, 2014
Sharing the Source of SLEEPER for MPE
SLEEPER is one of the best-known Contributed Software Library (CSL) programs in the history of the HP user group Interex. For years after the user group shut down, the CSL tapes -- created once a year and refreshed with new programs after every annual user meeting -- were the most significant Interex asset left to the community. But making the CSL a public asset, after the Interex shutdown, encountered a snag. How could those useful programs like SLEEPER get shared, if they didn't have clearance from the contributing companies?
SLEEPER and BOUNCER were the programs most often cited in the snags, since they were shared by way of the 3000 shop at Boeing. It was heard a lot, this caution, while people were searching for CSL collections after the Interex meltdown. "Oh, we can't go and release that swap tape," people like the Interex curators would say. "There's BOUNCER on there, and we'd have to get permission from Boeing." Well, that's not exactly true.
Maybe one email would've gotten the process of permission started. It's different times by now. The SPL source for SLEEP, a progeny of SLEEPER, was shared up on the 3000-L mailing list last week. (That's a link to the 3000-L archive message, containing the code in the message body.) Ray Legault offered the code when John Korb, a 3000 consultant, was asking around for it.
Korb thinks so highly of SLEEPER that wants to port the program to PHP, for use on a Linux system. "My goal is to create a PHP version of SLEEPER," he said, "something that can run on Linux desktops and servers and Windows 8.1 desktops -- so I can avoid cron on the Linux box, and avoid the Windows task scheduler."SLEEPER's powers are jobstream-related -- scheduling, after-execution error examination, and so forth. SLEEPER can handle some such scheduling, but the software is not as flexible as the MBF Scheduler written for Windows by MB Foster. That would also be something that can be run on Windows. On MPE/iX systems, the old Nobix Transpooler product is an improvement at sending spoolfiles to printers over the standard MPE/iX jobstream utilties.
However, compared to the built-in job management and scheduling powers of Linux or Windows, the 3000's tools are powerful and nuanced. So much so that MB Foster built Scheduler just for 3000 users who had thousands of jobs to be migrated to Windows application replacements. It's also for sale to non-3000 prospects, too.
Legault created that SLEEP program at a shop outside of Boeing. "The SLEEP I shared was written in Santa Fe drilling by a few of us, including Jay Zimmett and David Mendoza," he said. "I put in Y2K enhancements later on. I still use it today."
As for sharing the code for BOUNCER, Legault said he might be able to help there. BOUNCER logs users off after a specified amount of time, to free up seats on 3000s with limited license counts. VEsoft's LOGOFF also bounces users off a system after a specified amount of time. That is a good security practice, so users who abandon sessions by walking away from a keyboard don't put a system at risk.
The community's never gotten a replacement source for the hundreds of CSL programs that were shared for decades through swap tapes. There was so much value in shared programs that Interex made a business out of it, selling collections as a user group membership benefit. That software exists only inside former members' shops around the world, where DAT tapes live in cabinents or are programs stashed away in 3000 accounts -- ones that might be outside the awareness of IT managers who take on 3000 operations.
There was a time when HP was just as protective of its MPE-related creations, and the vendor still manages license transfers for the ultra-particular 3000 owners who need it. But SLEEPER and BOUNCER were meant to be shared. The CSL was an open source initiative before we ever used the term open source. It's good to see the sharing continue, more than 35 years after the first CSL tapes emerged.
November 07, 2014
Manual and Automatic Masters, Detailed
A few days ago we included a Hidden Value question about how manual and automatic masters work in TurboIMAGE. Our ally and friend Vladimir Volokh called to note that in part of the question, the system manager had found "one detail data set that has thousands of entries which do not appear to be connected to any master."
It wasn't exactly a question, but in a reply on the 3000-L mailing list and newsgroup, Roy Brown gave a fine tutorial on how these features do their jobs for MPE and the 3000 -- as well as how a detail dataset might have zero key fields.
Manual masters can contain data which you define, like Detail sets can, along with a single Key field. Automatic masters contain only the Key field.
In both cases, there can be only one record for a given key value in a Master dataset.
A Detail dataset contains data fields plus zero, one, or many key fields. There can be as many records as you like for a given key value, and these form a chain accessible from the Master record key value. This chain may be sorted, or it may just be in chronological order of adding the records.
Zero key fields in a Detail dataset would be unusual, but is permissible.
Brown explained that "Where there are keys, referential integrity demands that there are no Detail record entries with a key field that is not found in either a Manual or Automatic master, both Key name and Key value. So a Detail data set with Key fields that are not present in a Master record would be a sign of a seriously corrupted database."
However, I doubt this is the case, and when you do a QUERY FORM command, you will see which fields in Detail datasets are Keys, which fields are used to establish Sort orders, and which fields are data pure and simple.
From the Key name, you can determine which Master set links the keys.
As I said above, it is possible to have a Detail dataset with no keys, but these usually contain only a very few records, since direct access to them without keys is cumbersome, and you would otherwise have to trawl right through one to find any given entry.
So a Detail dataset with thousands of unconnected entries would be very unlikely.
The FORM output will allow you to check how the Detail dataset that you think might have unconnected entries is actually linked in.
November 06, 2014
Throwback: Today's Empire of Invent3K
Five years ago today we watched for notice about a fresh 3000 resource on the Web. Invent3K, a public access development server created by HP in 2001, was searching out a new home in November 2009. The vendor shut off Invent3K in November 2008, along with the Jazz website that hosted shareware utilities created by HP and the user community.
Invent3K was an OpenMPE adoption project five years ago. The community probably didn't need a public access development web server by the end of 2009. But replacing HP's withdrawn assets seemed important. Invent3K harkened back to a more hopeful time. 3000 developers were first offered access to MPE accounts on that HP server only about six months before the vendor announced it would end its 3000 programs.
Invent3K was unique in the 3000's history. The server was the first and only place that hosted free, development-use-only subsystem software from HP. Working from an Invent3K account, a developers employed COBOL II, TurboStore, and other HP-branded products while building apps or utilities.
For a time, OpenMPE wanted to sell $99 yearly development accounts on its replacement Invent3K. The community was not accustomed to paying for public access, so sales were slow. OpenMPE was trying to generate revenues for operating things like a Jazz replacement host where contributed tools could be accessed. By that time, much of Jazz had been re-hosted at servers owned by Client Systems and Speedware. Things were not hosted quite the same as on Jazz, though. HP insisted that those two vendors make users click through an End User License Agreement before using the contributed tools re-hosted from Jazz.
Last month, two of the replacement servers for delivering Jazz and Invent3K had online glitches. Speedware's server went offline for a weekend, so its hpmigrations.com website that hosts Jazz delivered only an error. The HP 3000 where Invent3K was headed in 2009 had a small hiccup, too: the 3000-based Empire 3.9 game server lost use of its domain name for awhile in October. Tracy Johnson is the caretaker for the Empire server and its parent -- Invent3K, whose domain name is invent3k.openmpe.com.
But Invent3K is operating today, at least for anyone who had an account established before OpenMPE curtailed its operations. Access is through any terminal emulator with Telnet or VT/Mgr protocol. Once you've configured your terminal emulator, connect to the address invent3k.empire.openmpe.com.After years of reduced 3000 development -- the result of many systems frozen to maintain stability -- Invent3K is more than a testament to shared effort of the community. It's a solution in search of a problem. Free access to the 3000's subsystem products for development wasn't much of a problem by 2010. Invent3K was devised as a means to deliver new software for MPE. The community was encouraged to help, back in 2001.
Advocates for MPE/iX waited longer than expected for invent3k.openmpe.com to come online. The wait was so lengthy that a dispute over who would control the server arose in the OpenMPE group. Ultimately the original openmpe.org domain was locked up, kept out of the hands of the OpenMPE board members. Allegro Consultants stepped up to donate the openmpe.com domain, which it had purchased long before Invent3K was up for adoption.
As a result of moving a consolidated version of Jazz out of HP's labs, the community now faces good news and bad for the Jazz web resources. The good news is there's plenty of redundancy, with Fresche Legacy (nee Speedware), Client Systems, and OpenMPE's volunteers like Johnson all hosting the programs.
The bad news is there are three sets of programs and command files and UDCs, some overlapping, and some not, among those redundant resources. Every host gets to use their own organizational map, so finding something specific probably requires a visit to all three sites. And some tools aren't on any of the servers, like the bash shell program for MPE/iX. Bash was the focus of the recent Shellshock hack, one that had administrators examining their servers for security vulnerability.
For the time being, the Jazz portals are located at:
OpenMPE Jazz: invent3k.openmpe.com/jazz/
Client Systems Jazz: www.clientsystems.com/jazzmain.html
Fresh Legacy/Speedware Jazz: hpmigrations.com/HPe3000_resources/HP_jazz
There is only one Invent3K, however. One might be enough, considering it's an HP 3000.
November 05, 2014
Migration plans: Rehost, Replace, or Retire
Migration begins with an inventory. Application by application, an IT manager does triage on every program on their 3000. The goal is to come up with a disposition for each app. Some ccan be rehosted. Apps built with PowerHouse might be moved to other servers, for example. For other apps, they can be replaced. Ecometry sites could adopt the CommercialWare application, in some cases. Or in other instances, an in-house program suite can be replaced by a Commercial Off The Shelf app. Migration services company MB Foster likes to call those replacements COTS. The company has a Wednesday webinar scheduled on Nov. 19 to explore planning to replace COTS and more.
Foster calls the strategy the Three R's of Migration: Rehost, Replace, Retire. At 2 PM Eastern Time in a couple of weeks (register here), Birket Foster will lead a slide talk with time for questions about what to do if you're migrating away from any one of the above types of applications. The word Retire is already on the minds of MPE-savvy managers, since most of them are older than 50. It turns out that applications can find the end of a career even before their caretakers do.
It's not always this way. At the San Bernadino County schools, the apps to run the California district will have a retirement date from the 3000 that falls after the district's 3000 expert. Sometimes, though, a migration can become easier when older programs that have fallen into disuse are simply erased. There's no need to migrate such an app.
However, there's only one sure way to discover these retirees: inventory and analysis. MB Foster's summary for the Nov. 19 webinar breaks that triage process into inventory of app environments; rating the functionality vs. business suitability of each app; then organizing and prioritizing with a study of interfaces and grouping used that applies to each app. COTS carries a different set of requirements to study during migration planning, the company says, than in-house apps.Look at the IT resources in your company as a portfolio, Foster advises. Create a profile for each asset in your portfolio, using an inventory of the documents, tools, and other parts of each application's environment. Pay attention to these elements across the lifecycle of the app.
The first stage in the lifecycle is a requirements document. What are the workflows being supported? Was the application built or bought, and what roles were considered as the project to charter the application began? If the application is built in house, then technical documentation for the app's design and delivery will be required. If the application is COTS software, then the requirements documentation changes, to include the software selection process and a Fit-Gap analysis.
Every application spends time crossing through three environments: Development, Test, and Operations. Migrating anything means that IT management must consider how to move away from Development elements such an IDE, or source control software, change control tools, scripting languages, and compilers.
Foster says in its summary for the webinar that making a transition away from an app's Test environment means analyzing the test tools, methodologies, test scripts, plus documentation for the app's unit test, integration test, and UAT (User Acceptance Test).
The inventory for analysis of an Operations environment studies "scripts for operating daily, weekly, monthly and other periodic functions," as well as "the toolkit for monitoring the production environment (unique or shared), the scheduler, and other production environment tools."
The analysis will allow the 3 R's of migration to be applied – Rehost, Replace or Retire. Once the application portfolio is divided into these categories, it is time to organize and prioritize the applications. Each application may be standalone or grouped – and the triage process will need to note any internal or external interfaces that will be impacted.
Foster says the webinar will examine and teach about those processes. It might be a way to bring something closer to retirement: an aging 3000, or even old business practices for IT.
November 04, 2014
Tips for Listing SCHEMAs, and FTP Listings
From my existing TurboIMAGE database, I want to generate a listing of data sets, data item names, and their relationships (master, detail). One detail data set has thousands of entries which do not appear to be connected to any master.
Oh, and I cannot remember the difference between manual and automatic masters.
Francois Desrochers first replies, "Use Query's FORM command."
PASSWORD = >> password
MODE = >> 5
Manual masters: programs have to explicitly add entries before you can add related entries in detail sets. Programs have to explicitly delete entries when there are no related detail entries left. In other words, you have to do master dataset maintenance.
Automatic masters: entries are automatically created when a related detail set entry is created. Entries in the master are automatically removed when the last related detail entry is deleted. IMAGE takes care of the maintenance.
Consultant Ron Horner adds, "If you have a tool like Adager or DBGeneral, it can create a file of the database schema. The other way is by using QUERY to get a listing."Horner also said, "The difference between a manual master and auto master is the following:
1. You have to add records to the manual master that contain the key data for any detail datasets that are linked to the master.
2. When working with automatic masters, you don't have to write data to them at all. IMAGE takes care of populating the master.
Ray Legault suggested using Allegro's free XSCHEMA, "A stripped down version of our commercial product DBHTML, this utility simply reads an Image root file and produces a DBSCHEMA-compatible text file as output."
Krikor Gullekian also noted that "With QUERY you can check the databases as long as you know the password. [Ed. note: Password advice is true, except when you're the database owner. No password is required then, just a semicolon.] FO SETS will give you a lot of details."
How can I tell the HP 3000's FTP server to use a standard 'ls -l', and not 'LISTFILE ,2' ?
Allegro's Donna Hofmeister said, The trick is to turn 'Posix' on.
Keven Miller of 3K Ranger added "check out FTPDOC.ARPA.SYS, mainly the POSIX section. It mentions how to set the default to ON."
November 03, 2014
10 years ago, was the 3000 halfway gone?
I revisited a set of user conference slides from 2004 recently, and I found a scheduling milestone. A presentation at that final Interex show in Chicago included a schedule warning. In that year, it was about halfway through the period HP gave the community to get away from their HP 3000s. The deadline for the end of HP support was December 31, 2006, as of the week the presentation rolled out in Chicago.
"Put your plan into the budget during this summer of 2004," one slide suggested. Using that timeline, an IT manager could commence migrating by the summer of '05 and complete a migration by the end of 2006. To be sure, 18 months is a swift migration schedule, but it's possible -- if a company can budget for some outside expertise for project management and coding. The tests will usually be handled in-house. That's up to one third of a migration project's timeline, according to migration services experts like MB Foster.
By the next fall, HP was pulling the carpet out from under such companies. That deadline for HP support got extended by 24 months. With that change, that halfway-gone point was still at the moment of HP's rescheduling. But now there were more than 36 months left. People had not migrated in enough numbers. A greater factor: HP made a business decision to keep support business alive and earning revenues for another two years. It was high-profit revenue. High enough that the deadline got moved again, out by another 24 months.
Since those days, we've seen many customers use the same sort of rolling-forward deadline for their migrations. They plan for the end of 2013, for example, then push out to the end of 2015. Just like HP, customers in places like California school districts and manufacturers in the Southwest are taking more time because they like the return on investment. They can afford to reset the halfway mark.
But what if 2004 was the start of 95 percent of migrations, and the decade since then represents halfway-gone? What's left by now in the user community still includes companies of serious size.Last week we heard of the farthest-out 3000 shutdown date ever scheduled: 2024. Sure, if that replacement project at the MANMAN site gets done sooner, the 3000 will go dark faster. But right now, 2014 is their halfway point, even if you only counted back a full decade ago.
Another customer, the San Bernadino, California school district, will not switch off until the end of 2017. So far away that its resident 3000 guru will already be retired. In that shop, even the exit of expertise won't move things along faster. Operations calendars play a role in scheduling.
The end of the calendar year is a typical time to switch things over for the schools that use the 3000. Classes end in December for a long break. E-commerce and retail companies like Musician's Friend, a former Ecometry site, consider the middle of summer to be their safe cutover time.
A one-month operations break, or even the 90 days before holiday retail time, is a short window compared to a full year. If a site doesn't get to its deadline window in time, there's an extra 6-12 months of 3000 operations to schedule. This mandates a Sustain plan.
Even back in 2004 the presentation advice included Sustain strategy. It included the same counsel given to migrators: Innovate before you migrate. Innovation is the one advantage of extending a halfway point. The IT staff can do some immediate-payback work. Innovation to existing systems helps as soon as it's tested.
Innovations in existing 3000 systems is not as crazy as it sounds. If you're aiming at innovation targets, these three were part of the 2004 slide set.
- Projects that affect operational efficiency and ROI
- Projects that build sills and experience on your staff
- Projects that will help with year-over-year comparisons in the future
If that final target isn't clear, it's a good practice to get another set of eyes on migration planning. IT stakeholders are one set of eyes. Another potential is the migration planners in your community. Knowing what to innovate before you go away: this is also preparing for a migration.