« April 2014 | Main | June 2014 »

May 30, 2014

Deleting 3000 System Disks That Go Bad

Hard-disk-headAs Hewlett-Packard's 3000s age, their disks go bad. It's the fate of any component with moving parts, but it's especially notable now that an emulated 3000 is a reality. The newest HP-built 3000 is at least 11 years old by now. Disks that boot these servers might be newer, but most of them are as old as the computer itself.

A CHARON-based 3000 will have newer drives in it, because it's a modern Intel server with current-day storage devices. However, for the nearly-total majority of the 3000 system managers without a CHARON HPA/3000, the drives in their 3000s are spinning -- ever-quicker -- to that day when they fail to answer the bell.

Even after replacing a faulty 3000 drive — which is not expensive at today's prices — there are a few software steps to perform. And thus, our tale of the failed system (bootup) disk.

Our disk was a MEMBER in MPEXL_SYSTEM_VOLUME_SET. I am trying to delete the disk off the system.  Upon startup of the machine is says that LDEV 4 is not available.  When going into SYSGEN, then IO, then DDEV 4 it gives me a warning that it is part of the system volume set — cannot be deleted.  I have done an INSTALL from tape (because some of the system files were on that device), which worked successfully. How do I get rid of this disk?

Gilles Schipper of GSA said that the INSTALL is something to watch while resetting 3000 system disks.

Sounds like the install did not leave you with only a single MPEXL_SYSTEM_VOLUME_SET disk. Could it be that you have more than one system volume after INSTALL because other, non-LDEV 1 volumes were added with the AVOL command of SYSGEN -- instead of the more traditional way of adding system volumes via the VOLUTIL utility?

You can check as follows:


If the resulting output shows more than one volume, that's the answer.

Schipper offered a repair solution, as well. 

Schipper's solution would use these steps:

1. Reboot with:


2. With SYSGEN, perform a DVOL for all non-LDEV1 volumes


4. Create a new System Load Tape (SLT)

5. Perform an INSTALL from the newly-created SLT

6. Add any non-LDEV1 system volumes with VOLUTIL. This will avoid such problems in future.

Those SLTs are also a crucial component to making serious backups of HP 3000s. VeSoft's Vladimir Volokh told us he saw a commonplace habit at one shop: Neglecting to read the advice they'd received.

"I don't know exactly what to do about my SLT," the manager told him. "HP built my first one using a CD. Do I need that CD?"

His answer was no, because HP was only using the most stable media to build that 3000's first SLT. But Vladimir had a question in reply. Do you read the NewsWire? "Yes, I get it in my email, and my mailbox," she said. But just like other tech resources, ours hadn't been consulted to advise on such procedures, even though we'd run an article about 10 days earlier that explained how to make CSLTs. That tape's rules are the same as SLT rules. Create one each time something changes in your configuration for your 3000.

Other managers figure they'd better be creating an SLT with every backup. Not needed, but there's one step that gets skipped in the process.

"I always say, 'Do and Check,' " Vladimir reports. The checking of your SLT for an error-free tape can be done with the 3000's included utilities. The venerable TELESUP account, which HP deployed to help its support engineers, has CHECKSLT for you to run and do the checking.

There's also the VSTORE command of MPE/iX to employ in 3000 checking. If your MPE references come from Google searches instead of reading your Newswire, you might find it a bit harder to locate HP's documentation for VSTORE. You won't find what you'd expect in a 7.5 manual. HP introduced VSTORE in MPE/iX 5.0, so that edition of the manual is where its details reside.  (Thanks to Digital Innovations' HP MM Support website for its enduring MPE/iX manual archives).

It's also standard practice to include VSTORE in every backup job's command process.

There's another kind of manager who won't be doing SLTs. That's the one who knows how, but doesn't do the maintenance. You can't make this kind of administrator do their job, not any more than you can make a subscriber read an article. There's lots to be gained by learning skills that keep that 3000 stable and available, even in the event of a disk crash.

12:40 PM in Hidden Value, Homesteading | Permalink | Comments (0)

May 29, 2014

They knew what they had before it was gone

In the classic Joni Mitchell song, she asks, "Don't it always seem to go, you don't know what you got 'till it's gone?" However, in the HP 3000 world, the advocates, fans and users know the special place the 3000 held in their lives -- and long before it was really gone.

Redwoods PicnicAt the now-defunct Boyle Engineering, the last in a long line of HP 3000s was sold for scrap this month, according to Harlan Lassiter. When Boyle was purchased in 2008, the site that housed the 3000 was closed down. Equipment was left behind, but Lassiter -- who worked at Boyle 27 years -- kept track of an abandoned 3000 Series 928. He reported he was sad to see it go. One last boot-up was all that Lassiter wanted at Boyle, whose services were engaged to plan, design, and construct infrastructure projects.

Last time I was in the building, in the corner of the raised floor computer room, was our HP 3000 928 system, console monitor and LPQ1200 printer. Yesterday it was gone. Apparently it was picked up late last week as scrap. Also picked up and sold for scrap from the room were about 50 Dell LCD monitors (some new, still in bubble wrap) and perhaps 30 Dell desktop computers, APC battery backup systems, server arrays, and other assorted computer equipment. Much of the equipment could have been donated to organizations that could use a computer system, even though it would not be the most current.  

That 928 was the last in a series of HP 3000 systems for the company, having begun with a Series II when I first started with Boyle in 1979 . We came a long way. I started as a programmer and left as the system manager. The system ran all of the company in-house accounting, finance, payroll and project tracking reports and engineering software.  All software was developed in-house and was written in FORTRAN. As FORTRAN evolved through the years, so did the software. Files were converted from serial (flat) files to KSAM and eventually to IMAGE databases. What used to take overnight to process took less than an hour in later days.

It was a great learning experience. I guess I was hoping to fire the system up one more time just for nostalgia's sake, since I am the only one left that would be able to do such a thing. 

Another piece of HP history, a living one that served both the 3000 and HP-UX systems, has been bulldozed, right off the ground of the old Hewlett-Packard Cupertino campus.

Apple HQApple now owns the acres of Cupertino where the HP 3000 grew into a business powerhouse. The HP buildings have been razed, and Jim Hawkins of HP reports that even the grove of redwood trees is no more. Apple's building a spaceship-like headquarters in its place. Employees and retirees held picnics there, along with the historic Glendenning Barn which HP maintained as a reminder of the property’s pioneer-era life as an apricot orchard and farm. Hawkins, one of the last 3000-focused engineers at Hewlett-Packard, celebrated those redwoods as a place of the 3000 community.

The HP Cupertino Site, home for (most of) the HP 3000 R&D teams, and manufacturing source of (most) pre-RISC MPE servers, is now scraped clean in preparation to land Apple's "Steve Jobs memorial spaceship." 

The redwood grove where execs used to serve us hamburgers during beer busts is all cut down, as are apparently all other trees except those on the borders of Pruneridge, Wolfe, Homestead, and Tantau streets.

After reading Lassiter's farewell, Ed Effinger shared a memorial in waiting. His was report of a forthcoming shutdown at Conestoga College in Kitchener, Ontario. "We have a similar story to what mine will be next March," Effinger said, "as we plan to pull the plug on our Series 929. We also started with HP in 1975-76, to replace our old Honeywell system -- and I too have done all things here."

These are customers of more than 35 years of MPE computing, and that redwood grove was servicing the community at HP's campus even before that time. At least these veterans of the ecosystem know what they're losing, and how much that loss stings. At the old HP campus, it looks like Apple's paving paradise to put up a an underground parking lot.

07:38 PM in History, Homesteading, News Outta HP | Permalink | Comments (0)

May 28, 2014

3000: Cards and punching and tape, oh-29!

The Hewlett-Packard System/3000 -- that's what the computer called the 3000 was first known as during the era when punched cards and tape could drive its data. The 3000-L mailing list popped back up to life last week with stories about the era when hanging chads and IBM 029 punch machines were a working part of MPE's four decades of historic service.

Reader-punchHistory for an active operating environment whose pedigree includes punched tape and punched cards -- that's pretty much exclusive to the HP 3000. Punching pedigree is a mark of utility and durability, even if those card readers are only in museums and garages today. One recently sold on eBay for more than $300 to a collector.

Maybe it was the debut of a System 360 mainframe on Mad Men's penultimate season that put punched cards into the minds of its longstanding users. Mark Ranft of Pro3K told a story last month about his first IT job as a System 360 operator in the US Marine Corps -- and how that led to a Nortel assignment with a card reader and paper tapes. "Thankfully they had a Series III [HP 3000].  As an operator, I was bored to death, so I read all the manuals.  That's how I got hooked on MPE."

PaperTape2About a month later, former OpenMPE secretary Tracy Johnson started the 3000-L readers down nostalgia lane by pointing to TELTAC: a Teletype tape-to-punched card conversion program. "Was there a Contributed Software Library program for that?" he asked. The MPE CSL was born as a swap tape, during this era of punched card holdouts. Gilles Schipper of GSA associates replied there was no need for a CSL program, because FCOPY has always had that capability.

The memories of cards and punching and the 3000 started to tumble out of the readers of the L. "If I recall correctly," said Terry Simpkins of Measurement Specialties, "when I was with HP's Disc Memory Division in Boise back in the early '80s, we actually had a card reader connected to one of our 3000s. I brought several boxes of cards with me from grad school, and we read them into EBCDIC files. Don't ask why I was carrying boxes of punch cards around the country."

The HP 3000, in its infancy, could use punched cards or paper tape. Those were two computing props not seen in Mad Men this spring. But they're remembered as durable data mediums, even by those of us who dropped a deck or two of them in front of a college computing center on the way to running a program.

"Why cards? asked Tracy Pierce. "A darn reliable medium. You never worried a sec about losing the data in those grad school cards. It's easy to mangle a card so it's not machineable, but darn difficult to really destroy its data. You can run cards through a shredder and still recover the data."

In just a matter of about eight hours, Jeff Kell of the University of Tennessee at Chatanooga had chipped in a thorough history of how data was sent to and from the earliest HP 3000s. The story included a speed measurement that used a holiday as comparison. The HP optical mark sense card reader was the tortoise in the data race.

As for the "mark-sense" reader... we had this grand plan to do grades on "mark-sense" cards. The idea was to "print" class cards (one card per student, sorted by instructor by class), and let them pencil-mark the corresponding grade for the student.  It was great in theory, but the mark-sense reader had much less than stellar performance and reliability (it sucked!). And having these "printed" cards burst on their perforations to yield the "card" left some rough edges, which the reader really, really hated. And it was slow as Christmas. Heck, it was slower than Leap Year.

We got an HP2000/Access system in Fall of 1975. It not only supported a card reader and printer, but also supported the remote job entry communications with the IBM at the University of Tennessee at Knoxville. So we had a card reader upstairs in the student keypunch lab, as well as a printer, and there was no more waiting for submission.  They could just feed their jobs directly to the reader, their printouts came back to the printer, and it was available constantly. Big step forward.

Later we got an HP 3000, and had a copy of MRJE/3000. Now students could enter their programs online via Editor/QEdit/Quad/whatever they prefer, submit their jobs via MRJE, and view their output in SPOOK before actually printing it out. Even better still.

Kell added that his campus kept the card reader for the 3000 for legacy purposes. This is a 3000 customer that only turned off its MPE systems last December.

Card readers for the 3000 lasted through the lifespan of the Series 70, which means into the early 1990s. 

"HP reluctantly supported a card reader through Series 70," said Bob Jankowski of Ideal Computer. "It was definitely available with HP-IB interface and required a dedicated GIC and an auto tap switcher for power. I remember working on these a few times -- and one of my current customers still has theirs in the computer room. One of the wearing parts was called a 'picker sector.' Try saying that 10 times fast. The HP 7260A was the optical mark sense reader. I remember it being a serial device used through MPE-V and being picky about what it would read.

Kell's colleague Tony Shepherd recalled the budget-conscious approach that a computing pro of the 1970s had to embrace. Carpentry power tools and rubber stamps were sometimes among the best data tools.

The perforated card edges were a problem. We wound up printing and bursting them, then putting them in card trays (3,000 cards per tray) and sanding the long edges. It took a little explaining to get management to understand why we needed to buy a dual-action orbital sander with integrated vacuum pickup in order to get grades to post. Sears had one for about $50 that did a great job. We had a good incentive to get the OpScan process developed quickly, and it was indeed much better.

In those days we were just staying ahead of the bleeding edge -- we had a small (but dedicated and very smart) staff and no money. Solutions had to be quick and cheap. For example, one office wanted a new system to record sales of parking bumper stickers. We spent some hours "studying" their needs, then presented them with a bound ledger book and a Bates numbering stamp. It fulfilled all their stated requirements.

The physical manifestation of data, cards had personality. "The first card stock we used had problems with curling," Walter Murray reported from his early 3000 days, "but the second stock we tried worked pretty well. Something to do with "long grain" versus "short grain," as I recall. You'd think we were buying rice."

As for the card reader on Murray's Series II HP 3000, Kell described it as hardware that reduced the footprint versus IBM's original designs.

In the HP card reader you loaded cards on the right into a diagonally-slanted tray, pushed the start button, and it had some sort of combination air driven / pick roller thing that swiped the card through the reader into the output stacker on the left. It was pretty darn quick about it too... not up to par with an IBM's speed, but not slouchy at all. And it fit easily on a tabletop, while the IBM version was the size of a chest freezer.

The work was obviously tedious. It might have helped develop an attention to detail in the earliest part of a 3000 pro's career. Kell has the last word on what a keypunch looked like from that era.

If I remember models correctly, there was the 029 (punched cards real-time), the 129 (buffered a card, you could "backspace," it only punched the card once you released it) -- and this service bureau I worked for had some key-to-disk things that "punched" (wrote) data to floppy diskettes. When they were done and verified, you loaded the diskettes into another IBM thing that loaded the diskettes to 9-track tapes that were used as data input on the mainframe.

09:59 PM in History, Homesteading | Permalink | Comments (0)

May 27, 2014

Does cleaning out HP desks lift its futures?

Migration sites in the 3000 community have a stake in the fortunes of Hewlett-Packard. We're not just talking about the companies that already have made their transition away from MPE and the 3000. The customers who know they're not going to end this decade with a 3000 are watching the vendor's transformation this year, and over the next, too.

Clean-Out-Your-DeskIt's a period when a company that got bloated to more that 340,000 companies will see its workforce cut to below 300,000 when all of the desks are cleaned out. The HP CEO noted that the vendor has been through massive change in the period while HP was cleaning out its HP 3000 desks. During the last decade, Meg Whitman pointed out last week, Compaq, EDS, Automomy, Mercury Interactive, Palm -- all became Hewlett-Packard properties. Whitman isn't divesting these companies, but the company will be shucking off 50 percent more jobs than first planned.

Some rewards arrived in the confidence of the shareholders since the announcement of 16,000 extra layoffs. HP stock is now trading at a 52-week high. It's actually priced at about the same value as the days after Mark Hurd was served walking papers in 2010. Whitman's had to do yeoman work in cost-cutting to keep the balance sheet from bleeding, because there's been no measureable sales growth since all 3000 operations ceased. It's a coincidence, yes, but that's also a marker the 3000 customer can recall easily.

When you're cutting out 50,000 jobs -- the grand total HP will lay off by the end of fiscal 2015 in October of next year -- there's no assured way of retaining key talent. Whitman said during the analyst conference call that everybody in HP has the same experience during these cuts. "Everyone understands the turnaround we're in," she said, "and everyone understands the market realities. I don't think anyone likes this."

These are professionals working for one of the largest computer companies in the world. They know how to keep their heads down in the trenches. But if you're in a position to make a change in your career, a shift away from a company like HP that's producing black ink on its ledger through cuts, you want to engage in work you like -- by moving toward security. In the near term, HP shareholders are betting that security will be attained by the prospect of a $128 billion company becoming nimble, as Whitman vowed last week.

In truth, becoming nimble isn't going to be as important to an HP enterprise customer as becoming innovative. Analysts are identifying cloud computing as the next frontier, one that's already got profitable outposts and the kind of big-name users HP's always counted in its corral. During an interview with NPR on the day after the job cuts rolled out, Michael Regan of Bloomberg News pointed out that most of HP's businesses have either slipped, like printers and PCs, or are under fire.

Servers are under a really big threat from cloud computing. HP formerly, you know, their business was to sell you the server so that you can store all your data yourself and have customers access the data right off of your server from the Internet.

The big shift over the last few years has been to put it on a cloud, where basically companies are renting space on a server, and consumers a lot of times aren't even buying any Web applications. They're renting them over the cloud, too. All three of [HP's] main business lines are really under a lot of competition from tablets and cloud computing.

This isn't good news for any customer whose IT career has been built around server management and application development and maintenance. Something will be replacing those in-house servers at any company that will permit change to overturn its technology strategy.

Cloud computing is a likely bet to replace traditional server architectures at companies using the HP gear. But it's a gamble right now to believe that HP's strength in traditional computing will translate to any dominance in cloud alternatives. IBM and Amazon and Google are farther in front on these offerings. That's especially true for the small to midsize company where an HP 3000 is likely to remain working this year.

During the NPR interview, Regan took note of the good work that's come from Whitman's command of the listing HP ship. But the stock price recovery is actually behind both the Standard & Poors average and the average for technology firms during Whitman's tenure. She's floating desks out the door, but that probably won't be enough to float the growth trend line upward. When extra cuts are needed to keep all those shareholders happy, one drooping branch could be the non-industry standard server business.

Any deeper investment in any HP strategy that relies on catching up with non-standard technology should float away from procurement desks for now.

11:01 PM in Migration, News Outta HP | Permalink | Comments (0)

May 23, 2014

Unicom calls PowerHouse users to mansion

Many things are on the table for change in the PowerHouse community, now that Unicom Global owns the software suite and contracts with customers. One of the more notable adjustments in the new order is a June 27 users conference, a single day's meeting to be held on the grounds of a Hollywood landmark.

From 8:30 to 3 that day at "the Legendary PickFair Estate in Beverly Hills," customers and developers using PowerHouse can attend a user conference. At the same time, the vendor's CEO is hand-picking from executive community members who want to serve on the first PowerHouse Customer Advisory Board. The vendor is calling customers over the phone, in addition to email notices and postings on LinkedIn and other web locations. For some customers, the Unicom calls will be the first PowerHouse outreach they've heard in many years.

PickFairThe meeting represents the launch of a PowerHouse user group, one of the first, if not a groundbreaker. I scanned through 20 years of HP 3000 reporting, and plumbed back another 10 while on watch at the HP Chronicle and as an independent editor, and couldn't recall a PowerHouse user group before now. The dim memory of a few Special Interest Group spin-offs from Interex comes to mind. We'd be glad to know if there's any PowerHouse history we overlooked.

The way this group differs from those other user group SIGs is that it's being founded by its vendor. In the days of Interex user groups -- from the early '70s through the end of the 20th Century -- that kind of leadership was considered too intrusive. But times have changed for user groups. They often need the support and attention only a vendor can deliver to a product's customers. HP and Encompass share the reins at HP Discover, the Hewlett-Packard enterprise user conference. Discover takes place June 10-12 at the Venetian Resort on the Las Vegas Strip. HP picks up the greatest share of the expenses at that meeting.

The PowerHouse meeting, a little more than two weeks later, calls users to a  mansion -- the former home of Hollywood icons Mary Pickford and Douglas Fairbanks. PickFair is part of the Unicom portfolio, another piece of the evidence that PowerHouse is in for a journey across new grounds.

Users are invited to attend, as well as make a statement about why they'd be a good part of the advisory board, at a Unicom webpage. The cost of the meeting is $500 per person, but if you register two or more attendees, the cost drops to $395 per person. The vendor is inviting customers to "attend the User Group and provide direct input into the PowerHouse roadmap."

There's a travel package deal available as well. Contact the corporation's Russ Guzzo -- who also happens to be leading the integration of PowerHouse into a company that has never sunsetted a product -- at 818.838.0606, or by email at russ.guzzo@unicomglobal.com.

09:56 PM in Homesteading, Newsmakers | Permalink | Comments (0)

May 22, 2014

HP's migration servers stand ground in Q2


The decline of HP's 3000 replacement products has halted
(click on graphic for details)

CEO Meg Whitman's 10th quarterly report today promised "HP's turnaround remains on track." So long as that turnaround simply must maintain sales levels, she's talking truth to investors. During a one-hour conference call, the vendor reported that its company-wide earnings before taxes had actually climbed by $240 million versus last year's second quarter. The Q2 2014 numbers also show that the quarter-to-quarter bleeding of the Business Critical Systems products has stopped.

But despite that numerical proof, Whitman and HP have already categorized BCS, home of the Linux and HP-UX replacement systems for 3000, as a shrinking business. The $230 million in Q2 sales from BCS represent "an expected decline." And with that, the CEO added that Hewlett-Packard believes its strategy for enterprise servers "has put this business on the right path."

The increased overall earnings for the quarter can be traced to a robust period for HP printers and PCs. Enterprise businesses -- the support and systems groups that engage with current or former 3000 users -- saw profits drop by more than 10 percent. HP BCS sales also fell, by 14 percent versus last year's Q2. But for the first time in years, the numbers hadn't dropped below the previous quarter's report.

The decline of enterprise server profits and sales isn't a new aspect of the HP picture. But the vendor also announced an new round of an extra 10,000-15,000 job eliminations. "We have to make HP a more nimble company," Whitman said. CFO Cathie Lesjack added that competing requires "lean organizations with a focus on strong performance management." The company started cutting jobs in 2012, and what it calls restructuring will eliminate up to 50,000 jobs before it's over in 2015.

Enterprise business remains at the heart of Hewlett-Packard's plans. It's true enough that the vendor noted the Enterprise Systems Group "revenue was lower than expected" even before the announcement of $27.3 billion overall Q2 revenues. The ESG disappointments appeared to be used to explain stalled HP sales growth.

But those stalled results are remarkable when considered against what Whitman inherited more than two years ago. Within a year, HP bottomed out its stock price at under $12 a share. It was fighting with an acquired Autonomy about how much the purchased company was worth, and was shucking off a purchase of Palm that would have put the vendor into the mobile systems derby.

If nothing else, Whitman's tenure as CEO -- now already half as long as Mark Hurd's -- contains none of the hubris and allegations of the Hurd mentality. After 32 months on the job, Whitman has faced what analysts are starting to call the glass cliff -- a desperate job leading a company working its way back from the brink, offered to a woman.

As the conference call opened on May 22, HP's stock was trading at close to three times its value during that darkest month of November, 2012. At $31 a share valuation, HPQ is still paying a dividend to shareholders. Meanwhile, the company said it has "a bias toward share repurchases" planned for the quarters to come.

There's still plenty of profit at HP. But the profits for the Enterprise Group, which includes blades and everything that runs an alternative to MPE, have been on a steady decline. A year ago before taxes they were $1.07 billion, last quarter they were $1 billion, and this quarter they're $961 million. Sales are tracking on the same trajectory.

Whitman noted the tough marketplace for selling its business servers in the current market. She also expressed faith in HP's system offerings. It's just that the vendor will have to offer them with fewer employees.

"I really like our product lineup. But we need to run this company more efficiently," she said. "We're going to have to be quicker and faster to compete in this new world order."

When an analyst asked Whitman about morale in the face of job cuts, she said people at HP understand the economic climate.

"No company likes to decrease the workforce," she said. "Our employees live with it every single day. Everyone understands the turnaround we're in, everyone understands the market realities. I don't think anyone likes this." HP believes the extra job cuts will free up an additional $1 billion a year, "and some of that will be reinvested back into the business."

There's also money being spent in R&D. At first during the Q&A session, the CFO Lesjack said that "the increase of R&D year over year is very broad-based" across many product lines. Whitman immediately added that there have been increases for R&D in HP's server lines. The servers which HP is able to sell are "mission-critical x86" systems. That's represents another report that the Integrity-based lineup continues to decline. BCS overall represents just 3 percent of all Enterprise Systems sales in this quarter.

HP's internal enterprise systems -- which were once managed by HP 3000s -- are in the process of a new round of replacements. SAP replaced internal systems at HP last decade. Whitman said the churn that started in 2001 with the Compaq purchase has put the vendor through significant changes, ones that HP must manage better.

"This company has been through a lot," Whitman said during analyst questioning. "The acquisition of Compaq. The acquisition of EDS. Eleven to 20 software acquisitions. It's a lot of change. We're putting in new ERP programs and technology to automate processes that frankly, have not been done in awhile."

07:16 PM in Migration, News Outta HP | Permalink | Comments (0)

May 21, 2014

Ops check: does a replacement application do the same caliber of power fail recovery?

Migrating away from an HP 3000 application means leaving behind some things you can replace. One example is robust scheduling and job management. You can get that under Windows, if your target application will run on that Microsoft OS. It's extra, but worth it, especially if the app you need to replace generates a great many jobs. We've heard of one that used 14,000.

Disk-bandaidA migrating site will also want to be sure about error recovery in case of a system failure. Looking at what's a given in the 3000 world is the bottom-rung bar to check on a new platform. This might not be an issue that app users care about -- until a brown-out takes down a server that doesn't have robust recovery. One HP 3000 system manager summed up the operations he needs to replace on HP's 3000 application server.

We're looking at recovery aspects if power is lost, or those that kick in whenever MPE crashes. On the 3000's critical applications, we can use DBCONTROL or FCONTROL to complete the I/O.  Another option would be to store down the datasets before the batch process takes place.

A couple of decades ago, this was a feature where the 3000's IMAGE database stood out in a startling, visual way. A database shootout in New Jersey pitted IMAGE and MPE against Unix and Oracle, or second-level entries such as Sybase or Informix. A tug on the power plug of the 3000 while it was processing data left the server in a no-data-loss state, when it could be rebooted. Not so much, way back then, for what we'd call today's replacement system databases.

Eloquence, the IMAGE workalike database, emulates this rock-solid recovery for any Windows or Linux applications that use that Marxmeier product. Whatever the replacement application will be for a mission-critical 3000 system, it needs to rely on the same caliber of crash or powerfail recovery. This isn't an obvious question to ask during the feature comparison phase of migration planning. But such recovery is not automatic on every platform that will take over for MPE.

Sometimes there's powerfail tools available for replacement application hosts, system-wide tools to aid in database recovery -- but ones that managers don't employ because of costs to performance. For example, a barrier is a system feature common in the Linux world. A barrier protects the state of the filesystem journal. Here's a bit of discussion from the Stack Exchange forum, where plenty of Linux admins seek solutions.

It is possible that the write to the journal on the disk is delayed, because it's more efficient from the head position currently to write in a different order to the one the operating system requested as the actual order -- meaning blocks can be committed before the journal is.

The way to resolve this is to make the operating system explicitly wait for the journal to have been committed before committing any more writes. This is known as a barrier. Most filesystems do not use this by default and would explicitly need enabling with a mount option.

mount -o barrier=1 /dev/sda /mntpnt

The big downside to barriers is they have a tendency to slow IO down, sometimes dramatically (around 30 percent) which is why they aren't enabled by default.

In the 3000 world, logging has been used as a similar recovery feature, focused on recovering IMAGE data. A long-running debate included concerns about whether logging penalized application performance. We've run a logging article written by Robelle's Bob Green that's worth a look.

Peering under the covers of any replacement application, to see the means to recover its data, is a best practice. Even if a manager doesn't have deep knowledge of the target environment, this peering is the kind of thing the typical experienced 3000 manager will embrace without question. Then they'll ask the powerfail recovery question.

07:53 PM in Migration | Permalink | Comments (0)

May 20, 2014

Who's SUSAN, and what's her CPUNAME?

The MPE operating system, first booted for genuine use some 40 years ago, is a most unique creature of the computer ecosystem. This is software that does not have its own license, specifically. According to HP, the ownership of any MPE/iX version is determined by owning an Hewlett-Packard 3000 server, one built to boot up MPE/iX.

SusanWe reached out for clarity about this when a very large aircraft maker tipped us off -- once again, it will examine replacing HP's 3000 iron with CHARON licenses on Intel systems. After the MPE/iX software is turned off on any replaced 3000 hardware, does its hardware-based license then expire? The operating system license, according to HP's MPE Technical Consultant Cathlene Mc Rae, is related to the HPSUSAN of the original HP hardware.

So wait a minute. Are these HPSUSAN numbers of 3000s considered de-licensed, even if they're going to be used on the CHARON emulator? Mc Rae explained.

The HPSUSAN number is different from the MPE/iX license, although there is a relation between the two. The ability to use MPE/iX on the emulator is a result of completing a Software License Transfer. The original MPE/iX license on the HP e3000 would then no longer exist. 

In the hardware world of HP 3000s, HPSUSAN takes the original serial and model numbers on the system. It remains the same, as long as the customer owns the system. This combination was used to ID the hardware and enable diagnostics for the correct system.

However, that transferred license for the MPE/iX installation on the CHARON emulator -- available via a $432 Software License Transfer Fee -- won't be getting a new HPSUSAN number during the process. HPSUSAN gets re-used, and so it leads us to see what HPSUSAN stands for, and how the HPCPUNAME is a key in emulator installations.

The U in HPSUSAN stands for Unique, as in System Unique Serially Assigned Number. Mc Rae said that HPSUSAN is one of a kind for HP-built 3000 systems. But SUSAN doesn't designate an MPE/iX license, even though MPE is licensed via hardware ownership. 

Mc Rae explained to us, and to the CHARON prospective user, "MPE hardware and software was created before the technology of  virtual systems and emulators, in the 1970s. Licenses were based on hardware ownership."

This sounds familiar. HP once compared the licensing of MPE/iX to license plates issued for a car. They could not be separated, these numbers and the car that was the HP 3000 iron. (Let's just put aside the common practice of those metal-plate days, when they'd give you a new number after your plate was older than 8 years in Texas.)

In 1999, HP was busy suing Hardware House and a few other resellers over the resellers' separation of HPSUSANs from HP's 3000 hardware cars. The House was taking other PA-RISC servers and pressing valid HPSUSAN numbers onto the non-3000 iron. People went to jail. Lo-jacks were ordered for ankles.

Thanks to the passage of 15 years' time, an HPSUSAN number can now move to a USB thumb drive plugged into a CHARON Intel- or AMD-based server. Those license plates can travel to a newer model of car. The emulator's HPCPUNAME, however, can only be designated as an A-Class or N-Class system, according to HP's knowledge. That'll likely be a reason to contact all software vendors whose products operate on the replaced HP 3000 iron.

You see, vendors use a combo of HPSUSAN and HPCPUNAME to control licensing. Products such as Infor's MANMAN or PowerHouse not only want to read HPSUSAN -- which you can move to CHARON -- but also HPCPUNAME. If you're moving off a Series 979, for example, "979-100" isn't an emulated system under CHARON. No 979-100 for HPCPUNAME. You've got to get license permission from your software vendors to enable an A-Class or N-Class HPCPUNAME.

The HPCPUNAME on the CHARON system may not be set to 979, Mc Rae said. "Based on the CHARON HPA/3000 family, it is assumed that the HPCPUNAME will be set to an A-Class or N-Class CPUNAME," she said. "For example: HPCPUNAME = SERIES e3000/A500-200-50. As far as I know, CHARON can only emulate A- and N-Class systems." That's true: a Series 9xx model isn't on the HPA/3000 product list.

The silver lining in this cloud is that you're only doing this contacting and CPUNAME-changing once. Moving to an A-Class or faster CPU from a 9x9 system is the last time you'll be changing from an unsupported CPUNAME to something included in the CHARON product line. 

In short, independent software vendors are going to have to be contacted, if they've licensed their products with the HPCPUNAME-HPSUSAN combo on a Series 9xx. Contacting your software vendors about a system upgrade is a fair business practice. But it's more than the right thing to do. Series 9xx users headed to the emulator look like they need that refresh to boot up their indie software.

07:06 PM in History, Homesteading, News Outta HP | Permalink | Comments (0)

May 19, 2014

PowerHouse users launch enhancement run

Years ago, the Interex users group for HP 3000 managers and owners provided a way to make MPE better. There wasn’t much that HP was willing to do to re-engineer its hardware servers — not working off the requests of customers. But ah, the operating system and its allied software subsystems were always open for system enhancement requests. They called it a System Improvement Ballot, and every year had an SIB.

In their day, these were much awaited missives from lovers of MPE to the heart of the OS, the HP labs. They were ranked and debated. The collection of a Gang of Six such requests made up the mission statement for OpenMPE from the first year of that group’s existence. When the labs went dark and that list was frozen, there was little hope of anything thawing the development stream.

That’s what makes the PowerHouse community so novel. After years of nothing new in the product line, the new owners have opened the doors to enhancement requests. The discussion of who’s going to manage the enhancement requests started bubbling up at the LinkedIn Cognos PowerHouse group. It tells a good deal about how slowly things were flowing at the time by looking at the name of that group. Cognos hasn’t been the owner of PowerHouse since 2009. Now that IBM has sold off the products and customer base, Unicom Global is using an established representative to build a wish list.

Bob Deskin has taken the discussion of enhancements onto the Powerhouse-L mailing list. If you're watchful about how much email fills your inbox, you can simply keep track of the list's archives without subscribing. Customers are giving the new PowerHouse management fresh improvement requests using that list.

There’s a lot of catching up and improvement to do. As one example, Fatal Errors of the software were “never documented in the manuals,” according to Bob Deskin, formerly the Cognos/IBM voice of PowerHouse products to the customer base. 

"More often than not," Deskin said of the Fatal Errors, “they simply represent something that should not have happened. And the most common cause for that was something else that happened but shouldn’t have that ended up causing the Fatal Error. That’s why many of them are so hard to trace.”

Details of what could be brought up to date in PowerHouse, shared to that mailing list this week, are going into deep specifics. But that’s what you expect from the creators of software. Deskin’s encouraging transparency.

As you can imagine, the UNICOM PowerHouse team is still in transition. That said, they are looking to the future and although they and I have some ideas, we’d like to hear yours big or small. You can post them here for everyone to see. That way everyone gets to see them and expand on them.

There’s no guarantee about how many of these requests will ever be acted upon. Or even which versions of PowerHouse products (MPE, or VMS, or AS/400) are eligible for wishes. But Bob Deskin, consulting with Unicom to moderate a dialogue with users, suggests that everybody with a PowerHouse request chip in, right out in public.

A colleague of Deskin’s, one who’d worked with him both at Cognos and then later at IBM, offered this testimonial to Deskin being the right fellow to listen at this moment. Matt Ohmes said nobody’s a better match for this role -- a pivot point for PowerHouse, happening at Unicom.

I’ve worked with Cognos, then IBM for 31 years, many of the early years especially using PowerHouse and gained quite a reputation myself. And I would like to say that there is not another person — literally — on earth who knows more, or is better qualified to answer questions about PowerHouse than Bob Deskin.

10:11 PM in History, Homesteading | Permalink | Comments (1)

May 16, 2014

Unicom returns PowerHouse expert to fold

CognosibmlogoBob Deskin fielded questions about the PowerHouse products for more than a decade on the PowerHouse-L mailing list. When a question from the vendor -- for many of those years, Cognos -- was required, Deskin did the answering. He was not able to speak for IBM in a formal capacity about the software. But he defined the scope of product performance, as well as soothed the concerns from a customer base when it felt abandoned.

UnicomgloballogoAfter retiring from IBM's PowerHouse ADT unit last year, Deskin's back in the field where he's best known. The new owners of the PowerHouse tools, Unicom Global, added him to the team in a consultant's capacity.

As part of UNICOM's commitment to the PowerHouse suite of products, I have been brought on board as a consultant to work with the UNICOM PowerHouse team to enhance the support and product direction efforts.

For anyone not familiar with my background, I started in this business in the early '70s as a programmer and systems analyst. I joined Cognos (then Quasar) in 1981 after evaluating QUIZ and beta testing QUICK for a large multinational. Over the years, I’ve been in customer support, technical liaison, quality control, education, documentation, and various advisory roles. For the past 12 years, until my retirement from IBM in 2013, I was the Product Manager for PowerHouse, PowerHouse Web, and Axiant.

New owners of classic MPE tools, like PowerHouse, are not always so savvy about keeping the tribal knowledge in place. In Deskin's case, he's been retained to know about the OpenVMS product owners' issues, as well as those from the 3000 community. There's another HP platform that's still available for PowerHouse customer, too.

The story behind the story of new PowerHouse ownership is a plan for enhancing the products, even as a site's sideways migration to a supported HP platform might be underway. In order to help retain a customer in the proprietary PowerHouse community, the new owners know they need to improve the products' capabilities

WainhomesballoonIn one example of such a sideways migration -- where PowerHouse remains the constant platform element while everything else changes -- Paul Stennett of house builder Wainhomes in the UK reported that an MPE-to-UX move began with a change in database. The target was not Oracle, either.

We actually migrated to Eloquence rather than Oracle, which meant the data conversion was pretty simple -- Eloquence emulates IMAGE on the HP 3000. The only issue was KSAM files which we couldn't migrate. However, Eloquence has a better solution, and allows third party indexes, and therefore generic key retrieval of data. For instance, surname = "Smi@" etc... Testing was around 3 months.

Our HP 3000 applications go back over 20 years and have been continually developed over time. I have had experience with other [replacement] packages for the housebuilding industry, in particular COINS, However, with the mission to keep the re-training and disruption to the business to a minimum, the migration option was the best route for us. 

I completely agree that you can gain major benefits from replacing a system [completely instead of migrating it.] I guess it depends on what type of business you are in. If you are an online retailer, for example, then technology can save costs and improve efficiency. As they say, it's horses for courses.

03:39 PM in Migration | Permalink | Comments (0)

May 15, 2014

Techniques for file copying, compressions

I need to submit a file to from an HP 3000 to my credit card processor, a file that is an 80-byte file. Before I submit it, I need to zip the file. I’m using the Posix shell and its zip program. I SFTP’d the file, but my vendor is not processing the file because it is supposedly 96 bytes long. If I unzip the file that I zipped, it becomes a bytestream file. I then check — by doing an FCOPY FROM=MYFILE;TO=;HEX;CHAR — and I see that no record exceeds 80 bytes. Why do they think it is an 96-byte file?

Barry Lake of Allegro replies

I would convert it to a bytestream file before zipping it 

:tobyte.hpbin.sys "-at /SG2VER/PUB/LCAUTHOT /SOME/NEW/FILE"

Mark Ranft adds

I would try copying the file to an intermediate server. Zip it. And SFTP it. See if that provides better results.

Tony Summers suggests there is good background instruction, to understand how MPE/iX files are different than those in Unix, at Robelle's MPE for Unix Users article.

I thought there was an option to FCOPY part of a record. If the record contains TODAY IS MONDAY and you want only columns 10-12, I thought there was an FCOPY subset option-- one that would result in just the characters in those positions (MON). Am I halucinating?

Francois Desrochiers replies

The SUBSET option is used to select records by record numbers or strings in certain columns, but you cannot select parts of records. It always works on complete records. You have to use other tools such as Suprtool, Qedit, Editor, Quad, or the Posix "cut" to extract columns.

Olav Kappert adds

You can also pipe the record into a variable and the parse whatever you want out.

12:53 PM in Hidden Value, Homesteading | Permalink | Comments (0)

May 14, 2014

TTerm Pro's latest version uses NS/VT


As we reported yesterday, the TTerm Pro app for HP 3000 emulation got an enhancement this month, one that makes the software very unique. NS/VT protocol support isn't exactly rocket science, but its not straightforward, either. The history of the 3000 is strewn with terminal emulator makers who didn't get this aspect all figured out.

Our ally Jon Diercks, who's the author of The MPE/iX System Administrator Handbook, updated his iPad app and gave the new 1.1.0 version a test. The short report: NS/VT seems to work, at first glance. Diercks added a second test to the first one of the app. He connected his iPad to the HPA202 freeware version of CHARON. With his exam, an HP 3000 terminal emulator was talking with an emulated HP 3000. He offered the screen shot above as proof.

Well, the 30-second report is ... it works! I fired up Charon, copied my previous TTerm telnet profile and changed to NS/VT, and the logon prompt came right up. The :SHOWVAR command above proves that NS/VT protocol is in use. I also launched NMMGR just to verify block mode still looks okay. I might play with it more later, but that's enough to satisfy my curiosity for now.

It's a marvel to consider how MPE has been carried into the future with this combination. The iOS operating system on the iPad is certain to have a longer life where it's improved than the alternatives based on desktops. By that, I mean I believe iOS has "got legs," as the saying goes among theatre people when they talk about a long-running show. You don't need a PC and Windows any more to emulate a 3000 terminal.

And with CHARON, you don't need the 3000 hardware anymore, either. All that's left is MPE and IMAGE, the bedrock of what we know as the 3000 experience.

12:03 PM in Homesteading, Newsmakers | Permalink | Comments (0)

May 13, 2014

iPad 3000 terminal emulator gains NS/VT

The only tablet-ready terminal emulator for HP 3000 users has crossed over even further into the language of MPE. The 1.1.0 version of TTerm Pro adds HP's 3000-specific Network Services/Virtual Terminal protocol. The new feature means that many more MPE applications will run without a flaw over the Apple iPad tablets.

TTerm Pro portrait view TelnetTo be exact, the latest version of TTerm Pro will run under iOS7, so it's possible that some other Apple mobile product could link up this app with a 3000. But a tablet is pretty much the minimum screen real estate for a terminal emulator. Jon Diercks, who tested the previous version of TTerm Pro, said in his review that an external keyboard connected via Bluetooth eased the use of tablet-based terminal emulation. But the screen capture at left -- collected back when TTerm Pro only did Telnet links -- shows you can even get a soft keyboard, plus function keys, onto an iPad's screen.

Turbosoft, which released a 3000-ready version of the iPad app last year, has lowered the price of TTerm Pro by 50 percent. It now sells for $24.95. Any 3000 managers who purchased the app last year can update it -- with its new 3000-savvy -- for free. NS/VT could be worth a lot more for any company that wants to preserve a 3000 application's capability to go mobile.

The earlier version of TTerm Pro supported only Telnet connectivity, which meant that the longest-standing 3000 apps would not run in the iPad-based emulator. Mind you, this is not an emulator of the base 3000 PA-RISC processor, a la CHARON. This iPad app emulates HP's proprietary terminals for the 3000, specifically the HP 700/92 series.

The MPE applications which were tuned the finest for 3000 users relied upon NS/VT protocols. The protocol was developed by HP as an emulation itself: NS/VT gave users on Local Area Networks the same kind of performance and reliability only available through an ATP card inside a 3000. By using NS/VT, an application didn't require that a 3000 have that card.

AICS Research developed a QCTerm emulator during the late 1990s which relied upon Telnet for its network protocols. But AICS founder Wirt Atmar knew very well how much advantage NS/VT held over Telnet. Full-duplex is being emulated via NS/VT, and that ensures the delivery of data.

Full-duplex has traditionally been by far and away the preferred protocol for communication with a host computer, because you have this very strong reassurance that the host did indeed receive the character. The host's retransmission of the character back to you is an explicit verification that it saw and absorbed the character you just typed.

NS/VT is an HP-proprietary client/server protocol — but it is also nothing more than a simple and obvious extension of the design philosophy that had begun with the ATP card, where a remote processor transmits a line of text to the HP3000's CPU only when that line of text is complete.

Two ways are commonly available today to use NS/VT-like services. One requires the use of a DTC (data terminal controller), the other a terminal emulator. In both, the function of the original ATP card is being faithfully recreated. When you serially communicate with a DTC, a processor located on the DTC's serial card is absorbing every character you type and echoing it back to you. Only when a termination character is typed, or the line buffer is full, or a time-out occurs, is your line of text transmitted to the HP3000 as a single packet of information via the LAN that connects the DTC to the HP3000.

An NS/VT-based terminal emulator is maintaining essentially full-duplex communication with the DTC serial card — or your PC's memory. Every character you type under NS/VT is immediately echoed back to your screen. Only when you strike the carriage return (or the enter key) is your line of text transmitted over the LAN to the HP 3000. "Turn-around times" are so quick on a LAN (if it's not too busy) that you don't tend to notice the nature of one-way communication inherent to a LAN.

According to the ubiquitous management manual The MPE/iX System Administrator's Handbook, NS/VT is a better choice for applications on 3000s. "It usually yields the best overall results, because it is optimized for the way most MPE applications work," the book states in its Getting Connected section. NS/VT is enough of an emulation specialty that Attachmate offers the WRQ-developed Reflection as a separate Reflection HP product. The chief difference between the rest of the Reflection line is that NS/VT is included in Reflection HP. There's an uplift in price for this capability.

TTerm Pro includes NS/VT along with Telnet protocol. It's pretty obvious a company isn't going to replace all of its terminal emulator desktops and laptops with iPads. But it's a real help to know the protocol optimized for the HP 3000 now has a way to run on mobile tablets. Consider that previous sentence for a moment. Then decide how often technology continues to flow back to the world of MPE. Instead of $249 a seat, terminal emulation now costs $24.95. And upgrades are free if you're using an Apple tablet.

07:11 PM in Homesteading, Newsmakers | Permalink | Comments (0)

May 12, 2014

3000 world loses dauntless Dunlop carrier

Hp3000linksDunlop Tires are a brand from England known for their breakthrough as tires which bore their weight on air. The pneumatic tire was crafted by John Dunlop to prevent headaches for bicycle riders. All tires to that point -- the British call them tyres -- used solid rubber instead of inflated designs. The 3000 and MPE community had its own Dunlop for decades: John Dunlop, founder of the headache-busting HP3000links.com website. Dunlop is an HP 3000 pro of more than 30 years standing, and more than 20 of it he spent posting to and reading the wisdom on the 3000-L mailing list. Last week, Dunlop reported he's moving out of the world of the 3000, since his server at work has been decommissioned.

Yesterday I turned off the HP3000 918 for the final time. It became surplus to requirements, finally.

It had been humming away quite happily for the last several years without much in the way of maintenance, and it did what it does best, being one of the best and most reliable online transaction processors ever built. For durability and reliability, it was without peer.

A rather sad event seeing as I have been working on HP3000's for the last 30-plus years, although very little in the last year or so.

Dunlop has only retired his HP 3000 career, and retains his life as an IT pro. But for more than a good decade of his 30-plus years in the community, he carried vital links to 3000 information and technique from his labor-of-love website. HP3000links.com pumped up the skill level of MPE owners and managers. Dunlop dedicated his career to the 3000 in other ways as well.

In the middle of the prior decade, Dunlop served as the webmaster for the OpenMPE advocacy group. This was a time when that group was proposing paid membership. That website would have been essential to providing service to paid members. In the middle of the last decade, openmpe.com still had one of the most extensive lists of 3000 owners. Even without the paid membership, Dunlop was posting meeting minutes on the site during a period when there was close scrutiny of OpenMPE developments.

His own site was a lively circus-page of links to technical papers as well as a gateway to many 3000 websites of the past decade. "In spite of all the discussion about dwindling HP 3000 resources, the links I have pulled together and maintained are still available," Dunlop said in 2006, "and demonstrate that there is still a lot out there for the HP 3000 user." Dunlop acted as an editor while he maintained the site for more than a decade. 

Four years ago he wondered why HP wanted to hang on to control of the 3000's configuration software after the vendor left the MPE market.

"There is software out there which will change HPSUSAN numbers," he said. "Surely HP would not be interested in chasing up anyone who used this software now, seeing as they have lost all interest in the HP3000?" Told that HP had just restated its forever-more control of SS_UPDATE -- the only 3000 support it will do on the record -- Dunlop replied, "I can't see why HP wants to retain control of this still, unless it's to try and milk a few more dollars out of the HP 3000 community."

The loss of a community member who knew the 3000 from the 1970s can feel like a death in the family, even though that person remains very much alive. The demise of HP3000links.com is very real, of course. Original material that it referenced is still alive on the web, in many cases. We did a survey of its vendor list during 2012, simply amazed at that time that it could have been so comprehensive.

Dunlop was signing off of 3000-L with his report. For years he's shared his wisdom while managing systems at Polimeri Europa UK Ltd. The company manufactures, among other things, synthetic rubber.

For many years, I have been mostly a lurker on this list but have benefitted greatly from the massed HP3000 knowledge so amptly demonstrated by the other members of this list.  To all the contributors, much thanks for all your help over the years.

So, sadly, this will be my final post to the list as I will no longer be seeking help in HP3000-land. To all, best of luck. Cheerio! 

One of his most notable contributions we could find in the 18-plus years of Newswire archives appeared in a 1999 article describing Posix startups under MPE/iX. The namespace for MPE which behaves most like Unix didn't always work properly on older systems:

Some sites are completely missing all of the HFS files (this is usually caused by an “incorrect” reload). From the MPE CI, try :LISTFILE /bin/. If no files are found, you will need to restore them from the FOS tape.

1. Restore the following from the tape:

restore *t;@.hp36431.support,i0036431.usl.sys;create;show

2. :STREAM I0036431.USL.SYS

3. After I0036431 finishes,


All of the HP-supplied HFS files will be restored, and the directory structure and permissions set to the defaults.

Note: if you just want to restore all HFS files on a backup tape, try “:RESTORE /-@.@.@;SHOW;KEEP;OLDDATE;CREATE”.

07:58 PM in Homesteading, Web Resources | Permalink | Comments (0)

May 09, 2014

HP bets "Hey! You'll, get onto your cloud!"

TalltorideHewlett-Packard announced that it will spend $1 billion over the next two years to help its customers build private cloud computing. Private clouds will need security, and they'll begin to behave more like the HP 3000 world everybody knows: management of internal resources. The difference will reside in a standard open source stack, OpenStack. It's not aimed at midsize or smaller firms. But aiding OpenStack might help open some minds about why clouds can be simple to build, as well as feature-rich.

This is an idea that still needs to lift off. Among the 3000 managers we talk with — who've been in computing since the 1980s — they are inclined to think of clouds much differently than time-sharing, or apps over the Internet. Clouds are still things in Rolling Stones or Judy Collins choruses.

The 3000 community that's moving still isn't embracing any ideal of running clouds in a serious way. Once vendor who's teeing up cloud computing as the next big hit is Kenandy. That's the company built around the IT experience and expertise of the creators of MANMAN. They've called their software social ERP, in part because it embraces the information exchange that happens on that social network level.

But from the viewpoint of Terry Floyd, founder of the manufacturing services firm The Support Group, Kenandy's still waiting for somebody from the 3000 world to hit that teed-up ball. Kenandy was on hand at the Computer History Museum for the last HP3000 Reunion. That gathering of companies now looks like the wrong size of ball to hit the Kenandy cloud ERP ball.

"Since we saw them at the Computer History Museum meeting, Kenandy seems to have has re-focused on large Fortune 1000 companies," Floyd said. There are scores of HP 3000 sites running MANMAN. But very few are measuring up as F1000 enterprises. Kenandy looks like it believes the typical 3000 site is not big enough to benefit from riding a cloud. There are many migrated companies who'd fit into that Fortune 1000 field. But then, they've already chosen their replacements.

The Kenandy solution relies on the force.com private cloud, operated by Salesforce.com. Smaller companies, the size of 3000 customers, use Salesforce. The vendor's got a force.com cloud for apps beyond CRM. But the magnitude of the commitment to Kenandy seems larger than the size of the remaining 3000 sites which manufacture using Infor's MANMAN app.

"Most MANMAN sites don't meet their size requirements," Floyd said. "I have a site that wants to consider Kenandy next year, but so far Kenandy is not very interested. We'll see if they are serious when the project kicks off next year, because we think Kenandy is a good fit for them."

The longer that small companies wait out such cloud developments as HP's $500 million per year, the better the value becomes for getting onto their cloud, migrating datacenter ops outside company walls. HP is investing to convince companies to build their own private clouds, instead of renting software from firms like Kenandy and Salesforce. Floyd and his company have said there's good value in switching to cloud-based ERP for some customers. Customization of the app becomes the most expensive issue.

This is the central decision in migrating to cloud-based ERP from a 3000. It's more important than how much the hardware to support the cloud will cost. HP's teaming up with Foxconn -- insert snarky joke here -- to drive down the expense of putting up cloud-optimized servers. But that venture is aimed at telecommunications companies and Internet service providers. When Comcast and Verizon, or Orange in Europe, are your targets, you know there's a size requirement.

You might think of the requirements for this sort of cloud -- something a customer would need to devote intense administrative resources to -- as that sign at the front of the best amusement park rides. "You must be Fortune 1000 tall to ride this ride," it might say. Maybe, over the period of HP's new cloud push, the number on the sign will get smaller.

10:04 AM in Migration, News Outta HP | Permalink | Comments (0)

May 08, 2014

A pretty fine book for MPE's after (HP) life

How could a vendor suggest that a widely-installed and mission-critical product be turned off? Have a look at what Microsoft is doing this year. The advice has been to turn Windows XP off, replace what's working. HP 3000 users got the same advisory in 2001.

FourelephantsandturtleThat was a momentous year for MPE users, but the year that followed contained the same confusion from the vendor that Microsoft is facing now. I noticed this as I dug into Jon Diercks' MPE/iX System Administration Handbook. It carries fine information, an opinion I expressed in our recent mini-lesson about BULDACCT and some automatic security that it provides. As I did my digging I found a stale message inside the book, but it wasn't one that Diercks created.

You might believe that nobody could apparently see what was about to happen to HP's 3000 business, considering what appears on pages xxi through xxiii. It's a foreword from the General Manager of HP's Commercial Systems Division, Winston Prather. A book that was released in 2002 -- yeah, months beyond that 2001 exit notice -- includes this advice about ownership.

Today, with technologies like Samba, Java, GUIs, our WebWise products and our partners, the HP e3000 still provides a great environment for the creation and support of new object-oriented, web-based applications, as well as e-service and e-commerce environments.

The book's readers absorbed that message for years after HP insisted that Prather was wrong. Or to be accurate, when Prather took pains to tell his customers the 3000 was not a great environment for any of the above tasks. It was probably as confusing as what Microsoft's done this month by releasing an XP security patch after it insisted it would not. Some writers believe that patch should not have been released. That's the kind of sentiment I continue to hear about HP twice-delaying its 3000 exit.

Follow the link from the top of yesterday's story and you'll find a writer who thinks "its a huge mistake" that Internet Explorer will not suffer from this month's zero-day exploit, even if the browser runs in XP.

IT admins, faced with the harsh reality of finally having to upgrade to a modern operating system, will sleep well knowing that Microsoft is a pushover and will continue to support XP while it has a significant number of users. The status quo is preserved.

Except it's not preserved, not any more than MPE/iX status was preserved during a pair of HP's two-year extensions. It's just that many companies -- perhaps the same percentage as the 3000 owners of 2002 -- find it beyond their budgets and resources to dump XP. And so even today, Diercks' book has value a-plenty to any company that still finds MPE to be the best tool for their circumstance. The book's not perfect, and Diercks always knew that was so, citing the rule all authors live by: omissions and errors will be in every creation. You must let your work go, however, learning from the creation and promising yourself to Do Better Next Time.

Put another way, perfect is the enemy of good. MPE was never perfect. If it were, than a mighty fine product like MPEX, an eXtension of MPE, would never have gained its candidacy as one of the elephants for the 3000 owner, developer, or administrator. Elephants, supported by a turtle.

Turtle? Elephants? And this has what exactly to do with MPE?

In a Hindu legend, the world is supported by four elephants, and those elephants ride on the back of a turtle. (It's a legend, so you'll have to take my word for this. Look at the picture above and you'll see four, with the fourth one tucked just behind the first.) But in this model of the world, MPE is that turtle. A few key independent software vendors are those elephants. You can decide for yourself who they might be, those elephants holding up the world -- which is the 3000 community.

But none of the elephants are mentioned in that fine MPE book. Diercks took care to note that he'd mention nobody's software except HP's in that book, no matter how fine the vendor's software performed. We've been in this situation, where including anyone just ensures that someone who's overlooked would be upset. You handle this situation a lot like Diercks did in his prologue: We know there are good products out there, but he wanted "to avoid being accused of unfairly representing (or failing to represent) any vendor or product."

One way of looking at the 3000's legend is to consider that MPE (and its database IMAGE), comprise the turtle in the Hindu story. Without MPE, there would be no Suprtool, no Adager, no MPEX. These might be considered elephants, and I'll leave you to fill in that fourth pachyderm. (If you're still with me, nominate a fourth elephant; that's going to be fun to share.)

ForewordWhile you're pondering all this, don't forget that Diercks' book is also something of a time machine. I mean that it contains a snapshot of the full faith of Hewlett-Packard in the 3000's MPE, right down to the hp.com/hpbooks webpage (now defunct) and HP logo on the title page Then there's Prather's best guess about the 3000's future, although it really was not written as a guess. See the language in the excerpt above; click for details.

Like Prather said in that foreword, things in the IT industry can change fast. Much faster than a printed book can be manufactured and released, with essential edits and reviews and distribution. If the publisher Prentice Hall had finished that book any later (because a publisher controls the birthdate of a book, not the writer) the MPE/iX System Administration Handbook might not even exist. That would be a great loss. At the least, this fine book wouldn't take us all back in time to HP's confusion about the 3000, confusion that mirrors Microsoft's of today.

08:42 PM in History, Homesteading | Permalink | Comments (0)

May 07, 2014

MPE automates (some) password security

IE-hackIt only took a matter of weeks to create an unpatched security threat to the world's single-most installed vendor operating system, Windows XP. At about a 30 percent penetration of all PCs, XP is still running on hundreds of millions of systems. A zero-day Internet Explorer bug got patched this month, however, reluctantly by Microsoft. Once it cut its software loose -- just like HP stopped all MPE patches at the end of 2008 -- Microsoft's XP became vulnerable in just 20 days.

MPE, on the other hand, makes a backup file of its account structure that will defy an attempt to steal its critical contents. HP 3000 users can count on the work of an anonymous developer of MPE, even more than five years after patch creation ceased.

The automated protection of MPE's passwords comes through jobstreams from a key backup program. These files, created by using the BULDACCT program, are jobstreams that can only be read by 3000 users with CR (the jobstream's CReator, who might be an operator) or SM (System Manager) privileges, according to Jon Diercks' MPE/iX System Administration Handbook. Diercks advises his readers, "Even if your backup software stores the system directory, you may want to use BULDACCT as an extra precaution, in case any problems interfere with your ability to restore the directory data normally." However, he adds, the BULDJOB files are powerful enough to warrant extra care. After all, they contain "every password for every user, group and account, and lockwords for UDC files where necessary."

Note: the jobstream files you build on your own -- not these BULDJOBs -- can be secured on your own. But you must do that explicitly. These user-created streams' protection is not automatic.

In any case, you should use BULDACCT every day, according to Vesoft's Vladimir Volokh, not just as an optional extra precaution. "Do it before -- well, before it happens," he says. What can happen is a messy manure of a failure of an LDEV, one that scrambles the system directory. 

Put the BULDACCT option into your backup's stream file, so its jobstreams are created before your backups. Daily backups, of course. You're doing daily backups, right? And then storing that tape someplace other than the top of the HP 3000. You'd be surprised, said Volokh, how many 3000 sites use that storage location for a backup tape.

The BULDACCT option includes the jobstreams in the backup tape. After your backup is complete, you should PURGE these two streams from your 3000's disk.

Those BULDACCT jobstreams (BULDJOB1 and BULDJOB2) are automatically secured at the file level. This protects BULDACCT streams  from hackers' pry-bars, a very good thing -- because this stream contains all system information including passwords.

You can then RESTORE these streams if you still have a disk error that leaves files intact, but ruins the directory structure. BULDJOB1 contains the instructions to rebuild directory structure, a job that runs before you RESTORE files. BULDJOB2 contains the SETCATALOG commands needed for to reassign all user, account and system UDCs, according to Diercks' fine book. Still available, by the way, online via O'Reilly's Safari e-book service.

Volokh says that if any of the above still seems unclear, 3000 managers can call him at Vesoft and he'll walk managers through the process. "For details, just call us. Don't chase the horse after the barn door has been opened."

09:41 PM in Hidden Value, Homesteading | Permalink | Comments (0)

May 06, 2014

PowerHouse users study migration flights

737-lineA sometimes surprising group of companies continue to use software from the PowerHouse fourth generation language lineup on their HP 3000s. At Boeing, for example -- a manufacturer whose Boeing 737 assembly line pushes out one aircraft's airframe every day -- the products are essential to one mission-critical application. Upgrade fees for PowerHouse became a crucial element in deciding whether to homestead on the CHARON emulator last year.

PowerHouse products have a stickiness to them that can surprise, here in 2014, because of the age of the underlying concept. But they're ingrained in IT operations to a degree that can make them linchpins. In a LinkedIn Group devoted to managing PowerHouse products, the topic of making a new era for 4GL has been discussed for the past week. Paul Stennett, a group systems manager with UK-based housebuilder Wainhomes, said that his company's transition to an HP-UX version of PowerHouse has worked more seamlessly -- so far -- than the prospect of replacing the PowerHouse MPE application with a package. 

"The main driver was not to disrupt the business, which at the end of the day pays for IT," Stennett said. "It did take around 18 months to complete, but was implemented over a weekend. So the users logged off on Friday on the old system, and logged onto the new system on Monday. From an application point of view all the screens, reports and processes were the same."

This is the lift-and-shift migration strategy, taken to a new level because the proprietary language driving these applications has not changed. Business processes -- which will get reviewed in any thorough migration to see if they're still needed -- have the highest level of pain to change. Sometimes companies conclude that the enhancements derived from a replacement package are more than offset by required changes to business processes.

Enter the version of PowerHouse that runs on HP's supported Unix environment. It was a realistic choice for Stennett's company because the 4GL has a new owner this year in Unicom.

"With the acquisition of PowerHouse by UNICOM, and their commitment to developing the product and therefore continued support," Stennett posted on the LinkedIn group, "is it better to migrate PowerHouse onto a supported platform (from HP 3000 to HP-UX) rather than go for a complete re-write in Java, with all of its risks. To the user was seamless, other than they have to use a different command to logon. The impact to the businesses day to day running was zero."

The discussion began with requests on information for porting PowerHouse apps to Java. The 4GL was created with a different goal in mind than Java's ideal of "write once, run anywhere." Productivity was the lure for users who moved to 4GLs such as PowerHouse, Speedware, and variants such as Protos and Transact. All but Protos now have support for other platforms.

And HP's venerated Transact -- which once ran the US Navy's Mark 85 torpedo facility at Keyport, Wash. -- can be replaced by ScreenJet's TransAction and then implemented on MPE. ScreenJet, which partnered with Transact's creator David Dummer to build this replacement, added that an MPE/iX TransAction implementation would work as a testing step toward an ultimate migration to other environments.

Bob Deskin, a former PowerHouse support manager who retired from IBM last year, sketched out why the fourth generation language is preserving so many in-house applications -- sometimes on platforms where the vendor has moved on, or set an exit date as with HP's OpenVMS.

Application systems, like many things, have inertia. They tend to obey Newton's first law. A body at rest tends to remain at rest unless acted upon by an outside force. The need for change was that outside force. When an application requires major change, the decision must be made to do extensive modifications to the existing system, to write a new system, or to buy a package. During the '90s, the answer was often to buy a package. 

But packages are expensive so companies are looking at leveraging what they have. If they feel that the current 4GL application can't give them what they need, but the internal logic is still viable, they look for migration or conversion tools. Rather than completely re-write, it may be easier to convert and add on now that Java and C++ programmers are readily available.

Deskin added as part of his opinion of what happened to 4GLs that they were never ubiquitous -- not even in an environment like the HP 3000's, where development in mainstream languages might take 10 times longer during the 1970s.

There weren't enough programmers to meet the demand. Along came 4GLs and their supposed promise of development without programmers. We know that didn't work out. But the idea of generating systems in 10 percent of the time appealed to many. If you needed 10 percent of the time, maybe you only needed 10 percent of the programmers. 

The 4GL heyday was the '80s. With computers being relatively inexpensive and demand for systems growing, something had to fill the void. Some programmers caught the 4GL bug, but most didn't. There was still more demand than supply, so studying mainstream languages almost guaranteed a job. 

Now even mainstream languages like COBOL and FORTRAN are out of vogue. COBOL was even declared extinct by one misinformed business podcast on NPR. The alternatives are, as one LinkedIn group member pointed out, often Microsoft's .NET or Oracle's Java. (Java wasn't considered a vendor's product until Oracle acquired it as part of its Sun pickup. These days Java is rarely discussed without mention of its owner, perhaps because the Oracle database is so ubiquitous in typical migration target environments.)

Migration away from a 4GL like PowerHouse -- to a complete revision with new front end, back end databases, reporting and middleware -- can be costly by one LinkedIn member's accounts. Krikor Gellekian, whose name surfaces frequently in the PowerHouse community, added that a company's competitive edge is the reward for the lengthy wade through the surf of 4GL departures.

"It is not simple, it takes time and is expensive, and the client should know that in advance," Gellekian wrote. "However, I always try to persuade my clients that IT modernization is not a single project; it is a process. And adopting it means staying competitive in their business."

Deskin approached the idea that 4GLs might be a concept as extinct as that podcast's summary of COBOL. 

Does this mean that the idea of a 4GL is dead? Absolutely not. The concept of specifying what you want to do rather than how to do it is still a modern concept. In effect, object-oriented languages [like Java] are attempting to do the same thing -- except they are trying to be all things to all people and work at a very low level. However, it takes more than a language these days to be successful. It also requires a modern interface. Here's hoping.

11:32 AM in Migration, User Reports | Permalink | Comments (0)

May 05, 2014

File ID errors mean a reach for BULDACCT

My system crashed. Now when I bring it back up it starts to behave strangely, indicating several system files cannot be accessed. I can sign on, as MANAGER.SYS, but most of the accounts that used to be on the system cannot be found. When I do a LISTF of PUB.SYS, most of the files have a message associated with them that reads as follows.


I believe the system disk experienced some “difficulties” at some point, and I’m not sure what happened or if it’s repairable. Of course I have a SYSGEN tape. But never having had to use one, I need to know if it contains the SYS account files necessary for me to begin reconstruction and reloading of accounts.

Paul Courry replies:

Bad UFID is a bad Universal File IDentifier. In other words, your file system is corrupted. You can try running FSCHECK.MPEXL.TELESUP (run with extreme care, reading the FSCHECK manual first). But considering the extent of the damage you probably will not be able to recover everything.

John Clogg replies:

Files, groups, and accounts on private volume sets are still there, but you will need to recreate the system directory entries for those accounts and groups. If you have BULDACCT output, that will make the job easier. It’s always a good idea to run BULDACCT periodically and store the result for just this eventuality.

Since you have missing accounts as well as the UFID problem, it seems your system directory is damaged. I think it’s a safe bet that your system volume set is clobbered. You need to do an INSTALL from your SLT. This will re-install your operating system and give you a brand new directory

You will also need to restore the contents of your system volume set. Make sure you use the KEEP option so you won’t lose any files created by the INSTALL. You might want to purge or rename COMMAND.PUB.SYS before the restore, so you get your SETCATALOG definitions restored along with the files.

Larry Barnes notes:

Your SYSGEN tape may or may not have the SYS account on it. It depends on how the tape was created. You can generate a SYSGEN tape and have it include certain accounts. I usually include sys and TELESUP on the tape.


11:42 PM in Hidden Value | Permalink | Comments (0)

May 02, 2014

Timing makes a difference to MPE futures

Coming to market with virtualized 3000s has been a lengthy road for Stromasys. How long is a matter of perspective. The view of an emulated 3000's lifespan can run from using it for just a few years to the foreseeable future. I heard about both ends of the emulator's continuum over the last few weeks.

StopwatchIn the Kern County Schools in Bakersfield, Calif., a 3000 manager said the timetable for his vendor's app migration is going to sideline any steps into using CHARON. Robert Canales, Business Information Systems Analyst in the Division of Administration and Finance, was an eager prospect for the software last May, when the company's Training Day unfolded out in the Bay Area. But the pace of migration demonstrated by his MPE software vendor, who's moving customers to Linux, showed his team that 3000 computing was not going to outlast the vendor's expected migration timetable.

Our main software vendor has since migrated several of their California K-12 education customers off of the 3000. We believe that our organization will be able to successfully migrate over to their Linux-based platform within the next 18-24 months. So from that perspective, we simply couldn't justify the financial investment, or the time for our very limited number of personnel, to focus on utilizing the CHARON solution for backup, testing or historical purposes.

The analysis at the district draws the conclusion that two more school years using available HP 3000 iron -- at most, while awaiting and then undertaking a migration -- will be a better use of manpower and budget than preserving MPE software. This is understandable when a commercial application drives IT. You follow your vendor's plan, or plan to replace something. Replacement could be either the physical hardware with an emulator, because the vendor's leaving your MPE app behind. Or everything: your OS environment as well as applications. Getting two years of emulator use, or maybe a bit more, isn't enough to fit the Kern County Schools resources and budget.

On the other side of that timetable, we can point out a comment from the recent CAMUS user group conference call. It suggests people will want to do more than mimic their 3000 power. They'll want to trade up for a longer-term installation.

An MB Foster analyst noted that as hardware moves upward, from one level of emulation to a more powerful option, the changes might trigger application upgrading. That's a long schedule of use, if you consider that horsepower increases usually happened on 3- or 5-year timetables back when MPE ran only on 3000s. That mirrors a schedule that emulator vendors have reported as commonplace: several decades of lifespan.

Arnie Kwong clarified what he said on that call: that moving upward in the CHARON license lineup might be reason for a vendor -- like some in the 3000 world -- to ask for upgrading fees.

My understanding on CHARON is 1) If you change processor class (for example, from an 'A' license to an 'N' license) then you are likely to get 'upticks' from your third party vendors.  

2) If you change to 'more processors' (for example, from one 'A' license to more than one 'A' license so that you can run separate reporting machines or year-end processing or the like) then you have more licenses as you are running more processors.

This isn't a change for anything that has been in place -- it's just a clarification of ours, that we haven't heard of anyone who isn't doing this the same way as its always been done. Stromasys is vending the 'hardware' and the software suppliers are providing the 'code' as things have always been.

We don't know how likely such upticks will be in the community. 3000 shops use an array of third party vendors. Some vendors do charge for processor uplifts. Others do not, and the number of vendors who will do this has not been confirmed by the installed CHARON base. We heard a report that a PowerHouse user was facing a six-figure fee to emulate their 3000. We heard that report before PowerHouse ownership changed at the end of 2013.

But if you think about that kind of scenario for a bit, you come up with a company that's extending its MPE power while it emulates. That's an investment to cover more than a few years. Emulating customers, just like the vendors who are offering this virtualization, are often into their applications for a very long ride. Before Stromasys emerged as the survivor in the emulation derby, there was Strobe Data. Willard West at that vendor talked about a multiple decades of a timetable for its HP 1000 and Digital emulation customer base.

"Our major competition has been the used hardware market," West said a decade ago. "We’ve out-survived that." At the time that we talked, Strobe was emulating Data General servers that were obsoleted 15 years earlier.

Emulation vendors know that time can be on their side if an application is customized and critical to a company. When time is on your side, the costs to revitalize an in-house application can be applied over enough years. Emulation mimics more than hardware platforms. It preserves IT business rules for returns on investment which have often been on MPE's side. MPE applications have outlasted their hardware and triggered upgrades. The clock on the ROI determines IT investments, just like it always has.

08:14 PM in Homesteading, Migration | Permalink | Comments (0)

May 01, 2014

3000 mailing list notes becoming fainter

Have you ever been down to your mailbox with anticipation, pulled open the door and find nothing new? The HP3000-L listserve, which we variously call the 3000 newsgroup and the 3000 mailing list, is having that kind of dry spell. Like the rainfall that we yearn for in Texas this spring, it's been close to two weeks since a single new note has been in that mailbox.

Silent RunningsThere's little point in comparisons but being the thieves of joy. However, the days of 1,500 messages a month were more joyful for the prospect of MPE and 3000 wisdom in those times, a torrent shared and shaped by a larger community. A goodly share of those messages, even in the heyday, covered the flotsam of politics, as well as more scandalous off-topic notes on climate science and treason. You could shop for a car or camera off of the advice, in those days.

The message count has drawn down despite a stable subscriber tally reported by the hosting system, servers at the University of Tennessee at Chatanooga. A little less than 600 readers are now receiving 3000-L mail. That is, however, the number of subscribers who were tallied nine years ago. And at least all of today's mail -- well, nearly all -- is related directly to HP 3000s. Off-topic noise has been all but eliminated.

We have a slavish devotion to the 3000-L, as the community veterans call it. Thousands subscribed to its messages for free, and I read that rich frontier of information in the early 1990s and could believe in a monthly newsletter for 3000s and MPE. We even devoted a column to summarizing and commentary about its traffic, for many years. John Burke was columnist for many years of those reports; the columns ran for more than 9 years in the printed edition of the Newswire. (Find them at the classic archives of the Newswire Tech Features, or type net.digest in our search page off the link at left.) Our caveat in passing along that expertise was "Advice offered from the messages here comes without warranty; test before you implement." If not for 3000-L, our last 18 years of work here might not have emerged.

A similar dry spell for the "L" took place in February, but the current one is the longest we've measured so far. It's simple enough to break the drought, simpler than what we face in Texas, anyway. Ask a question online -- you can do it via a web browser -- if you're subscribed (or sign up, from the website.) Then watch the wisdom echo back. In some ways, the L is like a canyon wall that won't speak until you shout out to it. Or futuristic drone robots, waiting for a command.

In years past, the mailing list was also a newsgroup. By using newsgroup reading software, and then later using a browser, readers of comp.sys.hp.mpe could enjoy all the wisdom, and wince or chuckle at the chaff. Alas, the synchronizing of listserv and newsgroup has broken down by now. You could not get a specific number in those days about readers. You knew how many subscribed via emails. But comp.sys.hp.mpe could be read and used by countless others.

After the previous dry spell, readers could learn how to lock a KSAM file in PowerHouse Quick, or get advice on how to rebuild a 3000's filesystem. The former is an arcane bit of technical knowledge, yes, but the latter is everyday wisdom. And the L offers a dialogue process, to follow up with additional questions.

Like the drone robots Huey and Dewey from the sci-fi classic Silent Running -- a movie so old that Bruce Dern was young while he starred in it -- the L is likely to run long after most people will find an everyday use for it. In an apt coincidence, Silent Running made its premeire the same year that HP did its first Series 3000 launch, in 1972. The 3000-L looks back for its wisdom, while the direction in which that film looked gave a view of one kind of future. Nobody can be certain when either of these stories will see their final showing. The Web, after all, remembers all.

07:09 PM in History, Homesteading, Web Resources | Permalink | Comments (0)