IBM i IWS or IAS Application Server Instance Does Not Start – CWWKE0002I Last Message Logged

Problem

IBM i Integrated Application Server v8.5 and Integrated Web Services v2.6 instances fail to start.  The associated IBM HTTP Server instance starts normally and the JVM appears to be active, but the /www/<server>/wlp/usr/servers/<server>/logs/messages.log shows the application server instance is hung up, not fully started, and did not become active.

Symptom

CWWKE0002I: The kernel started after x seconds” is the last message logged in the application server’s /www/<server>/wlp/usr/servers/<server>/logs/messages.log.
The /www/<server>/wlp/usr/servers/<server>/workarea/equinox.log contains the following exception:

!ENTRY com.ibm.ws.org.eclipse.equinox.region 4 0 2023-03-03 14:18:50.903

!MESSAGE FrameworkEvent ERROR

!STACK 0

org.osgi.framework.BundleException: Exception in class org.eclipse.equinox.internal.region.RegionManager.start() of bundle com.ibm.ws.org.eclipse.equinox.region.

Caused by: java.io.EOFException

    at java.base/java.io.DataInputStream.readUnsignedShort(DataInputStream.java:345)

    at java.base/java.io.DataInputStream.readUTF(DataInputStream.java:594)

    at java.base/java.io.DataInputStream.readUTF(DataInputStream.java:569)

    at org.eclipse.equinox.internal.region.StandardRegionDigraphPersistence.readRegionDigraph(StandardRegionDigraphPersistence.java:108)

    at org.eclipse.equinox.internal.region.RegionManager.loadRegionDigraph(RegionManager.java:106)

    at org.eclipse.equinox.internal.region.RegionManager.start(RegionManager.java:61)

    at org.eclipse.osgi.storage.FrameworkExtensionInstaller.startActivator(FrameworkExtensionInstaller.java:287)

    at org.eclipse.osgi.storage.FrameworkExtensionInstaller.startExtensionActivator(FrameworkExtensionInstaller.java:272)

    … 22 more

Cause

Issues reading the application server’s cached workarea data, which includes persistent cached information related to the individual application server instance, which includes OSGi resolver metadata and persistent OSGi bundle data.

Environment

IBM i OS

Integrated Web Application Server v8.5

Integrated Web Services v2.6

Diagnosing The Problem

If an individual IBM i Integrated Web Services v2.6 or Integrated Web Application Server v8.5 instance fails to start successfully, review the /www/<server>/wlp/usr/servers/<server>/logs/messages.log.  If “CWWKE0002I: The kernel started after x seconds” is the last message logged, then review the /www/<server>/wlp/usr/servers/<server>/workarea/equinox.log for the “org.osgi.framework.BundleException: Exception in class org.eclipse.equinox.internal.region.RegionManager.start() of bundle com.ibm.ws.org.eclipse.equinox.region” and “java.io.EOFException” exceptions.

Resolving The Problem

End the application server instance and start it up with the -clean option.

ENDTCPSVR *HTTP HTTPSVR(<server>)

STRTCPSVR *IAS INSTANCE(<server> ‘-clean’)

Then, see whether the <server> starts successfully by reviewing its /www/<server>/wlp/usr/servers/<server>/logs/messages.log file.  If it starts successfully, then end the application server and start the complete instance.

ENDTCPSVR *IAS INSTANCE(<server>)

STRTCPSVR *HTTP HTTPSVR(<server>)

 

Running “STRTCPSVR SERVER(*IAS) INSTANCE(<server> ‘-clean’)” cleans all persistent cached information (under workarea) that is related to the specified server instance, which includes OSGi resolver metadata and persistent OSGi bundle data. If you use this option, the server will be required to recompute any cached data at the next startup, which might take more time than a normal restart that can reuse cached data.

Only the ADMINx application server instances use –clean on every startup. Non-ADMINx application servers do not specify –clean at startup by default to improve application server startup performance, especially when numerous apps are deployed. As a result, it has to be manually specified to perform the cleanup of the workarea for non-ADMIN (e.g. IWS or IAS) servers.

[{“Type”:”MASTER”,”Line of Business”:{“code”:”LOB57″,”label”:”Power”},”Business Unit”:{“code”:”BU058″,”label”:”IBM Infrastructure w/TPS”},”Product”:{“code”:”SWG60″,”label”:”IBM i”},”ARM Category”:[{“code”:”a8m0z0000000CJLAA2″,”label”:”Integrated Application Server”},{“code”:”a8m0z0000000COsAAM”,”label”:”Web Services-u003EIntegrated Web Services”}],”ARM Case Number”:”TS012273706″,”Platform”:[{“code”:”PF012″,”label”:”IBM i”}],”Version”:”7.3.0;7.4.0;7.5.0″}]

How to recover from MSGCPDBCA1 “Handle is not valid” posted repeatedly in QTVTELNET after Certificate Authority deleted from DCM

When deleting certificate authority (CA) certificates from Digital Certificate Manager, if the Telnet Server has a CA Trust List defined and the CA certificate deleted is on the Telnet Server CA Trust List, new secure Telnet connections can quit working and message CPDBCA1 “Handle is not valid” is posted repeatedly to the Telnet Server QTVTELNET job log.

The IBM i OS Base Is Older Than We Think, But Moving Ahead

The IBM i OS Base Is Older Than We Think, But Moving Ahead

March 6, 2023

Timothy Prickett Morgan

Hardware upgrades are costly in terms of money and while it is always exiting to get a new system with that “new computer smell” – and you know what we are talking about – at least IBM i operating system and systems software upgrades don’t carry such a high price – provided you are on Software Maintenance and you do proper testing on your software before you make a jump to a new release.

The other great thing about IBM i systems software is that any given release not only runs on the current new Power Systems hardware at the time, but also on several prior generations of hardware, and older releases are updated to cover new hardware as it rolls out, too. Add in the Technology Refresh updates for the past several IBM i generations and it is considerably easier to stay current on IBM i software than it is on Power Systems hardware – and much less expensive, too.

The question is, how current are companies when it comes to their IBM i stack. We get some sense of that each January when the IBM i Marketplace Survey report comes out, which details the results of the survey that happened during the fall of the prior year. The survey, which is done by Fortra (formerly known as HelpSystems), has been in the field for nine years now and gives us a rich set of data from which to try to understand the IBM i base.

As we pointed out last week when we analyzed the Power Systems installed base among the IBM i faithful, we think that the data gathered by the survey reflects the state of the active part of the IBM i base, which we reckon is about 30,000 unique customers and does not indicate the vintage of hardware and systems software at the remaining 90,000 customers worldwide, what we call the laggards.

This active-laggard split is admittedly a hunch that we have, but it is that is informed by anecdotal evidence from customers and business partners who tell us far more vintage System/38, System/36, AS/400, iSeries, and System i iron has stayed in the field longer than any survey data ever shows. We think these laggard customers are stuck on vintage and unsupported software from third party vendors who may not even exist anymore or who charge too much money to upgrade applications to current releases. We think that active customers are the ones who tend to read publications like The Four Hundred, to go to COMMON and other user group meetings, and to take the time to do surveys. And we think that the Fortra survey, to be specific, is absolutely representative of what active customers are doing in terms of Power Systems and IBM i release levels.

We know that the laggards are quite a bit further back, although by how much is not clear. We reckon it is about four years or so behind the active part of the base, but following a similar path. Last week, we took the dataset of Power Systems distribution by Power CPU family from the nine Fortra reports and did a little witchcraft on it to figure out what the machine base at the remaining 90,000 sites might look like. This week, we are looking at the primary IBM i releases installed at sites.

The IBM i Marketplace Survey allows respondents to report on all Power Systems machines they have in their company, not just the primary machine’s type. But when it comes to the operating system, it asks for the primary one. This stands to reason, since if you have two machines (or three) in a high availability or disaster recovery configuration, you will probably be at the same hardware and software release. But some companies have separate machines for test and development and other uses, perhaps out in the field, and these can be at different hardware and software releases.

Here is what the raw data for primary IBM i operating system installed for the past five years:

In this raw data, we see that IBM i 7.5, which came out in May 2022, was cited by 2 percent of respondents, and we can see the rise of IBM i 7.4, which was cited by nearly half of respondents in the survey that ran last October. You will also note the decline of IBM i 7.3 after a pretty long run, and the decline of earlier releases as well.

We like to get an even longer view, and in more distinct colors and patterns that are easier to see, and so we put together this chart with all nine years of data:

We have estimated the distribution of earlier OS/400, i5/OS, and IBM i releases based on past data, when these were all called out separately. We think this is pretty representative of the OS distribution over time among the 30,000 active customers. As you can see, IBM i 7.5 is just starting out its climb.

But we do not think for a second that it is representative of what is going on at the laggards, and therefore for three-quarters of the installed base. We do not for one second believe that IBM i 7.4 is nearly half of the installed base and that IBM i 7.3 is nearly 40 percent of the base. No server operating system has 90 percent of its customers on the two current releases. Not Windows Server, not Linux. Not the IBM System z mainframe and its z/OS platform.

We created a model last year that showed that more than half of the base was actually on IBM i 7.1, which is consistent with the anecdotal evidence we get from customers and partners by time shifting those laggards to the distribution of operating systems that we saw on the IBM i Marketplace Survey four years ago. If you do that, you get a distribution of primary operating systems – including both actives and laggards added together – that looks like this:

Just for fun, we are going to let you see the model data that we generated for the 2022 and 2023 installed bases:

Here is what is interesting. The IBM i 7.4 base is growing very fast, and we think that this is customers moving up from IBM i 7.2 and in some cases IBM i 7.3. But funnily enough, the base of IBM i 7.2 is growing as customers upgrade from IBM i 6.1 and to a certain extent IBM i 7.1, and even though the IBM i 6.1 base is down, it is being propped up by those moving from i5/OS 5.4 and in some cases OS/400 V5R3. And IBM i 7.1, in our analysis, represents half of the IBM i base, not IBM i 7.4, and this has some important implications.

You can’t fault IBM for this, either. The company has done wonders with the Technology Refresh method of updating software functionality across three concurrently supported releases, and one of the side effects is that people are taking advantage of the seven years of support and can do so without sacrificing major functionality. That generosity is extending the effective life of IBM i releases. But the IBM i 7.1 base grew in the past 12 months – not by much, but by a few thousand customers. And if we are right, that is a big problem in the long run.

Last week, when talking about hardware, and in a bit of a haze because I was writing late at night, I said that IBM should worry less about what IBM i versions and releases it can support on PowerVM logical partitions on a particular Power10, Power9, and Power8 system and more about how it can use the Technology Independent Machine Interface, or TIMI, to make a Power10 processor look like a Power9, Power8, Power7+, or Power7 processor, and maybe older CPUs and their systems. Rather than trying to update everyone to more modern hardware and operating systems, why not convince older releases of OS/400, i5/OS, and IBM i that they are actually running on whatever hardware they expect? This would mostly mean hiding more modern features, not taking away old stuff.

So we ask again: Why can’t TIMI and microcode make any new Power processor look like any older Power processor? With such an approach, you could in fact move everyone to Power10 and just leave their software – operating system, database, systems stack, and applications – as they are. Why not?

RELATED STORIES

The Power Systems Base Is A Little Less Rusty (2023 Survey)

The IBM i Power10 Upgrade Cycle Forecast Looks Favorable (2023 Survey)

RPG Use “Skyrocketed” Says IBM i Marketplace Report (2023 Survey)

Cutting IT Costs Is Not A Priority, And That’s Good News (2023 Survey)

How Do You Stay In Touch With The IBM i Community? (2023 Survey)

Security Still Top Concern, IBM i Marketplace Study Says (2023 Survey)

The State Of The IBM Base 2022, Part Three: The Rusting Iron

The State Of The IBM i Base 2022, Part One: The Operating System

The State Of The IBM i Base 2022, Part Two: Upgrade Plans

The Real IBM i Legacy Is The People

IBM i Salaries: Underpaid, Yet Highly Valued And Hard To Replace

Marketplace Study Shows How IBM i Language Use Evolves

Security Again Top Concern in HelpSystems Marketplace Study

The Distinguished Professionals Of IBM i

Are You Experienced? IBM i Users Weigh In

Settling In With IBM i For The Long Haul

IBM i Has Been Getting With The Program For Years

The IBM i Base Did Indeed Move On Up

The IBM i Base Is Ready To Move On Up

Investment And Integration Indicators For IBM i

Security Still Dominates IBM i Discussion, HelpSystems’ 2018 Survey Reveals

The IBM i Base Not As Jumpy As It Has Been

The Feeds And Speeds Of The IBM i Base

IBM i Priorities For 2017: Pivot To Defense

IBM i Trends, Concerns, And Observations

IBM i Survey Gets Better As Numbers Grow

Where Do Those IBM i Machines Work?

Rocket Rolls Out IBM i Modernization Bundle

Rocket Rolls Out IBM i Modernization Bundle

March 6, 2023

Alex Woodie

Rocket Software last week unveiled a new collection of tools designed to help IBM i shops elevate their game across multiple IT disciplines. Dubbed the Rocket Modernization Suite for IBM i, the new bundle brings together existing Rocket offerings in the DevOps, APIs, test automation, HA/DR, terminal emulation, and Web-enablement spaces. It’s all about lowering the barrier to adopting Rocket tools and improving customer productivity when they do, the company says.

Like other software vendors in the space, Rocket Software has accumulated somewhat of a disparate collection of tools for the IBM i server. The Massachusetts vendor started its IBM i journey with the acquisition of Seagull Software and its screen-scraper and Web and mobile-enablement tools, now called Rocket Modern Experience. Then it added Datamirror’s iCluster high availability and disaster recovery tools, via a deal with IBM. It acquired Aldon’s change management tools for IBM i (now called Rocket DevOps), as well as the BlueZone emulator (Rocket Terminal Emulator. Along the way, it developed several brand new tools for IBM i.

The goal with Rocket Modernization Suite for IBM i is to bring all of those tools together in a more cohesive manner that benefits customers and better serves their overall modernization goals, according to Puneet Kohli, who was recently promoted to the position of president of the Application Modernization Business Unit, where the IBM i product sit.

“The real beauty of this launch is we put together a complete picture to say ‘This is what an IBM i portfolio from Rocket looks like, and this is how it helps the customer who is modernizing the i ecosystem on their end,’” he says.

Some Rocket customers have classic modernization requirements, insofar as they want to upgrade their greenscreen applications with Web and mobile interfaces, or even a Windows fat client. The Rocket Modern Experience, which is descended from Seagull’s JWalk and LegaSuite offerings, can provide that GUI upgrade.

It’s not uncommon for IBM i shops to have applications with 20,000 to 30,000 screens in their ERP and related systems. Not every 5250 screen needs to be upgraded with a sleek HTML interface, however. To figure out which screens an organization uses most, Rocket developed its own power tool, dubbed Rocket Process Insights, which it released in September 2021. That, too, is part of the new modernization suite.

But in many cases, customers’ modernization needs go beyond simply beautifying the look-and-feel of an IBM i application or even upgrading the RPG logic, Kohli says. For instance, the need to integrate IBM i applications with applications running on other platforms, such as ERP systems from SAP and other vendors, is also a form of modernization. Rocket API, which can automatically generate a REST API from a given applications, was developed in-house and released in June 2017 to provide that integration capability for shops without deep RPG knowledge.

Developer-heavy organizations can also benefit from the use of more modern code management techniques and technologies. So that puts Rocket DevOps into the modernization bucket.

“They’ve got applications running on i, they’ve got RPG code, they need to manage their RPG. So they need some DevOps capabilities,” Kohli says. “There are open source tools out there that even the i ecosystem needs to integrate with, and we made it easier for us to be able to integrate with JIRA, with Jenkins and with Git, and that’s part of this launch.”

Rocket added automated testing capabilities to its DevOps offering a year ago. Rocket DevOps Test helps manage the test scripts that developers create to ensure their programs run as designed, and also helps to ensure the security of data sets used in tests. As you might have guessed, testing also falls into the modernization rubric for Rocket.

Organizations running in the modern manner also don’t put up with unnecessary downtime, so high availability and disaster recovery are also a component of modernization.

“They obviously need HA/DR,” Kohli says. “We take a very pragmatic approach to HA/DR [with iCluster]. Not tons of bells and whistles, per se, but things that you really need. You need access to data when you need access to data. When a disaster strikes, you should be able to very quickly get access to it.”

Rocket has already built hooks that unite some of these products at a technical level. Developers who use Rocket Modern Experience to create new GUIs, for instance, can have their work automatically tracked with Rocket DevOps. Test scripts used to test the new applications can also be tracked with Rocket DevOps Test.

The company is currently working to create a common user experience across all of the seven products. According to Kohli, it’s being developed in the Carbon, which is IBM’s open source design system. All of the user interfaces will have a common look and feel, which will make it easier for users to adopt other Rocket products, he says.

There’s one product element missing from Rocket’s modernization journey: a development tool. The company has no plans to develop its own IDE, as others in the IBM i ecosystem have done (like its competitor, Remain Software, and its lightweight code editor, MiWorkplace).

Rocket’s tools are well-integrated into IBM’s Eclipse-based IDEs, with Rational Developer for IBM i (RDi) being chief among them. But Rocket is also increasingly seeing customers want to work in Web-based VSCode environments, which it is also supporting. Rocket’s tools work with Code for IBM i, the plug-in developed by Liam Allan.

The Rocket products do not currently work with IBM Merlin, which also offers a VSCode development tool, Kohli says. But the company is in talks with IBM and Rocket’s products, such as Rocket DevOps, could be integrated at some point in the future, he says.

IBM Merlin, of course, launched last year with built-in support for Rocket’s competitor, ARCAD Software and its DevOps tools. That clearly didn’t please Rocket and other providers of DevOps tools for IBM i, but IBM i CTO Steve Will said IBM may open Merlin up to other providers in the future.

“We are getting the sense they’re open to that,” Kohli says. “I think it would be good for the ecosystem anyway. I think the customers would appreciate that.”

Bundling is in the air these days, as Merlin clearly shows. Rocket sensed that customers are receptive to the bundling message too. In some cases, multiple departments within the same organizations were engaging Rocket separately, so having a single bundle helps drive more efficiency into customer engagement for Rocket (and vendor engagement for customers).

In some cases, customers that already have one Rocket product may see some savings by adopting the new Rocket bundle, according to Kohli.

“We have over 2,000 customers who are running one of these products on their i platform and that’s a huge opportunity for us to go in and talk to them about the capability of other products,” he says. “We felt that it was time for us to really talk about the IBM i ecosystem and bring these together.”

RELATED STORIES

Rocket Launches New DevOps Solution for IBM i

Rocket Maps IBM i Apps for Modernization Ventures

Rocket Rides Modernization, DevOps for Hybrid Cloud Adoption

Rocketing Ahead With An API Engine

Verified by MonsterInsights