Month: December 2022

As part of our AAG product we have been looking at how we can make the security bulletin checks easy so that users can get their systems status with respect to security exposures announced by IBM. The process uses a DB we manage to look up any security exposures that have been identified and any PTF’s that have been released by IBM to fix the exposure. This then sends notifications out to the user via the Nagios interfaces so you know exactly what security exposures your system is open to.

Sending the notifications is only part of the problem, you have to make sure that you download and install the fixes or the notification just keeps coming (annoying). We have always used Fix Central to download the latest CUM and PTF groups (Another check we run from AAG) so having the individual fix information would require a manual process to load the order via fix central. We wanted this to be a bit more like the other platforms where the fixes could be downloaded and installed with a single request from the IBM i, this is where SNDPTFORD comes in. (Please see our other Blog entry about setting up ECS on partitions hosted within another IBM i partition for the problem we encountered and the fix supplied by IBM).

We developed a test program called TSSECBUL that would carry out the same checks for the PTF’s directly on the IBM i as opposed to through the AAG process. This would the check if the required Licensed Program Product is installed (see note below) and if necessary send an order for the PTF using the SNDPTFORD process. The following shows the steps taken for a specific LPAR we needed to update.

You can see from the output below that AAG had found a number of PTF’s that were required to remove the exposures identified by the Security Bulletins announced by IBM. (All of the Security Bulletins we have listed in the DB are from a list that is provided from IT Jungle). This information is collected directly from the IBM i using DB so it it specific to the system we are checking. As you can see we had 12 security exposures identified.

Prior to the request to download the PTF’s

We Installed the new command and program on the target LPAR so that we could get the PTF’s from IBM. The whole thing relies on the SNDPTFORD being able to run plus the image catalog has to be available (The IBM documentation states that when the SNDPTFORD is run, if the Image Catalog does not exist it is created? In our checks this was not the case so we had to create the Image Catalog before running the command) but does not have to be connected to any virtual optical device.

New Command to check and download fixes

Once the command is run you will see output generated to screen about the CVE’s that are being checked and the relevant LPP and option, if the LPP and option are installed and the PTF is not installed the program will submit a job to go and get the PTF from IBM. Once all of the checks have been completed pressing enter clears the screen.

Review being run and orders placed

You can see if any orders were submitted by using the WRKSBMJOB command, the following is an example of what we saw for this particular LPAR. One jobs is the SNDPTFORD which is the one we submitted, this in-turn launches another job (QESECARE) that appears to manage the download of the image to the image catalog.

SNDPTFORD being run

Once all of these QESECARE jobs have finished you can check the image catalog and see the images have been downloaded and attached.

Orders placed waiting for downloads

You can now see the image catalog entries have been added.

Updates now in Image Catalog

Before you can load and apply the PTF’s you will need to load the image catalog to the virtual optical drive you have.

Load the Image Catalog to the optical device

Next we will verify the fixes.

Verify the fixes

Once everything has been verified we can then use the PTF menu (option8) to install the fixes from the image catalog.

Go to PTF Menu option 8

On the other systems we updated we did not need to IPL the system, as you can see later for this system an IPL was required to install all of the fixes, we were doing this early one morning so an IPL was not going to be a problem. You may have to consider this on your system to ensure you are not affecting your users while the IPL occurs.

Install the PTF’s

You will see the PTF’s being installed

PTF’s being installed

As mentioned above when installing the PTF’s it came across one or more that needed an IPL to apply fully. We were in the fortunate position of not having any problems with an IPL at the time.

IPL needed on this system

IPL is going to throw us off the system, no one else is on so no problem.

Confirm the IPL

Once the system had come back up we ran the SECBUL check against the system and we can see the system is now fully up to date and no security exposures exist.

All Security Bulletins now applied.

The whole process including the IPL took about 40 minutes on this LPAR, others have been as little as 10 minutes so this is definitely a time saver for us and the bonus is we can sleep happy at night knowing that our systems are not open to the exposures identified by the CVE’s.

We developed this program and command as a test for the one that we will add to the AAG product in an upcoming update, that command will be a lot more integrated with the product environment such as having its own job queue and job descriptions.

I think this alone makes the AAG product a worthwhile investment, saving all that time to investigate and download PTF’s to fix security exposures makes our life a lot simpler! We only have 12 LPAR’s to do this on internally, some of our customers are looking after 100’s of LPAR’s so their time savings will be huge. All our other platforms have a much simpler process for downloading and installing updates, now the IBM i is getting some of that capability.

PS: Don’t forget to clean up the downloaded PTF files once the PTF’s have been installed, they may not be huge but having them sitting around can take up a lot of Disk..

Happy Days.. Chris..

Notes:

We found out that the PTF’s listed in the CVE data are LPP Option dependent (IBM does not state which LPP Option the CVE relates to) if you download the PTF’s and run the install it will not install the PTF and you will keep seeing the notifications from AAG. We fixed up the DB to include the Option affected (after a lot of trial and error) so that checks would correctly omit any options that are not installed. I have asked IBM to add the affected option to the CVE data via the ideas portal.

We are running and IBM i hosting IBM i LPAR’s environment that we use for all of our development. The setup works perfectly for our purposes and even though we have since created a VIOS based environment on our Power9 this Power8 is our main development system.

One of the things we always try to do is keep the systems (individual LPAR’s) at the latest CUM and Group level so we use the Fix Central portal to download the latest CUM package and PTF Groups on a regular basis. Because we are now looking at the security bulletins published by IBM and the PTF’s required to fix the exposures, many of these are not included in a PTF group or CUM package. Having to sign onto the Fix Central portal and order the fixes individually is a long and manual process, especially where we need to do it for each of our partitions. The answer is the SNDPTFORD command which can be used to automatically download the PTF’s to an Image Catalog to allow installation.

We did this on our main hosting partition and everything worked perfectly, but when we tried in on the hosted partitions it failed every time! To make matters worse the only message that was sent indicated that we needed to review the job log to see the reason that the request failed, there were no other messages! So we placed a PMR with IBM asking for help, unfortunately one of the LPAR’s is running V7R2 and we needed a service extension before IBM would even consider looking at it. Thankfully we had the exact same problem on all of the LPARS running V7R3 and V7R4 so IBM could provide support against those LPARs.

This is the list of actions that IBM suggested we take to fix up the ECS links.

DLTSRVCFG DLTCMNCFG(*YES) (make sure the system value QRETSVRSEC is set to one for this to work)

WRKTCPPTP remove any profiles that are listed.

WRKLIND LIND(Q*) Remove the QESLINE/QTILINE objects (they are no longer needed)

RMVLNK OBJLNK(‘/QIBM/USERDATA/OS400/UniversalConnection/*’) this command fails with ‘Requested operation not allowed. Access problem.’ You can ignore this as the bits that need to be deleted are.

CRTSRVCFG ROLE(PRIMARY) CNNTYPE(DIRECT) CNTRYID(SELECT) STATE(SELECT) This will take you through a couple of screens where you need to select the country and province the system is located in.

SNDPTFORD PTFID((SF98720)) This simply orders a cover letter, does not matter that its for V7R2 in this instance its only used to test the download works.

SNDSRVRQS ACTION(*TEST) This will send out a PMR and close it. You will get emails about the PMR being raised and closed (3 in our case)

VFYSRVCFG SERVICE(ecs) VFYOPT(ALL) This runs some verification tests, just check the job log to make sure they all ran successfully.

Once all of the above had been done on each of the LPAR’s the SNDPTFORD was working fine except for our V7R2 LPAR, this is because IBM checks for the service extension as part of the process and if you do not have it the process fails (still same problem with absolutely no data to suggest why it failed).

This fixed the problems for us, your mileage may differ but hopefully it fixes the problem.

Happy Days… Chris…