As part of our AAG product we have been looking at how we can make the security bulletin checks easy so that users can get their systems status with respect to security exposures announced by IBM. The process uses a DB we manage to look up any security exposures that have been identified and any PTF’s that have been released by IBM to fix the exposure. This then sends notifications out to the user via the Nagios interfaces so you know exactly what security exposures your system is open to.
Sending the notifications is only part of the problem, you have to make sure that you download and install the fixes or the notification just keeps coming (annoying). We have always used Fix Central to download the latest CUM and PTF groups (Another check we run from AAG) so having the individual fix information would require a manual process to load the order via fix central. We wanted this to be a bit more like the other platforms where the fixes could be downloaded and installed with a single request from the IBM i, this is where SNDPTFORD comes in. (Please see our other Blog entry about setting up ECS on partitions hosted within another IBM i partition for the problem we encountered and the fix supplied by IBM).
We developed a test program called TSSECBUL that would carry out the same checks for the PTF’s directly on the IBM i as opposed to through the AAG process. This would the check if the required Licensed Program Product is installed (see note below) and if necessary send an order for the PTF using the SNDPTFORD process. The following shows the steps taken for a specific LPAR we needed to update.
You can see from the output below that AAG had found a number of PTF’s that were required to remove the exposures identified by the Security Bulletins announced by IBM. (All of the Security Bulletins we have listed in the DB are from a list that is provided from IT Jungle). This information is collected directly from the IBM i using DB so it it specific to the system we are checking. As you can see we had 12 security exposures identified.
Prior to the request to download the PTF’s
We Installed the new command and program on the target LPAR so that we could get the PTF’s from IBM. The whole thing relies on the SNDPTFORD being able to run plus the image catalog has to be available (The IBM documentation states that when the SNDPTFORD is run, if the Image Catalog does not exist it is created? In our checks this was not the case so we had to create the Image Catalog before running the command) but does not have to be connected to any virtual optical device.
New Command to check and download fixes
Once the command is run you will see output generated to screen about the CVE’s that are being checked and the relevant LPP and option, if the LPP and option are installed and the PTF is not installed the program will submit a job to go and get the PTF from IBM. Once all of the checks have been completed pressing enter clears the screen.
Review being run and orders placed
You can see if any orders were submitted by using the WRKSBMJOB command, the following is an example of what we saw for this particular LPAR. One jobs is the SNDPTFORD which is the one we submitted, this in-turn launches another job (QESECARE) that appears to manage the download of the image to the image catalog.
SNDPTFORD being run
Once all of these QESECARE jobs have finished you can check the image catalog and see the images have been downloaded and attached.
Orders placed waiting for downloads
You can now see the image catalog entries have been added.
Updates now in Image Catalog
Before you can load and apply the PTF’s you will need to load the image catalog to the virtual optical drive you have.
Load the Image Catalog to the optical device
Next we will verify the fixes.
Verify the fixes
Once everything has been verified we can then use the PTF menu (option8) to install the fixes from the image catalog.
Go to PTF Menu option 8
On the other systems we updated we did not need to IPL the system, as you can see later for this system an IPL was required to install all of the fixes, we were doing this early one morning so an IPL was not going to be a problem. You may have to consider this on your system to ensure you are not affecting your users while the IPL occurs.
Install the PTF’s
You will see the PTF’s being installed
PTF’s being installed
As mentioned above when installing the PTF’s it came across one or more that needed an IPL to apply fully. We were in the fortunate position of not having any problems with an IPL at the time.
IPL needed on this system
IPL is going to throw us off the system, no one else is on so no problem.
Confirm the IPL
Once the system had come back up we ran the SECBUL check against the system and we can see the system is now fully up to date and no security exposures exist.
All Security Bulletins now applied.
The whole process including the IPL took about 40 minutes on this LPAR, others have been as little as 10 minutes so this is definitely a time saver for us and the bonus is we can sleep happy at night knowing that our systems are not open to the exposures identified by the CVE’s.
We developed this program and command as a test for the one that we will add to the AAG product in an upcoming update, that command will be a lot more integrated with the product environment such as having its own job queue and job descriptions.
I think this alone makes the AAG product a worthwhile investment, saving all that time to investigate and download PTF’s to fix security exposures makes our life a lot simpler! We only have 12 LPAR’s to do this on internally, some of our customers are looking after 100’s of LPAR’s so their time savings will be huge. All our other platforms have a much simpler process for downloading and installing updates, now the IBM i is getting some of that capability.
PS: Don’t forget to clean up the downloaded PTF files once the PTF’s have been installed, they may not be huge but having them sitting around can take up a lot of Disk..
Happy Days.. Chris..
Notes:
We found out that the PTF’s listed in the CVE data are LPP Option dependent (IBM does not state which LPP Option the CVE relates to) if you download the PTF’s and run the install it will not install the PTF and you will keep seeing the notifications from AAG. We fixed up the DB to include the Option affected (after a lot of trial and error) so that checks would correctly omit any options that are not installed. I have asked IBM to add the affected option to the CVE data via the ideas portal.
