IBM and HCL partnership: a positive change for IBM Workload Scheduler people

After 18 years spent working for IBM, mainly on Workload Scheduler covering different roles, the time to leave and move to a different company has come.
On September 1st I've started working for a new company, HCL Technologies, a global IT services company with 100.000+ employees, but I'm not leaving IBM Workload Scheduler development.

This change is part of a 15+ years strategic partnership between HCL and IBM to invest together in IBM Workload Scheduler.

In this article I'll try to represent my personal view of this partnership and why I think this is a positive change for the product and all the people working with it.

All the information in this article are my own opinions that must not be consider coming from either IBM or HCL.

For an official statement from IBM and HCL about the partnership  you can look at the following posts on the HCL and IBM web sites.

Java DNS caching

I just spent some hours in the last days understanding an issue caused by the default Java behavior resolving hostnames, and I think sharing this configuration detail can help other people.

In my case I was experiencing failure due to some of our application servers that was unable to connect to another server. The problem was initially appearing random, some server was working, other not, even if they have the same configuration.
The failing servers was receiving a "connection refused" error while connecting to the backed server, but contacting the same URL from the command line was working successfully.
Restarting the application server was fixing the issue for that machine, but providing no clue about what had caused the issue.

Using tcpdump command I was able to trace the actual IP address used for the connections attempts, confirming that the application server was actually contacting an IP address different from the current one (the one returned by nslookup command). Investigating with the remote server team they confirmed that the other IP address is a backup system where the service was down at that moment.
My failing server was contacting an old server, currently inactive.

As we found, the HA (High Availability) architecture for the remote server is based on DNS resolution, with the hostname resolved to the IP address of the currently active server.
The default behavior of Java is to cache DNS resolution forever, with the result that our servers was continuing to use the IP address cached inside Java even if the active server has changed and the DNS has been updated.

This technote documents how to tune the JVM and change this behavior.
In our case we have changed the java.security file setting networkaddress.cache.ttl=30.

If HA strategy requires to update the DNS, this Java behavior can impact several scenario where TWS server or TWS agent have to contact a remote server using this strategy, e.g. a remote LDAP server or an application scheduled via plugin.

If you like this article and you find it useful, please share it on social media so other people may take advantage of it.

Using HTTP Server - Part 1: Introduction

During the development of our SaaS infrastructure, we have found very useful the usage of IBM HTTP Server in front of our WAS servers. Not only for load balancing on TDWC cluster, but also for security, performances and to modify some behaviors.

Setting up IBM HTTP Server is pretty simple and includes the following phases.

• Define architecture and SSL certificates
• Configure TDWC in cluster
• Install both IBM HTTP Server and Web Server Plugin
• Configure HTTP server
• Configure web server plugin

I'll dedicate a specific article to each of the above phases.

On our SaaS, in addition to TDWC access, we use the HTTP server also for connections from dynamic agents and to handle few redirects:

- to display a disclaimer at the beginning of each session

- to replace the logout page with a custom one.

HTTP server can also be used to set browser caching and reduce the network traffic and TDWC server load.

The presence of HTTP server improves TDWC scalability also because reduce the impact of network latency on the server. In this configuration TDWC can return the result back to TDWC very quickly, with HTTP server that will keep a thread active to return the data back to browser. This reduces the number of active threads in TDWC server.

If you like this article and you find it useful, please share it on social media so other people may take advantage of it.

Message flow and processes on FTAs and classic E2E

I've received a request from a customer for information about the flow of messages (events) in Tivoli Workload Scheduler for z/OS classic End-to-End.
With messages I'm referring to the information exchanged between TWS agents and servers in order to start and track job execution, submit new workload, modify existing workload, etc..
I'll not use the word event, that is sometime used in this context, to avoid confusion with the events of Event Driven Workload Automation (EDWA).

This is a very specific and technical topic, however understanding this flow was the first think I made when I started working on the TWSd code to start the porting on z/OS and integration with TWSz (OPC at that time) to make first release of the classic E2E. It was the year 2000 and TWS development was still in Santa Clara, while OPC was already here in Rome. I started creating the diagrams that I'll use in this article and they was on the wall in front of me for several months.
Even if this information is also available in the manuals, I think it could be useful to have this information also on this blog.

The picture above represents the basic message flows for a Fault Tolerant Agent (FTA).

Workload Service Assurance and Ideal Batch

Workload Service Assurance is one of the most powerful feature present in Tivoli Workload Scheduler, present both in TWS for z/OS and in TWS distributed.
This is also know as "Dynamic Critical Path" or with the acronym WSA.
This was made to address the need of taking under control the end of complex critical workflows.

Scheduling batch processes is a fundamental backbone of every IT infrastructure, from the smaller organizations to the larger one, and very critical processes are delegated to the scheduler, like creating payrolls, drive money transfers, calculate and distribute price lists, create financial statements, automatically process orders, process claims in insurance companies, etc...
Most of this processes are still scheduled and usually need to complete within a specific time, otherwise it will have significant business impacts and often will result in fees to pay. Also dynamic workload may have SLAs that imposes to process the request within a specific amount of time.
Traditionally this critical processes are constantly monitored by operations or application teams to assure they complete on time, the main challenge is to identify all the jobs that are part of the flows and assure that no one has issues that can impact the completion of the overall process. If there is an high number of jobs in the flow, the user try to identify the critical path that need to be monitored with more attention in order to react quickly to any issue. This remains a complex and time consuming work.
Workload Service Assurance is made to address those needs, but at the same time makes a step forward, removing the need to constantly monitor those processes.
The ideal batch is the one that you can forgive, that you can assume is working and that will provide the expected result on time, you should care about it only when there is an unexpected issue and in this case you should be notified and able to easily find where the issues is.