Historically, moving data between systems was typically a three-step process:
- Create a local file of the data to be moved.
- Transfer the data to the remote system (e.g. FTP).
- Create a file on the remote system for subsequent processing.
Using Parallel Data Mover (PDM), it’s possible to create and read data on remote systems directly from z/OS mainframes. On Linux/UNIX/Windows (LUW) systems, this can be accomplished using standard and named data pipes. z/OS, however, does not provide a standard equivalent facility. PDM fills the gap by providing similar capability on z/OS through its use of the Subsystem Interface (SSI).
How It Works
The z/OS Subsystem Interface (SSI)
A z/OS subsystem is code that provides services to other executing tasks. The most common subsystem is the Job Entry System (JES). JES is a “primary” subsystem that schedules jobs and provides spooling of input or output data. PDM installs on z/OS as a secondary subsystem. As such, PDM can exploit SSI facilities.
Typically, JCL is used to connect programs with datasets that reside on locally attached disk/tape storage devices. Read/write I/O requests are handled by PDM with minor changes to JCL statements. PDM satisfies read/write requests using data files on a remote system. The remote system can be another z/OS image or a LUW or Hadoop system potentially thousands of miles away.
The full capabilities of PDM are available as data streams between systems:
- Full compliance with existing data security and integrity facilities
- Data encryption and/or compression of data in flight
- Data manipulation
- EBCIDC/ASCII translation
- Add or remove line end characters
- Add or remove trailing blanks in records
- Monitoring, logging and error reporting for the transfer
This is an example of how the standard IBM z/OS utility (IEBCOMPR) compares two files. While the intended use of this program is to compare two local z/OS files, this example shows how one, or both files, may reside on a LUW system.
//USER1JOBB JOB.. //COMP EXEC PGM=IEBCOMPR //SYSPRINT DD SYSOUT=* //SYSUT1 DD DSN=’file1’, // DCB=(RECFM=FB,LRECL=80,BLKSIZE=8000), // SUBSYS=(PDM1,‘PROFILE=MYPROF’,‘HOST=unixhost’) //SYSUT2 DD DSN=USER1.LOCALDS,DISP=SHR //SYSIN DD DUMMY
The dataset “myuserid.DME.PROFILE(MYPROF)” for the example above contains the following:
* MY STANDARD PROFILE USER myid * My userid on host ‘unixhost’ PASS mypass * My password on host ‘unixhost’ PREFIX \tmp\ * File prefix to use DATATYPE A * Do ASCII/EBCDIC conversions
While all parameters can be specified directly in JCL, using PROFILE members achieves the following:
- Store parameters that are common to multiple jobs
- Stores parameters that may change over time (system names, file names, security credentials) making those changes quick and easy.
- Protects security credentials from being displayed in Job output logs
- This facility only applies to sequential data. The z/OS program can access other file types (VSAM, DB2) through different JCL statements, but the JCL statement pointing to PDM must be sequential processing (DSORG=PS)
- Processing must be “standard” sequential. The program opens the file once, reads or writes data starting at the beginning of the file and proceeds sequentially to the end then closes the file.
This facility streams data across platforms without creating interim files or provisioning storage. As a result, multi-step processes involving remote systems that create a local sequential files on z/OS file for one-time useare no longer needed. The most common example is unloading DB2 and VSAM data to a sequential file for transfer to a remote system. Benefits include:
- Saves time – unload processing and file transfer are concurrent
- Saves z/OS CPU time – eliminate overhead to create the local file
- Saves storage space – no space is needed for interim files
The ability of z/OS programs to directly read or create data files on remote systems is a standard feature of PDM. It requires simple changes to JCL statements to implement. No operating system or program changes are required. PDM’s use of the Subsystem Interface (SSI) is just one example of how Alebra optimizes technology to simplify the complex and save time and money.
For more information, contact Alebra at firstname.lastname@example.org.