Chapter 1. Introducing Linux

This chapter introduces Linux. Linux is the UNIX-like kernel of an operating system initially developed by Linus Torvalds at the University of Helsinki in the early 1990s. Hundreds of programmers that were connected through the Internet helped develop Linux into a fullfunction operating system. In this chapter we explore these questions:

  • What does Linux offer as an enterprise-class operating system?

  • What is the role of Open Source in Linux development?

  • How can Linux run on the mainframe architecture?


1.1 Benefits of the Linux operating system

Today, Linux is a worthy alternative to other well-known operating systems. As an operating system, Linux offers benefits such as:

  • Full functionality.

    Like other operating systems, Linux has components for systems management, security, and performance. Throughout its history, Linux has been enhanced with additional functions as they have been needed, at rapid speed.

  • Architecture independence.

    Linux started out as a personal computer (PC) operating system, but enthusiasts soon ported it to other platforms. Linus Torvalds himself ported it to the Alpha platform and, while doing so, introduced an abstraction layer between the hardware and the operating system. This allows the system to be compiled on different hardware architectures but still use the same source code. Thus, only small adjustments had to be made in the kernel to let Linux run on the mainframe. The way Linux is written allows for wide variety in hardware architecture. For example, when running on the mainframe, Linux uses the mainframe I/O structure with no problems.

  • Support for standards and new technologies.

    Thanks to the many active Linux developers, Linux is fast to adopt new technologies. Linux was one of the early operating systems to support 64-bit addressing. 64-bit addressing combined with sufficient random access memory (RAM) allows the central processing unit (CPU) to pull more data into memory and operate on it directly, thus increasing performance.

  • Linux is an Open Source project.

    Linux is an example of Open Source development with many active developers working together to provide functionality, enhance performance, and ensure support of devices.

To fully appreciate the last point, we need to discuss Open Source in more detail.


1.2 The role of the Open Source community

The idea of Open Source plays a major role in Linux development. In this section, we will look at what Open Source is, what it means for software to be Open Source, and what it means for Linux to be Open Source.

The definition of Open Source can be found on the organization's Web site, http://www.opensource.org.[1] Open Source code is licensed under a variety of licenses, all of which provide access to the source code and the right to modify it.

[1] A discussion of the Open Source philosophy is given in The Cathedral and the Bazaar by Eric Raymond.

1.2.1 Peer review means quality

How can something put together by a disparate group of programmers have high quality? Put yourself in their shoes. Any code you submit to the Open Source project is scrutinized by a lot of people. You cannot hide behind object code either, so delivering "spaghetti code" that works is not good enough. You do not want to be the laughing stock of the cybernetic neighborhood, do you? Therefore, you will work to create high quality code.

1.2.2 Licenses

Commercial software usually comes with a license that regulates its use. Open Software is no exception. The Open Source community recognized that when others use someone else's intellectual property, there must be rules that regulate users' rights and obligations. The Open Source Initiative provides a procedure for certifying software licenses as Open Source licenses.

One of the core ideas of Open Source software from a user's point of view is that a license grants you certain rights:

  • You have the right to receive the source code for the software along with any binary version of the software.

  • You have the right to modify the software according to your own needs or to expand the software with further functionality.

  • You have the right to redistribute the software, including your modifications or without them.

Because you receive the source code with the object code, you are not dependent on a software vendor for support of the code. You will not find yourself locked in a situation where you are forced to upgrade your software just because your current version goes out of service.

Two of the most prominent licenses are the GNU General Public License (GPL) and the BSD (from the University of California, Berkeley) license. While both licenses meet the Open Source definition (given at http://www.opensource.org), they differ substantially in their terms.

The Linux operating system consists of several portions of software called packages that are licensed under different Open Source licenses.

1.2.3 Linux as an Open Source project

When Linus Torvalds started work on "Linus' UNIX" operating system in 1991, he was aware of the work the Open Source community was doing. He derived his work from the academic MINIX operating system written by Andrew S. Tanenbaum. Although Torvalds' project started out as a private enterprise, it quickly became independent of the MINIX project and became a global, widespread project, especially when Torvalds released his work as Open Source software under the GPL. The Internet made this possible; it played an important role in the Linux and Open Source movements.

Open Source projects attract many of today's best programmers because of their simple and sensible premise: Let us work as a team, because together we are faster and smarter than each one alone.

A brief history of Linux

Linux was developed with the help of many volunteer programmers and "wizards" across the Internet, allowing anyone with enough know-how to develop and change the system. Table 1-1 shows the rapid development of Linux in the early years.

Table 1-1. Linux history: the early years

Year

Event

1991

  • August: Linus Torvalds sends a request over the Internet asking if other programmers are interested in helping him with the project. In the first six months, he received over 600 responses.

  • September: Torvalds puts version 0.01 out on an FTP server for others to look at.

  • October: Torvalds announces the first generally available version of Linux, version 0.02.

1992

  • March: Torvalds increases the version number to 0.95, to reflect his expectation that the system would soon be ready for an "official" release.

1993

  • December: The Linux kernel is still at version 0.99.pl14, slowly approaching 1.0. (Generally, software is not assigned the version number 1.0 until it is deemed complete or bug-free.)

1994

  • March: Linux 1.0 is announced at the Department of Computer Science at the University of Helsinki.

  • The first commercial Linux distribution, Red Hat Linux, is introduced.

What's in a name?

What is Linux? Ask three different people and you may get the following three answers:

  • The operating system kernel, offering the basic services of process scheduling, virtual memory, file management, and I/O—what Linus Torvalds started.

  • A complete system consisting of the kernel plus the many applications that it runs: compilers, editors, graphical interfaces, games, and so on—what Linus Torvalds and thousands of others are developing in the Open Source community.

  • What you get when you buy, for example, "SuSE Linux 8.0" in your local software store—what a distributor has chosen to combine into an easily installable product.

None of the above answers is right or wrong. Those who look at Linux as just the kernel of an operating system have a point, as other parts that make up the full operating system come from Open Source projects, such as GNU.

There are also people who think of Linux as what we refer to as a Linux distribution. Sometimes the association is so strong that they use the name of the distribution instead of Linux when talking about the operating system (Red Hat, SuSE, and Turbolinux, to name a few). This is easy to understand, given that a Linux distribution is what you are probably dealing with when you run Linux.

However, for the purposes of this book, we will use "Linux" to mean the complete operating system. When we talk about the kernel, we will call it the kernel or the Linux kernel. When we talk about a combination of programs you can buy from SuSE or others, we will call it a distribution.

IBM and Linux

IBM does not provide a Linux distribution. However, IBM is an active participant in the Open Source community. IBM delivers Open Source code that is either actively integrated into the official source code repositories of the various projects or packages or helps to integrate it.


1.3 The role of distributions

A distribution is a collection of various development projects (most of them are under constant development) that work together. Distributors select the packages with the versions that provide the right levels of new functionality, while at the same time assuring a level of stability, reliability, and maintainability. This must be achieved for the software packages of the individual projects and also for them working together. For some packages—for example, for the so-called core packages of the GNU development and buildenvironment—this is especially important.

A distribution includes applications with the operating-system packages. Among the most popular applications are Web-serving applications (such as Apache) and file and print serving applications (such as Samba). A lot of solutions can be built directly from the Open Source applications that come with the distribution. The distribution also contains installation scripts, configuration files, system administration tools, and documentation. Distributions often provide maintenance and support for the products they include, education, and sometimes certification.

Some distributions exist for very specific solutions. For Linux running on personal computer (PC) based systems, for example, there are distributions that focus on firewall solutions, while others focus on mail-serving, Web serving, or other solutions.

It is possible to create your own Linux by going to the Open Source projects and selecting what you want. However, building your own operating system is not a trivial task. When building your own Linux system, you pay with the work you put in, whereas a commercial distribution has a monetary cost. The cost covers, for example, the packaging, documentation, and testing which the distributor has done. Distributions often offer service and support.

1.3.1 Choosing a distribution

What criteria should you use when selecting a distribution? The answer depends on your situation. However, there are generally two ways to arrive at it: You can do an internal study or get consultant services.

Criteria to consider when choosing a Linux distribution are:

  • Does the distribution contain and support most of the functionality and applications that you need? One way to find this out is to install the distribution and test it in your environment.

  • What kind of middleware do you want to run? Who provides support for it? The middleware owner or your service provider?

Distributions that are available for Linux on the mainframe at the time of this writing include Red Hat, SuSE, and Turbolinux.

1.3.2 The role of a distributor

Distributors usually play an active part in Open Source development projects. The major distributors employ Open Source developers. For example, a distributor may employ compiler developers and developers working on the kernel.

Distributors actively drive some Linux development (for example, the compiler), depending on their business needs or the platforms they want to support. Distributors also support special projects where they see value for their customers (for example, a journaling file system). They run stress tests and actively search for security holes. They compile documentation and simplify installation. They provide support for their distribution and offer services and education.

Actively participating in Open Source projects means that distributors are giving something back to the community. In this way, everyone benefits from their work.

1.3.3 What about maintenance and support?

How do you plan to maintain an Open Source operating system like Linux? Are your employees going to do it? Are you getting a contract? To whom you turn for maintenance can depend on whether the system is for test or production purposes.

Maintenance from the Open Source community

Using Open Source software gives you options in the way the software is maintained. Normally, the provider of the software offers some level of maintenance. If not, you always have the choice (apart from looking for other software) of either maintaining it yourself or subcontracting the work to another company. The source is available for you to work with.

While chances are good that a request for a repair job to the Open Source community will get you a quick answer, there is no guarantee that anybody in the Open Source world will treat you as a 24-hour priority customer. If your business requires quick responses to maintenance requests, you may want a maintenance contract. Distributors normally offer maintenance and support for their distributions.

Maintenance from a subcontractor

If you are getting a contract, consider that you may have to recompile the Linux kernel in order to include needed functionality. This must be allowed under the terms of your maintenance contract. We will discuss maintenance and support in Chapter 16, "System Administrator Tasks."


1.4 Linux structure

The Linux operating system consists of a kernel and tools. Figure 1-1 shows the structure of the operating system with the kernel shaded. The diagram is not to scale. The kernel is significantly smaller than the tools and application portions.

Figure 1-1. The Linux structure

Linux kernel

A kernel is the program that controls the allocation of the machine's resources to the other programs that are running. Though a kernel is an essential part of an operating system, it is useless on its own. It can serve a useful purpose only as part of a complete operating system. Linux is a UNIX-like system and, as such, it contains most UNIX features such as multitasking, virtual memory, memory management, and TCP/IP networking.

GNU Compiler Collection (GCC)

GCC is a versatile and sophisticated compiler that can compile programs written in several different languages to run on many different hardware platforms. Languages supported include C, C++, Objective-C, Ada, Fortran, and Java. GCC is usually supplied with a Linux distribution, but is also available for download on the Internet.

GNU Binary Utilities

These are a set of utilities supplied as part of a Linux distribution. For example, the utility ar can be used to extract files from the archive file. Also included in the package are the linker ld and the assembler as. If you have acquired Linux as part of a distribution, binutils would have been included with the distribution. If you need to upgrade to the latest release of binutils, go to http://www.gnu.org/directory/index.html.

Note:

If you ever need to upgrade your kernel, GCC, glibc, or binutils, you must ensure that the new version is compatible with the rest of your operating system and that you follow the guidelines for compiling the new version. You may also need to recompile some of your programs, making sure you specify the correct libraries.

Linux applications

A distribution normally bundles various applications (such as word processors, spread sheets, Internet browsers, Web servers, fax programs, and graphics programs). Additionally, sophisticated applications are available from many suppliers.

Device drivers

A device driver is a file or program that enables communication between a computer and a specific peripheral device such as a printer. For Linux to be able to use a device, the associated device driver must be available to the Linux kernel. This can be accomplished either by compiling the device driver with the kernel or by making the driver available as a module. Details of the available mainframe device drivers and comprehensive information about their use are documented in Device Drivers and Installation Commands.[2]

[2] www.software.ibm.com/developerworks/opensource/linux390/.

Runtime environment

To run an application that has already been compiled, you need various support programs and files. A runtime environment provides these items. A runtime environment can simply be a directory that contains the necessary files, such as glibc or libgcj (created by compiler GCJ for Java) or the libg2c library (created by G77 Fortran compiler, part of GCC). Whereas a Java applet can rely on a Web browser to run a Java application, a user needs a Java virtual machine and various support programs and files. This collection of software constitutes a Java runtime environment. Various suppliers provide Java runtime environments for download. See, for example, IBM Developer Kit and Runtime Environment for Linux: http://www.ibm.com/developerworks/java/jdk/118/linux/jre-info.html, or Java 2 Platform, Standard Edition (J2SETM): http://java.sun.com/j2se/1.3/runtime.html.


1.5 Summary

Today, Linux is an enterprise-level operating system. It is robust and exploits numerous standards and new technologies.

Linux is an example of an Open Source project. Distributions include Open Source tools in their packages. The actual expense for using this operating system in your business can be significantly less than for other operating systems.

Open Source signifies freedom for developers to work on projects in which they take an interest. This is an attractive premise, and many talented developers work on Open Source projects. The scrutiny of many Open Source community members makes code quality high.

Distributions solve a perceived problem with Open Source: who is responsible for ongoing software support? Distributors (and others such as IBM Global Services) take care of support and maintenance in a legally binding way.


Chapter 2. Introducing the Mainframe

To understand what the mainframe offers to Linux, you need some basic knowledge of the mainframe's architecture and design. It is difficult to find a concise and easy-to-read mainframe overview. In this chapter, we provide a brief introduction for those who are new to the mainframe.

In this chapter, we discuss the following questions:

  • How did the mainframe develop?

  • What distinguishes the mainframe from other platforms?

  • What does the mainframe of today offer?


2.1 The mainframe's birth

Before 1940, a "computer" was a human being who did calculations using paper and pencil. Digital or electronic computers were developed to relieve the human computer of boring, iterative calculation work. An electronic computer performed calculations much faster than a human could and did not get tired. An important aspect of the electronic computer was that it could store a program. This distinguished the computer from an adding machine.

The first electronic computers were housed in several cabinets consisting of steel frames with metal covers. The most important cabinet—the main frame—housed the circuitry that performed calculations. The other cabinets housed power units and peripheral devices such as tape drives.

The very first IBM mainframe was installed half a century ago in IBM world headquarters (see Figure 2-1). It consisted of a room-sized set of cabinets housing the units for the 701 Defense Calculator. The 701 was the first large-scale, high-speed, stored-program electronic computer to be produced in volume by IBM.

Figure 2-1. The room-sized 701, IBM's first mainframe. IBM announced installation of the first machine in 1953. The 701 main frame is at the back of the room on the right. The other cabinets include three power units, a dual magnetic tape drive to the left, and card units to the right. (Photograph courtesy of IBM Archives.)

The 701 was capable of performing thousands of instructions per second: 2000 instructions when multiplying and dividing, and up to 16,000 when adding and subtracting. The average instruction execution rate was 14,000 instructions per second. Compared to the manual computing common at the time, this was an overwhelming achievement. One day of digital computer time was equivalent to 500 years of pencil and paper calculations.

The 701 may have been the first large-scale electronic mainframe, but it was built for special engineering and scientific purposes, not for commercial transaction processing. It did not have a general-purpose operating system (OS), and it could run only custom-made programs. However, its success encouraged IBM to make a computer for the commercial market. To find all-purpose mainframes as we know them today, we must go forward to the 1960s.


2.2 General-purpose computer architecture

Around 1960, there were many computer systems in the marketplace, each developed for a specific purpose. Some were optimized, for example, to do accounting, others to solve engineering problems or to help scientists with complex calculations. At the time, about 80% of the total customer cost of the IT environment was spent on hardware. Therefore, fully exploiting the capabilities of a piece of hardware was crucial for an efficient IT operation.

To efficiently use IT resources, it was sometimes desirable to move a program between computers. This was not easy. Figure 2-2 illustrates the relationship between the hardware, or electronic layer, and the program, application, or user layer at that time. To move between different computers, it was necessary to rewrite programs, because the interface to the programs was different for each machine.

Figure 2-2. Four computer systems, each with a different interface to applications and operating system

In response this problem, Gene Amdahl, Fred Brooks, and Gerry Blaauw envisioned and implemented the IBM System/360 (S/360)[3] architecture. The five-member S/360 family of computers was announced in 1964.

[3] The "360" in the name stands for the 360 degrees of a full circle (not for the 1960s when it was invented).

The basis for S/360 was the idea that computing could be abstracted so that a very broad range of problems could be solved within the confines of a single computer architecture. In fact, we can say that all computing can be reduced to the following operations:

  1. Get some data.

  2. Perform some computation (or manipulation) on that data.

  3. Save the data that resulted from the computation.

The IBM team distinguished between architecture, which is the theoretical behavior of the system, and design, which is an implementation of the architecture. They designed machines that implemented the general architecture in a way that allowed subsets of the problem space to be solved better (faster or more cost-effectively) on one model compared to another model. For example, some models of the S/360 family, such as the model 95, were considered to be supercomputers. The model 65 was the workhorse, top-of-the-line business computer, and the model 25 was for small businesses. All three used the same architecture across a wide range of applications.


2.3 Distinguishing features of the mainframe

Let us go deeper into the structure of three main resources defined by the S/360 architecture.[4] It is this structure and implementation that distinguishes the mainframe from other vendors' computer implementations and that can give some significant value to your business opportunity.

[4] In this book, we will use mainframe architecture to mean any of these architectures: S/360, S/370, S/390, or z/Architecture. If we refer specifically to the 64-bit architecture of the zSeries 900, we will use the term z/Architecture.

The mainframe is based on the von Neumann computer model (see Figure 2-3), with these main parts:

  • The CPU is the processing unit, responsible for executing instructions.

  • The memory[5] holds data to be manipulated.

    [5] Memory is called storage in the mainframe world.

  • The I/O system takes care of communication with external devices.

Figure 2-3. The von Neumann computer model

2.3.1 Central processing unit

In this section, we will take a look at how the architecture and design of the central processing unit (CPU) contributes to the reliability and scalability of the mainframe. The design aims to optimize the workloads that are run on the machine.

Architecture of CPU

The architecture of the CPU defines data integrity and work separation. Data integrity means that an instruction either completes accurately or the reason for the error can be found. One instruction is executed at a time. If it does not complete, the architecture defines how this will be handled. In other words, the mainframe is self-checking.

Work separation alludes to the fact that the CPU can run in either of two modes: supervisor (or privileged) mode and problem mode. The two states ensure that only properly authorized programs, like the operating system itself, can perform critical actions. At the same time, user programs can run without fear of their interfering with another program's or operating system's data. To protect data, storage protect keys were initially used. Later, technologies such as virtual memory and address spaces were developed.

The mainframe architecture defines the order in which instructions are to be processed as "next sequential instruction" (NSI). However, in the mainframe design there are two exceptions to the sequential rule: branching and interrupts. When branching, the program branches to an address specified in a branching instruction instead of executing the next instruction as specified. Interrupts signal external events that have an impact on the sequence in which instructions are executed.

A little known fact is that, statistically, every fifth instruction is a branch. So while the mainframe architecture defines it as an NSI machine, the CPU is designed around the fact that processing is constantly going down different paths. In the CPU design, a lot of effort has gone into ensuring that both branching and interrupts are handled as efficiently as possible. These efforts are ultimately the reason for the speed of processing.

Multitasking

Early systems processed work as it came in as a job unit. Each job was processed as a whole, and only one job was processed at a time. Jobs often had to wait (for example, for information from a printer or a tape drive), and the CPU was idle much of the time, as illustrated in Figure 2-4. The speed of the external devices was much slower than that of the CPU.

Figure 2-4. An active job causes the CPU to be idle while the program is waiting for I/O

In contrast to a personal computer, a mainframe is designed to support many users at the same time. On a PC, you might not really care how busy the CPU is, because you are the only user on it. On the mainframe, thousands of different pieces of work compete for processing resources. Finding a way to keep the CPU working while a job waited would increase the amount of work the CPU could process. The solution was multitasking. Multitasking means that when a job waits for I/O, another job can be processed, as illustrated in Figure 2-5.

Figure 2-5. Multitasking: While one job waits, another can be processed

In order to implement multitasking, a technique was needed to switch control from one job to another. This switching is achieved by interrupts. An interrupt is an event that alters the sequence in which the CPU executes instructions, as illustrated in Figure 2-6.

Figure 2-6. Interrupts alter the execution sequence of instructions

An interrupt can be planned (specifically requested by the program) or unplanned (caused by an event that is not related to the executing task). See Chapter 21, "Mainframe Reference" for more details on the interrupt technique. Interrupts and multitasking are possible because the relevant information about the interrupted program is captured and saved. This is called state handling and allows the CPU to continue executing a program where it left off.

The interrupt technique enables a program to wait for work without consuming significant resources, as the program does not need to constantly check if there is work to be processed. Programs will be called by an interrupt when data for processing are available.

Multiprocessing

Even the most efficient CPU can execute only one instruction at a time. As the workload increases, thousands of jobs can wait for a single CPU. In this situation, it might make sense to add CPUs. Two or more CPUs processing simultaneously make up a multiprocessor. Multiprocessing on the mainframe was introduced with the S/370 architecture in 1970.

Increasing the number of CPUs increases the demand on the operating system. Linux is an operating system that handles multiprocessors fairly well, as we shall see in Chapter 3, "Introducing Linux on the Mainframe."

Today's mainframe (for example, the zSeries 900) is a Symmetric Multiprocessor (SMP)[6] with up to 16 CPUs as illustrated in Figure 2-7.

[6] "Symmetric" means that from the point of view of any one CPU, the rest of the machine looks the same. At the same time, from the point of view of the memory, all CPUs behave identically. In other words, all CPUs can do the same tasks and have access to the same memory and the same I/O devices.

Figure 2-7. An SMP with 16 CPUs

A mainframe CPU can sustain a utilization of over 90% over long periods of time.

2.3.2 Memory

Memory's task is to serve data to the outside world. Memory either receives data and stores data safely, or gives data out. Memory serves many masters, including the CPUs and the I/O subsystem. This being the case, memory is optimized to do its job quickly and provide for data integrity while doing it. The memory subsystem does multitasking and knows how to keep track of all the requests for data. For example, when a program tries to use data that it shouldn't, an interrupt occurs, and a program is brought in to handle the problem.

Architecture of memory

Initially, the architecture defined a flat, real memory space. Memory was addressable by bytes, half words, full words, and double words (see Figure 2-8). Eventually, quadwords (4 words) and page were added for addressing, depending on the instruction or I/O operation being run.

Figure 2-8. Memory can be visualized as a horizontal string starting at zero and extending to the limit of memory

As CPU speed increased, it was only a matter of time before a CPU could execute more programs in a given period of time than it could hold in its own storage. Again, the CPU was idle for periods of time. The solution was to bring pieces of a program into memory (or real storage) instead of the entire program. The program pieces, called pages, were fixed in size (4 KB today). Each page was assigned a unique address defined by the first byte of the page. That address was called the virtual address. The CPU kept track of which pages were in real storage and which it would have to get by way of a table that listed the page addresses for the program that was executing. Thus, memory was no longer limited to the size of real storage, but to the number of virtual addresses (see Figure 2-9).

Figure 2-9. Virtual memory lets programs address a virtual address range. Only a subset may be residing in real memory at any time.

The benefit which programmers see from virtual addressing is that the program is freed from the constraints of real memory. Programmers can still address memory as if it were a flat array. The operating system resolves the addressing at runtime by paging.[7]

[7] Paging is I/O, so you might expect it to take a relatively long time. But paging is a much more efficient I/O than if the program itself would have to handle it.

Address spaces—full range of memory, yet isolation guaranteed

What is an address space on the mainframe? An address space is a range of contiguous virtual storage addresses that the operating system creates. A small portion of an address space is reserved for system data and programs, but most of an address space is available for user data and programs.

The first version of IBM's OS operating system to use virtual memory, the Single Virtual Storage (SVS) system, removed the constraint on real storage and provided a single 16-megabyte virtual address space to be divided among all tasks.

When more than one program is running on a computer, the issue of isolation becomes important. Programs must not be allowed to write into memory reserved by others. There are several techniques for doing this, including allocating a separate piece of real memory to every program. Real memory is divided into "n" pieces, plus an overhead for the operating system.

Only six months after the release of SVS in 1972, the Multiple Virtual Storage (MVS) operating system introduced the concept of one full range of virtual addresses per program running. That is, each address space would represent 16 MB of virtual memory, as illustrated in Figure 2-10.

Figure 2-10. S/370 architecture defined the address space to have the full range of addressing from 0 to 16 MB. MVS put each job or user in a separate address space.

Design of memory

From the beginning, the CPU was an expensive resource, and customers wanted to take full advantage of it. Hence, the goal of machine designers was to keep the CPU busy. The CPU had to be fed continually with data from memory, which in turn was fed from the I/O subsystem. The designers came up with the cache concept which allowed the CPU to access and process data faster. The mainframe today uses multiple levels of cache and separates data cache from instruction cache.

The architecture requires that each CPU has the same view of memory. In the design of the memory structure, it must be guaranteed that CPUs and I/O have the same view of memory. This is called cache coherence and requires some fairly complex design if done at the hardware level. Some architectures handle cache coherence at the programming level, which means that the cache must be "visible" to the programs. However, this can introduce problems with scaling and may lead to data integrity problems if the software misbehaves. The mainframe architecture thus does not define cache and cache is designed to hide the complexity of cache coherence. Thus, no operating system on the mainframe has to worry about cache, as it might on other architectures.

Table 2-1 shows an overview of memory development on the mainframe. Real storage addressing has progressed from 24-bit to 64-bit addressing. Virtual memory space has followed the increases in real addressing.

Table 2-1. Examples showing memory hierarchy development

Architecture

Memory addressing

Memory design (examples)

S/360

24-bit real addressing allows addressing of 16-MB memory.

Machine model 30 had a 64-KB memory. The 360/85 in 1969 was the first production machine with cache memory.

S/370

24-bit real addressing allows addressing of 16-MB memory.

Machine models 158 and 168 introduced multiprocessing and virtual memory.

S/390

31-bit real addressing allows addressing of 2-GB memory.

On the S/390 Parallel Enterprise Servers Generation 5 and 6, memory can be varied from 1 GB to 32 GB.

z/Architecture

64-bit addressing allows addressing of 16-EB[*] memory.

On the IBM eServer zSeries 900, memory varies from 5 GB on the 101 model to 64 GB on the 2C9 and 216 models.

[*] EB stands for "exa bytes" and equals 2 bytes or approximately 1 153 000 000 000 000 000 bytes.

2.3.3 Input/Output system

In contrast to other architectures that use I/O bus systems, the mainframe uses I/O channels. An I/O channel is a processor that manages the data movement between memory and external devices under the control of channel programs. In other words, I/O on the mainframe is managed independently of the CPUs. Architecturally, only the independent and overlapped execution aspects are specified and not whether a separate CPU is required for each real channel. For example, on the IBM zSeries machines, a single System Assist Processor (SAP) might control up to a hundred channels and the associated hardware interfaces.

Design of I/O

We have mentioned that it is possible for a mainframe CPU to be consistently busy more than 90% of the time. This requires that the CPU be given a steady stream of work to do. One of the potential bottlenecks in some system designs is the I/O design.

The central CPUs can process data at nanosecond speed. The rest of the computer system should be able to feed the CPUs information, and receive it from them, at the speed that they need it. Hence, there must be a balance between the CPUs, the memory, and the I/O. Let's take a look at data processing from the I/O perspective.

The simple, early approach to I/O was the following: Send out requests for data to a device, wait for data to be returned (the job is said to enter a wait state), and then continue processing. This is illustrated in Figure 2-11.

Figure 2-11. Simple I/O system design

Figure 2-11 shows quite a gap in processing while the I/O device is busy. In order to keep the CPU busy, mainframe designers decided to use this gap for other processing. The CPU would continue with other waiting work until an I/O interrupt signaled that the I/O operation was complete (Figure 2-12).

Figure 2-12. Mainframe I/O processing. The CPU can process other tasks while some tasks are waiting for I/O.

On early systems with 500 KB of real storage, three to five users (or programs) running simultaneously was the rule. As the CPU became faster, the idle time became significant again, as all programs were processed up to a point where they waited for I/O. It thus became possible for the CPU to load and initiate even more work. With the help of virtual memory (see 2.3.2, "Memory" in this chapter), it was possible to significantly increase CPU utilization.

Over the years, further improvements were made to I/O handling. Notice the two large gaps on the middle line (channel) in Figure 2-12. This means that the channel could execute another channel program that talks to a different device. By contrast, on the bottom line (device) the middle gap is small; the device should not do anything else until this data action is processed. While Figure 2-12 shows the principle of how a program is interrupted to wait for I/O, it doesn't show the proportions of time. I/O from DASD typically takes five orders of magnitude longer than a CPU cycle. For example, if we were to slow down the CPU so that a cycle lasted one second, the I/O operation would take days.

Over time, the mainframe's I/O system was optimized to keep feeding the CPUs data, allowing them to do other work while waiting for data.

Mainframe I/O processing

In the S/360 computing systems, a channel decodes the channel program, which in turn is composed of CCWs. Channels are not the same as the physical cabling; rather, they are a logical construct. The path, on the other hand, is the actual physical path that data can take to a device, as illustrated in Figure 2-13.

Figure 2-13. Mainframe I/O subsystem. Channels are connected to control units to which devices connect.

Each external device can be connected to one or more channels to allow for high bandwidth and reliability for data access on the external devices.

Channels make it possible to overlap requests to a disk to connect as soon as it is not active elsewhere and check the existing data on the disk to determine if and when what type of data transfer may occur—all done without any interaction with the CPU. Only upon completion of the channel program (or exception termination) is the CPU made aware of what has occurred. The CPU is not involved in the data movement.

As shown in Table 2-2 , the mainframe I/O subsystem has evolved from an initial maximum of 8 channels. Each channel is capable of transferring data as needed between memory and external devices independently and coincidently with other channel and CPU access to memory. This capability allows for the high data transfer rates required to sustain multiprocessing activities on up to 16 CPUs.

Table 2-2. Examples showing I/O development

Architecture

Channels

Machine models

S/360

8 Parallel

Model 67

S/390

Either of:

  • 96 Parallel

  • 256 ESCON

  • 36 FICON

S/390 Parallel Enterprise Servers Generation 5 and 6

z/Architecture

  • 88 Parallel

  • 256 ESCON

  • 96 FICON

  • 96 FICON Express

  • 24 OSA-Express

  • 4 HiperSockets

IBM eServer zSeries 900


2.4 From real to virtual

"Virtual" is a word used by other manufacturers today. Virtual on one machine is not the same as on other machines. On the mainframe it is not only the memory that can be virtualized. On the mainframe, virtualization has been part of the design for over 30 years. It is the key factor that allows you to do consolidation on the mainframe. For more details on virtualization, see Chapter 7, "The Value of Virtualization."

2.4.1 The roots of z/VM

In the 1960s, hardware was expensive, and it was imperative to use a computer as economically as possible, that is, to run it as much of the time as possible. The machine was needed to run existing business applications, and also to test new programs. Ideally, each tester needed a full machine with all the hardware and software available. Thus, there was a need for an operating system environment that allowed multiple users to use a single machine. Within IBM, two approaches were taken to meet the need: the Time-Sharing System (TSS) and what was to become the Virtual Machine (VM) operating system.

Time sharing allows multiple users to share the use of the operating system. Virtual machines allow multiple operating systems to share the same machine hardware.

2.4.2 Virtual machines

The VM operating system started its days as an unofficial project. Wanting to build a timesharing system for S/360 but realizing that the "official" TSS project would not be able to provide the sophisticated testing platform that they needed, developers at the Cambridge Scientific Center started work on the Cambridge Monitor System (CMS)[8] at the end of 1964.

[8] This is now known as the "Conversational Monitor System."

It was to be a new kind of operating system, a system that would provide not only virtual memory, but entire virtual machines. In the virtual machine model, the hardware resources of a computer system are managed by one computer program, the Control Program (CP). Users' activities are managed by an operating system (CMS is one such OS). Developers saw that the cleanest way to protect users from one another was to use the S/360 Principles of Operations manual to describe the user's interface to the CP. Each user would have a virtual S/360 machine image in software created by the CP. The virtual machine image appears to be actual computer hardware to the programs running in that virtual machine, including operating systems and application programs. The idea of a virtual S/360 was new, but what was really important about their concept was that nobody until then had seen how elegantly a virtual machine system could be built, with very minor hardware changes and not much software.

The CMS developers worked under a very early S/360 operating system until they got enough of CMS together so that they could start it stand-alone from a card reader. In 1972, IBM announced VM/370 as an operating system together with S/370 models 370/158 and 370/168. Today, this operating system has evolved into what is now called z/VM.

Virtualization, stemming from the time-sharing concept and the need to virtualize the hardware, made IBM go back and look at the mainframe architecture and extend it to ensure the isolation needed for the virtual machines and to allow the virtualization Control Program to optimize performance.

z/VM is an important reason why Linux on the mainframe is so attractive. The "guest operating system" capability of VM, along with the guest isolation provided by the virtual machine concept, makes it possible to run hundreds of Linux servers under z/VM.

Figure 2-14 shows a typical system with z/VM and Linux running as guest operating systems, side by side with other operating systems. z/VM itself can also run under z/VM (as a test environment, for example).

Figure 2-14. Linux as guest operating systems under z/VM

2.4.3 Logical partitioning (LPAR)

Some of the concepts of virtualization were so fundamental that it made sense to take them down below the architecture level. This became the logical partitioning (LPAR) and Processor Resource/Systems Manager (PR/SM) concepts.

Regardless of the number of central processors, a mainframe today can be divided logically into parts, giving the appearance of multiple different, totally separated and isolated computers to the users and to the system operator. A single zSeries 900 server today can be divided into as many as 15 logical partitions (LPARs).

For each LPAR, you can define the architecture and resources (central processors, central storage, and channels). A separate operating system can be installed and used in each LPAR. The operating systems in the LPARs can be different; you can run any mainframe operating system. An example is shown in Figure 2-15.

Figure 2-15. Linux images on one of two LPARS in a system

2.4.4 LPAR versus VM

With VM, resources can be dedicated to a guest operating system or virtualized. A virtualized resource appears to a program as a real resource that it can use fully. In fact, the resource is shared. With LPAR, resources can be dedicated (for the partition to use fully) or shared with other partitions. Table 2-3 shows the virtualization capabilities of VM and LPAR.

Table 2-3. Comparing LPAR and VM virtualization capabilities

VM

LPAR

CPU

Virtual or dedicated

Dedicated or shared

Memory

Virtual or dedicated

Dedicated

Channels or devices

Virtual or dedicated

Dedicated or shared

Number of images supported

Hundreds

15


2.5 Summary

The mainframe and hardware have undergone a long evolution. The evolution has resulted in the well-known attributes of reliability, availability, and scalability. From its 24-bit, single CPU, real storage origins it has developed into architectured, 64-bit, SMP machines with virtualization not only of storage, but also of CPU and I/O resources. A single server is capable of supporting hundreds of operating system images.

The mainframe today is a multipurpose machine, optimized for data processing in a business environment and it continues to evolve in response to new business needs.


Chapter 3. Introducing Linux on the Mainframe

From a project undertaken by a handful of developers, Linux on the mainframe has become an official component of IBM's mainframe strategy. This union of traditional proprietary computing and Open Source elements helps to bring e-business applications from distributed servers to a single platform where they can be more easily managed.

Looking at the Linux evolution and the evolution of the mainframe architecture, it becomes obvious that both have a specific background and their specific reasons for certain design decisions. In this chapter, we focus on bringing these two thoughts together. We will explore questions such as the following:

  • Why does Linux fit the mainframe?

  • How does Linux fit the mainframe?


3.1 Why Linux fits the mainframe

Linux can run on the mainframe because the mainframe is a general-purpose machine architected to support many programming models. Linux, in turn, is designed to be architecture-neutral except for a very thin architecture-dependent layer. Thus, the implementation of Linux on the mainframe came down to recompiling it for the mainframe architecture, and reimplementing Linux's architecture layer.

3.1.1 The mainframe is independent of the operating system

The IBM mainframe architecture was designed without a specific operating system in mind.[9] This is still true today. IBM's zSeries architects do not simply enrich the platform with new operating system features or new hardware functions; they primarily have the evolution of the platform in mind. Currently, the operating systems available for the zSeries mainframe are: z/OS, VSE/ESA, z/VM, TPF and, most recently, Linux.

[9] See Principles of Operation for an in-depth look at zSeries architecture.

3.1.2 Linux is independent of the hardware architecture

Linux grew to become a more architecture-independent operating system. Since the first port from Intel to the Alpha architecture in 1995, one design principle has been to clearly distinguish between architecture-dependent and architecture-independent code in the Linux kernel. The same holds for the GNU essential packages (like the C compiler and runtime library) which supported a variety of architectures even before Linux was released in 1991.

3.1.3 Fitting Linux into the mainframe portfolio

The Linux and Open Source development approach was taken for the port of Linux to the mainframe. But from the beginning, it was clear that production work had to run on Linux on the mainframe and that Linux had to run side by side with traditional mainframe operating systems in the data center. In this setting, qualities such as reliability, availability, serviceability, scalability, manageability, and security were important and had to be considered for Linux on the mainframe.

The unique hardware platform values of the mainframe are used by Linux. Fast time-to-market is also a factor when Linux is chosen in favor of a z/OS solution. Often a solution embraces both Linux and a traditional operating system, such as z/OS or VSE/ESA, in an integrated environment (that is, on the same machine). For example, since VSE/ESA does not provide a Java environment, Linux can host the Java components in an integrated VSE/ESA and Linux solution.


3.2 What was done to fit Linux onto the mainframe

Let us take a look at how the IBM team ported Linux to the mainframe and how the mainframe Linux distributions have developed.

IBM calls Linux on the mainframe "Linux for S/390" or "Linux for zSeries." Don't let the "for" confuse you. The Linux that IBM ported on the mainframe is the same Linux that is out there on the Internet. What the developers at IBM did was contribute a new architecture for inclusion in the Linux source tree—the zSeries (and S/390) architecture. Linux for zSeries supports the 64-bit z/Architecture in both real mode and virtual mode on zSeries CPUs. Linux for S/390 supports the 31-bit S/390 architecture and will also run on zSeries in 31-bit mode. The Linux code maintainers did not need to modify the common code, nor did they have to change the Linux system structure. (See Figure 3-1.) The new architecture code was accepted into the Linux code tree and is now a standard part of the Linux kernel.[10]

[10] You can even see this code on your PC Linux. It is in the code tree that ships with all Linux distributions.

Figure 3-1. Linux for zSeries system structure

For the friends of the mainframe, let it be said that no changes were needed in the zSeries or S/390 architecture. No adaptation layer was required. The bottom line is that Linux runs on the mainframe architecture, period. The goal of the developers at IBM is to combine the strengths of Linux and the mainframe, not to reinvent a mainframe-specific Linux.

So how much code was added by IBM? The patches to the kernel that are needed to build Linux for the mainframe, at the time of writing, boil down to between 0.5% in the areas of glibc, gdb, gcc, and strace, up to 0.8% for the binutils, and about 2% in the Linux kernel. These were changes due to the system environment, the CPU architecture, and device support.

3.2.1 Porting Linux to the mainframe

Despite the fact that Linux is mostly developed on PCs and the majority of users run Linux on a PC, the design of Linux has been architecture-independent since its early days. Given the availability of a well-defined and well-documented hardware architecture, a working platform-specific compiler, and an operating system that has proven to be easily portable, what else do you need to bring it to the mainframe? Just a team of skilled programmers who were very enthusiastic about Open Source, Linux, and how this could fit on the mainframe!

Within a few weeks, the programmers could show that there were no technical obstacles that prevented the port of Linux to the mainframe architecture. From the start, it was clear that this had to be more than just a proof of technology. Therefore, the Linux kernel design was carefully studied and analyzed. Two objectives had to be met:

  • The port had to become an official part of the Linux source. The platform-independent part of the Linux kernel must run unchanged on the zSeries architecture. The new code had to integrate seamlessly with the old, without impacting other architectures or violating the design and coding styles of Linux.

  • The port had to run on the basic architecture definition of zSeries as documented in the Principles of Operation, the reference description. No emulation layer, or other "glue" code, could be used, nor could any extensions to the existing zSeries architecture be required.

    While an emulation layer approach typically helps avoid porting problems and thus leads to fast results, it comes with some major disadvantages. You typically have a performance degradation, and you have another layer of code that has to be maintained and synchronized with both the layer beneath (the architecture) and the layer above (the operating system). The Linux for zSeries programmers wanted to avoid using an emulation layer and meet the challenge of a technically clean solution. The "right thing to do" was to implement Linux on the pure architecture definition.

The work progressed rapidly, thanks to the clear separation between architecture-specific and architecture-independent parts in Linux. (The code for the former is placed in subdirectories for each architecture.) Because it is all Open Source code, it could be used as template and be adapted smoothly for the S/390 architecture. The majority of the architecture-dependent part for S/390 was written by developers at the IBM laboratory in Boeblingen, Germany, in 1998 and 1999.

Some special considerations went into the design of unique mainframe characteristics and how they relate to the Linux design. One example is the I/O channel subsystem of S/390 and the interrupt concept of Linux. Other presumed issues simply vanished. One example was the assumption that S/390 computes EBCDIC data only. Few people realize this, but the S/390 architecture is actually code-page neutral. Thus Linux works as a pure ASCII operating system on the mainframe.

3.2.2 Spreading the word—the Linux install fest

The first set of patches from IBM that enabled Linux to run on a mainframe were released to the Open Source world in December 1999. In 2000, the first commercial Linux for S/390 distribution (from SuSE) based on these patches became available.

To bring the mainframe and the Linux worlds together, and to increase the number of installations, IBM sponsored an installation party with the potential administrators of Linux on the mainframe. It was called an "install fest" because it was run from Germany. (You may have heard of the October Fest in Munich.)

There was no way that customers could bring in their heavyweight mainframes and disk subsystems over long distances to the location of the install fest as they would for a Linux user group PC install fest. The install fest had to be run remotely and without impact on the parts of the system that were running production work.

Materials were sent to participants in advance—planning guides, installation documentation, Linux distributions for the mainframe and also for desktop systems—to help participants become familiar with Linux. The actual installation took place during conference calls with four to seven participants who were ready to install Linux on their mainframes. For some of them, this was their first contact with Linux. The phone calls were staged by time zone, as several continents were involved—North and South America, Europe, and Asia.

The idea worked well. There was a great overall satisfaction with the ease of the process, the quality of help, and the speed of the installation. A lot of people expected an installation of a mainframe operating system to be a tedious task, given their prior experience with traditional mainframe operating systems. Installing Linux during a three-hour phone conference was a new experience.

Many customers overcame the initial hurdle of dealing with a new operating system on their mainframes. Furthermore, some mainframe experts showed their Linux system to Linux experts who had never dealt with the mainframe platform before, but were familiar with Linux on the PC and UNIX worlds. To a great degree, experts from both sides could talk about the same thing despite their different backgrounds. These technical contacts created synergy by getting the various technical people together and having them learn from each other.

Table 3-1 shows the rapid development of Linux on the mainframe.

Table 3-1. A brief history of Linux on the mainframe

Year

Event

1998

  • December: Marist College in Poughkeepsie, New York, starts the Linux-390 mailing list: http://www.marist.edu/htbin/wlvindex?linux-390.

  • Several IBM employees participate in an unofficial capacity.

1999

  • December: IBM announces its source modifications to the Linux kernel and tool chain. Linux for S/390 will run under VM, in LPAR, and in basic mode.

  • December: Patches for Linux for S/390 are released to the Open Source community through the IBM DeveloperWorks server.

2000

  • January: Marist College releases pre-built files for a simplified install of Linux for S/390.

  • January: IBM initiates a Linux unit under senior executive Irving Wladawsky-Berger.

  • February: LinuxWorld in New York City and Paris. First technical demonstration of Linux on S/390.

  • May: SuSE and TurboLinux distributions announced.

  • August: SuSE Install Fest.

  • October: IBM commits support for 64-bit Linux (for zSeries).

  • November: TurboLinux Install Fest.

  • November: SuSE 31-bit distribution available.

  • December: IBM and other middleware for Linux on zSeries start roll-out.

  • December: 64-bit Linux patches submitted to Open Source.

2001

  • January: TurboLinux 31-bit available.

  • March: Tape and disk device driver enhancements released.

  • December: Red Hat 31-bit available.

2002

  • April: SuSE 64-bit available.

  • May: Exploitation of zSeries hardware (Cryptography and HiperSockets).

  • May: SCSI devices supported.

  • June: Red Hat 64-bit available.

  • November: SuSE SLES 8 (both 31-bit and 64-bit) powered by UnitedLinux launched.

2003

  • New offerings including middleware enhancements.


3.3 How Linux fits the mainframe

Once Linux was ported to the mainframe, it turned out that its behavior on this platform was quite good. In fact, with Linux on the mainframe, you can:

  • Scale with the workload

  • Utilize CPU power

3.3.1 Scale with the workload

It has been claimed by some experts in the industry that Linux running on systems with multi-CPU capability does not scale well. What would be the point in running Linux on a large mainframe with multiple CPUs? Performance measurements, however, clearly show that many Linux applications on a mainframe scale well.

Figure 3-2 illustrates a scalability experiment with a business intelligence workload performed on an IBM zSeries processor. When adding copies of the workload (up to eight), the time that one CPU takes to complete the work increases linearly. The diagram also indicates that when the workload level is kept constant at eight, the time taken to complete the work is halved when an additional CPU is added, then roughly halved again when CPUs are doubled. Finally, we reach a level of eight CPUs running eight workloads.

Figure 3-2. Scalability for Linux business intelligence workload. With increased load, response time increases linearly. With added CPUs, it decreases linearly on average.

The ideal scaling behavior of a multiprocessor would be when eight units of work on eight CPUs finish at the same time as one unit of work on one CPU. In Figure 3-2, if you compare the left-most and right-most measurements, they are not far off from that ideal. This experiment clearly indicates that on IBM's mainframe, Linux workloads scale well.

3.3.2 Utilize CPU power

On the mainframe, you can have hundreds of Linux images. The workload management capabilities of the mainframe let you utilize the mainframe to a higher degree than an average server performing a single task. As an example, consider the four stylized workloads depicted as functions of CPU utilization in Figure 3-3. Each workload exhibits a very different behavior from the others over time, but all of them use about 50% of one CPU. One workload uses 100% of the CPU at times and then goes back down to zero again. The others are less radical, but have their own characteristics.

Figure 3-3. Workloads using 50% of one CPU

Each of these workloads has a justification for the machine it is on. The two on the left need their machines due to peaks workloads. The two on the right may be running on systems that do not allow a higher utilization for sustained periods of time due to constraints in the application or another part of the system. As an experiment, we will now consolidate the four workloads on a 4-CPU machine.

When we build a composite of all four workloads, we get the graph shown in Figure 3-4. The average utilization is still 50%. Using a 4-CPU machine that can sustain a higher utilization than 50%, we can reclaim some of the "white-space" and put even more work on it.

Figure 3-4. Utilization after consolidation—50% utilization of four CPUs

Another alternative is to use only three CPUs. If these workloads were running on VM on a 3-way mainframe, VM would manage them in the following way. You would need to decide what workload was your least favored (for example, the topmost workload) and its peaks would get elongated to the right. Or, you could decide to slow all the workloads down the same amount. Notice that just after the highest peaks there is excess capacity. Despite the slowdown, work would be caught up very quickly.

When can you stay with the 3-way machine and when would you need a 4-way machine? If Figure 3-4 shows a 24-hour day starting at midnight, the first shift has the highest peaks. There is not much room to grow on the first shift. If you needed to add work on the first shift, that would be a reason to go to a larger machine. Similarly, if none of the first shift workloads could afford elongation (for example, if it was real-time work), you would need a 4-way machine.

Most of the time, there is capacity to spare. For the first quarter of the chart, utilization is at 2.2 to 2.3 CPUs. In the second quarter of the chart, average utilization is probably around 2.7 to 2.8. On a mainframe running z/VM or z/OS, that level of utilization is not a problem. In fact, there is room for even more work.

These sample workloads are obviously made up to show different behaviors rather than to provide a good consolidation example. Larger numbers of unpredictably behaving workloads turn out to be statistically favorable for a consolidation solution. Later (see Chapter 6, "Total Cost of Ownership: the Challenge") we will look at what kinds of workloads make a favorable consolidation case.


3.4 Six reasons to run Linux on the mainframe

Many more benefits of running Linux on the mainframe have crystallized than were expected at the time of the first proof of concept. In our talks with companies that use Linux on the mainframe, we have come across more benefits than we can list here. However, let us look at some of the most important ones. All of these will be discussed in greater detail later in the book.

Linux on the mainframe meets several needs of mainframe-based enterprises today:

  • Responsiveness.

    Many of the Open Source tools and applications that exist for Linux have become available on the mainframe. You can also test and use new functions earlier (in most cases) on Linux. You can try them out and promote or abandon them, depending upon the result of your tests.

    If a workload is broken up properly and distributed so that the right pieces of work are on Linux, time-to-market can be reduced. Securely isolated Linux images can be set up with ease, together with virtual communications that can be dynamically changed. This flexibility also adds to the time-to-market benefit.

  • Close integration.

    Today's applications are typically multitiered, with the back-ends firmly rooted in z/OS and other platforms vying for the other tiers. Linux makes it possible to move the middle-tier close to the z/OS mainframe back-end, which may improve performance and reduce complexity.

    In contrast to z/OS, VSE/ESA does not provide its own Java environment. For VSE/ESA shops, Linux provides the opportunity of running Java applications in close integration with existing VSE/ESA resources.

    Linux as an operating system can span a huge range of machines, from the mainframe to departmental machines to desktop PCs. Similar skills can cover deployment on different platforms. Support people speak a common language and communicate well.

  • Virtualization.

    With the virtualization technology found in IBM's z/VM product, you can run tens to hundreds of Linux images on a single mainframe, thus saving on hardware cost.

  • New applications.

    A broader selection of applications is available with Linux.

  • Safe experimentation.

    Running evaluations or proofs of concept are easily done on a mainframe. It is quite easy to fire up a test Linux system without impacting the production workload running on the same machine (provided the spare resources are available).

  • Skills are readily available.

    Since Linux is a UNIX-like operating system, a rich set of skills is available in the marketplace and from colleges and universities.


3.5 Summary

IBM is working to make sure that the mainframe can be used advantageously for Linux. Linux fits the mainframe well because of its hardware independence. Linux as an operating system behaves well on the mainframe. It scales and is able to fully utilize CPUs. Reasons to introduce Linux into your enterprise follow:

  • Linux inherits availability, reliability, and scalability characteristics from the mainframe hardware.

  • Linux is a source of new applications and new technology for e-business. Having new applications on the mainframe means you can use existing infrastructure for testing new ideas. In addition, having them on the mainframe can mean better integration into existing change and problem management procedures and service level agreements.


Chapter 4. Overview of What You Can Do with Linux on the Mainframe

Before you deploy Linux on the mainframe, you want to know what kind of business scenario can be advantageous. This book features two example companies: ISPCompany and StoreCompany. These companies demonstrate some typical Linux on the mainframe uses. One typical use is integrating middle-tier servers with z/OS or VSE/ESA. Another is consolidating stand-alone servers not needing integration with other systems. Scenarios that represent an in-between of these are, of course, also possible.

We have chosen two companies because with Linux on the mainframe we can observe two different styles of IT growth that we call horizontal and vertical. ISPCompany and StoreCompany represent typical businesses that are using Linux on the mainframe differently:

  • ISPCompany is an outsourcing business which is growing horizontally and offers server consolidation on a zSeries 900. Refer to Appendix A, "ISPCompany," for more details on this company.

  • StoreCompany is a retail company which is growing vertically, as the applications they use need more capacity. Refer to Appendix B, "StoreCompany," for more details on this company.

In this chapter we discuss:

  • What growth scenarios can Linux on the mainframe support?

  • What are typical Linux on the mainframe business applications?


4.1 Horizontal and vertical growth

What do we mean by horizontal and vertical growth? Horizontal growth is based on the concept of adding more of the same building blocks—in our case, adding server images alongside the existing server images. If you lined these servers up in a row, they would extend out to the horizon, as shown in Figure 4-1.

Figure 4-1. Horizontal growth

Such companies usually have a very segmented business where it is simple to segregate and grow by a cloning-like process. ISPCompany (for its outsourcing side of the business) experiences most of its growth from adding more images. For security reasons, each ISPCompany client needs a separate set of images. For ISPCompany's Internet service provider business, it is simple to grow by just adding more images for mail serving or Web serving as more clients sign up.

Vertical growth is based on the concept of an application needing more and more capacity. (See Figure 4-2.) In this case, it is easier to grow the image size than it is to split the work among a set of images. You could visualize this as an image growing upward:

Figure 4-2. Vertical growth

Both the horizontal and vertical growth scenarios can map well to zSeries:

  • Horizontal: It is relatively easy to create another Linux image using cloning techniques supported by z/VM. Some IT shops measure the time needed to set up a new server in a matter of minutes!

  • Vertical: A zSeries mainframe can currently grow to a 16-way SMP. Its memory and cache structure allow a single application to exploit that entire multiprocessor capacity. Such a machine is efficient at both utilizing multiprocessor capacity and running multiple processes. Linux is developing into an environment that can handle workloads using several CPUs efficiently. There are various Capacity Upgrade on Demand options available.


4.2 ISPCompany and its new business application

ISPCompany offers two types of services:

  • Mail serving and Web serving: infrastructure server consolidation of clients' low-utilized servers. (See Figure 4-3.) ISPCompany can put up to several hundred of these servers on one mainframe and can save substantially on overall cost. There is no need for stand-alone machines and their respective cables.

    Figure 4-3. ISPCompany infrastructure server consolidation

  • Outsourcing: complete or partial hosting of other companies' computer systems on the mainframe. ISPCompany offers a spectrum of services that ranges from supplying the bare iron through providing applications, helpdesk, and backup/restore services.

See Appendix A, "ISPCompany" for a more complete story of ISPCompany's development.

In our example, ISPCompany has been approached by a client to outsource its small server farm. This opportunity for ISPCompany is handled by creating a new set of unique Linux images needed by this client. This is an example of horizontal growth.


4.3 StoreCompany and its business application

StoreCompany is used in this book to demonstrate issues of security, performance, and system management. StoreCompany is the larger of the two example companies. StoreCompany is an established retail company that is looking at using excess capacity to test a new business opportunity.

StoreCompany illustrates two different possibilities for using Linux on the mainframe:

  • Back-end integration (Figure 4-4)

    Figure 4-4. StoreCompany's back-end integration

  • Vertical growth

StoreCompany, a leading department store chain with 85 stores, has been in business for decades, mostly in large cities in the eastern United States. Their IT organization has been very progressive over the years, moving from back office and support functions to actively managing sales and inventory. They have also become involved in business-to-business type transactions with their key suppliers.

Our examples demonstrate the growth of the mainframe side of StoreCompany's IT division as Linux enters the picture in a new business endeavour. After integrating and growing Linux applications on the mainframe, the logical setup of StoreCompany will be as in Figure 4-4 (simplified).

See Appendix B, "StoreCompany" for a more complete story of StoreCompany's development.

StoreCompany is looking to test a new business opportunity. The suggestion is to use spare IFL capacity on the mainframe that hosts a Web-based ordering service and add a catalog of gifts (called "One-of-a-Kind" or OaK). The catalog is based on the inventory database and customer database already in place on the mainframe. Most of the changes related to the implementation of this will turn out to be in the software. If the experiment turns out well, more IFL capacity will be added to a new business intelligence application.

This application will then grow vertically, and the new business opportunity will take off. The hardware for the new business opportunity is mostly in place. See B.6, "Project 3: OaK project" for more details.


4.4 Summary

In this chapter, we have presented two companies with typical Linux projects. ISPCompany supports the infrastructure of various customers with Linux images on the mainframe. This enables ISPCompany to offer lower rates to customers. ISPCompany grows horizontally, adding new Linux images on its mainframe.

StoreCompany, on the other hand, is integrating Linux into the fabric of its enterprise as a system for rapid deployment. Linux on the mainframe gives StoreCompany the opportunity to be very responsive to demands for new applications at a very low risk. StoreCompany will create new Linux images as new prototype projects require. Projects that succeed and grow can justify any additional IFL capacity. Projects that fail simply have their images deleted, and others can then use the resources.


Chapter 5. Sample Projects

After talking to customers who have finished at least one successful Linux-on-the-mainframe project, we realized that they all followed some basic project management rules. In this chapter, we will share our findings with you.

Some of the material may seem obvious to the experienced project manager, but it is surprising that a large number of projects fail to meet their original objectives because the simple rules-of-thumb presented here were not followed. Thus even the experienced may find this chapter useful as a quick review before moving on to the remaining parts of the book.

The trick is to choose the right project and use the opportunity of fast deployment that Linux gives you. In this chapter, we discuss the following questions:

  • What should a Linux-on-the-mainframe team look like?

  • What is a suitable project?


5.1 Building a team

In numerous project management classes, building a team for a project is seen as critical. It may be the first thing you consider, even before selecting a specific project.

On the Linux-on-the-mainframe team, you will need people with the right mix of technology skills. This might mean that your team members come from different departments across the company. First, the team will need general IT skills in the form of good system management and control processes. This skill is independent of their technical background and can be provided by people with either a mainframe or a Linux and UNIX background.

Second, the team will obviously need someone who knows how to prepare for a Linux image on the mainframe. Skills that such a person might have include setting up a mainframe, operating it, partitioning it, defining I/O, and setting up z/VM or LPAR.

Last, but not least, the team will profit from someone who has Linux, or at least UNIX, administrator skills.

Apart from these skills related to the technology used, you will also need some skill in the area of the project chosen.


5.2 Choosing the right Linux project

People tell us two things about Linux on the mainframe projects:

  • It is easy to get started.

  • Using a staged approach works best.

5.2.1 Start small

Our advice is to start small. The bigger the scope of the initial project, the more the entire company is involved, and the harder it will be to succeed. What would constitute a typical small project? In our experience, successful companies started with things like consolidating an infrastructure server on the mainframe or simple (static) Web serving. Choosing a low-attention project allows the team some success without the pressure of company politics. Once some success with Linux is established, that can be used to calm the fears of any doubters. It is important with new technology in a large company to have an early success. Even if the Linux project is simple, it gives you the proof that the technology works and the benefit of undisputable savings (for example, you did not have to buy another server). Once you have the proof that the technology works, you are ready to take the next step. StoreCompany started off with moving the firewall and proxy servers to Linux (see B.4, "Project 1: Firewall and proxy server"). That was a project of limited scope that proved successful.

The second reason for starting small is authority. The Linux-on-the-mainframe team needs the authority to move forward and to use the necessary resources. Big projects need significant resources, which again is often difficult to procure for as yet unproven technology in a company.

5.2.2 Initial server consolidation

For some, this level may be as far as they want to go. Here you might realize some hardware savings because you have consolidated servers on to the mainframe. This is also called simple server consolidation, taking the work from several servers and consolidating that work on one server. See Figure 5-1.

Figure 5-1. Simple server consolidation

With Linux on the mainframe, we continue to use the term server consolidation. It also means moving as much work as possible to one server. With Linux, what is often moved to the mainframe are whole images, one by one, including all the resources they need. Often this is done with the help of z/VM, in which case VM controls these guest systems.

Using Linux, this "work migration" is typically simple. There are no worries about how to pack all the things together, and usually the application we are talking about already exists on Linux. Thus, you do not need to migrate code from another operating system or talk your vendor into doing it. The only change that is made is where the image is hosted. Instead of on real, separate hardware, it is now on virtual hardware (LPAR and/or z/VM).

In this case, there frequently are savings on the hardware side, assuming that enough servers are consolidated. How many servers are "enough"? For example, one department store colocated 50 servers by going to Linux on the mainframe. The break-even point depends on your environment.

While what you include in a business case can vary from hardware and software costs to efficiency and ease of management, with Linux on the mainframe you can often make your case. What you have gained is a simplified setup, easy scalability, and improved availability.

Example

If there is already a mainframe present in the enterprise, then to achieve such a consolidation, the price of an Integrated Facility for Linux (IFL) should be considered. If a number of low-utilized servers, N, can be consolidated, that might make up the price for one IFL (see Figure 5-2).

Figure 5-2. Server consolidation cost example

Although this example is oversimplified, it illustrates why most of the larger consolidation cases you may hear about are on z/VM. With LPAR, you can have at the most 15 logical partitions in a z900 or z800, and with one server per partition, that is 15 servers. Under certain circumstances, even fewer images can pay off as a consolidation case.

What if you do not have a mainframe to start with? If you have to buy a whole new machine (say, a model z800 0LF that has one CPU), the cost will obviously be higher than adding one IFL to an existing mainframe. In this case, you need more Linux servers to consolidate for break-even. But the incremental growth path is attractive because if you need more capacity, then an additional CPU does not incur a great cost. By the way, the more Linux images you have, the more hardware sharing (such as sharing LAN adapters) will help save on cables and routers.

5.2.3 Full Linux integration in an enterprise

This second level of server consolidation is more complex because it involves integrating a Linux middle-tier with the legacy back-end systems. Because you have to use middleware and connectors and the like, this is a more complicated project. There are tuning issues as you allocate resources to the back-end and to the middle-tier. There are relative priorities that must be set to get the work done, and so on. Here IBM's WebSphere family of products can be effectively used for developing Linux applications and deploying them on the mainframe. For more information, see http://www.ibm.com/software/info1/websphere/index.jsp.


5.3 Summary

In this chapter, we shared with you some of the lessons we learned when talking with companies that have introduced Linux on the mainframe. Getting started with Linux on the mainframe is easy. We discussed building a team and the importance of choosing a suitable small project to start with. We mentioned cost and break-even considerations.

Customers tell us that Linux on the mainframe helps them streamline operations and simplify management. The mainframe might be more cost-efficient and easier to run than a physical server farm. For details of customer stories, see http://www.ibm.com/servers/eserver/zseries/os/linux/stories.html.


Chapter 6. Total Cost of Ownership: the Challenge

Numerous papers have discussed server consolidation in the last few years. The articles illustrate a renewed interest in the mainframe and make a clear financial case for server consolidation and for moving applications to the mainframe. The bottom line is that the initial cost of a mainframe is often counter-balanced and outweighed by long-term profitability. This especially applies to Linux on the mainframe.

If you are considering a project that involves Linux, you will already have a good idea of the business value to your company. To assess the return on investment for the project, you will need to balance the value against the total cost of ownership (TCO). This chapter discusses these questions:

  • What are factors to consider in a TCO equation?

  • How can the mainframe lower your TCO?

  • How can Linux lower your TCO?

  • What are examples of how companies save by deploying Linux on the mainframe?


6.1 Total cost-of-ownership factors

As indicated in 4.3, "StoreCompany and its business application," StoreCompany is looking at setting up a new Web catalog with third-party items that StoreCompany would not have to stock. Now it is up to you as "decision maker" to decide whether to give this new idea a try. Naturally, you ask, "What is the cost of this opportunity?" Linux on the mainframe brings some interesting factors to the TCO calculation.

Total cost of ownership attempts to give a picture of total cost of an IT system over a period of time, typically three to five years. A TCO analysis is especially of interest when you are considering any significant change, such as evaluating a new business opportunity, upgrading your servers, or moving locations. This can be contrasted with total cost of acquisition, which takes into account only the cost of buying new machinery and associated software but does not include any other costs.

A TCO study is made up of a multitude of factors. Some are quantifiable, such as the typical factors listed in Table 6-1. Other factors are of a more hidden nature and are listed in Table 6-2. Some factors have a particular importance when you are considering the combination of Linux and the mainframe.

In short, calculating TCO is a complicated business, and every organization ultimately must come up with its own method.

Table 6-1. Typical TCO factors (cost of acquisition)

Quantifiable cost factors

Hardware:

  • CPU, storage

  • Disks

  • Network, cables

  • Maintenance and support

Software:

  • Operating system

  • Middleware

  • Systems management programs

  • Application software

  • Maintenance and support

Occupancy:

  • Space

  • Power

  • Lighting, heating, cooling

People, or full-time equivalents (FTEs):

  • Operators

  • System administrators

  • Data administrators

Services:

  • Contract costs for new projects.

  • Consultants to guide the project design and deployment

Opportunity factors:

  • Cost of money (the interest on the loan you took, or the stock you issued)

  • The deferred costs of not pursuing other opportunities (this gets into the risk of prioritizing business choices)

A typical TCO equation thus sums the cost of hardware, software, occupancy, people, and services. In addition to this, you may want to consider costs of a more hidden nature such as those listed in Table 6-2.

Table 6-2. Hidden TCO factors

Hidden cost factors

Availability factors:

  • Duration and frequency of software outage

  • Cost of backup and restore

  • Cost of service, debugging, and repair

Hardware:

  • Reliability of hardware platform

  • Flexibility

  • Freedom of choice (hardware and software vendor)

Opportunity factor:

  • Time-to-market

  • Favorable mentions in the press or influential reference customer

Let us go quickly through the typical TCO factors before we look at what Linux and the mainframe bring to the equation. Note that we will not give you exact numbers for any of the parameters, because every one of these varies throughout the world. For example, labor costs are not uniform across the globe.

6.1.1 Hardware

Hardware usually includes:

  • Servers, including not just the machines you need for production, but all of the associated machines, like the backup, development, failover, network infrastructure (firewalls), and test machines.

  • Disks and tape (storage), including the data backup for this.

  • Network attachment (interface cards and routers and the cables, including fiber).

A physical server costs the same whether its utilization is 100% or 10%. Chances are that not only the test and supporting servers are low-utilized. Typically, the utilization of the production servers is in the 10 – 20% range. It may be worth looking into consolidating that whole set of servers on z/VM. Assuming you do not have work that requires new, dedicated capacity, the virtualization technology offered by z/VM can save you real hardware, such as CPUs, disks, and cabling.

6.1.2 Software

Software cost includes acquisition, plus service and support of the programs used on the system:

  • Operating system

  • Middleware

  • Systems management programs

  • Applications

Additionally, software costs generally include the backup software and essential network services such as directory services, messaging, routing security, and network management.

While it is possible to build a Linux system from sources on the Internet for free, this may not be suitable for production systems. With Linux, the operating system cost may be low, especially if the price is based on the number of processors you have (rather than the number of images). The cost of a distribution today may include maintenance.

In fact, for all of the software, you have the option of building it yourself (the cost will be in the development), buying it (and getting the service that goes with a contract), or using Open Source (some cost will be in the support, either in a support contract or as personnel cost).

6.1.3 Maintenance and support

The purpose of a support contract is for an enterprise to have access to a problem reporting mechanism (usually a phone number), including an assurance that problems reported will be fixed. Some software providers offer a single number to call for all of the products in the solution. In other words, they accept responsibility for the solution. Others cover only the products they sell.

The cost of support will be a trade-off between what you want to do yourself versus what the service contract should cover. Part of your TCO analysis might be to decide on the trade-off. It is the time of your staff versus an expense. The range of possibilities in the Linux space is wide. If, for example, you are happy with using mailing lists to find solutions to problems in Open Source programs, the cost of service can be low. You can decide on an umbrella contract that covers everything, such as IBM Global Services offers. Most Linux distributors offer support as well.

6.1.4 Services

Services involve a contract or contracts with a consultant or others who are hired to do something, such as develop a tool. An example of services for Linux on the mainframe could be getting z/VM operational, if you don't have the skills available. Or, if you don't have Linux skill, you could contract that out until your people get their education. Or you could get one of the distributors to come in and set up Linux for you.

You have a lot of control over services. The question is, for the particular solution you are considering, how many services are you going to buy to get the solution up and running?

6.1.5 Occupancy

Servers must be housed somewhere. The cost of the raised floor in square feet, space for access and the control center, and power are usually included in an occupancy parameter for TCO.

Part of what used to make the mainframe a "dinosaur" was the cooling, the raised floor, and the cabling. Today, a mainframe uses less of almost everything than a typical server farm. A mainframe no longer needs a raised floor and has a moderate footprint, as shown in Table 6-3.

Table 6-3. Mainframe (zSeries 900) space requirements

Minimum configuration, 1 frame

Maximum configuration, 2 frames

Weight

917 kg (2021 lbs)

1866 kg (4113 lbs)

Footprint

1.32 sq. meters (14.2 sq. feet)

2.81 sq. meters (30.3 sq. feet)

Height

200.4 cm (79.8 inches)

200.4 cm (79.8 inches)

Given the availability of high-speed fiber for interconnecting sites, the cost of occupancy may be a less critical factor in the future. You now have more choice as to where you want to locate the equipment. However, consolidating football fields of servers on a mainframe can mean substantial savings in square feet and power (power to run the machine, to cool the machine, power for your staff working there, etc.).

If you think about a typical low-end server (such as a PC) consuming, for example, a kilowatt, and the biggest z900 mainframe consuming around 15 KW and being capable of functionally replacing numerous low-utilized servers, the cost savings in power can be dramatic.

Example: While a typical configuration of a server farm can cost hundreds of dollars a day in electricity to run, a single z900 running the same workload can be an order of magnitude cheaper. The savings are even more dramatic when the floor space requirements of a server farm are considered. The average server farm can require up to some 10,000 square feet of floor space compared with the much smaller footprint of the mainframe including its peripheral devices, as shown in Table 6-3.

6.1.6 People

Personnel cost is normally measured in units of full-time equivalents (FTEs).

Personnel typically included might be operators, system administrators, and data administrators. The personnel cost can be calculated by multiplying the number of FTEs with a current average salary.

How many machines or servers or how much data one person can handle often depends on the type of task the servers are doing. FTE averages that are sometimes used are shown in Table 6-4.

In general, the more similar the servers in a server environment are, the more servers one FTE can manage.

Table 6-4. Different types of server tasks require different numbers of people per server

Server task

Number of servers per FTE

Large application server, running a CPU-intensive job such as business intelligence or ERP.

1.5 to 3 servers, with the average IT shop having around 2 servers per person.

Small application server, running more specialized applications such as accounting packages or HR programs.

2 to 4 servers, with the average IT shop supporting 3 servers per person.

File and print characterized by very low utilization rates.

10 to 40 servers, with the average person managing 20 servers.

Firewall servers typically characterized by very low utilization rates.

10 to 40 servers, with the average person managing 20 servers.

E-mail servers.

5 to 50 servers supported by each FTE. On average, 10 servers running a typical e-mail application can be supported by each FTE.

Web/Internet applications.

10 to 40 servers running a Web application can be supported by each IT staffer, with the average IT shop managing 15 servers per FTE.

Systems management involves applications that simplify network and applications management such as Tivoli Business Systems Manager.

1 FTE to support 5 to 15 servers. The typical IT staff member could manage 7 systems management servers.


6.2 The mainframe and the TCO equation

What are the mainframe attributes that make it valuable from a TCO perspective?

6.2.1 Availability factors

To assess the impact of an unavailable system and hence what it is worth to you to have the system available, you should consider these factors:

  • Staff productivity— If the system is down, parts of your staff cannot perform their jobs.The cost depends on the duration of the outage and how many employees were impacted. How long does it take before the system is up again, and what does a person-hour cost you?

  • Damage to the company brand— If the outage is spectacular enough, it will make the news.

  • Effect on the share value and hence the value of the company.

Sources of outages generally include software errors and hardware errors. The goal of mainframe hardware development is to have no downtime. Most changes to I/O and most memory upgrades can now be done on the fly, as can some CPU updates.

Statistically, outage-causing errors are most often found in the applications. Just as you decide what hardware to use, you decide what applications to use, whether to buy them, build them yourself, or use Open Source. Choosing software carefully can help in maintaining good availability.

6.2.2 Mainframe availability characteristics

Availability involves reducing the duration and frequency of outages.

The mean time between failures (MTBF) for the mainframe of up to 30 years can be attributed to the mainframe's unique combination of availability features (refer to Chapter 2, "Introducing the Mainframe" for details):

  • Redundancy is the most important contributing factor for availability, and it is also the reason why the mainframe is expensive from a cost-of-acquisition point of view. Redundancy gives the mainframe the ability to dynamically replace, frequently with no impact on running programs, failing hardware units like CPUs or memory.

  • Error correction code (ECC) is provided in all levels of memory and checking is provided on all buses. When errors are found in either data or logic, they are corrected on the fly, where possible.

Relevant chapter in book: Chapter 11, "Achieving Higher Availability" and Chapter 13, "Availability Management."

6.2.3 Partitioning and virtualization

The value of Linux on the mainframe is in large part due to the possibility of hosting multiple servers on a single hardware machine. On the mainframe, there are two methods for hosting multiple operating systems: logical partitioning (LPAR) and virtualization using z/VM. The mainframe offers the most mature and dynamic partitioning and virtualization technology in the industry.

zSeries LPAR implementation is unique in comparison to the other partitioning implementations available from other hardware vendors. LPAR can exploit the PR/SM microcode, which provides a flexibility superior to that of a static hardware solution. Logical partitions each have their own allocation of memory, and either dedicated or shared processors, as well as dedicated and shared channels for I/O operations.

z/VM presents a unique virtualization approach. It provides each end user with an individual working environment known as a virtual machine. The virtual machine simulates the existence of a dedicated real machine, including processor functions, storage, and I/O resources. For example, you can run multiple z/OS and Linux on zSeries images on the same z/VM system that is supporting z/VM applications and end users. As a result, application development, testing, and production environments can share a single physical computer.

Relevant chapter in book: Chapter 7, "The Value of Virtualization."

6.2.4 High utilization rates

The average server farm tends to have low CPU utilization. If you have low utilized Linux (or UNIX) servers, you could run them all on one mainframe, which would then run at a higher utilization. Using Linux guests on z/VM gives you the capability to utilize even more such logical servers. With z/VM, you can also manage the priority of the guests and provide more resources to important guests in a peak situation.

The CPU has historically been considered the "expensive" part of a (mainframe) computer, and the part to utilize fully. The rest of the system should feed the CPU work so that it can run at nearly 100%. It takes clever design to build a fast CPU and the supporting I/O and memory structure to allow "normal" work to be able to drive it to 100% capacity. When we talk about balance, we mean that memory and I/O support the CPU.

Thus, in utilizing a mainframe, it is not so much a question of how high the load can be (it can even be 100%). The question is, is there enough workload to keep it busy?

Relevant chapter in book: Chapter 7, "The Value of Virtualization" and Chapter 15, "Performance and Capacity Planning."


6.3 Linux and the TCO equation

Let us look at what Linux considerations can change the TCO equation to your benefit.

6.3.1 Opportunity factors

The importance of nearly 100% availability may argue for many projects being done in z/OS. However, when fast time-to-market is essential, Linux on the mainframe may come into play because for some applications Linux can have adequate availability characteristics while giving you the following:

  • Technology (and applications) not available on z/OS that you can exploit for time-to-market. (See Part 5, "Running Applications" for a discussion of the applications and middleware available.)

  • Opportunity for rapid change

6.3.2 Time-to-market

Time-to-market has become one of the most important attributes a company can have. If the IT infrastructure is impeding development (or whatever is important to your business, such as marketing), it can be costly in lost business. If this is the case, you might want to consider Linux. The benefits of Linux can be summarized as follows:

  • New applications can be available more quickly.

  • Applications are simpler to port, as many applications are developed on Linux or UNIX.

  • Many applications are available from the Open Source community.

Another kind of time-to-market is the possibility to provide new images quickly to the customer by using z/VM and cloning techniques.

Relevant part of book: Part 5, "Running Applications."


6.4 ISPCompany: TCO considerations

6.4.1 Software

ISPCompany buys most of its software through site licenses (when these are available) and shares savings with its clients. The benefit to their customers is that they get a larger choice for less cost to them.

ISPCompany offers a choice of two distributions to their customers: SuSE and Red Hat. The more consolidated servers there are, the cheaper they become. The more standardized the servers can be, the easier they are to maintain and create.

6.4.2 Occupancy

ISPCompany pays as part of its overhead for the IT infrastructure and utility costs. Its clients do not need the space and utility costs (and the headaches of redundancy, and so forth). They indirectly just pay a share of the costs.

Until the typical company grows big enough to have multiple sites, the IT site and personnel are usually kept where it normally does business, which tends to be a high-rent district. ISPCompany is big, and it is a business for businesses. Hence it does not have to locate in a high-rent district. It also has the ability to attract and keep more talent because it has more job opportunities in IT.

ISPCompany and its clients can save considerably when consolidating servers on the mainframe.

Refer to Appendix A, "ISPCompany" for more details on occupancy.


6.5 StoreCompany: TCO considerations

6.5.1 Software

StoreCompany's software costs will only increase when it adds IFLs or new priced software. In our OaK example, software costs are only an initial cost (probably mitigated by a test license) increase for their DB2 UDB. Software cost always depends on the license conditions. The other software costs are flat until the extra IFL capacity is brought online. And that cost is split amongst all the Linux images!

In our StoreCompany example, a clone can be made of an existing Linux image and some new code needs to be written. The new code manages the presentation of items in the OaK category, assuring that the order is sent to the right artist to be filled, making special offers to customers fitting a certain profile, and serving dynamic Web pages.

6.5.2 Hardware

When trying out a new project, one first tries to use spare capacity. On the mainframe, all spare capacity is aggregated into one machine. The amount of spare capacity to try out new ideas can mean avoiding new hardware acquisitions.

If one adds more capacity for a specific application and it turns out that the application does not need it, it is simple to reconfigure that capacity to the other images. If the new business opportunity does not work out well for StoreCompany, it is easy to reconfigure the system and use any excess capacity for other Linux applications. In contrast, other design implementations may mean that StoreCompany's business intelligence application would be implemented on a stand-alone machine. If the number of visitors to the new catalog turns out to be low, machine resources would be "wasted."


6.6 Summary

It is clear that the cost versus benefit equation for investment in server infrastructures extends well beyond initial acquisition. TCO deals with fundamental variables of IT effectiveness and business performance. Linux on the mainframe offers a highly reliable hardware platform. A large number of low-utilized Linux images can be consolidated with little effort. You can manage them flexibly using the resource-sharing techniques of the mainframe.

Linux on the mainframe is one alternative for implementing your project. You could also choose to implement an application using, for example, z/OS. The decision comes down to the alternatives available and the cost of each. Not only may a project have value in and of itself, but additional value may be realized when using Linux on the mainframe:

  • Using Linux as a development platform may give you a considerably shorter development time than other solutions and may offer revenue from a project sooner.

  • z/VM and Linux make it possible, for example, to create new Linux images in a matter of minutes instead of days.

What factors you include in a TCO for a Linux-on-the-mainframe project will depend on your model of IT value.


Chapter 7. The Value of Virtualization

Virtualization provides valuable support for running large numbers of Linux images on the mainframe. IBM mainframe virtualization technology is found in the z/VM product and in logical partitioning (LPAR). These technologies provide the infrastructure that enable, on one hand, numerous Linux images that use only small amounts of CPU, and, on the other hand, a few Linux images that consume large amounts of CPU to be hosted on a single mainframe. Such deployments might save your company money.

z/VM is the zSeries Virtual Machine operating system. This chapter explains why z/VM is ideally suited for large-scale Linux deployments and server consolidation. z/VM's ability to run hundreds of Linux images on a single machine opens a multitude of business possibilities. Because the powerful mainframe virtualization technology is little understood outside the mainframe world, this chapter should be especially useful for readers not familiar with the mainframe.

In this chapter, we discuss these questions:

  • What is z/VM, and how did its virtualization technology develop?

  • How does Linux run on z/VM?

  • What are the possibilities and benefits of running Linux under z/VM or on an LPAR?


7.1 What is z/VM?

To better understand what is going on today, it is sometimes helpful to revisit the past. In this section, we examine what drove the creation of z/VM and why it plays such a key role in Linux on the mainframe.

Virtual machine (VM) technology development started with the IBM System/360 in the mid-1960s. One reason for developing it was the need for testing facilities. Many software programmers could thus test new programs on the virtual hardware simultaneously.

Another factor driving VM development was the need for multiuser systems. Computers were large and costly. Thus, a need arose for an operating system environment that allowed multiple users to use one machine simultaneously in real-time.

To support testing and multiple users, researchers developed the unique concept of virtual machines. In the virtual machine model, the hardware resources of a computer system are managed by a hypervisor. VM's hypervisor is called Control Program (CP). Users' activities are managed by an operating system. When users log on to VM, CP creates a unique guest virtual machine for each user. Depending on the guest definition, CP then allows the user to start up an operating system within that guest.

The first commercial VM product offering from IBM, in 1972, was the VM/370 product. Apart from CP, it included a special-purpose operating system called CMS, the Conversational Monitoring System. CMS has been a component of VM products ever since the delivery of VM/370. The purpose of the CMS operating system is to provide multiple users with a powerful yet simple interactive interface that performs extremely well. Performing well means sub-second response time for hundreds of CMS instances on a single machine. The design for simplicity extends to all aspects of CMS, including the command set and file system. The CMS that offers the VM community a rich set of programming tools and utilities can help manage and automate a virtual Linux server farm running on the current VM product, z/VM.

Guests on z/VM are zSeries operating systems in their own right. The CMS operating system today is different from the other guests in that it no longer runs native on the hardware. The use of CMS has changed from being VM's only interactive end user operating system to also being a home for utility-like functions. For example, TCP/IP or performance management functions can be run in a guest of their own and can be used by other guests.

Similarly, in a Linux-on-the-mainframe environment with many Linux guests, CMS is where scripts get run to automatically restart failing images, or manage switch-over to a hot standby. A system administrator uses CMS to create new images (or CMS is the programmatic interface where the request for the creation of a new Linux image is routed).

Over time, IBM has made investments in hardware, architecture, and microcode, as well as in the VM product itself, to enhance the virtualization technology available with each successive line of mainframe computers.


7.2 How Linux can run on z/VM

At its core, the z/VM operating system is a hypervisor. The hypervisor can present virtual copies of the underlying hardware resources that it controls to operating systems running on a virtual machine. Users can run multiple images of other operating systems as "guests" of the hypervisor, sharing the same single set of real CPUs and I/O facilities, as shown in Figure 7-1.

Figure 7-1. Virtualization technology with z/VM. Resources can be virtualized and presented to the user as if each user had his or her own machine.

Any operating system that can run on an S/390 or z/Architecture mainframe can also run under the control of z/VM, that is, in a z/VM-created virtual machine. z/VM faithfully represents the complete mainframe architecture to the guest operating system.

Like any other operating system that runs on the mainframe architecture, Linux runs on z/VM without any further changes.


7.3 What does z/VM provide?

z/VM is a mainframe operating system. Linux on the mainframe can particularly benefit from what z/VM provides:

  • The hypervisor.

    z/VM's CP component is the hypervisor that manages resources for its guests. The hypervisor includes all the emulation code and an extremely efficient scheduler. The fact that hundreds of users can work concurrently and expect sub-second response times shows the efficiency of the scheduler.

  • The CMS operating system.

    CMS has a simple user interface and is used by system administrators to manage both z/VM itself and its guests. It is also used as a utility operating system for various server functions such as TCP/IP.

  • A large set of systems management tools.

    In the z/VM directory, system administrators can quickly and easily change the definitions of a guest or create a new guest. z/VM has both a user interface and a command line interface that can be driven by scripts to control the entire environment. z/VM's system administration facility includes functions that help the user create and manage multiple Linux images. Functions include assigning disks, starting and stopping Linux images, and more. System and hardware resources can be allocated among multiple Linux guests, which can also be managed by z/VM.

    A scripting language (REXX) allows for programmatic response to events; for example, you can use the programmable operator (PROP) to filter messages from the z/VM operator console. You can specify a REXX program (called a PROP exit) to be invoked whenever a guest logoff message appears. The message includes the name of the guest that terminated. The exit could then take whatever action was appropriate, such as restarting the guest. There are also extensive logging capabilities for performance tracking or debugging.

    z/VM has a large, active user community (much like the Open Source community) that provides tools and scripts for managing a z/VM environment.

  • Security and integrity.

    z/VM is built on the mainframe architecture that highly values those attributes. We examine z/VM security and integrity in Chapter 8.

To understand the difference of z/VM compared to other virtualization techniques, we need to take a look at some technical issues.

Importantly, z/VM allows its guests to take advantage of the underlying mainframe architecture. In the interest of achieving good performance, z/VM guests run as much as possible as if the hardware were real. Only when z/VM must run privileged instructions for the guest does it take control from the guest.

In 1985, IBM announced and delivered a microcode assist for its 3081, 3083, and 3084 processor models called Start Interpretative Execution (SIE). The SIE assist reduced the number of instructions VM had to execute on behalf of a guest operating system. With SIE, the processor (in interpretative execution mode) could handle more guest I/O instructions and associated interrupts. Less VM intervention was required, thus significantly improving guest performance.

The distinguishing factor of z/VM is that all resources can be virtualized. CPU, memory, disks, and network communications can be virtual (see Chapter 10). Additionally, certain hardware that is supported on some mainframe models can sometimes be virtualized on other models. For example, it is possible to use virtual HiperSockets on machine models that do not support real HiperSockets (such as G5, G6, and Multiprise 3000 servers).

In summary, the guest operating system believes that it owns and controls the hardware. CP allows the guest control until an instruction occurs that CP must interpret. From the outside observer's perspective, there is a "duet" of the CP and the guest.


7.4 What is logical partitioning?

Once VM was generally available in 1972, some companies used it to partition machines in order to run different workloads in different partitions. It turned out that for some companies, a simpler partitioning scheme would be just as useful. As a result, some fundamental virtualization concepts were included into the architecture. These concepts became the hardware hypervisor Processor Resource/Systems Manager (PR/SM) and the logical partitions (LPARs) into which PR/SM could divide a machine. A zSeries 900 or 800 mainframe can run in one of two modes: basic mode or LPAR mode. In basic mode, the entire machine is under the control of a single operating system. In LPAR mode, you can logically divide the machine into partitions so that multiple operating systems can run concurrently.

LPAR partitioning helps companies to exploit the resources of the machine better than if there were no partitioning. LPAR allows companies to have a single machine support both production and test requirements. Initially, companies would bring up the test partition only at off-peak times.

The PR/SM hypervisor of the zSeries 900 can manage up to 15 LPAR "guests." The CPU and channel paths are the two resources that can be logically shared. Central storage must be physically partitioned and assigned to each active partition.

Although the virtualization technology that is best known in the context of Linux on the mainframe is z/VM, there are some occasions where you might consider using an LPAR:

  • You might want to take advantage of the cost savings available with Integrated Facility for Linux (IFL). The IFL feature CPUs can be assigned only within an LPAR that is dedicated to Linux, or Linux running on z/VM (see 21.4, "Integrated Facility for Linux (IFL)").

  • You might want to have just a few Linux images. Then you might not require the ability of z/VM to handle hundreds of images.

  • You might require the certified security offered by LPAR. Typically, you might want to run a few Linux images each in their own LPAR (Figure 7-2). An independent authority has certified LPAR to confirm that LPARs are isolated from one another (see 23.1.1, "LPAR certification"). Depending on your corporate audit policy, a firewall might be an instance where you want an individual Linux in its own LPAR.

    Figure 7-2. Two LPARs, one running z/OS, the other running Linux

Companies can utilize their mainframe with more flexibility using LPARs than using native mode. However, LPAR provides far less flexibility and function than what is available with z/VM.


7.5 Why run Linux on z/VM?

The interest in Linux on the mainframe can be attributed to three things:

  • As a source of new technology for mainframe users

  • As an opportunity to reduce complexity when deploying an integrated server environment

  • As a possibility to host large server farms

z/VM is ideally suited to hosting an integrated Linux server environment, letting companies run applications across multiple images on a single hardware platform. Information technologists can exploit the benefits of having numerous images, each running only one application, and yet limit the costs associated with having unique hardware for each image, as shown in Figure 7-3.

Figure 7-3. Application images can be brought closer to the back-end z/OS (integrated environment)

Many years of product development have made z/VM ideally suited today for Linux deployments. Through the years of product development, z/VM has been designed to support:

  • Production systems. Typically, these are single-image operating systems or applications consuming large amounts of CPU, for example, VSE/ESA, or SAP R3 application servers.

  • Interactive users. z/VM can support hundreds of CMS users.

  • Test environments. These environments can be used for migration, or to test different operating system levels before deploying one.

7.5.1 Why z/VM is ideally suited for hosting large application Linux images

Even today, companies running large z/OS or VSE/ESA systems as guests of z/VM typically have only a few images (maybe just two: one for production and one for testing). The demand on z/VM is to maximize the amount of work that the z/OS or VSE/ESA guests can perform. In the past, this required VM developers to be very efficient in their support of new technologies such as large real memory, advanced disk storage systems, and processor facilities. The goal for supporting guest systems is to give each guest what it needs and get out of the way.

A characteristic of the z/VM operating system that benefits a Linux server environment is the high degree of resource sharing that is achieved with VM's CP. z/VM facilitates sharing processor capacity, main memory, disk devices, channel adapters, network adapters, and practically anything you can attach, connect, or plug into your mainframe among the virtual machines. Note that z/VM allows defining of guests dynamically; no static partitions need to be created ahead of time.

At the same time, if a guest needs sole access to a processor, device, or network resource, that resource can be dedicated to the guest. A hallmark of z/VM is its flexibility in supporting a wide variety of computing requirements. System administrators can mix and match shared resources with dedicated resources on the same z/VM system, and even in the same guest, changing the landscape of the virtual machine configurations, if needed, in a matter of minutes, not hours or days.

7.5.2 Why z/VM is ideally suited to Linux server consolidation

CMS workloads in the past were typically multi-image, interactive workloads. Many companies were deploying hundreds of CMS guests on their VM systems and were asking IBM for enhancements to support that workload. Today, Linux workloads tend to be spread across multiple images and be highly interactive, such as Web-serving workloads. Because of the similarities between Linux workloads and traditional VM workloads, the provisions made to accommodate these workloads also work well for Linux.

The refinements made to VM's scheduler and dispatcher over the course of decades are equally suited for a Linux server environment. This kind of environment requires the facilities to host a large number of images with an unpredictable set of workloads that must meet stringent response-time objectives.

The appeal of z/VM's resource sharing becomes obvious when one considers the amount of wasted resources that typically exist in a discrete server farm environment. It is not unusual for individual servers to have, on average, a utilization rate as low as 3%. Servers with such low utilization are, for example, domain name servers (DNS) where data does not have to be looked up often. A company can have many DNSs. ISPCompany, for example, could have a DNS for each client of its outsourcing business, which adds up to hundreds over time. In general, utilization rates depend on many factors, including:

  • The function provided by the application

  • Its I/O requirements

  • The networking bandwidth

  • The number of users or clients who access the application

  • The level of technology available in the server itself

Many network infrastructure servers, such as DNSs and firewalls, experience short bursts of activity followed by periods of inactivity. When a server is inactive, its resources are unavailable to other servers that could benefit from additional processor capacity, real memory, network or I/O bandwidth, or disk space. Purchasing server resources to handle application peaks compounds this problem. Processor speed and technology, memory requirements, and all other tangible capital costs associated with a server application are generally measured by how much capacity an application needs when user demand is high. To provide any less resource means potentially falling short of the computing capacity which the application needs at critical times. Unfortunately, this only compounds the waste of system resources when the server is idle or underutilized. (See 3.3.2, "Utilize CPU power.")

In a z/VM world, a server running in a virtual machine can grow or shrink its consumption of available system resources based on application demand. For example, processor cycles that would otherwise be wasted during times of low utilization are available for other images running on the same z/VM system. When resource requirements in a guest exceed expectations, z/VM lets a guest get more cycles or be backed by more real storage, provided no one else with higher priority needs it. Using CP commands, a system administrator can allocate more processor capacity, disk storage, or network bandwidth to the virtual machine when it needs it, not days later. This is possible provided that the virtual resources are available. (It usually takes days to provision a real server to replace the one that no longer meets the customer's need for computing power.)

Figure 7-4 shows many servers with low utilization being consolidated on a mainframe under z/VM.

Figure 7-4. Many servers with low utilization can be consolidated on z/VM

There are other costs associated with separate, seldom used servers. When a server goes idle, it still requires energy, floor space, and other environmental costs such as cooling to maintain its connection to the network and be ready for its next transaction. A guest on z/VM does not represent incremental environmental costs, whether active or idle. The environmental cost of a z/VM system is fixed for a given configuration of hardware and wastes resources only if the server farm in aggregate becomes idle or has low utilization. Studies have shown that environmental cost savings can be realized when running a multi-image server environment on z/VM versus a discrete system solution.


7.6 Summary

On a zSeries machine, using LPAR and z/VM for Linux deployments presents a multitude of possibilities not available anywhere else. There can be substantial hardware and floor space savings from using virtualization technology on a zSeries machine. Virtualization technology gives you a flexible server farm environment that can be easily managed and changed in minutes with a few keystrokes.

The purpose of z/VM's efficient virtualization technology is to:

  • Allow operating systems to run almost as if they were running on the hardware.

  • Maximize the logical sharing of the physical resources such as CPU and central storage.

Much of the appeal of running Linux on the mainframe comes from the ability to run tens to hundreds of Linux images on a single machine.


Chapter 8. Security Considerations

The details of mainframe security and Linux security are described in numerous technical papers, on Web sites, and in other books. This chapter covers those security aspects of Linux on the mainframe that are likely to influence a decision to run certain types of work, or store certain types of data on the Linux system.

On any platform, total security is an unattainable ideal, but security as a goal must be continually revisited and refined. Each platform comes with a degree of built-in security, and going beyond that means cost and effort. You need to balance the risk you are prepared to take against what you are willing to spend. Thus, when planning a Linux-on-the-mainframe project, you must ask yourself if the project will have sufficient security for a reasonable cost.

New projects and workloads can benefit from tools available in the Linux-on-the-mainframe environment. If you are moving from a traditional server farm, sometimes the tools that you have been using for your current environment are also available for Linux on the mainframe. For a list of security tools, see 25.3, "Security management tools."

Security management is really about risk management. Your goal when analyzing your system security is to reduce the perceived risk. To what level you want to reduce the risk is a question of how much you can spend to get a certain return on your investment. To build a security policy that is both functional and meets operational business needs, you need to balance a combination of physical security, software security, and trust in personnel. Here, it makes sense to spend effort on the areas that will give you the most benefit.

The technology involved in a Linux-on-the-mainframe solution offers some security benefits. In this chapter, we address these questions:

  • What role does a security policy play?

  • What security does the mainframe contribute to a Linux-on-the-mainframe solution?

  • What security do z/VM and Linux contribute to a Linux-on-the-mainframe solution?

  • What tools are available?


8.1 The role of security policy

Any computer environment has certain security characteristics, which can be accidental or intentional. For example, to make a computer totally secure, you might turn it off and place it in a locked closet. Arguably, such a computer is physically secure and under little threat of network attacks. It is not likely to be compromised or to affect other machines. However, it also accomplishes little of value to the company. If the computer is turned on, things get more interesting. And if it is attached to the Internet and running an e-business, then things get really interesting.

The mainframe is usually seen as having good security. However, as with any computer, the things required for good security on the mainframe do not come together magically. There is a need for a security policy, instead of letting accidental circumstances dictate security. A security policy is normally used to document desired system behavior. The policy should also guide the implementation of servers so that they have the desirable security characteristics that address various business risks.

A security policy is a comprehensive set of requirements that addresses how users are identified and authenticated, the roles and responsibilities of administrators, the capabilities of administrators and users, password policy, network policy, sharing policy, secure communication, administration of systems, maintenance requirements. Also included are resources and how they are defined and protected, hardening procedures and settings for each of the operating systems, auditing, logging, intrusion detection systems, and firewalls.

In the case of the orphan computer in the closet, the implicit context states that it is to do no work and have no users. Once the computer is turned on, a security policy is needed. The system administrator must determine if a human can manage this policy (typical in the home computer case), or if it must be documented as a procedure to be followed (done by many medium risk businesses), or if tools (like a policy manager) are required to enforce the policy.

Regardless of what kind of environment you are in, you probably already have a security policy in place. When considering a Linux-on-the-mainframe project, you will want to review your policy. Depending on the business value you are looking for from your project, you can follow different strategies. Two examples of objectives for Linux on the mainframe are:

  • Business value from consolidation: When Linux images are consolidated on the mainframe, you will probably also have z/VM to manage those images. Then z/VM must be considered in your policy as well. If, in the extreme case, your only goal is to consolidate as economically as possible, the isolation of images given by the hardware and z/VM may be all the additional security you need.

  • Business value from the speed-to-market of a business application: When applications running on Linux images interact with applications running on z/OS, the Linux applications should be close to the z/OS back-end, and connected to the z/OS images. In this case, you might need to review the policies that apply for your current environment and how they should be enforced on Linux on the mainframe. The implementation might be different for Linux from what it is for z/OS. For instance, by carefully segmenting the work, you might be able to isolate any new risks.

When evaluating the security policy for a Linux-on-the-mainframe project, you will want to know what risks are involved. We will look at risk assessment in the next section.


8.2 Risk assessment

Security is one of the first considerations for any server environment, whether it is a new system or an existing system that is used for new business opportunities.

All security is about managing risk. We have chosen a risk analysis approach as a framework for security. Using ISPCompany and StoreCompany as concrete examples, we first introduce some general aspects of security. Then we look at more details pertaining to the areas we introduce here, in particular, how they relate to Linux on the mainframe (see Figure 8-1).

Figure 8-1. Security space: what security covers

Linux is going to be tightly connected to many aspects of your e-business. Connecting the computer system to the Internet (that is, being connected to an external network) dramatically increases the risk of damage from intrusion.

Security is a trade-off between ease of use of the computing resources, business requirements for protection, and expense. There is no magic application or methodology that ensures the security of a system. Like all real-world technical designs, it is a practice of setting and meeting requirements and the consideration of limitations imposed by those requirements. Defending your Linux mainframe server is based on understanding these security requirements, best practices, and experience.

Naturally, the primary focus of a company is on its business and not on its systems. The goal of such systems is to support business and overall objectives. All business decisions, in IT or otherwise, are an exercise in the evaluation of the risk of inaction versus the cost of action to reduce risks (real or perceived).

8.2.1 What is risk assessment?

Risk can be defined as "being exposed to injury, pain, or loss." In every situation, your company is exposed to injury or loss. In the business context, there can never be zero exposure. The question is how to minimize that exposure and what level of exposure you can live with. Risk assessment, then, is the art of assessing what level of risk your company can accept.

Areas of risk assessment that pertain to Linux on the mainframe include:

  • Hardware: Ensuring physical security of the machine itself and its console

  • Network: Establishing secure communications, using encryption, protecting networks from being tapped, and from denial-of-service attacks

  • Technology: Using virtualization technology

  • Information: Protecting intellectual property (such as specialty applications) and protecting your data, which might be your most valuable asset

  • Human: Minimizing the threat of damage by malicious individuals

One way to approach security and make sure that all areas of importance are covered is by performing an extensive risk analysis. This chapter concentrates on problem areas that need to be considered in a Linux-on-the-mainframe solution.

Figure 8-2 shows a typical Web-serving setup, where the machines are separate and discrete. Some of the typical attacks on this kind of setup are illustrated.

Figure 8-2. Risk areas of typical Web-serving setup

Going back to the examples of business value from Linux on the mainframe, what are examples of risks in those scenarios?

If you are doing simple server consolidation, the images are no longer on individual hardware machines. What is the risk from shared hardware? ISPCompany might be wondering how to guarantee that the servers that used to be in separate cages cannot touch each other's data now that they are on one machine. We discuss this in 8.2.3, "Partitioning" and 8.2.4, "Virtual security."

If you are integrating Linux into the existing mainframe environment to work closely with current business applications, one of the Linux images could become compromised. What could an intruder do in a Linux image? We discuss this in 8.4, "Opening the doors."

A Linux-on-the-mainframe environment usually benefits from virtual networks. What are the risks of virtual networking? We discuss this in 8.4.3, "Secure and encrypted communication."

Let us now take a closer look at the risk areas, and see what our example companies have done to reduce risk.

8.2.2 Hardware security

The first security stage is physical access to the floor where the computers are housed. Assuming that ISPCompany, as the typical data center, has tens of servers in one room, the normal way to protect these is to tightly control access to that room. Only the trusted people who work with the servers are allowed there. Often, all the consoles are logged on to save time. As a result, if someone does break in, he or she could tamper with the servers.

Consolidating many servers on the mainframe simplifies the physical security challenge. There are fewer reasons for anybody needing actual access to the physical machine. The system administrator can do most work with the individual servers running under z/VM from the control room.

The mainframe system console

For a direct attack on a system, the (hardware or operating) system console may be the goal of intruders. The mainframe, in contrast to other systems, has a separate hardware system console. (The zSeries operating systems each have their own system consoles.) The physical mainframe system console is shipped with the machine in the form of two IBM laptop computers, one serving as a hot standby backup. They are called Support Elements (SEs). The SE communicates with the internal sub-systems in the machine. For example, it handles initial program loads (IPLs), I/O configuration, LPAR configuration, and microcode updates.

A Support Element is normally not used directly. Instead, commands are sent through the Hardware Management Console (HMC) to the SE, as illustrated in Figure 8-3.

Figure 8-3. The HMC, the SEs, and a mainframe. Commands for operating, monitoring, and maintaining the mainframe are sent through the HMC to the SE, which then issues commands to the central processor complex.

The HMC serves as a single point of control to operate the machines defined to it remotely. The HMC must be connected to the SEs through a LAN. To be physically secure, the LAN that connects the HMC to the SE should be contained within the raised floor and control room. Through the LAN, you can also operate the systems through a Web interface to the HMC. The HMC is often placed in the control room in case staff need to take some hardware-related actions. (Refer to IBM eServer zSeries 900 System Overview, SA22-1071, for rules on setting up the HMC.)

Apart from physical isolation in an access-controlled room, the SE and HMC are protected by role-based user IDs and passwords. The HMC has a role-based model for access that provides a second layer of security in addition to logon authentication. Role-based means that there are user IDs for certain roles. Even if you have a user ID and a password, you cannot do everything, but only what the role allows.

You can enable remote support for the HMC. This is not automatically enabled.

ISPCompany example

ISPCompany solves the physical access problem by using the conventional method for the outsourcing business. It puts customers' machines in cages, so that any customer has access only to his or her cage. Customer personnel wanting to work in their cage must sign in at the front desk, have badge access to the part of the floor where their cage is, and use a key to open their own cage.

With ISPCompany's mainframes, the mainframes' consoles are connected through a private LAN to the control room, where an HMC acts as a single point of control for those mainframes.

With applications or servers consolidated on the mainframe under Linux, the access problem is now one of logical access rather than physical access. There is little need to touch the actual mainframe. The exception is one or two highly trusted experts who, for example, need to plug in the fiber optic cables to connect new adapters on the server to the fiber network hub. These experts are typically ISPCompany employees. Other situations that call for access to the mainframe are hardware modifications, such as installation of cryptographic cards.

Moving a physical server farm with servers or applications to run on Linux images under z/VM, decreases the risk of physical attack. Instead, security under z/VM, which we examine in detail in 8.2.4, "Virtual security," becomes interesting. You also then need to manage the Linux images "remotely." This can be done with the help of, for example, a virtual private network (VPN) and secure shell (SSH) both of which we discuss later in this chapter.

8.2.3 Partitioning

Logical partitioning (LPAR) provides a flexible alternative for dividing a machine's CPU resources and also guarantees the isolation of the software images running in the individual partitions using the same concepts explained above. The isolation of the partitions is proven by an independent authority. For the most up-to-date information on the technology that provides logical partitioning, see http://www.ibm.com/servers/eserver/zseries/security/certification.html.

8.2.4 Virtual security

A common question asked by technologists interested in running multiple Linux images on z/VM for the first time is, "How good is z/VM's system integrity and security?" The short answer is "very good." The much longer answer offers some additional insight into the technology found in the z/VM product.

Because the z/VM Control Program (CP) and the virtual machine configurations are under the control of the z/VM system administrator, the actual level of system integrity that a company achieves depends on how the z/VM environment is set up and maintained. z/VM is specifically designed to maintain the isolation of one virtual machine from other virtual machines at all times.

The IBM ESA/390 architecture and z/Architecture are at the core of the system's ability to maintain integrity. One crucial aspect of this is the ability to keep each virtual machine isolated from every other virtual machine. This isolation even extends to the CP, because it is logically separate from all virtual machines in the system.

How z/VM keeps it all separate

A special component of the mainframe virtualization technology permits a virtual machine instruction stream to be run on the processor using a single instruction, start interpretive execution (SIE).

When the CP dispatches a virtual machine (that is, executes SIE), details about the virtual machine are provided to the hardware. The SIE instruction runs the virtual machine until the virtual machine receives an interrupt, or wants to perform an operation that either the hardware cannot virtualize or for which the CP must regain control. At that point, SIE performs an exit on behalf of the virtual machine and returns control to the CP, which simulates the instruction or performs the I/O (for example). Once this is done, control is returned to the virtual machine. In this way, the full capabilities and speed of the CPU are available to the virtual machine, and only those instructions that require assistance from or validation by the CP are intercepted.

This mechanism also enables the CP to limit the scope of many kinds of hardware or software failures. When the error can be isolated to a particular virtual machine, only that virtual machine fails. It can be re-initialized (rebooted) without affecting work running in other virtual machines. The CP is designed so that failures that occur in virtual machines do not affect the CP or other virtual machines.

Isolation through address translation

While most platforms today offer virtual memory, zSeries allows an operating system to create separate virtual address spaces. Address spaces enable memory isolation and management. For example, z/VM running natively on a zSeries machine creates a separate address space for each guest. Each address space has an associated set of region, segment, and page tables that contain precise information on the real memory locations being used for the guest. These tables are used by the address-translation hardware to convert virtual memory addresses to real memory addresses. Because these tables are maintained by the operating system and are not accessible by the guests themselves, it is not possible for a guest to read from or write to memory that is used by the operating system itself or another guest.

zSeries takes this capability a step further by supporting two levels of address translation. An operating system running in a virtual machine under z/VM constructs its address-translation tables as usual to isolate and contain the memory for its users. The entire memory of this virtual machine, although viewed by the guest operating system as real memory, is in fact virtual storage as well, defined by another set of translation tables managed by the z/VM CP. Even if an application running on a guest operating system were able to compromise the integrity of the guest, the damage would be limited to that one virtual machine, because of the separate layer of protection provided by zSeries hardware and z/VM.

A virtual machine cannot access an address space owned by another virtual machine unless the address space owner explicitly allows the virtual machine to do so. This capability is controlled through the z/VM directory entry for each guest.


8.3 Before opening the doors: hardening

A process sometimes used in the UNIX and Linux community is hardening. A "hardened" system is presumed to be impervious to any currently known attacks, exposures, or vulnerabilities. Every system, Linux or otherwise, should be hardened before being placed on any active LAN. Hardening is not a one-time task. Depending on the level of acceptable risk, triggers must be identified to ensure that the system configuration is revisited as needed. The security policy should contain criteria that trigger a reassessment of the configuration. For example, if a new Internet worm is identified and is known to attack the e-mail server, the security policy must be checked, and systems must be updated and rehardened.

Develop a patch policy

System hardening includes making sure that the latest software patches are applied to the system, especially if they are security-related or apply to any of the security or system management products that are critical to the solution. Once all the patches are installed, the mechanisms into and out of the solution must be evaluated with regards to the solution. This makes it necessary to have a patch policy within the security policy.

Get rid of the extra fat

Only those mechanisms required by the solution should remain active. For example, if the work that needs to be done is Web serving and the system will be managed remotely through SSH, it would be prudent to turn off the telnet and mail daemons (among others), because they are not part of the solution and could provide intruders unnecessary routes of attack.

Most standard Linux distributions include features and services to enable a wide scope of functionality and value—usually far more then you need and want on a typical Linux server deployment. Many of the latest Linux distribution releases provide warnings if you attempt to install a service that poses a security risk, such as telnet. We highly recommend that you pay attention to these warnings. Validate that the services running are critical for the function of the Linux server you are configuring.

Related information is in the SANS/FBI list (http://www.sans.org/top20/), including one of the top general vulnerabilities, "Default Install of Operating System and Applications."

Remove unused accounts

Removing insecure user accounts is also part of the hardening process. Some system images are provided with well-known accounts and passwords. Some application install processes leave behind accounts that are well-known and vulnerable. Unnecessary user accounts should be removed, and default or weak passwords should be reset to strong passwords. User authority to access system or application code should be limited as well.

For z/VM, consider including these issues in your security policy:

  • After installing a new z/VM system, remember to change the default logon and minidisk passwords for all users in the system directory.

  • Do not give virtual machines more authority than they require.

8.3.1 Managing hardening

The hardening of a system must be linked with the management of that system. Every time the kernel is patched or upgraded, or a new application is installed or modified, the state of the system with regard to hardening must be reviewed. This integral step can be performed by any number of scripts available in the Open Source community or from service organizations skilled in this area. For example, scripts are available from Bastille at http://www.bastille-linux.org/. These scripts are yet another example of how Open Source code helps Linux be more secure.

In some scenarios, once an image is hardened it can be used as the starting point for cloning new images as needed. This cloning capability is easily implemented and its value is realized when z/VM manages the images.

The early broad-brush approach to enabling many popular network services, such as telnet or FTP, has left many systems vulnerable. The traditional zSeries customer would rather enable function as needed, and the Linux trend is shifting in this direction.

8.3.2 Tools for managing security on z/VM: DirMaint and RACF

Software products exist that can be used to further enhance the integrity and security of a z/VM system. Two of those products from IBM are optional features of z/VM Version 4 called IBM Directory Maintenance for z/VM (DirMaint) and IBM Resource Access Control Facility for z/VM (RACF).

DirMaint provides a safe, efficient, and user-friendly way to maintain the z/VM system directory. Through its command line or full-screen interface, you can quickly and easily add, modify, or delete users from the system directory. See 23.3.1, "DirMaint" for more details. In any z/VM installation where large numbers of virtual servers are being deployed, DirMaint is useful.

RACF is an external security manager. It provides comprehensive security capabilities that extend the standard security implemented by the base z/VM product. RACF controls user access to the VM system, checks authorization for use of both system and virtual machine resources, and audits the use of those resources. Like DirMaint, RACF is packaged as a priced feature of z/VM Version 4 and is pre-installed on the system installation media. See 23.3.2, "RACF" for more details.


8.4 Opening the doors

Now that we have covered some physical and virtual security aspects, let's go to the next step in protecting a machine: allowing users entry.

When StoreCompany created its Linux-on-the-mainframe environment with Linux images and z/OS working together on one machine, one risk (arguably very small) it considered was that of a Linux image being taken over. What could then happen? The Linux image has a connection to z/OS, and someone could theoretically get access to z/OS resources. However, like most z/OS systems, StoreCompany's z/OS resources are under the control of the RACF security manager. It would not be possible to take over z/OS.

If someone very knowledgeable got as far as taking over a Linux image with a connection to z/OS, that person could send commands to z/OS. But the effect of such commands is limited to what the Linux image is authorized to do. For example, if the image belongs to user George, it might be authorized to look at all files associated with George's orders. However, if George's image asks z/OS for the payroll, z/OS recognizes that the image has no authorization to look at this file and will deny access. Hence, with the standard authentication and authorization of the mainframe in place, even if an image were taken over, it would have a very limited scope of access to other systems.

This section covers considerations of allowing users access to the system: authorization, access control, and secure network communications.

8.4.1 Identification and authentication

Identification and authentication provide a first level of protection when permitting entry to the machine. Users identify themselves and give some piece of information that authenticates who they are. For example, users identify themselves by giving a user ID and then give the password associated with that ID.

Identification and authentication in general can be done by many different means, such as certificates, user IDs, and passwords.

When the machine knows who you are, it can decide what you are allowed to do—this is authorization. For example, a user ID and password combination gives users access to files, with which they are thus authorized to work. In this case, the files are password protected. The file system provides the security service of allowing access to the file only when presented with the correct password. This method was typical of, for example, batch-oriented S/360 systems. This method has some drawbacks in a large multiuser system, however:

  • All files (or programs, or other resources) have their own authentication code. If you want to change it, you must recompile all the programs or all the resources that reference the method.

  • If multiple users need access to one file, they must all know the password. It is then hard to tell who did what with that file.

  • If users need access to multiple files, multiple passwords are needed.

Instead of permitting access on a per file basis, modern operating systems (like z/OS and Linux) authenticate the user to use an identity. Identities are typically stored in a user registry. Both Linux and other mainframe operating systems make use of registries.

For z/OS and z/VM, the user registry of choice is the Resource Access Control Facility (RACF). (Note, however, that RACF for z/OS is different from RACF for z/VM.) RACF is the chief function of the mainframe's Security Server and incorporates user identification and authentication, as well as access control, which is discussed in 8.4.2, "Access control." In a Linux-on-the-mainframe context with Linux images running under z/VM, you can use RACF to secure z/VM user IDs and passwords.

Instead of each program having its own authentication code, many Linux distributions now come with Pluggable Authentication Modules (PAMs). If a program is PAM enabled, youcan easily switch between several authentication methods. If the Linux images with connectors to z/OS use PAMs, it might be possible to share user ID definitions and passwords with the Linux images.

Example of centralized authentication services

On Linux, the Lightweight Directory Access Protocol (LDAP) can be used with a user registry. An example of how to use PAMs and LDAP to share existing z/OS RACF data is shown here.

An LDAP server can be any general-purpose distributed directory server and can contain many different types of information, from distributed application descriptions and configuration information to user and group definitions.

Imagine an environment like that of StoreCompany with many Linux images running on a mainframe where z/OS is also running. User Peter has an account on z/OS as well as on several of the Linux images. How can you manage many users like Peter from a central point? Using a user registry is one answer. Assume user information is stored centrally in an z/OS LDAP server. PAMs can then be configured to pass authentication requests to the LDAP client, and from the LDAP client to the server. Thus you can share existing RACF information with the Linux systems and keep, for example, passwords protected by RACF, using a special feature of the z/OS LDAP server.

If z/OS runs on the same hardware as the Linux images, HiperSockets communications can be used. HiperSockets is an internal communication method with no physical network parts. Thus no information can be "sniffed." This scenario is illustrated in Figure 8-4.

Figure 8-4. RACF information can be shared with Linux systems through using PAM and LDAP

8.4.2 Access control

The use of access control has migrated from a mainframe-only environment to become an integral part of many platforms' security solutions and their security policy management systems. As Linux continued to mature and the number of applications grew, the availability of and need for resource access control became a critical component in the deployment of Linux-based environments.

Access control mechanisms on Linux systems

"Who can do what, where?" is the task that access control performs. There are two types of access control mechanisms:

  • Discretionary Access Control (DAC)

  • Mandatory Access Control (MAC)

DAC can be used to manage access control in its simplest form in a low-risk environment. The supplied Linux DAC is known as "user," "group," and "other" permission bits on the file system. These bits determine what user or group is allowed to read, write, or execute a particular resource (file, directory, or device). For example, when you access a printer, you are actually writing to a file in the file system. This can be controlled through the "user," "group," and "other" bits. In the simple server consolidation case, this may well be enough.

MACs are used when securing highly confidential data. The user has no control over the classification or labeling of the resources. The security policy requires a security manager, human or automated, to assign sensitivity labels to every subject (users and programs) and all objects (files, directories, and devices). MACs, along with the proper labeling, make up the core of a multilevel security (MLS) policy. The automated implementation of rigorous MACs is found in sophisticated policy management systems where risk must be minimized.

Using DAC, users can choose to share or give away access to a resource under their control. With MAC, this decision is out of their control. At a minimum, groups should be identified and group permissions used to isolate security functions such as user administration, audit, system and application maintenance, system backup, kernel and key operating system executables.

Access control lists (ACLs): Access control lists (ACLs) are another form of the more robust access control case, MAC. They allow the user more flexibility to manage individual resources, but require more knowledge of the system. ACLs give a security administrator the ability to control each user's access at an individual resource level. They list users and what they can do. Taking the "user," "group," and "other" mechanism to the next level by including ACLs provides an environment with less risk, but yields more system administration overhead.

Certain Linux journaling file systems, such as ext3, are ACL-enabled. These are included in, for example, the SuSE SLES 8 distribution.

Tools for access control

For an access control solution to successfully enforce the access control list (and hence the access policy of the system), it must be closely integrated with the kernel's decision points to avoid various system attacks or spoofing attempts. In the vendor arena, access control solutions are provided, for example, by IBM Tivoli Access Manager and CA's eTrust suite. See 25.6, "Tools Web sites" for a list of tools and their Web sites.

In the Open Source area, BestBits, LoMAC, SE Linux, and others all provide various levels of access control functionality and can run on a zSeries Linux server.

8.4.3 Secure and encrypted communication

Linux provides some potentially useful schemes for more secure communication which are important for today's e-business environment. In this section, we will cover the most important aspects of secure communication:

  • Secure physical communication using Secure Socket Layer (SSL), Virtual Private Network (VPN), and mainframe cryptographic functions

  • Secure virtual communications over HiperSockets and other virtual communication connections

SSL and key setup

The Secure Sockets Layer (SSL), which is the core technology used to secure Internet transactions, uses cryptography both for authentication of clients and servers and for data confidentiality.

One shortcoming of the user ID and password (basic authentication) model is that the password has to be passed from the end point, where the user is, to the physical location where the application and security registry reside, which may be far away. It is important to ensure security of the password in transit. On the mainframe, the SSL protocol can be used to encrypt the password while in transit.

The Open Source version, OpenSSL, is included in many distributions and can be used with Web servers, such as Apache. OpenSSL is a robust, commercial-grade, full-featured, and Open Source toolkit that implements the Secure Sockets Layer and Transport Layer Security protocols, as well as a full-strength general-purpose cryptography library. On the mainframe, you can use the hardware cryptographic functions for authentication through OpenSSL.

SSL is a protocol for transmitting documents securely through the Internet. SSL works by using a public key to encrypt data that are used to agree on symmetric encryption keys. These keys, in turn, are used to encrypt data that are transferred over the SSL connection. SSL has several characteristics that can be used by an e-commerce application to communicate with large numbers of users via common Internet browser software:

  • SSL can be used to identify and authenticate (with a high degree of trust) server applications to users of the application. For example, this function would be important for a Web marketing application that requests a customer's credit card number. The customer wants to be assured that the application is the one intended and not a "Trojan horse" imposter that is stealing credit card numbers. SSL provides this highly useful security function.

  • SSL can be used to establish a private cryptographic communication channel through the Internet between the communicating parties (for example, the customer on the Internet and the Web application server). This channel enables credit card numbers to pass from the customer to the marketing application without being observed. Note that VPNs, which we will discuss later, also provide Internet communication privacy. The difference is that VPNs are generally used for point-to-point communication links that can be shared by multiple applications and that continue for a longer duration than those typically seen with common browsers.

  • SSL can be used to identify and authenticate (with a high degree of trust) users of applications to Web application servers. This function is most useful when the application provides function that requires the user to be known to the application. An example is when the application has administrative support function that needs to be individually authorized for execution by only specified individuals (such as payroll functions).

SSL has emerged as the preferred method for securing Web-based transactions on the Internet. It is widely supported in browsers and most Web servers. The first two functions are in common use worldwide, while the third (user identification and authentication using X.509 version 3 digital certificates) is being applied in an increasing number of cases.

For a description of how SSL works, see http://developer.netscape.com/tech/security/ssl/howitworks.html.

Virtual private networks

Virtual private networks (VPNs) allow remote users safe access to critical system resources and provide safe remote communication, for example, for systems management. VPNs provide a valuable means of doing remote system management securely over the Internet.

VPNs enable encrypted connections from one system to another over the Internet. The part that constitutes the actual connection over the Internet is encrypted, often using OpenSSH (see Figure 8-5) or the IP Security Protocol (IPSec).

Figure 8-5. Virtual private networks send data over the Internet securely encrypted with SSH or IPSec

There are several uses of VPNs: hardware-based VPNs (usually for encrypting routers), firewall-based VPNs (which include address translation, authentication, alarms, and logging), and stand-alone software VPNs (ideal to use when the connection endpoints are not controlled by the same organization).

Virtual networks should be planned with the same care and attention to security as would be taken for a real, physical network. Networks, virtual or real, must be designed and implemented so that no unauthorized access to data or resources is possible. For system administration tasks, a separate network with secure access is recommended. The ability to define multiple virtual routers gives the ability to completely isolate traffic moving in and out of the S/390 or zSeries server. Highly secure communication between LPARs can be easily handled by using zSeries HiperSockets connections.

Available virtual communication paths under z/VM are summarized in Table 8-1. For more details about them, see Chapter 10, "Communicating in a Virtual Environment."

Table 8-1. Available virtual communication paths with z/VM

Communication

Functionality

Guest LANs

Guest LANs provide multipoint any-to-any virtual shared media connections between guests. As many Guest LANs as are needed can be defined and used simultaneously. To prevent unauthorized connection to a Guest LAN, the creator of the LAN can define it to be restricted, permitting only specific virtual machines to connect to it.

vCTC

vCTC emulates a real CTC adapter (IBM 3088). Each virtual machine defines a vCTCA and then uses the CP COUPLE command to connect the two endpoints. I/O operations to the device are intercepted by the CP, which moves the data between two different virtual machines. Again, data are moved at near memory speeds.

Whenever you use virtual communications technology, there may be no need to encrypt the traffic because the information never leaves the machine. An overview of secure communications can be found in the whitepaper z/VM Security and Integrity, GM13-0145. It is available at http://www.ibm.com/servers/eserver/zseries/library/techpapers/gm130145.html.

Cryptography

The use of cryptography has become a common requirement in today's computing environment. Cryptography can be used for confidentiality, authentication, and data integrity. Cryptography is crucial for security on the Internet, banking and finance, and e-business.

Confidentiality is gained when a cryptographic algorithm is used to make the data available to only a trusted group of users. The data are transformed from clear text into cipher text by using an encryption algorithm and a key. The data are unintelligible in cipher text form and can be decrypted only using the correct algorithm and the correct decryption key.

One use of cryptography on Linux on the mainframe is digital signatures. Earlier, authentication was thought of as being done by users entering his or her user ID and password. This is still valid today, but the trend goes towards using digital signatures. This relatively new technology is available and deployable on Linux on the mainframe. Although there are several methods for digital signatures, the most common method in use today is RSA public key cryptography.

Another use of cryptography is the validation of data integrity. Integrity can be validated through the use of a one-way cryptographic function. A hash function is sometimes used by the owner of critical data to come up with a "digest" of the data. Any other user or program can then validate that the data have not been modified by using the same hash function and obtaining the same digest.

zSeries and S/390 servers offer specialized processors for cryptographic operations. In an ISP or ASP environment such as ISPCompany, cryptographic procedures are frequently used for highly secure TCP/IP connections between the server and a user somewhere on the Internet. Applications for firewalls, Web serving, and mail serving also have the requirement to protect data.

Linux and mainframe cryptography: Linux on the mainframe can use the mainframe cryptographic services, thanks to a single device driver for both the newer IBM PCI Cryptographic Accelerator (PCICA) and the older PCI Cryptographic Coprocessor (PCICC). The new PCICA feature is designed to address the high-performance SSL needs of e-business applications. It provides accelerated performance of the complex RSA cryptographic operations used in the SSL protocol.

The older PCICC allows implementation of different cryptographic algorithms and can be used for specialized financial applications, or for secure long-term storage of keys and secrets. Both PCICA and PCICC have earned Federal Information Processing Standard (FIPS) 140-1 Level 4 certification, the highest certification for commercial security awarded by the United States government.

To use a cryptographic coprocessor, the operating system must be able to recognize application requests for cryptography, pass the requests to the cryptographic hardware, and return the result to the application. The driver available for Linux on zSeries and S/390 exploits the PCICA and PCICC cryptographic hardware for the algorithms used by SSL (see Figure 8-6). This exploitation results in significant performance improvements for SSL transactions.

Figure 8-6. The cryptographic device driver for Linux on the mainframe enables openLDAP or openSSL requests

The code for cryptographic support and associated functionality is included in, for example, the SuSE SLES 8 distribution.

Virtual cryptography support—Linux and z/VM: The PCICC and PCICA cards can be shared and used by any number of Linux guests as shown in Figure 8-7. Each operation is discrete and independent of those that precede or follow it. z/VM manages the queue of requests so that a guest can see only its own request, as with a shared processor or a shared channel.

Figure 8-7. Virtualized cryptographic functions

ISPCompany example: secure networking

In a service bureau situation such as at ISPCompany, where there are physical farms of machines in separate caged areas, there is sometimes a LAN that connects all the customer machines. Assuming a client were to exchange the standard network adapter for one that allows all the network traffic to be read, the client could read not only his or her own traffic but also everybody else's.

To prevent this, ISPCompany has point-to-point connections from the incoming data source to the different machines. This costs more, but is more secure than a LAN.

When ISPCompany virtualizes the adapters by putting its customers into z/VM, there is no physical adapter for intruders to sniff. z/VM does not let guests see any traffic other than their own when they share an adapter.

By using HiperSockets, ISPCompany can eliminate the physical risk of someone tapping into the network. Other benefits of HiperSockets include high speed of transfer and close to zero latency. Some savings can be made on not having actual cables and other hardware needed for the physical network.


8.5 Preventing attacks

To detect and prevent attacks on your system, you need to use firewalls, intrusion detection tools, and anti-virus tools.

Firewalls block certain traffic from entering your environment and examine network traffic. Intrusion detection tools examine network behavior behind the firewall for abnormal patterns. Anti-virus tools examine data content for suspicious modifications and virus signatures.

8.5.1 Firewall considerations for Linux on the mainframe

One of the threats you need to assess in any environment, including a Linux-on-the-mainframe environment, is the denial-of-service attack. One part of the defense against denial-of-service attacks is using firewall techniques.

A firewall is a secure and trusted system that acts as a barrier between private and public networks. A firewall, when combined with a VPN, can provide secure, encrypted communications with sites outside the firewall.

Firewall techniques include:

  • Establishing a demilitarized zone (DMZ), illustrated in Figure 8-8.

    Figure 8-8. Standard firewall implementation with demilitarized zone. The Web server is protected first by one firewall that has connectivity to the Internet. If users are legitimate, they are redirected by a proxy server through the second firewall.

  • IP packet filtering, static and dynamic (state-dependent)

  • IP datagrams are inspected; the filter decides if a datagram needs to be processed or discarded. Filtering can be done on protocol type, port, datagram type, IP address, and so on.

  • Application gateway (or proxy)

    The proxy understands the application protocol, performs logging, and can provide authentication and caching capabilities. An example of a proxy caching server is squid (http://www.squid-cache.org/), which provides access control lists for ports, URLs, and subnets.

DMZ considerations

Where does your Web server reside within the infrastructure? Traditionally, a Web server is located between filtering devices, such as firewalls and routers. This places it in a "demilitarized" zone (DMZ), which offers some protection from Internet intrusions, as illustrated in Figure 8-8.

The internal network receives protection from the Web server. Outbound filtering devices permit, for example, only ports 80 and 443 to reach the Web server; other potentially dangerous traffic is inhibited. The inside filtering device is important, because the Web server itself should not rely on other hosts within the secure LAN. Although you must allow potentially dangerous parties access to your Web server, careful measures should be taken to prevent unwanted entry. (There are many reference books that address the different DMZ methods in more detail, such as Building Internet Firewalls by D. Brent Chapman and Elizabeth D. Zwicky and Linux Firewalls by Robert L. Ziegler.)

An example of when a DMZ is needed is to separate Web applications from the Web server, as StoreCompany does for its Web catalog.

With Linux on the mainframe, it is possible to consolidate firewalls on the mainframe. The network firewall can be outside the mainframe to save mainframe resources and the other DMZ firewall can be on a separate image. A firewall can be implemented either on an LPAR or under z/VM (Figure 8-9). This consolidation makes administration of the firewall easier and eliminates the need for separate hardware.

Figure 8-9. Using a firewall in a Linux-on-the-mainframe environment

Linux firewall capabilities

Some firewall capabilities have been integrated into the Linux kernel. These include:

  • IP packet filtering, which either accepts or rejects network packets based on the information in the packet header. Packet filtering can be used through the newer iptable tool (kernel version 2.4).

    In a Linux-on-the-mainframe environment, you can use IP filtering. On the mainframe, data that never go outside the machine can never be physically compromised.

  • Network address translation (NAT), which modifies the network address in a datagram, thus "hiding" the internal network addresses.

  • IP masquerading, which is a special form of NAT, can be used in a hot standby takeover process. When a hot standby image takes over a failing image, it can also take over the IP address. The standby server thus masquerades as the server it replaces.

Using only these capabilities and Open Source tools, you can implement firewalls on Linux. Most distributions deliver some sort of firewall support.

You can also turn to commercial firewalls. For information about firewalls for Linux on the mainframe, refer to IBM's ISV support for mainframe servers Web site: http://www.ibm.com/servers/eserver/zseries/solutions/s390da/.

While a firewall blocks certain traffic from entering the company network, intrusion detection recognizes strange behavior within the network.

8.5.2 Intrusion detection

The goal of intrusion detection is to detect malicious attacks on your infrastructure, such as denial-of-service attacks.

Denial-of-service attacks can occur in different ways. For example, a compromised image could try to use all the CPU resources or all the network resources. Although the damage that a compromised Linux image can cause could be limited by z/VM definitions, the better solution is to detect an intrusion before damage is caused.

Intrusion detection systems (IDS) can be designed for network-based and host-based systems.

  • Network-based IDS are attached to the network. They detect attacks by analyzing the content of network packets sent over the wire. An unusually high number of TCP, UDP, or ICMP packets sent to a single destination can easily be detected. IDS are configured to determine if these packets should be considered attacks or normal traffic.

  • Host-based IDS are software components that attempt to detect attacks against the computers on which the IDS is installed. Host-based IDS can analyze the network packets received on the network interface, as well as the log files written by the operating system or by applications running on the computer. Typically, a host-based IDS can detect denial-of-service attacks against a Web server by analyzing its log in real-time.

Sites should install both network-based and host-based detection systems. Network analyzers should be available to help determine the nature of an incident and to help formulate possible filtering and rate-limiting responses in the event of an actual denial-of-service attack.

The Linux community and ISVs have developed some sophisticated, useful tools for Linux that work on Linux on the mainframe as well. The leading Open Source intrusion detection solutions include Tripwire and Snort. It is important to note, however, that IDS systems come in different flavors. For instance, Snort (available with SuSE SLES8) is more of a network IDS, while Linux Intrusion Detection System (LIDS) is targeted more to the protection of, and the intrusion detection in, the Linux image.

Tripwire detects and reports file and directory modifications. This can help to detect Trojan horses[11] and modified software (for example, for sniffing out passwords).

[11] A Trojan horse is a program that cannot operate unless it is invoked (unintentionally) by a user.

IBM offers a Tivoli IDS solution. Tivoli Risk Manager is a network-based IDS that can recognize basic attacks and prevent denial-of-service incidents.

8.5.3 Anti-virus protection

With access control in place and isolation of power between root and user, Linux is not so susceptible to common viruses. (A virus is a program that actively operates, independently of a user, to attack various system resources.) It is reportedly unusual for UNIX-based operating systems to suffer system-level damage from a virus, because most viral code cannot get access to low-level system functions. However, it is possible for a Linux server to host an infected file or e-mail and send it to a Windows user. For this reason, it is still a good idea to routinely check incoming e-mail and downloaded files for viruses.

It is important to note that the commercial definition of "anti-virus" includes not only viruses, but also Trojan horses, worms, and other similar threats. Although it is not technically or semantically accurate, this discussion includes these other threats as part of the "virus" context. The threat of virus can be broken down into three basic areas:

  • Viruses targeted at other platforms stored on Linux on the mainframe in an e-mail server.

    The application of most interest to hackers is e-mail clients running on popular software on common platforms. This type of threat can be neutralized by anti-virus applications running on the target platform. A filtering anti-virus tool to neutralize these types of threats while stored in a mainframe e-mail server would be a plus.

  • Viruses written in interpretative languages (like Java, HTML, and PERL).

    These are dangerous because they can be executed on any platform that has an interpreter.

  • Linux viruses that run on the mainframe.

    Viruses that actually execute on a zSeries server could be a concern. This type of threat would require that the code be placed on an image and be executed, either manually or through some misdirection. This type of exposure would be contained by developing and managing good security policy, limiting the access to production images via a firewall, and monitoring the network access with current intrusion detection techniques.

Careful use of standard Linux features can reduce the risk of virus attacks. If a system is serving static files (non-writeable), the file system can be mounted as read-only.

When considering anti-virus protection on a server platform, you need to identify the work that is going to be done and what needs to be protected. If the customer is running a mail server on Linux on zSeries, his or her definition of anti-virus protection is likely to be a mail-scanning anti-virus tool that can be integrated with his or her mail server of choice. The mail clients are likely to be PC-based, so the security manager should check for viruses in the customer's stored mail prior to distribution or delivery. This would be necessary to prevent the zSeries server from spreading a virus.

Commercial anti-virus tools are available for Linux on the mainframe. For information about anti-virus tools, refer to IBM's ISV support for mainframe servers Web site: http://www.ibm.com/servers/eserver/zseries/solutions/s390da/.


8.6 Keeping up to date on security issues

Distributors maintain mailing lists and sites for fixes that relate to their Linux distributions. Here are some useful addresses for general information on security alerts:

SecurityFocus

Visit http://www.securityfocus.com/ for general security issues. From this site, you can also link to the BugTraq mailing list for the discussion and announcement of computer security vulnerabilities.

CERT Coordination Center

Visit http://www.cert.org/ for general information, and http://www.cert.org/contact_cert/certmaillist.html for a mailing list with current security fixes.

SecuriTeam

Visit http://www.securiteam.com/ for a mailing list with news on current security issues and tools.

SecurityTracker

Visit http://www.securitytracker.com/ for news on security. You can list the issues according to vendors, impacts, and other categories.

SANS/FBI list

Visit http://www.sans.org/top20/ for a list of the top vulnerabilities of computers.


8.7 Summary

When considering security for Linux on the mainframe, remember that:

  • The hardware itself enjoys a higher level of physical security than the average server farm through virtualization and reduced physical resources.

  • z/VM can help by using:

    - Defined roles that scope authority

    - Guest definitions to diversify types of guests and their authorizations

    - Virtual communications that eliminate the risk of wire-tapping

The use of a mainframe and z/VM can lower the total cost of ownership:

  • Because fewer people need access to the physical machine, the cost of repairing damages caused by intentional or unintentional tampering with the system console, cables, and other hardware is lowered.

  • z/VM ensures consistency and speed when defining the secure environment and making copies of a known secure environment. Thus, z/VM can reduce the time spent by administrators identifying, applying, and supporting security patches.

  • z/VM ensures the isolation of images you require, in most cases without the need to buy and set up individual machines. Individual machines bring with them costs and interconnection complexity, resulting in additional risks in the configuration that would not be present in a z/VM environment. Corporate rules sometimes state that a specific piece of work must run on a physically separate piece of real hardware. Recall, however, that LPAR, having a certified level of isolation, should be considered as separate, isolated hardware and can make Linux on the mainframe a real option.


Chapter 9. Setting Up Linux on the Mainframe

In this chapter, we discuss considerations when setting up Linux on the mainframe. These considerations cover obtaining a distribution, installing it, and customizing it. The customization section discusses options for creating Linux images for different purposes in an efficient way using z/VM. Customization also covers I/O considerations when setting up the environment.

In general, installing the Linux operating system on the mainframe is no different from other platforms. Setting up the hardware for Linux on the mainframe is different because of virtualization technology. Setting up virtual hardware can be easier (and potentially less costly) than setting up real hardware.

Installing Linux itself on the mainframe is relatively easy, as was demonstrated by the "Install Fests" (see 3.2.2, "Spreading the word—the Linux install fest"). The initial installation is described in the respective distributor's documentation and is not repeated here. Instead, we focus on configuration choices that can give you some value from Linux on the mainframe.

With Linux on the mainframe, you have an interesting opportunity during the configuration process to address the key areas of availability, security, and management. You can set up for availability early when creating your images. You can also set up for security to a certain degree, and Linux on the mainframe can make it easier to manage both software change and operations. For example, operation management can be simplified if z/VM is set up to be the health monitor. These are choices that you can make when configuring z/VM and the Linux guests.

In this chapter we explore these questions:

  • Where can you obtain a Linux distribution for the mainframe, and what are your criteria for choosing one over another?

  • What are the runtime environment options for Linux on the mainframe?

  • What should you consider when customizing the environment for availability?


9.1 Distribution considerations

The fact that there are several distributions gives you options. The distributions include Open Source packages; it is possible to modify, exchange one distribution for another, and upgrade at your own pace.

How do you choose a distribution? You may already have criteria for selection. In this section, we present some criteria that may be of use. We have found that companies tend to consider the amount of modification they need to make in order for a distribution to fit their needs. The amount of modification needed depends on how well the base distribution fills your needs, what device drivers it comes with, and so forth.

The Linux distributions for the mainframe all come with the unique extra functions for mainframe device drivers and communications features such as HiperSockets. You can choose between 31-bit distributions (Linux for S/390) and 64-bit distributions (Linux for zSeries). Table 9-1 lists some commercial mainframe distributions.

Table 9-1. Mainframe Linux distributions

Company

Web site

Red Hat, Inc.

http://www.redhat.com/

SuSE Linux AG

http://www.suse.com/

Turbolinux, Inc.

http://www.turbolinux.com/

While other mainframe distributions may be available, those listed in Table 9-1 have been tested, not only by the suppliers but by IBM as well.

Any revision of a distribution may at any given time exploit more or less of the full Linux for S/390 or zSeries capabilities. IBM first released its new S/390 or zSeries technology patches to IBM's DeveloperWorks Web site. From there, distributors can download the code.

The distributors each have their own schedule for building new versions of Linux on the mainframe. Distributors "leap frog" each other with the software version they include. For example, Red Hat might have a newer version of glibc, but SuSE might have a newer kernel.

9.1.1 Choosing the right distribution

The price of the distribution will be part of your TCO equation. Contributing factors to the cost of a commercial Linux distribution are the charge for the support and maintenance as well as the rights to future upgrades

A goal of the Free Standards Group workgroup Linux Standard Base (LSB)[12] is to promote compatibility among Linux distributions and thereby enable software applications to run on different Linux systems. The LSB defines a standard against which both the distribution and the middleware/application products can be certified. Ensuring compliance of any distribution (and application) you use with the LSB helps you avoid problems when you change distributions or need to move an application to run on another distribution.

[12] http://www.linuxbase.org

Documentation can be another important aspect of your choice of distribution. Inspect it for coverage in those areas where you have the greatest need for clear information.

9.1.2 Modifying a Linux distribution

As explained in Chapter 1, "Introducing Linux," Linux consists of numerous Open Source packages. Different distributions bundle different versions of various packages. Together with specific installation and administration tools, configuration files, documentation, and so on, this bundle of packages makes up your Linux system. At some point, you may need to make a change to your Linux distribution. You probably already have a change management policy in place for most of your other operating system images, and maybe even for Linux. This section presents a few things unique to Linux that you might want to ensure are addressed.

You might want to change your chosen distributor's basic installation in order to:

  • Set up specialized systems for specific tasks.

  • Improve security by reducing the number of services on the system.

  • Reduce the size of the system.

  • Do performance and tuning.

  • Include certain patches for enhanced functionality.

  • Address dependencies of middleware or applications on certain versions of packages.

Some changes, such as installing kernel patches, may invalidate the support contract. It is important that any changes made (for example, to the kernel, to the distribution, to the middleware, to third party applications, and so forth) are acceptable under the terms of the respective maintenance and support contract. Methods of change that usually are acceptable include:

  • Changing parameters in configuration files (adding or removing modules, changing module parameters)

  • Adding or removing packages

  • Choosing different versions of packages (that is, updating the kernel or runtime environment)

Starting with a distribution that is closest to your desired system helps you minimize changes. If you do need to make changes, the preferred method is to use the specific configuration tools of the distribution.

Change management as a discipline is discussed in 16.2, "Change management."


9.2 Running Linux on the mainframe

On the mainframe, there are three options for running Linux:

  • Natively on the mainframe hardware

  • In a logical partition (LPAR)

  • As a guest operating system under z/VM

Of these, the most interesting for the server consolidation case is the z/VM option.

Using LPARs, you can define up to 15 partitions (with the S/390 and zSeries machines), each of which hosts a separate server. If an LPAR contains CPUs that can run any operating system (so called general purpose engines), any mainframe operating system can run in that partition. The hardware resources of the mainframe (CPU, I/O, and memory) are divided according to the definition of each LPAR. zSeries and S/390 hardware have the computing power coupled with adequate memory and I/O to support hosting multiple operating-system images at one time.

Using z/VM, virtualization goes further than with LPAR to allow you dynamic definition of CPU, memory, and I/O channels as well as real sharing of memory, channels, and disks. Hundreds of images can run as guest operating systems under VM, as illustrated by consolidating entire server farms to Linux under z/VM by ISPCompany.

Software pricing for operating systems such as z/OS, z/VM, or VSE/ESA is generally based on the processing power of the machine. Adding CPUs normally increases this processing capability and, therefore, software prices. This was an early inhibitor to using Linux on the mainframe. IBM introduced Integrated Facility for Linux (IFL) to allow you to add processing power for Linux without increasing software pricing for traditional operating systems and other software. IFL features are managed by PR/SM as one or more logical partitions with dedicated CPUs. In IFL LPARs, you can have Linux images only or Linux and z/VM. Using the IFL, you can provide capacity dedicated to Linux workloads, as shown in Figure 9-1.

Figure 9-1. An IFL feature can be used to run Linux workloads

    [ Team LiB ]


9.3 Creating images

As a server farm grows, day-to-day administration can become cumbersome. With Linux on the mainframe, you can plan your installation with a view to ease installation and administration of new servers. z/VM gives you unique possibilities to create virtual Linux servers in a matter of minutes.

Before you install Linux, you can benefit from planning ahead. List your Linux images and decide what categories they fit in, and think about whether you want to take advantage of code sharing. Code sharing can be direct, by having a read-only disk that all images can access, or by copying from one golden image. z/VM can support all these choices.

Creating a virtual Linux server is a two-stage affair. You first create a virtual machine, then you can install a Linux image to run in it.

Normally, the system administrator logs on to a CMS guest console and uses the control program functions to configure the virtual hardware for the new virtual machine (guest). Each virtual machine, or guest, in z/VM is known by a unique ID that is used as part of the logon process. In the Linux usage of z/VM, we refer to this logon ID as the guest name. The software and data that reside on disks are then made available to the new guest.

These disks are typically of two types: system disks that contain the system executable libraries and system parameters and data disks that contain the unique data to be used by the Linux image while it is running its applications.

Then the administrator can boot the Linux image that resides on a disk defined to the new guest. (The booting of the Linux image is called Initial Program Load (IPL) in the mainframe world.) z/VM isolates the guests from each other (see 8.2.4, "Virtual security") so that if one guest should crash, the others are not impacted.

You can create the virtual machine and define disk or minidisks with z/VM's DirMaint or a similar program product. DirMaint allows you to use profiles and prototype files, so you can create hundreds of virtual servers quickly and easily with no conflicts of resources. (See Linux on IBM zSeries and S/390: ISP/ASP Solutions for detailed instructions.)

Having created a virtual machine, you can proceed to install a Linux image in it. For a small number of images, you could install each Linux guest from scratch, as you would with a set of CDs from one physical server to the next. If you require a large number of images, then some Linux distributions offer a means of repeated installs (such as the YaST shortcuts in SuSE and the mkkickstart utility in Red Hat). However, using z/VM, there is another simple way to avoid such repeated installs.

Without even having a single Linux image running, the z/VM utilities allow you to copy and duplicate a guest's disks. Companies can install and configure one Linux image; and when they are satisfied with that image, they use the z/VM copy utilities to create a master copy, called the golden image. The golden image is then copied to produce the next Linux image. This process of copying is often referred to as cloning Linux images. See Figure 9-2.

Figure 9-2. Cloning Linux images

When cloning Linux images, a small number of modifications must be made in order for the new image to be useful. For example, the host name and IP address must be changed. An efficient cloning scheme keeps these changes to a minimum. Even if more changes are needed, it can simplify setup to use the golden image as a base. However, an image with many modifications, for example, a different application suite or a different level of glibc, is not strictly a clone, but a new unique image. Each unique image requires separate administration effort. Keeping the number of unique images down saves administration cost.

When cloning z/VM Linux images, you need to copy the virtual machine definition and the disks. Typically, the disks that contain the system executable libraries and much of the system parameters are cloned, while the local data disks are just defined and initialized. The new guest must have a unique identity. Hence z/VM unique items (such as the guest name and local disks, on the one hand) and Linux items (such as the host name, IP address and root password, on the other hand) need to be changed.

Figure 9-3 shows a guest being cloned into another guest. The first ("golden") guest has two disks with system data and one disk with image-specific data. For example, A000 could contain the kernel and glib, A001 could contain data that needs customizing (such as the IP address) and A002 could contain the applications or databases. In the copy process, the system data are copied over. The second disk will need some customization pertaining to the new image. The third will be filled by the owner of the guest and is separate from the cloning process.

Figure 9-3. Cloning Linux images by using VM guests

Cloning with z/VM gives you a unique opportunity to save disk space. When you create Linux images by copying minidisks, if you were to place the system data on one minidisk and then for each image simply define a read-only link to it, you in effect have only one real instance. This is shown in Figure 9-4. Hundreds of users can then share the disk. In addition to saving disk space, having a read-only disk makes it easier to administer the environment and control change. You must, however, check your particular distribution for what parts of the file system can easily be shared.

Figure 9-4. Saving disk space by linking read-only disks

9.3.1 Disk space requirements and file system layout

Almost all currently available Open Source executable programs fit in less than 4 GB of disk space. The file system is usually organized into a hierarchy, depending on the files. Concentrating files that need read-only access in one file system makes it possible to save disk space by sharing the file system among Linux guests. Some typical directories found in a Linux file system are:

/usr | The /usr directory contains commands and applications. This directory is a candidate for sharing among Linux guests.
/opt | The /opt directory typically contains additional software packages, such as KDE3 or gnome. The benefit of installing applications under a separate file system is that you can upgrade Linux without re-installing the applications, and vice versa. This directory is a candidate for sharing among Linux guests.
/var | The /var directory is often allocated as a separate file system (not under /). A separate /var file system can help avoid a system halt due to overflowing log, spool, or mail files. Such problems are easier to track and take corrective action when /var is a separate file system. Also, it is possible to set certain file system parameters for the /var file system in order to speed up creation and deletion of file blocks. This directory cannot be shared among Linux guests.

A planned code hierarchy simplifies setup of servers and helps simplify both a cloning process and software maintenance. A more detailed description of the Linux directory structure is in 20.2, "Overview of Linux directory structure."

9.3.2 Example Linux guest directory

The example Linux guest directory below shows the standard definition used by ISPCompany. The standard guest has 256 MB of storage, a Guest LAN, a HiperSockets connection, links one read-only disk, and defines two read-write minidisks. Customization for this directory is limited to changing the user ID (or guest name) and passwords. For ISPCompany, these changes can easily be done (for example, over the phone, while the customer is waiting).

USER LINUXE XXXXXX 256M 1024M G
   INCLUDE LINDFLT                         /Links standard DIRMAINT definitions
   SPECIAL EE00 QDIO 3 SYSTEM LINQDIO      /Adapter for QDIO Guest LAN
   SPECIAL FF00 HIPER 3 SYSTEM LINHIPER    /Adapter for Hipersocket Guest  LAN
   LINK LINUX0 A000 A000 RR                /Link the read-only common disk
   MDISK A001 3390 1 2000 LIN008 MR RXXX WXXX MXXXX      /Define minidisk A001
   MDISK A002 3390 1 2000 LIN009 MR RXXX WXXX MXXXX      /Define minidisk A002

Standard settings that are common across all the Linux guest directory entries can be made available through the INCLUDE command.


9.4 Purpose of Linux images

You may not want to have all identical Linux servers. A typical server farm needs several kinds of servers. Each instance of the Linux operating system we call an image. As you customize a Linux image for its particular function, it becomes a specific server. For example, you can have an image that is configured to be a domain name server (DNS) serving the internal local area network, or an Apache HTTP or https server, or a firewall. Before implementing a server farm on Linux, it is advantageous to think about what kinds of servers are needed. Code sharing across servers might be possible, thus making software management easier.

Let us look at ISPCompany for an example of code sharing and easier software management. ISPCompany offers clients a standard Linux image. A custom contract gives clients the right to change the standard image, but a normal contract states that the image should remain unchanged. The standard image contains code on a read-only disk, where ISPCompany can easily support and manage the software. Client programs are on other, read-write disks. For example, if ISPCompany needs to apply a security patch, sometimes it can put the patch on a new read-only copy of the golden image disk. Each Linux image that boots from this disk will pick up the change. You can use z/VM to mark disks as read-only and to share one set of real disks among multiple Linux images as read-only.

If you are moving images from an existing server farm, you probably already know how many images you need, what they do, and which ones are similar. For security reasons, a Linux image should have only the operating system functions needed by the application it supports. Dividing up your servers according to the type of application they run gives you a rough view of what sets of images you need to create. Then, as you consider the dependencies which the applications have on distributions and other software, you can refine your view of what your images need.

Once you have determined how many unique images you need, you can construct them, create the unique golden copies, and then, with DirMaint, create the guests you need.

You can add resources such as processors, memory, minidisks, and so forth, to an existing Linux image by simple configuration changes which you can make dynamically. The initial guest definition can always be changed later.

The cost equation for doing hot standby can be drastically improved by using z/VM. One type of Linux image you are likely to consider in you design are low-cost hot-standby images. These images are redundant images that can share the file system, as shown in Figure 9-5.

Figure 9-5. Hot-standby Linux image under z/VM

If an image fails and has a hot-standby image, shared disks allow the hot standby to take over quickly. (If data are read-only, such as static Web pages, there is no need for data duplication or RAID-type storage.) You must decide on your particular schemes so that you can configure your system accordingly at the time you are setting up. Availability is discussed in detail in Chapter 11, "Achieving Higher Availability."


9.5 Setting up for secure and efficient I/O

A Linux image needs all three resources of a computer system: CPU, memory, and I/O. Let us present a couple of methods that can give you value in your setup. Resource sharing for Linux guests under z/VM is one of the strong points of the Linux-on-the-mainframe solution. For a guest, you can define the number of CPUs or parts of CPUs it should have, as well as how much memory it should have. In contrast to other solutions (based on individual real processors), a decision to give a guest more CPU capacity does not have to be expensive. Using the guest directory definition, administrators can change resources allocated to a guest in a matter of minutes.

Compared to over-configuring real hardware (such as CPUs), z/VM allows you to simply change the definition to either add more or take some away. Resources taken from one guest can easily be made available to others.

The third resource, I/O, can also be set up to be shared in different ways. I/O on the mainframe is set up by defining the paths, the devices, and the relationship of devices to paths to the mainframe hardware, to z/VM, and to Linux.

The mainframe accesses devices by means of channel paths. The hardware looks only for those devices that are defined to it in a configuration file called IOCDS (I/O configuration data set). The definitions for all LPARs and the number of LPARs are defined in the IOCDS file. If a device and its path are not defined in the IOCDS, the device is invisible and inaccessible to the hardware and all operating systems and programs that run on the LPAR, even if there is an actual, physical cable connection. If a device is defined in the IOCDS, it will be visible and usable as long as it is physically present. A device can be defined in the IOCDS file and not be physically present without causing any errors. In fact, it is common practice to overconfigure the I/O paths to support availability and future growth. For example, you can define a fiber channel with a new set of ESS storage in the IOCDS. You can then later "hot-plug" the ESS without having to change the configuration file.

z/VM can see the devices on all channel paths known to its LPAR (or the machine, if z/VM runs natively). A z/VM guest can see only those devices that z/VM defines to it. These definitions can be changed dynamically from z/VM. z/VM can also split the real devices into smaller virtual devices and control the guest's access at the level of the virtual devices.

On Linux, you have two possibilities. One alternative is to use the auto-detect capability. Linux attempts to access all devices it can find. Thus, if there is an actual cable connection to a device, and Linux has a channel path that can access it, the device will be accessed. In an environment where resources are heavily shared, you might want to avoid this. For instance, you can use the z/VM guest setup to rule out what Linux should not access.

The alternative is to instruct a Linux image to look only for specific devices. Linux is then unable to see any other devices, even if the hardware definitions and z/VM have made them available.

It is possible to overconfigure the paths for the hardware, overconfigure z/VM, and (in the guest definition) restrict Linux' access to devices. By constraining each of the guest's definitions to only the devices actually needed, you effectively protect the data on the devices from misuse by other guests. I/O definitions for a guest can be changed dynamically.

9.5.1 Booting and automatic find

While Linux on the mainframe gives you great flexibility when configuring channels and devices, it may be helpful to know something about how Linux assigns devices so that you can use the flexibility to your advantage.

Unless the kernel parameter file contains an entry for disk or minidisks, Linux on the mainframe assigns disk devices (disks and minidisks) based on the order in which the disks are found at boot time. If there is no such entry, the disks are ordered according to the subchannel number allocation that takes place at boot. To be on the safe side, it is strongly recommended that you list the disk and minidisks that you want Linux to use in the kernel parameter file. This assures a well-known naming convention that commands and applications can safely assume. Here is an example of a disk entry in the kernel parameter file:

dasd=200 mdisk=201,202,203 root=/dev/mndc ro

This entry ensures that dasda maps to disk 200, mnda maps to minidisk 201, mndb to 202 and mndc to 203. Also, mndc (203) is where the root file system will be mounted. With this entry in the kernel parameter file, these allocations are assured at every boot. Without it, you could have unpredictable allocations, especially if you (or someone else) change the I/O configurations in Linux.

9.5.2 Network considerations

The network setup is central to the design of the solution. Linux on the mainframe offers two unique values for network setup.

  • You can share the network interface card. z/VM lets you share the adapter across virtual servers. For example, an OSA-Express card with the Gigabit Ethernet feature allows up to 15 LPARs in a z900 to share one physical port.

  • Alternatively, you can easily set up a unique Linux virtual router. A virtual router avoids the cost of a real router. Details on configuring networks in a box and internal LANs is covered in Chapter 10, "Communicating in a Virtual Environment."

9.5.3 File system considerations

Availability of a system involves two key aspects: avoiding system failures and if the system does fail, ensuring a quick recovery from the outage. The file system recovery can add a significant amount of time to the recovery process.

Several different file systems are available on Linux. Broadly speaking, file systems fall into two distinct categories: conventional and journaled.

Conventional file systems

A conventional file system must be contained within one physical device. This type of file system uses a delayed write protocol which implies that recovering the file system after an outage requires scanning the disk and reconstructing the file system. The advantage of conventional file systems is that these are relatively fast in I/O processing. One down side is that these file systems typically limit the file size to the device capacity. Additionally, if a check or repair of a conventional file system were necessary, the process can take a very long time if it is on a large disk. ext2 is an example of a conventional file system. With the Linux 2.4 kernel, it is the de facto standard Linux file system. (Some distributors are now changing the default file system from ext2 to the journaled file system ext3.)

Journaled file systems

A journaled file system keeps a record of all changes that are made to data held on the file. In the event of a software failure or any data integrity problems, recovery and repair can be completed quickly using the journal instead of restoring by hand. For example, restoring a file system to a consistent state takes seconds or minutes, rather than the hours or days it can take with a large non-journaled file system. However, journaled file systems do incur a performance hit because all changes to the data must be recorded. Examples of journaled file systems are:

  • ext3 is a journaled follow-on of the ext2 file system. It is compatible with the ext2 file system and requires kernel 2.4. (http://www.ibm.com/developerworks/linux/library/l-fs7.html)

  • The Journaled File System (JFS). (http://oss.software.ibm.com/developerworks/opensource/jfs/)

  • The Reiser file system (reiserfs). (http://www.namesys.com/)

Other file systems

There are a great many other types of file systems available. Some of the more common are:

  • nfs, the network file system, is used for accessing remote file systems over the network.

  • swap, the swap file system, is not mounted. It is used to page out any unused memory pages.

  • procfs is a virtual file system that exists in memory. The kernel uses this file system to make certain that key system information is available to programs through standard file operations.

  • smbfs is the Samba file system, which allows file sharing with Windows clients.


9.6 Summary

In most cases, setting up Linux images on the mainframe hardware means setting up virtual hardware. In many cases, setting up virtual hardware is easier, more flexible, and cheaper than working with real hardware; for example, network setup can be easier because there is no need to install physical machines and cables. You can add and remove resources to meet your needs without having to commit new, extra hardware resources. Additionally, virtual over-configuration makes the environment flexible.

Redundancy can be configured into the LPAR and z/VM guests to allow for recovery from most hardware single points of failure. Hot-standby configurations are economically attractive.

If your company doesn't already have a predisposition toward a distributor, you will have the luxury of choosing the distributor based on what minimizes your need for making changes and getting the types of additional support you might need. The service contract should allow for the changes you plan to make.

The configuration capabilities of the mainframe, z/VM and Linux give you enormous advantage when you need to add extra virtual servers or resources, or if you need to try out new things. You can do so quickly, easily, and at no extra hardware cost. For example, you can create a new Linux server within minutes to try out some new software or idea.


Chapter 10. Communicating in a Virtual Environment

In a large server farm environment, server-to-server communication is crucial. In stand-alone server environments, this is done with a physical network, real cabling, routers, and associated hardware. Acquisition and operational costs are involved, and as the size of the environment grows, so does its complexity.

A unique communication method available in z/VM involves using the memory bus to implement the communication paths. The memory bus replaces hardware cables, and memory-to-memory transfer replaces hardware adapters.

In this chapter, we explore these questions:

  • What communication methods are there for Linux on the mainframe?

  • How can a virtual network reduce complexity and cost for server farms?

  • How is security improved by using virtual networks for Linux servers?


10.1 Communication methods under z/VM

One advantage of consolidating servers on the mainframe is the elimination of physical network interconnections between virtual servers. Replacing the physical cabling with memory buses inside the machine also improves the data rate and lowers the latency. In zSeries servers, a high-speed interconnection for TCP/IP communication (HiperSockets) enables TCP/IP traffic to flow directly between partitions. HiperSockets exploit the "network in a box" concept that transmits LAN-type data at near memory speeds.

The z/VM computing model, with multiple operating systems each running applications as separate guests, yet with a need to communicate among themselves, naturally led to thinking of a mainframe system as an extensive network of cooperating virtual nodes. This was the springboard to the early development of mechanisms for communicating between virtual machines, managing such communications, and ensuring the integrity of the system's data in such an environment. In fact, because this computing model required an architecture where one virtual machine requested some system service of another virtual machine, VM could be considered the first implementation of a client/server architecture. This natural affinity between the virtual machine computing model and today's network computing model makes z/VM an ideal infrastructure for network-connected Linux systems.

z/VM offers options for communication among virtual servers that greatly reduce complexity. A "virtual network cable" is nothing more than a Control Program service that transfers data from one memory location to another, making the communicating virtual server images believe that they are sending and receiving data over a real network connection. Memory-to-memory data transfers are fast, and they get faster as processor speeds increase. Upgrade to a faster real processor, and the data transfer rate of your virtual network automatically increases in speed.

z/VM provides several types of communication. However, the ones you will probably use in a Linux-on-the-mainframe environment, besides the real connection to the Internet or intranet, are virtual channel-to-channel adapter (CTCA), virtual Network Interface Cards (NICs), and Guest LANs.

10.1.1 Virtual CTCA

Channel-to-channel (CTC) communication was developed before today's high-speed networks. Originally, CTC simply enabled two S/360 or S/370 processors to communicate using a special protocol called CTCA. The processors used channel paths for the connection to replace, for example, communication over a telephone line. In other words, CTC is a point-to-point connection. A CTC connection is considered a secure communication path, because it involves only two parties directly connected only to each other through a channel path that is entirely on the secure raised floor.

Today, a real channel-to-channel adapter is a common method used to interconnect pairs of mainframe systems. z/VM has virtualized this connection. z/VM provides the ability to define virtual channel-to-channel adapters, which can be useful in connecting virtual machines running the CTCA protocol without requiring real CTCA hardware. The virtualization makes a CTC faster. In a Linux-on-the-mainframe environment, this is useful in connecting Linux guests to other guests, such as VSE/ESA, or z/OS. Virtual CTCAs can also be used to connect Linux guests to each other.

Although virtual CTC lets you establish point-to-point connections between pairs of images, it might be more practical to use Guest LANs when many images need to communicate.

10.1.2 Network in a box: NIC and Guest LANs

z/VM Version 4 includes support for virtual NICs. NICs are used to connect to a local area network (LAN). Specific NIC protocols supported by z/VM include HiperSockets and OSA-Express QDIO mode. Real HiperSockets are used to connect server images in separate LPARs on a HiperSockets LAN. z/OS, VSE/ESA, Linux on zSeries, and z/VM all support the HiperSockets protocol. Like a real HiperSockets connection, virtual HiperSockets are used to connect server images in virtual machines, but data are transferred at near memory speeds. The same is true for virtual QDIO connections. Virtual HiperSockets and virtual QDIO NICs are used to connect a guest to a "virtual LAN"—a LAN totally inside z/VM.

In z/VM Version 4, these virtual LANs are referred to as Guest LANs. A server image running on z/VM can connect to a Guest LAN using virtual HiperSockets or queued direct I/O (QDIO) NIC. Virtual HiperSockets and QDIO connections look just like the "real thing" to a Linux server or to any other software that supports the real connections. Unlike the real thing, however, there is no limit on the number of virtual HiperSockets or QDIO connections that can be defined in a z/VM environment. And there is no predefined limit on the number of Guest LANs that can be created.

Guest LAN support eliminates some of the point-to-point network management challenges that exist when using virtual CTCA while providing almost all the value. A Linux server connected to a Guest LAN can communicate with other virtual machines connected to that LAN (see Figure 10-1). Unicast, multicast, and broadcast communications are possible with a Guest LAN. (OSA Express Guest LANs support unicast, multicast, and broadcast. HiperSockets Guest LANs support unicast and multicast.)

Figure 10-1. A Guest LAN connects z/VM guests

z/VM is designed to enable virtual NICs and Guest LANs on mainframes that do not support real HiperSockets, such as the IBM 9672 G5/G6 models and the Multiprise 3000. This enhances the virtual networking environment on those machines, while letting companies prepare for a real HiperSockets environment before moving to a zSeries server.

There can be "system" Guest LANs and Guest LANs that are associated with a specific z/VM guest (for example, a Linux image). System Guest LANs exist independently of any active, logged-on guest, while Guest LANs associated with a guest exist only as long as the guest is active.

For example, assume ISPCompany (as an ISP and outsourcing provider) has a client that has six servers. ISPCompany can host these six servers on a mainframe and map the real LAN that connected the servers to a Guest LAN on the mainframe (see Figure 10-1).

While connected to a Guest LAN, a Linux server can also connect to a real HiperSocket connection and communicate with other servers in a different LPAR on the same mainframe. This offers companies a way to route Guest LAN traffic to another LPAR without requiring each Linux image on the Guest LAN to access a real HiperSockets connection. For example, in order to let one Linux server communicate with the back-end z/OS, StoreCompany uses a real HiperSockets connection (Figure 10-2).

Figure 10-2. HiperSockets connect a Linux server to an image on another LPAR

10.1.3 The value of HiperSockets for Linux

Many data-center environments today have multitiered server applications that include middle-tier servers surrounding a zSeries data and transaction server. Connecting this multitude of servers requires the complexity and cost of many networking components. Furthermore, the performance and availability of interserver communication depend on the performance and stability of this set of connections.

Consolidating this middle-tier workload as multiple Linux virtual servers on a zSeries machine requires a very reliable, high-speed network over which these servers can communicate. zSeries HiperSockets provide this. In addition, these consolidated servers also have direct high-speed access to database or transaction servers running under z/OS on the same zSeries machine.

The left side of Figure 10-3 shows a server farm surrounding a zSeries machine with its corporate data and transaction servers on z/OS.

Figure 10-3. An overview of server consolidation based on Linux

The right side of Figure 10-3 shows a Linux-based server consolidation on a zSeries machine. Servers can communicate on the zSeries machine via HiperSockets. In addition, the network connection for all servers is concentrated over a few high-speed OSA-Express interfaces.


10.2 Scenario: networking in a virtual environment

The following scenario illustrates how useful z/VM virtual networking can be:

Figure 10-4 shows a typical multi-tier application infrastructure that many companies have deployed using discrete, real server hardware and networking equipment. StoreCompany deploys this configuration on its mainframe for its online catalog business. The significant difference in this configuration is that the server images shown are actually running on a single IBM zSeries machine. Two logical partitions (LPARs) are configured on the zSeries machine. One is running z/OS and hosts StoreCompany's regular business applications unchanged, including the DB2 database server, the CICS transaction server, and the Websphere Application Server (WAS). The other LPAR runs fifteen Linux images as guests under z/VM, hosting a mix of application and networking software.

Figure 10-4. StoreCompany virtual network (z/OS network not shown)

A real HiperSockets connection (HLAN1) passes network traffic between the Linux images on z/VM and the z/OS database server. Linux firewall servers control access to the real HiperSockets LAN that connects to the z/OS LPAR. Two z/VM Guest LANs (GLAN1 and GLAN2) are high-speed, easy-to-use virtual LANs that isolate and control network traffic among the various applications that are running in the Linux images. Two of the Linux images running on z/VM function as network routers. They are connected to real network adapters (OSA cards) and route traffic between the "outside world" (XLAN1) and the virtual server environment being hosted on z/VM.

Let us follow the flow of traffic into this virtual server environment from an end-user perspective. This scenario could be deployed as an intranet or Internet solution. In either case, users access the StoreCompany environment by specifying a Web address (URL) on their browsers. The switch passes connection requests to one of the z/VM Linux routers attached to the external LAN (XLAN1). Several physical Internet connections come in through the switch, providing redundant external access possibilities. Having two Linux routers configured for this environment provides a level of availability for the services offered. If one of the routers fails (or, more likely, requires service or a software upgrade), the other Linux router then distributes the requests over the HTTP servers. A second Linux server running on z/VM does not require the duplication of hardware expense normally incurred when deploying this environment on real servers.

The Linux routers are connected to z/VM Guest LAN number 1 (GLAN1). Also attached to GLAN1 are the HTTP servers that process Web requests and perform load balancing of the requests. The Linux router that receives an HTTP call passes the request to one of the Web servers connected to GLAN1. Assuming this is a user's first access to the services offered by this environment, the Web server probably sends back a home page to be displayed on the browser, reversing the flow of network traffic that connected the user to the Web server. When a user clicks on a link that requests a service, the HTTP call is returned to one of the HTTP servers. If the request requires access to one of the Linux WebSphere Application Servers (WAS) running in the virtual environment, the HTTP server calls the WAS server, which processes the request.

The Linux WAS servers in our scenario are connected to a second z/VM Guest LAN (GLAN2). Communication between the HTTP servers and the WAS servers is controlled by the two Linux firewall servers that are connected to both Guest LANs. Once again, system availability is ensured with a redundant set of Linux images providing firewall services between the LANs.

Once the request has reached the WAS server, it accesses additional server images as needed, depending on the function it is going to perform.

If a WAS server needs to access data found in the z/OS DB2 database, it connects to one of the Linux firewall servers for GLAN2, then accesses the real HiperSockets LAN in order to pass the data request to the DB2 server on z/OS, which is also connected to the real HiperSockets LAN. This connection is also used when an CICS request comes in through WAS that requires processing by the z/OS CICS server. Firewall redundancy has been configured for the HiperSockets LAN as well.

This environment provides a number of benefits:

  • Hosting multiple, independent server images on a single machine reduces complexity.

  • Redundant servers can increase availability, improve the quality of service, and simplify the maintenance of a virtual server environment.

  • Virtual networking saves the expense of real networking equipment, while achieving high-speed communications and easing configuration and reconfiguration.

  • Standard networking skills can create a virtual server environment on z/VM. A virtual network looks just like a real network on paper but does not have the cost of a real network.


10.3 Virtual failover solutions

Another consideration for server farm operations is failover. Because z/VM controls the hardware, it can be seen as a single point of failure. Therefore, you might want a failover system in case z/VM suffers an outage.

Depending on company policy, there are three possibilities for a backup system: a duplicate image, a duplicate z/VM on another LPAR, or duplicate machines.

To maintain service during a system failure, company policy might state that a duplicate server must be available to back up the primary server. A physical hot-standby configuration duplicates acquisition and environmental costs for a server that you hope will never have to be online. With z/VM, the costs associated with a backup Linux virtual machine waiting to become active are minimal. If a hot-standby server in the form of a duplicate image is activated, it simply uses the system capacity relinquished by the primary server that just failed.

zSeries 800 or 900 machines can be partitioned into up to 15 logical partitions using the PR/SM feature. One partition can contain a z/VM that acts as a hot standby for the z/VM system in another logical partition. This may require some custom programming (for example, developing a "heartbeat" monitor to verify the operational status of the systems, and programmable operator scripts to initiate procedures on the standby system). Linux-based failover solutions can also incorporate a secondary z/VM system as part of the approach. See Chapter 11, "Achieving Higher Availability," for more information.


10.4 Communicating with the outside world

We have covered communications inside the mainframe; let us now look at communications with the outside world. High-speed communications between the mainframe and external networks are primarily supported through these communication adapters:

  • Open Systems Adapter-Express (OSA-Express)

    This TCP/IP-based adapter is important for server consolidation, because it is designed to communicate between a consolidated server environment (that uses virtual adapters and IP addressing) and the "outside world." The OSA-Express adapter can use the QDIO architecture to provide direct connectivity between applications running in a Linux guest or Linux LPAR, and the other platforms on the attached network. The OSA-Express adapter consists of different features: Gigabit Ethernet, Fast Ethernet, ATM (asynchronous transfer mode), and Token Ring.

  • Channel-to-channel (CTC) adapters (ESCON and FICON)

    A CTC connection is the typical point-to-point high-speed connection between mainframes. The Linux on zSeries CTC device drivers can be used to establish a point-to-point TCP/IP connection between two mainframe Linux systems, or between a Linux system and another operating system such as z/VM, VSE/ESA, or z/OS.

For a description of the features and adapters, see 20.4.8, "Network device drivers and adapters."


10.5 Summary

This chapter talked about internal communication and communication with the outside world. There are several ways of communicating in a Linux-on-the-mainframe environment. z/VM provides high-performance communication among guests running Linux and other operating systems. Point-to-point connections can be implemented with real or virtual CTC connections. LAN-type connections can be created with real or virtual HiperSockets, or by using virtual Guest LANs. Virtual LANs are cheaper, easier to set up, faster, and more flexible than real LANs. The benefits of virtual networks for your business include:

  • Decreased hardware cost of hubs, switches, and cables.

  • Improved performance from latency and bandwidth of memory-to-memory transfers.

  • Improved security due to the lack of hardware cables to "tap."

  • Simplified administration. The logical administration is the same (nothing new for the network administrator to learn to get the value) and the physical administration practically disappears.

  • Reduced network outages due to failures of physical components.

An important communication adapter for the communication with the world outside the mainframe is the Gigabit Ethernet OSA-Express adapter.


Chapter 11. Achieving Higher Availability

The mainframe enjoys the reputation as the most reliable server hardware on the market. One of your reasons for considering Linux on the mainframe might be the outlook of deploying highly available Linux application servers.

In contrast to what many people believe, high availability is not a matter of all or nothing. High availability is driven by trade-offs between business needs and costs. Before working towards high availability, it is important to have a clear idea of what exactly is required. For example, high-availability solutions are often not meant to cover disaster scenarios. In this chapter we examine some availability requirements in relation to what we mean by high availability.

This chapter builds on the other chapters in this part of the book. It describes configuration considerations that might help you to improve the availability characteristics of your Linux servers. We look at various configuration options of the zSeries hardware and the z/VM virtualization that can decrease the probability of an outage and speed up the recovery of a failed Linux image. The corresponding management tasks for keeping an already-running Linux server available are covered in the next part of the book.

We use examples to show how the methods that are available today allow some of the Linux on the mainframe server farms to be configured for specific application workloads, such that any likely failures would not affect the users' access to the services.

There is a rapid evolution of tools for creating and managing highly available Linux servers. In the last topic of this chapter we point to some promising developments that are in under way as we write this book.

We will explore questions such as:

  • What is meant by high availability?

  • How does z/VM help with configuring for higher availability?

  • What types of availability configurations are the easiest to implement with Linux on the mainframe?

  • What will be coming next for Linux high availability?


11.1 What is high availability?

Availability is an attribute of an entire system, and is always measured from the user's perspective. High availability is a complex term to define. Most people agree that high availability is an attribute of a system, whereby the system is available to its target users almost all the time. High availability starts with a minimum of 99.99% availability. That is, a particular server configuration is said to be "highly available" if the combination of both planned and unplanned outages in a year is less than one hour. All the pieces of a high-availability solution need to be simultaneously available, as shown in Figure 11-1.

Figure 11-1. A typical two-tier application solution

High availability does not imply that there are never any failures within the system. Instead, the design of a high-availability system accepts that failures occur and focuses the effort on providing continuing service in spite of the failure. So, besides a focus on preventing failures by having robust components, a design for high availability also requires that recovery from failures is so fast as to not allow the failure be visible to the user as an outage.

Building a high-availability application server is a complex process that starts with the definition of the level of service required. Typically, the analysis includes a list of the sources for potential solution failure, and evaluates cost/benefit trade-offs of various recovery designs for potential failures. The higher the availability required, the more it is likely to cost in terms of development, deployment and keeping the solution running. Starting with the right components improves the likelihood of success at an acceptable cost.

A system is only as good as its weakest part. Linux and the mainframe are only a strong foundation to aid in the construction of a high-availability solution. Much of the effort in building such a solution will go into the application software, the middleware, and the management tools that are parts of the system. Here we will focus on what Linux and the mainframe can contribute to an overall high-availability solution.

zSeries and z/VM, having evolved over decades, are both inherently reliable and come with a rich set of tools. Linux, despite being only about a decade old, has earned a reputation for availability and is vigorously growing its tool-set that aids the rest of the software stack in constructing the environment for high availability. The current "state of the art" with Linux is still a distance away from allowing the easy configuration of application servers that meet such a stringent requirement as less than one hour of downtime a year. Even with a less-than-complete Linux tool-set, the existing tool-set, together with the capabilities of zSeries and z/VM, allow you to create application servers where users will not perceive a service outage even if a processor, disk, or Linux image itself, were to fail.

The cost of configuring for high availability is directly related to which risks of failure you want your system to be able to survive. As you can well imagine, it is a lot harder to ensure availability across an earthquake, explosion, or other such major catastrophe, than across a hardware or application failure. Is it enough to have two independent connections to the Internet backbone? Our example ISPCompany did not think so. They installed four separate backbone connections from four different suppliers. They also saw two independent potentials for failure: the bankruptcy of a supplier, and a potential for off-site damage to the fiber connection. Furthermore, they needed to ensure that they had the required bandwidth in case of failure.

To provide a high-availability solution, your design team needs to clearly understand the risks-versus-cost trade-offs that are important to your business. An application solution design is complex, and highly dependent on the particular situation. This chapter focuses on unique opportunities that Linux on the mainframe can contribute to your high-availability design.


11.2 The zSeries hardware availability

At the turn of the century when IBM reorganized its disparate processor divisions into the eServer brand, each of the four server teams got to choose a letter to precede the word "Series." The old S/390 division chose the letter "z" for near-zero downtime, to reflect its focus on a design for continuous availability of the processors.

The design of the zSeries processor takes into account single points of failure in many ways. All levels of memory have a type of ECC (error correction code) that ensures that single-bit (often multiple-bit) failures are corrected "on the fly." The instruction processing unit itself has two identical sets of logic executing every instruction. If the results do not agree, then the instruction is retried. If the processor itself fails, a spare CPU is automatically "configured in" to replace it. In fact a number of types of CPU failures, for example the "adder logic" fails, are completely transparent to the software. The replacement processor simply receives a copy of all the current registers and begins executing again at the point where the other processor had failed.

There are times when an almost-perfectly running system simply needs additional hardware resources to avoid impacting availability. The zSeries family also has some interesting solutions in this arena. In 17.2, "Simple server hardware consolidation," there was a brief discussion of Capacity Upgrade on Demand and the ability to "configure in" additional main storage. The I/O adapters are hot pluggable.

In other words, Linux on the mainframe gets to inherit all these mainframe availability characteristics that come with zSeries. Some find the risks of hardware failure in the zSeries machine so low that having a redundant zSeries machine is only considered during disaster recovery discussions.


11.3 Redundancy and single points of failure

The typical application solution, which consists of hardware and software, has a multitude of parts that all have some failure rate. A single point of failure (SPOF) is that part of a solution, that if it fails, makes the entire solution no longer available. A typical high-availability design focuses on both the availability characteristics of its parts, as well as assuring the continuous availability the face of individual part failures.

Redundancy, having appropriate spare parts available for use, is one design technique used to address single points of failure.[13] Of course, only those with extremely large budgets can afford to have one or more instances of every resource. So the challenge in any high-availability design is determining the single points of failure within the proposed solution, and finding a way at reasonable cost to ensure a quick recovery from the failure of those parts that are important.

[13] Systems that are responsible for controlling human life-support usually have a backup system with a completely unique design, just in case the "failure mechanism" is in the original design! Redundancy is not the only way to handle single points of failure.

Given the definition of 99.99% solution availability, system owners are forced to deal withan exponentially rising costs as various design alternative look to cover more of the potential failure scenarios. High availability is really the business practice of "good enough, and no more." One common approach to high availability is to ignore the catastrophic scenarios. Many find that addressing the set of highly probable single points of failure is their cost-effective answer. This approach can be simplified even further with the assumption that one only needs to deal with one failure at a time. In other words, single failure is used during the design phase to understand failure problems, and as a design assumption to control the complexity and cost of solution development.

Interestingly, redundancy can occur at many levels. You might be surprised to know that the internals of a processor contain many redundancies. Starting with redundancy at the "processor" level, there are further levels of redundancy all the way up to completely redundant application servers. At this higher level for recovery from failures, such a design chooses to disregard which part of an application server failed and simply replaces the entire server with another one of similar capabilities. A set of redundant application servers, interconnected in a redundant way sharing the data on disk is called a cluster. In 11.4, "High availability for the ISPCompany example" we explore how ISPCompany uses a load balancing cluster to provide a high-availability, static, Web-page serving solution.

11.3.1 Linux image redundancy

It is also possible to configure redundant Linux guest images, where availability is achieved by having another Linux image take over work from the failing image. There are two different approaches taken, one called hot standby and the other clustering.

The hot-standby approach to availability has an alternate, already-booted but idle, Linux image just waiting to take over the workload if the primary image fails. With z/VM, a hotstandby image is simply an idle Linux image that z/VM will have swapped out, not consuming real resources until the point of failure of the primary server. But at that precise time of failure, the primary server stops using z/VM resources. Those resources will begin to be used by the hot standby. The z/VM approach to hot-standby images has an extremely low cost, since the hot standby does not require additional computing resources. This is especially true if you compare this to a discrete server farm implementation, where the hot-standby image is a completely-configured duplicate hardware box. Figure 11-2 is a simplified drawing of this approach on z/VM.

Figure 11-2. A simplified Linux hot-standby environment

It is important to recognize that this hot-standby approach is a special case where read/write file sharing is safe. The hot standby never writes while it is idle, so there is only one "writer" at any one time. Writing only begins when it is driven to recover. After the recovery takeover process, it is safe to write to these shared disks because part of the process is ensuring that the failed primary image can no longer write to disk.

With z/VM, this approach to Linux image redundancy has a number of significant advantages over a discrete server farm implementation. First, there is no need for additional server hardware or cabling which can be a significant cost savings. When we discussed the various types of server consolidation in Chapter 7, "The Value of Virtualization," we emphasized the efficiency with which z/VM managed the logical resources for the hundreds of guests. Here, because an idle guests put almost no resource load on z/VM, the low cost factor makes it reasonable to consider having more production images each having their own unique hot-standby image.

Configuring for hot standby with Linux on the mainframe is also easy. One uses a cloningprocess to configure the hot-standby image. In this case, the clone will also have the identical set of disks for the data as the primary server. The heartbeat process is a simple communication path between the two images exchanging an "I am well" message. Ensuring that the failed image is in fact no longer able to do I/O is as simple as requesting z/VM CP to "stop" and not reboot the particular virtual machine. The speed for recovery comes down to how fast the application data can be returned to a safe state. The implementation of the data recovery depends upon the application and the specific Linux file system.

There is a very similar story for the cluster approach to Linux image redundancy. This design is based on there being a significant number of equally capable application servers. A few, typically two, servers in the configuration are unique in that they act as workload managers, receiving the incoming work and sending it to one of the many application servers. The cluster approach has two unique advantages. First, an individual server failure does not affect the application availability, so there is zero recovery time. Only the active units of work in that server are lost. The workload manager will no longer send incoming work to a failed server. The cluster just has a little less capacity. The second advantage is that the overall capacity of the cluster can be increased with no availability impact by simply adding yet another application server which, once identified to the workload manager, becomes a candidate for its share of the incoming work.

A Linux cluster is fairly simple to implement on z/VM, and there is a related Open Source project called Linux Virtual Server (LVS).[14] The workload manager is actually likely to be implemented as two Linux images, one as a hot standby (Figure 11-3). The Open Source module, FAKE, is used to handle taking over the failed workload manager's IP address so that users do not notice that the application server has moved to a new Linux image. The cluster is a set of cloned Linux images, with a Linux file system that allows them to share a common set of disks. The connectivity among the cluster elements would be a z/VM Guest LAN. Only the two workload managers would have redundant connections to real network adapters.

[14] Table 25-6 provides a pointer to further information about this project.

Figure 11-3. A simplified Linux cluster environment

When it is required to increase the work capacity of the cluster, an additional interesting option exists with this type of z/VM Linux cluster. As with a real server farm implementation, one could define yet another Linux image. However, one has the new option to just increase the CPU shares for each of the cluster members. And if the environment was already "maxed out" on the available real CPU capacity, one could use the zSeries feature of Capacity Upgrade on Demand to dynamically add the real CPU capacity to z/VM, and thus to the cluster. See 11.4, "High availability for the ISPCompany example" for details of how ISPCompany uses LVS.

11.3.2 z/VM: Redundancy and recovery time

First we will discuss how z/VM itself uses redundancy, and also lets its guests use redundancy. Then we will discuss having a redundant z/VM.

z/VM inherits the zSeries hardware availability advantages (as described in 11.2, "The zSeries hardware availability") and can be configured to pass them on to its guests. In the case where the hardware is architecturally required to notify the software about a failure, z/VM will process that information. Most of the time it completely hides the failure event from the Linux guest, or, in some worst case scenarios, z/VM will fail the Linux guest. z/VM can then have its own automation control for a recovery of the Linux guest (which usually involves a re-IPL of the Linux guest).

The hardware configuration file (IOCDS), which controls the environment where z/VM executes, should always contain definitions for at least two of every hardware resource:

  • There should be at least two CPUs defined.

  • Every disk should have at least two paths defined.

  • There should be at least two LAN connections.

Similarly, if the intent is to have a Linux guest able to recover from single points of virtual hardware failure, each Linux guest will have the same generic list of redundancy needs as z/VM itself.

Figure 11-4 shows how the three Linux guests (Linux X, Linux Y, and Linux Z) have been defined. Each Linux guest has two virtual CPUs that it can use (CPU 00 and CPU 01). Since none of these Linux guests has dedicated real paths and devices, there is no need to specify the path redundancy. z/VM automatically handles that aspect of recovery for the Linux guest.

Figure 11-4. Sample statements that show the CPU redundancy for the Linux guests

As with any part of a solution, there is the possibility that z/VM becomes unavailable. z/VM is a very stable operating system, and although it is possible that z/VM itself crashes, it is more likely that factors such as human error or z/VM maintenance or software upgrade might cause an interruption. The same schemes as for Linux image redundancy, hot standby and clusters, can be implemented at the z/VM level.

The first item to identify with a z/VM availability discussion is the source of the risk of z/VM failure. Almost equally important is the speed for recovery from the failure. For example, if it was human error that stopped a z/VM image, it is possible to immediately restart the z/VM and then all the Linux images, and so on. While this approach is simple and inexpensive, it is so lengthy as to not fit any high-availability scenario. If a more timely recovery process is required, then a second z/VM image is needed.

The secondary z/VM image could be in another LPAR on the same machine. This approach is effective providing the zSeries hardware itself was not the cause of the outage. If you need to include the risk of the hardware failure, then the secondary z/VM must be in another zSeries machine, and possibly even at another site, all depending upon which causes of outage you want to survive.

The hot-standby choice for z/VM redundancy makes a lot of sense in the situation where all the lost Linux images can be brought up fast enough to meet the recovery time requirements. A hot-standby z/VM can even share all the disks with the primary z/VM. By using the two LPARs on the same machine approach and sharing the identical set of resources with both the primary and hot-standby LPARs, the failed primary LPAR's resources can be used by the hot standby. The same set of resources adequately serves both LPARs because of the serial nature of the requirements.

The cluster approach of two z/VM images sharing the workload is another approach for z/VM image redundancy. It is going to be required if the recovery time is tight. However, clusters imply a certain amount of actively-shared data across the z/VM cluster and are more challenging to implement.

11.3.3 Data redundancy and recovery time

The data constitute another component of the solution whose high availability must also be assured. There are two levels that need to be addressed: the hardware (the physical layer), and the Linux file systems (the logical layer) that provide access to the data. Having highly available data brings with it the need to address how quickly the data (the logical, consistent view they represent) can be recovered following the interruption of access.

Access to data on the zSeries has two hardware components: the channel path to the device and the device itself where the data resides. Simply assuring that the hardware (and virtual machine) configuration includes multiple paths to each device will assure the appropriate recovery from a channel path failure.

There are a number of approaches used with disk devices based on the concept of redundancy. By eliminating the single point of failure at the hardware level, redundancy ensures continued data availability in face of the loss of a single disk. One such approach is the Redundant Array of Independent Disks, better known as RAID. For example, with disks configured for RAID 5, one adds an extra disk to the set of data disks and keeps a type of error-correcting code for each unit of data stored across the array of data disks. If any single disk in the array fails, there is sufficient information on the other disks to rebuild "on the fly" all the data within the array. The recovery time in this implementation is zero.

Another common, but slightly more expensive, scheme goes by the names of "mirroring" or "dual copy." The concept is that every write-to-disk operation causes the data to be placed on two disks. This dual placement assures that if any single disk fails, there is an up-to-date copy available on the mirror disk. Depending upon the specific implementation when there is a failure of the primary copy, the data may be immediately available, or some specific action, usually automated, needs to be taken to make the second copy available to the running system. This duplication scheme can be extended over significant distances. This latter type of remote dual copy is typically used for availability across a single site disaster.[15]

[15] There also is a Linux Open Source project related to data redundancy that might be of use in certain types of solution design. rsync is a tool that can be used to create and manage mirror data across two remote sites. The tool is run periodically to resynchronize the two instances of the data. This approach might prove useful for a disaster recovery design, but would not handle a timely recovery of the data that would meet high-availability requirements.

There is another way that access to the data might be lost to the application. What if the software fails, leaving the data on disk inconsistent? Having a RAID cluster or a dual copy of the data will not help. What additional considerations are needed for a highly available application where the data on disk are damaged?

The typical Linux file systems are byte-stream oriented file systems that have the characteristic that the data from individual files are interleaved throughout the entire disk space managed by the file system. The allocation map as to where the data for each file reside is critical to being able to use any file. The Linux ext2 file system, in the interest of high-speed performance, does not always have this allocation map (that correctly represents that data) actually on the disk with the file system data. If Linux crashes, crucial information about each of the open ext2 file systems can be lost. In the boot process, Linux checks each file system to see if it had been closed properly and if not, then it runs a special file-check program which attempts to rebuild the allocation map. The larger the file system and the more writing done to the file system, the longer it will take to recover access to the data. And some data or files may be non-recoverable!

The Open Source community has developed a number of unique file systems,[16] each addressing some aspects of delivering both high performance while in normal use, and reasonable speed for data recovery in the event of a Linux system crash. The latest emphasis has been on journaled file systems, like JFS, ext3 and ReiserFS. These newer file systems are now showing up in the Linux distributions that support the mainframe. If you consider there to be an availability risk from a Linux crash, then when you install your Linux on the mainframe you should consider exploiting one of these journaled file systems.

[16] http://www.linuxgazette.com/issue55/florido.html is a good tutorial on various Linux file systems.


11.4 High availability for the ISPCompany example

ISPCompany's ISP division offers its small company clients a very successful static Web-page-serving service, by using the marketing slogan "Anyone in the world, day or night, can find out about your company and its services." By guaranteeing the 24x7 availability, they have differentiated themselves from the competition and have been able to attract a number of smaller businesses to using this service. The real value to ISPCompany is that these clients become excellent candidates for outsourcing a dynamic Web-page-serving environment that is even more profitable.

The design in Figure 11-5 is their implementation of the Linux virtual server (LVS) cluster technology. Most of this design is a pretty obvious cluster implementation with a workload balancer front-end (in its own hot-standby configuration). In the simple discussion in 11.3.1, "Linux image redundancy" we glossed over a rather interesting challenge. When the primary workload balancer image fails, how do users who already know its IP address manage to connect to the hot-standby image?

Figure 11-5. Implementing high availability for ISPCompany

Many users of LVS use the program FAKE to connect users to the hot-standby image. In Figure 11-5 you probably noticed the existence of an extra server, called the DNS (Domain Name Server). Among its functions is helping others to locate a specific machine by giving a translation from the IP address to the corresponding hardware LAN adapter address. The DNS can be primed with a lookup table, or dynamically learn the relationship from machines as they become active. There is a type of IP message, an address resolution protocol (ARP), that lets a machine "shout" to the local world: "I'm IP address x.y.z and my physical adapter LAN address is 123456789" (actually it is a large hex number).

FAKE relies on the fact that the DNS will listen to the latest "shout" and use it to further route messages. So, in our particular case, both the primary and secondary workload servers have two IP adapters (for symmetry, both have two.) The normal adapter on the hot standby is the address by which anyone can talk to it, in its standby role. However, once the primary image fails, the secondary image starts broadcasting ARPs, using the primary image's IP address and substituting the hardware address of its own backup LAN adapter. In fact, all the new workload manager's traffic will be done in this "fake" mode.

The Linux Primary image of Figure 11-6 uses the service's IP name as its normal attachment to the LAN. The backup attachment to the LAN is not really needed.

Figure 11-6. Performing a FAKE failover for ISPCompany

The Linux Secondary image of Figure 11-6 uses its unique real name as its normal attachment to the LAN. Therefore, it is possible to perform a heartbeat and communicate with this Linux Secondary image via this LAN attachment. The backup attachment to the LAN is used to ARP-out the Linux Primary image's IP address, claiming that it should be resolved to this backup LAN MAC address.

A source for additional information on the FAKE process can be found by looking for the LVS tool in Table 25-6 in Chapter 25, "Systems Management Tools."

There is another, more elegant and effective approach for IP takeover available on the zSeries, called VIPA (Virtual IP Address). For a source of more information on VIPA, see Table 25-6 in Chapter 25, "Systems Management Tools."


11.5 High availability for the StoreCompany OaK example

The purpose of this example is to stress that application design is a crucial part of choosing what types of availability issues need addressing when creating a design for high availability.

The One-of-A-Kind (OaK) StoreCompany project was conceived to validate a marketing idea. It costs StoreCompany little to enter special art items into its standard catalog structure and have the items drop-shipped by the artist. The proposal made by marketing was that by using business intelligence (BI) schemes they could identify, in real time, those customers likely to buy expensive OaK items. They could also identify those types of items that would be of interest by comparing the customers' buying habits with those of other customers. The approach chosen was to give the market development team a few Linux images of their own: the datamart and business intelligence images in Figure 11-7.

Figure 11-7. Implementing high availability for StoreCompany

StoreCompany considers the availability of its online catalog store to be important to its corporate image. So the design of even the OaK prototype needed to consider high availability. Since the art items are kept in the online catalog, that aspect of this prototype inherited the existing high-availability characteristics of the catalog-ordering system, as described in Appendix B, "StoreCompany."

One of the key items being investigated with the prototype was whether this datamart/BI scheme would work at all. And if it did work, could the real-time BI query be done fast enough in most cases? The purpose of the query was to produce a pop-up window to the customer. This window contained the results of a unique catalog search of the OaK items, based on the "mined" prior preferences of this and similar customers.

The availability consideration netted down to the following conclusion: there was no special treatment for availability needed in the prototype. If the BI query produced its intended results (a catalog search results page), then the customer was directed to the search facility of the catalog ordering system. Therefore, availability from that point onwards was already taken care of. If the new Linux image failed before producing the page, the result was no different, from the customer's perspective, from the query's not returning fast enough.

A restart of the prototype's Linux images would be quite fast because the two Linux images reference only static data in the datamart.


11.6 A quick look at the future

At the time of writing this book, there is already a fair amount of capabilities for building some highly available application solutions using Linux on the mainframe. This chapter has laid out some of the key areas where implementation is both feasible and cost-effective. In the dynamic world of Linux on the mainframe, there is still much going on.

The Open Source community has two major projects dedicated to high availability. This chapter has already discussed the LVS approach from the Linux Virtual Server project. The other key project is the High-Availability Linux project that is building key infrastructure for peer cluster environments. IBM, through its Linux Technology Center, is a very active participant in this latter project, as are some other companies.

The area of high availability has attracted significant attention from both small and large companies, as you can easily see from a Web search such as: Linux + "high availability." Each of the major Linux distributors is also addressing aspects of high availability as part of its mainframe offering.

The IBM Tivoli System Automation (ITSA) for Linux provides the high-level tools that make it much easier to build high-availability clusters from Linux images. These tools are particularly helpful for a cluster built on z/VM virtualization. ITSA for Linux uses the abstractions of: resource types, resource managers, grouping services, and automation policies. There is one resource manager for one or more resource types. Resource types are, for example, network adapters or file systems and disks. ITSA for Linux monitors changes to the resources contained in resource groups, and provides complex proactive and reactive management beyond simple failover capabilities. It moves beyond simple scripts to create policies that can be implemented throughout a cluster. These policies can be used to implement processes in which complex environments are described and automated.


11.7 Summary

High availability begins with defining the failure types for the application solution for which recovery is required, the speed with which the recovery should happen, and the various hardware, middleware, and application parts that constitute the high-availability solution.

The mainframe already has multiple levels of built-in hardware redundancy that contribute to its intrinsic high availability. You can configure the hardware so that the software can directly exploit various levels of hardware redundancy, such as the CPUs and channel paths to devices. Using z/VM virtualization, you can create various types of Linux configurations that provide software redundancy to an application. For example, you can configure hot-standby Linux images without investing in additional hardware.

It is possible today to implement some high-availability solutions with the existing Linux-on-the-mainframe tools. Besides developing their own high-availability products, IBM and other vendors are also supporting several Open Source projects that are working to further improve the high-availability tool set. Additional full-blown products are likely to become available during the year 2003.


Chapter 12. Systems Management

Systems management[17] aims at controlling an IT infrastructure in order to meet given business objectives. This chapter outlines in more detail what we mean by systems management in the context of this book. It discusses the main strategies and principles of systems management and how you might get additional value from Linux on the mainframe.

[17] In the UNIX world, systems management is more commonly referred to as network management. This reflects the paradigm of an environment with many physical hardware machines that are connected via a network of cables. The mainframe paradigm is that of a small number of machines that contain the critical components or systems.

With today's environments, which are growing both in complexity and in the number of systems to be managed, systems management becomes increasingly challenging. Accordingly, systems management takes a greater share of the TCO of IT infrastructures. Also, because system administration skill is sometimes difficult to find, overworked IT staff is a common phenomenon.

Most operating systems are designed to run alone, on dedicated hardware. Thus, the scope of management functions that come with them tends to be a single system. On the other hand, mainframes with their virtualization technology have been running multiple operating system images concurrently for decades. Consequently, tools for multi-system administration have been growing around the mainframe operating systems for a long time.

In recognition of the difficulties that systems managers encounter today, IBM announced that it would further develop its existing management functions. Through IBM's Autonomic Computing initiative, the vision is to deliver self-managing technologies across the entire IBM product line, including the IBM eServer zSeries platform. Autonomic technologies are self-optimizing, self-configuring, self-protecting, and self-healing. Once set up, they are envisaged to run with little to no human intervention while still managing work according to business-level objectives. All operating systems that run on IBM eServer machines benefit, including Linux on the mainframe. For more information on autonomic computing visit: http://www.research.ibm.com/autonomic/index_nf.html.

In this and the following chapters, we will look at how to manage IT infrastructures that include Linux on the mainframe to meet given business objectives. We will see how policies, procedures, and tools define the orderly scheme of how people and tools work together to achieve system availability. In this chapter, we will explore:

  • Can virtual Linux images on the mainframe be controlled effectively?

  • What can you gain from policies?

  • What tool options are available?

  • Which systems management frameworks embrace Linux on the mainframe?

  • Where does the unique manageability of Linux on the mainframe give additional value to your business?


12.1 Controlling the cost of systems administration

Systems management, like all other aspects of a business, is subject to economic constraints. A significant contribution to the TCO of an IT infrastructure is the number of people required to manage it. The number of FTE (Full Time Equivalent) employees required not only depends on the size of the installation, but also on the diversity of the hardware and software and on the desired characteristics, such as security and availability.

For example, ISPCompany runs hundreds of Linux images and expects more images as its business grows. It is inconceivable to manage all of them without the use of tools. Generally, the more numerous and uniform the systems to be managed, the more beneficial it is to use tools.

Let's make the absurd assumption that a large number of systems are managed individually and that all interventions take place through a command line at the respective system's console. Assuming that one FTE can handle a given number of systems, the cost of maintaining the systems is roughly proportional to the number of systems to be maintained. This would mean that administrators would have to be hired and trained at the rate at which the systems increase. Consequently, the potential for growth would be capped by the number of available administrators.

Tools increase the efficiency of administrators and thus keep the number of FTEs down, but are themselves associated with cost. The cost of a new tool is not only the purchase or development cost but includes more hidden aspects like deployment, training, and maintenance. Typically, the expenditure for a tool is high at the time of introduction[18] and comparatively low while it is in use. Applying the same tool to an increased number of systems might mean higher license fees, but it usually means a less than proportional increase in overall costs.

[18] A rule of thumb for the cost of deploying a sophisticated tool solution is to plan for three times the purchase cost.

Figure 12-1 illustrates the cost characteristics of people-dominated system administration versus tool-dominated administration. Introducing zSeries hardware and Linux as new elements into an IT environment is likely to mean that either existing tools must be adapted or alternative tools must be acquired to handle the new environment. Tools that accommodate Linux on the mainframe are already available. Linux has been designed as a simple system to manage; that is, it does not come with too many control points that you have to take care of. On the other hand, you can apply system or kernel patches to include any additional controls you need.

Figure 12-1. Cost of system administration for a growing number of systems

Looking for new tools implies the chance to build in the capability for handling a growing number of systems. For example, our hypothetical StoreCompany initially introduces only a few experimental Linux images. As StoreCompany gains confidence and experience with Linux, implementing the departmental servers across the company on Linux could be an attractive option. If the tooling put in place for the first experimental Linux images is chosen with that in mind, moving 50 or 100 departmental servers to Linux images can be an easy exercise. Moving to Linux on the mainframe is also an opportunity to examine whether the tools you are using can be replaced by something that is more suitable for what you want to attain.

Because systems management must be driven by business needs and not by the capabilities of a tool or set of tools, it is essential to know exactly how the business needs translate to management tasks. We argue that this is best achieved by first mapping the business objectives to policies and then to procedures. With procedures at hand, you can then assess how suitable a particular tool is for your purpose. Before we do that, we will map out the territory of systems management.


12.2 Systems management disciplines

In contrast to what you may be used to on a UNIX system, systems management on the mainframe goes far beyond just managing a few users. Systems management covers an extensive range of policies, procedures, and tools. We will divide systems management into essentially arbitrary disciplines that, nonetheless, reflect the division of responsibilities in many large IT installations. Furthermore, the disciplines we are going to use also map to the areas in which many tool providers specialize.

We will distinguish these disciplines:

  • Availability management

  • Data management

  • Security management

  • Performance and capacity planning

  • System administrator tasks

The following subsections provide an overview of these disciplines. For each discipline, there is a reference to a chapter with more detailed information.

12.2.1 Availability management

Availability management covers all the strategies and activities that have the prime objective of ensuring that systems and applications are available to the target end users for their intended use. The more central your systems are to your ability to do any business at all, the greater are the demands on availability.

Running applications on zSeries gives you hardware for which the mean time between failures (MTBF) of a machine is currently measured in decades, and Linux is itself widely valued as a solid operating system. Linux on the mainframe has the potential for excellent availability and we argue that it can be managed to leverage that potential.

With that foundation, the risks involved in quickly deploying a new application are centered on what you have most control of: the application itself.

Availability is always a high priority, but it is not the only issue in systems management. There are other aspects, such as security and performance, that compete for importance which might require trade-offs with availability. How to rank these aspects and where to strike the inevitable compromises for a particular installation can be inferred from the business objectives that the installation has been set up to serve.

While StoreCompany also benefits from good availability of a new Internet sales project, its prime concern is to ensure that the new project does not compromise existing procedures and the integrity of its database. Security issues and isolation requirements for the new project thus override the new project's availability considerations.

Chapter 13, "Availability Management," addresses the question of which level of availability is appropriate for a given project or application. It also points to some common availability management procedures that have unique considerations in an environment with Linux on the mainframe.

12.2.2 Data management

Data management covers all efforts to have the data available that are to be processed by systems and applications. In a Linux-on-the-mainframe environment, the z/VM virtualization technology can help you to share data between Linux images. In particular, both an image and a hot standby that are ready for failover can share access to data.

A company's business data are a major and vital asset that often cannot readily be recreated. Consequently, many firms have substantial procedures in place to ensure that backups are kept in safe places in case of damage to the currently active set of data. Many governments also have laid down rules on privacy and on record-keeping that have to be covered by the data management procedures.

Our focus is on the logical aspects of managing the data, rather than the physical aspects. We discuss the physical devices only to show what is there for you to use and what consequences the choice of physical devices might have on your options for handling the data. Some of what we discuss under data management is also classified as storage management.

Data management activities typically include planning for space requirements for increasing amounts of data. It also involves aspects of security management in ensuring that data are visible and accessible only to programs and persons with specific authorization to the data.

Data management needs to address the interrelationships of sets of data. Sometimes it is critical to have sets of files backed up with the content they had at the same point in time. Such backups require significant coordination with application availability.

Data management is tightly coupled to availability management. If the processor is up and the application is ready to do work, yet the data to be processed are inaccessible, then the end user sees a broken system. The mainframe allows you to attain high data availability, for example, through redundant data access paths and efficient backup tools.

Chapter 14, "Data Management," explores this discipline with respect to Linux on the mainframe.

12.2.3 Security management

The challenge of security management is to make a system as secure as possible without unduly affecting the legitimate user.

Security management protects systems from intended or accidental damage and thus supports availability. Security, however, often also conflicts with availability because it restricts accessibility and deploys barriers that only intended users and applications can overcome. Forgotten passwords can mean that StoreCompany customers temporarily cannot access their accounts with the StoreCompany Internet sales application. Trade-offs have to be made, depending on the overriding concern.

Security issues must be considered when conceiving the layout of a system, so we covered much of the security management discussion in Part 3, "Is Linux on the Mainframe for Me?," which includes Chapter 8, "Security Considerations," and Chapter 9, "Setting Up Linux on the Mainframe." Because that discussion also includes operational issues, there is no separate chapter on security in this part.

12.2.4 Performance and capacity planning

For the sake of discussions in this book, those aspects of "performance" that are used to determine system health are covered under availability management. The performance management and capacity planning chapter addresses those activities that are less directly associated with running systems.

Performance management aims to understand the behavior of the IT infrastructure (such as hardware, software, networks, etc.) to assure that resources are used in an effective way and SLAs (Service Level Agreements) are complied with. Where the results of performance analysis call for changes to the system structure, the running and the availability of the system are affected. Here, performance management intersects with availability management.

Capacity planning takes the analysis of performance data one step further. The objective in capacity planning is to anticipate the impact of changes to workload and resources in the system. The first three disciplines focus on activities that work together to keep everything running smoothly under stable conditions. Capacity planning is about predicting the behavior of the system if something were to change. It is used to answer challenging questions such as, "What happens to response time if we put this new product marketing campaign up on System 3?"

Chapter 15, "Performance and Capacity Planning," explores this discipline with respect to Linux on the mainframe.

12.2.5 System administrator tasks

Rather than introducing a multitude of additional disciplines, we have grouped all activities that do not fall into the above four disciplines into a single fifth category. It includes various activities typically focused on the underlying software infrastructure and the hardware, performed by the most skilled members of the IT staff.[19]

[19] For those with a mainframe background, this is the system programmer we are talking about. In a small UNIX shop, typically one person, rather than an entire department or division, is responsible for all the activities associated with the care and feeding of one or maybe even ten UNIX systems. Even though the focus in this book is on the mainframe environment where there will be potentially hundreds of Linux images and possibly separate departments handling the other disciplines, we chose to stay with the UNIX terminology for this specialized work.

Time-to-market for new ideas is one good reason for having Linux on the mainframe. How this potential is leveraged depends on your key staff of administrators and the tasks that they perform.

Chapter 16, "System Administrator Tasks," delves into some of the cultural (organizational) challenges that occur when marrying the tightly controlled mainframe environment with the Linux mentality of exploiting the latest and greatest. The issue of change management is the common thread across Part 4, "Making the Most of Linux on the Mainframe," in general and Chapter 16 in particular. With Linux on the mainframe, we are now dealing with a large number of Linux images sharing resources where change and control are both key to success.

12.2.6 Putting it all together

The ultimate goal of systems management is to have an IT infrastructure available to support a business. Figure 12-2 illustrates how we see the different disciplines help to support this goal.

Figure 12-2. Systems management disciplines

Data management and security management directly support availability by providing meaningful content for processing and by protecting the system from damage. These three disciplines are basically operational and require constant attention.

Performance and capacity planning and the set of tasks that have been summarized under system administrator tasks provide the foundation to assure that the system will also be available in the future. These tasks can typically be done at scheduled times and do not require a person to be on call continuously.


12.3 Policies

System administrators cannot operate with just the business objectives as guidelines. There is no setting on Linux or any other operating system where you can adjust a company's revenue or market share and business strategies. After all, market analysis is not the expected expertise of the system administrator.

System administrators need a set of guidelines that are derived from business goals and represent implementation of these goals in more technical terms. Management usually communicates such guidelines in the form of policies. To keep the focus on the key requirements, policies are most effective if they are few.

12.3.1 How to phrase a policy?

In practice, all system administrators follow policies, written or unwritten. The more extensive an installation is, the more obvious becomes the need to have formal, written, and agreed-to policies. Like laws in society, policies are needed as a guideline for potentially contentious issues within an organization. And like laws, policies can be phrased in very general terms and remain so, if they express a commonly agreed principle. Where there is less agreement, the policy is refined to a more specific statement.

For example, a common issue of contention is that of storage space usage. While available space, no matter how much there is, tends to get filled up as if by a law of nature, it is the job of the storage administrator to make sure that the company's key applications do not run out of storage. The goal of keeping storage available must be achieved within the limits of a budget. What could a policy look like that is to contain storage requirements?

A general policy could state that "All users must use space economically." This is flawed in two ways. For one, it implies that users are doing something wrong. Thus, it is likely to antagonize the very people who are being asked to cooperate—not a good approach.[20] Secondly, it does not account for application data or for the gray zone where users are consuming space through applications they use.

[20] At the following URL, a white paper explores the psychology of policies and provides a detailed example for a storage policy and a matching SLA: http://www.ntpsoftware.com/WhitePapers/docs/DoYouNeedAStoragePolicy.pdf.

Assuming that we are in an environment where users access mainframe storage through the Linux images that they use and through the applications that run on these images, a more diplomatically phrased policy could state: "All Linux images are assigned to storage groups. At the end of each month, any group approaching 90% of its storage limit is reviewed to see if it is appropriate to raise the ceiling." This raises the awareness that there is a ceiling for what can be used at any time without pointing an accusing finger at anyone. Which image is allocated to which storage group is negotiable. For storage issues, this is probably still too general. z/VM could add a refinement of this policy, since it provides the possibility to balance the relative usage of space within a group and prevent a single image from seizing most of the available space.

A more specific policy could state that there are fixed quotas for the amount of space for each Linux image. ISPCompany provides Linux images to its clients and must be very specific with what a client is entitled to. Policies are normally intended to be long-lived, so the actual quotas are usually recorded elsewhere, for example, in a service level agreement (SLA). An SLA is an agreement between two parties, for example, the IT division and the sales division of a company or between a company and its clients. It states, in explicit terms, what one party commits to deliver to the other. In our example, this could include the space quotas for users.

To manage its space requirements, ISPCompany sets space quotas in the SLAs for the standard Linux images which it offers to clients. For example, under the terms of its standard contract, ISPCompany allows 100 megabytes of storage for a Web-hosting image that is intended for static content. Linux can be installed to include the function for assigning quotas to users and groups, and most distributions includes this function.

12.3.2 Why use policies with your Linux projects?

Why do we discuss policies in a book on Linux on the mainframe? Using Linux on the mainframe means change and might call for adaptations in your policies. Especially for companies that have been operating successfully with established procedures, change can be seen as a threat to the company's success. New policies are a means of enabling the required change and at the same time confining it to where it is necessary. Thus, those in charge of the Linux project have the freedom they need without everybody else feeling that all rules are being overturned.

We asserted that Linux could provide a platform for rapidly deploying new applications in mainframe shops. StoreCompany might well have policies in place that require lengthy test runs before a system can go into production. While this is perfectly appropriate for a z/OS system that processes financial transactions, it could defeat the purpose for having Linux as a fast deployment platform. Rather than scrapping the policy for z/OS systems, the policy can be altered. It could apply different rules, depending on what kind of system and application is to run. An acceptable solution to this simplified problem could be to continue with utmost care for transactions but to add a fast path for other applications, such as new Web interfaces, to be run on Linux.

ISPCompany has a business objective of being able to quickly respond to client requests for setting up one of three types of standard Linux images. To ensure that this goal can be attained, a ISPCompany policy states that there is a fixed set of services and an SLA for each of these types. No alterations or special provisions, no matter how simple, can be made to the three low-cost types. Otherwise, staff might get carried away with accommodating minor requests for customization and affect the speed of deployment as well as maintenance. After all, ISPCompany offers a separate category where both the fee and delivery time depend on the extent of the requested customization.

Let's assume that the SLA for one of the types also states the following: "Since we do the availability management for you, you are eligible for a rebate on your monthly bill if your server is unavailable for more than a one hour period." We will build on this example in the following sections on procedures and tools.


12.4 Procedures

A procedure outlines what is to be done, under which conditions, by whom, and when, to satisfy policies and SLAs. With the example of the SLA statement in the previous section, a procedure could look like this:

  1. If an image goes down as seen by z/VM, z/VM initiates a restart, and, if successful, logs that a successful automatic restart has taken place for the image.

  2. If the automatic restart is not successful, the operations staff is notified through the health monitor GUI, and an e-mail is sent to the client contact.

  3. If the recovery is not completed after 30 minutes, a page and an e-mail are sent to the responsible system administrator.

An obvious next step to increase productivity is to automate those parts of a procedure that follow a fixed repetitive pattern. For example, you could write a script that issues a set series of commands. A more elaborate piece of code might handle more complex parts of a procedure. Step one of our example is already automated through z/VM.


12.5 Using tools

This section covers considerations and decision points about tools in general terms. There are hundreds of tools available. To preserve the flow of our discussion, we will restrict ourselves to only occasionally mentioning a specific tool. Chapter 25, "Systems Management Tools," provides a more extensive table of individual tools for each systems management discipline.

With procedures in place for the different systems management disciplines, the decisions on how to proceed in all anticipated situations are laid out. The person who follows a procedure need not be as experienced as the person who devised it. Parts of a procedure might be so well-defined that they leave no room for variation. Normally, system administrators identify such parts and write a tool that takes care of a fixed part of a procedure.

Environments with a large number of images cannot be administered at a reasonable cost without the use of tools. Using tools has other advantages, too. A suitable tool can perform a procedure faster, more consistently, and be less error-prone than a human. Hence, the question is not whether to use tools but rather where to get them and how to deploy them in a heterogeneous environment, which Linux on the mainframe necessarily is.

An example of such an environment is that of StoreCompany as represented by Figure 12-3. It does not need many Linux images and runs other operating systems in some LPARs on the same zSeries machine. On z/VM where it runs its Linux images as guests, there is also a CMS (Conversational Monitor System)[21] guest.

[21] If z/VM runs on an IFL (Integrated Facility for Linux), then CMS is limited to supporting Linux (for example, for systems management).

Figure 12-3. StoreCompany environment

The zSeries hardware is controlled by an HMC (Hardware Management Console). z/VM is controlled from a CP console. Both run on workstations in a control room.

The need for systems management tools has given rise to a number of small companies that offer excellent solutions for specific systems management tasks. There are also larger companies that offer suites of integrated tools. Linux, mainframe or not, has long been on their agenda, and the number of available tools for Linux is growing rapidly. If what you are looking for is not available today, it is worth checking again in a few months, not years. It is also worthwhile to check out the Open Source community. Someone else might have come across the same problem already and may have built a solution that is available for you to use.

Generally, you have these options for acquiring tools:

  • Use z/VM

  • Get your tools from the Open Source community

  • Develop your own tools

  • Buy your tools

z/VM is listed first because it is an obvious choice for anyone who already has or can acquire z/VM skills and because it is a platform where you can readily deploy scripts that automate other tools. However, you do not have to use z/VM. The most attractive source for tools to you depends entirely on your individual circumstances, preferences, and concerns.

12.5.1 Use z/VM

If you run your Linux images under z/VM, you already have a single point of control for them. The virtual machine control program (CP) is the place where you control the virtual hardware and do much of what you normally do at the hardware level. For example, you can start and stop your systems, assign system resources to your guests, reconfigure real or virtual hardware, or access the guests' consoles. z/VM can serve as a basic availability and performance management tool.

Having a single point of control is a great advantage if you want to monitor numerous Linux images. Instead of gyrating around the raised floor on roller skates and trying to spot red LEDs, your system administrator can check a single screen where all messages on failed images are routed.

Figure 12-4 represents the environment of ISPCompany. It wants as many Linux images as possible and uses a small number of LPARs with z/VM. One LPAR is a hot standby, ready to take over from another LPAR where critical Linux images run, in case z/VM fails.

Figure 12-4. ISPCompany environment

Because ISPCompany's main concern is to manage the virtual hardware and its availability to its clients, it chooses to do most of its systems management activities through z/VM. Using z/VM functions saves the expense and overhead of deploying tools on every single Linux image.

12.5.2 Get your tools from the Open Source community

This option carries some of the same advantages that you also get from using Linux. Open Source tools are free and quick to obtain by download. Because they are Open Source, you can tailor them to your needs, and you can exchange ideas and possibly get help from others in the community with similar needs.

You can find a large number of free tools. Some have been around for a long time for UNIX. One example is sar (system analysis report).

Other free tools have been provided for Linux by the Open Source community or contributed by companies that promote Linux. One such contributor is SGI (Silicon Graphics, Inc.)[22] with CSA (Comprehensive System Accounting) and PCP (Performance Co-Pilot).

[22] For a list of Linux projects by SGI, visit http://oss.sgi.com/projects/.

Generally, you can find good tools for specific tasks. If you have the need and the skill, you can combine these tools into an integrated solution that is tailored to your requirements.

To assure that you do not have to modify the tools for specific zSeries configurations, use the tools that come with your distribution. One decision point for using or not using a tool might be the degree to which service is available for it.

If you are prepared to compile and possibly make modifications to a tool, here are sites you might want to check:

  • http://www.linux.org/

  • http://sourceforge.net/

  • http://freshmeat.net/

Tools that are available in the Open Source community have mailing lists. You can use the mailing list to get answers for a wide range of questions that are not covered in the tool documentation or that emerge after examining the tool.

If you have to alter a tool, you might have to maintain it yourself. Hence, you are approaching the option of developing your own tools. You could also give your contribution back to the Open Source community by suggesting the alteration to the owner of the tool. If the alterations are accepted, the tool with your alterations can be maintained by the owner.

12.5.3 Develop your own tools

The advantage of a home-grown tool is that it is made specifically for your needs, and you decide what is developed and when.

ISPCompany, as a service provider, has people with the skills to develop tools. It also expects a competitive advantage from having tools in place that are not available on the market.

Some of ISPCompany's clients outsource their infrastructure to ISPCompany but want to stay in control of some aspects of their images. For example, they want to be able to do health monitoring, user management, back up and restore data, and, if needed, reboot their images. The current process starts with a client request, by note or a phone call to the responsible ISPCompany operations person. That person then acts on behalf of the client and performs the requested action from z/VM. ISPCompany is developing a tool that will enable it to automate this process. The tool includes a Web interface through which the client can pass a request to the tool on a Linux image. The tools first does a security check and then passes the request to z/VM for execution.

It is important to remember that tools not only need to be developed but also maintained and constantly adapted to a changing context. For example, new hardware and softwareneed to be integrated as they become relevant to the procedure that the tool covers. Generally, this approach is most useful where fast deployment of specialized tools is critical and where the skill to develop tools is present in the company.

12.5.4 Buy your tools

Buying the tools you need allows you to focus on your core competence (unless tools are your business). Along with a tool, you can also purchase support for its deployment and maintenance. On the other hand, you also create dependencies on the tools you buy, since you cannot easily detach yourself from release cycles and the costs associated with installing new releases.

StoreCompany, for example, established an IT infrastructure not because it specializes in computing, but because it is the most efficient means for most administrative aspects of its business. Later StoreCompany embraced the opportunities of the Internet to gain new customers and boost sales, and the role of its IT expanded beyond administration. Because StoreCompany does not intend to be in the systems management tools business, it turns to a vendor who sells integrated systems management solutions. StoreCompany puts much stress on security, and it chose Tivoli because of the way security control had been implemented within the management structure.

12.5.5 Tool options summary

The following table summarizes the options you have for tools:

Use z/VM | ... to manage the virtual hardware and do accounting at the image level.
Get tools from the Open Source community | ... to obtain tools at low cost without having to develop them yourself.
Develop your own tools | ... if you need special functions fast and if you can use these tools to a competitive advantage.
Buy tools | ... if the functions you need are available on the market and you do not want to diverge into building and maintaining tools development skills.

You guessed it, the options are not exclusive, and there is no virtue in a purist approach. However, it is useful to choose the main source of tools and then other possibilities can be used to fill the gaps. In most cases, the least you will have to do yourself is to customize tools you have bought or obtained from the Open Source community to fit your needs. Also, if you have decided to develop your own tools, you will usually find a useful Open Source project to start with. If need be, you can guide your approach to tools by specific policies that address their role within the IT infrastructure.


12.6 Using a framework

With Linux on the mainframe, you are likely to have a heterogeneous environment with multiple operating systems, including traditional mainframe operating systems such as z/VM, z/OS, and VSE/ESA. Systems management frameworks address the question of how to accomplish an integrated management of such heterogeneous environments. This section first provides a general introduction to frameworks and then gives examples for frameworks that are suitable for a Linux-on-the-mainframe environment and encompass Linux and the traditional mainframe operating systems.

During the late 1980s and early 1990s, major vendors of systems management tools began to develop substructures for integrating tools and handling the diversity of heterogeneous environments. Initially, the goal was to make the vendor's own disparate tools work together in a single, integrated, monolithic solution and to provide a single point of control from where an administrator could monitor and control all systems.

Monolithic solutions soon proved too rigid to meet the demands of highly dynamic and diverse customer environments. As a response, the major providers separated the substructure from their tools. With its interfaces published, the substructure became a framework that allowed other vendors and customers to plug in their own tools.

An open framework provides the basis for suitable plug-ins to be assembled into a set of tools that spans all systems management disciplines and all systems in a heterogeneous environment. The framework/plug-in model not only allows tool integration but also the selection of only those tools that are needed in a particular environment.

To a framework provider, the framework is a selling point for the systems management tools that come with it. The suite of tools that the framework provider offers is more attractive to customers if the framework allows other tools to be integrated. Framework providers, therefore, tend to publish their interfaces to enable other vendors and home-grown tools to plug in.

12.6.1 What is in a framework?

A systems management framework is an architecture that allows a user interface, managers, and agents to work together. Managers and agents are plug-ins that communicate with the framework through APIs (Application Programming Interfaces). Managers provide platform-independent management functions and use the agents to run system-specific code on the systems to be managed. For example, a manager that runs a system health monitor could send a request to the agents on each system to find out about free space in the file system. As a response, the agents could use a method that is specific to a particular operating system to get the data. For example:

  • Issue a command (such as df on Linux)

  • Call an API

  • Pass a control block (on z/OS)

In all cases, the agent then sends the results back to the manager. On each image, multiple agents can plug into the framework.

A framework usually includes:

Common services

that are available on all platforms, such as task management, memory management, or an interface to a security manager.

Extending a framework to support Linux merely means writing the code to map the common services to the ones provided by Linux. And then, in most cases, all one needs to do is retarget the compile to the respective hardware platform.

Connectivity

among the plug-ins. The communication is typically based on TCP/IP. It can include encryption and usually provides secure communication across systems.

A data repository

for information that is shared across the framework. For example, this could be an LDAP-based directory with an underlying SQL database. The database contains common definitions and information about the state of managed objects (for example, processors or applications).

A user interface

that provides the user with a view of the data in the management data repository and through which the user interacts with other plug-ins. The interface usually runs on a PC or laptop away from the machine room (for example, in a control room or in the system administrator's office). The user interface also connects to the framework through an API, so alternatives are possible. User interfaces are often graphical interfaces but can also be command interfaces.

APIs

for getting to common services and communication with other user interfaces, managers, agents, and the data repository.

Figure 12-5 illustrates how a framework spans the StoreCompany infrastructure.

Figure 12-5. Framework for StoreCompany

The framework is installed on all managed systems and the systems where the managers run. It offers interfaces to agents and managers and connectivity with other systems that run the framework. The managers can run on any system for which the system requirements for the framework and the manager are fulfilled. Potentially, a manager could run on one of the systems being managed, but it could just as well be on a different continent.

12.6.2 Frameworks and standards

Systems management vendors with frameworks usually provide a suite of managing functions that plug into the framework. You can usually get plug-ins by vendors other than the framework provider, and there is room for you to plug in your own management tools if you have functions to complement what is provided.

To allow other vendors to use a framework and to enable you to plug in your own functions, framework owners publish the APIs that define their architecture. The architecture describes the behavior of the framework.

There is an interaction between architectures and standards. An architecture can be raised to a standard by approval through a standards body. Framework architects, on the other hand, look to existing standards to make their frameworks compatible with tools that adhere to the same standards.

It is no surprise that different vendors often use competing standards. Some vendor APIs are proprietary, others are open standards like SNMP (Simple Network Management Protocol) or the Distributed Management Task Force (DMTF) CIM (Common Information Model). See http://www.dmtf.org for details.

The DMTF is the industry organization that is leading the development, adoption, and unification of management standards and initiatives for desktop, enterprise, and Internet environments. CIM is a common data model of an implementation-neutral schema for describing overall management information in a network/enterprise environment. It includes specifications that define details for integration with other management models, while the schema provides the actual model descriptions.

SNMP is a standard for a systems management protocol that is not only supported by many management tools but also by many "dumb" devices like printers, where the framework cannot be installed. If a tool or framework supports SNMP, it can manage these devices. The Web site for the SNMP standards body is at http://www.ietf.org.

12.6.3 Existing frameworks

Examples of frameworks that cover Linux today are:

IBM Tivoli Management Framework

The Tivoli Management Framework implementation is object-oriented. Earlier implementations were based on the Object Management Group / Common Object Request Broker Architecture (OMG/CORBA) specification (http://www.omg.org/). Its APIs were adopted by DMTF and the Open Group.

More recent development shows a shift to Java, Java Beans, and the CIM data model.

The Tivoli Management Framework has a 3-tier architecture with a TMR (Tivoli Management Region) server at the top. The TMR server can support up to about 200 gateways, each of which can connect to thousands of endpoints. This design provides scalability to a large number of systems and keeps the framework code to be installed on the managed endpoints very small.

Tivoli Management Framework comes with a base set of management functions. There are additional functions that you can optionally plug in, including a plug-in for SAP R3 support. In addition, Tivoli provides a toolkit that helps you to write your own plug-ins.

An administrator with sufficient rights can add a new endpoint to a Tivoli controlled environment by instructing a Tivoli manager or gateway to install the framework code on the new machine. The new machine then communicates with its assigned gateway or registers with a suitable gateway by means of a broadcast.

The Tivoli Management Framework includes a security concept that is based on user roles. These user roles are mapped to regions that are managed by policies.

For the Tivoli Management Framework, Linux has been given increasing attention as a platform for running mainframe managers. For example, IBM Tivoli Storage Manager runs on Linux.

For more information on Tivoli Management Framework visit: http://www.ibm.com/software/tivoli/products/mgt-framework/.

CA Unicenter TNG Framework

The Unicenter TNG Framework provides a base set of management tools, plus optional management tools that you can buy. Optional tools are offered by several vendors, including Computer Associates International, Inc. The Unicenter TNG Framework also provides the Unicenter TNG Software Developer Kit for integrating home-grown tools. Several major system vendors ship systems with the Unicenter.

TNG provides "auto discovery" when an image with a TNG agent registers with the corresponding managers. Unicenter plug-ins can use policies and correlation techniques to attain a degree of autonomous behavior and to reduce network traffic.

The Unicenter GUI and the manager functions run on Linux, so you can control your entire enterprise from a Linux image.

For more information on Unicenter TNG Framework visit: http://www3.ca.com/press/PressRelease.asp?CID=37246.

Other organizations that are worth following in this context are:

BMC PATROL

PATROL does not claim to be a framework but fits our definition of 12.6.1, "What is in a framework?." It takes a more decentralized approach than the other frameworks we have discussed.

It combines what we have called managers and agents into "knowledge modules." These plug into what we would call the framework, but BMC calls an agent. The agent/framework provides, for example, for cross-system communication.

The knowledge modules are platform- and function-specific. Because they act fairly autonomously, once they are set up on a machine, PATROL creates less data traffic than some other frameworks. Knowledge modules take a set of parameters that can be specified in a plain text configuration file.

PATROL comes with a basic set of knowledge modules. There is also a toolkit that allows customers and vendors to write their own knowledge modules. PATROL knowledge modules identify themselves by listening at a specific port.

PATROL does not have knowledge modules for the mainframe operating systems. Instead, there is a separate product, MainView, that provides administrative functions for these systems.

Because the knowledge modules can act independently, PATROL does not provide a GUI. However, PATROL provides knowledge modules that can send messages to the MainView interface, the TNG console, or the Tivoli GUI. Thus, either of these can be a control point, if needed.

For more information on BMC PATROL visit: http://www.bmc.com/homepage_elements/ii/idc/ema_bmc_patrol_brief.pdf.

OpenNMS

OpenNMS is an Open Source group that started from the network view of the world and based much of its work on SNMP.

Its code is based on today's state-of-the art open standard technologies XML (Extensible Markup Language), SML (Simple Markup Language),[23] Java, Perl, and a PostgreSQL object-based relational database.

[23] SML: http://www.xml.com/pub/a/1999/11/sml/

For more information on OpenNMS visit: http://opennms.org.

We refrain from a detailed comparison of the merits of the various systems management frameworks that can work in a Linux-on-the-mainframe environment because they are subject to rapid change. Which framework is best suited to your environment depends on how well it addresses your current business and operational needs, and on the ease with which you can integrate plug-ins to cover future needs.


12.7 Summary

We have seen that tools are available to cover the systems management disciplines of Linux on the mainframe. We have stressed that managing hundreds of Linux images calls for tools. We have argued that business objectives come first and that the suitability of tools can be assessed only once the business objectives have been translated into policies and procedures.

The systems management disciplines we have introduced are essentially arbitrary, but they still reflect the division of responsibilities in many larger IT installations. The borders between them are not sharply defined, and there are considerable overlaps and interdependencies.

Security, as one of the disciplines, has already been covered to a large extent in Part 3, "Is Linux on the Mainframe for Me?," because it must be designed into an environment from the outset. Because there is no separate chapter on Security in this part of the book, Figure 12-6 summarizes the security tasks here.

Figure 12-6. Security discipline

The following chapters each focus on one of the disciplines and point to some important decision points to consider when working out policies and procedures to govern Linux on the mainframe.


Chapter 13. Availability Management

Availability management starts with configuring the various hardware and software components with their individual reliability characteristics into a system that meets the availability requirements of the end user. It covers all the tasks associated with the delivery of service to the end users.

You might expect that Linux on the mainframe enables you to provide a higher available service at reduced cost when compared to alternative server farm configurations. In this chapter, we will cover some of the areas where we believe you will find key leverage points for this value potential.

We will regard a server as available if the end user can perform the desired functions. This definition of availability spans a wide range of failure points, from power to the hardware, the processors running, the disks spinning, and the network running to the operating system, the middleware, and applications all running and functioning together.

We will discuss all tasks listed in Figure 13-1 to some degree, but we will primarily focus on the following three availability tasks because they potentially have some unique value for you in a Linux-on-the-mainframe environment:

  • Monitoring the individual components and end user experiences. This monitoring is frequently called health monitoring.

  • Running automatic procedures to recover from various forms of failure. These automatic procedures are usually called automation.

  • Defined policies and procedures for handling changes to the system. This area of effort is called change management.

Figure 13-1. Availability management tasks

We will focus on how Linux, the mainframe, and z/VM can all contribute to the management tasks of keeping the system available according to your policies. We will explore questions such as:

  • Are there simple ways to determine the health of hundreds of Linux images on a mainframe?

  • Can Linux-on-the-mainframe applications provide a 99.9% level of availability?

  • What tools are available to manage a high-availability set of Linux images?

  • If the required level of availability is greater than what comes with Linux, what options are there?


13.1 Availability policy

Most organizations have a service level agreement (SLA) with their end users, often in the form of an explicit contract. The intent of an SLA is to define what system resources will be provided, on what schedules and, possibly, even at what level of performance. For example, StoreCompany has the following SLA for its internal mail servers:

StoreCompany SLA

The mail server will be available from 7 a. m. to midnight, and no outage should last longer than 10 minutes.

These service level agreements become the availability policy that drives the availability management decisions for the various services.

However, given that availability is now usually defined from the end users' perspectives, there are some very interesting secondary effects that can aid in a system design based on Linux, z/VM, and the mainframe. Here are some examples that show the valuable benefits that both end users and the IT shop can have from properly crafted policy statements.

Based on the simple SLA for StoreCompany's mail server, even if that server is down between 3 a. m. and 5 a. m., service could still be considered 100% available because the scope of the availability is 7 a. m. to midnight. The system administrator can use the non-production time for change activities such as applying preventive maintenance to forestall problems. Not needing to apply changes during the defined system available times permits a less complicated system design, which in turn tends to improve the overall system reliability. Currently, most servers do not need to provide their service 100% of the time in order to meet the requirements of end users.

How often have you entered a URL, seen some activity occurring at the bottom of the Web browser, and then the activity seemed to stall? Most of us just simply try it again. We do not say that the server is unavailable if we manage to connect after a few tries. In truth, during that brief period the Web server might have been down. Then the proxy server, noticing that one server was down, routed all new requests (for example, our second attempt) to other servers that were still functioning properly. Because the user does not require the service from a particular server, the IT shop can use this less expensive design involving the proxy server to meet availability (see also 11.4, "High availability for the ISPCompany example").

Managing availability is a juggling act of providing service to end users in a way that they are satisfied, based on the SLA. Providing higher levels of availability than what is required may, in fact, be a waste of valuable corporate resources.


13.2 Health monitoring

Most IT organizations try to anticipate and avert system problems before calls come into the help desk. The larger the organization and number of servers, the more likely it is that the organization has one or more display windows whose colored icons represent the health of components of various services provided to end users. These health monitors usually come with two functions:

  • Agents that reside close to the service to be monitored and that send out either regular status or exception status

  • A presentation manager that collects status information from various agents and aggregates that data into a meaningful presentation format (including, perhaps, a page or e-mail to the staff alerting them that a problem needs to be fixed)

What is unique with Linux on the mainframe is that due to virtualization, z/VM has some very useful data about both the individual guests and the aggregate usage of the real and virtual hardware resources. z/VM needs this data in order to manage the guests. Thus, no significant additional cycles are needed to share this data with a health monitoring tool. To the extent that most outages of service can be seen with hardware level performance metrics (for example, the CPU is no longer busy or the outbound LAN traffic has gone to zero), the operations staff can use this easily available data as a first level indicator of potential problems. While some other servers do have some rudimentary hardware monitoring capabilities, it is usually quite expensive to tie that information into the health monitor.

Most organizations also include two other crucial functions that move health monitoring into true health management:

  • A manager that collects and logs data about availability

  • A presentation manager that allows both a global and specific drill-down control of the managed environment

A tool that includes all four functions of health management becomes a single point of control for the system with all of its disparate images. Some installations have built their Linux systems' health management tool on top of z/VM because z/VM covers all four functions quite well.

In the following section, we will expand on the "control" aspect of health management.


13.3 Automation

Availability management aims at responding to anomalous events fast enough that the damage is contained and repaired before the SLA criteria are missed. Typically, humans take at least a few seconds to observe, analyze, and react to a situation.[24] With the speed of modern processors, a lot of damage can occur in a few seconds.

[24] As today's systems continue to improve, these anomalous events happen less and less frequently—a good thing. As a result, we tend to forget what the proper procedure is to respond to the event—a bad thing. Typically, there is a written procedure on how to recover. But a site is almost guaranteed to have a lengthy outage if the operator first has to read up on what to do.

Automation refers to a program suite that performs the "observe, analyze, and react" processes at machine speeds. At the heart of the automation tool is the automation engine that gathers events, correlates information from various sources, and manages the system state changes with scripts. The key value of an automation tool is the speed at which a new system state (for example, the database manager is down) is recognized and at which it causes the predefined transition policies[25] to be executed. At the time of the writing of this book, automation tools cannot provide recovery itself. Instead, they provide a framework where you can plug in your recovery (state change) processes, typically in the form of a program script.

[25] Policy is one of those overused words today. So far in the book we have used it exclusively to refer to a set of corporate directives that guide how IT is practiced in the company. Here we are using policy to mean one or more scripts that implement a set of procedures that respond to an event. The concept here is that someone has thought through what procedures need to be followed when some event occurs. Loosely speaking, a policy on how to respond was developed.

Linux on the mainframe has an ideal environment for managing availability because a number of automation tools exist. For example:

  • In z/VM, both CP and CMS have a broad range of commands to control one, all, or a subset of the guests.

  • REXX is a very powerful scripting language that runs in the CP and CMS environments.

  • A programmed operator interface (called PROP in z/VM) to each guest's console facilitates direct (not over a LAN interface) interaction by REXX with a guest.

A number of automation tools are also available from system management vendors. See Chapter 25, "Systems Management Tools," for some other options.


13.4 Change management

Those in the computer business who are responsible for availability often see system behavior as: "If the system ran yesterday and no one made a change to it, that system will probably keep running today." Sometimes this same understanding of change gets expressed by: "If the system fails today, the probable cause of the failure is the latest change made to the system."

While change poses a threat to availability, it is critical for continued existence in business. It is, in fact, the acceleration of change that is the hallmark of business today and one of the key drivers for deploying a Linux-on-the-mainframe environment.

Change management in most IT organizations is typically handled by some of the most skilled persons on the staff: the system administrators. We cover this topic in more detail in Chapter 16, "System Administrator Tasks."

With Linux on the mainframe, you typically apply changes to more than one image at a given time. The scripting capability of automation tools, and especially of z/VM, is one of the ways that provides significant control for the orderly introduction of complex changes into a large number of active systems. The use of automation is important for managing change in the Linux-on-the-mainframe environment.


13.5 Key factors to consider in availability

Reliability is an attribute of a component or system and is typically defined as a probability of that component or system being up and running when it should be. It does not include the time when something is planned to be down. Availability focuses on the end user's perspective, which tends to be an "all or nothing" attitude. The system is either up and usable, or it is considered not available.

Most IT organizations include only unplanned outages (system crash) as part of their availability computations. High-availability systems usually have such stringent availability requirements that there is no distinguishing between planned (system administrator time) and unplanned outages. Some end user groups are beginning to require systems that are continuously available, that is, no outage of service ever, for any reason.

Availability management always uses a dual strategy:

  • Avoiding an outage in the first place

  • Rapid recovery once an outage does occur

As part of any TCO analysis for a new IT project, you should consider the tangible (for example, lost productivity, lost business) and hidden (for example, loss of reputation, bad press) costs of an outage.[26] When these costs are compared with the cost of avoiding various potential outages and the probability of such an outage, you have the basis to build a plan for availability management. The technical team is then able to focus its efforts on both failure avoidance and time to recover in an order defined by business requirements.

[26] See Gartner Research Note, "24x7 is a Management Thing," at http://www.availability.com/resource/pdfs/in4.pdf.

13.5.1 Levels of availability

The costs associated with making a service more available can grow exponentially. That cost explosion leads IT directors to examine closely the availability requirements statement and the cost of achieving that goal. Table 13-1 lists various levels of availability.

Table 13-1. Levels of availability

Level of availability

Time of outage allowed

90%

2 shifts per week

95%

1 shift per week

99%

1 shift per month

99.5%

6 shifts per year

99.9%

1 shift per year

99.99%

1 shift per decade (less than one hour per year)

99.999%

1 shift per century

The built-in availability functions in each system component make a significant difference in how costly it is to provide a given level of availability. This aspect of the cost of improved availability is not often discussed. When deciding where to host the various parts of a new application, it is as important to understand what the system components offer for availability functions as it is to have a clear idea about what availability is required. If a system component falls short of what is required, you must expect additional costs in the implementation of your solution.

The system functions required for building an application with a given level of availability also depend heavily on the needs of the application. Most systems today provide at least a 90% availability right out of the box for any application they support. With all the lament about the reliability of our PC hardware and software, most of us do not experience 8 hours of downtime per month when working with a typical PC application, which translates to a respectable 99% availability.

Today's Linux comes with enough features to allow you to build a highly available static Web page server without undue effort. But building a 99.99% available e-business application with a 100-gigabyte relational database would be a significant challenge with Linux. Linux today lacks the built-in capabilities of z/OS parallel sysplex for shared read/write access to large databases. It will cost you to derive your own substitute for those functions that come already built-in with z/OS. Figure 13-2 illustrates how the cost of an availability solution rises steeply as you go beyond what the operating system provides.

Figure 13-2. The cost of an availability solution

Depending on what level of availability your end users require and on the application you intend to host, Linux might provide what you need.

While the current Linux distributions for zSeries are missing some of the more advanced functions for constructing and managing high-availability servers, numerous tool sets are available from various vendors. For pointers to some of these tools, see Chapter 25, "Systems Management Tools." A number of Open Source projects in this area are expected to culminate in significant functional improvements in future Linux kernel releases.

StoreCompany with its new Internet sales project specifically chose to leave the parts of the application with complex availability requirements on z/OS and keep on Linux only those parts suited to the Linux availability functions.

13.5.2 Outage avoidance

Outages are typically caused by:

  • Deficiencies in the system components

  • System complexity

  • Human error

  • Acts of God (such as storms or earthquakes)

The last two items are fairly independent of the system choices you make, especially if you take significant advantage of automation. We are going to ignore them here.[27] How does Linux on the mainframe help for the first two items? Let's look at some of the key components in a server farm:

[27] Most systems use automation to minimize the possibilities for operator errors. Linux on the mainframe lends itself to that type of automation.

  • The hardware: CPU, disks, cables, network

  • The operating system

  • The middleware

  • The application

The mainframe has a well-earned reputation for its availability. In Chapter 2, "Introducing the Mainframe," we explained the architectural and design basis that leads to exceeding IBM's 2002 design point for the IBM z900 of greater than 30 years mean time between failures. For those who have experienced life with a server farm of 200 machines, a farm built with z/VM guests on zSeries hardware will be a pleasant experience. The mainframe hardware can almost be ignored as a source of outage.[28]

[28] A component failure that brings down a mainframe machine unexpectedly is a low probability source of outage and for many planning purposes can be ignored. Other hardware-related sources of problems (for example, power grid problems) are still there.

The operating system is at the core of the software environment. Contrasting factors are involved here:

  • The more function the operating system provides, the less complex the middleware and applications tend to be. Hence, these higher layers are less prone to failure.

  • The larger and more complex the operating system, the more likely that there are cases in which the operating system still has hidden bugs.

By choosing Linux as the core of your available system, you pick up some very strong characteristics that complement the zSeries hardware. Linux is an interesting operating system in that it has robust function yet remains relatively small (as compared to z/OS or Windows XP) in size. It also has a huge user base that is using the system, maybe two orders of magnitude more installed instances than z/OS and rapidly approaching the latest install levels of Windows. The unique aspect of Linux that is important to its availability is the size of the community working at fixing problems found and the speed at which these changes get rolled into the base code.

If you find that the latest levels released by one of the main distributors (Turbolinux, Red Hat, and SuSE) meet your functional requirements, you will have a stable operating system. If you want to have the added security of an organization guaranteeing to fix your Linux problems, then you should investigate signing a service contract with either your distributor or IBM's Global Services.

If some new function that is still in the developmental stages (for example, code from one of the Linux high-availability projects), then you have a choice. In your case, Linux might not be ready for your needs. Or you will need to analyze and accept the various risks with being on the leading edge. Probably you will find that such functional needs can be isolated to only a few Linux images, and that the rest of the images fall under the stable and contractually supported release levels.

13.5.3 Rapid recovery

Table 13-2 shows typical recovery times for some failure types.

Table 13-2. Recovery times

Failing component

Approximate recovery time

Hardware

hours to repair; 5 or more minutes to restart

Linux (boot)

1 minute

Middleware startup (for example, WebSphere Application Server)

3 minutes

Database manager

5 minutes

Application

depending on the application, 1 minute or more

A typical Service Level Agreement (SLA) that specifies the maximum permitted outage time goes a long way to dictating the availability management strategy for a given server.

In large server farms of PCs (say, more than 200), it is not unusual to experience a hardware failure on the average of once a week. Typically, end users would not be satisfied with a strategy built on repairing or replacing the failed server, since that might take hours.

Based on the numbers in Table 13-2 it would be difficult to meet an SLA of recovery under five minutes with reboot as the recovery strategy. Starting all components adds up to more than five minutes, even if the recovery action is driven by an automation tool! The restart figures can differ for different setups, but there is always a limit to the degree of availability you can achieve with restart as the sole means of recovery.

Similarly, if the SLA calls for 24x7 service with 99.9% availability, the system administrator is going to find it more difficult to apply maintenance and changes than on a server that requires only 99% availability.

A critical decision that you must make in availability discussions is what failures you intend to recover from. For example, if your customers can accept, and do not hold you responsible for, power grid failures that last for more than one day, you can configure for a combination of battery and generator backup. In this case, you do not have to plan recovery actions for an area-wide flood and keep only a day's supply of diesel fuel at your site.

The more failure scenarios you want to cover and the more 9's you need, the harder and more costly it is going to be to accomplish a high level of availability, even with a superb suite of availability management tools.

ISPCompany, based on its established procedures and tested systems, is able to guarantee a 99.99% availability to clients doing static Web page serving. It is relatively easy to configure a number of Linux images where two images act as proxy servers (Figure 13-3).

Figure 13-3. ISPCompany high-availability setup for static Web serving

The proxies spray incoming requests to a number of Linux Apache servers that share, read-only, the files with the static Web pages. Any single image failure will be transparent to the end user.[29]

[29] Redbooks publication, Linux on IBM zSeries and S/390: High Availability for z/VM and Linux, REDP0220, defines a number of high-availability configurations.

z/VM is a great aid in this environment. First, it provides a highly optimized path, HiperSockets, for the traffic from the proxy servers to the Apache servers. Second, it also has, through the use of REXX scripts, the ability to add a new Apache server instance to take over the lost capacity from a failing Web server. With z/VM, there is no need to have excess capacity in this case because the failing server no longer consumes its portion of the zSeries cycles. They will just be picked up by the replacement image.

13.5.4 High availability

In general, providing availability greater than two 9's, or specifying outages taking less than five minutes, is going to require significant planning and will probably result in more levels of redundancy. The costs of providing such levels of service also typically grow exponentially with the complexity.

As already stated, it is fairly straightforward to provide a high-availability solution for static Web page serving. On the other hand, if you had a read/write database application, the middleware and Linux file systems are still relatively restrictive on sharing a multi-write environment. It is this added state information that complicates the system design and makes it much harder with today's levels of Linux and middleware to achieve a 99.99% availability.

A failure of a database manager that has some state information not yet completely written to the database is going to take more than a few minutes (and maybe even hours) to recover. Much of the design of this type of high availability in Linux is still left to you to analyze and to implement. This is not to say it is impossible; rather, such a solution is likely to be time-intensive for your staff.

Linux has many parameters and install options that can be used to allow for speedy recovery. For example, journaled file systems provide for speedier recovery than other types of file systems. The penalty is a small but noticeable performance degradation.

You normally start from a given availability that you require for a particular application. Figure 13-4 illustrates how the cost for attaining this availability varies with different operating systems (OSi, OSii, and OSiii). OSi represents an operating system with the weakest and OSiii with the strongest availability features. The curve shows how the cost explodes for operating systems that offer too little availability functions to support the application. For operating systems that fall in the middle section of the curve, solutions can be devised at a reasonable cost. If the operating system provides all you need, like OSiii in our example, the solution will cost you next to nothing. Note that the curve would change for a different level of availability, a different application, and different hardware. As operating systems evolve, they generally provide better availability features so that a comparison is also valid only for a particular point in time.

Figure 13-4. The cost of availability solutions depending on the operating system

An operating system like z/OS (an example of an OSiii type operating system) has many availability features built into it. z/OS can also work with a Coupling Facility that makes it significantly easier to build even 99.999% available systems with a geographically dispersed cluster of images. At the time of this writing, if you had a need for such high levels of availability for systems with shared write to data, you would probably be better off basing the design on z/OS than on Linux on the mainframe. Many organizations are actively pursuing high-availability solutions for Linux, and most of these would make good sense if deployed on z/VM and zSeries hardware.


13.6 Summary

What you want for availability from your Linux images is up to you to define in conjunction with your end users' requirements. System availability is not free. It will take effort, and that effort starts with a clear definition of the availability requirements, for example, the various service level agreements with the end users.

Linux as distributed by one of the large distributors with zSeries support is a highly stable operating system benefiting from being both relatively small and having an enthusiastic and skilled group of people dedicated to making it better.

z/VM is a fantastic base on which to run a farm of Linux images. In the first place, you automatically inherit the reliability of the zSeries hardware. z/VM contributes availability tools (such as REXX scripting capabilities) to ensure real time response to events showing up in one or more Linux images. Since z/VM is aware of all the Linux images, it is much easier to build recovery scripts involving related Linux images using z/VM and REXX than it is on most other platforms.

A number of availability management tools from software vendors and the Open Source community also can help you build the level of available Linux servers you need.

At the time of this writing, Linux on the mainframe is not a panacea for all availability problems. However, it is continuously improving and there are many areas where Linux and z/VM are a great pair for delivering the desired levels of availability.


Chapter 14. Data Management

Data are crucial to a business and the mainframe enjoys the reputation as the best keeper of corporate data. In this chapter, we will turn to examples of ISPCompany and StoreCompany to show how Linux on the mainframe can use that mainframe potential and what Linux provides for managing data. But before going into details, let's summarize the scope of data management.

Ultimately, data management ensures that data are available to legitimate users and applications when they need them, while protecting them from misuse or damage through unauthorized access. Data management governs the entire life cycle, from creating new data to archiving or destroying obsolete data.

Data management also covers the physical aspects of storage, including devices, cabling, and floor space. Data for a mainframe Linux image can reside on a variety of devices, including traditional mainframe devices. The mainframe also provides flexibility in the way devices can be attached, the possibility for access path redundancy, and capabilities for dynamic reconfiguration.

In this chapter, we will first discuss some important technical concepts about storing data on the mainframe. Building on this discussion, we will then explore what Linux on the mainframe has to offer to data management. Along with the key data management tasks of Figure 14-1, we will address these questions:

  • How can I ensure that data are available only to legitimate users?

  • Can I use SCSI devices?

  • Do I have to acquire new devices, or can I use devices I already have?

  • How can I protect my data from loss or damage from user, application, or hardware errors?

  • How do I attain satisfactory performance when disks and channel paths are shared?

Figure 14-1. Data management discipline

    [ Team LiB ]


14.1 Keeping data on the mainframe

In this section, we introduce the devices that can be used by Linux on the mainframe. Then, we discuss some technical aspects of keeping data on the mainframe. This technical information can help you decide what to bring to Linux on the mainframe. The sections on the data management tasks that follow build on these technical concepts.

14.1.1 Which devices are there for you to use?

Because Linux on the mainframe supports both traditional mainframe devices and Open Standard storage devices, you have much freedom to choose where to place your data.

Traditional mainframe devices

The basic functions of mainframe devices tend to be covered by generic device drivers. However, because manufacturers generally treat their device interfaces as proprietary, special device features and full error recovery are usually not supported by generic drivers. To close this gap, some manufacturers provide device specific drivers as Object Code Only (OCO) modules.

With the exception of some older devices, mainframe shops can continue to use the devices they already have with Linux.

Disk devices: Table 14-1 shows some of the disk devices (DASD, in mainframe language) you can use with Linux on the mainframe (the '**' signifies any specification):

Table 14-1. Supported IBM disk devices

Control unit type/model

Device type/model

Restrictions

3990(2105)/**

3380/**

3990(2105)/**

3390/**

9343/**

9345/**

Basic error recovery only

6310/**

9336/**

VM virtual disk in storage only

3880/**

3370/**

Basic error recovery only

Also, Linux on the mainframe supports more recent devices such as the IBM TotalStorage Enterprise Storage Server (ESS).[30]

[30] IBM TotalStorage Enterprise Storage Server is also known as "Shark."

Because operating systems and their file systems tend to have unique data formats, Linux has its own special formatting of zSeries disks. As of the writing of this book, there is no common data format for Linux and mainframe operating systems such as z/OS or z/VM that allows both operating systems to read or write common files.

There is a disk format (Compatible Disk Layout) that allows Linux file systems to be managed by zSeries storage management products, for example, DFSMS (Data Facility Storage Management Subsystem) for backup and restore, or GDPS (Geographically Dispersed Parallel Sysplex) for disaster recovery. However, this disk format does not allow Linux data to be processed from operating systems other than Linux.

Tape drives: You can use tape drives that are compatible with IBM 3480, 3490, and 3590 magnetic tape subsystems.

FCP attached devices

Beginning in late 2002, Open Systems devices can be attached to a zSeries machine through a Fibre Channel Protocol (FCP) link. The generic SCSI device drivers provide basic support for most devices such as tape drives and libraries, or fiber-attached SCSI disks. However, the generic drivers are subject to restrictions with regard to functionality and reliability. Such restrictions can be overcome by using vendor supplied device drivers (for example, "ibmtape" provided by IBM for tape devices).

The immense test effort involved in confirming the scores of SCSI-compliant devices that this FCP attachment opens up to Linux on the mainframe is the limiting factor for "official" support. Many devices are known to run, but have not been tested exhaustively. On the other hand, there is also a small number of devices with limited SCSI compliance that are not suitable for Linux on the mainframe.

At the time of writing, we are anticipating an increasing number of "official" support statements by IBM and other vendors to be issued as qualification tests are completed for specific devices.

SAN

You can also attach a Storage Area Network (SAN) through an FCP link. However, operating system images, even on the mainframe, inherit the open security and management issues that SANs suffer on other platforms.

SANs are considered enterprise-ready in environments where a single operating system controls the entire SAN. As of the writing of this book, SANs are still subject to data isolation problems in heterogeneous environments and for usage across security domains. Intense work is under way to address these problems. In anticipation of a breakthrough on these management issues, Linux on the mainframe already supports SANs.

14.1.2 Data sharing

Here, we are not using sharing to mean a particular technical implementation, but rather to describe the logical instances in which multiple users need access to a common set of data.

With Linux on the mainframe, it is likely that you are not dealing with a single Linux image and its data, but with data that are needed by multiple Linux images and maybe even a z/OS image. This sharing can be through NFS, by getting a point-in-time copy of the file of interest, or by making a program call to another application for the required data. It is not only possible but sometimes also desirable to keep data on z/OS. The mainframe provides the communication methods and there are numerous connectors (see 19.3, "Connectors to back-end systems") that facilitate the use of data in z/OS resources from Linux on the mainframe.

Sharing data raises questions about who owns which data and who is responsible for managing them. Data could be owned and controlled by an individual user, but also by a Linux image or by an application.

Where data are created by an application, it is usually the application that owns them. Especially where interrelated data are created and used in multiple locations, the application is best suited to determine what constitutes a consistent set of data (for example, for backup purposes). If the Linux image shares the use of data with a z/OS image, the data are probably directly attached to z/OS by a channel path so that you can exploit z/OS data management capabilities.

For data management, it is important that all parties are aware of who owns the data, who uses the data, and who is responsible for which aspects of their management.

Data sharing reduces the amount of required storage, both for the original data and for backup copies. It can, however, introduce latency for getting data from the owner, which might lead to unacceptable response times.

Data sharing need not be the best solution for providing the same data to multiple applications or users. Where sharing is advantageous, Linux on the mainframe can exploit the mainframe technology with its fast internal communication methods for accessing the data of other operating systems on the same machine. The next section is about the opposite of sharing: privacy.

14.1.3 Data isolation in a virtualized environment

Multiple operating systems have been running on the mainframe long before the emergence of Linux and Linux on the mainframe. Consequently, the question of isolating data and protecting each operating system's data from other operating systems is not a new concern. It has long been addressed and robust isolation mechanisms are at hand.

Channel paths

To understand data isolation and access control on the mainframe, we need to know a few things about channel paths. Channel paths are the physical access paths with their associated logic and controls that the mainframe machines use to access data (see Figure 2-13). The zSeries architecture supports 256 channel paths.

The number of devices you can attach to a channel path depends on the path type. In total, a maximum of 64 K devices can be attached to a single hardware machine. Even in the old days of 3 gigabyte-sized disks, you could directly attach close to 200 terabytes of data on disk. In Open System environments, comparable capacities can be provided by using a smaller number of large disks, rather than a large number of small disks, as typical for the mainframe.

You can share channel paths among operating systems. Path sharing is a necessity if you intend to run hundreds of Linux images. Path sharing is also a means to save on cabling and adapters (channel cards). Conversely, it is common practice to provide more than one channel path for accessing a particular device (multipathing). This redundancy eliminates cabling as a single point of failure and can boost performance.

Controlling access to devices

With a zSeries machine running numerous Linux images, you might ask how one image's data could possibly be kept safe from the other images. There are several lines of defense:

  • The hardware definitions

  • z/VM

  • The Linux images

The combination of these control mechanisms provides flexibility for addressing the individual data isolation and sharing needs of different installations.

The hardware definitions: The mainframe accesses devices by means of channel paths, that is, the work queue, the involved system assist processor, and the cabling. A channel path constitutes the hardware logic and processing required to access a device through a particular communication line (for example, a parallel link or a serial ESCON or FICON link). The hardware looks only for channel paths and devices that are defined to it in a special file, the IOCDS (I/O configuration data set). Devices must be defined in the IOCDS to be detected by the zSeries hardware. Undefined devices are inaccessible to the hardware and all software that runs on it.

To isolate data, you can dedicate a device, that is, define it in the IOCDS only to a particular LPAR. As a result, programs or operating systems that run in other LPARs cannot access the isolated data.

You also can dedicate a channel path with all of its devices to a particular LPAR. Channel path dedication is often used for allocating bandwidth to one LPAR in favor of others. For data isolation, all possible paths to a device would have to be dedicated.

z/VM: z/VM can detect the defined devices on all channel paths that are defined to its LPAR (or the machine, if z/VM runs natively). A guest can use only the devices that z/VM defines to it. These definitions can be changed dynamically from z/VM. z/VM also can split the real devices into smaller virtual devices (minidisks) and control the guest's access at the granularity of the virtual devices.

The Linux images: With root access to Linux, you have two more ways to control data access: at the device level and at the user level. Both can be suitable control mechanisms if only trusted personnel or applications have root access.

You can instruct a Linux image to look only for specific devices. As a result, Linux will not be able to detect any other devices, even if the hardware definitions and z/VM have made them available.

In addition, Linux can restrict the access rights of individual users to the data in Linux file systems.

Example: Using z/VM for access control

One widely-used method is to over-define the relatively static hardware definitions and the Linux definitions and then use the dynamic z/VM definitions as the effective control. Over-defining means defining devices that do currently not exist but may be added in the future. A newly added device that has already been defined to the hardware can be brought online without taking the machine down.

Figure 14-2 illustrates a z/VM that knows nine devices (A000 through A008) and has three Linux guests (Lx, Ly, and Lz). Figure 14-2 also shows the statements that z/VM uses to selectively make devices available to the guests.

Figure 14-2. z/VM control of storage devices

z/VM can give a guest exclusive access (ATTach) or shared access (LINK) to a device. Figure 14-3 shows the logical views of the devices that the guest systems have according to the statements of Figure 14-2.

Figure 14-3. Linux guests' view of devices

z/VM assures that each guest can detect only the devices that are defined to it.


14.2 Introduction to backup and restore

Companies establish backup and restore procedures against loss of data if:

  • Users unintentionally delete or damage data.[31]

    [31] Loss of data due to erroneously hitting a key is also known as a "finger check." In an early version of the S/360 Time Sharing Option (TSO), the command for editing a file was the single character E, while the command for deleting a file was a D. A glance at your keyboard will tell you why this short command for deletion was soon abandoned for the three-character command DEL.

  • An operating system or application crashes.

  • Hardware fails (for example, a disk head crash).

Linux on the mainframe allows for the usual schemes, including tape libraries and dual copy facilities, that are used for backup, restore, and disaster recovery. z/VM's data management tools can help you to physically isolate critical data on particular disks that can then be handled by z/VM disk copy tools.

The standard backup method is to copy data to tape at regular intervals. Tape allows you to recover an earlier version of data that have been lost by accidental deletion or damaged by a failing application. However, with this method, you lose all changes that have occurred since the time the last backup was made.

Maintaining two synchronized copies of current data (for example, by dual copy) is a backup method that always assures a current copy of your data. This protects you against a head crash, but not against accidental deletion or damage through a malfunctioning application.

Because backup activities can be disruptive and consume processing power, communication, and media resources, backup rarely covers all of a company's data. Backup is often limited to important data, which cannot readily be recreated. Generally, the more critical and dynamic the data, the more frequent the backups.

For a selection of tools to handle your backup and restore procedures, your choices range from developing your own solution, to leveraging the Open Source community or buying a solution. Amanda is one of the better-known examples of Open Source tools.

tar and ftp from Linux can supplement the formal backup and restore process for smaller amounts of data that are controlled by single or small groups of individuals.

14.2.1 Tape versus disks

In the past, backup meant tape. With disk space becoming cheaper, backups on disk are becoming a viable alternative to tape for some types of data.

Backups on disk can be restored with less latency than backups on tape. In addition, the mainframe allows you to connect an enormous number of storage devices, so that disk as a backup medium becomes an option.

Tape prevails where backups must be taken off-site and where floor space is limited.

14.2.2 Layout of backup infrastructure

Most backups are still written to tape. Because highly reliable tape devices can be costly, the typical server farm approach is to have a LAN-attached backup server with an attached tape device or library (Figure 14-4).

Figure 14-4. Backup server in a distributed server farm environment

If you have consolidated a server farm using Linux on the mainframe, you can continue to use the backup server on the LAN with its tape library (Figure 14-5).

Figure 14-5. Consolidated server farm with the backup server on the LAN

However, a LAN-attached backup server has disadvantages, including:

  • Backup activities can overload a LAN and sometimes necessitate a separate LAN.[32]

    [32] With Gigabit Ethernet, you are less likely to experience LAN overloads from the backup data.

  • Moving backup data across the LAN is a potential security exposure.

With a consolidated server farm on a zSeries machine, you can also have directly attached tape devices (Figure 14-6). You can move data to the backup server through HiperSockets, which are secure, provide high bandwidth, and, being virtual, require no cabling.

Figure 14-6. Backup server on the mainframe

You can attach multiple tape devices to the same zSeries machine and perform multiple backup operations concurrently.

14.2.3 Restore

An important decision point for selecting a backup/restore tool is whether it satisfies your security requirements (for example, to control who can retrieve backed-up data). Before buying a tool, be sure that it can be integrated with your security checks.

Restoring a single file usually takes longer than backing it up, especially when the restore medium is a tape. The tape must be mounted, and the required data must be located by winding the tape to the correct position. The greater the tape capacity, the longer is the expected response time.

You can have faster response times by keeping the backed-up files on disk. By keeping the disk available to a Linux guest image or by making it accessible through z/VM, you can provide rapid access to a backed-up file system.

14.2.4 Disaster recovery

Disaster recovery is a business survival scheme that is used in the event that an installation is partially or totally destroyed. Disaster recovery includes contingency plans for a range of possible disaster scenarios. These plans involve physical aspects, such as the location from where to operate and the machines and devices to be used.

z/VM tools and hardware facilities such as FlashCopy for Enterprise Storage Server allow rapid copying at the device level. If you ensure that all critical applications keep their data on a specific set of physical devices, you can create your data copies for disaster recovery at the physical level and take advantage of these facilities.

For most disaster scenarios you must keep data copies off-site, usually on tape. If the initial copy is on disk at the primary site, a slower but non-disruptive secondary copy to tape should follow the initial fast copy. If you are using a remote dual copy, you might not need to go to tape. ESS, for example, provides the remote dual copy features Peer-to-Peer Remote Copy (PPRC) and eXtended Remote Copy (XRC).

This section and the disaster recovery examples of our hypothetical companies are not a full discussion of disaster recovery. Our view is limited to aspects of data management in which special considerations apply to Linux or to the mainframe.

With z/OS Parallel Sysplex,[33] the mainframe offers an unsurpassed availability scheme for z/OS images. You cannot build sysplexes with Linux images; but if you already run a Parallel Sysplex, you might be able to exploit the sysplex capabilities to protect critical data.

[33] A Parallel Sysplex is a mainframe clustering technology that can also involve geographically distant systems. One or more Coupling Facilities assure integrity as data are shared among constituent systems. If a shared storage device fails, the data can be reconstructed from the Coupling Facilities' log streams in conjunction with the application and the middleware that owns the data.

At StoreCompany, for example, Linux works with data that are copied from z/OS. The z/OS copy is the master copy (see 14.4.2, "StoreCompany data management") and is included in the disaster recovery scheme. No additional precaution is required to safeguard the survival of data on the Linux side of its operations.

14.2.5 Backup tools summary

Table 14-2 summarizes some of the tool options you have when taking backups:

Table 14-2. Entities to be backed up

Backup level

Tool

Disk

DASD dump restore

Minidisks

DASD dump restore

CMS files

CMS copy command

Linux files

Tivoli Storage Manager

Incremental database backups

Database utility

Cross-resource increments

Application that owns the resources

z/VM can take care of entities that it knows, for example, the physical disks. Where the entities are of a more logical nature, such as a Linux file system, you need tools that operate at the same abstraction level as the entities to be backed up. See also 25.2, "Data management tools."


14.3 Quota

In this section, we use quota as any way of constraining the use of disk storage by operating system images, applications, or users. Constraints can be applied for the available disk capacity at different levels: the machine level, the z/VM level, or within a Linux image.

Potentially, there is a large number of individual devices attached to a mainframe machine. The hardware definitions limit what is available to each LPAR, for example, an LPAR with a z/VM.

z/VM can allocate logical portions of its available storage to individual guests. In z/VM, such logical portions of storage are referred to as z/VM minidisks. Access to a minidisk can be restricted to a single image, or the minidisk can be shared by multiple images. Usually only one image has write access to a particular minidisk. Within the constraints that result from the physical devices, you can also allocate a minidisk of a moderate size and increase it on demand. Thus, you can divide the available real disk space into well-known portions, and you can keep some portions inaccessible as a reserve that no one can use without your active intervention.

If you place all the spare disk space into a single pool that the storage administrator can use for any of the systems, you can realize some significant disk storage savings. In a server farm where each system has its own discrete devices, each system must also have its own excess capacity that allows for unexpected storage needs. Because it is highly unlikely that unexpected needs arise on all systems in a server farm simultaneously, much less overall excess capacity is required if spare storage can be made available wherever it is needed. Attaching an extra physical disk to a small machine means a reboot. Adding a z/VM minidisk to a guest system does not disrupt operations.

Users who perceive storage as an inexhaustible commodity for which they are not accountable are unlikely to use it economically. In a multi-user environment where storage is provided centrally, containing the erratic behavior of individuals is an important aspect of storage administration. User quotas are one way of returning a sense of responsibility for the consumed storage resources to the individual user. Linux images can be installed with the capability to have any Linux system administrator assign quotas to users and groups in that image.

Linux quota and disk allocation schemes must give special attention to the various Linux system logs and temporary files. These two file categories have a tendency to grow relentlessly over time and a housekeeping strategy is essential for them. Often the cron daemon is used for an automated approach where tasks scheduled at intervals redirect the log streams to new files, archive current logs, and discard obsolete files.

A log file that has filled all available space can cause a Linux image to crash. Therefore, managing log files is typically part of availability management schemes. Availability schemes usually have tools in place that intercept warning messages indicating that the available space has fallen below a critical mark and either automatically assign new storage, or inform the storage administrator, or both.


14.4 Data, policies, and tools

Enterprises know how vital their business data are to them. Accordingly, most companies use policies (and an SLA) to control how its data are handled.

Because there is no universally valid answer to the question of which policy and SLA are suited to the data that are required and generated by an application on Linux, let us turn to our hypothetical companies to explore some interesting possibilities for Linux on the mainframe.

14.4.1 ISPCompany data management

ISPCompany, as an ISP and outsourcing partner, is the keeper of the corporate and personal data of a large number of clients. ISPCompany faces the challenge of isolating different clients' data and of catering to a wide spectrum of needs for backup, disaster recovery, and space management.

As the guiding principles for its data management, ISPCompany follows these policies:

ISPCompany data management policies

  • The client's privacy is paramount. This applies to all clients.

  • Mail on ISPCompany mail servers is managed uniformly for all ISP mail customers.

  • Clients of ISPCompany's outsourcing services have a choice of multiple service levels for each of the following: backup and restore, disaster recovery, and storage.

ISPCompany maintains two geographically distant sites, each with a zSeries machine. Our discussion focuses on the primary site, where ISPCompany manages a total of 14 terabytes of data. The primary site is illustrated in Figure 14-7. By far, most of the data is client data (Web content, mail, applications, and various business data). A much smaller amount is ISPCompany's own data. This includes its production capital (master copies of its z/VM and Linux operating systems, configuration files, the middleware and applications it offers to clients, and the Linux images it uses to manage itself). The rest is ISPCompany's business data (accounts receivable, payable, in-house developed code, the log files it uses for accounting, and other minor proprietary items).

Figure 14-7. ISPCompany storage devices and connections at its primary site

ISPCompany devices

As illustrated in Figure 14-7, ISPCompany uses two IBM TotalStorage Enterprise Storage Server (ESS),[34] ESS1 and ESS2, to store 14 terabytes of data at its primary site. Approximately 8 terabytes reside on ESS1 with 6 terabytes on ESS2. Each ESS has about 500 gigabytes of empty disk space. There is redundancy for all critical parts (such as control units) to eliminate single points of failure.

[34] At the time of writing of this book, a single ESS can be fitted with a maximum capacity of 55.9 terabytes.

To cope with the massive amount of data for archiving and backup and restore purposes, three IBM TotalStorage LTO Ultrium tape libraries are available.

Each ESS is divided into two sections: one that acts as a SAN and another with direct-attached virtual 3390 disk devices. The SAN on ESS1 is owned by the mail servers. Only the mail servers and a trusted ISPCompany Linux image that handles backups can access this SAN.

The SAN on ESS2 provides storage for a client of ISPCompany's outsourcing section. The client has made special arrangements with ISPCompany to be able to experiment with a SAN. No one but this client's images have access to the SAN. All further security aspects are left to the discretion of the client.

Some clients of the outsourcing section have their own devices that are accommodated in their respective client cages. Clients can also include in their contract the ability to attach to devices in the ESS2.

ISPCompany data security

ISPCompany has to meet the challenge of protecting its many clients from each other, while assuring that all clients have access to their own data. To isolate data for medium and large client accounts, ISPCompany uses dedicated disks that can be accessed only from the owners' Linux images.

ISPCompany uses z/VM as the place from which to do the physical data security management. No client has access to z/VM. This means that accounts which control their own Linux images cannot boot them directly. Instead, such clients send a message to a z/VM tool. The tool checks if the request originates from an authorized source. If so, the tool performs the requested IPL and logs it for auditing purposes.

The thousands of e-mail accounts do not get pre-allocated dedicated disk space. These customers access their data through the mail server instead of directly. ISPCompany uses the mail server's security to protect the data.

In keeping with its privacy policy for client data, ISPCompany logs each occasion in which an ISPCompany employee accesses client data. Accessing client data is permissible only when necessary for debugging and is mostly restricted to viewing diagnostic data. Breach of these rules is seen as serious enough by ISPCompany to warrant dismissal.

ISPCompany data sharing

The large accounts in the outsourcing business have their data on dedicated disks that can be accessed only by a client's own images. ISPCompany's tools see only disks and partitions, not file systems and files, unless the tools actually log in to the client's image.

Medium-sized accounts in the outsourcing business use standard Linux images and share (read-only) the respective disk with Linux code libraries. The same rules apply to some of the standard applications. The unique data of these accounts are kept on dedicated disks.

ISPCompany backup and restore

The clients of ISPCompany's outsourcing services have an enormous amount of data to back up. To minimize disruptions to these clients' businesses and stay within acceptable backup windows, ISPCompany uses a carefully devised scheme:

  • Full backups are taken only at agreed times when minimal impact is expected for a particular client.

  • Incremental backups are taken at more frequent intervals. Linux provides this capability for the changed files in Linux file systems, for example, through GNU-tar.

  • Peer-to-Peer Remote Copy reduces the backup window for the most critical data to almost nothing and protects against outages due to device failure. Peer-to-Peer Remote Copy is an IBM TotalStorage Enterprise Storage Server advanced feature.

  • Fast backups to disk are used as a first backup level. zSeries hardware with its enormous number of directly attachable devices makes this a viable option. These backups to disks provide an interim backup until the subsequent slower backup to tape.

ISPCompany divides the data into three regions, each attended to by an IBM Tivoli Storage Manager on a Linux image. Agents on the clients' Linux images determine which files to back up. Each manager schedules and coordinates the different backup operations across the ISPCompany clients within its region. To cope with the workload—backups for the three regions—ISPCompany usually performs backups simultaneously.

The mail server's data are backed up only to protect the mail that has not been replicated to the users' mail clients. The responsibility for backing up mail that has been replicated to a user client is transferred to the user. Backups are taken every day and kept for a month.

The medium and large business accounts can directly request a restore from the responsible Tivoli Storage Manager. The manager performs an authentication and an authority check to determine if a request is legitimate before it routes the request to the tape library.

ISPCompany data handling for disaster recovery

Every month, ISPCompany uses a z/VM tool, DASD Dump/Restore (DDR), to dump to tape all of the data that it has earmarked for disaster recovery. The tape device is a high-speed IBM LTO Ultrium. DDR is invoked from a REXX script on z/VM, where the script defines which disks are to be copied. Incremental backups of the disaster recovery data are taken daily. Because these increments do not map to complete devices, DDR is not used.

After the data are written to tape, the tapes are removed and sent to the second ISPCompany site for safekeeping. The secondary site acts as a recovery site. Disaster recovery tapes are exchanged between the sites so that each site keeps the recovery tapes of the other site.

Each z900 has an idle LPAR that is ready to be IPLed with a z/VM system laid out on disks. At the expense of a mere three 3390 devices' worth of storage for the z/VM system data and with no extra hardware, a z/VM for the most essential ISPCompany and client Linux images has been deployed, ready for IPL, at the recovery site.

Resumption of full services depends on the scale of the disaster. Resumption also involves special agreements for fast delivery of replacement machines and devices to a contracted site.

ISPCompany database management

ISPCompany uses DB2 databases, both for its own purposes and on behalf of its clients. ISPCompany performs backup for all databases that it controls. It uses Tivoli Storage Manager to drive the backups. Clients who have opted to manage their databases themselves can schedule these activities from their Linux images (for example, using the cron daemon).

ISPCompany space quotas

ISPCompany can control disk space usage at the image level and at the Linux user level.

At the image level, ISPCompany uses z/VM disk storage virtualization as a means of providing storage to clients who use the company's outsourcing facilities. A Linux image under z/VM is given access to the amount of storage agreed to in the respective SLA. Space can be allocated as full disks or as z/VM minidisks.

To limit the storage of individual Linux users, the Linux kernel can be installed with a quota function. Most of the Linux images which ISPCompany provides to its clients and the Linux images which ISPCompany maintains for its ISP clients use the Linux quota facility.

14.4.2 StoreCompany data management

StoreCompany has been running its department store chain with the support of a z/OS Parallel Sysplex at each of its major sites. At the site we are going to examine, StoreCompany had been using hundreds of 3390 disks for storing its data. StoreCompany has also recently purchased an IBM TotalStorage Enterprise Storage Server (ESS) with a capacity of 1.25 terabytes.

The challenge for StoreCompany is to experiment with its new Linux project without endangering the integrity of its corporate data. StoreCompany responds to this challenge (and largely sidesteps the overhead for managing data for its new Linux project) by adopting a policy that uses existing procedures and tools.

StoreCompany data management policy

A Linux image owns no critical business data. All business data remain under the ownership of a z/OS image, and no alterations are made to managing these data.

StoreCompany devices

StoreCompany has approximately 900 gigabytes of data, most of which is owned by its z/OS sysplex. There is enough spare capacity for operations, plus some room for future expansion. Besides the ESS, there are the older 3390 disks and a tape library for backup and restore (Figure 14-8).

Figure 14-8. StoreCompany storage devices

StoreCompany keeps the z/OS data on the ESS. To conveniently provide a separate space for the new Linux project, StoreCompany places the system data for these Linux images and the local data for the new Linux application on some of the spare 3390s.

StoreCompany data integrity

Through its policy that the new Linux application must not be responsible for business data, StoreCompany eliminates any new risk of losing data. The application on Linux does not generate new data. The application can feed data back to the online sales application on z/OS only through the existing order entry process and, hence, is protected by the existing order entry's management of these data. If the new application fails, no data are lost. In the worst-case scenario, an order does not make it to the sales application and would have to be re-entered by the end user.

StoreCompany data sharing

Store inventory and client accounts data are located in the z/OS DB2 and are shared across the sysplex. Linux accesses (shares) these data only through interaction with applications that run on the sysplex, using connectors to CICS and DB2.

StoreCompany backup and restore

Because the new Linux images own no business data, the impact on the existing StoreCompany backup and restore processes is minimal.

The 3390 disks with Linux system data and the new application are backed up at the file level as part of the scheduled weekly Linux backup. As the need arises, important intermittent changes to configuration files or the application are handled by the Linux development team and are backed up with the help of the tar and ftp commands to a remote system.

The new application requests a data extract from the corporate data warehouse once a week. The query is performed by the database manager on z/OS. The extract is written to a separate tablespace on z/OS and is then sent to the Linux image by ftp. If the data get lost or damaged on the Linux side, it can easily be restored from the original on z/OS.

StoreCompany disaster recovery

StoreCompany covers its disaster recovery needs by maintaining a similar infrastructure at both its two major sites, including similar sysplexes and storage devices.

The Linux project is not part of the immediate disaster recovery plan, but could be recreated from the backups that are compressed with tar and then transmitted to a remote system with ftp.


14.5 Database management

In the context of an application, backups must present consistent records across all of the data repositories that are used by the application. For example, a transaction must either be reflected in all of the affected data repositories or in none. This can be ensured by driving the backup from the application and relying on the intelligence of the application, or by quiescing the application and all associated data repositories during the backup.

General-purpose backup tools usually provide backup capability at the volume level or at the file level. For example, FlashCopy for Enterprise Storage Server, with a subsequent copy to tape, can back up a volume from z/VM, or a Tivoli Storage Manager agent can back up single files in a file system. Both the storage manager and the agent can reside on a Linux image. To the agent, a database looks just like any file. However, taking a full backup of a database is time-consuming and, if taken at close intervals, can also be wasteful of storage space.

Industry standard databases come with tools or APIs, or both, to facilitate data backup and recovery. These tools and APIs can be integrated with standard backup/recovery storage management products to support a rich set of backup/recovery storage services for relational database data. For example, DB2 has an API for incremental backups at the tablespace level. DB2 also includes a backup utility for non-disruptive DB2 backup that is integrated as a Tivoli Storage Manager client. Hence, you can run both full backups and incremental non-disruptive backups from Tivoli Storage Manager on a Linux image's databases.

A UNIX application that is to be consolidated on Linux on the mainframe might include databases on UNIX. For an application that has been set up properly, it can be assumed that a management solution for full and possibly incremental backups of the database is in place. When porting the application to Linux, it can be useful to port the entire application, including the database, in one stride. However, depending on your priorities and goals, different scenarios are conceivable. For example, you can migrate the application only and use a connector (for example, IBM DB2 Connect) to connect to an existing database on UNIX.


14.6 Performance tuning and capacity planning

Performance tuning aims to alleviate imbalances in system resources where one resource is underutilized because of constraints introduced by other resources. This section discusses I/O as one of the resources to be balanced. Although the diagnostics that reveal I/O shortcomings belong to the performance management discipline, I/O is discussed here because any corrective measures are usually the responsibility of the data administrator. For a more general discussion, see Chapter 15, "Performance and Capacity Planning."

In the mainframe tradition, the CPU is regarded as the most important part of the system, which must be kept busy whenever work is to be done. The mainframe's I/O subsystem has, therefore, evolved to live up to this demand. On the mainframe, providing paths is inexpensive compared to adding memory or processor capacity, and a skilled mainframe system architect will tend to deliberately overconfigure I/O.

It is also common practice to provide redundant data paths to ensure availability in case a path fails. A welcome side effect of multipathing is an improvement in the average response time for I/O. At the customary path utilization,[35] the probability of finding multiple paths occupied simultaneously is low. z/VM manages the available paths to provide more effective bandwidth to the Linux images. Multipathing can also compensate for a possible performance degradation that might result when multiple images share data.

[35] 20% utilization is a common rate.

For newly established systems, you should not need to worry about I/O bottlenecks that are rooted in your system layout. For example, if you consolidate a server farm on a zSeries machine, you will have an idea of the required bandwidth for different components, and multipathing gives you the means of removing bottlenecks you might have had in the server farm.

However, I/O bottlenecks can develop if the load profile of a system changes over time, for example, as you add new Linux images. You will have to discern trend lines on the utilization of your I/O resources. Both z/VM and Linux provide the means to gather the required data. Which operating system is more useful depends on the scope of what you want to assess.

z/VM is better suited for getting summary data on all of the Linux images that run under it, or for summary data for an entire image. From Linux, you can gather data on the absolute I/O usage of individual Linux users in the image, which might be required for accounting purposes. Where a breakdown for individual users is not required but, for example, summary data on an application that runs on the Linux image is of interest, z/VM provides the simpler means of getting the data.

Linux allows you to gather the performance data that you expect from UNIX. In particular, you can use the iostat facility (see Chapter 25, "Systems Management Tools") to gather I/O data for the scope of a particular Linux image. However, while a Linux image can give you useful data for the resources that are dedicated to it, its limited view of shared virtualized resources can be misleading. For example, Linux might indicate a low utilization for a channel path that is actually heavily used by other images.

z/VM is designed to pool and allocate resources. To do so efficiently, z/VM gathers performance data at the virtual and real hardware levels for devices and channel paths. Without incurring a great performance penalty, you can write these data to a log stream for analysis. Getting a comprehensive report on all of the systems in a distributed server farm would require some special software and probably even hardware. A typical Linux-on-the-mainframe environment already includes z/VM with its diagnostic capabilities.

Moreover, z/VM provides the means to take corrective action. Not only can you identify potential bottlenecks, you also can re-allocate resources to alleviate them. For example, you can dedicate a path to a particular image that suffers I/O delays. You can do this from a z/VM terminal without having to go to the machine room to physically make changes to the cabling and even without having to restart any Linux images.


14.7 Summary

Linux on the mainframe can use all supported mainframe devices, such as IBM TotalStorage Enterprise Storage Server (ESS), or the more traditional 3390. Also, support for fiber SCSI over an FCP link will allow Linux images to reach SCSI devices that are common in the UNIX world.

At the end of 2002, it became possible to use SANs in the context of Linux on the mainframe. However, until current security and management issues are resolved, the use of SANs should be restricted to constrained environments.

The mainframe with its channel paths allows an enormous number of directly attached devices. Because your Linux images run in the same physical machine, you can share physical access paths. The large number of configurable paths also provides for multiple channel paths leading to the same device. Multipathing can increase availability and bandwidth.

Although physical connections might be in place, a Linux image cannot access a device unless the mainframe's hardware definition and z/VM definitions permit it. z/VM definitions can be changed dynamically without the need to restart any images or to change any cabling on the machine floor.


Chapter 15. Performance and Capacity Planning

The decision to run a particular set of work on the mainframe entails many factors, one of which is the performance of the environment. Part of the value you can get from the Linux-on-the-mainframe environment is the efficiency of the mainframe to support the diverse workload of a server hardware consolidation environment. Since we expect the majority of the Linux-on-the-mainframe deployments to be on z/VM, this chapter focuses on performance and capacity planning in this environment.

Sometimes, performance and capacity planning on a physical server farm means watching for response time problems. When response times reach the limit of acceptability, one solution might be to split the workload and buy a new machine, complete with more CPU, memory, I/O, and cables.

Linux on the mainframe allows you to rethink your methods for performance and capacity planning. The tasks included in the performance and capacity planning discipline are listed in Figure 15-1. Tools are available on the mainframe for these tasks, both with z/VM and the Linux mainframe distributions.

Figure 15-1. What performance covers

For the simpler Linux-on-the-mainframe start-up projects such as domain name server (DNS) or static Web page serving, the focus is primarily on getting familiar with the Linux-on-the-mainframe environment. Performance issues will typically be of secondary importance in a server farm environment. When consolidating servers on the mainframe, planning for performance enables you to make efficient use of the mainframe resources. For more complex projects, for example, a business intelligence application like StoreCompany's OaK (Appendix B, "StoreCompany"), you might want to put in the necessary effort when you set up your system to ensure that you meet your performance goals.

The tools and techniques that performance experts need are readily available, and z/VM gives you great flexibility in achieving performance objectives.

In this chapter, we explore the following questions about Linux on the mainframe:

  • How are performance data collected?

  • How are hardware tuning and software tuning performed?

  • How is capacity planning performed?

  • What is unique about performance in this environment?


15.1 Day-to-day performance

Performance tuning aims to satisfy service level agreements that reflect users' requirements pertaining to performance.

A key advantage of running Linux on the mainframe is the decades of experience developing the right knobs and meters for performance tuning. These well-documented knobs and meters are equally valuable for providing the insight for performance tuning in the new server consolidation Linux-on-the-mainframe world. The knobs are the parameters that the analyst changes and the meters are the reports by which the analyst can observe the effects of the parameter changes. These knobs and meters are realized with the various performance management tools.

Performance measurement and tuning are costly. Why would you do it, and when is a good time to do it? In a server farm environment, when the cost of purchasing a new system is often less than the cost of measurement, it might not be necessary to take any measurements. However, in a Linux-on-the-mainframe environment, it might be a good idea for the following reasons:

  • To maintain and demonstrate compliance with the service level agreements.

    Where there are service level agreements, performance monitoring becomes essential to ensure that the agreements are met and service does not degrade over time. For example, StoreCompany might decide to monitor the new OaK application to learn what to expect on resource consumption with its first application.

  • To prevent surprise resource shortages and to plan capacity for the future.

    If changes in your workload cause over-utilization of a real resource, it might indicate that the application does not scale and needs to be modified. Under these circumstances, routinely monitoring performance and gathering historical data are essentialtasks if you are to anticipate problems and deal with them before they reach crisis proportions. For example, StoreCompany might decide to monitor the new OaK application to gather the data during the beta phase and use that understanding to project the resources required if OaK matures.

  • To ensure that adequate resources are kept in reserve. Reserve resources are needed to handle the various peaks of the guests or even the introduction of new workloads.

    Like ISPCompany, if your installation has several different clients, you might want to monitor each server to ensure that resources are not wasted in the local Linux image (and hence are not consuming unneeded resources in z/VM). Or you might want to measure only the aggregate use across all of a client's servers to ensure that the SLA is being met. ISPCompany wants to closely monitor resources because it gets market share by charging lower fees than the competition for similar levels of service. To maximize profits, it is important to optimize the use of assets, whether human or machine.

For these and other reasons, a mainframe installation requires both real-time monitoring and analysis as well as a structured methodology for capacity planning. z/VM has the built-in tools to monitor and collect performance data and other data that you can use for billing, input to capacity planning, analyzing bottlenecks, and tuning purposes.

See Chapter 26, "Performance Reference," for more details about the available tools.

15.1.1 Performance metrics

Many metrics are used for performance tuning and capacity planning. The following metrics are easily available with Linux on the mainframe and might help you to manage your SLA in accordance with agreement:

Response time

This is a measure of the time taken to perform a certain task, such as running a specific job. Response time is related to external throughput rate (ETR).

CPU utilization

This is a measure of how busy the processor is over any given period of time, for example, 24 hours. To obtain maximum return on investment, it is usual to purchase a system whose capacity matches the requirements as closely as possible plus whatever near-term growth is expected during this budget cycle. CPU utilization is a factor in the internal throughput rate (ITR).

Memory utilization

This is a measure of how the real memory and the individual guest virtual memory are used, and is typically derived from page usage rates.

I/O utilization

I/O utilization covers two areas: the utilization rates of paths and devices and the amount of disk space currently in use. He yssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss


Appendix A. ISPCompany

ISPCompany is a hypothetical company that we created to show many of the characteristics of real companies. This company and StoreCompany are used in this book to demonstrate issues of security, performance, and system management in a realistic way.


A.1 Corporate profile of ISPCompany

ISPCompany started out as a service bureau. It is now also in the business of outsourcing IT services to companies who do not want to maintain a server themselves. Founded in 1985, ISPCompany is a pioneer in the outsourced IT management business for small and medium-sized companies. Recently, ISPCompany has recognized that companies are moving into the "e-business on demand" space and need variable amounts of computing capacity for certain aspects of their business computing. In the future, ISPCompany also hopes to outsource applications. The company has one site outside the city limits of Pancake City, New Jersey, on the east coast of the United States (Figure A-1) and one site in southwest Germany.

Figure A-1. ISPCompany's building in Pancake City, New Jersey

The company's core activity has been providing infrastructure, systems, and managed network services to large and medium-sized enterprises. Due to rapid changes and increasing complexities in information technology, outsourcing is an efficient solution for many businesses.

ISPCompany helps customers bridge the gap between their mainframe/legacy operations and the Internet with a comprehensive suite of IT-managed services that assure the highest levels of availability of mission-critical applications. The company also provides integration expertise to link customers' online businesses and e-commerce functions with existing proprietary back office systems. ISPCompany's ability to remotely monitor all aspects of a company's operations greatly reduces the number of people needed to perform the management function. Managed services are offered on an outsourced basis to companies seeking optimal performance, security, reliability, and scalability of their computer systems without the large cash outlays required to maintain this function in-house. In today's challenging business climate, when executives are looking for ways to cut costs and maximize their business potential, ISPCompany offers the expertise and advanced tools to effectively control their entire IT operation.

ISPCompany has two locations, but in this chapter we will concentrate on only one of them.

The customers of ISPCompany range from Fortune 1000 to Fortune 200-type companies.

A.1.1 Business objectives

The company's objective is to provide a comprehensive alternative to meet all or part of its clients' information technology and mission-critical Internet requirements. Typically, the company enters into contracts with clients providing for automatic renewal unless prior written notice is given. The contracts have varying terms, typically ranging from one to five years. The rates for the company's services vary according to factors such as the volume and types of services used by a particular client. Industry-specific outsourcing applications and services are developed so that the company's in-depth knowledge of a particular industry can be applied to servicing multiple clients in that field. The company currently provides outsourcing services to approximately 500 clients in such diverse fields as financial services, publishing, home health care, apparel, and consumer products.


A.2 Offerings

The company offers a spectrum of services that ranges from supplying the bare iron through applications, helpdesk, and backup/restore services. The company provides data system outsourcing services in the following areas:

  1. Internet Data Center and server-hosting services

  2. A full suite of automated integrated managed services that assure optimal performance, security, reliability, and scalability of a customer's computer systems operations

  3. Information technology (IT) hosting services

  4. Systems infrastructure and operations consulting

These offerings correspond to how much the customers might want to do themselves, which ranges from doing everything themselves to having everything, including user management, done by ISPCompany. Examples of offerings are shown in Table A-1.

Table A-1. Spectrum of ISPCompany offerings

What ISPCompany offers

What the customer can do

Base system

Everything

Base system plus the operating system

Health monitoring, all applications, all user management

Base system, OS, middleware

Install packages

Base system, OS, middleware, applications

User management

Base system, OS, middleware, applications, helpdesk

End-user tasks only

Base system, OS, middleware, applications, helpdesk, backup/restore

End-user tasks only

ISPCompany offers Linux distributions from the following companies:

  • Red Hat

  • SuSE

ISPCompany offers the following middleware:

  • Apache

  • IBM WebSphere Application Server (WAS)

  • SendMail and Bynari Insight Server

  • DB2 UDB

A.2.1 The Internet service offering

The ISP offering is a stable part of ISPCompany's revenue model. It is low-risk with reasonable returns, but offers only moderate growth opportunity. ISPCompany has a few thousand mail customers, some of whom also have some static Web pages to be served. The company has both low-speed and high-speed lines that its clients can use. Its rates are very competitive and it emphasizes the use of the static Web pages to present a company image along with an associated mail account that Web users can contact for additional information. The mail service side of the ISP business allows many of the clients to avoid owning and managing a mail server at their business site.

A.2.2 The data systems outsourcing offering

The outsourcing of data systems services, whereby a client company obtains all or part of its information processing requirements (including systems design, software management and hardware, network communications, training, maintenance, and support) from providers such as ISPCompany, continues to be a growing trend. ISPCompany believes that it is generally 10% to 50% more cost-effective and efficient for its clients to outsource information processing services to the company than it would be to provide equivalent services for themselves by purchasing or leasing in-house systems and hiring or contracting for service and support personnel.

Outsourcing provides clients with the following benefits:

  • The refocus of personnel, financial, and technological resources on core business and client-related activities

  • Access to highly skilled personnel and newer technology resources

  • Access to resources that support technological re-engineering

  • Reduction of operating costs

  • Reduction of future investment in infrastructure not directly related to the core business activity


A.3 Description of environment

ISPCompany began offering Internet data center (IDC) services in the first quarter of fiscal 2000 by retooling a portion of its state-of-the-art mainframe outsourcing data center into an IDC. Hosting Internet systems demands many of the same skills, procedures, and physical requirements as for mainframe and midrange environments.

The company's IDC features state-of-the-art physical and network infrastructure, including:

  • Multi-layered security with 24x7 guard station, video surveillance, and restricted physical (secure equipment cages) and network access

  • Redundant power systems with two tiers of backup (two UPSs and two diesel power generators) to ensure continuous operation in the event of a power outage

  • Leading edge fire suppression as well as smoke and fire detection systems, and sprinklers

  • Redundant climate control systems in a raised floor environment

  • Four high bandwidth, network access points with physically diverse connections into the IDC

A.3.1 Physical environment of Pancake City site

The 67,000 square feet (6700 sqm) building at the Pancake City site has two floors. The first floor (shown in Figure A-2) houses the data center, a conference room with one-way windows out to the secured equipment area and into the control room, and the building entrance with a security guard. The second floor houses the offices.

Figure A-2. Floor plan of Pancake City site

A.3.2 Logical setup of Pancake City site

ISPCompany's logical setup is shown in Figure A-3, and consists of:

  • Customer cages containing equipment owned by customers of ISPCompany

  • A zSeries box for server consolidation, connected to disk and tape storage

  • A private LAN connecting all machines with the control center (remember the HMC of zSeries)

  • Entire setup connected through an Internet backbone with fiber (multiple connections)

Figure A-3. Hardware and software logical setup of ISPCompany

ISPCompany handles many different platforms. The company uses IBM Tivoli as a systems management platform.

At the rate that the SAN management technology is improving, ISPCompany's New Jersey site is looking at deploying a shared SAN for a subset of its outsourcing clients in late 2003. The first sets of clients moved to SAN will be those with over 100 GB of data who need their own dedicated channel path.

IBM Tivoli Storage Manager server is running in a Linux image on z900. This is used to back up customer data on tape.


A.4 Example new client

This customer approaches ISPCompany about outsourcing:

  • Small server farms that do back-office-type things

  • The beginnings of this client's Web business, as it doesn't want to get into Web administration or system administration (they have a business to run).

The client used to have only static Web pages (information about their business) with ISPCompany; but really it wants to go to dynamic page interaction with the rest of its business and is considering a complete outsourcing. But as any conservative business would do, the first step is to move the hardware, then buy some services, and finally, contract it all. This client was into horizontal growth. And ISPCompany, as it adds more clients of this type, is itself into horizontal growth.


Appendix B. StoreCompany

StoreCompany is used in this book to demonstrate issues of security, performance, and system management. StoreCompany is the larger of the two example companies, being a Fortune 1000 company.

StoreCompany demonstrates different possibilities for using Linux on the mainframe:

  • Integration of the middle-tier with the back-end

  • Vertical and horizontal growth

  • Extending existing applications by using new technologies on Linux

StoreCompany, an upmarket department store chain with 85 stores, has been in business for decades, mostly in large cities in the eastern United States. Its IT organization has been very progressive over the years moving from just back-office and support functions to actively managing sales and inventory in real-time. It also has moved into business-to-business type transactions with its key suppliers. StoreCompany deploys the following:

  • WebSphere Application Server (WAS)

  • DB2

  • CICS

  • VSAM

At the end of 2001, StoreCompany decided to expand into making its store items (basically its inventory database) available online to shoppers. As the very first Linux on the mainframe project, a firewall server and a proxy server were created as Linux images. See B.4, "Project 1: Firewall and proxy server."

By extending their use of WebSphere Application Server (WAS) and WebSphere Commerce Suite (WCS), StoreCompany next was able to quickly create a Web store using a few Linux images for the Web front-end part, and leveraging all the CICS, DB2, and VSAM databases on the z/OS side for the critical parts of the new order confirmation and fulfillment system. See B.5, "Project 2: Web application server."

Project 1 and Project 2 are examples of back-end integration.

Given how successful Projects 1 and 2 have been and how quickly they went from pilot to production, it was decided to give the marketing organization an opportunity to explore new opportunities. Any new opportunity would have to isolate the IT-required changes within a small number of Linux images with no changes to the normal production applications on z/OS.

The new business opportunity is to enter the specialized gift category, leveraging the StoreCompany brand name. A new catalog store will be created within the existing online store. It will be called One-of-a-Kind (OaK) and will offer items in the $300 to $10,000 price range from artists specializing in paintings, sculpture (stone and wood), and jewelry. While the items (with full color photos) will appear in the online catalog, StoreCompany will not actually stock the items. Instead, when an item is purchased, the order is electronically routed to the artist for scheduling and shipping. Customer billing is handled by StoreCompany. To the StoreCompany customer, it looks and feels like an order from StoreCompany, including StoreCompany's customer satisfaction guarantee. See B.6, "Project 3: OaK project."


B.1 Corporate profile of StoreCompany

StoreCompany started out as a small western Pennsylvania department store chain in the 1960s. It has since expanded its network of stores over most of the eastern United States.

While merchandising is the heart of StoreCompany's business, it relies on its infrastructure for support. Its management information systems help store clerks, increase accuracy and efficiency of the four distribution centers, and manage the details of product range planning. Figure B-1 shows an overview organization chart.

Figure B-1. Organization chart of StoreCompany

    [ Team LiB ]


B.2 Description of environment

B.2.1 Logical setup

Figure B-2 shows the logical setup of StoreCompany. StoreCompany has two zSeries 900 Model 105 (with five CPUs) machines configured in a Parallel Sysplex. The Coupling Facility in the second LPAR of the second machine (LPAR-E) uses an Integrated Coupling Facility (ICF). The business applications run under z/OS with DB2, CICS, VSAM and WAS.

Figure B-2. Logical view of StoreCompany, before Project 1

    [ Team LiB ]


B.3 Programming model and middleware platform

StoreCompany made a strategic decision to develop an object-oriented model of its business processes.

The advantage of object-oriented program modules is that their clearly defined interfaces make them easy to reuse. There are also effective tools that support object-oriented programming. Enhancing and changing object-oriented programs is much faster than with traditional programming methods. An object-oriented approach allows for the rapid development and deployment of IT based projects that StoreCompany needs to turn its IT capabilities into a competitive edge.

StoreCompany chose IBM WebSphere Application Server (WAS) as the middleware plat- form for its object-oriented applications. WAS offers a solid basis with rich security interfaces and transaction capabilities. For StoreCompany it was important that the spectrum of supported operating systems include z/OS and Linux. StoreCompany's vital business data and transaction processing are all on z/OS with its strong availability and recovery features. Mainframe Linux is intended as the initial integration point and rapid deployment platform for new technology ideas.

This dual approach of running some application parts on Linux and others on z/OS is tied together logically by WAS and physically by the mainframe hardware with its LPARs and HiperSockets. As an application evolves and its requirements to the underlying operating system change, StoreCompany can use WAS to redeploy the application or parts of it on a different operating system.

Because Linux and WAS subscribe to open standards, the flexibility and potential for integration that StoreCompany gains goes beyond its current infrastructure. Open standards facilitate assimilation of future additions to its infrastructure. Open standards also pave the way to take advantage of current and future developments, for example, business to business applications or on demand computing.


B.4 Project 1: Firewall and proxy server

StoreCompany's desire to add its catalog to the Internet sparked an audit into security and feasibility of the project. A firewall and demilitarized zone would be needed to separate the zSeries from the Internet. The conclusion was that infrastructure servers could be located as Linux guests under z/VM on the mainframe. This would have three desirable results: a standard setup for all guests, some savings in actual hardware, and a fairly simple project for introducing Linux on the mainframe into the production environment.

The team doing the Linux infrastructure work was composed of both mainframe and UNIX/Linux people. The infrastructure work went well. It was fast (75 days) and it was on budget. After the conclusion of Project 1, security had the structure needed to give the green light to the more complicated online catalog project.

As a follow up from Project 1, StoreCompany then decided to move its departmental Samba servers from UNIX and Windows servers to the mainframe. These servers hold data for the various StoreCompany departments.

B.4.1 Changes needed

  • Project 1 needs only 10% or so of an engine, but to prepare for future projects, one IFL is placed on order.

  • As soon as the IFL arrives, z/VM is set up on a new LPAR with the IFL, LPAR-C.

  • Acquire a Linux distribution, in this case, SuSE.

  • Configure firewall and proxy.

  • Test firewall and proxy.

  • Test IP filtering.

B.4.2 Implementation

Work started out with Linux on z/VM under the test LPAR, LPAR-A. Once that worked, an IFL was ordered as a trial, knowing that it could be sent back if things did not work. While the IFL order process was wending its way through StoreCompany, the project went into production using the test LPAR just before the IFL was to show up to accelerate delivery. Once the IFL was in place and the second z/VM was set up, ISPCompany just moved over the Linux image and the network attachment, and so forth. The final implementation is shown in Figure B-3.

  • The firewall and the proxy server were located on a new LPAR, LPAR-C, which uses an IFL engine.

    The proxy re-addresses requests from the Internet to a machine that can handle requests. That machine is on a network that is not visible to the outside. In other words, the proxy does two things: re-direction, and occasionally workload management.

  • As a second line of defense, IP filtering is used between proxy and the application images.

    IP filtering is used to ensure that traffic in and out is only from servers that have business here. For example, only traffic from the proxy is allowed in. This is the minimum needed to define the end of the DMZ.

  • Intrusion detection systems (IDS) will be implemented on three levels: Network, host, and Web.

Figure B-3. Logical setup of StoreCompany after locating the firewall and the proxy on the zSeries machine

    [ Team LiB ]


B.5 Project 2: Web application server

The next project StoreCompany undertakes is to implement a Web catalog. The Web catalog project can exploit the work done by the successful team from Project 1.

B.5.1 Changes needed

  • WAS and WCS for Linux. WCS provides everything StoreCompany needs to set up an online catalog.

  • Three new Linux images: one for development and test, two catalog Web servers.

B.5.2 Implementation

The Web server which StoreCompany uses is Apache. Once the Web business was started, a second IFL was immediately ordered to ensure that there would be no hardware single point of failure for the Linux images. The final implementation is shown in Figure B-4.

Figure B-4. Linux Web server added

    [ Team LiB ]


B.6 Project 3: OaK project

Linux has created a beachhead in StoreCompany through two successful projects. The first was a simple consolidation of some security-related infrastructure images (firewall and proxy servers) onto a few Linux images on one of the z900 machines. The second project, described above, was the use of WCS to create an online store.

The new marketing programs manager at StoreCompany has suggested that it should be a relatively simple thing to offer a new brand of products: a category of exclusive gifts called One-of-a-Kind (OaK). They would be presented on the StoreCompany Web site, but would actually not be in stock at StoreCompany. Instead, third-party vendors (craftspeople, artists) would offer their products for inclusion in the OaK Web catalog. This would give the third parties the advantage of this new marketing channel, as well as the brand recognition of StoreCompany.

A small team consisting of one person familiar with the WAS interfaces to the order entry and inventory systems, one person from the Linux WCS side, one Web-page designer, and the lead marketing person whose idea spawned the project will design and implement these ideas. The intent is to have a first deployment for customers in one month, with three experimental trial periods lasting an aggregate of six months. At the end of the project, a recommendation on how and whether to add this opportunity to the standard portfolio will be made to the management committee.

B.6.1 Concept

Artists supply StoreCompany with information about the articles for sale. StoreCompany presents the articles in its Web catalog. Customers can then browse the catalog and place orders. When an order comes in through the catalog to StoreCompany, it is sent directly through to the third party who creates the item and then ships it to the customer. The process is illustrated in Figure B-5.

Figure B-5. Flow of new business opportunity "One-of-a-Kind"

StoreCompany adds the articles into their Web catalog under the existing category of "specialty items." The lead OaK person wants to measure the probability of a purchase when certain types of customers are presented with search results tailored to their known preferences.

Using business intelligence (BI) methods to mine customers' buying patterns it is possible to both determine an appropriate target customer and identify OaK items that are likely toappeal. When a customer from this special set enters their catalog site, a BI query is triggered to create a pseudo search page result from the OaK set of items. The page is presented to the customer in a pop-up window with the heading "a new service for you."

Market opportunity and benefits

For StoreCompany, the following benefits are expected:

  • Keep high-end customers and encourage customers to buy more

  • Receive a higher profit margin per transaction

For the artists and craftspeople, the following benefits can be expected:

  • A new channel to reach clients the company wouldn't normally see

  • Benefit from the brand name of "StoreCompany"

  • Little interaction with the end client during the business transaction process (more time for art or crafts)

For the customer, the following benefits are expected:

  • One-of-a-Kind special items from a trusted, respected retailer

  • Ease in finding the perfect present

B.6.2 Business objectives

  • Allow the marketing department to rapidly try out new ideas for creating new retail business opportunities by making experimental applications available to customers

  • Increase revenue and profit per individual sale.

B.6.3 Policies

  • There will be risk management plans in place for all "market trials."

  • All changes are to be implemented within new Linux images exploiting only IFL engines. The new Linux images should place less than a 10% peak and a 1% average utilization increase on the z/OS image. Thus it cannot impact normal business commitments.

  • No business data should be kept on the new Linux images. That is, the items for sale are part of the sales database on DB2; the transactions for completing a sale will be the standard CICS transactions, and any inventory information for the new opportunities is kept in the inventory database on z/OS.

  • The marketing department owns the new Linux images—that is, it owns the availability requirements and any resulting impact to its business experiment. The IT organization will support the operation of those images.

B.6.4 Procedures

The OaK team has established the following:

  • All new business data representing the OaK products, customers, and vendors will be managed as part of the standard backup/restore/disaster recovery processes in place on z/OS for the respective databases.

  • The OaK catalog starting point will be the only change allowed on the production catalog system. During this trial phase, control and information will be given over to one of the new Linux images.

B.6.5 Changes needed

  • New business intelligence (BI) image that coordinates all the activities of OaK, such as order entry and delivery.

  • Change to production system: authorizing the new BI image to make certain requests of the production system that result in the customer data being forwarded to the BI image for further processing.

  • Customer data to artist or craftsperson for shipping; shipping process data back to customer.

  • New code will be required to implement dynamic concatenation of suitable items (such as Amazon.com's "Customers who bought this also bought").

B.6.6 Implementation

A new BI application was written to handle gift offering in connection with the OaK. Some of its functions are:

  • Based on shopping history and customer profile, find and display list of gifts in suitable interest area and price range.

  • When customer orders, return customer to main application to conclude sales. Send shipping information to third-party vendor for handling.

A datamart is a convenient way to get information from all your data. A datamart approach was chosen to handle the different pieces of information that need to be consolidated into a new Datamart database:

  • A batch job will run on z/OS that extracts the information needed for the new OaK query from the various databases, creating a unique database that will be transferred to DB2 UDB on a new Linux image that has all the data optimized for real time queries of customer buying interests.

  • Change to production side: Schedule the datamart building job at some "quiet time" on the production machine. Since only a small portion of StoreCompany's business is 24x7 (the catalog) and the larger portion is still store-based, the company does have these quiet times.

The BI application and the datamart were added to StoreCompany's set up, as shown in Figure B-6.

Figure B-6. The new datamart image and Linux image for business intelligence are added

Other changes include:

  • WAS to implement the cross-system interactions between the new Linux images and z/OS.

  • BI transactions that finds customer profile will be implemented in new image.

  • The BI image will not have an external IP address.

  • The existing accounting and ordering applications are not changed and are separated from the BI image (and so is most of the rest of this customer's catalog experience).

  • A HiperSockets LAN and Guest LANs are used for the most secure communications, as illustrated in Figure B-7:

    - A HiperSockets LAN connects the z/OS applications with the new business intelligence image, as well as the Web application image, the Web server, and the proxy.

    - One Guest LAN connects the firewall image with the proxy server, and another connects the datamart and business application images with the Web server.

    - Cross-memory services provide communication among the z/OS applications.

    Figure B-7. Logical implementation of new business opportunity, One-of-a-Kind gifts

B.6.7 Opportunity cost

  • Development and deployment cost:

    - Excess capacity of the second IFL will be used, as the online catalog needs the first one, and growth is expected

    - Project staff

    - One new DB2 UDB license

  • Cost for including articles in catalog: small, just the storage it takes to describe one.

  • Cost of finding articles to include: advertising cost and part of the cost of one specialized buyer for selection of appropriate items.

  • Cost of failure: Low. Some of the new code will be re-usable for other experiments.


Further Reading

The available information related to Linux on the mainframe is rapidly changing and new information becomes available every day. Nevertheless, because the topic is new, we put together a selective list of what we think might be of interest to you when starting with a Linux on the mainframe project.

In contrast, the bibliography provides a more comprehensive list.

You can obtain IBM documentation through the IBM Publication Center at: http://www.elink.ibmlink.ibm.com/public/applications/publications/cgibin/pbi.cgi.

The suggested literature includes IBM Redbook publications. To obtain a list of all available Redbook publications about Linux on the mainframe, visit:

http://publib-b.boulder.ibm.com/cgi-bin/searchsite.cgi?query=linux+AND+zseries.

For a list of all IBM Redbook publications on Linux, visit:

http://publib-b.boulder.ibm.com/cgi-bin/searchsite.cgi?query=linux.

For other suggested literature, we have either provided a URL where the publication is available for download in a printable format or the ISBN to help you order it through a retailer.

Part 1, "Linux on the Mainframe – an Introduction"

Hall, Mark Evolution of IBM's Open-Source Strategy. A Time Line of Linux and Open Source Developments. In: Computerworld from IDG.net, October 2001. Available on Internet: http://www.computerworld.com/softwaretopics/os/linux/story/0,10801,65073,00.html.

Provides a time line of Linux and Open Source developments through 2001 and points to other relevant articles.

Hoskins, Jim, and Bob Frank Exploring IBM eServer zSeries and S/390 Servers: See Why IBM's Redesigned Mainframe Computer Family Has Become More Popular than Ever! Florida: Maximum Press, 8th Edition, 2003. (ISBN: 1-855068-91-3)

Provides a thorough look at mainframe hardware and software.

IBM IBM eServer zSeries 900 and z/OS. IBM Reference Guide, August 2002. (IBM Form Number: G326-3092-03)

IBM's technical guide to the z900 machine and z/OS.

Raymond, Eric S. The Cathedral and the Bazaar. O'Reilly & Associates, February 2001. (ISBN: 0-596001-08-8)

An easy-to-read book for anyone interested in learning how the Open Source community functions. The author is an early community member. From a sociologist's point of view, he explains how the "society" is formed and manages to accomplish tasks that others thought could not possibly happen.

Weiss, George Conditions for Strategic Linux Adoption. Gartner Research Note, May 2002. Available at the Gartner Site: http://www.gartner.com.

Discusses Linux as a enterprise-ready operating system.

Part 3, "Is Linux on the Mainframe for Me?"

Altmark, Alan, and Cliff Laking The Value of z/VM: Security and Integrity. IBM Technical Paper, May 2002. (IBM Form Number: GM13-0145-00)

Provides an overview of the security and integrity characteristics of the z/VM hypervisor.

Alves, L.C, et al. RAS Design for the IBM eServer z900. In: IBM Journal of Research and Development, Volume 46, Numbers 4/5, Jul/Sept 2002, pp. 503–522. Available on Internet:

http://www.research.ibm.com/journal/rd/464/alves.html.

Amrehn, Erich, et al. Linux on IBM zSeries and S/390: High Availability for z/VM and Linux. IBM Redpaper, 21 June 2002. (IBM Form Number: REDP0220)

Helps planning for and installing a high-availability solution for Linux for zSeries running under z/VM. Discusses the high-availability possibilities of a Linux for zSeries environment.

Geiselhart, Gregory, et al. Linux on IBM zSeries and S/390: Cloning Linux Images in z/VM. IBM Redpaper, 3 September 2002. (IBM Form Number: REDP0301)

Detailed how-to description of a cloning implementation.

Ogden, Bill, and Bill White Getting Started with zSeries Fibre Channel Protocol. IBM Redpaper, 13 August 2002. (IBM Form Number: REDP0205)

Helps you understand the concepts of zSeries Fibre Channel Protocol support and shows how various SCSI devices can be configured to build a zSeries FCP environment.

Schneier, Bruce Applied Cryptography. Protocols, Algorithms and Source Code. C. John Wiley & Sons, 2nd. Edition, 1995. (ISBN: 0-471117-09-9)

A comprehensive guide to modern day cryptography. Guides the user as well as the professional cryptographer through the algorithms, uses, and applications needed to secure data and communications.

Thomas, Stephen SSL and TLS Essentials: Securing the Web. John Wiley & Sons, 2000. (ISBN: 0-471383-54-6)

An introduction to Secure Socket Layer technology providing an end-to-end view of the technology behind e-commerce and the Web.

White, Bill, et al. zSeries HiperSockets. IBM Redbook. (IBM Form Number: SG24-6816)

This book offers a broad description of the architecture, microcode function, and operating systems support of HiperSockets. It enables you to plan and install a HiperSockets network.

Ziegler, Robert Linux Firewalls. New Riders Publishing, 1999. (ISBN: 0-735709-00-9)

Provides guidance for single- and multiple-system firewall solutions using Linux, along with the basics of packet filtering and the DMZ. In addition, aspects of basic system hardening, intrusion detection, and system monitoring are covered.

Zwicky, Elizabeth, Simon Cooper, and Brent Chapman Building Internet Firewalls. O'Reilly & Associates: 2nd Edition, June 2000. (ISBN: 1-565928-71-7)

A general guide to firewall technology. Introducing various network communications, why they need to be protected and how to build a firewall to do the job.

Part 4, "Making the Most of Linux on the Mainframe"

Compaq Business Value Methodology in Support of Networked Storage Architectures. Compaq White Paper. Prepared by Enterprise Storage Group Compaq Computer Corporation. February 2002. Available on Internet: ftp://ftp.compaq.com/pub/products/storageworks/whitepapers/15ZV-1201A-WWEN.pdf.

Explains the various types of network-based storage methods (for example, NAS, SAN). It is not specifically about Linux, but it lays out much of the terminology in an understandable way. It also shows how a slavish approach to TCO/ROI might lead one to wrong conclusions on the best approach to solve storage needs.

Geiselhart, Gregory, Tung-Sing Chong, and Michael Donovan Cloning Linux Images in z/VM. IBM Redpaper, 03 September 2002. (IBM Form Number: REDP0301)

In a LinuxWorld demonstration, IBM created thousands of Linux images averaging one new image per minute. This Redpaper gives an insight into exactly what those images could do and how the cloning process was managed.

IBM Large Systems Performance for IBM eServer zSeries and S/390. Available on Internet: http://www.ibm.com/servers/eserver/zseries/lspr/.

Explains the background of the Large Systems Performance Report process. Includes descriptions of the workloads as well as a recent set of measurements. The document is typically updated as new mainframe processors become available.

NTP Software Do you need a Storage Management Policy? NTP Software White Paper, Rev. 2.1, August 2001. Available on Internet: http://www.ntpsoftware.com/WhitePapers/docs/DoYouNeedAStoragePolicy.pdf.

This white paper argues that companies' crucial data call for explicit storage management policies to guide the organizational interaction with these data. The paper also explores how to formulate policy in a way that makes people comply with it.

Ogden, Bill, and Bill White Getting Started with zSeries Fibre Channel Protocol. IBM Redpaper, 13 August 2002. (IBM Form Number: REDP0205)

Gives a description on how to set up the SCSI over FCP environment for Linux on the mainframe.

Rayns, Chris, et al. Linux on zSeries and S/390: System Management. IBM Redbook, 04 December 2002. (IBM Form Number: SG24-6820)

Explains the "how to" aspects of Linux-on-the-mainframe systems management. Apart from a description of IBM Tivoli, this book provides descriptions of CA and BMC tools that are written by technical people from the respective companies.

Part 5, "Running Applications"

IBM Porting UNIX Applications to Linux. Hints and Tips. IBM Technical Paper, January 2002. (IBM Form Number: GM13-0115)

Provides hands-on information and practical tips for those interested in the technical aspects of porting an application to Linux on the mainframe.

Wahli, Ueli, et al. Enterprise Integration with IBM Connectors and Adapters. IBM Redbook, 22 February 2002. (IBM Form Number: SG24-6122)

Explores how IBM connectors and adapters can be used to solve the integration problems many enterprises face when deploying e-business solutions. The book uses scenarios to show how to use a J2EE connector with an existing application and how a J2EE connector or the MQSeries Adapter Offering can be used in creating a new application.

Wakelin, Phil, et al. Java Connectors for CICS: Featuring the J2EE Connector Architecture. IBM Redbook, 22 March 2002. (IBM Form Number: SG24-6401)

A practical guide with technical details for integrating Java applications with CICS back-ends. The book explores several possibilities, with a focus on the J2EE Connector Architecture. It includes comprehensive code samples.


Bibliography

    Books

    IBM Redbooks

    IBM articles and papers

    Other IBM publications

    Other articles and papers


Books

Hoskins, Jim, and Bob Frank Exploring IBM eServer zSeries and S/390 Servers: See Why IBM's Redesigned Mainframe Computer Family Has Become More Popular than Ever! Florida: Maximum Press, 8th Edition, 2003. (ISBN: 1–855068–91–3)

Raymond, Eric S. The Cathedral and the Bazaar. O'Reilly & Associates, February 2001. (ISBN: 0–596001–08–8)

Schneier, Bruce Applied Cryptography. Protocols, Algorithms and Source Code. C. John Wiley & Sons, 2nd. Edition, 1995. (ISBN: 0–471117–09–9)

Ziegler, Robert Linux Firewalls. New Riders Publishing, 1999. (ISBN: 0–735709–00–9)

Zwicky, Elizabeth, Simon Cooper, and Brent Chapman Building Internet Firewalls. O'Reilly & Associates: 2nd Edition, June 2000. (ISBN: 1–565928–71–7)


IBM Redbooks

The following publications can be found on Internet: http://www.redbooks.ibm.com by giving the correspondant IBM Form Number in the Search Command.

Amrehn, Erich, et al. Linux on IBM zSeries and S/390: High Availability for z/VM and Linux. Redpaper, 21 June 2002. IBM Form Number: REDP0220.

Amrehn, Erich, et al. Linux on IBM zSeries and S/390: Server Consolidation with Linux for zSeries. Redpaper, 30 July 2002. IBM Form Number: REDP0222.

Amrehn, Erich, Ulrich Boche, and Manfred Gnirss Linux on IBM zSeries and S/390: Securing Linux for zSeries with a Central z/OS (RACF) LDAP Server. Redpaper, 21 June 2002. IBM Form Number: REDP0221.

Endrei, Mark, et al. IBM WebSphere V4.0 Advanced Edition Handbook. Redbook, 13 March 2002. IBM Form Number: SG24-6176.

Geiselhart, Gregory, Tung-Sing Chong, and Michael Donovan Cloning Linux Images in z/VM. Redpaper, 03 September 2002. IBM Form Number: REDP0301.

Geiselhart, Gregory, et al. Linux on IBM eServer zSeries and S/390: Application Development. Redbook, 5 August 2002. IBM Form Number: SG24-6807.

Injey, Frank IBM eServer zSeries 900 Technical Guide. Redbook, 06 September 2002. IBM Form Number: SG24-5975.

MacIsaac, Michael, et al. Linux for IBM eServer zSeries and S/390: Distributions. Redbook, 20 September 2001. IBM Form Number: SG24-6264.

MacIsaac, Michael, et al. Linux on IBM zSeries and S/390: ISP/ASP Solutions. Redbook, 20 December 2001. IBM Form Number: SG24-6299.

Ogden, Bill, and Bill White Getting Started with zSeries Fibre Channel Protocol. Redpaper, 13 August 2002. IBM Form Number: REDP0205.

Rayns, Chris, et al. Linux on zSeries and S/390: Systems Management. Redbook, 31 October 2002. IBM Form Number: SG24-6820.

Wahli, Ueli, et al. Enterprise Integration with IBM Connectors and Adapters. Redbook, 22 February 2002. IBM Form Number: SG24-6122.

Wakelin, Phil, et al. Java Connectors for CICS: Featuring the J2EE Connector Architecture. Redbook, 22 March 2002. IBM Form Number: SG24-6401.


IBM articles and papers

Adair, R.J., et al. A Virtual Machine System for the 360/40. IBM Cambridge Scientific Center Report 320-2007, Cambridge, Mass., May 1966.

Altmark, Alan, and Cliff Laking The Value of z/VM: Security and Integrity. IBM Technical Paper, May 2002. (IBM Form Number: GM13-0145-00). Available on Internet: http://www.ibm.com/servers/eserver/zseries/library/techpapers/.

Alves, Luiz, et al. RAS Design for the IBM eServer z900. In: IBM Journal of Research and Development, Vol. 46, No. 4/5, Jul/Sept 2002, pp. 503–522. Available on Internet: http://www.research.ibm.com/journal/rd/464/alves.html.

Amdahl, G.M., G.A. Blaauw, and F.P. Brooks, Jr. Architecture of the IBM System/360. In: IBM Journal of Research and Development, Vol. 8, No.2, 1964, pp. 87–101. (Reprinted in IBM Journal of Research and Development, Vol. 44, No.1/2, Jan/Mar 2000). Available on Internet: http://researchweb.watson.ibm.com/journal/rd/441/amdahl.pdf.

Auslander, M.A., D.C. Larkin, and A.L. Scherr The Evolution of the MVS Operating System. In: IBM Journal of Research and Development, Vol. 25, No. 5, Sept. 1981, pp. 471-482. Available on Internet: http://www.research.ibm.com/journal/rd/255/auslander.pdf.

Guski, Richard, et al.Security on z/OS: Comprehensive, Current, and Flexible. In: IBM Systems Journal, Vol. 40, No. 3, May 2001, pp. 696– 720. Available on Internet: http://www.research.ibm.com/journal/sj/403/guski.html.

Mealy, G. H., B.I. Witt, and W.A. Clark The Functional Structure of OS/360 (Parts I, II, and III). In: IBM Systems Journal, Vol. 5, No. 1, 1966, pp. 3-51. Available on Internet: http://www.research.ibm.com/journal/sj/.

Mueller, Michael, et al. RAS Strategy for IBM S/390 G5 and G6. In: IBM Journal of Research and Development, Vol. 43, No. 5/6, 1999, pp. 875–888. Available on Internet: http://www.research.ibm.com/journal/rd/435/muellaut.html.


Other IBM publications

14K Days, A History of the Poughkeepsie Laboratory. 1984.

IBM eServer zSeries Performance Reference, October 2002. Available on Internet: http://www.ibm.com/servers/de/downloads/brochures/G511_4172_06.pdf.

IBM eServer zSeries 900 and z/OS. IBM Reference Guide, August 2002. (IBM Form Number: G326-3092-03). Available on Internet: http://www.ibm.com/servers/eserver/zseries/library/refguides/pdf/g3263092.pdf.

IBM WebSphere Portal for Multiplatforms V4.1 Adds Support for Linux on zSeries. IBM Software Announcement, 24 September 2002. Available on Internet: http://www.ibm.com/software/webservers/portal/news.html.

IBM zSeries 900 and z/OS Reference Guide. 2001. (IBM Form Number: G326-3092)

Large Systems Performance for IBM eServer zSeries and S/390. IBM Technical Paper. Available on Internet: http://www.ibm.com/servers/eserver/zseries/lspr/.

Linux for S/390. Device Drivers and Installation Commands. 4th Edition, 2001. Available on Internet: http://oss.software.ibm.com/linux390/docu/l390dd03.pdf.

Porting UNIX Applications to Linux. Hints and Tips. IBM Technical Paper, January 2002. (IBM Form Number: GM13-0115). Available on Internet: http://www.ibm.com/servers/eserver/zseries/library/techpapers/gm130115.html.

Samba on Linux for zSeries. IBM Product Description, June 2001. (IBM Form Number: GM13-0088). Available on Internet: http://www.ibm.com/servers/eserver/zseries/library/specsheets/gm130088.html.

The Value of z/VM for Linux. IBM eServer zSeries Announcement, March 2002. Available on Internet: http://www.ibm.com/servers/eserver/zseries/os/linux/pdf/VM_Value_Guide.pdf.

Tivoli Storage Management Best Practices. IBM White Paper, 19 April 2001. (IBM Form Number: G325-1691)

z/Architecture Principles of Operation. December 2000. (IBM Form Number: SA22-7832). Available on Internet: http://publibfp.boulder.ibm.com/pubs/pdfs/os390/dz9zr000.pdf.

zSeries 900 System Overview. December 2000. (IBM Form Number: SA22-1027).

z/VM Performance Report. IBM VM Performance Document, 18 June 2002. Available on Internet: http://www.vm.ibm.com/perf/docs/zvmperf.html.


Other articles and papers

Aberdeen Group CA's Unicenter TNG Framework: Entry-Level for the Industry's Best Enterprise Management Software Solution. Announcement Profile.

Amdahl, G.M. Validity of the single-processor approach to achieving large scale computing capabilities. In: AFIPS Conference Proceedings, Vol. 30 (Atlantic City, N.J., Apr. 18-20). AFIPS Press, Reston, Va., 1967, pp. 483-485.

Backa, Bruce Why have storage policies? In: SearchStorage Tips & Newsletters, 22 January 2002. Available on Internet: http://searchstorage.techtarget.com/tip/1,289483,sid5_gci798099,00.html

Cap Gemini Ernst&Young US LLC Angelo Salerno and Peter Kostrobala: IBM zSeries Announcement. Changing the Cost of Ownership Paradigm for Enterprise Computing. White Paper, October 2000. Available on Internet: http://www.ibm.com/servers/eserver/zseries/library/whitepapers/pdf/ibm_zseries.pdf.

Claybrook, Bill Linux is on the move - Up! AberdeenGroup InSight, 24 July 2001. Available on Internet: http://www.ibm.com/linux/LinuxInSight.pdf

Compaq Business Value Methodology in Support of Networked Storage Architectures. Compaq White Paper. Prepared by Enterprise Storage Group Compaq Computer Corporation, February 2002.

Computer Associates CA Common Services Architecture Overview. White Paper, 24 January 2002. Available on Internet: http://www3.ca.com/Solutions/Collateral.asp?ID=1313&=2848.

da Cruz, Frank Columbia University Computing History. In: Academic Information Systems of the Columbia University, Feb 2001. Available on Internet: http://www.columbia.edu/acis/history/timeline.html.

Enabling Technologies Group Fred Bothwell: Meeting the Business Challenges of the 21st Century. Comparative Large System Capabilities and Attributes IBM eServer zSeries 900 - HPSun. ETG White Paper Supplement, February 2001. Available on Internet: http://www.etginc.com/whitepap/z900s_final2.pdf.

Gartner Kyte, A.: 24x7 Is a Management Thing. Gartner Research Note, 17 May 2001.

Gartner Weiss, G.: Conditions for Strategic Linux Adoption.Gartner Research Note, May 2002.

Gribbin, Jeff Development of 360/370 Architecture—A Plain Man's View. Electronic Data Systems, February 1989. Available on Internet: http://pucc.princeton.edu/%7Emelinda/gribbin.pdf.

Hall, Mark Evolution of IBM's Open-Source Strategy. A Time Line of Linux and Open Source Developments. In: Computerworld from IDG.net, October 2001.

Hilf, Bill Securing Linux Servers for Service Providers. IBM Linux Technology Center, December 2001. Available on Internet: http://oss.software.ibm.com/linux/papers/security/Securing_Linux_Servers_xSP.pdf.

Hilton, Paul Assessing the Cost/Benefit Ratio for Storage Quotas. In: SearchStorage Tips & Newsletters, 26 March 2002. Available on Internet: http://searchstorage.techtarget.com/tip/0,289483,sid5_gci812453,00.html

Hurwitz Group zSeries: Economic e-business Server. Hurwitz Group, Inc., March 2001.

Illuminata Dianne McAdam: Storage as Risk Management. Illuminata Publication, 05 February 2002. Available on Internet: http://www.illuminata.com/cgi-local/pub.cgi?docid=storagerisk.

International Technology Group Value Proposition for e-Infrastructures: Cost/Benefit Case for IBM eServer. ITG Report, May 2002. Available on Internet: http://www.ibm.com/servers/solutions/serverconsolidation/pdf/itg.pdf.

Linuxcare Levanta: Simplifying Server Consolidation with Linux on z/VM. Linuxcare White Paper, 01 August 2002.

Meta Group Consulting ERP Platform-Related Analysis Total Cost of Ownership Study. A Platform-Related Cost Analysis of ERP Applications on-Going Support Costs in the Mid-Tier. 11 February 2000. Available on Internet: http://www.sun.com/servers/workgroup/tco/meta_study.pdf.

NTP Software Do You Need a Storage Management Policy? NTP Software White Paper, Rev. 2.1, August 2001. Available on Internet: http://www.ntpsoftware.com/WhitePapers/docs/DoYouNeedAStoragePolicy.pdf.

OPEN NMS The Latest Released Version of OpenNMS is 1.0.1.Homepage of OpenNMS. Available on Internet: http://www.opennms.org/.

Pasch, Eberhard Optimizing with gcc on Linux for S/390. IBM Article, Oct 2001. Available on Internet: http://www.ibm.com/servers/esdd/articles/gcc_opt/.

Van Vleck, Tom UNIX and Multics. 05 February 1995. Available on Internet: http://www.multicians.org/unix.html.

Varian, Melinda VM and the VM Community: Past, Present, and Future. SHARE 89, Sessions 9059-9061, August 1997. Available on Internet: http://pucc.princeton.edu/melinda/25paper.pdf.