Linux Troubleshooting Bible

Christopher Negus and
Thomas Weeks
Published by Wiley Publishing, Inc.
10475 Crosspoint Boulevard,
Indianapolis , IN 46256
www.wiley.com

Copyright © 2004 by Wiley Publishing, Inc., Indianapolis, Indiana

Published simultaneously in Canada

ISBN: 0-7645-6997-X

Manufactured in the United States of America

10 9 8 7 6 5 4 3 2

No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning or otherwise, except as permitted under Sections 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 646-8600. Requests to the Publisher for permission should be addressed to the Legal Department, Wiley Publishing, Inc., 10475 Crosspoint Blvd., Indianapolis, IN 46256, (317) 572-3447, fax (317) 572-4447, E-Mail: permcoordinator@wiley.com.

LIMIT OF LIABILITY/DISCLAIMER OF WARRANTY: THE PUBLISHER AND THE AUTHOR MAKE NO REPRESENTATIONS OR WARRANTIES WITH RESPECT TO THE ACCURACY OR COMPLETENESS OF THE CONTENTS OF THIS WORK AND SPECIFICALLY DISCLAIM ALL WARRANTIES, INCLUDING WITHOUT LIMITATION WARRANTIES OF FITNESS FOR A PARTICULAR PURPOSE. NO WARRANTY MAY BE CREATED OR EXTENDED BY SALES OR PROMOTIONAL MATERIALS. THE ADVICE AND STRATEGIES CONTAINED HEREIN MAY NOT BE SUITABLE FOR EVERY SITUATION. THIS WORK IS SOLD WITH THE UNDERSTANDING THAT THE PUBLISHER IS NOT ENGAGED IN RENDERING LEGAL, ACCOUNTING, OR OTHER PROFESSIONAL SERVICES. IF PROFESSIONAL ASSISTANCE IS REQUIRED, THE SERVICES OF A COMPETENT PROFESSIONAL PERSON SHOULD BE SOUGHT. NEITHER THE PUBLISHER NOR THE AUTHOR SHALL BE LIABLE FOR DAMAGES ARISING HEREFROM. THE FACT THAT AN ORGANIZATION OR WEBSITE IS REFERRED TO IN THIS WORK AS A CITATION AND/OR A POTENTIAL SOURCE OF FURTHER INFORMATION DOES NOT MEAN THAT THE AUTHOR OR THE PUBLISHER ENDORSES THE INFORMATION THE ORGANIZATION OR WEBSITE MAY PROVIDE OR RECOMMENDATIONS IT MAY MAKE. FURTHER, READERS SHOULD BE AWARE THAT INTERNET WEBSITES LISTED IN THIS WORK MAY HAVE CHANGED OR DISAPPEARED BETWEEN WHEN THIS WORK WAS WRITTEN AND WHEN IT IS READ.

For general information on our other products and services or to obtain technical support, please contact our Customer Care Department within the U.S. at (800) 762-2974, outside the U.S. at (317) 572-3993 or fax (317) 572-4002.

Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books.

Library of Congress Cataloging-in-Publication Data available upon request.

Trademarks: Wiley, the Wiley Publishing logo and related trade dress are trademarks or registered trademarks of John Wiley & Sons, Inc., and/or its affiliates, in the United States and other countries, and may not be used without written permission. Red Hat and Fedora are trademarks of Red Hat, Inc. Used by permission. Linux is a registered trademark of Linus Torvalds. All other trademarks are the property of their respective owners. Wiley Publishing, Inc., is not associated with any product or vendor mentioned in this book.

About the Authors

Christopher Negus is the author of Red Hat Linux Bible (all editions), Linux Toys, and dozens of other books on Linux and UNIX systems. Chris began his career writing about UNIX systems at AT& T more than two decades ago. His work with AT& T included an 8-year run with Bell Laboratories and UNIX System Laboratories, where he worked directly with the developers of the UNIX System V operating system. Later, Chris followed the UNIX source code to Novell, Inc., in Utah, where he helped develop UnixWare documentation and wrote several books on UNIX and UnixWare.

Thomas ("Tweeks") Weeks holds a BS-EET/Telecom degree from Texas A& M, has worked for several large government and IT/security contractors in the positions of Test and Integration lab coordinator and general sysadmin, sysadmin technical trainer and course developer, as well as electrical and systems engineer. He has been working with Rackspace Managed Hosting since 1999 in the roles of Sys-Admin, Corporate Technical Trainer, and has acted as liaison between customer support/security/product/engineering departments. Tweeks has also been president of the San Antonio user group, X-otic Computer Systems of San Antonio (xcssa.org) since 1996.

Credits

Acquisitions Editor
Debra Williams Cauley

Development Editor
Sara Shlaer

Production Editor
Eric Newman

Technical Editor
François Caen

Contributing Author
Jesse Keating

Editorial Manager
Mary Beth Wakefield

Vice President & Executive Group Publisher
Richard Swadley

Vice President and Executive Publisher
Bob Ipsen

Vice President and Publisher
Joseph B. Wikert

Executive Editorial Director
Mary Bednarek

Permissions Editor
Laura Moss

Media Development Specialist
Kit Malone

As always, I dedicate this book to my wife, Sheree.

-C.N.

To my loving and supporting wife, Teri, who likes firewalls, especially green ones. Without her, I would be an incomplete geek.

-T.W.

Acknowledgments

Chris Negus: With the change from Red Hat Linux to Fedora Core and a rampant feeling that those using free Linux systems weren't going to get any official support if things went wrong, we wanted to get a Linux troubleshooting book out to you quickly. To that end, we asked a lot of people to work really hard in a short period of time.

I was thrilled when Thomas Weeks agreed to coauthor this book. His years of experience working with and training the people who support thousands of Red Hat Linux systems at Rackspace Managed Hosting (www.rackspace.com) have added incredible depth to the content of this book. I'd like to thank him for the long hours, late nights, and weekends he consumed to get this book out to you.

Getting Jesse Keating to write the chapter on upgrading and updating, as well as an appendix on Fedora software repositories, was another coup. As leader of the Fedora Legacy project, there is no better authority than Jesse on getting the critical software updates you need to continue to use free Red Hat Linux and Fedora distributions as reliable operating systems. My thanks to Jesse for the countless hours spent outside his day job to make this book a reality.

Special thanks go to François Caen (spidermaker.com), whose RHCE skills added a wonderful dimension to his technical editing of the book. Rounding out the writing team, Kate and Joe Merlino have shown the value of having technically savvy professional writers on a Linux book project. The Kate and Joe team had to work through large amounts of raw material from Thomas and myself, which I feel they did with great success.

As any Linux book should, I would like to acknowledge Linux kernel developers (kernel.org) and the GNU Project (gnu.org) as major contributors to Linux and the whole open source movement. Likewise, I would like to thank Red Hat, Inc., (www.redhat.com) and members of the Fedora Project (http://fedora.redhat.com) for producing high-quality Linux distributions.

I'd like to thank Debra Williams Cauley at Wiley for suggesting that I do this project, instead of just working on more Linux Toys (www.linuxtoys.net). Thanks to Sara Shlaer for keeping us on track with a tight schedule. Thanks to Margot Maley Hutchison at Waterside Productions for contracting the book with Wiley and the other writers on this project.

And finally, thanks to my wife, Sheree, for being my rock, and Caleb and Seth for being the best boys a dad could ask for.

Thomas Weeks: I am honored to have been asked by Chris to participate in this project. He's a great resource to the Linux community, and he and I have really come to know each other well during this frenzied project. Technical Editor François Caen and I used to work together at Rackspace Managed Hosting and have been good friends for years. Between our long hours of work during the early years at Rackspace, I would try to get his wife to let him come to my ubergeeky LUG-ish user group here in town: X-otic Computer Systems of San Antonio (xcssa.org). But now that François has settled into the Tacoma, Washington, area, he has become heavily involved with his own local LUG there, taclug.org, and has been elected president!

My point here is that I am a strong believer in that people who use and love Open Source products (and specifically GPL products) need to contribute back to this planet-wide, utopian, software development model that we've built in whatever form they can. If you have programming skills, get involved by joining an Open Source/GPL-based project development group that interests you (sourceforge.net). For others, you might just join and support your local user group, assist at Linux Install-Fests, College LinuxFest events, or work for an incredible company that believes in and supports Open Source, such as Rackspace.

I could not have done this book without all the others that did their part. I would like to thank Chris for having the faith (or sheer workload-induced stupor) to ask for my help on this impossibly quick, monstrous undertaking. Working with him and Francois, both on the book and on helping to host the Linux Toys site (linuxtoys.net) community effort has been a lot of fun. I'm very glad that our paths all crossed when they did.

Also big thanks go to Joe Merlino and Kate Wrightson for spending countless hours trying to interpret and mold my raw research into something usable, to Sara for listening to my mail list whining about timelines, and to Debra for all her work in getting all this contract material worked out for me.

I would like to thank Rackspace-working there is the most fun I've ever had while getting paid! Plus it's such a super-rich learning environment to boot! It's been a wild ride since 1999. Thanks Rackspace, for allowing us to get some of our "in the trenches" expertise from the Fanatical Support™ Department in this book. This is a great way to give back to the community! Dirk, Pat, Richard-keep on "doing the needful" guys.

I would also be remiss if I didn't thank my parents and my Creator. Mom, Dad, thank you for bringing me up, keeping me engaged, challenged, and always supporting me. Thank you, God, for carrying me though the rough parts. It's to you I give all glory.

Most of all, I want to thank my wife, Teri, for giving up all the evenings and weekends that it took me to complete my portion of this work. Teri, I love you, and yes, I would marry you all over again.

Team LiB


Preface

Simplified install screens, rich desktop interfaces, and GUI administration tools have made Linux an easy operating system to use... unless something goes wrong.

If installation fails, you might find yourself figuring out what kernel options to add at an installation boot prompt. If you check a Linux forum with a server problem, most times your friends will push aside your graphical tools and have you type shell commands and hand-edit configuration file.

With official support options disappearing for Red Hat Linux, and Red Hat's free Fedora Project offered with no long-term official Red Hat support, some people continuing to use free Linux distributions are feeling left out in the cold. If you are devoted to the idea of using free Linux distributions that don't have commercial support options, you have another option-learn to troubleshoot Linux yourself.

Getting in Deeper

Troubleshooting is an often-neglected area of system administration. After books and manuals have gone through the steps to configure and use a feature, usually they've filled up their chapter and are on to the next topic. So what happens if you get to the end of a setup procedure and the feature doesn't work?

Linux Troubleshooting Bible was created to help you troubleshoot failures that you may encounter as you use Linux. For most features covered in Linux Troubleshooting Bible, we take you through some quick, basic steps of setting up that feature. In many cases, running the basic setup will uncover some step you forgot to do. For more details on basic uses of the feature, we point you to books (such as the latest edition of Red Hat Fedora Linux Bible), mailing lists, and websites.

After you have done what should have worked, and found that it didn't work, we tell you how to go a bit deeper. For example, nearly every type of Linux service outputs messages to log files. Daemon processes have verbose or debug modes to spew out reams of messages for tracing down a problem. There are commands that come with many software packages that let you check the status of a feature and see what is broken. The trick is to know where the troubleshooting tools are and how to use them.


Troubleshooting Fedora and Red Hat Linux

We chose Fedora Core 1 as the Linux operating system to illustrate Linux troubleshooting. We go through some of the reasoning behind this choice in more depth in Chapter 1. Basically, however, we chose Fedora because we felt it was the best free Linux you could use to learn troubleshooting techniques that would span from home desktop systems to corporate enterprise computers.

Despite many people's fear that Fedora Linux might lack quality without official long-term support from Red Hat, Inc., we found the first release of Fedora (Fedora Core 1) to be quite reliable. There is also a strong (and growing) community to support Fedora Linux distributions.

If your goal is to be a professional Linux system administrator, learning Fedora will help you learn skills that are immediately transferable to Red Hat Linux (versions 8 and 9 in particular) as well as Red Hat Enterprise Linux (version 3 in particular). All of those operating systems use the same installer (Anaconda), graphical administration tools, and desktops (bluecurve theme on KDE or GNOME desktop environments).

You may want to build a business around providing computers installed with Fedora distributions to small businesses, schools, or nonprofit organizations, as many people are doing today. Or, you may want to learn skills that will transfer easily to large Red Hat Enterprise Linux installations. By going through the procedures and learning the tips in this book, you will learn tools and techniques to help you trace down and repair problems with Linux, as well as learn how to prevent problems before they occur.


Transitioning to SUSE or Debian Linux

You need a Fedora or other Red Hat Linux system in front of you to make best use of the procedures and tips in this book. However, given the turmoil surrounding Red Hat's changes to the name, support options, documentation, and training for its free Linux system, we recognize that some previously faithful Red Hat supporters are now exploring other Linux systems.

Whether you plan to drop Red Hat Linux distributions altogether or simply have requirements (or desire) to use other Linuxes, we want to help transition your troubleshooting skills to those Linux systems. In particular, we want to help you understand the challenges you will face moving to SUSE and Debian Linux systems.

SUSE Linux

SUSE is an obvious alternative to Red Hat Linux distributions for those with an eye toward the enterprise. Since the acquisition of SUSE Linux by Novell, Inc., in January 2004, SUSE has become hooked in with the strong, established international sales, marketing, and development resources of Novell. SUSE is also RPM-based, so the tools for managing software packages are the same as those in Fedora and other Red Hat Linux systems.

As a Red Hat Linux troubleshooter transitioning your skills to SUSE, you will need to learn about differences in software packaging, system administration and installation tools (provided by the YaST facility in SUSE), and licensing and support issues. These and other features that you need to understand to transition from Fedora and Red Hat troubleshooting practices to SUSE troubleshooting are covered in Appendix B.

Debian Linux

When Red Hat, Inc., dropped the official Red Hat Linux product, Debian Linux was one of the first distributions that people evaluated as an alternative to Red Hat,-particularly for small business servers. The Debian "stable" distribution (with the code name Woody), has a reputation for being rock solid, if slightly slow in implementing the latest Linux software.

Despite its stability, however, some consider Debian to be less suited for mission-critical applications because it doesn't have big-company support behind it (such as Red Hat and Novell). Installation is not as simplified as you will find in Red Hat distributions. Likewise, hardware detection can require more manual activity than it does in Red Hat. On the whole, Debian has a stronger draw for people who are more technically inclined. You won't find as many books or support options available with Debian as you will with Red Hat systems-but that's okay, since most Debian users prefer to read man pages anyway.

Appendix C covers Debian troubleshooting issues that will help you understand what you will face if you want to transition your Fedora and Red Hat Linux troubleshooting skills to Debian.


Using Linux Troubleshooting Bible

Our primary goal with Linux Troubleshooting Bible is to give you the resources you need to overcome any problems you might have using and administering Fedora Linux (primarily), as well as other Red Hat Linux systems. We take two approaches to attaining this goal:

  • Teaching you how to troubleshoot Fedora on your own

  • Getting you in touch with other Fedora resources

By focusing mostly on command-line tools and text-based configuration files, you'll learn basic troubleshooting techniques that will apply to most Linux distributions. For example, whether you use an Apache web server from Fedora, RHEL, SUSE, or another Linux distribution, the httpd.conf file will still be the basic file you will troubleshoot if something goes wrong with your web server.

For the features we cover, we try to give you multiple commands and options to try out. Because we know we can't cover every issue that might come up, for most subjects we will point you to mailing lists, FAQs, or other online resources you can turn to for more information.

Approaching Linux Troubleshooting

You can use this book as a reference (check it when you have a problem) or a guide (step through the procedure, to learn how to troubleshoot a feature). We have divided the book into six major parts.

  • Part I : Getting Going-Part I contains the most Fedora-specific information in the book. In addition to an introductory chapter (Chapter 1), the two other chapters in this part describe how to troubleshoot problems installing Fedora Core 1 (Chapter 2) and how to get the updates and do upgrades you need to keep Fedora patched and secure (Chapter 3).

  • Part II : Preventing Problems-The two chapters in this part focus on how to prevent problems from happening in the first place. Chapter 4 describes how to lock down both desktop and server Fedora systems before you deploy them. Chapter 5 focuses on how to prepare good backup policies to protect your data.

  • Part III : Troubleshooting Basic Features-Many features that apply to troubleshooting both desktop and server systems are covered in this part. Basic tips and tools that can be used to troubleshoot many different Linux features are described in Chapter 6. Chapter 7 tells how to troubleshoot your video, mouse, and keyboard. Tools for managing your software packages and correcting problems are described in Chapter 8. Troubleshooting disks and file systems is covered in Chapter 9.

  • Part IV : Troubleshooting the Network-Techniques for making your network secure and keep it running smoothly are contained in this part. Security issues are covered in Chapter 10. Chapter 11 tells how to troubleshoot your firewalls. Chapter 12 describes how to troubleshoot problems with serving domain names from a DNS server. Chapter 13 covers how to troubleshoot modem problems.

  • Part V : Troubleshooting Internal Services-Network services that are typically offered behind a company firewall are covered in this part. The part includes chapters for troubleshooting printers (Chapter 14), Samba (Chapter 15), and NFS (Chapter 16).

  • Part VI : Troubleshooting External Services-Popular services that are often made available on public networks are covered in this part. Troubleshooting techniques are described for web servers (Chapter 17), file transfer servers, such as FTP (Chapter 18), and e-mail servers (Chapter 19).

In addition, the book contains three appendixes.

  • Appendix A : Fedora Software Repositories describes several repositories for Fedora packages.

  • Appendix B : Troubleshooting SUSE Linux is intended to help transition the Fedora and Red Hat Linux troubleshooting techniques learned in this book to SUSE Linux systems.

  • Appendix C : Troubleshooting Debian/GNU Linux aims at helping Fedora troubleshooters transition their skills to Debian Linux.

Many of the chapters in Linux Troubleshooting Bible will cover the same basic types of information. This information includes the following:

  • Overview-Provides an overview of the feature being covered in the chapter. This often includes a quick review of the steps needed to get each basic feature running.

  • Procedures-Provides step-by-step procedures for walking through techniques for troubleshooting a feature. When you are done, you should have a good sense of how to feel your way around a feature and look for potential problems.

  • Tools-Provides additional commands, options, and interfaces for working with a particular feature. Once you understand the basic tools needed to work with a feature, you should be able to refer to man pages and info pages to get more options you can use with those tools.

  • Tips-Provides tips for troubleshooting some of the most common problems you might encounter.

  • References-Provides refererence information to help you find things like the location of mailing lists, project home pages, and FAQS for the feature being described.

If you find that you are stuck on a problem, you might try one of the many Fedora and Red Hat Linux online forums listed in Chapter 1 for further information.

Conventions Used in this Book

Throughout the book, a special typeface indicates code or commands. Commands and code are shown in monospace type:

   This is how code looks.

In the event that an example includes both input and output, monospace is still used, but input is presented in bold type to distinguish the two. Here is an example:

   $  ftp ftp.linuxtoys.net
   Name (home:jake):  jake
   Password:  ******

The following special mark-up is used to call your attention to points that are particularly important.

Note

A Note provides extra information to which you need to pay special attention.
Tip

A Tip shows a special way of performing a particular task, or introduces a shortcut to ease your way.
Caution

A Caution alerts you to take special care when executing a procedure, or damage to your computer hardware or software could result.
Cross-Reference

A Cross-Reference refers you to further information on a subject that you can find outside the current chapter.


Reach Out

The FedoraTrouble.com website contains updates and errata information for Linux Troubleshooting Bible. You can also go to that site to download some of the code examples that are contained in this book.

If you have any questions or comments about this book, feel free to contact Chris by e-mail at chris@linuxtoys.net. Thomas (Tweeks) can be reached at tweeks-bible@theweeks.org.


Part I: Getting Going

Chapter List

Chapter 1: Introduction to Linux Troubleshooting
Chapter 2: Troubleshooting Installation
Chapter 3: Updating and Upgrading Fedora


Chapter 1: Introduction to Linux Troubleshooting

Overview

Linux troubleshooting skills that were once useful only for hackers and ubergeeks have become necessities for computer professionals and serious technology enthusiasts. Once confined to the computers of free software proponents, Linux systems have found their way into desktop computers, small office servers, hand-held computers, enterprise operations, and anywhere else computer processors might be found.

If you can troubleshoot Linux, demand for your skills will grow exponentially in the time to come.

Despite the fact that there are now scores of different Linux systems, the core components of those systems are still strikingly similar. The kernel, hundreds of basic commands, and the approach you need to take to track down problems are very much the same among most Linux systems.

The goals of Linux Troubleshooting Bible are to help you:

  • Set up and fix Linux desktop, laptop, workstation, and server systems.

  • Develop Linux troubleshooting expertise that can easily transition to many different Linux systems.

To provide detailed, working instructions to learn Linux troubleshooting, we chose Fedora Core 1 as the example Linux distribution for this book. In fact, we are strong proponents of Fedora for those who want to:

  • Become proficient in a free Linux distribution that can be used in the same ways that Red Hat Linux was traditionally used.

  • Learn how to dig up resources that can help you maintain Linux systems yourself, without paying expensive subscription fees.

  • Develop a set of troubleshooting skills that will transition up to enterprise-quality Linux distributions.

Most of the techniques described in this book will work exactly the same on Red Hat Enterprise and Red Hat Linux (versions 8 and 9, in particular) as well. In the next section, we explain why we believe Fedora is an excellent choice for both learning Linux troubleshooting and for using as desktop and server systems, in many situations.

Once you have studied the troubleshooting techniques in chapters of this book, you may find that you want to spread those skills to other Linux distributions. For that reason, we provide two appendixes (Appendixes B and C) to help you understand the challenges you will face transitioning to two other popular Linux distributions: SUSE and Debian.

For the moment, however, I recommend that you have a Fedora Core system in front of you (or a Red Hat Enterprise Linux system) if you care to step through the information laid out in this book.


Introducing Fedora Linux

In September 2003, while in the middle of developing the successor to Red Hat Linux 9 (presumably Red Hat Linux 10), Red Hat, Inc., made a startling announcement. It was merging its Red Hat Linux development with Fedora Linux (www.fedora.us) into a new initiative: The Fedora Project.

Suddenly, it seemed, there was no flagship Red Hat Linux product anymore. Users would have to choose between the subscription-based Red Hat Enterprise Linux and the community-supported Fedora Project. The Red Hat Linux product in development was renamed and eventually released as Fedora Core 1.

Common reactions to the abandonment of Red Hat Linux for Fedora ranged from surprise, to confusion, to anger. As the dust settles, however, the logic of this move and the potential upside for the open-source community has already begun to emerge.

Learning Linux Troubleshooting with Fedora

The Fedora Project is being touted as a Red Hat-sponsored and community-supported open-source project. Its mission is to produce a Linux distribution called Fedora Core (sometimes also referred to as Fedora Linux), which in its first release was built from the Red Hat Linux code base. The distribution would be re-released every 4 to 6 months.

With the name change and its association with Red Hat, Inc., changing so suddenly, there has been a lot of speculation about what exactly Fedora is and how well it can be relied upon. Let's start by separating the facts from the speculation about Fedora:

  • Fedora is essentially Red Hat Linux 10-A good case can be made for this statement. Up until the last 2 months of the development process, what is now called Fedora Core 1 was being developed as the next release of Red Hat Linux. Most of the last-minute changes had to do with changing logos and expanding access to software repositories (which we will discuss later).

  • Fedora Core 1 is a solid Linux system-Hundreds of bug fixes and improvements were made during the development process. There were not a tremendous number of new features added to the Red Hat Linux 9 base. By most accounts, Fedora Core 1 is a rock-solid Linux distribution.

  • Red Hat, Inc., supports Fedora-It is clear that Red Hat, Inc., wants to create high-quality software from the Fedora Project. Most of the technology in Red Hat Enterprise Linux 3 matches almost exactly the same software packages in Fedora Core 1. Red Hat, Inc., plans to use Fedora as a proving ground for its enterprise products. It doesn't have the resources to maintain and enhance the entire Linux operating system and related applications itself. Fedora is critical to Red Hat's success.

  • Fedora is a path to Red Hat Enterprise-Some people like to use a freely distributed Linux as a means of showing a reluctant boss or IT department that Linux will work in their business. With Fedora Core 1, you can demonstrate many of the same features that are in Red Hat Enterprise Linux 3. Future releases of Fedora could be used to evaluate new technologies in which a company may be interested. (See the description of Fedora Core 2 later in this chapter.)

  • Fedora offers more software-For someone interested in trying software that is outside of the corporate software model (such as games, audio, and video), Fedora Core offers dozens more packages than does Red Hat Enterprise Linux. It also contains tools, such as yum, for downloading complete sets of applications from software repositories in Red Hat RPM format.

  • Fedora has the latest technology-Here's where some of the risks (and opportunities) lie with Fedora. Because Fedora is slated to come out two to three times per year, it can incorporate the latest software available for Linux. This should make Fedora Core a great operating system for keeping your knowledge and troubleshooting skills up to date. However, this should also have the effect of making some releases of Fedora less stable. You might find yourself picking and choosing a Fedora Core release to use as your server.

  • Fedora offers critical updates and patches-This has been the biggest concern for those who have been using Red Hat Linux as a server for their businesses or organizations. The amount of time that critical updates will be officially supported by Red Hat for Fedora Core is shorter than it was for Red Hat Linux. The Fedora Legacy Project (www.fedoralegacy.org), as well as other organizations, is stepping up to deal with this issue. (See Chapter 3 for discussions of how to solve the update and upgrade issues with Fedora.)

The bottom line is that Fedora offers great technology in a freely distributed Linux system. Yes, there are fewer guarantees from Red Hat, Inc., if you want to bet your business on Fedora. However, if the prospect of going it alone with Fedora is exciting but somewhat daunting to you, we are here to tell you that you don't have to go it alone. If you decide to go with Fedora (or at least look into it further), Linux Troubleshooting Bible is here to give you (or help you find) the resources you need to support it.

Finding Opportunities with Fedora

If you take the time to learn how to troubleshoot Fedora, what kinds of Linux systems can you support with your new skills? The short answer is: any kind of system you could set up with Red Hat Linux. The software in Fedora supports desktop, workstation, and server systems, and is being used for a variety of specialty uses.

Fedora Linux Desktop

As with most Linux and UNIX systems, Fedora Core uses the X Window System as the foundation for its graphical user interface (GUI). It also offers the GNOME and KDE desktop environments. If you don't want to use a full-blown desktop environment, you can use other window managers that come with Fedora instead, such as the Tab Window manager and Motif Window manager.

Figure 1-1 illustrates the GNOME desktop, including two panels, the Nautilus graphical shell and the Konqueror web browser/file manager.

Click To expand
Figure 1-1: Choose between GNOME and KDE desktops, based on the X Window System.

Fedora comes with all the basic desktop applications you would expect to find on a Linux desktop: editors, web browsers, office productivity applications, music players, and graphics programs. Using yum and apt repositories, you can add to the hundreds of desktop applications that come with Fedora Core.

Whether you are supporting dozens of desktop systems or just a few, this book covers how to troubleshoot installation (Chapter 2), your video card, mouse, and keyboard (Chapter 7), and software packages (Chapter 8). Chapter 4 tells how to lock down a Fedora desktop system, and Chapter 3 describes how to get critical updates for your desktop system.

Fedora Linux Servers

All the basic server features that came with Red Hat Linux are in Fedora Core. Graphical configuration tools developed for Red Hat Linux are included in Fedora Core, as are a variety of commands and graphical utilities for troubleshooting your servers.

Figure 1-2 illustrates the categories of server software in Fedora from the Package Management window.

Click To expand
Figure 1-2: Advanced network server software once in Red Hat Linux is included in Fedora.

Despite the lack of guarantees from Red Hat, Inc., for supporting more than a short update period for Fedora systems, a group of people committed to using Fedora as a server have rallied around the Fedora Legacy Project (see Chapter 3 for details). Their goal is to provide the long-term support for Fedora that it needs to be a viable server operating system.

Most of the troubleshooting skills you will gain from this book are aimed at server troubleshooting. While we briefly touch on Red Hat-specific graphical administration tools, techniques for working with server configuration files and command-line tools will help you develop a skill set that applies to most Linux and UNIX systems.

For small office servers and personal servers, you can probably use Fedora Core. For larger installations, you might consider purchasing Red Hat Enterprise Linux server subscriptions. In either case, the skills for troubleshooting both distributions will be the same.

In terms of how server topics are covered in this book, we have divided the chapters into three major sections. The chapters in Part IV focus on how to troubleshoot your network with an eye toward security (Chapter 10 covers how to detect and deal with intruders), and then describe specific techniques to troubleshoot your firewalls (Chapter 11), domain name system server (Chapter 12), and modems (Chapter 13).

Fedora Linux Workstation

A Linux workstation is basically thought of as a desktop system used for software development. There are well over 100 software packages that come with Fedora Core that fall under the heading of development packages (see Figure 1-3 for an illustration of software development package groups in Fedora).

Click To expand
Figure 1-3: Software development tools are spread across more than 100 software packages in Fedora.

Troubleshooting skills needed to support Fedora Linux workstations are pretty much the same skills you would need to support any desktop system.

Specialty Fedora Linux Systems

Because there are no licensing fees associated with Fedora Core, people have already begun using it to create their own specialized Linux systems. For example, bootable live CD Linux distributions created from Fedora include ADIOS Linux Boot CD (http://dc.qut.edu.au/adios) and RPM Live Linux CD (http://nwst.de/livelinuxcd).

Many companies creating devices that use embedded Linux systems have leaned toward Red Hat Linux in the past. According to the 2003 LinuxDevices.com Embedded Linux Market Survey, Red Hat Linux was the preferred Linux source/vendor for embedded Linux systems. With 14.4 percent of embedded Linux systems being based on Red Hat Linux (with only homegrown Linux systems beating it, with an 18 percent share), Red Hat Linux captured a substantial share of the embedded systems market. The fact that most surveyed (63.9 percent) considered no royalties as being the most important feature for choosing a Linux distribution for creating their embedded systems means that we can expect Fedora to carry on that tradition. Whether you hope to build your own embedded or other specialty Linux system some day, the skills you learn in Linux Troubleshooting Bible will help you not only get your systems working, but understand the range of features that lie beneath the surface of Linux. Even if it's just in the area of fun projects, such as those included with Linux Toys (www.linuxtoys.net), learning to troubleshoot Fedora can be a useful skill. Figure 1-4 illustrates the Linux Toys Television Recorder/Player.

Click To expand
Figure 1-4: Linux Toys and other hobbyist applications will run on Fedora Core.


Going with Fedora (Without Going it Alone)

What some seem to forget as they worry about how well Red Hat, Inc., will support Fedora is that Linux was originally created and supported by a community of open-source developers, administrators, and integrators. Right now, the community of people and companies out there that are ready and able to support the software that comes in Fedora is as strong as ever.

While calling Red Hat, Inc., for every problem with Fedora may not be an option, the trick is to know what the Fedora Project itself is offering (with the support of Red Hat, Inc.) and how to look elsewhere for the help you need. The first thing to do, however, is to get your hands on Fedora.

Choosing a Fedora Version

By the time this book is released, Fedora Core 1 would have been available for several months and Fedora Core 2 would just be hitting the market. Although versions 1 and 2 are similar in many ways, we based most of this book on Fedora Core 1 for the following reasons:

  • Fedora Core 1 had gone through nearly a full Red Hat testing cycle before becoming Fedora Core 1. There were not major feature additions over Red Hat Linux 9, while there were many bug fixes that were included.

  • Fedora Core 2 included many major new features and hadn't been out long enough to know if all the major bugs had been fully shaken out yet.

For these reasons, we felt that Fedora Core 1 made a more stable environment for those who wanted to use the skill taught in this book to set up and use reliable desktop and server systems. If, however, you want to use Fedora Core 2 instead with this book, here are some of the differences you can expect:

  • Fedora Core 2 includes the new 2.6 kernel by default. While there are not many features about the new kernel that change how you use Linux, it's possible that support for some hardware drivers may not have been carried over to the 2.6 kernel yet. The new kernel also has brought some changes in the types of boot media you can use with Fedora Core 2. (Refer to http://fedora.redhat.com/download for details on installation media and install procedures.)

  • All the redhat-config-* graphical administration utilities had their names changed to system-config-*. So, anywhere in the book that we discuss the commands that launch graphical administration tools, you should change redhat to system when you type the command.

  • The XFree86 X server used in Fedora Core 1 has been replaced by a new X server from X.org. During descriptions of troubleshooting video in Chapter 7, you may find differences in how some of the X-related commands behave.

Likewise, in the course of normal improvements to Linux software (such as new versions of KDE and GNOME desktops in Fedora Core 2), some of the steps and options described in this book (which was based on Fedora Core 1) may differ.

Getting Fedora

You need Fedora Core Linux operating system to use this book. When Red Hat, Inc., transitioned its operating system from Red Hat Linux to Fedora, it stopped producing boxed sets of that product. To follow along with this book, you should get the three-CD Fedora Core 1 set in one of the following ways:

  • Red Hat Linux Bible: Fedora and Enterprise Edition-The complete three-CD installation set for Fedora Core 1 comes with this edition of Red Hat Linux Bible. If you are new to Red Hat Linux, Fedora, or Linux in general, Red Hat Linux Bible covers the basics you need to use, administer, and set up servers for Fedora Core. Linux Troubleshooting Bible builds on those basics.

  • Downloading-You can download the ISO images of the CDs from the Fedora Project website (http://fedora.redhat.com/download/). The site describes the process of downloading, verifying, and burning the CD images.

  • Find a friend-Many Linux enthusiasts or Linux user groups will be happy to get you a copy of Fedora Core, or any other Linux that you desire. Remember, Fedora is free and distributable.

Fedora Core can be installed on most Pentium class PCs. Unless otherwise noted, all the software needed in this book is available on the Fedora distribution CDs.

Note

Chapter 2 contains information on installing Fedora Core and tips on troubleshooting installation.

To learn about what is new in Fedora Core, especially if you want to know the differences between Fedora Core and Red Hat Linux 9, you can refer to the Release Notes document. That document describes the hardware requirements for running Fedora, provides notes on installing and using Fedora Core, and lists new and removed software packages. You can get the Release Notes on the first disk of the three-CD Fedora Core set or from the Fedora Project website (http://fedora.redhat.com/docs/release-notes).

Connecting with the Fedora Project

The best resource for the latest news and policies relating to Fedora is the Fedora Project website: http://fedora.redhat.com. From that site, you can get the latest information on release schedules, project objectives, and ways of participating in the project.

You should read the Fedora Project FAQ (click the FAQ link from the home page). You can also click the About link to learn about the people and structure that supports the project.

Getting Fedora Documentation

The Red Hat Linux documentation is one component that didn't come across to the Fedora Project. In fact, one of the major reasons for this book is to fill the void left by the lack of official documentation coming out of Red Hat, Inc., for Fedora.

Instead of porting the Red Hat Linux Documentation to Fedora, the Fedora Docs project is focusing its first efforts on producing an Installation Guide. Beyond that, the Fedora Docs project hopes to produce (and encourage others to produce) smaller tutorial and HOWTO documents.

Because the first Fedora release (Fedora Core 1) was very similar in many ways to the previous release (Red Hat Linux 9) and the recent enterprise release (Red Hat Enterprise Linux 3), you can find answers to many questions in existing Red Hat documentation for those products. Another resource for Fedora documentation is documents that come with each software package in the Fedora distribution itself. Here are some places to look:

  • Red Hat Linux 9 Doc Set-Manuals for installing, customizing, and administering Red Hat Linux 9 are on the Red Hat website (www.redhat.com/docs/manuals/linux). You can download the Installation Guide, Getting Started Guide, Customization Guide, Reference Guide, Security Guide, System Administration Primer, Glossary, and Release Notes. Most of the documents are available in PDF, RPM, and HTML (tarball) formats.

  • Red Hat Enterprise Linux Doc Set-Manuals for the Red Hat Enterprise Linux products are also available online (www.redhat.com/docs/manuals/enterprise). Several of these manuals are quite generic. So most of the content in, for example, Introduction to System Administration, System Administration Guide, Reference Guide, Security Guide, and Step by Step Guide would be appropriate for Fedora as well.

  • /usr/share/doc-Software packages that come with documentation often put that documentation in subdirectories of the /usr/share/doc directory. Much of the documentation is in HTML format, and so you can read it from your web browser.

  • Man pages-Traditional UNIX and Linux documentation was done on per component documents referred to as man pages. For a Fedora troubleshooter, man pages are a great resource for understanding all the options that come with each command. Man pages are typically stored in the /usr/share/man subdirectories for commands (man1), system calls (man2), programming functions (man3), special files (man4), file formats (man5), games (man6), miscellaneous components (man7), and administration commands (man8).

  • Info pages-Like man pages, info pages are text-based descriptions of components in Fedora. However, the info utility, which displays info pages, is more menu-oriented than are man pages.

Checking Documents with Man and Info

The man and info commands are used to display man pages and info pages, respectively. For example, to see the man page for the cat command, type

   #   man cat

To start the info utility, you can simply type the info command. From there, scroll up and down to find the command or other component that interests you. With the cursor on the item you want, press Enter to display the information.

Finding Documents in Software Packages

If you would like to see what documentation is included with a particular software package, you can use the rpm command. For example, to see what documentation comes with the bash shell software (bash package), you could type the following:

   #  rpm -qd bash |less

This command will show you any manual pages, HOWTOs, release notes, or other documents associated with a Fedora software package.

Checking Project Sites

Nearly every project represented by software packages in Fedora has its own website, at which you can find further information and documentation on the project. You can usually find the address of a project's website by checking the README file or the RPM description that comes with the software package in Fedora.

For example, to find out about the Rhythmbox multimedia player in Fedora, I paged through the Rhythmbox README file:

   # less /usr/share/doc/rhythmbox-*/README

In that file, I saw that the official website is http://web.rhythmbox.org and that there is a mailing list at http://lists.gnome.org/mailman/listinfo/rhythmbox-devel. If you can't find information about the project's website from its README file, you can try listing information about the package to find the site. For example, to list information about the rdesktop package (a Windows NT terminal server client), you could type the following:

   #  rpm -qi rdesktop

Besides showing a description of the package, this command showed that the rdesktop website is at www.rdesktop.org. Links from this site point you to documents that describe the protocols used in rdesktop, if you want to research further how the software works.

Getting on Fedora Mailing Lists

Many of the resources slated to support the Fedora Project were created to suit its charter to be a community-supported Linux. While Fedora mailing lists have been churning away happily from the get-go, community involvement in other areas will, by their nature, take more time to develop.

The Fedora Project has an active set of mailing lists that are available through Red Hat, Inc. To access the mailing lists (both archives and e-mailed versions are available to everyone), visit the following site:

  • http://redhat.com/mailman/listinfo/

There is very active participation from Red Hat, Inc., developers on all of these lists, so you can often get information straight from those people developing and making decisions about the direction of Fedora. Here's a rundown of what you can expect from each of the Fedora mailing lists:

  • General Fedora questions (fedora-list)-The fedora-list is for general questions about Fedora. Of the available mailing lists, fedora-list is the most active. If you sign up for this list, you can expect 100 to 200 posts per day (or more) devoted to questions about Fedora. Although there are all types of Fedora questions on this list, the list is especially good for "how do I get" and "where do I find" types of questions.

  • Fedora testing discussions (fedora-test-list)-The fedora-test-list tracks the development of Fedora software. For the troubleshooter, this is a good list to talk about software that has broken down and is in the process of being fixed.

  • Fedora announcements (fedora-announce-list)-If you are using Fedora, you should at least subscribe to fedora-announce-list. This list announces the availability of critical updates for your Fedora systems.

  • Fedora desktop (fedora-desktop-list)-For issues related to troubleshooting desktop problems or discussing desktop enhancements, the fedora-desktop-list is a good place to start.

  • Fedora software updates (fedora-legacy-list)-To get in on discussions of how updates will be available for Fedora (and older Red Hat Linux systems as well) beyond the officially supported update plans from Red Hat, Inc., you should join the fedora-legacy-list.

To keep up with the ongoing development of Fedora, there are also other lists that might interest you. Fedora-devel-list contains information on newly built packages and lets people comment on feature changes. For development issues related directly to configuration tools, there is fedora-config-list. If you want to contribute to the Fedora documentation effort, you can join fedora-docs-list.

Getting on Fedora Internet Relay Chat

If you have never used Internet Relay Chat (IRC) before, it offers a great way to participate in (or just watch) discussions by typing messages to a scrolling list. Messages are transferred in real time, so anyone on the IRC channel can participate right at the moment.

The Fedora Project offers several IRC channels through the Freenode IRC Network (www.freenode.net). Channels include:

  • Fedora general chat (#fedora)-Open to end users or any participants in the Fedora Project.

  • Fedora development chat (#fedora-devel)-For chatting about problems and suggestions related to Fedora software development.

  • Fedora problem reporting chat (#fedora-bugs)-To talk about bugs you encounter while using Fedora and find out if workarounds or fixes are available.

There is also a Spanish language chat channel (#fedora-es) and other public and private channels associated with Fedora projects. If you have not used IRC before, refer to an IRC tutorial, such as the one at www.irchelp.org/irchelp/irctutorial.htm.

Here's a quick procedure for getting into a Fedora IRC channel:

  1. With a computer connected to the Internet, open an IRC client. For example, in Fedora you can open an X-Chat window by clicking the Red Hat menu and selecting Internet⇨More Internet Applications⇨IRC Client.

  2. Add a Nick Name (and two alternate nicknames, in case yours is taken), a User Name, and a Real Name, and click Connect.

  3. Once you are connected to the server, join the list you choose by typing /join and the channel name. For example, to join the general Fedora chat channel, type

    / join # fedora

Getting More Software

While the plans to broaden the range of software available for Fedora are still being developed, new features in Fedora already make it easier to get additional software packages for Fedora than it was for Red Hat Linux. Right now, you can:

  • Use up2date-The up2date facility that you used to get software updates for Red Hat Linux has been enhanced for Fedora Core. Beginning with Fedora Core 1, software repositories created for the yum or apt package managers can be accessed through the up2date facility. By default, up2date is configured to use yum repositories to get updates for Fedora systems, while Red Hat Enterprise Linux systems still use the Red Hat Network to get official updates from Red Hat, Inc.

  • Use yum or apt-The yum package manager (which comes with Fedora Core) and apt package manager (which is easily attainable) can be used to get additional software packages that are not in Fedora Core.

High-quality software packages, in RPM format, are already available from yum and apt repositories. The www.fedora.us site (which merged with Red Hat Linux to form the Fedora Project) has software repositories built for Fedora (www.fedora.us/wiki/FedoraPackageLists). The Freshrpms.net site is the first place many Fedora users go to get additional software packages. There are also other repositories that contain more software that may not meet Fedora's or Red Hat's standards of quality (or legality).

There is a Fedora Project Developer's Guide (http://fedora.redhat.com/participate/developers-guide) for those who want to contribute their software to the Fedora Project. Plans for software packages that don't make it into the main Fedora Core distribution, but still want to be sanctioned to some extent by the Fedora Project, can be added to Fedora repositories in one of the following ways:

  • Fedora Extras-These are additional software packages that don't replace any of the basic Fedora Core packages. Eventually, the www.fedora.us site is slated to hold an official set of Fedora Extras packages.

  • Fedora Alternatives-These are software packages that can replace existing software packages that are in the Fedora Core.

  • Fedora Legacy-These are packages that contain fixes to older versions of Fedora Core software.

The guidelines for submitting Fedora software packages, and having them sanctioned by the Fedora Project, are still being developed. For more on software package definitions, visit the Fedora Project Terminology page (http://fedora.redhat.com/participate/terminology.html). As I noted earlier, you should also keep in mind that there are some (and will be more) software repositories that will be available that won't fall under Fedora Project guidelines. You will be on your own to evaluate the quality and legality of those packages.

Cross-Reference

See Chapter 3 for more complete information on getting Fedora software packages and updates. See Appendix A for a list of popular third-party software repositories.


Giving Back to Fedora

Contributing to the open-source community is the best way to ensure that Linux, in general, and Fedora, in particular, will continue to improve and grow in the future. Even if you are not a kernel hacker, or a software developer at all, there are plenty of other ways you can contribute.

Finding Bugs and Filing Reports

The easiest way that anyone using Fedora can help improve it is to report problems. As you use Fedora, you will certainly bump into things that don't exactly work right. Using a facility called Bugzilla, which is maintained by Red Hat, Inc., for the Fedora Project, you can:

  • Search-You can search the Bugzilla database to see if anyone has encountered the same problem you have. If someone has, there may already be a workaround or software fix available to overcome the problem. If you can't find the exact bug you encountered in the database, you might also want to check a Fedora mailing list on the topic to see if the problem has been encountered, before you file a bug report. To search Bugzilla, go to the Red Hat Bugzilla site (http://bugzilla.redhat.com/bugzilla/) and enter a search term.

  • Report-If you feel that the bug has not been reported yet, you can enter a Bugzilla report into the database. When you do, make sure that you have as many details available about the problem as possible before you enter the report.

    To enter a bug report, you need to give a valid e-mail address and password. To get an account, or just log in to enter a bug report, go to the following web address: http://bugzilla.redhat.com/bugzilla/enter_bug.cgi.

    Once you log in, check the frequently reported bug list and check the latest errata updates to see if your bug has already been dealt with. Then select Fedora Core to enter a new bug report by selecting the component (software package), severity, platform, priority, summary, description, and other information about the bug.

    Note

    If you have never entered a bug report before, click the bug writing guidelines link from the Enter New Bug page. Those guidelines will give you good advice on how to pass on the most important information about your bug to help the developers reproduce and fix the problem.

  • Fix-If you believe that you have a code fix or other solution to the problem, it's a good idea to submit that fix along with the bug report. Red Hat, Inc., recommends that most fixes be delivered in the form of unified diffs (so the maintainer can see exactly what code changes you are suggesting).

    After you have entered the bug report, you are given an opportunity to attach a patch or other text to the bug report. Select Create an Attachment and enter the filename of your patch (and a brief summary description) along with the attachment.

I can't stress enough how important it is to enter bug reports to get important fixes done. When a package maintainer goes through the list of fixes needed to a software package, a report entered into Bugzilla will carry a lot more weight than ranting in a mailing list or just firing off e-mails to the maintainer. Bug reports make for orderly fixes.

Joining Fedora Projects

The Fedora Project website already lists a variety of software projects you can become involved with. Signing on to one of the Fedora mailing lists (described earlier in this chapter) is a great way to connect up with the people at Red Hat, Inc., who are running the ongoing Fedora projects.

Some of the best features of Red Hat Linux that have made their transition over to Fedora are the Red Hat installation program, the graphically oriented redhat-config tools (called system-config in Fedora Core 2), and the Red Hat desktop. Current projects being sponsored by Red Hat as part of the Fedora Project that include these features are:

  • Red Hat Installation Program (Anaconda)-Its installation program named Anaconda, with both graphical and text-based versions, has been one of the strong points of Red Hat Linux for the past few years. Anaconda also includes a kick-start feature that lets someone installing Fedora preconfigure installation selections so that the install can be done with little or no live input. Anaconda is written in Python.

    Currently, the Fedora Project is asking for bug reports and requests for enhancements for Anaconda. If you are interested in contributing to this project, I recommend that you join the anaconda-devel-list and/or kickstart-list mailing lists to see how you can become involved.

  • Configuration Tools-Red Hat's own home-grown graphical configuration tools (many of which also have text-based interfaces) have made great strides in recent Red Hat Linux (and now Fedora) releases toward simplifying Linux system administration. These let you configure your network, file sharing, video cards, printers, and many other features.

    While I count about 40 Red Hat configuration tools, there is still room to simplify many other administrative features. The Fedora Configuration Tools Project is recommending a list of features that could benefit from a GUI. These include GUI tools for partitioning your disks, scheduling tasks, configuring more refined firewalls, and setting up boot servers (such as DHCP and NIS).

    To find out how you can contribute to the Configuration Tools Project, you can start by joining the fedora-config-list mailing list.

  • Desktop Project-Red Hat brought the concept of a standard look-and-feel across the two major desktop environments (GNOME and KDE) to Fedora in the form of the Fedora Desktop Project. The standard Red Hat look-and-feel is referred to as Blue Curve. Particular areas where Red Hat is looking for input related to the Desktop Project is in the area of creating desktop themes, creating splash screens, and choosing which applications are set up as defaults.

If you are fond of writing, or have enough insight into a feature that you could write a tutorial on it, the Fedora Docs Project is looking for contributors. As noted earlier, you could join the fedora-docs-list mailing list to find out more. There is also an ongoing effort to find people to translate Fedora into different languages. The fedora-trans-list mailing list is a good way to get further information.


Moving to Fedora Core 2

Fedora Core 2 reflects Red Hat's commitment to making the Fedora Project a platform for rapid development and adoption of Linux technology. Although we are recommending Fedora Core 1 as a more stable Linux distribution to follow along with this book, you may want to use Fedora Core 2 to take advantage of some of its new features.

If you are sitting down to use this book with Fedora Core 2, you should be aware of several differences from Fedora Core 1. Here are some of the major ones:

  • Linux 2.6.5 kernel-The Linux 2.6 kernel is a major upgrade from the 2.4.22 kernel included in Fedora Core 1. New features should result in improved performance, more supported hardware, and better scalability. For the most part, the new kernel shouldn't change how you use this book. However, visible reflections of the new kernel include a new /sys file system (where you can view and change kernel information) and device files represented by .ko instead of .o suffixes.

  • Advanced Linux Sound Architecture (ALSA)-The underlying mechanism for sound in Fedora Core 2 is ALSA, replacing the Open Source Sound system (OSS) used in Fedora Core 1 and earlier Red Hat Linux distributions.

  • system-config tools-To give a more generic feel, commands to launch graphical administration tools that previously began with redhat-config now begin with system-config. For example, the redhat-config-network command is now system-config-network. Anytime you encounter a command beginning with redhat-config, try system-config in Fedora Core 2 instead.

  • Security Enhanced Linux-The SE Linux feature pervades many aspects of the Fedora Core 2 operating system. This new security model can be used to overcome the monolithic "root owns the world" approach to security in earlier Linux and UNIX systems. Using access control lists (ACLs), a system with SE Linux enabled can finely define administrative control over the files, processes, and devices in that system. By default, SE Linux is turned off in Fedora Core 2.

  • X.org X server-The X server delivered with Fedora Core 2 changed from the XFree86 server to the X server software available from X.org. While this shouldn't have much impact on the average desktop user (who sees mostly the GNOME or KDE desktop environment, colored by the metacity window manager and Bluecurve theme), it does change some of the tools available for configuring your video cards from the command line.

  • Many new packages-More than 200 new packages were added to Fedora Core 2 after Fedora Core 1. While many of these packages were added to improve support for application development (especially Java applications), some new administrative features were added such as IPSEC (VPN software replacing cipe) and the exim mail transfer agent (which could be used instead of sendmail or postfix).


Summary

Despite the change in both name and how it is produced, the Linux operating system referred to as Fedora Core is a good, free Linux operating system for learning Linux troubleshooting. Red Hat's flagship Red Hat Linux product formed the foundation for the new Fedora Core distribution that is available today.

Linux Troubleshooting Bible is dedicated to bringing you the skills you need to prevent, find, and fix problems with Fedora Core, in particular. The same skills you learn in troubleshooting Fedora can be used for troubleshooting older Red Hat Linux systems and newer Red Hat Enterprise Linux systems, as well as a variety of other Linux systems.

Besides the information in this book, there are many other resources available if you want to support Fedora Core systems. The Fedora Project itself sponsors several very active mailing lists where you can ask questions about Fedora features, development, and software fixes. And despite not having a full documentation set dedicated to Fedora, there are many useful documents from Red Hat Linux 9, Red Hat Enterprise Linux, and from individual software projects (included on the Fedora Core CDs) that can be used to find information that applies to Fedora.

Because Fedora is a community-supported distribution, there are many ways you can contribute to the project itself. You can start by filing bug reports when you encounter problems. The Fedora Project also encourages you to contribute code fixes and request for enhancements to help improve Fedora going forward.


Chapter 2: Troubleshooting Installation

Overview

A failed Linux installation will stop you cold. I've heard too many stories of people giving up on Linux because they couldn't get it to install. In many cases, just knowing a few tricks can get you through.

Installation is one of the most distribution-specific features described in this book. The topics in this chapter cover how to fix or avoid problems when installing Fedora Core. However, because the same installer is used with Red Hat Linux 9 (as well as earlier versions) and Red Hat Enterprise Linux, most of the information contained here will cover those issues as well.

For other versions of Linux, such as SUSE and Debian, the installation process is much different. However, many of the installation troubleshooting concepts (passing options to the kernel, dealing with unsupported hardware, and so on) will help you know what to do if you have trouble installing those other Linux distributions. (See Appendixes B and C for descriptions of SUSE and Debian, respectively.)


Understanding Installation

Each Red Hat Linux distribution (Fedora Core, Red Hat Linux 9, or Red Hat Enterprise Linux) is made up of groups of RPM packages, bootable images, and other assorted software and README files. These files are commonly gathered together in one of the following forms:

  • Sets of CDs or DVDs-CDs are the most common medium for installing Red Hat Linux systems, although in the coming years DVDs will probably become more popular. Fedora Core 1 and Red Hat Linux 9 each consists of three installation CDs and three source code CDs. Red Hat Enterprise Linux is packaged in different CD combinations for WS, AS, and ES installations.

  • ISO images-These are images created to fit on a particular medium, such as CD or DVD. Typically, users can download a set of ISO images and burn them to CD or DVD themselves to use for installation. This creates, bit-for-bit, the same CDs or DVDs that were packaged with Red Hat Linux 9 boxed set or books on Fedora.

    As an alternative to burning ISO images to CD or DVD, you can copy these images to hard disk. The Red Hat installation procedures allow you to install images from a local hard disk or by using a network protocol, such as NFS.

  • Software repositories-The directory structure of a Red Hat Linux distribution can also be copied into a network server and installed using NFS, HTTP, or FTP protocols. RPM packages that would be on multiple CDs are merged into a single RedHat or Fedora directory (depending on which distribution used).

With the installation CDs in hand (or available on a server), you begin the installation process by booting from a CD or floppy disk. The installation program used on Fedora and all other Red Hat Linux distributions is referred to as Anaconda. Here's what happens when you start the installation process:

  • A kernel is started with a minimum set of installed drivers, and then additional modules are loaded from the initrd (initial RAM disk).

  • Installation begins using default values (graphical install from CD) or in a mode you enter at the boot prompt. The modes you enter determine if the install is run in text or graphical mode, if installation is done interactively or from a kickstart file, and whether the software is installed from the CD, network server (NFS, HTTP, or FTP), to name a few choices.

You can choose the modes Anaconda can run in to suit your situation. For example, installation normally runs in graphical mode, but you can run it in text mode to skip over any problems you might have with a video card. You can also add kernel options from the installation boot prompt. This can help you overcome some of the most common factors related to installation failures, which are due to unsupported, improperly probed, or buggy computer hardware.

Books such as Red Hat Linux Bible and the installation manuals that come with Fedora or Red Hat Linux do a good job of taking you through the steps to install Red Hat Linux distribution. Because our aim here is to help you troubleshoot, the following list is just to remind you of the choices you'll face during Fedora or Red Hat Linux installation:

  • Check media-You are given the option to verify the integrity of each of the CDs in your installation set before starting installation.

  • Language-You need to choose which language to use during installation. You can add other languages later.

  • Keyboard-Different languages and countries use different keyboards. Choosing the wrong keyboard type can result in letters not being where you expect and special characters you need (such as an umlaut or a circumflex not being available).

  • Mouse-Generic mouse selections based on your mouse type (serial, USB, and so on) will probably work in most cases. However, if your mouse device offers a wheel or special buttons, look for an exact match by manufacturer and model to get those features working correctly. Figure 2-1 shows an example of the Mouse Configuration window during Fedora Core installation.

  • Monitor-If you can't find an exact match for your monitor, consider using a generic selection based on screen resolution and monitor type (such as CRT or LCD). If you can't get a selection to work, try some of the options suggested in the troubleshooting tips ahead (such as vga=). After that, consider running installation in text mode and dealing with the issue later (see Chapter 7 for information on troubleshooting video cards and monitors). Figure 2-2 shows the Monitor Configuration screen during installation.

  • Upgrade or new install-Issues related to upgrading to Fedora from an earlier Red Hat Linux distribution are described in Chapter 3. Figure 2-3 shows the Upgrade Examine screen during installation.

  • Install type-Fedora offers special software package groups and configuration settings for Personal Desktop, Workstation, and Server installs. Advanced users typically choose Custom install where they can install everything, a minimum set of packages, or any combination of packages they choose. (Advanced users who repeat the same install process multiple times can use the kickstart feature described in Chapter 4.)

  • Disk partitioning-The Disk Druid screen lets you add, modify, or delete partitions on any of your hard disks. Advanced users can switch to a different virtual console (type Ctrl+Alt+F2) to get to a shell and then use the fdisk command to work with disk partitions. Disk Druid also supports RAID and LVM partitions. If you want to resize a partition as well, you can use the parted command.

  • GRUB boot loader-The GRUB boot loader can be configured so you can launch any bootable partition when you boot your computer. You can add a password to the boot loader (to protect from one who can boot your system) and you can include kernel boot options for special hardware needs. (Kernel boot options are described later in this chapter.)

    The LILO boot loader is no longer offered as an option, although you can replace GRUB with LILO by installing the lilo RPM package from CD 3 of the Fedora Core 1 installation set.

  • Configure networking-If your computer has an ethernet interface, you can configure that at install time. You can just leave the defaults in place, so that the computer will be set up as a DHCP client (and automatically start up if there is a DHCP server on the network).

  • Firewall-Red Hat Linux distributions presently all use iptables, by default, to configure firewalls. Selections during installation let you get a same set of rules to begin with (with high, medium, or no firewall security). See Chapter 11 for more information on firewalls. All firewall features on Red Hat Linux distributions use the iptables service by default.

  • Additional languages-You can select to have more languages supported than just the one you chose during installation

  • Date and time zone-If the computer has an Internet connection, you can use network time protocol (NTP) to automatically keep your computer's clock correct. You need to choose your time zone.

  • Root password-At least one user (root) is required for every Linux system. You enter (and verify) the root password here.

  • Packages-With custom installs, you can go through and select the exact set of software packages you want to install.

  • Boot disk-If your computer has a floppy disk drive, you have the option of creating a boot disk (which is recommended).

Click To expand
Figure 2-1: When configuring a mouse during installation, select the manufacturer and model number when possible.
Click To expand
Figure 2-2: If your monitor isn't listed on the Monitor Configuration screen, choose a generic CRT or LCD display option.
Click To expand
Figure 2-3: Fedora detects an earlier Fedora or Red Hat Linux installation and gives you the option of upgrading.

If installation failed at some point, go through the sections on choosing hardware and troubleshooting installation problems to help track down and solve the problem.


Choosing Hardware

By using computer hardware that is both supported and contains the recommended amount of power (RAM, CPU, and so forth), you have the best chance of successfully installing a Red Hat Linux system. This section presents some of the issues related to making sure you have computer hardware that will work for Fedora and Red Hat Linux systems.

Random Access Memory (RAM)

Having enough RAM installed on your computer is critical to a successful installation of Fedora or Red Hat Linux. Recommended memory amounts for Fedora Core are as follows:

  • Text mode: 64MB (minimum)

  • Graphical mode: 192MB (mimimum)

  • Graphical mode: 256MB (recommended)

If you are running in graphical mode with less than 192MB, you will almost certainly be unhappy with performance at some point. To run a Red Hat Linux system on less than 64MB of RAM, you might consider looking into the RULE project (www.rule-project.org). RULE (Run Up2date Linux Everywhere) is a version of Red Hat Linux that was modified to run on less powerful computers.

Note

According to the RULE project (speaking on an earlier version). Red Hat Linux will abso-lutely not install with less than 20MB of RAM. Using RULE, the project claims that you can install on a Pentium 200 MHz with 12MB of RAM in about 1 hour. So, if you are trying to install on low-end hardware, RULE might be the way to go.

If you meet or exceed the minimum RAM requirement, in most cases the Anaconda installer will detect it. So, the installation should just work without any special consideration of RAM. However, there are some cases where the kernel cannot detect the actual amount of RAM. In these cases, you can run a memory test to find out how much RAM you actually have (as explained in the next section), then provide an option when you boot the install process to tell the kernel how much memory to use.

Testing Memory From Installation Boot Prompt

To run a memory test, boot from the first installation CD (Fedora or Red Hat Linux). Then type the following at the boot prompt:

   boot:  memtest86

The memtest86 feature runs the memtest86 stand-alone memory diagnostic utility. Besides accurately reporting the total amount of available memory, memtest86 also tests for memory failures, level 1 and level2 cache memory, and reserved memory. It also reports on the memory chipset and method used to find memory size (e820 by default). For more information on memtest86, refer to www.memtest86.com.

Checking Memory after Installation

After Fedora or Red Hat Linux is installed, you can find out how much memory is being used by checking /proc/meminfo as follows:

   $  cat /proc/meminfo
Note

You can also view the contents of /proc/meminfo using the free command or the top command.

If you are about to do an upgrade or a fresh install, you can compare the amount shown in the cache size line to the Memory line shown in the memtest86 output. If not all the memory is being detected, you might consider providing the higher amount of RAM as an option when you boot Fedora or Red Hat Linux installation. (See the Selecting Memory Sizes section later in this chapter.)

Installation Media (CD-ROM)

CD-ROMs are still the most common medium for installing Linux systems. Red Hat Linux 9 and Fedora Core 1 each comes on a set of three CDs that you can get by downloading from the Internet or with books such as Red Hat Linux Bible. If you download the CD ISO images on your own, then burn them on to CDs, here are a few tips to help insure that you make good copies of those media:

  • Consider purchasing media or getting copies from a friend if you have a dial-up connection. Even a broadband connection will take 1 to 3 hours for each CD to download.

  • Run the md5sum command on each ISO image. For example, if you downloaded Fedora Core 1 (code-named yarrow), with the images in the current directory, type

    #  md5sum yarrow*iso > myMD5SUM

    Compare the contents of myMD5SUM to the MD5SUM you downloaded along with the ISO images (MD5SUM is a small text file contained in the same directory as the ISOs).

  • The most recent recommendation from Red Hat, Inc., for a command line to burn CD is to use the cdrecord command in dao mode. Here's that command line showing how to burn the first Fedora Core 1 CD from the current directory (repeat this step for the other two CDs in the installation set):

    #  cdrecord -v -dao -eject -pad padsize-150s yarrow-i386-disc1.iso

  • If you are burning bad copies, slow down the burn process (even as low as 2× or 4×).

  • Use CD-R instead of CD-RW media.

Video Cards

If your video card or monitor is improperly detected, the screen may be too garbled to be able to complete installation. Here are some suggestions if the install process is not able to properly probe and configure your video hardware:

  • Run the install in text mode (linux text)-After you boot the computer, you can fix the GUI using redhat-config-xfree86 or the XFree86 tools. As long as you are able to see the screen in text mode, you can continue with the installation. (This issue is addressed in Failing to Start the Install Process section later in this chapter.)

  • Try setting the vga=option-If you have a nonstandard size monitor or undetectable video card, you can try to set the vga= option at the boot prompt when you run the install. Laptop computers are particularly at risk for not being detected properly.

    Using the vga= option at the boot prompt, Anaconda can use the video BIOS to modify the display mode being used. This takes place before the kernel is even booted to start the install process. To display a list of available VGA settings to choose from, you can boot the install process using the following vga= option:

    boot:  linux vga-ask

    When you see the first message, press Enter to see a list of VGA settings you can choose from and it will appear as follows:

    Press <RETURN> to see video modes available.
SPACE to continue or wait 30 secs
Video adapter: VESA VGA
Mode: COLSxROWS:
0    0F00 80x25
1    0F01 80x50
2    0F02 80x43
3    0F03 80x28
4    0F05 80x30
5    0F06 80x34
6    0F07 80x60
7    0100 40x25
Enter node number or 'scan':

    You can type scan to scan for additional modes or type a mode number (from 0 to 7 in the example above) to choose a particular mode. As an alternative, you can choose a video mode directly with the vga= option. Table 2-1 contains some examples of vga options and the values they represent.

    Table 2-1: VESA Framebuffer Console

    VGA Value

    Video Modes Each Represents

    vga=769

    640 × 480 × 256K

    vga=771

    800 × 600 × 256

    vga=773

    1024 × 768 × 256K

    vga=784

    640 × 480 × 32K

    vga=785

    640 × 480 × 64K

    vga=787

    800 × 600 × 32K

    vga=788

    800 × 600 × 64K

    vga=790

    1024 × 768 × 32K

    vga=791

    1024 × 768 × 64K

The vga values can also be entered as hexadecimal numbers. Table 2-2 shows the hexadecimal values for different resolutions and numbers of colors (this table is from the Framebuffer HOWTO).

Table 2-2: Setting VGA Modes
Colors | 640 × 400 | 640 × 480 | 800 × 600 | 1024 × 768 | 1152 × 864 | 1280 × 1024 | 1600 × 1200
4 bits | ? | ? | 0 × 302 | ? | ? | ? | ?
8 bits | 0 × 300 | 0 × 301 | 0 × 303 | 0 × 305 | 0 × 161 | 0 × 307 | 0 × 31C
15 bits | ? | 0 × 310 | 0 × 313 | 0 × 316 | 0 × 162 | 0 × 319 | 0 × 31D
16 bits | ? | 0 × 311 | 0 × 314 | 0 × 317 | 0 × 163 | 0 × 31A | 0 × 31E
24 bits | ? | 0 × 312 | 0× 315 | 0 × 318 | ? | 0 × 31B | 0 × 31F
32 bits | ? | ? | ? | ? | 0 × 164 | ? |

For more information on VGA modes, refer to the svga.txt file in /usr/src/linux-2.*/Documentation directory. Refer to Chapter 7 for more in-depth coverage on troubleshooting video cards.

Laptop Problems

Some specific problems known to cause laptop installations to fail in Fedora Core 1 include the following:

  • Boot fails on LCD displays-Some LCD displays may prevent the installation process from starting. To get around the problem you can try disabling the frame buffer as follows:

    boot: linux nofb

  • Installation not starting on Sony VAIO notebooks-You may need to temporarily disable some PCI devices on Sony VAIO notebooks to get Fedora to install properly. Here are options you can try to add to the boot prompt to get installation to work:

    boot:  linux pci-off idel = 0x180, 0x386

If all Else Fails...

Even the sections you just read will help you take you through the type of hardware and media you should have used to install Fedora or Red Hat Linux; it's quite possible that some hardware that is up to specifications will not work. If you have gone through the information in this chapter and are still having failures installing Linux, I suggest you do the following:

  • Bugzilla-For now, Fedora and other Red Hat Linux distributions all log bug reports into Bugzilla database of Red Hat, Inc.: http://bugzilla.redhat.com. Try searching the Bugzilla database for your computer hardware to see if someone else has had trouble installing on it. If nothing turns up, consider entering a bug report yourself. Then track the bug to see how people on the project respond to the report.

  • Google-Use www.google.com or your favorite search engine to search for the word Linux, along with the name of your motherboard, chipset, or computer model. Although you may not be able to get Linux to install on your particular PC hardware, chances are someone else has tried and has shared their experiences on a website or mailing list somewhere. Googling might tell you about other peoples' results installing on the hardware you have.

  • Installation mailing lists-There are mailing lists devoted to answering questions about Fedora and Red Hat Linux. You can access the Red Hat installation list from www.redhat.com/mailman/listinfo/redhat-install-list. For Fedora, try the general.list: www.redhat.com/mailman/listinfo/fedora-list.


Overcoming Installation Problems

This section takes you through common (and some uncommon) failures that can occur during installation and provides advice to get through those problems.

Note

Many of the features for installing Fedora Core or Red Hat Linux are described in detail in Red Hat Linux Bible. If you are not familiar with techniques for installing over the network, creating install servers, making boot floppies, doing kickstart installations, and other installation topics, you can find these and other installation topics covered in Red Hat Linux Bible. Information on doing kickstart installations is included in Chapter 4.

Failing to Start the Install Process

There are some cases where you don't even make it to the boot: prompt on the installation screen. If that's the case, here are a few things you can try:

  • CD drive not booting-Some computers are not set up to boot from a CD. This is more often true of older PCs where you were expected to boot from a floppy disk. You can often fix this problem by changing the BIOS.

    To change the BIOS, boot the computer and look early on for a message that tells you how to enter setup (usually by pressing Del, F1, or F2). After entering setup, look for a place to change the boot order so that your CD drive is first on the list of media to boot (at least, before hard disk).

  • CD drive not found-It is possible that the installation process might be able to boot the CD from your IDE (ATAPI) CD drive, but not find the drive when it goes to install Fedora. To get around this problem, you can identify the CD drive from the installation boot prompt. For example, if your CD driverwere the slave drive on your first IDE controller, you could enter the following line to identify your CD drive:

    boot:  linux hdb=cdrom

    The master on the first IDE controller would be hda; the master on the second IDE controller, hdc; and the slave on the second IDE controller would be hdd.

  • Not bootable CD drive-If there is no way to make your CD drive bootable, you can create a bootable floppy disk image from the Red Hat Linux or Fedora installation media. You can find the bootable floppy disk images on the first Fedora Core CD in the images directory. The README file in that same directory provides a brief description of each image.

  • No bootable CD drive or floppy-An even slimmer possibility is that you have a computer with a nonbootable CD drive and no floppy disk drive at all. In that case, you might be able to boot installation from MS-DOS (that's MS-DOS directly and not from a DOS window in Windows). With DOS running, insert the CD into the CD drive and type the following:

    C:\ >  D:
D:\ >  cd \ dosutils
D:\ DOSUTTLS>  autoboot.bat

    Another alternative if your computer has a network connection is a Preboot eXecution Environment (PXE) install. To be able to do a PXE install, you must first set up a boot server to let the client boot the necessary install image. (PXE install images are contained on CD 1 for Fedora and Red Hat Linux systems in the images/pxeboot directory.) Then set up an install server (as described in Red Hat Linux Bible). If PXE support is built into the network interface card on the machine you want to install from, you can enable PXE in that computer's BIOS. When you boot the computer, select to boot from PXE. When prompted, identify the install method you want to use, and continue as you would with any network install.

    Note

    There is a white paper from HP on doing a PXE install. You can find it here:http://h18000.www1.hp.com/products/servers/linux/redhdt-whitepapers.html. The Linux on an IBM ThinkPad page (www.solarblue.net/docs/x31.htm) provides a good description of how to do a PXE install, including how to set up the PXE boot server.

  • No CD drive-If there is no CD drive on the computer, you can install from any of several other media types. By first creating a floppy boot disk (as mentioned in the previous point), you can install from software that is located on an HTTP, FTP, or NFS server. An ethernet card is required for each of those network install types. If you can get copies on to the hard disk, you can install from that medium as well.

  • Checking media-The media could be so bad that you may not even get to a boot prompt. To find out if the medium is truly damaged beyond repair, try inserting the first boot CD into any running computer (Linux or Windows) and try to browse the contents of the CD. If you can access the CD and some of the files on that CD, chances are that you should be able to boot the CD. (The CD could still be corrupted, but you can find that out with a media check when you boot the install process.)

  • Screen is unreadable-My first suggestion to installing on a machine where the GUI is unreadable is to run the installation in text mode:

    boot:  linux text

    Some experienced system administrators prefer text-based installs because it allows them to deal with their video cards after the operating system is installed. Instead of having the system boot to runlevel 5 with a GUI that may or may not work, the computer will boot runlevel 3 (text login prompt) and let the administrator get the GUI going on a working system.

    Note

    I always do a text-based install when I have an NVIDIA-based card since I know I'm going to replace the video driver after installation with an NVIDIA proprietary driver.

    If you are having trouble with the screen not being readable in text mode, there are some ways of telling the install process something about your graphical hardware. See the description of the vga= option earlier in this chapter for further information.

From the Boot Prompt

If you are able to successfully boot the Fedora Core or Red Hat Linux CD, you should see the installation boot screen (with a boot: prompt at the bottom). In most cases, you will choose from one of the following two installation types:

  • Graphical installation-If you just press Enter (or wait a few seconds for it to start automatically), Fedora Core 1 will start up the graphical installation process and expect to install the software from the three-CD set. This is the default way to install Red Hat Linux systems.

  • Text-based install-If you have a low memory computer, unsupported graphics card, or simply don't feel the need for colors and icons, you can do a text-based install. To start a text-based install, type linux text from the boot prompt.

Starting the installation process essentially means booting a small Linux kernel that, in turn, runs the Anaconda installer. Because you are starting an install process, there are several different install-related options you can add to direct how the installation process should behave. Likewise, you can also pass options to the kernel that can specifically tell the kernel about how to deal with your hardware (especially if the installation process is unable to properly detect your hardware).

Trying Red Hat Installation Options

The following points describe different problems that can arise during installation and that you might be able to correct by trying different options from the installation boot prompt.

  • Corrupted CDs-Before you install Fedora Core or Red Hat Linux from CD, you can verify that the contents of the CD have not been damaged by running mediacheck from the boot prompt. Even if you have no reason to suspect that the CDs are bad, checking the entire set of CDs can help you avoid having to deal with a partially completed installation.

    The mediacheck performs an md5sum checksum of the contents of any of the Red Hat installation CDs you provide. Because the exact same content (an ISO image) is used in an NFS or hard disk install, mediacheck allows you to validate those installation media as well. To start mediacheck from a boot prompt, type

    boot:  linux mediacheck

  • CD-ROM installation hangs transferring disk image-When booting from a CD, it is possible that your computer will hang with the message "Transferring Install Image to Hard Disk." There are two recommended courses to try: Turning on DMA for your CD or turning it off completely for your system. To turn on DMA for your CD, type the following from the installation boot prompt:

    boot:  linux allowcddma
    Note

    To run the same option for a text-based install, type linux text allowcddma. This also applies to any other option you want to add to a text-based install type.

    To try turning off DMA (during installation at least) to fix the problem, type the following at the boot prompt:

    boot:  linux nodma

    With DMA turned off, installation will go slower, but failures may be less likely to occur. After Linux is installed, you may want to remove the nodma option from the /boot/grub/grub.conf file to see if your computer can run with DMA on (to improve performance).

    One reported fix for this problem is also to reduce the amount of memory on computers (either by physically removing one of the RAM sticks or by making less memory available using the mem= option). See the Selecting Memory Sizes section later in this chapter for information on the mem option.

  • Hangs when installing on SMP system-If your computer hangs while trying to install on a system with multiple processors, you can try disabling IO-APIC. The noapic option causes an SMP kernel to disable some interrupt controller features on multiprocessor computers. Here's how to disable 10-APIC:

    boot:  linux noapic

    If your board doesn't create a working multiprocessor table, you might be able to create one manually using the pirq boot option. For information on how to do this, refer to the I0-APIC.txt file in the /usr/src/linux-2.*/Documentation directory.

  • Hangs probing PCMCIA devices-You can disable the probing of PCMCIA devices by adding the nopcmcia option as follows:

    boot:  linux nopcmcia

  • Hangs probing Firewire devices-You can disable the probing of Firewire devices by adding the nofirewire option as follows:

    boot:  linux nofirewire

  • Hardware not detected (or improperly detected)-The Anaconda installer does an amazingly good job of probing and identifying your computer's hardware. However, if you have reason to suspect that your hardware is not being properly identified, you can boot using the noprobe option.

    Before you start a noprobe install, you should know what hardware you have in your computer. If the drivers are not on the CD, you need to have the drivers you need available on floppy. Here's how to start a noprobe install:

    boot:  linux noprobe

    When prompted, you need to select the drivers to match your hardware or provide the drivers yourself on a floppy.

  • No floppy to start network or hard disk install-Earlier releases of Red Hat Linux required that you create a boot floppy to start an FTP, HTTP, NFS, or hard disk install (assuming, I suppose, that if you have a CD-ROM drive you can just install from there). In Fedora Core 1 (and other recent Red Hat Linux distributions), you can start a network or hard disk install from the first installation CD boot prompt using the askmethod option as follows:

    boot:  linux askmethod

    After entering your language and keyboard type, the installation screen asks you to select the type of install (HTTP, FTP, NFS, or hard disk). You might want to do this to save yourself from the trouble of inserting each CD.

  • Unable to upgrade-Any change to the /etc/fedora-release or /etc/redhat-release files (depending on which system you are upgrading from) may cause the upgrade process to fail to offer you the chance to upgrade your system to your chosen version of Fedora. If that is the case, you can force Fedora to allow you to upgrade by entering the upgradeany option at the boot prompt as follows:

    boot:  linux upgradeany

Trying Kernel Installation Options

There are literally hundreds of options you can pass to the kernel when you begin installation. While we go over some of those options here, you can find a more complete list in the documentation that comes with the kernel source code. For Fedora Core 1, refer to the following file for descriptions of kernel options:

   /usr/src/linux-2.*/Documentation/kernel-parameters.txt

Options you can pass to the kernel are also described, in some detail, in the BootPrompt HOWTO. You can find that document here

  • http://www.ibiblio.org/mdw/IIOWTO/text/BootPrompt-HOWTO

Documentation associated with particular drivers can be found in files and subdirectories of the /usr/src/linux-2*/Documentation directory. For the most part, you will not need to add module-specific options to the installation boot prompt. Special settings that need to be done to individual drivers can be done after installation (see Chapter 6 for information on adding loadable modules).

Caution

While most options that you can pass to the kernel from the boot prompt will cause no harm, some have the potential for damaging your hardware. For example, some improper video settings can damage your video card and options that effect CPU activity (such as no-hlt) can result in overheating.

In most cases, you don't need to enter any kernel options to get the Fedora or Red Hat Linux installation to work. Hardware will be detected properly and same defaults will be set. However, cases where you would want to add kernel parameters include the following:

  • Not able to probe-In some cases, the kernel is not able to properly identify the hardware you are using or fully determine the features it does or doesn't support.

  • Extra tuning-Some options that can be passed to the kernel can be used to tune your kernel to make most efficient use of your hardware.

  • Override the defaults-You may want to access hardware in a different way than it is used by default (for example, to change the default screen resolution).

Once you find kernel options that work during installation, write down those options. You may need to add those same options to the kernel boot process after Fedora or Red Hat Linux is installed so that your system can boot properly (see the Adding Kernel Options After Installation section later in this chapter).

The following sections break down different kernel options you can use at the boot prompt for Fedora and Red Hat Linux installation.

Selecting Memory Sizes

Sometimes CPU bugs, old BIOS limitations, and other issues can prevent Linux from properly detecting the amount of memory available on your computer. Not having all available memory useable by the install process can considerably slow down the install process. The cachesize= and mem= options can be used to tell the kernel how much memory to use during the install process.

Note

Before you start installing, you can test the amount of available memory using the mem-ory test feature. See the Random Access Memory section earlier in this chapter for information on using the memtest feature from the installation boot prompt.

  • Set CPU cache size-If the kernel can't determine the cache size correctly from the CPU (typically because of a CPU bug), you can override the level two CPU cache size and enter the proper amount in kilobytes. Here's an example where the level two cache is set to 512kB:

    boot:  linux cachesize=512k

    Be sure to not set this number higher than the actual available level two CPU cache size.

  • Set memory size-If you have added memory to an older machine (beyond the 64MB limit originally defined in PC BIOS), the boot process may not detect that fact. Using the mem= option, you can set how much memory the kernel has available to it. Don't set this number higher than the actual amount of RAM available or your computer will crash at some point.

Changing Power Mode Options

The hlt instructions tell CPU to change modes to use less power. The CPU essentially goes to sleep during some periods of disuse. Some older processors, particularly 486 processors, had problems coming out of this low-power mode. If you think this is the case for you, try entering no - hlt to the linux boot prompt as follows:

   boot:  linux no-hlt

One downside to using the no-hlt option is that the CPU stays on continuously. This could cause the CPU to overheat, especially in situations where you are overclocking the processor.

Some laptops have bugs in battery query support. If the machine crashes while entering a graphical install, you can try turning off advanced power management support as follows:

   boot:  linux apm=off

Partitioning Problems

When you create the disk partitions on which you install your Fedora or Red Hat Linux system, you need to make sure that those partitions provide enough space not only for the current installation, but also for the space you need for data. Here are some tips related to partitioning your hard disk:

  • Allow enough total space-An Everything installation of Fedora Core 1 can take up about 5GB of disk space. So, you need to allow at least that much space plus room for your data if you do a complete installation.

  • Allow enough space for installation on each partition-If you separate your hard disk into multiple partitions to keep areas of the files system, be sure to allow enough space in each partition to handle the data needed. For example, some parts of the file system you may want to assign to their own partitions are /bin, /var, /home, /usr, and /tmp.

    If you have 5GB available for a full install, but separate that space evenly across several partitions, chances are you will not have enough disk space where you need it. For example, /usr can require most of that 5GB of disk space itself, while /bin may require about 5MB of disk space.

  • Allow enough space for data on each partition-If you are splitting up partitions on your hard disk, be sure that you have taken account of the amount of space you need for data. Typically, content for your Web server, FTP server, and other data servers are contained under your /var directory. User data tends to be store under the /home directory.

  • Allow for overhead-An ext3 journaling file system (the default file system typed used by Fedora and Red Hat Linux systems), requires about 30MB of overhead for each partition. You need to take that into account if you are creating multiple partitions for your Linux installation.

  • The /boot partition-Although a 100MB partition is recommended for Fedora, you can get by with a much smaller boot partition if you are tight on space. A typical bootable linux kernel will consume between 1 and 4MB of disk space. So even with several different kernels, if the space is tight you can probably get by with as little as 20MB of space in /boot.


General Installation Troubleshooting Tips

This section contains some random tips to help you through the installation process.

  • Escaping to virtual consoles-During installation, it's possible to find out more about what is happening with the installation process than you can see from the main install screens. Using a set of Ctrl+Alt keystrokes, you can switch to different virtual terminals to see installation-related messages or even go to a shell prompt to run commands. Table 2-3 shows different key sequences (hold Ctrl and Alt keys as you press a function key).

    Table 2-3: Keys for Switching Virtual Terminals

    Keys

    Terminal

    Ctrl+Alt+F1

    Virtual console 1: Contains running installation dialog

    Ctrl+Alt+F2

    Virtual console 2: Offers shell prompt to run commands as root user

    Ctrl+Alt+F3

    Virtual console 3: Shows install log messages from installation program

    Ctrl+Alt+F4

    Virtual console 4: Displays system-related messages

    Ctrl+Alt+F5

    Virtual console 5: Displays other messages

    Ctrl+Alt+F7

    Virtual console 7: Return to graphical installation screen
    Note

    You can use these same key sequences to switch between virtual consoles after instal-lation is completed. You can use these virtual consoles to have multiple login sessions, even while a graphical desktop is running.

  • Changing partitions with fdisk or parted-Advanced users can use the parted, fdisk, or sfdisk commands to change partitions during the installation process itself. You do this by changing to a virtual console (Ctrl+Alt+F2) to get to a shell, then simply typing the command. Some features available through fdisk are not offered with Disk Druid; for example, more file system types are available through fdisk. Disk Druid also cannot resize partitions as parted can.

  • Getting help during installation-During the installation process, you can view online help text by pressing F1, F2, F3, F4, F5, or F6 buttons. Each provides a different type of help information.

  • Reading Release Notes-There is a lot of excellent information about the installation process in the Fedora Release Notes. To view the Release Notes from within any GUI install screen, click the Release Notes button on the bottom of the screen.

  • Grabbing install screens-During graphical installs you can grab screenshots of any installation screen. To do this, press Shift+Print Screen to save the current screen to hard disk. You can repeat this as many times as you like. When installation is completed, you can find the screenshot(s) in the /root/anaconda-screenshots directory. (This feature is not supported in text-based installs.)

  • Get the ks.cfg file for kickstart-You can use a kickstart file to answer installation questions, so you can run unattended installations of Fedora Core or Red Hat Linux distributions. After you have installed Fedora or Red Hat Linux, a file named anaconda-ks.cfg is created in the /root directory. You can use that kickstart file to reproduce the installation you just completed on another computer.

    Later, you can edit the anaconda-ks.cfg file with any text editor. You can also use the redhat-config-kickstart window to modify that file or create a new kickstart file.

    Note

    Refer to Chapter 4 for information on using the kickstart feature to create your own custom installs that you can use to automate the installation process for many computers.


Getting Linux to Boot after Installation

You may find that after you have finished installing Fedora or Red Hat Linux, your system will not boot. There are several things you can try if that is the case. The first suggestion would be to pass options to the kernel. After that, you can try to go into rescue mode to correct the problem.

Adding Kernel Options after Installation

If you need to add options to the kernel to get the installation process to work, you may also need to pass those same options to the kernel after the operating system is installed. Otherwise, some piece of hardware may not work properly or the entire boot process may fail. You can try out kernel options before you add them permanently to your boot loader configuration file. Procedures for adding kernel options temporarily and permanently are described in the following two sections.

Adding Options at Boot Time

To try out kernel options before adding them permanently, do the following:

  1. Reboot your computer.

  2. When you see the boot screen (GRUB by default), quickly press the arrow key to highlight the operating system you want to boot.

  3. Press e to be able to edit the boot lines for that operating system.

  4. Move the cursor to highlight the kernel line and press e to edit the kernel line.

  5. With the cursor at the end of the kernel line, type the options you want to add and press Enter.

  6. Press b to continue the boot process, including the new options.

If the options you added fixed your problem, you can consider adding them to your boot loader configuration file so that they can be used permanently. The following section describes how to add kernel options to the GRUB boot loader.

Adding Options to the GRUB Boot Loader

If you have decided that you want to permanently add boot options to your computer, and you are using the GRUB boot loader, you can put those options directly into the /boot/grub/grub.conf file. The following example shows what a grub.conf looks like:

   # grub.conf generated by anaconda
   #
   # Note that you do not have to rerun grub after making changes to this file
   # NOTICE: You have a /boot partition. This means that
   #         all kernel and initrd paths are relative to /boot/, eg.
   #         root (hd0.0)
   #         kernel /vmlinuz-version ro root=/dev/hda6
   #         initrd /initrd-version.img
   #boot=/dev/hda
   default=0
   timeout=13
   splash image=(hd0,0)/grub/splash.xpm.gz
   title Fedora Core (2.4.22-1.2115.nptl)
   root (hd0.0)
   kernel /vmlinuz-2.4.22-1.2115.nptl ro root=LABEL=/ hdb=ide-scsi
   #initrd /ini trd-2.4.22-1.2115.nptl.img

If you want to add boot options to this file, you can put them right at the end of the kernel line. For example, if you wanted to tell the kernel to use 256MB of RAM, you could add the mem=256M option to the end of the kernel line so that it appears as follows:

   kernel /vml inuz-2.4.22-1.2115.nptl ro root=LABEL=/ hdb=ide-scsi mem=256M

When Fedora boots, the mem= values will be read and the kernel will attempt to use the assigned amount of memory.

Entering Rescue Mode

The same boot CD you used to install Fedora Core or Red Hat Linux can be used to boot your computer in rescue mode. To enter rescue mode, insert the CD and boot the computer as you would start a regular installation. When the boot prompt appears, type the following:

   boot: linux rescue

Rescue mode does what it can to start up a network connection for your computer, locate your Fedora or Red Hat Linux installation, and mount the file system so that you access your files. If you can access your computer file system at that point, you can try to modify your configuration files to get Fedora to boot.

The first thing you should do is change the root directory to/mnt/sys image (where the rescue CD should have put the root directory of your Fedora Core installation). To do that, type

   # chroot /mnt/sysimage

With your file system now accessible as it would be during a regular system boot, you can try to correct whatever problem is preventing your system from starting up properly. The things you might want to do include the following:

  • Disabling services that may be hanging the computer-If there is a service keeping your computer from booting, you can simply disable it. For example, once you have access to a shell, you could disable the Samba service by typing

    # chkconfig smb off

    The next time you boot your computer, it will not try to start the Samba service.

  • Change the default run level-If the GUI is not starting up or crashing, you can edit the /etc/in ittabflle to make the default a text-based login. If the in it default line is booting to state 5, change the line to read as follows so it will boot to init 3:

    id:3:initdefault:

  • Add boot options-You can edit the /etc/grub/grub.conffile to add options to the kernel line, as described in the preceding section.


Installation Troubleshooting Reference

Table 2-4 lists some resources where you can find more information on troubleshooting Fedora and Red Hat Linux installation.

Table 2-4: Red Hat Linux and Fedora Installation Information
Resource | Location
Anaconda installer project | http://fedora.redhat.com/projects/anaconda-installer
Installation guides | Red Hat Enterprise Linux Installation Guides: www.redhat.com/docs/manuals/enterprise Red Hat Linux Installation Guides: www.redhat.com/docs/manuals/linux
Red Hat installation mailing lists | www.redhat.com/mailman/listinfo/redhat-install-list
Anaconda documentation in Fedora | /usr/share/doc/anaconda-9.2/
Anaconda development mailing list | www.redhat.com/mailman/listinfo/anaconda-devel-list
Kickstart mailing list | www.redhat.com/mailman/listinfo/kickstart-list


Summary

Fedora and Red Hat Linux operating systems rely on the Anaconda installer to handle the installation of the operating system. Both graphical and text-based installation processes are supported, as are different installation methods (including CD, NFS, HTTP, FTP, and hard disk installs).

Linux has improved dramatically in recent years in its ability to properly probe and configure hardware during installation. In those cases where Linux cannot detect your hardware or you want to fine-tune how hardware is configured, you can pass options to the installation process from the boot prompt.


Chapter 3: Updating and Upgrading Fedora

Overview

One of the most overlooked aspects of running a Linux system is keeping it up to date. Keeping a system up to date involves installing new versions of the software on your system. This can seem like a daunting task, given the huge amount of software included with Fedora Core. Luckily, there are automated updating systems both included with Fedora Core and available for Fedora Core as a third-party add-on.

Another process that needs attention is upgrading of your existing Red Hat Linux to Fedora Core. This process involves obtaining the installation media, performing the upgrade, and verifying your software.

Sometimes upgrading a system is more of a challenge than you want to handle, or maybe you've just gotten too comfortable with your current version to change to a new one. This chapter provides some information that helps you keep your legacy Red Hat Linux system running strong.


Keeping Fedora Core up to Date

Because no software is perfect, software developers are constantly trying to find ways to improve and enhance their products. The two most important reasons you should be vigilant about updating your system are security and stability.

  • Security-With the ever-growing popularity of Linux, discovering and correcting the security vulnerabilities of the software used on Linux systems becomes paramount. The fixes that emerge are made available to the community by your distribution vendor in the package format chosen for that distribution. Fedora Core is no different. Updates are provided in RPM format for end-user consumption.

  • Stability-Bugfix updates, while of lower priority than are security updates, are still very important as they directly affect the stability of your system. For the most part, minor bugs are fixed upstream by the software developers and do not spawn an update rpm for the current Fedora Core release. As the severity of the bug increases, so does the possibility of an update rpm being issued for the current Fedora Core release.

There are many ways to keep your Fedora Core system up to date. These include:

  • Automated updates-Using tools that automate the installation of updates, as well as resolving dependencies for the updates. These methods can be totally automated (with scheduled times for automatically processing updates), or triggered manually.

  • Manual updates-Manually seeking out available updates and processing them by hand. This method is satisfactory for single updates, but becomes tiresome when you need to handle lots of updates and/or large numbers of systems.

  • Source updates-Using source code to update existing software. This method may be useful when you need an update before a new package is made available, but it is also highly discouraged as it will lead to the irrelevance (what's actually on your system doesn't match what's in your RPM database) of your package management system as it's only able to track what you install/update in RPM format.

Caution

While it may seem on the surface that source installs will make only that package irrel-evant in your RPM database, all the packages that rely on this package or interact with it will also suffer from expecting one version of the package while a newer version is installed via source. This can cause a cascading effect as you gradually replace more and more packaged software with source to overcome the irrelevance of that first single package. While this may not happen with every user or package, it's necessary to warn against it.

Using Automated Updates

The preferred method of keeping your Fedora Core system up to date is with automated update tools. Two such tools are included with Fedora Core: up2date and yum.

up2date

up2date is the familiar Red Hat-developed tool that has both a command-line interface and a graphical user interface (GUI). up2date has been given some pretty significant enhancements with Fedora Core, including the ability to access yum, apt-rpm, and dir repositories. (yum and apt-rpm are discussed later in this chapter.) While up2date is technically able to communicate with Red Hat Network (RHN), currently there are no Fedora Core channels available at RHN and thus this functionality is wholly unused.

To go along with these new features, up2date has a new configuration file. Given that RHN functionality is no longer used in Fedora Core, you can safely ignore all other up2date configuration files, save the new one. This file holds a list of the yum/apt-rpm/dir repository you wish to use. The format is one repository entry per line. The first word on each line reflects the type of repository. The /etc/sysconfig/rhn/sources file contains the following active entries:

   yum fedora-core-1 http://fedora.redhat.com/releases/fedora- core-1
   yum updates-released
   http://fedora.redhat.com/updates/released/fedora-core-1

The first line shows a yum style repository named fedora-core-1 for the base OS packages (all the packages you'd find on Fedora Core CDs 1-3). The second line shows another yum style repository named updates-released for the released updates for Fedora Core 1. You can add as many repository lines as you wish, even linking to third-party repository for all your extra software.

Note

There's a line in /etc/sysconfig/rhn/sources that contains up2date default. It is safe to ignore this line, which is a vestigial artifact of the development process. If you want the full explanation, read on.

The line is for use with RHN servers. It is included, and the client supports RHN servers, because at the time of the initial beta cycles, the decision had not yet been made as to whether RHN would carry Fedora content. The decision was later made not to carry Fedora content, but the application still supports RHN protocol.

Each repository type has its own syntax. The syntax of each repository that can appear in the sources file used by up2date is covered in Table 3-1.

Table 3-1: up2date Sources Syntax
Repository Type | Syntax | Example [*]
yum | Type + channel-label + url | yum fedora-core-1 http://fedora.redhat.com/releases/ fedora-core-1
apt | Type + channel-label + service:server + path + repo-name | apt arjan-2.6-kernel http://people.redhat.com~arjan/2.5 kernel
dir | Type + repo-label + path | dir my-local-rpms /home/buiIdman/RPMS/
[*]Each repository name should be one line, with no breaks.

Multiple versions of all repositories can be used, aside from RHN repositories. Dependencies can be resolved cross-repository if need be, meaning that if you wish to install a package from one configured repository, and it requires a package located in a different repository, the second package can be found and installed from within its repository. This means that each repository does not have to be self-hosting, and can depend on other repositories to fulfill package requirements.

By default, up2date is configured to use the yum repository at fedora.redhat.com. If you've used up2date in the past, the usage syntax has not changed, even though the backend repository has. To check for available updates, type

   # /usr/sbin/up2date -1

or

   # /usr/sbin/up2date --list

If you are using a GUI when you first call up2date, it will bring up a screen to help you configure up2date. Each option is self-explanatory, and most often the defaults will suffice. It will also bring up a box asking you to import the Fedora GPG key into your rpm keyring. This will allow rpm to make an extra check of the package validity.

The next time you run up2date (configured for yum) it will need some time to download all the available package headers from the repository. Do not be alarmed if up2date seems to stall; check for network activity before killing any processes.

If there are any updates available, up2date will list them for you. You can update them individually, or choose to install all available updates. To update a single package, type

   # /usr/sbin/up2date -u package

where package is the update you want to install.

Caution

When using up2date to update single packages, beware that if the package you pass to up2date is not currently installed, up2date will install that package, and any dependencies it may need.

If you wish to process all the available updates at once, just type

   # /usr/sbin/up2date -u

If you use the GNOME or KDE desktop environments, you'll have a panel icon that can look like a blue ball with a white checkmark, or like a red ball with a white exclamation point, depending on whether any updates are available. This is the Red Hat Network Alert Notification Tool. This tool uses the repositories configured in /etc/sysconfig/rhn/sources to periodically check for available updates, changing from the blue check (no updates available) to the red exclamation point (updates available). Clicking the icon brings up a window that shows available updates and allows you to launch a graphical up2date interface to process the available updates.

This interface does not offer as much functionality as the command-line interface, but can be used to check for updates and install individual updates or all available updates. Figure 3-1 shows the graphical up2date with an update available, and Figure 3-2 shows the graphical up2date downloading an update.

Click To expand
Figure 3-1: Graphical up2date with an update available.
Click To expand
Figure 3-2: Graphical up2date downloading an update.

Once you understand the basics of what up2date does, you might want to look into some of the extra features that come with it. Here are a few tips and tricks that might interest you:

  • Package search-up2date has the ability to display all the packages available for download. The command to see this is up2date-showall. You can pipe the output of this through grep to search for a specific package or packages.

  • Source package retrival-up2date can retrieve a given srpm without trying to resolve dependencies. This option is -get-src.

  • Force install-up2date by default is set to skip kernel packages. To override this setting without editing the /etc/sysconfig/rhn/up2date configuration file, you can supply the - f flag on the command line. This will force up2date to ignore package exclusion rules and install/update what you ask it to.

  • Test run-up2date can be ran in a test mode, where it will process its task without actually doing the task. This can help you see the expected outcome of a complicated task without actually making any changes. The option for this is -dry - run.

There are many more options and configurations for up2date. Consult the up2date man page and the core configuration file /etc/sysconfig/rhn/up2date for more information.

yum

yum stands for Yellowdog Updater, Modified. You could say that yum started its life as a tool called yup or Yellowdog Updater for the Yellowdog Linux distribution. Yellowdog Linux is a Linux distribution built for Apple hardware and based on Red Hat Linux. The description of yum from its home page (http://linux.duke.edu/projects/yum) is:

  • yum is an automatic updater and package installer/remover for rpm systems. It automatically computes dependencies and figures out what things should occur to install packages. It makes it easier to maintain groups of machines without having to manually update each one using rpm.

Originally a very popular third-party package management tool, yum was officially included in Red Hat systems starting with Fedora Core 1. One of yum's best features is how easy it is to set up a yum server.

Using a yum Repository

For clients to be able to use a yum repository, they need to define that repository in their /etc/yum.conf configuration file. There are two sections to this configuration file: main and servers. The main section defines all the global configuration options, while the server section defines the entries for each repository. Table 3-2 shows the main configuration items and their default values. You can read more about these and other options on the yum.conf man page.

Table 3-2: yum Main Configuration
Options | Usage | Default Value
Cachedir | Directory that holds the cached header info | /var/cache/yum
Debug level | Controls the amount of debug output | 2
Logfile | File that yum log into | /var/log/yum.log
Pkgpolicy | Package sorting order (for use with multiple repositories) | newest (install the newest version found)
Distroverpkg | Package used to determine distribution release | fedora-release
Tolerant | How to treat errors on the command-line options | 0 (not tolerant)
Exactarch | Force yum to be strict with package archs | 1 (enabled)

Following the main options is the repository configuration section. This section lists the repositories one by one, each usually in a three-line set per repository. The first line is the serverid, contained in square ([]) brackets. This must be a unique name, all one word. The next line holds the human readable name for the repository. This is the name you'll see on the command line. The syntax for this line is name=My Repository Name. The final line is the actual URL to the repository. This can be in the form of http://, ftp://, or file://. The syntax is baseurl=url://server/path/to/repository.

Note

Don't forget the third slash (/) when defining a file-based repository, for example, baseurl=file:///var/packages.

You can also specify more than one URL, to set up a failover array of URLs, by just adding more URL lines. (Don't add another baseurl line or you'll get bizarre results from yum) By default, there are two servers configured for your Fedora system.

Cross-Reference

See Appendix A for a list of third-party repositories you can use with your Fedora Core system and yum. yum repositories can be added to your /etc/yum.conf file. Please consult the yum.conf man page as well as /etc/yum.conf for examples.

Now that you have your configuration file just the way you like it, it's time to actually use the application. yum is incredibly simple to use; simplicity is one of its strong points, and one of the goals behind the development of yum. To check for available updates, type the following:

   #  /usr/bin/yum list updates

If there are any updates available, yum will list them for you. You can update them individually, or choose to install all available updates. To update a single package type

   #  /usr/bin/yum update package

where package is the update you want to install. In order to process all the available updates at once, type

   #  /usr/bin/yum update
Note

This command will ask you to verify that you wish to install each update. This can get very tiresome if there are a lot of updates available. To assume yes for every question, pass yum the -y option. This will make yum fully automatic.
Setting up a yum Server

Setting up your own yum server has many benefits. Updates can be downloaded once to a local system, and then other local systems can update from it, saving wasted bandwidth from duplicate downloads. Custom repositories can be created with special content. Repositories can be created for systems that don't ship with yum.

There is no specific protocol for yum. yum supports http, ftp, local directory, and just about any remote file system protocol. There is no specific directory structure either. You can create any kind of file structure, and just make the top directory (and the subdirectories) available by one of the available protocols. The following instructions show how to set up a yum repository on an HTTP server.

  1. Copy the software packages to the directory structure you want to use. Assuming the packages you want are in the current directory, you could type

    # mkdir /var/www/html/mypackages-1
# cp *.rpm /var/www/html/mypackages-1

  2. Create the metadata needed by the yum client as follows:

    # yum-arch /var/www/html/mypackages-1/

  3. Enable the httpd server if you haven't already done so.

As shown in the code example, yum-arch takes one main argument: the location of the RPMS you want to scan for. This argument will scan from the argument directory down through all its subdirectories, gathering information about all the RPMs it finds along the way. yum-arch will create a directory named headers/. This directory holds the compressed RPM header information. This directory tree is now ready to be a yum repository.

Tips on using Yum

Although on the surface yum appears to be a simple tool, it is remarkably powerful and useful. Here are a few neat things that yum supports:

  • Package information-yum has the ability to display a package summary and description, even if the package is not installed. The option for this is info. It takes a package name or names as arguments, for example, # /usr/bin/yum info kernel.

  • Package removal-yum has the ability to remove packages, as well as any other packages that might depend on the package you're trying to remove. This can be extremely handy when trying to slim down an installation base. This option is remove. It takes a package name or names as arguments, for example, # /usr/bin/yum remove gimp.

  • Alternative configuration file-yum can be told to use an alternative configuration file for an operation. This comes in handy when you need to use a separate repository that is not normally used. Also, with yum's ability to read a configuration file over the Internet (http/ftp), you can maintain a single configuration file for an entire work group of computers, minimizing maintenance tasks when needing to make changes to yum's configuration. The option to use a different configuration file is - c and it takes a file location or an http/ftp file location as its argument, for example, # /usr/bin/yum -c http://mywebserver/myyumconf.conf update.

yum also supports some advanced features like package group operations and package listing options. For more information consult the yum and yum.conf man pages and the /etc/yum.conf configuration file.

Manual Updates

In some situations, you may prefer manual updates to automatic updates. This could be because you wish to quickly update a critical package without waiting for a possibly lengthy update program to run, or you wish to update a system that does not have any external Internet access.

In order to update your system manually, you'll need to know what updates are available and where you can download them from. Typically you can use the location of available updates to glean what updates are available, just by looking at the packages within the updates directory. There is also a Red Hat-sponsored mailing list setup for update announcements at www.redhat.com /mailman/listinfo/fedora-announce-list. This page also has an archive of past messages sent to the list.

Messages sent to this list have two main formats. One format is for security-related updates, and the other format is for non-security-related updates. Listing 3-1 shows a security update for the Linux kernel, and Listing 3-2 shows a nonsecurity update for the kernel as well.

Listing 3-1: Security-Related Update
Start example
   From: Dave Jones <davej redhat com>
   To: fedora-announce-list redhat com
   Subject: [SECURITY] Updated kernel resolves security vulnerability
   Date: Tue, 6 Jan 2004 13:18:20 +0000

   ------------------------------------------------------------------
   Fedora Update Notification
   FEDORA-2003-046
   2004-01-05
   ------------------------------------------------------------------
   Name        : kernel
   Version     : 2.4.22
   Release     : 1.2138.nptl
   Summary     : The Linux kernel (the core of the Linux operating system)
   Description :The kernel package contains the Linux kernel (vmlinuz), the
   core of your Red Hat Linux operating system. The kernel handles the basic
   functions of the operating system: memory allocation, process allocation,
   device input and output, etc.

   Paul Starzetz discovered a flaw in bounds checking in mremap() in the Linux
   kernel versions 2.4.23 and previous which may allow a local attacker to
   gain root privileges. No exploit is currently available; however, it is
   believed that this issue is exploitable (although not trivially.) The
   Common Vulnerabilities and Exposures project (cve.mitre.org) has assigned
   the name CAN-2003-0985 to this issue.

   All users are advised to upgrade to these errata packages, which contain a
   backported security patch that corrects this issue.

   Red Hat would like to thank Paul Starzetz from ISEC for disclosing this
   issue as well as Andrea Arcangeli and Solar Designer for working on the
   patch.

   These packages also contain a fix for a minor information leak in the real
   time clock (rtc) routines. The Common Vulnerabilities and Exposures project
   (cve.mitre.org) has assigned the name CAN-2003-0984 to this issue.

   -------------------------------------------------------------------
   * Wed Dec 24 2003 Dave Jones <davej redhat com>
   - Fix mremap corner case.

   * Tue Dec 23 2003 Dave Jones <davej redhat com>
   - Numerous USB fixes (#110307, #90442, #107929, #110872)

   * Tue Dec 16 2003 Dave Jones <davej redhat com>
   - Fix leak in CDROM IOCTL. (#112249)

   ------------------------------------------------------------------
   This update can be downloaded from:
     http://download.fedora.redhat.com/pub/fedora/linux/core/updates/1/
   C9a94fe6d6a3cffb2604a17bf42839e2 SRPMS/kernel-2.4.22-1.2138.nptl.src.rpm
   841d46869Z37a9acce1c5e667e13807e i3867kernel-source-2.4.22-
   1.2138.nptl.i386.rpm
   ad2e530db065ddc5cceef49ff825d606 i386/kernel-doc-2.4.22-
   1.2138.nptl.i386.rpm
   ed345bd737557d25437d8a06ff10c6ed i386/kernel-BOOT-2.4.22-
   1.2138.nptl.i386.rpm
   bc4dab2573386726679014c6fa2eb059 i386/debug/kernel-debuginfo-2.4.22-
   1.2138.nptl.i386.rpm
   88bb242ald86c54674c8be1b2a5d918e i386/kernel-2.4.22-1.2138.nptl.i586.rpm
   e5f61174a8586eb7487453dae7ea1875 i386/debug/kernel-debuginfo-2.4.22-
   1.2138.nptl.i586.rpm
   ee7ac4fferdfa5ceff18e10d9c5b776b i386/kernel-2.4.22-1.2138.nptl.i686.rpm
   50a7aabcbb8642c878714e3c58a49dff i386/kernel-smp-2.4.22-
   1.2138.nptl.i686.rpm
   fdd33ad20548d78295e59a3d3c7d1f5b i386/debug/kernel-debuginfo-2.4.22-
   1.2138.nptl.i686.rpm
   8b6ee386df3bcb68dd1c149c91f677e5 i386/kernel-2.4.22-1.2138.nptl.athlon.rpm
   01 af9cea1186fefc4e844c4d78cc7a06 i386/kernel-smp-2.4.22-
   1.2138.nptl.athlon.rpm
   01592b67d16d33f577c4de864f3799e5 i386/debug/kernel-debuginfo-2.4.22-
   1.2138.nptl.athlon.rpm

   This update can also be installed with the Update Agent; you can
   launch the Update Agent with the 'up2date' command.
   ------------------------------------------------------------------
   References:
   http://cve.mitre.org/cgi-bin/cvename.cgi?name=CAN-2003-0985
   http://isec.pl/vulnerabilities/isec-0013-mremap.txt
End example
Listing 3-2: Non-Security-Related Update
Start example
   From   :Dave Jones <davej redhat com>
   To     :fedora-announce-list redhat com
   Subject: Fedora Core 1 kernel update.
   Date   : Tue, 13 Jan 2004 20:24:56 +0000
   ------------------------------- -------------------------------------
   Fedora Update Notification
   FEDORA-2003-048
   2004-01-13
   --------------------------------------------------------------------
   Name        : kernel
   Version     : 2.4.22
   Release     : 1.2149.nptl
   Summary     : The Linux kernel (the core of the Linux operating system)
   Description :
   The kernel package contains the Linux kernel (vmlinuz), the core of your
   Fedora Core Linux operating system. The kernel handles the basic functions
   of the operating system: memory allocation, process allocation, device
   input and output, etc.

   ------------------------------------------------------------------

   * Wed Jan 07 2004 Dave Jones <davej redhat com>

   - Merge several EXT2/3 fixes from 2.4.25pre
     - EXT2/3 fixes.
      - Reclaim pages in truncate
      - 2.6 EA symlink compatibility
      - forward-compatibility: online resizing
      - Allow filesystems with expanded inodes to be mounted
      - Handle j_commit_interval == 0
     - IDE timeout race fix
   - Merge some 2.4.23pre patches that were missed.
     - Make root a special case for per-user process limits.
     - out_of_memory() locking
     - Drop module count if lockd reclaimer thread failed to start
     - Fix potential fsync() race condition
   - s/Red Hat/Fedora/ in specfile (#112992)
   - Add PCI ident for new Intel e1000 card. (#105046)
   - Actually wire up 3c59x ethtool ioctl.
   - Fix up numeric sysctls to match mainline.

   ------------------------------------------------------------------

   This update can be downloaded from:
     http://download.fedora.redhat.com/pub/fedora/linux/core/updates/1/

   0528ff2ef4b077e34a6e9f0bdc4c4c43  SRPMS/kernel-2.4.22-1.2149.nptl.src.rpm
   cf2da4c300650d6a68aeb3141e8de93e  i386/kernel-source-2.4.22-
   1.2149.nptl.i386.rpm
   b6ffe9lb11cb64af23f08de19c965677  i386/kernel-doc-2.4.22-
   1.2149.nptl.i386.rpm
   37bdb02f23cd936fc6a68c8b2f176275  i386/kernel-BOOT-2.4.22-
   1.2149.nptl.i386.rpm
   fe89710f267b238c9e5fcdf3d2658383  i386/debug/kernel-debuginfo-2.4.22-
   1.2149.nptl.i386.rpm
   68de6d015862dfc63d40b68a8fa1affa  i386/kernel-2.4.22-1.2149.nptl.i586.rpm
   9714ae57c042a42400336c06ae07f2c3  i386/debug/kernel-debuginfo-2.4.22-
   1.2149.nptl.i586.rpm
   ccc831fbb9ffa04ed7504f058411febc  i386/kernel-2.4.22-1.2149.nptl.1686.rpm
   257145cc01flea38fbcf22304b93d566  i386/kernel-smp-2.4.22-
   1.2149.nptl.i686.rpm
   baff2d70eb4e24b626511b9f3feacccd  i386/debug/kernel-debuginfo-2.4.22-
   1.2149.nptl.i686.rpm
   70b1314d932ff4283cab39a289b7aedc  i386/kernel-2.4.22-1.2149.nptl.athlon.rpm
   af27b81477979c6ff42a9e1475adaf3b  i386/kernel-smp-2.4.22-
   1.2149.nptl.athlon.rpm
   307fb0efbbc54a55e3d0c2a4b134c691  i386/debug/kernel-debuginfo-2.4.22-
   1.2149.nptl.athlon.rpm

   This update can also be installed with the Update Agent; you can launch the
   Update Agent with the 'up2date' command.
End example

The main differences between the two formats are the subject lines, the update description, and the references. Security updates get a special [SECURITY] prefix to their subject lines. This makes it easier for an administrator to pay close attention to security-based updates. The nature of a security update means there is more information given in the update description. The security flaw is described, and any external security tracking cases are referenced as well (like CVE or Common Vulnerabilities and Exposures). At the bottom of the notification, the external references are listed once more, with specific URLs.

These messages provide URLs to the updated packages, and you can find a mirror list at http://fedora.redhat.com/download/mirrors.html. You can manually download the packages and install them, using the -Uvh option, to rpm:

   #  rpm -Uvh gaim-0.74-5.i386.rpm

Updating packages manually means that you'll have to resolve the dependencies manually as well. This can sometimes be tricky. Thankfully, Fedora Core's rpm system is now able to make package suggestions for missing dependencies if you have the rpmdb-fedora package installed. (If you don't have this package installed, and you plan on manually updating your system, I highly recommend you install it.) Here's an example of rpm's ability to suggest resolutions:

   # rpm -ivh xsane-gimp-0.91-1.i386.rpm
   error: Failed dependencies:
           libieee1284.so.3 is needed by xsane-gimp-0.91-1
           libsane.so.1 is needed by xsane-gimp-0.91-1
       Suggested resolutions:
           libieee1284-0.2.8-1.i386.rpm
           sane-backends-1.0.12-4.i386.rpm

The output suggests that we install libieee1284 and sane-backends to satisfy the dependency requirements of xsane-gimp.

Source Updates

Source updates, while technically possible, are highly discouraged. Updating via source tarballs involves downloading the latest source from a project website, compiling the source, and installing over the current software set. This method is not very clean, as the default source install may not put files in the same places that the previous package did, does not update your rpm database, and may be missing critical patches that are necessary for operation on your Fedora Core system.

If you must go the source route, I recommend that you use a tool that will keep your RPM database up to date. Checkinstall (http://asic-linux.com.mx/~izto/checkinstall/) provides a way to take a set of sources and build RPMs out of them. It's still not as clean as RPM updates provided by the Fedora Project, but it is cleaner than straight source installs.

Troubleshooting Update Problems

Many problems can occur when updating your system. Many of the errors involve network problems of one kind or another. When you do run into update problems, make sure you can access the server you have configured. Try using mirrors of the server. If you are using a development tree, beware that the tree can sometimes be inconsistent and not all dependency resolutions are complete.


Upgrading to Fedora Core

Upgrading may seem like a daunting task. "Will my programs be ok?" "Will I lose my settings?" "Will I lose my custom applications?" These are some of the questions that get asked over and over again when preparing for an upgrade. Answers to these questions and many more can be found in the following sections.

Before you upgrade your system, there are a few things you should do:

  • Back up your data-One thing that can't be stressed enough is the importance of backing up your data prior to performing an upgrade. This step will ensure that in the case of upgrade failure, you can still recover your data. In most cases, your entire data set is held within your home directory, so making a backup of your home directory will suffice. If you've stored data elsewhere on the file system, you must back it up as well prior to upgrading your system.

  • Remove unnecessary software-Removing unused or unnecessary software will greatly reduce the time needed to complete your upgrade. Running /bin/rpm ~qa |sort |less will give you an alphabetical list of all the RPMs that are installed on your system that you can page through. Removing the packages you no longer need is recommended.

  • Obtain the proper media-Once an upgrade has begun, you cannot roll back to your previous Red Hat Linux release. Thus, it is vital that you have all the media that you will need to perform your upgrade. Upgrades can be done over any of the media that installs can be done from.

The process of upgrading to Fedora Core can be performed in many different ways. This chapter covers the three ways:

  • Upgrading using the Fedora Core installer (Anaconda)

  • Upgrading using Yellowdog Updater, Modified (yum)

  • Upgrading using apt-rpm (apt-get)

I also touch on some issues that may come up as you upgrade your system.

The process of upgrading differs from a new installation in that all the packages on the system are upgraded to the new version, but none of the settings are overwritten. This provides a way for you to get the new release installed without spending a lot of time reconfiguring your system to the settings it had prior to the upgrade.

Upgrading using the Fedora Core Installer

The Fedora Core installer, like previous incarnations of Red Hat Linux, offers the capability to upgrade from previous Red Hat Linux systems to Fedora Core. This method is arguably the most tested and most supported way of upgrading your system.

Starting the Upgrade

To start an upgrade process using the Fedora Core installer, boot your PC to the installation media (CD-ROM, DVD-ROM, floppy, PXE). This will bring up Anaconda, much like a normal Fedora Core install. After selecting your language, your mouse and keyboard, and your monitor, Anaconda will search for existing Red Hat Linux installs. If it finds any, you'll be given the choice to either upgrade or do a fresh install. Figure 3-3 shows the Anaconda screen where you can choose to upgrade an existing Red Hat Linux install.

Click To expand
Figure 3-3: Upgrade an existing Red Hat Linux install.

Once you choose to upgrade your existing install, you'll see some different screens and choices than you would with a standard install. The first screen you may see asks if you'd like to migrate your ext2 partitions to ext3. Migration is safe and causes no data loss. Figure 3-4 shows this screen, asking if I'd like to migrate my ext2 /boot partition to ext3.

Click To expand
Figure 3-4: Migrate ext2 to ext3.

The next screen asks you how you'd like to handle the boot loader. You have a few options:

  • Update your boot loader-This will update your current boot loader. This is the recommended option.

  • Skip boot loader updating-This will make no changes to boot loader configuration. If you are using a third-party boot loader (like Microsoft Windows), you should choose this.

  • Creat new boot loader configuration-This will let you create a new boot loader configuration. If you wish to switch boot loaders, you should choose this.

Next the installer will check your installed packages for things that it needs to upgrade. This process may take a while so don't get worried if you don't see anything happen on the screen other than the hourglass rotating.

On the next screen, click Next to upgrade to Fedora Core. A complete log of your upgrade will be saved to /root/upgrade.log.Based upon the package set Anaconda found to upgrade, you'll get a pop-up window showing which CDs you'll need to have to complete the upgrade. Figure 3-5 shows this screen.

Click To expand
Figure 3-5: Required install media.

If you don't have the required media handy, choose the reboot option to cancel the upgrade process, and start over when you have all the required media. If you click Continue, Anaconda will display a progress screen where you can track the progress of the upgrade process, as well as see the individual package names that are upgraded. Be sure to watch for requests to insert the next CD.

After all the packages are updated, you'll see two brief pop-up windows: the first tells you that the post-install configuration is being done, and the second notifies you that Anaconda is installing the boot loader. After these processes are complete, you'll see a screen that lets you know that the upgrade has finished, and you can reboot. Now you can reboot into your freshly upgraded Fedora Core 1.

Note

Because of the nature of upgrades, you will not see any new software, such as the graphical boot screens of Fedora Core 1. An upgrade will only upgrade existing packages, and only install new packages if they are a dependency of an upgraded package. If you wish to add newly available software after you perform the upgrade, use the redhat-config-packages tool after you have completed the upgrade.
Checking the Upgrade

All that's left now is playing with your new Fedora Core system, and configuring everything just the way you like it. Be sure to review your upgrade log files. Your /root directory will contain an upgrade.log file, and possibly an upgrade.log.syslog file. upgrade.log is a log file of how many packages were upgraded, which ones, and all the packages that were available on the Fedora Core CDs that were not used as part of the upgrade process. You can quickly look through the packages that did not get used to see what you might be missing. Within the section of packages that did get upgraded, there may be some comments about configuration files and how the new ones were treated during installation. For example:

   Upgrading openldap-2.1.22 8.i386.
   warning: /etc/openldap/ldap.conf created as
   /etc/openldap/ldap.conf.rpmnew

This lets you know that the ldap.conf file that came with the new ldap server package was saved with the .rpmnew extension, so as to not overwrite your current configuration. While this is normally a good thing, there may be key elements to that new file that you need to replicate in your existing file. It's a good practice to compare the two files and make the appropriate changes.

An upgrade.log.syslog file is created if any of the package upgrades caused a message to get logged to the syslog facility. Here's an excerpt from an upgrade that I recently did:

   <21>Jan 24 12:b4:47 sendmail[1191]: alias database
   /etc/aliases rebuilt by root

   <22>Jan 24 12:54:47 sendmail[1191]: /etc/aliases: 63 aliases,
   longest 10 bytes, 625 bytes total

   <86>Jan 24 12:55:53 useradd[1259]: new group: name=dbus,
   gid=81

   <86>Jan 24 12:55:53 useradd[1259]: new user: name=dbus,
   uid=81, gid=81, home=/, shell=/sbin/nologin

These are messages that would have normally appeared in /var/log/messages, but were added to the upgrade.log.syslog file for you to review after the upgrade was complete.

Upgrading using Yum

A very popular, although not supported, method of upgrading your system is using the yum system. This allows "live" upgrades to happen, often without a reboot. At the most basic levels, you fool yum into thinking that the system it's installed on is actually Fedora Core, and then just tell yum to install all available updates.

Caution

Given the unsupported nature of this type of upgrade, I cannot stress the importance of making a good backup of your data prior to attempting an upgrade.

An advantage to this upgrade method is that you upgrade automatically to the latest updates available, instead of upgrading to the initially released packages, then processing all the available updates. If you use any third-party package repositories, then with the proper configuration, you can update all the non-Red Hat packages on your system to the proper version along with your Red Hat packages.

When upgrading a Red Hat Linux 9 system to Fedora Core, it's best to use the yum that is included with Fedora Core, as the configuration will already be pointing to Fedora Core repositories. This RPM can be found at:

   http://download.fedora.redhat.com/pub/fedora/linux/core/1/i38
   6/os/Fedora/RPMS/yum-2.0.4-2.noarch.rpm

Download this file and install it with rpm. You may have to install the libxml2-python package from your Red Hat 9 media. Now that you have yum installed, you need to fool yum into thinking that you are running Fedora Core. To accomplish this, you need to install the fedora-release package from the same directory listed above. For this package you'll want to use the rpm -Uvh command as it will be upgrading the redhat-release package already on your system.

Once the fedora-release package is installed, you can issue the standard yum update command:

   # /usr/bin/yum update

After yum spends some time downloading all the header files for all the new RPMs (and updates), it will try to solve the dependency tree. Once it has done that, it will begin to gather all the RPMs. This can be quite a lengthy task, depending on how many packages you have installed, and where you're getting the updated packages from. If you're going to do this task for more than one system, it's a good idea to set up a local yum repository with all the necessary content to avoid the lengthy download multiple times. After yum gathers all the RPMs, it will attempt a test transaction. If this passes, it will proceed into updating all your packages. As each package transaction occurs, you'll see a comment to the right, something like syslinux 100% done 143/929. The numbers indicate how many actions have occurred versus how many are left to process. An action can be a package installation, a package removal, or a package %post script completing. A log of the upgrade process can be found at /var/log/yum.log. Once yum completes its task, you should reboot to the new kernel and your new Fedora Core system.

Upgrading using apt-rpm (apt-get)

Upgrading to Fedora Core using apt-get is very similar to using yum. You have to install the apt package, upgrade to the fedora-release package, and tell apt to update your system. This method is also unsupported, although nearly as popular, maybe even more so, than yum-based upgrades. Debian folk have been upgrading their systems with the Debian version of apt-get for many years, and swear by this method. Although apt-rpm and Debian's apt-get share a lot of the same syntaxes and theories, the methods by which packages are installed are quite different. This stems from the inherent differences between Debian's package platform (.deb) and Red Hat's RPM. Because of these differences, apt-rpm has to do a lot of different work to reach the same goal as apt-get. While this is not necessarily a bad thing, it is a new method, and thus shouldn't be as trusted as the tried-and-true Debian system. So again, make backups before you attempt to upgrade your system via apt-get.

Use the following instructions to upgrade with apt-rpm:

  1. Make a backup of your system in case of problems.

  2. The next step is to obtain apt-get. I recommend the apt package that Fedora.us provides, available at the following URL (which should be one line, with no spaces):

    http://download.fedora.us/fedora/redhat/9/i386/
RPMS.stable/apt-0.5.5cnc6-0.fdr.8.rh90.i386.rpm.

  3. To fool apt into thinking it is on a Fedora Core system, you need to edit its sources configuration file, /etc/apt/sources.list. In this file find these two uncommented lines:

    rpm http://download.fedora.us/fedora redhat/9/i386
os updates stable

rpm-src http://download.fedora.us/fedora redhat/9/i386
os updates stable

    Change these to point to the Fedora Core 1 repository, as follows:

    rpm http://download.fedora.us/fedora fedora/1/i386
os updates stable

rpm-src http://download.fedora.us/fedora fedora/1/i386
os updates stable

  4. Next, tell apt that you'd like your kernel upgraded when you do a dist-upgrade. To do this, you need to edit the main apt configuration file, /etc/apt/apt.conf. Within this file, find the line

    Upgrade-Kernel "false";

    and change it to true:

    Upgrade-Kernel "true";

  5. Next, install the fedora-release package found at http://download. fedora.redhat .com/pub/fedora/linux/core/l/i386/os/Fedora/RPMS/fedora-release-1-3.i386. rpm (the URL should be one line, with no spaces). This package is an upgrade to the existing redhat-release package, so use the upgrade syntax of rpm (rpm -Uvh). This package includes the Fedora Core gpg key, which needs to be imported into your rpm system. After you upgrade to the fedora-release package, run the following command:

    # /bin/rpm -import /usr/share/doc/fedora-release-
1/RPM-GPG-KEY-fedora

  6. Now you're ready for the actual upgrade. You issue two commands: The first command updates apt's package information, and the second acts upon this package information to actually process the upgrade:

    # /usr/bin/apt-get update
# /usr/bin/apt-get dist-upgrade

    apt will check against its local package database and tell you which packages it will update, which packages will be replaced by a package with a different name, and which new packages will be added. It will also tell you what the size of your total download will be, and how much more disk space these packages will take once installed. Press the y key to proceed with the update.

    Much like yum, apt will then download all the packages it needs, and proceed to install them. As apt installs the package, you will get progress indications on your screen. You will see the individual package progress indication, as well as the total task indication marked in percentage format:

       210:kudzu
   #################################### [ 43%]

    This line tells us that the 210th package update, kudzu, is 100 percent complete with the installation, and this brings us up to 43 percent completion for the entire upgrade.

  7. When the upgrade is complete, you will need to edit the /etc/grub.conf file. apt will not make the new kernel the default option. The default= stanza of the grub.conf file needs to be set to 0 for the first kernel listed. After that you can reboot, and enjoy your Fedora Core system.

Start Sidebar
Upgrading Using up2date: Not Recommended

A less common method of upgrading is with up2date. Since the up2date in Fedora Core can support yum or apt repositories, it can be used to upgrade your system using one of these two subsystems. Because of the added complexity of up2date on top of either yum or apt, I strongly recommend not using up2date to perform your upgrade, and instead using one of the three methods explained in the chapter.

End Sidebar


Troubleshooting Upgrade Problems

Some issues may come up during your upgrade. The majority of problems that arise in upgrading a Linux release involves the third-party software. Since the Linux distribution can only really know about the software it provides, often third-party software can be ignored. However, when the third-party software replaces some distribution-provided software, problems can arise. The biggest example of this is Ximian Desktop software.

Updating a Ximian Desktop

Ximian desktop software is designed to be complete from Ximian. Because of this, it replaces many core components to GNOME that came with your Red Hat Linux system. This means that the predetermined dependency tree is no longer valid, and upgrades tend to break or ignore the entire GNOME tree, leaving you with a nonworking desktop environment. The following excerpt from the Fedora Core 1 release notes comments on this problem:

There have been issues observed when upgrading Red Hat Linux 7.<x>, 8.0, 9 and Fedora Core 1 systems running Ximian GNOME. The issue is caused by version overlap between the official Red Hat Linux RPMs (or the ones from the Fedora Project) and the Ximian RPMs. This configuration is not supported. You have several choices in resolving this issue:

  1. You may remove Ximian GNOME from your system prior to upgrading to Fedora Core.

  2. You may upgrade your system, and then immediately reinstall Ximian GNOME.

  3. You may upgrade your system, and then immediately remove all remaining Ximian RPMs, replacing them with the corresponding Fedora Core RPMs.

    You must resolve the version overlap using one of the above choices. Failure to do so will result in an unstable GNOME configuration.

The same strategy applies to most other third-party software.

Checking Third-Party Updates

The yum and apt upgrade process has a unique ability to handle third-party software issues. yum and apt can be configured to provide updates for that software, so your third-party software is updated along with your Red Hat base.

A quick way to determine which non-Red Hat packages are on your system is to run the following command:

   # /bin/rpm -qa -queryformat "%{NAME}-%{VERSI0N}-
    %{RELEASE} Vendor: %{VENDOR)\ n"| grep -v "Red Hat, Inc."

This will query all your RPMs, with an output format of the name-version-release, followed by the vendor. You pipe this output into grep, which removes all entries containing Red Hat, Inc., leaving the non-Red Hat or third-party packages.

Breaking Graphics Drivers

Another issue that comes up quite often is that the graphical interface will not work after upgrading. Most often this is because the system was using third-party graphics drivers, like the NVIDIA 3D graphics module. These third-party graphics modules have to be built specifically for the kernel they will run on, but part of the distribution upgrade includes a new kernel. If you are using a third-party graphics module, consult the documentation for the driver on how to upgrade the driver for your new kernel.


How to Keep Your Current Red Hat Linux Systems Running

One challenge faced by many system administrators these days is what to do with legacy OS systems. At some point, known as the system end of life, a system may still run smoothly, but Red Hat no longer produces security or bug fix updates for the product.

Red Hat Linux systems have enjoyed a very lengthy support term, but that term is running short. Table 3-3 shows the recent Red Hat Linux releases, and their respective end of life.

Table 3-3: Red Hat Linux End-of-Life Dates
Red Hat Linux Release | End-of-Life Date
Red Hat Linux 7.1 | December 31, 2003
Red Hat Linux 7.2 | December 31, 2003
Red Hat Linux 7.3 | December 31, 2003
Red Hat Linux 8.0 | December 31, 2003
Red Hat Linux 9 | April 30, 2004

Some systems will not be hurt by the drop in support, as they are used only internally, have no Internet access, and continue to run smoothly. Other systems, however, rely heavily on Red Hat support as they are public-facing, have heavy usage, and continue to need bug fixes for the software used.

The alternatives for a system administrator faced with this dilemma include upgrading the product to a more recent offering, leaving the system as is for as long as possible, or using a third-party updating service. As of this writing, there are two choices for third-party updates: the free and open Fedora Legacy Project, and the pay service provided by Progeny Linux Systems.

Fedora Legacy

The Fedora Legacy Project can be found at www.fedoralegacy.org. The following description is from the home page of the Project:

  • The Fedora Legacy Project is a community-supported open source project. It is not a supported project of Red Hat, Inc. although Red Hat, Inc. does provide some support services for it.

  • The goal of The Fedora Legacy Project is to work with the Linux community to provide security and critical bug fix errata packages for select End of Life Red Hat Linux and Fedora Core distributions. This will allow for a longer effective life for those releases.

Through a community effort, the Fedora Legacy Project keeps track of the software that is contained in certain end-of-life Red Hat products. Currently, the Fedora Legacy Project supports Red Hat Linux 7.2, 7.3, and 8.0. According to its website, the Project plans on supporting Red Hat Linux 9 when it goes end of life, as well as extending the life of the Fedora Core releases.

The Project does all of its development and communication through open media channels. Two public mailing lists are located at www.red hat.com/mailman /listinfo/fedora -legacy-list/ and www.redhat.com/archives/fedora-legacy-announce. The Fedora Legacy List is for general discussions about the Fedora Legacy Project, as well as development discussions and test update announcements. The Fedora Legacy Announce List is a read-only (no posting by members is allowed) mailing list used to announce updates as they become available. The Project also has an IRC (Internet Relay Chat) channel where developers and end users can discuss issues and progress of package releases. Their channel is #fedora-legacy and it can be found on the Freenode IRC network (www.freenode.org/).

Currently, the Fedora Legacy Project supports package delivery methods of http, yum, and apt-get.

Progeny Linux Systems

Progeny Linux Systems provides a pay service called the Progeny Transition Service. The Progeny Transition Service offers software updates for Red Hat Linux 7.2, 7.3, and 8.0, with support for Red Hat Linux 9 scheduled to start early May 2004. The service is designed to provide a flexible migration path for Red Hat Linux systems.

Progeny's service fee is $5 per month, per system, or a flat fee of $2500 per month for unlimited machines. The minimal contract length is currently 6 months.

The Transition Service exists to help customers who are in the process of transitioning from their legacy Red Hat systems to a supported platform. Currently, the Progeny Transition Service supports package delivery methods of http, and Novell's Ximian Red Carpet Enterprise version 2.0.

More information about Progeny Transition Service can be found on its website at http://transition.progeny.com/.


Summary

Keeping your Linux system updated is vital to the health and security of your system. You can obtain vendor-provided updates for your system, and even third-party-provided updates for end-of-life products. Fedora Core 1 includes yum and up2date as tools for keeping your system updated.

Upgrading to Fedora Core need not be a scary prospect. There are multiple ways to accomplish the task, and workarounds available for most problems. Backing up your data prior to upgrading your system is absolutely critical. Loosing your data once will solidify this concept, but you'll be happier if you accept it on faith, rather than through sad experience.


Part II: Preventing Problems

Chapter List

Chapter 4: Securing and Automating Desktop and Server Installs
Chapter 5: Preparing for Backups and Migration


Chapter 4: Securing and Automating Desktop and Server Installs

Highlights

This chapter is targeted at IT administrators who are responsible for maintaining and installing large numbers of PCs. While Linux on the desktop is becoming more and more popular nowadays, without serious methods for standardizing, securing, and automating the install process, the interoperability of the desktop itself is only half the issue at hand.

This chapter builds upon the installation-related skills learned in Chapter 2 to help you secure your standardized desktop, workstation, and server OS installs. The kickstart technology covered here will enable you to create standardized desktop and server loads, customize those standard loads, and then roll them out to machines over the network. This client/server kickstart automated provisioning system comes with all Fedora Core-and Red Hat-based systems.

This type of OS install technology, when combined with OS hardening skills we cover in this chapter, is exactly what it takes to be able to roll out, or provision, Linux systems in the enterprise environment. This level of provisioning automation has been built into Red Hat-based systems for years and is one of the many features that really helps to decrease a Linux shop's Total Cost of Ownership (TCO), while increasing your bosses Return on Investment (ROI) (that is, justification for your salary).

We start off with a nice leisurely overview of the kickstart technology, and then get into how to customize your template machine loads. We review some best practices for locking down and securing your desktop and server template machines, go through the steps for creating your own kickstart provisioning server, and explain how to run a kickstart network install via boot floppy or CD. We wrap up with a couple of common kickstart problems (and their solutions).


A Kickstart Overview

Kickstarting a machine refers to the client/server method by which you can set up a centralized server with the OS Install CD content on it and offer this data out to the network (either via FTP, HTTP, or NFS services), and use this to automate your client OS loads. On the client side, you use special boot floppy or CD media to get a new PC up and then do either a manual interactive or fully automated desktop or server load.

Kickstart is Red Hat's own client/server technology. It lets you do local or network-based OS installs with as little as a network connection and a boot floppy. Kickstart builds upon older networking service technologies along with Red Hat's own client program, Anaconda. All these, combined with either a syslinux boot media or a PXE network boot environment, make up the complete kickstart automated provisioning system.

Note

Syslinux is a mini-Linux OS boot environment that can get a machine up and running with a basic network connection. Red Hat-based systems use it along with the Anaconda client side program to perform an OS load, either from CD, the hard drive, or over the network. For more information on syslinux or isolinux, go to http://syslinux.zytor.com/.
Start Sidebar
PXE Booting

PXE, short for Pre-Boot Execution Environment and pronounced "pixie," is a client-side boot specification by Intel that is usually implemented by network card hardware, and uses the network server-side DHCP/BOOTP protocols. If you have a network card that features PXE booting and a BIOS that can support booting from PXE, then you have the ability to "network boot" a blank machine (assuming you've set up a DHCP/BOOTP boot server). From there you can perform network-based installs, hardware testing, or cluster provisioning, without even inserting so much as a floppy or a CD. PXE is not recommended for a production LAN as accidental PXE-first booting on a production machine could prove catastrophic if a new OS is installed over a user's or administrator's production machine. For more information on PXE check these sites: www.linux-mag.com/2002-10/netbooting_01.html and ftp://download.intel.com/labs/manage/wfm/download/pxespec.pdf.

End Sidebar


Starting Your Kickstart Templates

The nice thing about kickstart is that if you've ever installed Red Hat on a machine, you've created a kickstart config file without even knowing it. This file contains all the options that you entered during the install. After any Fedora Core- or Red Hat-based install, you should see your new kickstart config file (called anaconda-ks.cfg) in the root user's home directory /root/:

   # ls -la /root/an*
   -rw-r--r--    1 root root 1877 Dec 16 12:12 /root/anaconda-ks.cfg

You can also use Red Hat's GUI config tool to create a kickstart config file from scratch (see the Using the Red Hat Kickstart Configuration Tool section later in this chapter for details).

Looking Inside Your ks.cfg File

A kickstart or ks.cfg file is divided into four main parts:

  • The Install Command Area-This portion of the file has the configuration information and commands, such as the new system's settings, the install media source (CD, network, or hard drive based), the new system's firewall settings, root password, authentication configuration, new install versus upgrade, and so on. This area also contains information such as whether the drive is to be wiped clean before starting the install, what partition types and sizes are to be set up, whether the system will use the MBR for booting, and if you're setting up drive/partition level options like software RAID or LVM.

  • The New System Packages Area-This portion of the file lists all the RPM packages or metapackages (groups of packages) that are to be installed.

  • The Pre-Install Script-This portion contains shell or programming commands that you want to run before the install begins.

  • The Post-Install Script-This portion contains the shell or programming commands that run after the basic install, but before the end of the install or the reboot.

In short, this file has all the installation settings covered in Chapter 2. In fact, if you did an install on your machine, you should now have your own /root/anaconda-ks.cfg file that will enable you to install the same configuration settings on other machines. This anaconda-generated file is, essentially, your first kickstart template file. The next step is to back up your anaconda-ks.cfg file so that you can easily identify the original file and so that it won't be overwritten later:

   #  cd /root
   #  cp -a anaconda-ks.cfg ks-my-first-install.cfg
Tip

You can name these kickstart config files anything you want, but when you create boot disks later that will use them, by default anaconda looks for the file name ks.cfg on the media that you point it to. If you are running several different kickstart config files, it is wise to break away from the ks.cfg standard naming and use something such as ks-workstation-date.cfg so that you'll know exactly what file you're about to apply to a given system.

Take a look at the following /root/anaconda-ks.cfg config file generated when I did a server load from CD. My added comments appear in boldface italics (# - The Install Command Area):

# Kickstart file automatically generated by anaconda.
# - The Install Command Area
# -----------------------------------------
Install
cdrom                                  #- <--- The install source
lang en_US.UTF-8
langsupport --default en_US.UTF-8 en_US.UTF-8

keyboard us
mouse alpsps/2 --device psaux --emulthree
xconfig --card "S3 ProSavage KM133" --videoram 31680 --hsync 1.5-48.5 -
vsync 40-70 --resolution 1024x768 --depth 16 --startxonboot --
defaultdesktop gnome                   # - <--- xconfig all one line...
network --device eth0 --bootproto dhcp # - <--- Network setup
rootpw --iscrypted $1$td8M3NHj$1S.0kU.Pz.rGT2VzZpRND/
firewall --enabled --port=http:tcp --port=ssh:tcp
authconfig --enableshadow --enablemd5
timezone America/Winnipeg
bootloader --location=mbr --append hdc=ide-scsi rhgb
# The following is the partition information you requested
# Note that any partitions you deleted are not expressed
# here so unless you clear all partitions first, this is
# not guaranteed to work
#clearpart --linux --drives=had
#part /boot --fstype ext3 --size=100 --ondisk=had
#part / --fstype ext3 --size=1024 --grow -asprimary
#part swap --size=514 -asprimary

#-Notice above that even if you do a fresh install, wipe all the
#-existing partitions and create new partitions, that this area gets
#-commended out in your anaconda-ks.cfg file. This is for safety
#-reasons. If someone tests their new ks.cfg file and doesn't quite
#-understand what they're doing, this will save them from nuking
#-their own system. DO NOT FORGET TO UNCOMMENT THESE IN
#- YOUR FINAL TEST INSTALL

# - New System Packages Area
# ---------------------------------------
%packages
@ kde-software-development  #- These are all meta packages
@ office                    #- or logical groupings of packages
@ kde-desktop
@ network-server
@ web-server
@ engineering-and-scientific
@ x-software-development
@ mail-server
@ dns-server
@ server-cfg
@ dialup
@ sound-and-video
@ sql-server
@ editors
@ admin-tools
@ system-tools
@ authoring-and-publishing
@ base-x
@ gnome-desktop
@ gnome-software-development
@ graphics
@ ftp-server
@ kernel-development
@ smb-server
@ development-tools
@ ruby
@ printing
@ games
@ text-internet
@ graphical-internet
vnc-server            #-These are the individual RPM
7squirrelmail          #- packages that were added on
postfix                #- through the GUI installer when I
tsclient               #- set up this server load.
elinks
vim-XII
xpdf
ethereal-gnome
xmms-skins
dhcp
mailman
xcdroast
shapecfg
redhat-config-boot
tftp-server
koffice
gtkam-gimp
grip
nmap-frontend
kdepim
kdegraphics
rdesktop
gnomemeeting
kernel-pcmcia-cs
mysql-server
grub
abiword
spamassassin
fsh
kernel
talk-server
kdemultimedia
php-mysql
amanda-server
ncftp
vnc
amanda-client
imap
quanta
telnet-server
redhat-switch-mail

#-Pre Install Script
#-----------------------t
%pre
echo Pre-Install bash commands go here...

#-Post Install Script
#--------------------------
%post
echo Post-Install bash commands go here...
echo
Tip

Sometimes manually adding RPMs to the Packages section will lead to dependency issues and can cause your automatic install to fail from dependency errors. To handle this and automatically install any required subdependency packages, you can add the --resolvedeps switch to the packages header like this:

  %packages --resolvedeps

For more information on this topic, see Chapter 9 of Red Hat's online RH-EL3 Administration Guide documentation at www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/sysadmin-guide/ch-kickstart2.html.
Note

Notice this file has no mention of the terms server or workstation, as in the original interactive Fedora Core installer. This is because these terms are just abstractions of the metapackages and RPMs or that define "groupings of groupings" of RPMs. The listing of packages in the config file example is the standard Server load, with a number of metapackages and RPMs added. These metapackages are all grouped logically and defined in the base comps XML file located on CD 1 in Fedora/base/comps.xml. You can see all of these metapackage names with the following command (typed as one line) while Fedora Core CD 1 is mounted:

  $ cat /mnt/cdrom/Fedora/base/comps.xml | grep \ <id
  |  grep -v '\ -support'

In these metapackages and groupings you see groups such as games and office as well as our groups of groups server and workstation-common.

Network Enabling the ks.cfg File

Now that you have an idea of what a kickstart config file looks like, the next step is to make this config file usable as a part of a network install boot disk.

Note

At Rackspace Managed Hosting, one "Geek Rites of Passage" for all new Linux system administrators is to load their own workstation without using any CDs. This is done by downloading a Red Hat boot disk image from the one-line Red Hat mirror content images directory, using dd to write it out to a floppy, and then performing a CD-less network-based install from our Red Hat mirror server. Many people do not realize that they can install most Linux distros totally over the Internet or a LAN with nothing more than a network boot floppy or other network-aware boot environment (such as a PXE NIC/LAN setup). For more information on using dd to write out floppy disk images, see the Creating Customized Boot Floppies section.

Here are the lines in the ks.cfg file that we need to change to make the install pull the CD files from the network:

   install
   cdrom
   lang en_US.UTF-8
   langsupport --default en_US.UTF-8 en_US.UTF-8
   keyboard us
   .
   .
   .
   network --device eth0 --bootproto dhcp
   rootpw --iscrypted $1$td8M3NHj$1S.0kU.Pz.rGT2VzZpRND/

The following block of code shows the changes required to have the install look to a network-based ftp kickstart server:

   # TWW: 2004-02-22, Network install of the Fedora CD Media
   # files from my FTP server...
    install
    url --url ftp://ftp.example.com/pub/fedora/core/l/i386/os/
   lang en_US.UTF-8
   langsupport --default en_US.UTF-8 en_US.UTF-8
   keyboard us
   .
   .
   .
   network --device eth0 --bootproto dhcp
   rootpw --iscrypted $1$td8M3NHj$1S.0kU.Pz.rGT2VzZpRND/

Notice that first we've changed the type of install from a CD-ROM-based to a URL or network install. Also notice that we've defined the URL from which the new machine is to get its install files. (We have not yet set up that FTP file share, but will do so in the Creating a Kickstart Provisioning Server section later in this chapter.)

Tip

If you want to do your own network install of Fedora Core Linux from the Internet with nothing but a floppy and a PC, just go out to a Fedora mirror site, download the bootdisk.img and drvnet.img (or just the bootdisk.iso CD image), copy them to the correct media (more on this in the Creating Your Kickstart Provisioning Boot Media section), and boot off that media. You simply follow the prompts to do an FTP, HTTP, or NFS-based install, and point the install to one of the mirror sites, including the full path to the directory above the Fedora directory. For example, the full path to the Fedora directory (case matters, and all on one line) is

  • ftp://mirrors.kernel.org/fedora/core/1/i386/os/Fedora/

Then point your manual floppy/CD network install to

  • ftp://mirrors.kernel.org/fedora/core/1/i386/os

Whatever directory path you give it, it will expect to find the Fedora directory inside it. The same goes for Red Hat wanting the directory that houses the RedHat directory.

Using the Red Hat Kickstart Configuration Tool

The other "easy way" of creating a kickstart template file is via the Red Hat kickstart configuration GUI tool: redhat-config-kickstart. This tool is nice in that it gives you a GUI front end to that comps.xml file. Just take a peek inside that file and you'll want some form of abstraction. Trust me.

You can start Red Hat's kickstart configuration tool either from the command line (as shown in Figure 4-1) or by the clicking on Red Hat⇨System Tools⇨Kickstart.

Click To expand
Figure 4-1: The redhat-config-kickstart GUI config tool makes a ks.cfg file for automating Fedora Core or Red Hat Enterprise network-based OS installs.
Note

You do not have to be root to run the config file generation tool. It just is not in the $PATH of nonroot users, so Figure 4-1 shows how to start it after becoming root with su -. I also could have run it as user bob by simply running it by its absolute path of /usr/sbin/redhat-config-kickstart, or via the desktop menu Red Hat⇨System Tools⇨Kickstart.

Although redhat-config-kickstart tool looks like the solution to all the headaches of manually playing with raw ks.cfg files, don't be fooled. This GUI utility has its shortcomings. First, it's not that stable and has problems reading existing ks.cfg files. GUI tools are nice to look at and occasionally handy for whipping up something quickly, but be careful.

Caution

Some Red Hat GUI tools do not work well, have problems reading existing config files, or in the worst cases (as with the apache config tool) can even overwrite your existing config file. Just be careful when using them and back up your original flat files before trying them out.

With this caveat noted, this tool is very convenient for throwing together a base ks.cfg file without having to do a full install on a box. After using the tool to get started, you can open the new ks.cfg file in your favorite text editor and further refine and tweak it.

Although learning to use the tool is fairly easy, a complete breakdown of its usage is available online at Red Hat's EL3 Admin Guide documentation, Chapter 10, at www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/sysadmin-gulde/ch-redhat-conflg-kickstart.html.

Saving Your ks Files

Now that you have the base kickstart template file(s) set up and network ready, take a moment to name them accurately and organize them for easy retrieval. I've used my first template and the config tool to create a few specific OS loads (for example, desktops, workstations, and servers) for use throughout my company. Typically a directory of these ks.cfg files should be neatly organized to look something like this:

  ks-everything.cfg
  ks-IT-desktop_2004-02-16.cfg
  ks-IT-desktop_2004-02-20.cfg
  ks-my-first-install_2004-02-21.cfg
  ks-server-laptop-install_2004-02-22.cfg
  ks-workstation_2004-02-19.cfg
  ks-workstation_2004-02-20.cfg
Tip

Besides naming your files with descriptive names and date stamps, you should also consider keeping an internal comment-based change log so that you can track down problems to specific changes made to your ks.cfg file on specific dates.

The next section shows you how to lock down your desktop and server loads by modifying these ks.cfg files to include some common "lock down best practices" settings. This will empower you to secure your OS loads by guaranteeing that all machines that you provision with this system will have your exact configuration settings.


Locking Down Desktops and Servers

This section covers some example settings you might want to add to your template workstation or server loads to lock them down, secure them, or just customize or standardize them for your enterprise environment.

Anything that you could normally run or do on a client PC or a new server install from the command line can be put into your ks.cfg's postinstall script section to be automated. To begin, you'll need to think through the kind of standardized settings your enterprise PC and server installs will require.

Identifying Your Enterprise Desktop Needs

Just pointing some of your client PCs to different DNS servers, giving them all the same internal home page, bookmarks, or company directory can be invaluable. And how about creating standard command aliases or desktop icons for common company tasks, or "back door" administrative accounts, or a help desk VNC remote desktop setup? Wouldn't you like your people to have standardized corporate NFS/SMB drive mappings, printer settings, and antivirus package configurations? Not to mention things like desktop backups and system health warnings. These are just a few of the things a customized installation template can provide.

Most experienced IT administrators already know the importance of building a customized OS install template. Many inexperienced IT guys learn this the hard way: by months of fighting with users over the phone, countless visits to the cube farms, manual matching, reinstalls, and troubleshooting sessions. A smart set-up design can help you avoid wasted hours and frayed tempers.

Identifying Your Enterprise Server Needs

This is an area where you really want to sink some thought: Would you like all of your servers to have nightly cron jobs to download and apply the latest security patches? Does a root-kit or intrusion security check and e-mail you anything out of the norm? Send you an e-mail if the machine is rebooted? Tar up and backup, to a single place on the network, a set of important config files, logs, or database files? What about setting up trusted SSH keys in every new server that goes online so that you can log into any new server, even if the root password has changed? These are all things that can be put into the %postscript area of your ks.cfg file.

Planning and defining each of your enterprise server issues is really the key. It's also essential to discuss these plans with all the system administrators in your company to make sure everyone understands the benefits of a standardized kickstart file and to generate ideas for useful %postscript options.

Every company's needs are different. You need to sit down, hash out, and plan for your departmental and organizational needs. Now with all that out of the way, let's take a look at what you can do to make a standard install of Fedora Core Linux (or most any Linux), a bit more tight and secure.

Locking Down Your Template Installs

In this section are some good rules of thumb for locking down and securing the common services that you have running on your machines. What you use here depends on your security needs. You might find it useful to refer to the iptable/firewalling configurations, discussed in Chapter 11, to set up and implement some basic iptable automation for all of your server loads. Just follow along and you'll quickly get an idea of what's possible.

Note

If you have not yet done a basic install of your server template machine, you should do one before proceeding. The commands that we are going to use in our %postscripting section will need to be run and tested as if from the actual install machine itself. As such, you will need to have a test machine to run these commands on to verify proper operation before implementing them in your own ks.cfg file. To assemble a reliable set of %postscripting commands, you will need to run the various test commands, take notes, and copy and paste your good commands into what will end up being the final %postinstall script section within your ks.cfg file.

The main install changes that you want to examine right off for your servers and perhaps for some desktop systems, include the following:

  • Shutting off unneeded services

  • Locking down networking/IPv4 settings

  • Patching automation

  • System "Health checks"

  • Basic intrusion detection or root-kit checks

  • On-disk hardware testing packages

The following subsections cover each of these items.

Shutting Off Unneeded Services

Before you can lock down services, you should first see what your default runlevel is. On most Red Hat/Fedora Core-based Linux systems it will either be a default runlevel of 3 (for console/text logins) or 5 (for Xll GUI display manager logins). Reboot your template server and type the following:

  # cat /etc/inittab | grep ^id:
  id:5:initdefault:
  #
  # runlevel
  N 5

If you're on a Red Hat/Fedora machine and both results are 5, you are running a X11 GUI-based login machine. If you prefer a text login only server, edit /etc/inittab and change the line you see above to an initdefault of 3, reboot, and check it again. Runlevel 3 should be most servers' default runlevel if a run-time GUI is not desired. In the examples here, I'm operating on a server load installed on my laptop, so runlevel 5 will be the default.

Tip

You want to have a good reason before running a server in runlevel 5 with X11 GUI running, not only is this a security issue but it also eats up system resources, there are serious performance issues with running a server with X windows in place. Just think about some 3-D screen savers and the processing power they eat up. Do you really want your web server or mail server that's under attack from the latest virus having to tread water while competing for system memory and CPU resources with the latest 3-D "The Matrix" screen saver? I think not.

The following chkconfig listing shows the services I have running in my runlevel 5:

  # chkconfig --list|grep 5:on|sort
  acpid          0:off   1:off    2:off   3:on   4:on   5:on   6:off
  anacron        0:off   1:off    2:on    3:on   4:on   5:on   6:off
  apmd           0:off   1:off    2:on    3:on   4:on   5:on   6:off
  atd            0:off   1:off    2:off   3:on   4:on   5:on   6:off
  autofs         0:off   1:off    2:off   3:on   4:on   5:on   6:off
  crond          0:off   1:off    2:on    3:on   4:on   5:on   6:off
  cups           0:off   1:off    2:on    3:on   4:on   5:on   6:off
  gpm            0:off   1:off    2:on    3:on   4:on   5:on   6:off
  httpd          0:off   1:off    2:on    3:on   4:on   5:on   6:off
  iptables       0:off   l:off    2:on    3:on   4:on   5:on   6:off
  irqbalance     0:off   1:off    2:off   3:on   4:on   5:on   6:off
  isdn           0:off   1:off    2:on    3:on   4:on   5:on   6:off
  kudzu          0:off   1:off    2:off   3:on   4:on   5:on   6:off
  messagebus     0:off   1:off    2:off   3:on   4:on   5:on   6:off
  microcode_ctl  0:off   1:off    2:on    3:on   4:on   5:on   6:off
  named          0:off   1:off    2:on    3:on   4:on   5:on   6:off
  netfs          0:off   1:off    2:off   3:on   4:on   5:on   6:off
  network        0:off   1:off    2:on    3:on   4:on   5:on   6:off
  nfslock        0:off   1:off    2:off   3:on   4:on   5:on   6:off
  pcmcia         0:off   1:off    2:on    3:on   4:on   5:on   6:off
  random         0:off   1:off    2:on    3:on   4:on   5:on   6:off
  rawdevices     0:off   1:off    2:off   3:on   4:on   5:on   6:off
  rhnsd          0:off   1:off    2:off   3:on   4:on   5:on   6:off
  sendmail       0:off   1:off    2:on    3:on   4:on   5:on   6:off
  smartd         0:off   1:off    2:on    3:on   4:on   5:on   6:off
  spamassassin   0:off   1:off    2:on    3:on   4:on   5:on   6:off
  sshd           0:off   1:off    2:on    3:on   4:on   5:on   6:off
  syslog         0:off   1:off    2:on    3:on   4:on   5:on   6:off
  touchpad       0:off   1:off    2:off   3:on   4:off  5:on   6:off
  vsftpd         0:off   1:off    2:on    3:on   4:on   5:on   6:off
  xfs            0:off   1:off    2:on    3:on   4:on   5:on   6:off
  xinetd         0:off   1:off    2:on    3:on   4:on   5:on   6:off

That's a lot of services configured to be running. Also, don't forget about all the xinetd spawned daemons. These subdaemons will either be configured to be on or off at the same runlevel that xinetd is configured to be on:

  #  export LC_ALL=C
  #  chkconfig --list| grep [a-z]: | grep on$| sort
           cups-lpd:               on
           finger:                 on
           imap:                   on
           imaps:                  on
           pop3s:                  on
           sgi_fam:                on

If any of the aforementioned services are unfamiliar, take note of them. We'll tell you how to look up and determine what unfamiliar services are, so that you can decide on whether to shut them down or not.

You should also check which services are bound to the network IP addresses, with the netstat command:

  # netstat -at
  Active Internet connections (servers and established)
  Proto Recv-Q Send-Q Local Address        Foreign Address         State
  tcp        0   0 *:imaps                 *:*                     LISTEN
  tcp        0   0 *:pop3s                 *:*                     LISTEN
  tcp        0   0 *:printer               *:*                     LISTEN
  tcp        0   0 *:pop3                  *:*                     LISTEN
  tcp        0   0 localhost.localdoma:783 *:*                     LISTEN
  tcp        0   0 *:imap                  *:*                     LISTEN
  tcp        0   0 *:finger                *:*                     LISTEN
  tcp        0   0 *:http                  *:*                     LISTEN
  tcp        0   0 *:ftp                   *:*                     LISTEN
  tcp        0   0 dav.mydomain.com:domain *:*                     LISTEN
  tcp        0   0 mydomain.com:domain     *:*                     LISTEN
  tcp        0   0 10.10.10.1:domain       *:*                     LISTEN
  tcp        0   0 192.168.128.25:domain   *:*                     LISTEN
  tcp        0   0 localhost.locald:domain *:*                     LISTEN
  tcp        0   0 *:ssh                   *:*                     LISTEN
  tcp        0   0 localhost.localdoma:ipp *:*                     LISTEN
  tcp        0   0 *:smtp                  *:*                     LISTEN
  tcp        0   0 localhost.localdom:rndc *:*                     LISTEN
  tcp        0   0 *:https                 *:*                     LISTEN
  tcp        0   0 localhost.localdoma:ipp localhost.localdo:32963 FIN_WAIT2
  tcp        1   0 192.168.128.25:32926    216.239.39.104:http     CLOSE_WAIT
  tcp        1   0 192.168.128.25:32925    216.239.57.99:http      CLOSE_WAIT
  tcp        1   0 localhost.localdo:32963 localhost.localdoma:ipp CLOSE_WAIT
  tcp        1   0 localhost.localdo:32921 localhost.localdoma:ipp CLOSE_WAIT

In this netstat, you see quite a few protocols listening on IP address 0.0.0.0, or in nonnetworking talk, all IPs. Do you want all of these services exposed to the outside? Again, take notes on these template machine settings.

First, you want to turn off the services that you don't want running in your default runlevel, or for good measure, runlevels 2 to 5. For example, browsing though the first chkconfig listing above, I know that the services autofs, cups, isdn, kudzu, and nfslock don't need to be running. Configure them to be off for all usable runlevels (2, 3, 4, and 5) with the chkconfig serviceoff command, and then stop the ones that are currently running by calling the daemon init script directly with the stop option; do this for each service that you don't want running:

  #  chkconfig autofs off; /etc/init.d/autofs stop
  Stopping automount:                [ OK ]
  #  chkconfig cups off ; /etc/init.d/cups stop
  Stopping cups:                     [ OK ]
  .
  .
  #  chkconfig nfslock off; /etc/init.d/nfslock stop
  Stopping NFS statd:                [FAILED]

Don't worry about FAILED messages. This is usually because of an interrelated service dependency or an issue that a reboot will clear up. Do another netstat -at if you're not sure whether you just locked clown those services.

And for xinetd subdaemons, if you don't want cups-lpd, finger, or imap running on your server load by default either, switching xinetd services to off will stop them immediately:

  #  chkconfig cups-lpd off
  #  chkconfig finger off
  #  chkconfig imap off

Now do a netstat listing again with netstat -at and see what services are now bound to your IPs. Then, do another chkconfig-list | grep5:on listing and now look as to what's configured to come up at boot time. Better?

Tip

If you're not sure what certain services do and you want a quick run down of each, Red Hat- and Fedora Core-based systems offer a quick, easy-to-use ncurses TUI (or text-based GUI) interface tool called ntsysv. After running ntsysv, it has a concise F1 help description for every service installed on your system. Invoke it as root by running ntsysv, or use the new, handsome, full GUI version redhat-config-services. It gives you a very good description of each service on your machine. You can read up and then decide whether to turn off each service or leave it running. WARNING: If you're not sure about a given service, ask someone or leave it on until you learn more.

So the chkconfig service off ; /etc/init.d/service stop commands that you just ran to lock down the services on this template server did a good job. Now copy/paste them into a text file called /root/post-script.txt or the like for eventual inclusion in your ks.cfg file later. Your file should look something like the following:

  # Shut off all the services we don't want running....
  chkconfig autofs off ; /etc/init.d/autofs stop
  chkconfig cups off ; /etc/init.d/cups stop
  chkconfig isdn off ; /etc/init.d/isdn stop
  chkconfig kudzu off ; /etc/init.d/kudzu stop
  chkconfig nfslock off ; /etc/init.d/nfslock stop
  chkconfig cups-lpd off
  chkconfig finger off
  chkconfig imap off

Now that you have a basic starting place for your ks.cfg file's %post config, let's continue to see what you can do to tighten your kickstart-based OS install.

Locking Down Networking/Ip Stack Settings

Professionals who know Linux love it when it comes to networking because you have control over the networking and TCP/IP stack more so than any other production operating system. You interact with networking and the IPv4 settings in a number of ways (such as with iptables, as you'll see in Chapter 11), but when dealing with the real-time kernel settings, you're usually reading and writing the settings to some kernel variable under the the Procfs, aka the /proc virtual filesystem. There are a lot of settings for the IPv4 networking system under /proc. Most of these /proc parameters can be controlled very nicely with the sysctl system via the /etc/sysctl.conf file. This file is run every time the system boots, and is the more modern and safer way of working with /proc changes (as compared to the old school method of using post-boot echo variable> /proc/sys/... commands in the rc.local file). If you need any kernel run-time parameters changed, do it here when possible. Many times when people need a kernel recompiled to make some adjustment to the kernel runtime parameters, for things like running Oracle, memory hungry apps, or specialized networking apps, sysctl is what they really need (since it will remain active even after a kernel upgrade or the like).

Note

In the old days, administrators used to control the kernel settings either with a kernel recompile or via /proc directly by echoing changes out to specific settings within proc from the rc.local startup file, as in this snippet:

   # stops SYN flood attacks
   echo 1 > /proc/sys/net/ipv4/tcp_syncookies

This is old school, rather dirty, and not really safe since most administrators did this when the /etc/rc.d/rc.local file was run, which is very late in the boot process. This creates time-related vulnerabilities, and in some cases (such as Oracle) simply cannot be done. Instead of echoing values like this out to /proc, simply poke such values into the /etc/sysctl.conf file. Read the man sysctl.conf page for more information.

To help secure your TCP/IP stack on a standard Linux mahcine, you will want to add the following lines to your /etc/sysctl.conf file:

   ##TWW: 2004-02-22
   # stop syn-flood attacks
   net.ipv4.tcp_syncookies = 1

   # Throws off people trying to fingerprint your OS by its TTL (def=64)
   net.ipv4.ip_default_ttl = 61
   # stops the routing table from being modified by packets non-sourc routed
   net.ipv4.conf.all.accept_redirects = 0

   # same as above but stops your system doing the same to others
   net.ipv4.conf.all.send_redirects = 0

You can add this to your script by storing these settings on a local FTP server (that the kickstart will have access to), and in the %post config pull them down from your ftp.example.com FTP server (for example) and append it on the end of the existing /etc/sysctl.conf:

   # Get all of our sysctl/networking adjustments
   wget ftp://ftp.example.com/pub/installs/sysctl - append
   cat sysctl-append >> /etc/sysctl.conf

So with this in your %post config script, you will append the contents of your standard sysctl-append file to the end of your new install's /etc/sysctl.conf file, thus enabling these settings every time one of these systems is booted-a very nice way to ensure all of your systems are just that much more secure.

Additional System Lock Downs

Here are some common security measures that some system administrators like to see in place on their servers. These are basically a collection of shell-based commands that will help lock down and secure various aspects of the system.

   # restricts those not in the root group from being able to su to root
   chmod 4750 /bin/su

   # restricts the mount/umount commands to root group only users
   chmod 4750 /bin/mount
   chmod 4750 /bin/umount

   #Install "scponly" scp replacement for bash and add to /etc/shells
   rpm -ivh ftp://ftp.example.com/pub/installs/scponly-3.8-1.i586.rpm
   echo /usr/bin/scponly >>/etc/shells

   #make scponly the default user shell
   wget ftp://ftp.example.com/pub/installs/useradd
   mv -f useradd /etc/default/useradd

And finally, here are some details we would like to add on all servers that get put online with our kickstart file:

   #Send an email whenever this box is rebooted
   echo "echo | mail -s\ "Server \ $HOSTNAME Rebooted\ " \
   pager@mail.example.com">>/etc/rc.d/rc.local
   #Document which kickstart file this is:
   echo ks-server-install_2004-02-22.crg > /etc/ks-release

Note that the Rebooted line will work with either the trailing \ on the first line or as a single unwrapped line without the trailing \.

These two lines are very nice. The first will send you an e-mail whenever one of the servers that it's installed on is rebooted, and the second will allow you to track which kickstart created a given server.

You may want to add other lock clown measures, such as these:

  • Adding SSH-key based authentication (Chapter 18)

  • iptable/firewall service restrictions by network (Chapter 11)

  • On-disk hardware testers such as memtest86 (www.memtest86.com), smartctl (http://smartmontools.sourceforge.net), or cpuburnin (http://users.bigpond.net.au/cpuburn/)

You can add these in your %postscript and test them out.

The only real limit to what you can do with the %post config scripts in an enterprise environment is how general you want your ks.cfg files to be (that is, how many or few you wish to maintain), and how much time you have to spend on fine-tuning them.

Tip

If you want good support for getting great lock-down information from other departments, make their system administrators and professionals a part of the effort. Show them what is possible with package adds, lock downs, security settings, and so on-and let them "own" their own little section of the script that deals with their work (security, FTP/Web admin, DNS admins, backup folks, and so forth). Not only will you get better acceptance of this methodology and be including others in setting de facto policy, but you'll get some really great scripts out of it!

The Final %postscript

Frankly, you can spend years tweaking this type of script. Let's take a look at what we have after all the changes thus far. I have also added some sections that are "works in progress." Some of these sections are commented out but left in so that you can see how they work:

   %post
   ###########Setup###########
   exec < /dev/tty6 > /dev/tty6 2>/dev/tty6 #Set up chvt to handle VTtoggles
   chvt 6                                   #put this in all post-scripts
   mkdir /tmp/install                       #Set up temp file space
   #cd /tmp/install
   #######iptable-ssh from MY LAN#######
   # May have to do this in the "first boot" after install
   #/etc/init.d/iptables restart
   #iptables -I RH-Firewall-1-INPUT 7 -s 10.1.1.0/24 -p \
   #tcp --dport 22 -j ACCEPT
   #iptables-save >/etc/sysconfig/iptables
   ###########Services Stuff###########
   # Shut off all the services we don't want running....
   chkconfig autofs off ; /etc/init.d/autofs stop
   chkconfig cups off ; /etc/init.d/cups stop
   chkconfig isdn off : /etc/init.d/isdn stop
   chkconfig kudzu off ; /etc/init.d/kudzu stop
   chkconfig nfslock off ; /etc/init.d/nfslock stop
   chkconfig cups-lpd off                    #These are xinetd based
   chkconfig finger off
   chkconfig imap off
   ###########Networking############
   # Get all of our sysctl/networking adjustments
   service network start                      #Turn on the network
   wget ftp://ftp.example.com/pub/installs/sysctl-append
   cat sysctl-append >> /etc/sysctl.conf
   ###########System Stuff###########
   # Only root group has access to su
   chmod 4750 /bin/su
   # Only root group users have mount/umount
   chmod 4750 /bin/mount
   chmod 4750 /bin/umount
   # Install "scponly" scp replacement for bash
   rpm -ivh ftp://ftp.example.com/pub/installs/scponly-3.8-1.i586.rpm
   # And add it to /etc/shells
   echo /usr/bin/scponly >>/etc/shells
   # Make scponly the default user shell
   wget ftp://ftp.example.com/pub/installs/useradd
   mv -f useradd /etc/default/useradd
   ###########Intrustion Detection########
   ## Need to get latest chkrootkit and put here....
   ## Get from http://www.chkrootkit.org/ and
   #wget ftp://ftp.pangeia.com.br/pub/seg/pac/chkrootkit.tar.gz
   # Untar it:
   #tar xzvf chkrootkit.tar.gz
   # Compile it:
   #cd chkrootkit-*/
   #make sense
   #cd .. ; my chkrootkit-* /root/
   # Clean up
   #/bin/rm -rf chkrootk*
   # Can also set up automation scan & updates via cron
   # Also see chapter 10 on how to "hide" chkrootkit for
   # better post-hack trustworthiness of the binary.
   #########AutoPatching/Up2date###########
   # uncomment when we're ready to use with registration
   #up2date-nox -uf
   ###########Final Custom Config########
   # Send an email whenever this box is rebooted
   echo "echo | mail -s\ "Server \ $HOSTNAME Rebooted\ " \
   pager@mail.example.com">>/etc/rc.d/rc.local
   #Document which kickstart file this is:
   echo ks-server-install_2004-02-22.cfg > /etc/ks-release
   ###########CLEAN UP##############
   cd /root
   /bin/rm -rf /tmp/install
   echo | mail --s"Done With Kickstart on $HOST" admin@example.com
   chvt 1                                #Tear down the fancy chvt
Note

Installing things like chkrootkit at install time is rather cool, but ultimately not very useful. This is because once a box is cracked (a.k.a. hacked), you cannot trust anything on the box. Many of the newer rootkits out there can change the way the kernel works and modify system memory (infecting seemingly secure parts of the system) systemically. The chkrootkit install in this kickstart %postsection is just a demonstration to just let you see what's possible from within this environment (on the fly downloading, decompressing, and compiling and installation of a package). For more information on the modern issues of crackers and the threats of rootkits, see this article: http://neworder.box.sk/newsread.php?newsid=4182.

That's it. Save the changes to the %post config area of your ks-server.cfg file to do some testing in the next section.


Creating a Kickstart Provisioning Server

You're going to love this part. All you have to do to create a kickstart server is to set up an FTP, HTTP, or NFS server and dump the CDs on it in an easy-to-reach place. It can even be password protected if you want.

All the footwork is really done on the client side with our boot media, anaconda, and our ks.cfg files.

Tip

Some administrators who like to centrally administer their kickstart files and who don't want a lot of outdated kickstart floppies or the like out there floating around, put their actual production ks.cfg files out on the network in a centralized place. This way when they make an update to the centralized ks.cfg file(s), all new installs make use of them, regardless of what old boot disks are out there.

The first step to setting up your new kickstart content server is to put the install content of the three Fedora Core CDs in place on the FTP server.

  1. Sit clown at your FTP server with the Fedora Core install CDs in hand.

  2. Insert Fedora Core CD 1 and run the following commands:

    # mount /mnt/cdrom/
# cp -af /mnt/cdrom/* /var/ftp/pub/
# eject

  3. Insert Fedora Core CD 2 and run these commands:

    # mount /mnt/cdrom/
# cp -af /mnt/cdrom/Fedora/ /var/ftp/pub/
# eject

    Insert Fedora Core CD 3 and run the same commands as with CD 2.

    Note

    Depending on your distro, you may not need to issue the mount commands. Some newer Linux distros have automounting services configured to do this for you.

  4. Clean up all the TBL files in the FTP area:

    # find /var/ftp/pub/ -name "*.TBL" -exec rm {} \;

  5. Make sure that your FTP daemon is on and configured:

    # /etc/init.d/vsftpd start ; chkconfig vsftpd on
Starting vsftpd for vsftpd:                    [ OK ]
#chkconfig --list vsftpd
vsftpd     0:off  1:off   2:on  3:on  4:on 5:on 6:off

Congratulations, you've just set up a Fedora Core Linux Kickstart Provisioning Server! Setting up Red Hat distro kickstart servers such as RH 9 or Red Hat Enterprise Linux; these are identical to Fedora except that they use the directory name RedHat instead of Fedora.

This can also be done via username-protected FTP space or even via login-protected httpd/web share.


Centralizing and Standardizing Your Kickstart Files

Now that you have the CD content in place, I recommend also setting up your ks.cfg flies for your server, workstation, laptop loads, and so on out on a centralized web share with standard link names that point to the latest version (you'll see). Some place that anyone can get to (unless you have security concerns, and then it can be password protected). This will enable you to literally install a desktop from around the world over the Internet if the FTP file share and web share of ks.cfg files are available, all with nothing more than a network connection and a boot floppy. Very powerful. Very nice.

To make web-based file shares where our own (non-CD) kickstart-related files will go, simply make a directory on the web server called installs (we're using ssh remotely to do this for us) and then copy (scp) the ks.cfg files over to your web server's default website under your installs directory:

   #  ssh root@www.example.com mkdir /var/www/html/installs
   root@www.example.com's password:
   #  scp ks-files/ks-*.cfg root@www.example.com:/var/www/html/installs/
   root@www.example.com's password:
   ks-everything.cfg                             100% 1812    67.2KB/s    00:00
   ks-IT-desktop_2004-02-16.cfg                  100% 1446    98.0KB/s    00:00
   ks-IT-desktop_2004-02-20.cfg                  100% 1446     1.7MB/s    00:00
   ks-my-first-install_2004-02-21.cfg            100% 1877    89.0KB/s    00:00
   ks-server-laptop-install_2004-02-22.cfg       100% 1877     1.9MB/s    00:00
   ks-workstation_2004-02-19.cfg                 100% 1446     1.5MB/s    00:00
   ks-workstation_2004-02-20.cfg                 100% 1446     1.5MB/s    00:00
Note

In the kickstart example in this chapter we are using a centralized ks.cfg file that is accessed via HTTP. Once you load your ks.cfg file (either locally from disk or remotely) and get the kickstart up, you see (as somewhat seen in our ks.cfg file) how you can access data, scripts, and packages located on other data shares including HTTP, NFS, and FTP. This is done just to illustrate the various ways in which you can load your data over the network. So if you're going to set up your own kickstart server, you can mix and match data stores of NFS, FTP, or HTTP throughout the install process, or keep them in one unified share such as HTTP or NFS. It's up to you.

And then log in to the web server and set up standard symlink pointers from standard names that you reference from the client side of the kickstart, to point to the specific file names that you change, update, and have date stamps on the centralized web server:

   # ssh root@www.example.com
   root@www.example.com's password:
   Last login: Thu Feb 19 23:17:12 2004
   # cd /var/www/html/installs/
   #
   #  ls
   ks-everything.cfg                   ks-server-laptop-install_2004-02-22.cfg
   ks-IT-desktop_2004-02-16.cfg        ks-workstation_2004-02-19.cfg
   ks-IT-desktop_2004-02-20.cfg        ks-workstation_2004-02-20.cfg
   ks-my-first-install_2004-02-21.cfg

Set up your symlink pointers:

   # ln -s ks-server-laptop-install_2004-02-22.cfg ks-server.cfg
   # ln -s ks-workstation_2004-02-20.cfg ks-workstation.cfg
   # ln -s ks-IT-desktop_2004-02-20.cfg ks-desktop.cfg
   # ls -la | tr -s " "| cut -f9,10,11 -d" "
   .
   ..
   ks-desktop.cfg -> ks-IT-desktop_2004-02-20.cfg
   ks-everything.cfg
   ks-IT-desktop_2004-02-16.cfg
   ks-IT-desktop_2004-02-20.cfg
   ks-my-first-install_2004-02-21.cfg
   ks-server.cfg -> ks-server-laptop-install_2004-02-22.cfg
   ks-server-laptop-install_2004-02-22.cfg
   ks-workstation_2004-02-19.cfg
   ks-workstation_2004-02-20.cfg
   ks-workstation.cfg -> ks-workstation_2004-02-20.cfg
   # exit

The ls command was to keep the output clean and understandable. The point here is how the standard names that you are going to reference from your kickstart clients will be standardized file names, which are simply pointers to the actual newest version of each particular ks.cfg file.

Now your client site boot media will be able to point to standard ks.cfg file names like

  • http://www.example.com/installs/ks-server.cfg

or

  • http://www.example.com/installs/ks-workstation.cfg

while keeping some form of sanity and organization on the back side of things. Next comes the fun part: creating and testing the client-side boot media.


Creating Your Kickstart Provisioning Boot Media

You've got the kickstart server set up. You've modified the various ks.cfg files and have ks-server.cfg file and a ks-workstation.cfg files ready to be used remotely. Now let's take a moment and talk about Red Hat's (and now Fedora Core's) standards for client side, kickstart install boot media.

Caution

Many people who set up production kickstart servers put the ks.cfg files on the boot media itself. In practice, especially the IT environment, this is begging for trouble. With this setup, every time you make an update to a ks.cfg file, you have to contact everyone in the company and get them to download your latest and greatest boot floppy or boot CD and discard or erase their old boot disks with the old ks.cfg files. In short, it doesn't work, and those old boot floppies just keep popping up and messing things up for everyone. Just take a tip from those who have gone before you and centralize your ks.cfg files-pointing to standardized web reference names (or links) from your modified and automated boot media. The result is that by doing it this way, your "boot media version" is now irrelevant. The only better solution is either a homegrown integrated PXE-LAN environment (diskless kickstarts) OS provisioning system or Red Hat's own Satellite/Provisioning server, which huge provisioning shops like Rackspace might run. For more information on the Red Hat servers, see www.redhat.com/software/rhn/table/#architecture

Finding Red Hat Boot Images

Red Hat boot floppies used to be divided into local media boot floppy images called boot.img and network-capable boot images called bootnet.img. These can still be seen in the older (nonenterprise) sections of Red Hat mirror sites such as the one we maintain at Rackspace:

  • ftp://mirror.rackspace.com/redhat/linux/7.3/en/os/i386/images/

This system was nice, but with all the motherboard chip sets, IDE controllers, and network card drivers, Red Hat quickly ran out of room on that bootnet.img floppy image.

To remedy this, Red Hat now provides only one boot floppy image, called bootdisk.img. It allows for basic local HD- and CD-based installs to IDE drives. For network installs or installs that require network drivers (FTP, HTTP, NFS installs), SCSI/RAID or PCMICA mass storage drivers (that is, any third party drivers), bootdisk.img needs to load the drivers (after booting from bootdisk.img) from either the drvnet.img, drvblock.img, or pcmciadd.img driver disk, respectively. This process can be a nuisance, so Red Hat also has a bootable CD image that has all of these drivers built in. Figure 4-2 shows the boot screen from the bootdisk.img or the boot.iso (CD).

Click To expand
Figure 4-2: The Fedora Core CD/floppy/PXE boot/install screen.

These boot media images are available on all Fedora Core FTP mirror sites as well as on Fedora Core CD 1. See Table 4-1 for a description of what each image does.

Table 4-1: Fedora Core Boot Images on CD and Mirror Sites
Media Files | Description
Bootdisk.img | Required boot floppy image for Fedora Core installs from CD or hard drive media. Also base boot for network and block device (SCSI/RAID) installs (1.44M).
Drvblock.img | Supplemental block device drivers such as storage controllers, SCSI, and RAID controllers (1.44M).
Drvnet.img | Supplemental network device drivers such as NIC (network interface card) drivers (1.44M).
Pcmciadd.img | Supplemental PCMCIA/PC-Card driver disk (1.44M)
Pxeboot/ | The directory of files to put on your PXE/DHCP server to allow pure network-based boot and installs (~3-4M)
Boot.iso | A standalone CD image that includes all the boot and driver info from bootdisk.img, the drvblock.img, drvnet.img, and the pcmpciadd.img floppy images. Good for doing machine boots for single media network installs.

To get a listing of Fedora Core mirror sites where you can download these boot images, go to http://fedora.redhat.com/download/mirrors.html.

Considering Network Install Options

There are three general boot options for performing a network install: floppy disks, CDs, and PXE.

  • Stock or custom boot floppy network installs: This type of installation uses the bootdisk.img floppy image, along with one to three standard driver disks. These driver disks enable the install time environment to get to network or media that has other install-related files (such as kickstart config files). Within such floppy-based installations you have two alternatives:

    • Stock: Use the stock bootdisk.img boot floppy image and whichever of the three driver disks you might require (block device driver, network driver, and PCMCIA driver disk images). This method can require manually swapping floppies at least once, so this makes for a poor automated install process. You also have to make your clients manually point the boot/install process to a centralized network share of your kickstart config file(s) where all of your install parameters are defined. This method is very prone to typos and not very user friendly.

    • Slightly Customized: This method uses the same one to four driver floppies as with the Stock method, but you slightly modify the boot disk to have your boot config file automatically point to the correct network location for the kickstart config file(s). Of the two main forms of floppy installs, this tends to be the more popular since it is more automated.

      Note

      There is a third, less common method that can totally automate the entire install process all from a single boot floppy (with network and other drivers built-in), but it is technically very complex and beyond the scope of this chapter. This is a method by which you can make your own custom, automated, single, hybrid boot/driver floppy with all the drivers that you need. This single boot/install/driver floppy has your own custom initrd (initial RAM disk) file needed to automate everything from a single boot disk. If you don't know what an initrd file is, then you probably don't want to even think about this option. If you're still interested more information on this useful subject can be found at http://linuxcourse.rutgers.edu/bootdisk.html.

      The fact is we are more and more moving away from floppies as a common media, especially for specialized boot environments like kickstart. Some newer laptops no longer come with floppy drives. Keep in mind, however, that if you prefer using floppies, the first of these alternatives is simpler for you (creating the floppies), but more difficult and error prone for the end user doing the install. The second method is probably the way most people should do floppy-based network installs, as it is much easier for the end user, still requires a bit of floppy swapping by the user, but only slight customization on your behalf. Details on the second option are covered in the Creating Customized Boot Floppies section later in the chapter.

    Note

    It should also be noted that while it is considered a technically bad management practice (in an enterprise at least) to put your ks.cfg file(s) on the floppy itself, this is still an option available to you. If you decide to do this, the boot option to point anaconda to the floppy for the ks.cfg file from the boot: prompt is linux ks=floppy.

  • Stock or Custom Boot CD: Use the boot.iso CD image, and either have your clients manually point to a centralized HTTP share of your ks.cfg files or make your own slightly customized CD image that totally automates this. Either way, you have a single media network install solution. These procedures are covered in the Making a Network Install Boot CD section.

  • LAN PXE boot solution: This solution is available only if all of your people have a PXE-capable network card and have the card and machines BIOS set up for PXE network booting. Since most IT administrators do not have access (or clearance) to set up a full PXE boot environment, and since this configuration would also have to be duplicated across all company LANs (since PXE/DHCP/bootp is a LAN broadcast only solution), this solution falls outside the scope of this book. If interested in this option, see www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/sysadmin-guide/ch-pxe.html.

Given all this, the boot CD seems to be the easiest and more forward-compatible network install option for doing Red Hat Enterprise or Fedora Core kickstart/anaconda installs. Since all CDs and machines are pretty much bootable nowadays, it's usually easier to either use a stock boot.iso boot CD and point it manually to your network-based ks.cfg file(s) or to modify the boot.iso image for automatic kickstarts for fire-and-forget installs. Rebuilding the floppy image with a single disk hybrid OS/kernel/initrd setup is beyond the scope of this chapter, but we give a link to more information on doing this in one of the aforementioned notes. Let's start with the older and more unruly to manage multifloppies method, and then we'll turn to the more elegant modern CDs method.

Creating Customized Boot Floppies

If you want to do your network-based OS install using the three floppy images, first copy the bootdisk.img, drvnet.img, and drvblock.img, from the CD 1 (in Fedora/images/) or from a mirror site. Then follow these steps:

  1. Write your floppy images out to three blank/formatted disks using dd:

    # dd if=bootdisk.img of=/dev/fd0

# dd if=drvnet.img of=/dev/fd0

# dd if=drvblock.img of=/dev/fd0
    Note

    The drvblock floppy is needed only if you're installing to or from mass media storage such as SCSI, RAID, or specialized storage hardware. It's not bad to have, but usually not required on most PCs.

  2. Insert and mount the bootdisk floppy:

    #  mount /dev/fd0 /mnt/floppy/
#  cd /mnt/floppy/
#  ls
boot.msg       initrd.img    options.msg   rescue.msg   splash.lss   vmlinuz
general.msg    ldlinux.sys   param.msg     snake.msg    syslinux.cfg

    Now with your new boot floppy mounted, you can start editing and customizing it to point to your kickstart files out on the network.

    Tip

    If you don't want to have to do this every time you want another identical boot disk, then after you work all the bugs out and tweak out your bootdisk just so, you can loopback mount the actual img file as a loopback file system (with the -o loop mount option):

       # mount -o loop,wr bootdisk.img /mnt/floppy/

    cd into /mnt/floppy and edit away on your "phantom floppy" image. Then when you're done:

      # cd
  # umount /mnt/floppy

    and from then on when you dd a copy of the img file to a disk you will have all of your changes! Try doing that on a stock Windows box!

  3. After mounting your new bootdisk floppy, you need to make some space on the disk by deleting the .msg files:

    # /bin/rm *.msg
    Tip

    Keep a few of these .msg files around. These would be handy later when you decide that you want an interactive user menu with directions and help options.

  4. Now edit the syslinux.cfg file to look like this and add the bold bit (all on one line):

    default linux
prompt 1
timeout 1
label linux
  kernel vmlinuz
  append initrd=initrd.img ramdisk_size=8192
          ks=http://example.com/installs/ks-server.cfg
          ksdevice=eth0 dd <--- Note

label memtest86
  kernel memtest
  append -
    Note

    The trailing dd boot option in the isolinux.cfg file has nothing to do with the dd command that we used earlier. The dd boot option is what tells anaconda to ask for the drvnet and/or drvblock driver disks so that it can either get online or install SCSI or RAID card drivers-must in our case since we need the dvrnet disk's network card drivers to get online and our kickstart file! This required "floppy swapping" illustrates how floppy's time has come and gone, and we will see how using the single boot CD boot.iso is a much nicer and more modern option.

  5. After you're done with your edits, cd back home and unmount the floppy:

    # cd
# umount /mnt/floppy
Tip

You can keep the boot.msg if you want to have user messages and a time out with pointers to the various ks.cfg files that you have on your web server. It's up to you, but just watch the space on the floppy. You may have to do your editing off disk and then copy your final edits back over.

Also, if you use an editor like vi or vim (which most of you will), and you want to quickly do your edits in place (on the floppy), then invoke vi/vim like this: vi -n/mnt/floppy/syslinux.cfg. The -n will keep a temporary swap file from being created by vim (which can create really problems when trying to edit files when you have very little elbow room).

Finally, unlike most UNIX/Linux systems, these config files are standard (or nonstandard) MS-DOS CR/LF text files, so you can actually get away with using DOS/Windows type editors on them, such as notepad. The filesystem is also FAT(12) so it can be accessed from just about any OS without much of a problem.

Now take your new bootdisk and driver disks to a test machine and see how they work!

Start Sidebar
Floppies. Laptops, and Kernals

A couple things to know about laptops: If you expect to use the PCMCIA or PCCard slots during the install, you will also need the pcmcia.img floppy driver disk (unless you're using the CD boot media, which includes it). If you don't need the PCMCIA hardware during install time, don't worry about it. After the postboot OS downloads kicks in and begins to run, the same Fedora installer that installs from CD will be running, and will detect any supported hardware and install the correct drivers for your machine. pcmcia.img (as well as the other driver disks) is just an install time thing in case you need a PCMCIA network card, external CD-ROM, or SCSI/RAID driver support to gain access to the ks.cfg file and the remaining bootstrap image itself.

This being said, there have been some strange problems with Fedora Core's support of PCMCIA/PC-card controller technology as well as with some older network cards. If you are experiencing similar hardware problems, monitor the buzilla bug lists at https://bugzilla.redhat.com/bugzilla/query.cgi?format=simple.

After going to this URL, just search on the "Product: Fedora Core," "Version: <your version>", and "Component: kernel-pcmcia-cs" for an up-to-date status of these issues. It was still in a state of flux at the time of this writing.

On newer laptops, if you also experience problems with battery monitoring, PCMCIA/PCCard hardware, network service lockups, and other such Laptop-isms, try adding the acpi=on kernel boot option to your grub.conf kernel line (the one below is wrapped):

   title Fedora Core (2.4.22-1.2129.nptl)
           root (hd0,1)
           kernel /vmlinuz-2.4.22-1.2129.nptl ro root=LABEL=/
              hdc=ide-scsi acpi=on
           initrd /initrd-2.4.22-1.2129.nptl.img

Adding the acpi=on will ensure that your laptop's ACPI functions will be enabled. As with Fedora Core (1 at least) this kernel option is compiled to be off by default. Additionally, if you have a built-in CDRW drive and you're having problems writing to it, you may also want to add the hdX=ide-scsi option (where X is the device letter of your CDRW drive). Hopefully, many of these issues will be fixed in newer versions of Fedora Core and by the next release of Enterprise Linux.

End Sidebar

Making a Network Install Boot CD

Given the problems with floppy-based network kickstart installs, let's look at the CD option via the stock CD boot image boot.iso from our own CD 1's path Fedora/images/boot.iso.

Note

You can get this CD boot image boot.iso from one of the Fedora Core mirror sites if you don't have the Fedora stock install CDs handy.

We're going to cover a couple of different ways of installing from a CD-an easy (stock) way and a not-so-easy (custom) way. The easy way is, well, easier for the admin upfront, but it forces the clients (or installer person) to type in special pointers to point to your centralized ks.cfg file(s) out on the network. The second, not-so-easy, way is cool because it requires no intervention from the install person at all (other than inserting a CD), but it takes a lot more footwork on your part.

Creating a Boot CD the Easy Way

First mount CD 1 and copy the stock boot.iso file from the Fedora/images/ directory:

   # cd /root
   # mkdir tmp ; cd tmp
   # cp -a /mnt/cdrom/images/boot.iso

Next, insert a blank CD-R into your burner. Since this is only a 3 to -4MB image file, I recommend using the small "business card" CD-Rs (more impressive too). Proceed to burn the boot.iso with the following command:

   # cdrecord dev=0,0 /root/tmp/boot.iso ; eject

See the man page for cdrecord and use cdrecord -scanbus to see your CDR/W drive id number, or if more convenient, iso files can also be burned by Windows-based CDR burning software such as Nero.

Start Sidebar
CD Burning Errors

If you try to burn your CD with cdrecord and get errors like this

   # cdrecord dev=0,0,0
   /var/ftp/pub/fedora/core/l/i386/os/images/boot.iso ; eject
   Cdrecord-Clone 2.01a19 (i686-redhat-linux-gnu) Copyright (C)
   1995-2003 Jrg Schilling
   scsidev: '0,0,0'
   scsibus: 0 target: 0 lun: 0
   Linux sg driver version: 3.1.25
   Error trying to open /dev/scd0 exclusively ... retrying in 1
   second.
   [...]
   Error trying to open /dev/scd0 exclusively ... retrying in 1
   second.
   Error trying to open /dev/scd0 exclusively ... retrying in 1
   second.
   bailing out..

then something else may have control of your CD-R device. Sometimes processes such as kscd or the like may have a hold on it. You can detect and kill such processes like this:

   # ps auxw|grep [k]scd
   tweeks 4187 0.0 4.1 24984 9272 ? S 15:09 0:00 kscd -session
   117f000001000107214522900000042380008_1080711996_7
   26640
   # killall kscd
   # ps auxw | grep [k]scd

Now your cdrecord session should work fine.

End Sidebar

Next, try a test boot/install on a test client machine. When you get to the boot prompt, type

Boot:  linux ks=http://  www.example.com/installs/ks-server.cfg ksdevice-eth0

substituting the http path where you've placed a copy of your ks-server.cfg file. (This is why it takes a bit more work on the client side. One typo by the client, and the install will fail to start.)

Creating a Boot CD the Not-So-Easy Way

You can make things easier for the person doing the client side of the install by creating your own boot.iso disk with the http:// pointer built in. To do this, you need to have the boot CD automatically have its ks= pointer pointing out to your web server's ks files. The following instructions show you how to do that:

  1. Mount the boot.iso image to some mount point (such as /mnt/tmp or even /mnt/cdrom):

    #  mount -o loop,rw /root/fedora-core-1-boot.iso /mnt/tmp
#  is -la /mnt/tmp/isolinux
total 4257
drwxr-xr-x      2 root      root         2048 Nov 3 17:45 .
drwxr-xr-x      3 root      root         2048 Nov 3 17:45 ..
-r- r--r--      1 root      root         2048 Nov 3 17:45 boot.cat
-rw-r--r--      2 root      root          292 Nov 3 17:45 boot.msg
-rw-r--r--      2 root      root         1034 Nov 3 17:45 general.msg
-rw-r--r--      4 root      root      3012543 Nov 3 17:45 initrd.img
-r--r--r--      2 root      root         9428 Nov 3 17:45 isolinux.bin
-r-xr-xr-x      2 root      root          569 Nov 3 17:45 isolinux.cfg
-r--r--r--      2 root      root        84420 Nov 3 17:45 memtest
-rw-r--r--      2 root      root          788 Nov 3 17:45 options.msg
-rwr--r--r--    2 root      root          872 Nov 3 17:45 param.msg
-rw-r--r--      2 root      root          490 Nov 3 17:45 rescue.msg
-rw-r--r--      2 root      root          549 Nov 3 17:45 snake.msg
-rw-r--r--      2 root      root         6773 Nov 3 17:45 splash.lss
-r--r--r--      1 root      root         2880 Nov 3 17:45 TRANS.TBL
-rw-r--r--      4 root      root      1228344 Nov 3 17:45 vmlinuz

  2. Copy all the files to a "scratch area" for editing:

    #  cd /root
#  mkdir boot-iso-files
#  cp -a /mnt/tmp/* boot-iso-files/
#  umount /mnt/tmp
#  cd boot-iso-files/
#  cd isolinux/
#  ls
boot.cat    initrd.img   memtest     rescue.msg TRANS.TBL
boot.msg    isolinux.bin options.msg snake.msg vmlinuz
general.msg isolinux.cfg param.msg   splash.lss
    Note

    The isolinux.bin, initrd.img, and vmlinuz together are what boots the system here. The inslinux.cfg is like a lilo.conf file that you're going to edit to provide control over the automatic ks=... parameters.

  3. Edit the isolinux.cfg file down to this and add the bold bit (all on one line):

    default linux
prompt 1
timeout 300            <--- Timeout in seconds before it starts
label linux
  kernel vmlinuz
  append initrd=initrd.img ramdisk_size=8192
     ks=http://example.com/installs/ks-server.cfg
     ksdevice=eth0

label memtest86
  kernel memtest
  append -

    Include and change the boot message files if you want.

    Caution

    The append line and the two following lines should all be a single line. Allow it to simply wrap around the screen. Do not break the line.
    Tip

    The memtest86 you see listed is a built-in hardware memory tester, one of the best out there for Linux. These CDs are also very useful for doing memory burn in tests. Remember this the next time you have some suspected bad RAM you need checked out.

  4. cd back one dir and make your new iso image to save out to my-server-boot.iso on your hard drive:

    #  cd ..
#  1s
isolinux TRANS.TBL
#
# mkisofs -o ../my-server-boot.iso -V 'Fedora Core Server
Install' -b isolinux/isolinux.bin -c isolinux/boot.cat
-no-emul-boot -boot-load-size -boot-info-table
 -R -J -T.

  5. Insert a blank CD-R disc and burn your new boot.iso image to CD:

    # ls -lah      my-server-boot.iso
-rw-r--r--   1 root   root         4.3M Feb 23 10:00 my-server-boot.iso
#  cdrecord dev=6,0 speed=6 my-server-boot.iso

When cdrecord finishes, you should be able to boot a test system from your new automated boot.iso disc. If your customized kickstart file is correctly located at http://example.com/installs/ks-server.cfg, as I had shared it out previously (or whatever URL/share you are using), then everything should autoinstall just fine with zero human intervention aside from inserting the CD and powering up.

Tip

These instructions just covered the bare essentials. You might want to spend a little extra time and create a network boot and install CD that can allow various install time options of using a server load, a desktop load, or a workstation load via different ks.cfg files-all located out on your http server via standardized symlink names. Then you can use the built-in menus (in isolinux.cfg) on the boot.iso to give the install person some more options, while you still centrally control what happens with each one of those loads from the ks.cfg links and files you have stored online at www.example.com/installs/.

More Floppy and CD Customization Information

For more information on customizing your floppy and CD images, look at the following links:

  • http://linuxcourse.rutgers.edu/bootdisk.html

  • www.linuxjournal.com/article.php?sid=6473

  • http://birger.sh/computing/RedHat_9_customized_install_media_howto.html

  • http://acd.ucar.edu/~fredrick/linux/linux-kickstart/


Testing Your New Kickstart Media

Testing your kickstart boot media is fairly simple. Pop in the bootdisk.img floppy or boot.iso CD, verify that the machine BIOS is set to boot off that media, that your networking hardware is all connected, and that you're on a DHCP LAN (if you're doing nonstatic IP network loads), and power up.

Floppy Installs

Remember, if you're using the bootdisk.img and the network, you're going to have the driver disk in hand.

If you're using a stock bootdisk.img-based floppy, before you get any drivers loaded up you will still need to tell the program from where to get its ks.cfg file. In this example, we manually point it to the web share that we've created when it gives up the boot prompt:

boot:  linux ks=http://  example.com/instalIs/ks-server.cfg ksdevice=eth0

You will then have to manually feed it the drvnet driver floppy so it can see the network and go to the ks.cfg file that you pointed it to.

If you're running a custom build (automated) floppy and configured your boot floppy correctly, it should default to the proper URL for the ks.cfg (even though you may not see it), load the syslinux boot environment, and then prompt you for your driver disk(s). At this time you'll need to start feeding it floppies for whatever hardware you need access to during the install (usually just the drvnet disk).

Regardless of the type of floppy install you're going with (manual or automatic), you will need to insert the pcmcia.img disk (if you have a PCMCIA-based network card) and/or your drvnet.img disk for network drivers. Also, if you are installing any block device or mass storage devices such as SCSI drives or RAID controllers or the like, you'll also want the dvrblock.img floppy handy.

After these drivers are all loaded and if you're running a customized/automated bootdisk.img floppy, the Anaconda installer should take off and start following your kickstart config file and scripts. No more intervention should be required.

Note

If the automation aspect of the installer stops somewhere and it starts prompting you for questions that are supposed to be automated (such as the keyboard, mouse, partition layout, and so on), this indicates that there is a problem in your kickstart config file. It's probably a typo, or maybe you left something commented out that shouldn't be (the clearpart or part lines are common mistakes). Go back and check your install script line by line.

CD Installs

CD installs are a bit nicer if you ask me. If you've set up your own custom boot.iso CD with the preconfigured ks.cfg network pointers, then this is really a fire-and-forget operation. Just be sure not to walk away and forget about it if you have configured the kickstart to reboot when it's done. Doing so would put the machine into a perpetual install-reboot-install-reboot loop that will most likely get you a visit from your NOC (network operation center) wanting to know who's been DoSing the network and eating up the network bandwidth all day. This type of event usually lands one a colorful nickname with your cohorts as well.

If you are using a stock boot.iso-based CD image, you will still have to give only one piece of information at the initial boot: prompt:

boot:  linux ks=http://example.com/installs/ks-server.cfg ksdevice=eth0
Tip

If your install does stop, hang, or start asking you questions that should be automated, use the virtual terminals detailed in Table 2-3 to look at the various parts of the install mechanism. After the install has begun, CTRL+ALT+F2 will give you a root bash prompt so you can see the hard drive where the files are being installed, poke around the kernel modules, config files, and so on. Just be aware that this shell is a very sparse minimalist implementation and is for basic troubleshooting only-no man pages or advanced functionality. However, you can do things like FTP out, download packages, and affect changes on the system. Just be careful. ALT+F3 gets you the install log, ALT+F4 all kernel messaging, and ALT+F5 all partitioning and formatting information. These can really help you when tracking down problems (and some times fix them) during your first few install troubleshooting sessions. Above all, take your time and read the logs to help you troubleshoot problems in your ks.cfg file. It usually takes several attempts to come up with a perfect install script.

You are now armed with all the information that you need to kickstart your own desktop, workstation, server, or customized systems automatically and from any network enabled PC.


Troubleshooting Common Kickstart Problems

Kickstarting systems entail a ton of client-side, boot media, network-related, and server-related issues. It can be a real bear to track clown kickstart provisioning issues even for experienced administrators, much less the newly initiated! Here are a few of the most common problems users face with kickstart, and some tips for resolving them.

  • How do I make a bootable floppy or bootable CD from the bootdisk.img or boot.iso file, respectively? If you have downloaded the bootdisk.img and other files, follow the procedures here to create a bootable disk.

    • To create a bootable floppy disk from Linux:

      1. insert a blank floppy

      2. type the following command:

        #  dd if=bootdisk.img of=/dev/fd0

    • To create a bootable floppy disk from Windows:

      1. type the following line to get the rawrite command from the install CD 1 (where D: is your CD-ROM drive):

        C:\temp\>  copy D:\dosutils\raw*.*

      2. Insert a blank/formatted floppy and run the following command with your disk image file:

        C:\temp\>  rawrite -f bootdisk.img

      3. Reboot

      4. If it does not boot off the floppy, verify that your BIOS is set up to boot from the floppy first and hard drive second, and then reboot.

    • To create a bootable CD from Linux with a blank CD-R in your burner, as root type

      #  cdrecord dev= 0 ,0 /root/boot.iso

      See the man page for cdrecord and use cdrecord -scanbus to see your CDR/W drive id number (the first zero in the preceding code).

    • To create a bootable CD from Windows use your favorite burning software to burn the boot.iso file (type ISO-9660) to a blank CDR

  • The system locks up when some people try to connect to the FTP shares on the kickstart .server. This can be caused by firewalls, or especially iptables, interfering with the underlying FTP, HTTP, or NFS-based protocols. Try getting around this problem by using an HTTP/web file share instead of an FTP share, or just by quickly shutting off iptables on the client, and then the server (if this is feasible). If the problem clears up, then this was the cause. If you're running FTP and want to try using HTTP quickly instead, just put your CD install files (the Fedora or Red Hat directories from your CDs) into a web directory like /var/www/html/kick-data and reconfigure your ks.cfg file's url -url line accordingly:

    .
install
url --url  http://example.com/kick-data/
lang en_US.UTF-8
.
.

    Put the ks.cfg file back on the floppy or on your network share and try it again.

  • My kickstart gets to the same place every time and then just stops. It never starts actually installing files. The problem could be a number of variables, from a typo in your ks.cfg file to problems with the network, DNS, your server, or firewalls. One great troubleshooting method is to use the virtual terminals to switch over and see what's going on. Remember to use (CTRL+)ALT+F2 to get to a bash prompt to poke around the system while it's being installed, ALT+F3 to view the actual install log, ALT+F4 to see all kernel messaging as it's happening (good to identify many errors), and ALT+F5 to see any partitioning and formatting information in real time. ALT+F7 will take you back to the GUI install screen (if GUI install mode is being used). If you are not able to change even virtual terminals, then you're not all the way through the network boot stage and you probably have a typo on your kickstart config file that is preventing it from coming all the way up (look specifically at your url -url and network lines).

  • My kickstart always dumps me out of the automated install and makes me enter the keyboard type, network settings, and media install source (CDROM, FTP, NFS, and so on)-just as if I were doing a completely manual install. This is a clear indicator that you have a problem either in the syslinux.cfg/isolinux.cfg file not pointing to your ks.cfg file correctly (either locally or over the network), or there is a typo in your ks.cfg file. Check them all and follow the flow of pointers.

  • My kickstart floppy install just keeps asking me for driver disks, never loads my network card driver, and never allows me to even get a DHCP address to get on the network! This is usually associated with unsupported hardware or hardware that is having problems being autodetected by the syslinux bootstrap environment. The problem is sometimes as simple as reseating your network card, or setting the machine's BIOS to reissue all of the PCI bus IRQs. Sometimes it will start working by simply moving a PCI NIC (and telling the BIOS to shuffle IRQs), because a PCI shared IRQ bug with your PCI/motherboard chipset and your cards. This latter problem seems to crop up more in systems with mass storage cards such as SCSI or RAID cards, but you might see this also with other types of cards such as sound cards, which have several IRQs. Try the hardware IRQ tricks first. If this does not clear things up, also try the boot: prompt kernel modifier variations of acpi=on, pci=biosirq , and pci=noacpi. If nothing works there, then it's time to hit Google, or visit http://bugzilla.redhat.com/.

  • My kickstart starts off just fine, and then half way through the install I get an error message saying something about unresolved dependencies. This is usually because you manually added RPM packages to the %packages section of your ks.cfg file. If you're not sure to also install any packages that that package may need, then you can get this error. The proper thing to do is to check out the dependencies a given package has before adding it to your kickstart (check this via rpm -q -requires <packagename>). That being said, the quick fix is to add the ks.cfg option %packages -resolvedeps to the packages line, which will hunt through the XML and RPM dependency lists to discover everything you need to install a given package, get those packages and install them first, as well as any of their sub-dependencies! A very nice change from the old days of manually tracking this stuff down.


Kickstart Troubleshooting References

It's hard to find good information on kickstart installs if you don't even know what questions to ask. So Table 4-2 contains a few time-tried resources that will help round out some of the more detailed aspects and options of setting up a kickstart provisioning environment.

Table 4-2: Kickstart Troubleshooting References
Reference | Location (addresses should be one single line)
Red Hat EL3 System Administration Guide: Ch9, Kickstart Installations | www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/sysadmin-guide/ch-kickstart2.html
Red Hat EL3 System Administration Guide: Ch10, Kickstart Configuration | www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/sysadmin-guide/ch-redhat-config-kickstart.html
Red Hat EL3 Security Guide | www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/security-guide/
Creating a boot disk for unattended kickstart installation of RH 9: Great technical info on creating your own customer-driver initrd for a single boot/install kickstart floppy. | http://linuxcourse.rutgers.edu/bootdisk.html


Summary

The ability to standardize and automate your enterprise OS installs is critical for doing business effectively. Red Hat/Fedora Core allows you to add value by allowing detailed customization of automated enterprise installs with open ended, open source tools such as syslinux/isolinux/anaconda and general networking protocols such as NFS, HTTP, and FTP that together form the backbone of kickstart-based technology.

By leveraging common protocols such as FTP, HTTP, NFS, kickstart allows you perform your own custom, stock, LAN, or even Internet-based OS installs from just about anywhere in the world. The key to making it work for you and keeping it manageable is in the departmental planning, work, and maintenance that goes into the ks.cfg files for your various OS install target platforms. Kickstart is just the tool by which you are able to execute these corporation-wide technical strategies.


Chapter 5: Preparing for Backups and Migration

Overview

Any system administrator who is serious about running servers in a production environment must have plans for backups and restores. Backups are often overlooked by less experienced IT managers, or are relegated to the new guy as a minor task.

I tell people that backups are a lot like insurance. You don't know how much you really need them until you need them. You can often tell experienced administrators from inexperienced ones when they're putting together productions systems. Those who are experienced (or have been burned in the past) will always ask about backups, backup types, media rotation, retention times, and doing test restores. Those who may not be as experienced may try to save money by including only limited backups, thinking RAID addresses backup-related issues, or even trimming this important aspect of the production server environment from the budget completely.

This chapter starts off with a general discussion of backup types and media, then gets into the included and more traditional backup tools. We explain how to use them, and how to test and actually do restores on a system. We discuss how to migrate data between systems (either old system to new system or data from backups to a reinstalled OS/server), and along the way we touch upon some common backup-related troubleshooting issues that system administrators see in the real world.


Backup Types

Backups can consist of tar-balling your data up and sticking it on another server, dumping a live filesystem to tape, or e-mailing yourself 562MB uuencoded tar files to be reassembled at home. Although none of these are really production quality forms of backup, the whole point of backups is being able to get your system back to its original operating condition after some catastrophic event.

Most production systems require a regular and strictly regimented backup schedule to a set of tapes. To be effective, your plan should include backing up regularly and often, and occasionally rotating sets of tapes off site for safekeeping in case of disaster.

Before you learn how to work with the software tools to get the job done, you need to understand the basics of various types of backups (levels), backup methodology, media, and various related technologies. Then you can start making intelligent decisions about what backups tools to use and how to use them most effectively.

Start Sidebar
The Myth of RAID Backups

RAID stands for Redundant Array of Inexpensive/Independent Disks. It's a set of hardware and/or software configuration(s) that grants you higher storage hardware availability (from failed disks), storage capacity, and/or gets you storage speed-nothing more. Some people are under the misconception that RAID can function as a backup for their system. This is simply not true.

RAID gets you hardware availability with respect to hardware failures (that is, it keeps things running after one of your hard drives die). Backups get you data availability with respect to time. RAID will NOT save your assets if you accidentally rm -rf /home (don't ever type that!), or if you accidentally rpm - Upgrade to the latest kernel instead of -installing it, nor will RAID help you get your data back if you get hacked, cracked, or rooted. When data is written to a RAID array, it gets written to ALL the disks. RAID will save you if one of your hard drives fails. That's all. Hardware availability with respect to, well, hardware!

Backups get you data availability for when your data fails (that is, rm - rf *, rpm - i kern*, or h4x0r3d). I've heard from many people who were baffled as to why they couldn't just recover their system from RAID. (These are likely the same unfortunate souls who thought that a firewall would protect them from the CodeRed or Nimda Internet Worms.) In short, if you want to protect your system from being down from blown drives, get RAID. If you want to protect your data and OS configuration from, well, yourself and others, get backups.

You may have heard of some newer forms of pseudo-RAID/backup-ish type technologies such as snapshots, LVM, and off line RAID-1 that do give some measure of "data availability" or backup type of functionality. But these are not very common, are beyond the scope of this chapter, and should not be confused with regular RAID-1 or RAID-5. Plus, these solutions do not get you all the security and flexibility of off-site rotated backup tapes.

End Sidebar

Types of Backups

What are commonly referred to as types of backups should really be called levels of backups. What level of backup you want dictates how many files make it to tape (or your backup media of choice). The following sections describe the three main types of backups: full backup, incremental backup, and differential backup.

Full Backup

A full backup is simply backing up every file in the system or target directories, regardless of the previous backup time or changes since the last backup. A full backup is what most nonadministrators think of when they think of backups; however, this is not what you should do every time you want to run backups of your system. The time it requires and amount of tapes it needs makes this solution something you don't want to do every time backups are needed, but it is still required once in a while to achieve a good overall backup strategy.

If you have a corporate file server, for example, running full backups of it every night of the year just is not feasible-not to mention all of your other systems in the corporation. Let's say you have a 300GB RAID-5 corporate file server and you want to back this up every night to your local, measly little four 12/24GB tape drives. If you get a 70% compression rate (yielding ~20GB/tape), this would take you around 15 tapes per night at a cost of around $60/night with a standard 90 data day retention time-over $5,400 for 3 months of tape. And that's just one of your servers. You may have only spent half that much the server itself. The bad thing is over 90 percent of the data on all those tapes is all the same. This is not an efficient method to use in a production environment.

Another issue with full backups is the time required. If each of your four little 12/24GB tape drives chugs along at 35MB/min transfer rate (real world), the effective transfer rate is 140MB/min (assuming that you have your four tape drives on separate SCSI busses running full speed, with a constant data stream). All things being equal, your daily 300GB file server full backup should take you around 36 hours of continuous backing up, or around 1.5 days! (And this is assuming that you're there for the full backup, swapping tapes when prompted, and making runs to Taco Bell between tapes.) As you can see, this is not feasible in most real world environments. But more than this, it's wasteful and will turn you into a gray, bitter, old adminstrator who mumbles to himself in the hallway. Don't do it.

But do you really need full backups every night? After all, how much of your data really changes every day? From experience, I can tell you that that the delta (or amount of file change) on our example would generally be less than 1 percent. So in the real world, you could probably just get away backing up the delta of no more than 3GB of data per night (you could get a whole week on one tape!). This leads us to consider other alternatives-incremental and differential backups.

Incremental Backup

Incremental backups are more frugal than full backups. Simply put, incremental backups record everything that's changed since the last backup. To do a restore, you will need the most recent full backup tape(s), plus the tapes of each incremental backup since the full. So with incrementals, you start with a full backup, and then build on that with each incremental only backing up what's changed from night to night. This translates to less data to back up per night, hence less tape used. In fact, you can probably squeeze more than one day's backup per tape if you're careful. However, when implementing an incremental backup system, if you have to do a full restore on Friday, you need the tapes from Thursday, Wednesday, Tuesday, Monday, and all the full tapes from Saturday. Figure 5-1 shows what a daily incremental-based backup looks like.

Click To expand
Figure 5-1: Example of using full and incremental backups together.
Note

If you don't have people working weekends and can afford to skip backups on Sunday, consider keeping this "the day of rest" (for the tape drives anyway). It gives you an emergency time slot to do large unexpected backups, restores, or run into "full backup overflow" from Saturday if need be so that you don't eat into Monday morning when people start coming in. Nothing slows down live file server access more than running a remote backup of it while people are trying to get to it and use it.

Since more tapes are required to do a full and current restore from any given day with this scheme, (today's tape plus each tape back to and including the full backup) this can make for serious trouble if you can't locate and retrieve all of the tapes required. Even so, an incremental-based backup system can be used in a production environment; it's just a nightmare to restore from and increases restore failure risks. It can effectively allow you to get the job done, especially if your department is in a financial pinch and doesn't want to buy all those tapes for full. Just keep in mind: there is not "one best solution" here. The pros and cons of each backup scenario must be considered for your given situation. As we'll see a bit further down, occasional incrementals make for a good diverse backup strategy, they're just not ideal to rely on as your sole backup strategy.

Tip

If you don't always have budget for new tapes, get them when you can. You'll need them the most when you're totally out. Trust me.

Differential Backups

While an incremental backup records whatever has changed since the last backup of any type, a differential backup only backs up what has changed since the last full backup. So if we first backup the 300GB file server with a full backup on Saturday, and then every night we backup using differentials, we will typically have something like in the graph shown in Figure 5-2.

Click To expand
Figure 5-2: Example of using full and differential backups together.
Tip

When scheduling backups with cron via crontab -e, many inexperienced administrators just schedule a time such as midnight or 1:00 A.M. for the backup job. Don't do this. Pick the hour you want your backups to start, and then add a random number of minutes from -12 to +12 to come up with your backup start time (for instance, 1:08 A.M.). Scheduling your backups like this will keep you from running into the classic "top of the hour" problem that less experienced administrators face when their system starts an even-hour database dump, runs backups, fires off a massive tar ball download, and starts scanning the system for file alterations all at once. All this activity starting at the exact same second is bad and can take down an otherwise stable server. Also, note the system's own standard mass cronjob times in /etc/crontab and avoid those times as well.

Figure 5-2 shows a full backup on Saturday, with differential backups beginning on Monday. Each day you back up what has changed since the full backup on Saturday (differentials). The great thing about this is that to do a full restore of your system, all you need is the previous night's differential tape, as well as the last full backup set (from Saturday in this case). The major disadvantage of this scenario is that you are still backing up data that has been backed up multiple times (on weeknights), and so you still really use more tape space (or tapes) than the absolute minimum required to get the job done.


Which Backup Strategy Should You Use?

So incremental backups are great in that they use less tape, but you need ALL of them plus the full backup tapes to do a restore. Differentials are great because you only need the latest tape, plus the full backup tape to do a restore. But man, that's a lot of wasted tape.

Wouldn't it be nice if you could use less tape and not need so many tapes to be able to do a current restore? You can, using both backup concepts together with backup levels in what's called the Tower of Hanoi backup strategy.

The Classic Elegance of Level-Based Backups

Some of the more expensive backup suites today have tried to make backups easier and prettier. They wrap a GUI around the backup engine, abstract you from what's really going on at the file/schedule level, and paste slick marketing terminology throughout the tech manual, mixing up or inverting terms like differential and incremental, and even inventing their own terms such as occasional incremental or true differential. Such muddying of terminology, marketing spin, and sales-speak only add to the confusion when you're planning your company's data recovery strategy. Rather than relying on these commercial suites, consider an established system used by serious UNIX administrators: backup levels.

Backup levels are a simple system of numbers from 0 to 9 that allow you to describe and arrange exactly what, how, and when you want your files put to tape. Level 0 is your full backup and levels 1 to 9 are "levels of incrementals" that back up the data changed since the next lower backup level used. See the bird's eye view in Table 5-1.

Table 5-1: Backup Level
Backup Level | Description
Level 0 | A full backup; backs up all files.
Level 1 | The first incremental level; gets all files that have changed since the last level 0 backup. It acts like a differential by getting everything that has changed since the level 0/full backup.
Levels 2-9 | Backs up whatever files have changed since the next lower level backup. Can work as an incremental if used sequentially, or as a differential if all the same number is used. Can also be "mixed" to create hybrid backup strategies.

The idea here is that a backup level of 1 to 9 will backup everything that has changed since the next lower level that it can find. For example, a backup of level 3 will backup everything that has changed since the last level 2 backup. If there was no level 2 backup done, then it backs up to the last level 1; if no level 1, then level 0. Look at Figure 5-3 to see how you could use backup levels to provide a pure incremental backup strategy.

Click To expand
Figure 5-3: Using sequential levels to achieve incremental backups.

This level-based backup schedule looks just like the Figure 5-1 incremental backup configuration, right?

Now, knowing how these levels work, imagine what you would get if you ran the following level-based backup schedule:

S | S | M | T | W | T | F
0 | - | 2 | 2 | 2 | 2 | 2

Since a backup level seeks the file changes since the next level lower than itself, a 0 2 2 2 2 2 backup schedule would behave like a pure differential backup, backing up all changes since the last full backup every night.

So, with numerical backup levels, we can control how much data actually goes to tape. Let's mix and match to see if we can get the most effective and efficient blend: full, incrementals, and differentials.

The Power of TOH

The classic "best scenario" for balancing the most efficient use of tape, data assurance, and file redundancy over time is referred to as the Tower of Hanoi (TOH) backup schedule. A formal description of the full TOH schedule can be found at www.pcmag.com/article2/0,1759,1155464,00.asp.

Implementing a full TOH schedule can be a real pain with a basic command-line tool like dump and no advanced index/tape pool tracking software. However, if you're on a budget and don't want to use the fancier backup suites that we discuss later, consider the simplified and modified TOH tape rotation schedule that we show here. It does nicely for a production server environment, using nothing but the included dump and restore backup utilities. The TOH schedule proposed here will keep your data safe, secure, and will keep you from having to run full/level 0 or level 1 backups on weekdays (which would slow your system(s) and possibly network access). I am representing our modified TOH schedule here in a 3-week cycle, which looks something like this (this schedule assumes an early morning start time, such as 1:34 A.M.):

Sat | Su | M | T | W | T | F |
0C | - | 3 | 2 | 5 | 4 | 1 |
1R | - | 3 | 2 | 5 | 4 | 7 |
1R | - | 3 | 2 | 5 | 4 | 7 | -> start over

The idea here is that you start off the 3-week cycle with a level-0 (full) backup, making a dated Copy of this first level-0 for later. The rest of the first week is followed by the modified TOH schedule. On the second Saturday, you perform a level-1 and date it, which gets everything back to the original level-0. At this point you are able to safely do a full restore with nothing more than your current level-1 and last Saturday's level-0. Next you Rotate the first week's tape set (with the dated copy of your level-0) off-site for safe keeping in a fire safe for disaster recovery. For the rest of the second week, follow the 1, 3, 2, 5, 4, 7 schedule. The third Saturday you repeat the previous Saturday's tasks-complete level-1, date it, and rotate out the second week of tapes for off-site storage. You could now do a full restore to date with your second level-1 and the original level-0. Continue with the TOH schedule through the end of the third week, bringing back the first week's tapes (minus the copy of the level-0) to do it all over again.

Tip

I recommend rotating tape sets out like this and when bringing them back for reuse, keeping Saturday's 0-1-1 tapes all at the remote storage site. This allows you to go back to any week of the year and fetch files from that week's Saturday with no more than a copy of the level-0 that they kept, or the level-0 plus level-1. This method will take only around 70 tapes per year and will give you an unlimited archive ability. At the end of each year you can even reabsorb all the archive tapes for reuse except for a level-0 for each quarter.

While this is not a pure TOH implementation (it's much more simple), this modified TOH backup strategy has several advantages:

  • Less restore work and less to go wrong-If you need to do a restore on Friday of the first week, for example, then you only need tapes 0, 2, 4, and last night's 7-instead of 0, 1, 2, 3, 4, and 5 as would be required if doing a pure incremental (0 to 5 as in Figure 5-3).

  • Less tape waste-You're not copying the same files onto the tape night after night as with the differentials or nightly full backups. Less waste of tape and backup time means saved money and added capacity for growth.

  • Off site rotation-Into the second week of the schedule, simply keep the original level-0 tape, and rotate a copy of tape 0 along with Monday-Friday tapes off-site to a fire safe or vault for disaster recovery. You can still perform a full restore with the original level-0 and the first Saturday's level-1 (and the 3, 2, 5... moving forward).

  • Data security-Files that have changed since the level 0 make it to tape at least twice in this scheme.

Note

After each level-0 backup, you need to make a "dated clone" of your level-0 tape to go with your off-site rotation tape set, keeping the original onsite for restores. This will guarantee that anyone needing to do a full recovery from the off-site tape set will be able to do so with your safe, clearly labeled and dated 0, (1), 2, 4, 7 off-site tapes.

The main disadvantage of a TOH/levels-based backup strategy is that it usually takes more than one night's tape(s) to do a full restore. And that's about it really.

If you walk into most enterprise environments, you'll find that even if they're running some high end GUI based backup suite, this type of backup strategy is what the little software man behind the GUI curtain is actually doing.

You can take a closer look at how to implement this form of backup strategy in the Backup Tools and How to Use Them section.

Tip

A 4- or 8-mm tape on a helical scanning backup drive can be used a couple of hundred times (for in-depth info, see www.datman.com/tbul/dmtb_035.htm), but after this you need to replace the tape with a new one before it starts encountering errors. Any time you introduce a new tape to your facility, you need some system to tell when it came into use and thus when it needs to be replaced. Most higher end backup suites do this for you automatically, but a simple manual tape-aging system that I've found useful is color-coded circle stickers from any office supply stores. Each color represents a given month. When you put a tape into commission, write the year in the color dot so you'll know how long you've been using it (only put the dot on the front edge of the tape, never on the flat body). Starting out, to help determine your usage frequency, put a tick mark on the circle every time you back up to it. After a few months you'll know how many uses/months your tapes get for your given backup strategy, so you can figure out how long it takes for tape in your backup schedule to "expire." After this, you can just put a color dot and year on all new tapes and know (ahead of time) when to dispose of them and get new tapes.

Backup Media Types and Hardware

You can find a huge array of modern tape drives and media types to choose from. Not one media is best for all applications; you need the right tool for the right job. For example, while a CD-RW might be fine for backing up your mom's My Documents folder, a few binaries, and config files on your server, it is really not considered to be a production-quality backup solution. A DVD-I-/-RW is fine for offloading your MP3 collection to archive or maybe even cloning your boot partition for doing emergency boot recoveries, but these forms of backup are limited and can be inherently expensive and too fragile for long-term use. They're not really suited for continual server backups, unless you're doing something special such as boot images or random access configuration file archiving. CD and DVD+/-RWs can be considered in some low-end backup scenarios, but are not considered a safe or viable solution for single production server or multiserver backup arrangements.

Some people prefer using hard disk drive based storage for either disk-to-disk cloning, or archiving, or nightly full backups. While this practice may be okay in the mind of new sys-admins, it really is not practicing safe backups. Unless something like external or hot swap drives are used, the media is not easily removable/storable; it usually stays in the machine spun up and running. This means that problems such as electrical damage, fire, or flooding can all render both source and backup drives inoperable. Even if you use some form of external or removable media, if you were to try to employ a large scale removable hard drive backup and rotation scenario, you would quickly find it to be very cost prohibitive and dangerous with the fragile nature of hard drives. In fact, if you need rotated and secure copies of data, this is actually one of the most expensive ways of backing up large volumes of data over time-and also clumsily awkward and frustrating to implement. The only way drive-based backups tend to be done today is via external SAN (storage area network), which we'll talk about later, but SAN is not really considered to be a classic disk-to-disk solution.

Tip

Some determinedly creative administrators create a hybrid disk and tape backup solution that can have all the benefit of on line or "drag-n-drop" access (via the backup disk), while using a local or remote tape solution to backup the backup disk for tape rotation and off-site storage purposes. This offers a nice functional compromise, as the backup drive does not have "open file" issues that more traditional live backups have. This type of backup can be easily scripted, and you can even do prebackup database dumps or "hot copies" (such as using mysqlhotcopy) to static files on the backup drive before the backup drive-to-tape portion of the script runs, thus backing up the database also.

So what type of backup media do you need? This varies a lot based on your data size, type of data, pocketbook, and the scale of your operation. Let's take a look at a few of the various common business models out there and see which fits you best and based on that and other variables, which resulting media type you should consider.

SmalltoMedium Backups

Single or multitape drive backups are commonly used for small to medium server configurations. This is generally effective for a system of four to six servers; beyond that, you should consider using a tape library or mass robotic tape changer type arrangement. The up-front costs of implementing small, medium, or large tape backup arrangements tend to be higher than that of other less traditional forms of backup (such as DVD+/-R or disk-to-disk); however, tapes are used because they end up costing less per GB and are more robust than other "neat idea" solutions like disk-to-disk or CD/DVDs. When it comes down to it, the cost of tapes over the long haul is cheaper and they are easier to handle, rotate, and move. They also give you greater flexibility when growing into a large-scale backup, rotation, and off-site storage arrangement than do most other forms of backup. Table 5-2 compares the advantages and disadvantages of various tape drive and media types for small-, medium-, and large-scale backups.

Table 5-2: Backup Media Type Comparison
Media | Drive Price | Media Price | Capacity Nat/Comp. | $/GB | $/10TB
CD-RW | $50 | $0.45 | 0.700GB | $0.64 | $6,450
Pro: Good for small workstation/personal data
Con: Expensive and cumbersome for large/servers, fragile (scratches). Recent studies also indicate media degradation in as little as 18 months
DVD-RW | $100 | $1.50 | 4.7GB | $0.39 | $4,000
Pro: Good for server config backups, DBs, and recovery data and boot tools
Con: Still expensive, easy to outgrow, fragile (scratches)
Hard Disk | $150 | N/a | 150GB | $1.00 | $10,000
Pro: Good for "instant recoveries," live/instant random access. Fast
Con: Very expensive and awkward for archiving/rotation, easy to outgrow
DDS3 Tape | $400 | $4 | 12/24GB | $0.20 | $2,400
Pro: Cheap drive and price/GB, good for small jobs, easy to find media
Con: Outdated, limited growth potential, slow backup/restore speed
DDS4 | $500 | $8 | 20/40GB | $0.24 | $2,900
Pro: Cheaper drive and media. Good current solution for small or multiple small servers, fairly common media
Con: Represents last drive in "end of life" technology. 2-4 years remaining maximum
AIT-1 Tape | $700 | $40 | 35/70GB | $0.67 | $7,400
Pro: Cheaper end of AIT drives, good for secondary and small-medium servers
Con: Comparative ROI of media over time is low because of low capacity
AIT-2 | $1,100 | $45 | 50/100GB | $0.53 | $6,400
Pro: Good middle road tape with growth potential. Good for changers
Con: Media still expensive/GB, in decline of acceptance
AIT-3 | $3,000 | $50 | 100/200GB | $0.29 | $5,900
Pro: Higher end of AIT drives, great for growth servers and changers, good tape/capacity ROI
Con: Expensive initial cost of drive somewhat prohibitive
LTO-1 Tape | $2,800 | $35 | 100/200GB | $0.21 | $4,900
Pro: Great cost/GB, good for for mid-high end servers, and for streaming
Con: Newer tech, hard to find media, not good for start/stop
LTO-2 Tape | $3,800 | $75 | 200/400GB | $0.22 | $6,000
Pro: Fair costs, good future growth direction for changers, and for streaming
Con: Expensive drive, newer tech, hard to find media, not good for start/stop
Note

Price per giga byte of the media is probably the most important figure to look at when starting production-grade backups. However, keep in mind that newer more expensive tape prices (such as AIT-3 and LTO) often come down quickly 2 to 3 years after introduction. Plan ahead.

The rightmost column in Table 5-2 includes the price of the drive. This figure is not a linear comparison but just a ballpark figure to show much your first 10TB of backups on this drive and media will cost you. You really need to account for your giga bytes/month needs and project it over the course or a year to get a useful comparison for your scenario.
Note

Streaming means that when you're taking a backup, you feed the tape drive an uninterrupted stream of data. If the tape drive is much faster than the data it is being fed, then the tape will have to slow, stop, reverse (while buffering the now incoming data), pick up where it left off, and start going again. This causes unneeded wear and tear on the drive and tape, and depending on your scenario, can greatly slow your backup process. Fast tape LTO and DLT tape drives are a good fit for streaming backups, while helical scan tape drives such as DDS, AIT, or Mammoth are better for start/stop or nonstreaming scenarios.

More information on various Linux-compatible tape drive tests and specs can be found at the Linux Tape Device Certification Program site: www.linuxtapecert.org/drives.php.

Medium- to Large-Scale Backup Solutions

It's ironic that as the computing industry has gone from a centralized (mainframe) environment to a distributed one (PCs and servers) over the past 30 years, backups have moved the opposite direction. Even 10 years ago, each new server usually came bundled with its own tape backup drive. Now, doing backups to a local tape drive on every server is considered wasteful. Even from a purely administrative perspective, backup administrators would rather have centralized backup control and tape management over multiple remote systems. Innovations in the networking arena in the past 10 years have made this natural shift toward remote centralized backups possible.

Some medium to large corporations find that implementing network backups is a good way of minimizing tape drive costs and the complexity of tracking dozens of tape pools. In a nutshell, this process is handled by each machine running a network-backup client program that talks to a centralized backup server. This backup server is in turn hooked up to a single large robotic tape library (that is, a bunch of tapes, tape drives, and one or more robotic arms in a big cabinet). In this arrangement, your many servers all talk to the single centralized backup server daemon that is controlled, scheduled, and monitored centrally. This is often implemented with either DDS3/4 4-mm tape (on the low end) or AIT/Mammoth/LTO tape and drives (at the high end). This type of setup can be configured with tapes and all for under $20,000. Another larger enterprise backup solution is a Backup SAN (Storage Area Network) or what's sometimes called LAN-Free Backup. In the same way that a LAN allows distributed access to computing resources, a SAN allows for distributed access to storage resources. SAN is a more pricey arrangement by which all the production servers or machines to be backed up use a common (but "off-LAN") pool of high-speed fibre channel or iSCSI (Internet SCSI that runs over a second dedicated TCP/IP backup network) attached tape storage that forms a Backup SAN (see Figure 5-4). This is nice as it gives you centralized backup management, without the normal network/LAN congestion and timing issues you have when backing up over the corporate LAN. A tape backup SAN is segmented up for each of the production servers or machines. Each of these machines on this SAN has a Fibre Channel HBA (or dedicated high-speed NIC with iSCSI) and the backup client software installed and configured. The backup server sees all of the tape SAN client machines and schedules and directs the backup clients on the machines to the tape SAN for back up locally. The tape backup library's large streaming set of parallel tape drives and/or tape changer and library makes quick work of getting the data to tape, indexed, and accounted for. This type of arrangement (streaming or continuous writing of data) makes good use of high speed, linear tape technologies such as DLT (Digital Linear Tape) or LTO (Linear Tape Open standard, a.k.a. "Ultrim") because the high speeds needed for nonstop streaming is guaranteed by the throughput of the SAN to backup system. This is a higher end solution and can be pulled together for more on the scale of $80 to 120k on the low end to several million dollars on the high end.

Click To expand
Figure 5-4: A Linux server/machine running a backup client talking to a centralized backup SAN via high-speed fibre channel.

Other Backup Hardware Tips

Here are several words of wisdom regarding hardware-related backup issues, so you can use the pain and suffering of others in your pursuit of backup Zen.

SCSI or IDE Tape Drives?

Most serious administrators use SCSI tape drives. Nowadays, even higher end tape drives such as the AIT drives are coming with an optional cost-saving IDE/ATAPI interface. With blazing fast ATA-UDMA speeds being what they are these days, IDE/ATAPI tape drives are looking less vile than they once were, especially since ATA with UDMA is not burdening the system with Processor taxing Input/Output mode (PIO) based data movement any longer. However, running ATAPI on Linux is still not an ideal setup. Sometimes ATAPI drives work, but they often fail or lock up. It's been my observation that administrators who choose ATAPI tape drives for backup tend to care more about the price of their system more than they do the quality of their backups. That being said, if you enjoy living on the edge and want ATAPI tape drive based backups anyway, just be sure to disable ide-tape kernel driver support (rmmod ide-tape) and use SCSI emulation instead (insmod scsi_mod; insmod ide-scsi; insmod st). This will change your tape device from /dev/ht0 to /dev/st0. And be sure to make these changes permanent in your /etc/modules.conf file. Good luck.

Tip

If you're going to risk it and run IDE/ATAPI interface tape drives, at least don't put them on the primary ATA interface with your system drive. Either put it by itself (or with a CD-ROM) on the secondary interface (cable) or get yourself a separate PCI ATA hard drive card for $20 and put it on that. You don't want your tape drive on the same controller as your production OS and data drives. Doing so will simply slow down your entire system.
Caution

Feeling lucky with IDE/ATAPI tape drives? If you want to try these, the ATAPI AIT drives are fairly popular. Just do not use the low-end IDE/ATAPI (or even SCSI) versions of Travan-or QIC-based tape drives for production quality backups. These drives have caused many problems in the production backup arena and using them is begging for trouble. Besides lack of error feedback and high bit error rates, they require regular tape retensioning (via commands like mt -f /dev/st0 reten) and will plague you with frequent backup failures, or even worse, no "failures"-just bad backups! And did I mention that the average Travan tape costs $30 to 40 as compared to a DDS3/4 tape price of $4 to 8?

Dedicated Backup Interface

If you're running SCSI-based tape drives, be sure to put the SCSI tape drive on its own SCSI bus. I don't mean just external, or on the narrow external connector of a wide channel, I mean have more than one SCSI bus-usually another dedicated SCSI card or separate SCSI bus on a multichannel SCSI controller. You definitely don't want to put a SCSI tape drive (or other slow r/w devices) on the same SCSI bus as your high-speed SCSI or RAID devices. Doing so will just slow down your whole system, especially during backups. Just get yourself a cheap little SCSI card that will work with your drive and use that. Be sure to get a narrow card if you have a narrow tape drive, or a wide card if you have a wide tape drive. Some server-grade motherboards actually have a secondary SCSI bus built in just for this purpose. Some newer SCSI tape drives even run high-speed SCSI LVD or UltraXXX. Be sure to check this out on the drive and card specs before making the purchase. Almost any decent (or even older) SCSI card will work with the Linux kernel now days, at least for tape backups.

Tape Drive Error Flash Codes

Be sure to get a good tape drive that has some type of "LED flash codes" or "error codes" so that you can notice problems just by looking at the tape drive. Problems such as incorrect tape type, dirty/cleaning required, and other r/w errors are most often detected and fixed when a drive communicates them directly to the backup administrator or tape rotation staff via direct visual cues or LED error codes. Many times when your tape drives are remote from the backup software, your backup software may not report on such errors correctly, or the guy changing the tapes may just leave a problem drive alone if it keeps spitting a tape out or the drive just locks up on him. These types of critical errors are best reported directly from the drive itself and error flash codes seem to be the best way to communicate them.

Tape Drive Cleaning

Oh, don't forget to keep your tape drive clean and failure-free! When you buy backup tapes, spend $60 or more and get yourself a ten-pack of high-quality cleaning tapes too. They're a must in a production backup environment. Always keep at least two on hand for your specific tape drive. Consult your tape drive manual for further cleaning and maintenance details.

Set up a policy-based system so that after every X number of backups you load a cleaning tape and let the drive take care of itself. Follow your specific tape drives recommended cleaning schedule.

One more caution: Don't lend your cleaning tapes out! If you lend it to someone in another department, you never know where it has been or what it has cleaned when you get it back. You don't want an 8-mm cleaning tape back that just cleaned some guy's coffee-soaked Hi8 CamCorder coming back to your department to clean your server tape drives! Keep a couple extra on hand to just give to people who want to borrow one of your production cleaning tapes.


Backup Strategies

Before you decide how you want to backup your system(s), you need to ask yourself some basic questions: "What can I not afford to lose?" "How much money could it cost my company if this data was irretrievable?" "How long do I need to keep my data?" Only after considering questions like these can you start to look at your company's business and business data intelligently to address things such as the following:

  • Data to be backed up

  • Retention times

  • Tape rotation and off-site storage policy

  • Processes and policies surrounding backups

All of this is essential to ensure that the unspeakable partial or complete data loss never happens to you.

The following sections get directly into the technical details of what is generally important to backup on a Linux system, and what to do about things like databases, log files, e-mail, and other system content that is always in a state of flux or changing.

What to Back Up?

Your backup strategy needs to focus on what you want to back up, or in some cases, what not to back up.

As you can see in Table 5-3, some parts of the OS hard drive are important to back up, others are less important to backup on a regular basis, while still other directories don't need backup at all.

Table 5-3: Backing Up System Directories
Definite, frequent backups required
/etc/ | System-wide OS/software/daemon config files
/home/ | User home directory space
/var/ | System variable files; logs, mail/printer queue, and so on
/root/ | Root's "home" directory
/var/www/ | web file system (servers only) on newer Linux
/van/ftp/ | FTP file system (servers only) on newer Linux
Backup rarely or never
/mnt/ | File system mount points (floppy, CD-ROM, NFS, and so on)
/proc/ | Kernel/OS dynamically created information
Backup less frequently or only if a full OS backup is desired
/bin/ | OS essential binaries
/boot/ | Boot files, kernel files, and so on
/dev/ | OS device files (hard drive, floppy, serial, and so on)
/lib/ | OS shared library (program) files, pam, and kernel modules
/lost+found/ | Corrupt and unlinked file storage directory for / filesystem
/opt/ | Third-party and FHS compliant software (such as Open Office)
/sbin/ | OS binaries (usually only run by root or system)
/tmp/ | Application, OS, and user temp file space
/usr/ | Distro installed/OS, binaries, docs, source, shared, X11, and so on

If you have any valuable system configuration data, be sure to back up /etc/. This is the most common area for system-wide configuration data. Don't forget it.

Other directories you don't really need to back up unless you're doing a full backup. These include /opt/ (for third party and FHS based applications), /usr/local, /bin/, /boot/, /dev/, and the other directories in the last section of Table 5-3.

Note

Some people do not back up /lost+found/ because this is where corrupted files end up being stored. However, in some gradual disk crash scenarios where you're slowing losing hard drive blocks over time, this type of directory could be useful for getting back those lost and partially corrupted files. Don't just discount it without considering such issues.

For some directories you just do not want to back up, ever; of these, the two big ones that come to mind are /proc/ and /mnt/./proc/ is a special directory that does nothing more than give the kernel a place to mount itself so that user space apps can "talk" to it, look at, and in some cases change running kernel parameters on the fly. If you boot from a Linux boot floppy and take a look in your system's /proc/ directory on the hard drive, there's nothing in there. And if you try to back it up, you may run into problems.

The /mnt/ directory is kind of special. While the subdirectory names in this directory can be important, most of the important (system-wide ones) should be referenced in the /etc/fstab file and should be able to be restored fairly easily.

Caution

If you know what you're doing with your specific backup suite, then you can back up the /mnt/ directory and its top-level directories, if you're careful. However, if you're not careful and you have things like CD-ROMs, floppies, NFS, and SAMBA/Windows drive mappings to this directory, you could quickly find yourself backing up your own removable media and the corporate file server! Just watch it with this directory.

Backing up the Entire Operating System

Why wouldn't you want to backup the whole OS on a server? The UNIX-/Linux-based file systems are clearly divided by function. So when doing a recovery, many people assume that they'll be doing an OS reinstall anyway-so they don't bother backing up the parts of the OS that never change after an install is complete. This stance varies depending on what you define as critical for your environment. If you lose the main and only drive in a server, do you think you'll just be getting a new drive, reinstalling, reconfiguring fresh, and then restoring the data? Or do you want to restore everything (including the OS) from tape? Most administrators don't bother with OS backups of their servers. They get the users, application configurations, and just apply those to a fresh OS install.

When might you want to backup the entire OS? If you have a highly specialized OS nonserver or workstation install with many customized user application and OS settings (such as a developer workstation or remote desktop terminal server) you probably want to get the whole OS and all of its installed and configured apps.

When might you want not to backup everything? A likely example would be on a single-purpose server such as a user home-dir fileserver, vanilla web server, BIND/DNS server, or the like. These types of application configurations can usually be quickly brought back up by an OS reinstall, then using up2date (for RPM systems) or apt-get (for Debian systems) to bring the system up to a fully patched state, and finally putting the backed up data back in place from tape.

For example, after a reinstall a BIND DNS server would simply need its DNS zone files put back into place in either /var/named/ or /var/named/chroot/var/named/ (for chroot'd systems) along with the main named config file /etc/named.conf. Restart the named service and you're back up and running. It's kind of a waste to back up the rest of the OS (several GB) every week when it is not really warranted.

Most home/desktop systems only require backup of user data in /home/ and /etc/ (configuration data), and possible /var/ if you store e-mail, web, and/or ftp data there. In such cases only /home/ should probably get backed up, since home Linux users tend to do a lot of OS switching anyway. In fact, one very convenient thing I have found is to just keep your /home/ files ystem on its own dedicated drive. This way when you go to reinstall a new Linux distribution on your system, you just unplug the /home/ drive, install the OS on your main drive, and then plug in and mount the old /home/ drive back into place. Be sure you get the new /home/username/ directories' content copied over into the old home drive. Many desktop apps and environments need their newer config directories in place and will balk if you try to forcefeed them the older counterpart's config directories and settings.

Other Important System Config Files

Even if you do a total system backup, there are some important configurational aspects of your system that you should backup, such as your hard drive's partition layout, your important /etc/ configuration files, and so on.

To make your Linux system back up its partition table(s) to a flat text file, you can use the following commands:

# mkdir /etc/config-data ; chmod 700 /etc/config-data ;
  chown root.root/etc/ config-data
# fdisk -1/dev/hda > /etc/config-data/hda-fdisk.txt
# dd if=/dev/hda of=/etc/config-data/hda-part-boot.img bs=512 count=1
1+0 records in
1+0 records out
# fdisk -1 /dev/hdc > /etc/config-data/hdc-fdisk.txt
# dd if=/dev/hdc of=/etc/config-data/hdc-part-boot.img bs=512 count=1 1+0 records in
1+0 records out

The first command sets up a /etc/config-data/ directory and root-restricted permissions. The next fdisk line grabs the human-readable partition table config from drive /dev/hda (IDE primary-master) and dumps it out to the flat text file /etc/config-data/hda-fdisk.txt. The third line then backs up the actual partition table and MBR out to the file /etc/config-data/ hda-part-boot.img. Likewise, the last two command lines do the same for the IDE secondary-master or hdc.

Doing this will ensure that you have backups of your drive partition layouts as well as your actual master boot record (MBR), which can be used in case someone ever corrupts and renders your system unbootable. And since you always backup /etc/ to tape, you should have these archived safely.

Backing up Databases

Backing up a database (or any other file that is in a state of change) is like photocopying a sheet of paper. If you wiggle it while copying it, then you get a corrupted copy. If you want a good backup of a database (or other such "wiggling" file), you need to choose one of the following options:

  • Backup directly with a backup agent-This is a special piece of backup software (usually added on in the form of a commercial backup module) that "talks" to both the database and to the backup software. The backup software starts copying the database and when the agent senses a change in the database, it finds out what that change was and communicates it back to the backup suite and so the uncorrupted change can be accounted for in the final backup. The database backup agents are not always available for all database systems or backup suites or may cost several hundred dollars, plus their use varies from vendor to vendor.

  • Stop, copy, and back up-You stop the database or service, quickly copy the related file(s) somewhere else, and then restart the service and get your backup. This technique is appropriate for backing up files like mySQL/database,sendmail, and syslog.

  • To use this method, see the Shutting off services section later in this chapter. Shutting off services is great for getting everything related to that service on tape. However, if you're on a production server, you may not be able to shut off mission-critical services such as web and databases.

  • Freeze, copy, and back up-This method is generally more difficult, but you can usually lock your databases using simple script-driven programs like mysqldump or mysqlhotcopy (more on these later). This method uses the same concept as the previous technique, but you don't stop the services; you just use these tools to "freeze" them (read-only) while you quickly copy them out, and then unfreeze them when you're done with your copy. See the rest of this section for more information on how this is done with MySQL.

The Backup Agent method may not always be available to you, depending on what type of database and backup suite you're using. Some Linux databases are starting to get third-party commercial backup agents that will talk to the database and the backup system to effectively back up a live, or hot, database. (A MySQL backup agent for ARCserve is available at www3.ca.com/Solutions/ProductOption.asp?ID=4691.) However, MySQL comes with a couple of utilities to help you use the Freeze and Copy method.

For the Freeze, Copy, and Backup database backup method, there are various options depending on what database you are using. Because MySQL is the most popular open source database, let's take a look at that. MySQL is good in that there are a few tools with which to get the job done. The first, older program is called mysqldump. It does not get binary log files (required in some cases), and it dumps to flat .sq1 files (which can take up more room than the actual data base files). If neither of these issues bothers you, then this mini-app might be just what the doctor ordered. mysqldump's general shell usage looks like this:

# mysqldump --opt --all-databases --user=root >
  /root/all-databases_$( date +%Y-%m-%d).sql

This will freeze and copy all of the databases out to a time-stamped flat file under /root/.

In newer MySQL database Freeze and Copy option is another included tool called mysqlhotcopy. This does a table level or all-database read-only lock effectively, clones everything in the mysql data directory (/var/lib/mysql/for most RPM-based systems) out to a directory of your liking, and then unlocks the databases when done. This is what the shell usage looks like:

# mysqlhotcopy --user=root --allowold --regexp=".*" /root/mysql-backup

Your regular system backups will now get the static database files in /root/mysql-backup/ and you'll be good to go.

mysqlhotcopy is great if you're running MyISAM storage engine table types. But if you're running innodb table tapes, you really do need to shut down the whole service and do a cp-a /var/lib/mysql/*/root/mysql-backup/, otherwise you will not get your innodb database content. If you're not sure which engine/table type you're running, then you're probably running the default MyISAM type and you have nothing to worry about.

For more info, do a mysqlhotcopy--h or perldoc/usr/bin/mysqlhotcopy.

Note

As of the Fedora Core RPM of mysql-3.23.58-4, there is a known bug with Red Hat's copy of mysqlhotcopy that creates an error when trying to use the program. As this book goes to press, there seems to be a patch file for this on bugzilla that should be released in MySQL 3.23.58-8 along with Fedora Core 2. See Red Hat bugzilla and mysql for the latest info on this: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=112693.

Setting up Tape Drive Devices

Commands from dump, restore, mt, and tar will all want to see the device file called /dev/ tape in place to backup and restore from. For example, mt rewind and mt-f /dev/st0 rewind should do the same thing if you have a symlink set up from /dev/tape to /dev/st0. But with some of these commands, if no device file is given /dev/tape is the default, so you should have this link set up. If you don't and you are using a SCSI tape drive (/dev/st0) set that up now, using the following commands:

# ln -s /dev/st0 /dev/tape
# ls -la /dev/tape lrwxrwxrwx lroot root 8 Mar 14 15:17 /dev/tape->/dev/st0
Note

When working with tape devices, you have two device names (besides /dev/tape) that signify two different tape drive behaviors. The device name /dev/st0 will be your first auto-rewinding SCSI tape device, and the device /dev/nst0 will be the same device but it will not auto-rewind at the end of a backup. This is significant in that multiple backups to st0 will overwrite each other (what most backup software expects). Nst0 will not overwrite, but will continue to append backups to the end of the previous backup. In this chapter, we will be assuming that /dev/tape is always pointing to /dev/st0 and rewinding after each backup.

Shutting Off Services

As previously mentioned, you don't want to back up a system with a bunch of actively changing files on it. You get around this by shutting down these services or using the Stop and Copy method.

One of the most orderly ways of doing this is via runlevel 4. In most Linux distributions, runlevel 3 is the command line or headless mode in which you might run a web or e-mail server; you don't need a GUI/X11 display (as you would on a desktop). Desktops and workstations, on the other hand, need to run in a default runlevel of 5 (at least for Red Hat-based systems).

Using Runlevel 4

If you want good system backups, you will probably want to turn off your "disk chatty" services in runlevel 4, and then use this runlevel as your system backup runlevel to run your backups.

Here's a list of some services that you might want to shut down on a server for backups:

  • sendmail

  • mysqld

  • apache

  • vsftpd

  • syslog

  • lpd or CUPS (printing services)

  • rhnsd (Red Hat Network daemon)

  • autofs

  • sshd (to keep others off, but only if you're feeling confident in your backup's stability)

  • xinetd

You can quickly get a listing of your services running in runlevel 4 with the following command:

# chkconfig --list | grep -e 4:on -e on$

This will show you everything running in runlevel 4 plus the services running under xinetd. These problem services can be configured to be off in runlevel 4 individually with the command syntax:

# chkconfig --level 4 syslog off
# chkconfig --level 4 sendmail off
# chkconfig --level 4 xinetd off
# chkconfig --level 4 mysqld off
...

Then runlevel 4 will be ready to be used as a backup runlevel that you toggle to just before backups, and back out of when they're complete.

Tip

If you're not sure which services are which, or what each one in /etc/init.d/ actually does, then on Red Hat Fedora systems you can simple run the command ntsysv from the command line and flip through all the service names, hitting Fl for the ones that you would like a description of.

The command runlevel will tell you what runlevel you are currently in, and the init 4 command will put you in whatever runlevel you want to go to (runlevel 4 in this usage).

Caution

Changing run levels affects the entire system. If you shut off most things (including ssh or networking) in runlevel 4, this can have drastic effects on other users and other services. Only change runlevels and services on a production server when you're mindfully thinking of who and what you may affect. If unsure, just do a chkconfig -list|grep -e 4 : on to see who and what you may affect on that runlevel. It is common for people to remotely change runlevels and by doing so effectively locking themselves off of the server. After an "oops!" like that, usually only a visit to the physical console can reestablish connectivity.

After you have your services all figured out and shut off in runlevel 4, a simple script such as this one will be able to back up the majority of your system, using tar, to a local tape drive:

#!/bin/bash

## Do a MYSQL-HOT-COPY to /root/mysql-bak
mysqlhotcopy --user=root --allowold --regexp=".*" /root/mysql-backup

## Rewind tape
mt rewind

## Change Run levels to 4, backup, back to 3
init 4
sleep 10
tar -czf /dev/tape /bin /boot /dev /etc /home /lib /lost+found
  /opt /root /sbin /tmp /usr /var > /dev/null 2>&1
echo -Backups complete...
## Eject tape...
mt eject
sleep 5
init 3
echo -Done

This script is more an example of using runlevels with backups than it is a hardened backup tool. Even though you can backup your system with tar, it's not something that I would recommend doing for all of your production servers. Lets take a look at some of the other options out there.


Backup Tools and How to Use Them

This section covers the basic backup tools that come with most all Linux distributions, explains how they work, and discusses the pros and cons of each.

Tar or Star

Tar is short for Tape ARchive and it's been around a very long time. The GNU version of tar that comes on all Linux systems is a feature-laden miracle worker that most praise for its convenience and rich feature set, and a few curse for its non-POSIX compatibility. Star (pronounced "ess-tar") is pretty much like tar, but much faster for doing local streaming to tape as well as more POSIX-compliant. It also handles file attributes like creation time, access time, and modification time, which makes it a bit more powerful than plain old tar.

Tar and related backup utilities hit the system through the filesystem, just as a user would access files. As a result, it can be a bit slower than some other form of raw device backup such as dump, cat, or dd (depending on the usage).

Tar functions very well as a general-purpose file system/directory archiving tool, as well as doing network-based backups in conjunction with things like ssh. However, in and of itself it does not support "level backups" and so must be used creatively with other tools to create a full TOH or level 0 to 9 backup script.

Tar is also nice in that it comes bundled with gzip compression compatibility built in; however, this needs to be used with caution on live filesystems. If a file system changes underneath the tar/gzip session, this can break the compression stream and break the tar session all together. We'll examine ways to get around this later.

Dump/Restore

Dump and restore are actually two different programs that are included in the dump package. Dump is one of the more popular OS-included, production level, filesystem-based backup utility packages out there. Dump, used in conjunction with other backup programs or by itself, is used across most all UNIX systems to do full system backups with level 0 to 9/differential/ incremental support.

Dump talks to the raw block devices that underlie your filesystems, and as such is never supposed to be used on a live, mounted r/w, nonstatic, filesystem. That being said, many people choose to do this anyway. Red Hat and even Linus Torvolds warns against using it like this (see warnings about using clump at http://dump.sourceforge.net/isdumpdeprecated.html, along with info on how to use it safely). However, even with such authoritative foreboding, you can still safely use it and get great results if you do so carefully, and prepare the system with a little admin sleight of hand.

Besides having a default "batch mode" for scripting, both dump and restore's other features include an interactive command line interface. Restore's interface allows one to browse back into the backup dump file-cd'ing through directories, adding files to your restore list, and then restoring them with the interactive extract command. Dump/restore also handles multivolume (tape) support, post-/pretape scripting (for changers), file/device/stdout output, and much more.

Major issues to watch out for with dump are (as previously mentioned) copying live and changing filesystems (active logs, e-mail, databases, and so on), as well as dump and restore binary compatibility issues. Dump and restore are tied at the hip. Never try to do a dump with one version and a restore with another. This will often fail. Be sure you know what versions of dump you have on all of your systems and make sure that you have the corresponding version of restore. In fact, I recommend getting a copy of your dump/restore binaries off your backups system for safekeeping, so that if the restore you're using doesn't seem to be reading the data off the tape, you'll have another copy to fall back on. Red Hat Fedora is currently coming with dump/restore version 0.4b34. You can see this with the following commands:

# which dump restore
/sbin/dump
/sbin/restore
# rpm -q dump
dump-0.4b34-1
# dump 2>&1 | grep ^dump
dump 0.4b34 (using libext2fs 1.34 of 25-Jul-2003)
# restore 2>&1 | grep ^restore
restore 0.4b34 (using libext2fs 1.34 of 25-Jul-2003)

Be sure that you have both binaries on your system, that they're the same version on all of your servers that you're doing dumps and restores on, and that there are no other dump/ restore binaries anywhere in your system path, like this:

# locate bin/dump | grep dump$
/sbin/dump
# locate bin/restore | grep restore$
/sbin/restore
Note

Most backup-related packages, including dump, expect to use a standard device file or symlink to be set up to point to the tape drive. The link name expected is /dev/tape. If you do not have it, then you will have to manually point the system to the tape device /dev/st0 (for example). To fix this, create a symlink with the ln --s command like this:

# ln -s /dev/st0 /dev/tape
# ls -la /dev/tape
lrwxrwxrwx lroot root 8 Mar 14 15:17 /dev/tape->/dev/st0

Amanda Client/Server

Short for Advanced Maryland Automatic Network Disk Archiver, Amanda is a pretty slick and polished automated, production client/server backup package that's free and comes installed on many Linux systems out of the box, including Red Hat/Fedora systems.

Amanda is a suite of command-line tools that brings together the best backup commodity tools such as dump/restore, tar, ssh, and production quality backup concepts such as parallel client dumps, a caching "holding disk" for caching client dumps to disk while tape busy, and many other advanced features.

Note

Amanda is a very advanced backup suite, and replaces much of the manual level-x and TOH work shown in this chapter, but it also demands a more complex setup configuration. See Table 5-4 at the end of this chapter for more information on setting up the client/server configuration for Amanda.

Table 5-4: Backup-Related Resources
Resource | Location (addresses should be one single line)
Backup resource#1 | www.address.com/stuff
Red Hat EL3 backup tools and examples | www.redhat.com/docs/manuals/enterprise/RHEL-3-Manual/admin-guide/sl-disaster-rhlspec.html
Data Media Pocket Guide (Sony) | http://www.cbtsoft.fi/cbtware/docs/Sony_DataMediaPocketGuide.pdf
 | Good reference, albeit vendor-biased, for comparing many forms of backup media.
Linux-backup | http://linux-backup.net/ Great collection backup scripts, articles, and related information. All free.
AMANDA client/server | http://amanda.org http://backupcentral.com/amanda.html The more advanced AMANDA client/server backup suite that performs dynamic level-x backups given a fixed number of tapes in a backup set. Very nice for production environment backups.

The server config files are located in /etc/amanda/, where each backup set is defined in its own subdirectory.

The daemon is configured on the centralized backup server, and the service turned on like this:

# chkconfig --list | grep amanda
      amandaidx: off
      amanda: on
# chkconfig amanda on
# chkconfig --list amanda
amanda          on
Note

As you can see from the grep, it's on, but there are no runlevels visible. This means that it is actually a subservice (on Fedora anyway) of the xinetd service.

One the best walkthroughs in setting up and using this comprehensive client/server backup package is located at http://backupcentral.com/amanda.html.

Other Useful Tools

One of the most useful tools you'll find out there for implementing great backup feats of wonder is the find command. It epitomizes the UNIX philosophy of many small tools that do one thing well. For example, it can be used to find all files on your file server that have not been accessed in over a year, and move them off the system and back them up to tape:

# find /mnt/fileserver/ -atime +365 -exec mv-and-backup.sh {} \ ;

Or, how about finding, backing up, and deleting all files on a hard drive that are owned by an ex-employee?

# find / -user bob -exec backup-n-rm.sh {} \ ;
Note

These examples are using an imaginary backup script. In this example, you could substitute the script for the command "tar -czvf/dev/tape {}; rm -rf {}" or the like.

If you want to master UNIX or Linux, find is one of those commands that can greatly assist you, while making you appear all-powerful to those who are less inclined to use a non-GUI interface.

The command mt is another tool you'll need to understand to do backups well. It stands for magnetic tape, and it allows you to send commands to and talk to the tape drive. The thing to remember is that it expects to talk to the tape device /dev/tape. If that device is not in place, then it will not work without pointing it elsewhere. Be sure you have that set up. It should look something like this:

# ls -la /dev/tape
 lrwxrwxrwx lroot root 8 Mar 14 15:17 /dev/tape->/dev/st0

If you do not have this symlink set up, please refer back to the Dump/Restore section for details on how to fix this. After this is set up, you can issue simple tape drive control commands such as rewind, erase, and status:

# mt status
SCSI 2 tape drive:
File number=0, block number=0, partition=0.
Tape block size 512 bytes. Density code 0x13 (DDS (61000 bpi)).
Soft error count since last status=0
General status bits on (41010000):
 BOT ONLINE IM_REP_EN

If you're working with tape changer libraries (with robot arm changers and the like), you'll need to familiarize yourself with the mtx package. This is the software that allows you to talk directly to the tape changer and tape drive using standardized commands.


Backup Examples

You've got the theory of backups under your belt, and you know some of the tools. Now let's get down to it and start giving some examples of useful methods in which to perform various types of backups.

Using Tar Locally

You may have seen this done, but here is how to tar up all the user's home directories in the /home directory:

   # tar -czvf home-dir-userdirs.tgz /home

The flags (czvf) stand for create, z compress, verbose, and output file. Then the two (or more) options the command takes are the target tgz file (including the file extension) and then the director(ies) to be tar-balled into the target file.

Want something more creative? How about combining the find tool with tar. Use them together to find all new files on the hard drive that belong to the user tweeks and are less than 30 days old, and put them in backup archive:

   # find / -user tweeks -ctime -30 -exec tar Azvf /home/BACKUPS/
   tweeks-new- files.tgz {}\ ;

Or use tar to back up all the standard OS and user files and directories on a system to tape locally:

   # tar czvf /dev/tape /bin /boot /dev /etc /home /lib /lost+found
       /opt /root /sbin /tmp /usr /var

That will back up everything to the tape. One nice command, no unneeded fancy backup software to get in the way.

Caution

This last tar command is not guaranteed to backup active and changing files. You still must shut off important services and stop writing to key files that you want backed up. The undefined system runlevel of 4 is perfect for shutting down active disk writing services and doing backups. Look back to the Shutting Off Services" section to see how this is accomplished.

Scripting Tar with Runlevel 4

Using the customized runlevel 4 Stop and Back up method (not a Freeze and Copy; that was for the database), we can now apply a simple backup script such as this one to go to runlevel 4, run a set of full tar backups, and e-mail us the results.

#!/bin/bash
## backup-rl -4.sh
## A simple script that switches to runlevel 4, tars up the dirs
## to tape, emails the results, and changes back to the previous runlevel.
## Use the COMPRESS and BACKUPDIRS variables below to change how it runs
## Tape device or symlink expected is /dev/tape.

######## User Variables ###########
RLBAK="4"
##COMPRESS=""                  # compression off=""
COMPRESS="z"                   # compression on="z"
BACKUPDIRS="/bin /boot /dev /etc /home /lib /lost+found /opt /root
  /sbin /tmp /usr /var"
## All dirs "/bin /boot /dev /etc /home /lib /lost+found /opt /root
  /sbin /tmp /usr /var"
MYEMAIL="admin@example.com"
TAPE=/dev/tape

########### Do not change ###########
RL="$(runlevel | cut -f2 -d" ")"
RLORIG="$RL"
CURDIR=$(pwd)

## Do a MySQL-HOT-COPY to /root/mysql-bak
#######################################
echo -Performing a MySQL Hot Copy to /root/mysql-backup
mysqlhotcopy --user=root --allowold --regexp=".*" /root/mysql-backup
echo -MySQL hotcopy complete...

## Change Run Levels and Log Activity
######################################
echo
echo -Current runlevel is $RLORIG
logger -t BACKUP "Runlevel 4 tar backup script was run..."
echo -Changing run levels to $RLBAK for backups
logger -t BACKUP "Changing run levels to $RLBAK for backups"
init "$RLBAK"
sleep 10
echo -Now in runlevel $(runlevel | cut -f2 -d" "), starting backups
logger -t BACKUP "Now in runlel $(runlevel | cut -f2 -d" "), starting backups"
echo " -preparing to backup directories $BACKUPDIRS to tape"
logger it BACKUP " -preparing to backup directories $BACKUPDIRS to tape"

## Do the needful
#######################################
echo "   -rewinding tape"
mt rewind
echo     -tarring content to tape
tar -c$COMPRESS -f $TAPE $BACKUPDIRS > /dev/null 2>&l ERROR=$?
if [ $ERROR -ne 0 ] ; then
  #logger -t BACKUP "Failure in backup script couldn't run tar" $ERROR
  echo "tar errored out this " $ERROR | mail -s "BACKUP FAIL: tar
    could not run" $MYEMAIL
  exit " -FAILURE doing tar" $ERROR
fi
## Cleanup...
###################################
echo -Backups complete...
logger -t BACKUP "Backups complete..."
echo $BACKUPDIRS | mail -s "BACKUP GOOD: directories listed below" $MYEMAIL
echo -Ejecting tape...
mt eject
echo
echo -Returning to runlevel $RLORIG
logger -t BACKUP "Returning to runlevel $RLORIG"
init $RLORIG
echo
sleep 5

echo    Now back in runlevel $Rl
echo -Done
Note

If you are going to use this type of script, be sure that you are not shutting off syslog if you use the logger logging function to record backup status to the system log.

This is a good example of a backup script that works for performing simple, full system backups. It will get mysql using mysqlhotcopy and everything else (assuming that you have shut such services off in runlevel 4). There are tons of good backup scripts out there for you to choose from-most of them in shell or perl scripts for free. You can find many good backup scripts at http://linux-backup.net.

Tar can handle incremental backups. However, if you want to do level-based incremental/ differential or TOH backups using this script, then you need to do this with dump (see the Using Dump section later in this chapter).

Note

For more information on tar, see the man tar page, or check out http://linux-backup.net/full_inc.gwif.html.

Using Tar Remotely

A very nice tar trick I like to show is how to use tar (and other such tools) remotely. Most sysadmins are very familiar with ssh (or its older insecure cousin, rsh), but not all have ever used it to transport standard output (or stdout) from a Linux server, over the network to a backup server. This is particularly useful for systems without tape drives.

   # tar cvf - /home /var /etc | ssh root@10.1.1.10 "cat
   >/mnt/backup/remoteserver1_$(date +%Y-%m-%d).tar"
Tip

Some geek types like to use tar/ssh to go directly to a remote tape device, but I would recommend against using tar/ssh this way. I've seen this cause backup stability problems. I would instead recommend using remote tarring by tarring up your data to what's called a remote holding disk on the remote tape backup machine (as we discussed earlier). After all of your remote servers have pushed their data up to individual holding disk directories on the tape backup server, just have the backup server back up all your remote content from disk to tape. You'll get much better performance and better system predictability.

Using Dump

Dump is probably one of the more fun backup packages. However, you need to be mindful that it is not backing up the file system through the file system as you and other programs see it, but at the block device level. But what makes dump even more sensitive than tar and other such tools to disk content changes is that dump looks at the block device before the backup begins and at all the inodes that it needs to back up. If many of those inodes change before it gets them to tape, these excessive errors can cause dump to abort its backup all together.

This means that before you actually perform a backup, you should definitely use runlevel 4 or a similar method to shut down disk-chatty services, lock and copy your databases (or just shut them off completely), and quiet anything else on the system before using dump to back up to tape. If you do all of this (and are running a supported file system (such as ext2/ext3), then you shouldn't have any problems with dump.

Here's the basic usage for a local dump session doing a level-0 (full) backup to tape by the hard drive device name /dev/hda1:

# dump -Ouf /dev/tape /dev/hda1

This will back up the first file system (/ boot) on the first hard drive to tape (on this particular system). It's fairly fast and will create few (if any) errors if you've sufficiently quieted your system.

Note

Notice the u option. This is important as it allows you to keep track of the backup dates and levels automatically in the /etc/dumpdates file.

If you have three file systems on your server, /boot, /, and /home (for example), then you do not want to rewind the tape after each backup. This would be bad in that it would only yield the last partition you backed up on the tape. So if you have three partitions on a given disk, you need to use the /dev/nst0 device for at least the first two partitions and then the /dev/st0 or /dev/tape for the last backup on that given tape. These are the three commands that you might want to execute for the first level-0 backup:

dump -Ouf /dev/nst0 /dev/dha1
dump -Ouf /dev/nst0 /dev/dha2
dump -Ouf /dev/st0 /dev/dha3

You might call this your Saturday backup. If you are using the TOH backup plan, what do you think your Monday backup would look like? Probably something like this:

dump -3uf /dev/nst0 /dev/dha1
dump -3uf /dev/nst0 /dev/dha2
dump -3uf /dev/st0 /dev/dha3

If you want to get fancy and also give each day's tape its own dump label, this might help things a bit if tapes get mixed up:

dump -3u -L "L3_2004-03-04_Mon" -f /dev/nst0 /dev/dha1
...

Based on this general usage, we can create a total system backup script. Using some fancy shell script conditionals and the date command, you can autodetect the correct date in our 3-week TOH backup schedule, use the correct related runlevel, and with tools like mysqlhotcopy can build a simple backup script that can pretty much take care of your whole system. Here's a simple example script that employs everything that we've discussed thus far:

#!/bin/bash
# Name:    dump-disks.sh
# Version: v2004-04-26 by tweeks
# Takes:   /dev/dev1 [/dev/dev2] [/dev/dev3]
# Does:    Does modified Tower of Hanoi tape backup on three week rotation
# Details: Looks at day of the week and day of the month and implements a
#          modified TOH dump of /dev/dev1 [/dev/dev2] [/dev/dev3] as per:
#          S S M T W T F
#          0 - 3 2 5 4 7
#          1 - 3 2 5 4 7
#          1 - 3 2 5 4 7
# Output: log to /tmp/backuplog.txt emailed to $MYEMAIL
# Returns: "1" on Sunday error

######## User Variables ############
RLBAK="4"
MYEMAIL="backups@example.com"
TAPEDEV=/dev/nst0       # use the "n" device for no rewind (e.g. /dev/nst0)
MYSQL=/etc/init.d/mysqld
########### Do not change ###########

RL="$(runlevel | cut -f2 -d" ")"
RLORIG="$RL"
CURDIR="$(pwd)"
LEVEL=""
DEV1="$1"
DEV2="$2"
DEV3="$3"

## Detect Day of Week/Month
####################################
# Thanks to Henry Pugsley/Rackspace
dow='date "+%a"' # Day of Week
dom='date "+%d"' # Day of Month
#disks="/dev/hda1 /dev/hda2 /dev/hda3"
if [ $dow == "Sat" ] && [ $dom -lt 7 ] ; then LEVEL=0
elif [ $dow == "Sun" ] ; then echo "Not on Sunday" && exit 1
elif [ $dow == "Sat" ] ; then LEVEL=1
elif [ $dow == "Mon" ] ; then LEVEL=3
elif [ $dow == "Tue" ] ; then LEVEL=2
elif [ $dow == "Wed" ] ; then LEVEL=5
elif [ $dow == "Thu" ] ; then LEVEL=4
elif [ $dow == "Fri" ] ; then LEVEL=7
fl

## Command Line Error Check/Usage
##################################
if [ "$DEV1" = "" ] ; then
  echo
  echo ERROR
  echo "Usage: dump-disks /dev/dev1 [/dev/dev2] [/dev/dev3]"
  echo "Modified TOH backups run Mon-Sat. Will abort on Sundays..."
  exit
fi

#### sanity check ####
echo -----IS THIS RIGHT?------
echo dump-disks.sh about to run like this:
echo    Backup Level=$LEVEL, Backup Devices= $DEV1 , $DEV2 , $DEV3
echo
echo Backup will be:
echo dump -$LEVEL -auL "L$LEVEL_$(date +%Y-%m-%d)" -f $DEV1 $DEV2 $DEV3
echo -----CTRL-C TO STOP------
sleep 10
echo

## Set up logfile: /tmp/backuplog.txt
#######################################
echo dump-disk.sh run on $(date +%Y-%m-%d_%H:%M) by $USERNAME
>/tmp/backuplog.txt
echo"                    o--------------------------------o"
 >>/tmp/backuplog.txt

## Do a MySQL-HOT-COPY to /root/mysql-bak
#######################################
echo -Performing a MySQL Hot Copy to /root/mysql-backup
echo -Performing a MySQL Hot Copy to /root/mysql-backup>>/tmp/backuplog.txt
mkdir -p /root/mysql-backup
# ensure mysql is actually running...
$MYSQL start
mysqlhotcopy --user=root --allowold --regexp=".*" /root/mysql-backup
 >>/tmp/backuplog.txt 2>&1
$MYSQL stop
echo -MySQL hotcopy complete...>>/ tmp/backuplog.txt
echo "                   o-------------------------------o"
 >>/tmp/backuplog.txt

## Change Run Levels and Log Activity
#######################################
echo
echo -Current runlevel is $RLORIG
echo -Changing run levels to $RLBAK for backups init "$RLBAK"
sleep 10
echo -Now in runlel $(runlevel | cut -f2 -d" "), starting backups
echo "-preparing to L-$LEVEL dump Devices=$DEV1, $DEV2, $DEV3 to tape" echo "-preparing to L-$LEVEL dump Devices=$DEV1, $DEV2, $DEV3 to tape"
 >>/tmp/backuplog.txt
## Do the needful
####################################
echo " -rewinding tape"
mt -f $TAPEDEV rewind
# ----------------
if [ "$DEV1" != "" ] ; then
   echo -dump of $DEV1...
   dump -$LEVEL -auL "L$LEVEL_$(date +%Y-%m-%d)" -f $TAPEDEV $DEV1
 >>/tmp/backuplog.txt 2>&1
   echo "                   o--------------------------------o"
 >>/tmp/backuplog.txt
fi
# ----------------
if [ "$DEV?" != ""] ; then
   echo -dump of $DEV2...
   dump -$LEVEL -auL "L$LEVEL_$(date +%Y-%m-%d)" -f $TAPEDEV $DEV2

 >>/tmp/backuplog.txt 2>&1
   echo "                   o------------------------------o"
 >>/tmp/backuplog.txt
fi
# ----------------
if [ "$DEV3" != ""] ; then
   echo -dump of $DEV3...
   dump -$LEVEL -auL "L$LEVEL_$(date +%Y-%m-%d)" -f $TAPEDEV $DEV3
 >>/tmp/backuplog.txt 2>&1
   echo "                   o--------------------------------o"
 >>/tmp/backuplog.txt
fi

## Cleanup...
###################################
echo -Backups complete...
echo -Ejecting tape...
mt -f $TAPEDEV rewind
echo
echo -Returning to runlevel $RLORIG
init $RLORIG
echo
sleep 5
echo Now back in runlevel $RL
echo -Done >>/tmp/backuplog.txt
echo -Done
if [ "$LEVEL" = "0" ] ; then
   echo | mail -s"Following Backup was Level=0 dump. Make a copy."
 $MYEMAIL
fi
cat /tmp/backuplog.txt | mail -s "$(date +%a) Dump status" $MYEMAIL

The command line usage for backing up to three partitions with this script looks like this:

  # dump-disks.sh /dev/hda1 /dev/hda2 /dev/hda3

And this will give us a modified TOH backup based on the day of the week and the calendar date. Additionally, the first Saturday of the month will always get a level-0 dump and the script will even send us an e-mail reminder to tell us to clone our level-0 tape! This is a configuration that we can fire and forget, and it will be completely automated (except for the insertion of the tapes).

The nice thing about this date smart script configuration is that now we can easily automate this into a daily cronjob and it will know what to do. This is all you need in root's cronjob (by editing with contab -e as root):

########### Smart Three Week TOH Backup Script ############
34 1 * * *      /root/bin/dump-disks.sh /dev/hda1 /dev/hda2 /dev/hda3

And that's it. The smart TOH dump script will now run every morning at 1:34 A.M., and will even autoabort on Sundays (see the script for how this works).

As you can see, with the backup tools provided in most Linux distributions, it doesn't take that much to get an intelligent, hands off, backup configuration set up that makes good use of backup levels, tape usage, and easy recovery strategies. You just need to sit down and plan these things out a bit, and automate things with scripts whenever possible.

If the aforementioned backup script is still a little to "in the rough" for your tastes, you definitely owe it to yourself to check out Amanda at http://backupcentral.com/amanda.html and www.amanda.org. Newer backup suites such as Amanda even go beyond backup levels and the TOH as automate things by simply asking you questions about number of tapes, retention times, and data sets. Very nice indeed!

Using Dump Remotely

Although it is not often done, you can run dump remotely over ssh. You do this with a special usage of dd on the machine with the tape drive on it. The stream is pushed over ssh, into dd on the tape machine, and get put directly to tape. The usage looks like this:

# dump -f - /dev/hda2 | ssh root@backupserver.example.com dd \
 obs=10k of=/dev/st0

Pretty neat! You can also use a pull method from the tape/backup server, like this:

# ssh -n root@desktop.example.com dump f - /dev/hda2 | dd \
 obs=10k of=/dev/nst0

These can be very convenient if you can't be in more than one place at a time.


Testing and Doing Restores

Never restore files back into their original location! Doing so is wrong and can crash your system even worse than a regular crash. If you use tar or dump, always create a special RESTORE directory wherein you put your restored files in. Then as you need to place system or data files back into place, just use cp - a to copy them from place to place.

Also, when restoring system files, you must be in single-user mode. You may have to remount your root partition in r/w mode; however, this is a must for a stable full restore.

Restoring From Tar

Tar is very straightforward. If done from a file, it looks like this:

# pwd
/root/RESTORE
# ls
home.tgz
# tar xzvf home.tgz
home/bob/
home/bob/.kde/
home/bob/.kde/Autostart/
[...]
home/bob/.htpasswd-BAJ
home/bob/.Xauthority
# cd ./home
# pwd
/root/RESTORE/home

We put the backup tar-ball into /root/RESTORE/, untarred it, and cd'd into /root/RESTORE/ home/ before we starting doing file copying around the system.

Un-tarring from tape usage is pretty much the same, but you should do so to a clearly marked RESTORE directory of some type. Here's an example of rewinding the tape with mt, then restoring some files for a user's (bob) tape. The operation shows how you can have tar eXtract with compreZsion Verbosely (xzv) from the file /dev/tape, into the directory /home/bob/ RESTORE/:

# mt -f /dev/tape rewind
# tar xzvf /dev/tape /home/bob/RESTORE/

It's that simple.

Restoring From Dump

For system-wide restores, the same words of wisdom regarding the RESTORE directory apply here. The idea is to restore files into a temporary holding place that is clear in its purpose, but not viewable by the world. This is why we're putting the RESTORE directory in /root/ in this case:

# cd /root/RESTORE/

The usage of the dump package's restore command is rather strange. It's a real time, interactive restore where you are in the restore shell, interacting with the backup file in real time. The idea is that you look around the inside of the backup file, pass the restore package parameters, and have it add the files to your restore (or extraction list). Then when you're ready to restore, you just type extract and the files pop out of the dump archive and into your current directory with a new relative path from your current location.

It's easier to show than to describe:

# mt rewind
# restore  -i -sl -f /dev/tape

This restore command we tell to go into interactive mode (-i) and to source (-s) the first file on the tape. This source file number is simply the sequential order of the data files on the tape. If we had three backed up partitions on the system of /dev/hda2 /dev/hda3 and /dev/hda6, they were laid down to tape in that order, and we wanted to restore from the second partition file on tape, we would tell restore to source -s2 this time.

Now that we're in the restore environment looking at the dump archive in real time, we can browse it as if it were a live file system:

restore > ls
.:
System.map                     lost+found/
System.map-2.4.18-24.7.x       message
System.map-2.4.18-26.7.x       module-info
System.map-2.4.18-27.7.x       module-info-2.4.18-24.7.x
System.map-2.4.18-27.7.xsmp    module-info-2.4.18-26.7.x
System.map-2.4.18-3smp         module-info-2.4.18-27.7.x
boot.b                         module-info-2.4.18-27.7.xsmp
chain.b                        module-info-2.4.18-3smp
config-2.4.18-24.7.x           os2_d.b
config-2.4.18-26.7.x           vmlinux-2.4.18-24.7.x
config-2.4.18-27.7.x           vmlinux-2.4.18-26.7.x
config-2.4.18-27.7.xsmp        vmlinux-2.4.18-27.7.x
config-2.4.18-3smp             vmlinux-2.4.18-27.7.xsmp
grub/                          vmlinux-2.4.18-3smp
initrd-2.4.18-24.7.x.img       vmlinuz
initrd-2.4.18-26.7.x.img       vmlinuz-2.4.18-24.7.x
initrd-2.4.18-27.7.x.img       vmlinuz-2.4.18-26.7.x
initrd-2.4.18-27.7.xsmp.img    vmlinuz-2.4.18-27.7.x
initrd-2.4.18-3smp.img         vmlinuz-2.4.18-27.7.xsmp
kernel.h                       vmlinuz-2.4.18-3smp

restore > cd grub/
restore > ls
./grub:
device.map              jfs_stage1_5           stage1
e2fs_stage1_5           menu.lst               stage2
fat_stage1_5            minix_stage1_5         tnt_whitemount.xpm.gz
ffs_stage1_5            reiserfs_stage1_5      vstafs_stage1_5
grub.conf               splash.xpm.gz          xfs_stage1_5

After we find what we're looking for (grub.confin this case), we add it to our archive extraction list, add any other files or directories that we might want, and then we extract them.

restore >  add grub.conf
restore >  extract
You have not read any volumes yet.
Unless you know which volume your file(s) are on you should start
with the last volume and work towards the first.
Specify next volume # (none if no more volumes): 1
set owner/mode for '.'? [yn] y
restore > quit

Now we're out of the interactive restore session and back in our shell:

# pwd
/root/RESTORE
# ls -la
total 1204
drwxr-xr-x   3 root    root        4096 Mar 15 04:00 .
drwxr-xr-x  29 root    root        4096 Mar 15 04:00 ..
drwxr-xr-x   2 root    root        4096 Apr 10 2003 grub
# cd grub/
# ls
grub.conf
#ls -la
total 12
drwxr-xr-x   2 root    root        4096 Apr 26 06:29 .
drwxr-xr-x   3 root    root        4096 Apr 26 06:29 ..
-rw-r--r--   1 root    root        884 Mar 21 23:13 grub.conf
# pwd
/root/RESTORE/grub
#

That's about all there is to restore. Just remember that you need to add all of the files that you want while in the restore shell before you run the extract command. Also, remember that it will create a relative set of directories to contain the file(s) that you added for extraction. This is another reason that we use a RESTORE directory; it's safer. We wouldn't want to happen to be in the / directory and extract backup files over top live operating OS files. That would be a bad thing.

Restoring MySQL Databases

To restore a backed up database from a mysqldump-created .sql file requires only a one line command:

# mysql < all-databases_2004-03-15.sql

Just make sure that your database is running when you issue this mysql client command. If the mysql client cannot talk to the mysql server database, it will not be able to recreate all the tables and data.

To restore a backed-up database from mysqlhotcopy cloned files (or restored files from tape), use the following command:

# /bin/rm -rf /root/mysql-backup/*old

This makes sure that any old copies of databases are wiped out. Then shut off the mysqld service, copy the files back, and turn the service back on like this:

# /etc/init.d/mysqld stop
Stopping MySQL:                                 [ OK ]
# cp -af /root/mysql-backup/* /var/lib/mysql /
# /etc/init.d/mysqld start
Starting MySQL:                                 [ OK ]

The database is now restored and operational.


Server Migrations

Sometimes you just need to back up everything, reinstall a new or different OS, and restore all your data onto the new server load. This can be done, but it's tricky when you're not going right back into the exact same environment. Here are some things to watch for if you find yourself in this situation.

Disk-to-Disk Magic

If you're being pressed into a massive server upgrade, don't risk it all with the dangers of an in place upgrade. I've done in place upgrades on other people's systems just to see if they really work. Sometimes they do and some times not. But at the other end of the spectrum, why bother moving all the existing server data to tape when you're just going to have to move it all right back into place? Tape is slow; so there is no need to do it that way. If you're able to get a couple extra hard drives lined up, there's a much easier way.

First, get yourself a couple of extra drives. One of the new drives will be your new OS drive-the one you're going to install a fresh new OS on. The second drive is going to be your insurance. Take your second drive and clone your original old OS/data drive with imaging software such as Symantec Ghost or Drive Image Pro (this is assuming an X86 environment, but Ghost can also clone non-x86 filesystems). Either Ghost or DIP will allow you to make a block-level copy of your old drive to your temporary migration drive. Now take that original old data/OS drive, clearly label and date it, and put it in a safe place for at least 6 months, just in case.

Now take your ghosted drive and install it in your new system under somewhere obvious like /mnt/old-drive/, mounted read only. It should look something like this in your /etc/fstab file:

/dev/hda3     /               ext3        defaults       1 1
/dev/hda2     /boot           ext3        defaults       1 2
/dev/hdc6     /mnt/old-drive  ext2        defaults,ro    0 0

After you have the drive mounted, begin migrating your data carefully and slowly. If you don't want to overwrite new server configuration data and files with your old stuff, DO NOT just blindly copy of large chunks of data such as /home directories, websites, users, and config files. Many times, new software package (such as apache 2's httpd.conf file) is not compatible with its older cousins. The same goes for even some user content. Be sure to inspect /home/username directories before just blindly copying them over. You should usually to spend a few minutes on each one before letting loose with your cp -a /mnt/backup/ home /home. In fact, sometimes before starting to move over the larger chunks, I just go ahead and back up the new stuff just in case (if I have the space):

# cp -a /home /home-BAK

If you don't feel so comfortable with doing a lot of this on the command line, you can use a GUI environment such as KDE, GNOME, or even a cool shell tool such as Midnight Commander (or mc on the command line).

Software Issues

The software and server daemons on the source and target servers make up a major part of the server migration checklist. In short, you need to do your homework and make sure that the content and web programs you are moving do not have application version reliance that could make things difficult for you later (such as perl versus mod_perl, php versions, and so on). From a bird's-eye view, some of the technical issues that you need to check on the new server before the move begins include the following:

  • OS install and supported libraries (for example, Linux and glib versions)

  • Versions of apache (or other web server service used)

  • Version of Sendmail (or mail service used)

  • Apache modules, Perl, and CGI engines installed and configured

  • BIND/DNS software (formatting of zone files)

  • Same or greater patch levels (old versus new) for network visible services

  • Same or greater hardware configuration

  • Same security/access rights (TCP-Wrappers, ACLs, and so on)

  • Any software or configuration dependencies on your IP address

  • Any commercial packages that register their key based on IP, FQDN, or server name

  • Roughly the same OS and filesystem UID/GID (user/group) permissions

This last point is usually one of the trickier ones. You need to examine your old /etc (now on /mnt/old-disk/etc/) and start looking for differences with users (/etc/passwd and /etc/shadows) and groups (/etc/group). Then recreate the users on the new system with the same UID/GIDs if possible.

Caution

No matter how tempting it is, DO NOT just copy over the old /etc/passwd on top of the new one. There are often many services that will conflict between the old and new systems. You may want to copy over one or two user lines that have UID and GIDs of 500 or higher. After you get a feel for this, you may be able to step this up a bit. If the old and new OS versions are close enough, and you can verify that the system crypt/hash functions are the same (that is, the passwords are encrypted using the same algorithm), you may be able to copy/paste parts of the old /etc/passwd, shadow, and groups files over to the new system. Just be careful to match up up user and group ids (UID/GIDs). Sometimes you may actually be able to keep the system and user passwords intact! Just remember NOT to copy entire /etc config files over the new install's config files.

Here's an example and breakdown of what the user accounts in /etc/passwd look like, and which column does what:

bsmith:x:503:503:Bob Smith:/home/bsimith:/bin/bash
  ^^^    ^^  ^^    ^^^       ^^^^          ^^^^
  user   UID GTD   username  home dir      login shell

The file system and movement of data can be the hardest parts of a migration, because every file in the system maps to a specific user/UID and group/GID (not to mention special file links, security settings, and so on). Simply tar'ing or zipping-up and moving all the files from the source server over to the target server and then manually trying to match everything up can be a daunting job-even for a professional. Even though cp -awill preserve user/group IDs perfectly, your new OS install will often have slightly different directory, group, and security settings that you will have to keep battling, sometimes months after the migration is complete.

Web/Apache Gotchas

Here are a few of the things to watch out for with regards to apache migrations:

  • Comparing the old and new httpd.conf configurations and modules of your two servers (assuming UNIX/apache)

  • Verifying that system automation scripts, that may, for example, restart the web server, use the correct start/stop commands and paths

  • Checking the paths of logs that are written and read (if you're doing any log analysis reporting)

  • Checking all user and system crontab files, moving them over, and verifying each absolute path reference

  • Apache 1.3 versus 2.0 module incompatibilities

One big web migration issue that is often overlooked is checking to make sure that your SSL engines (underlying SSL server programs) on your old and new servers are the same or at least certificate-compatible. If they are not, you could be looking at either regenerating and repurchasing all of your SSL certificates (which could take up to 2 weeks per certificate, and hundreds of dollars each), or installing a different SSL engine on the new server.

Also, if you're bringing new SSL sites to your server that has existing SSL certificates that they've paid for, you will need both the SSL key and certificate for each site. Otherwise, the certificate is useless. I also recommend getting the cert request if it is still available. This can sometimes save the site owner money if the cert needs to be recreated.

Security

This too is a big category, so I'll just mention the basics of what you'll want to check out and compare on the new target server. This includes the following:

  • TCP-Wrappers or IP based ACL (access) controls

  • IP-Chain/IP-Table configurations

  • FTP access rules

  • Mail/SMTP/SMTP-Auth settings and relaying rules

  • Mail/POP-locks and access rules

  • Firewall ports, IP rules

  • DNS zone transfer lock-downs/allows

Usually security ends up taking a back seat to getting everything moved over and working in the migration priorities. While this might end up being your de facto strategy, just don't forget to follow up with your server's security settings or you may end up experiencing a not so pleasant reminder in the form of hacked users accounts and/or compromised systems.


Backup-Related Resources

Table 5-4 provides a list of resources for more information on backups.


Summary

Running production-grade backups takes a lot of planning, pricing, and consideration of policies on things like data retention and tape rotation issues. Doing it well in a large enterprise often draws on technical coordination among departments and server administrators, especially if you're running centralized backups.

There are a number of useful tools that come with Red Hat, Fedora Core, and most other Linux distributions. Using them effectively not only with the OS but also with your applications such as databases (such as MySQL) is critical for getting backups that you can rely on. In addition, knowing what to backup, what not to, and how to restore is also important in ensuring that there are no technical glitches along the way.

Just remember, with backups you must always be thinking at least 6 months to a year ahead. Plan for new tapes, server growth, tape pools, and rotations. With careful planning, you'll be ready when they come to you to get the full restore of that critical system, and then it will all be worthwhile.


Part III: Troubleshooting Basic Features

Chapter List

Chapter 6: Troubleshooting Tools
Chapter 7: X Troubleshooting: Video, Mouse, and Keyboard
Chapter 8: Software Package Troubleshooting
Chapter 9: File System, Disk, and Power Troubleshooting


Chapter 6: Troubleshooting Tools

Overview

To this point in the book, we have focused on procedures and tips for getting your Fedora or other Red Hat Linux distribution installed and ready to deploy-sort of the "ounce of prevention" approach to troubleshooting. For the rest of the book, the focus is on learning the skills to troubleshoot running Linux systems.

You will be able to get up to speed much faster if you bring some Linux skills to the table. The first section of this chapter quickly goes over some skills you need to troubleshoot Linux (the shell, ways of gaining root access, and so on), but if you are fairly new to Linux you should refer to some basic Linux text for more help in learning the general skills you need.

Note

Red Hat Linux Bible: Fedora and Enterprise Edition should help provide the foundation of knowledge you need to move on to some of the more advanced troubleshooting techniques described in this book.

While you might want to jump right to a chapter on a particular feature you want to troubleshoot, there are some tools that are useful across a variety of Linux features. For example, tools such as modprobe, lsmod, and modinfo are useful for working with modules when your hardware needs to be manually configured. Tools such as ssh and screen can help you connect to computers on the network, regardless of the particular feature you are troubleshooting on the remote machine. Those and other helpful Linux troubleshooting tools are described in this chapter.


Getting Around with the Shell

The next few pages cover the basic tools and concepts that a Linux system administrator and troubleshooter needs, to get around the shell command-line interpreter-or just shell for short. If you are a Linux pro, you can probably skip ahead to some of the less common administration tools.

Knowing your way around the shell, for typing commands and traversing the file system, will make it possible for you to do most of the tasks laid out in the rest of this book. There are a few tricks you can use to get to and work with Linux through shell interfaces. I go through some ways of starting up, switching to, and working with shells in this section.

Using Fancy Shell Tricks

To troubleshoot a Linux system, you won't be able to stay with the graphical interfaces. Being able to get to a shell command-line interface, or more likely several interfaces at once, is a critical component of Linux system administration.

On a Linux system that has no graphical interface (or boots to runlevel 3), the shell is all you will have to start with. Login from your text-based login screen and the next thing you will see is a shell prompt. From a graphical user interface (GUI) desktop, however, there are lots of different ways of accessing shells and tricks for using shells efficiently.

Ways to Get to a Shell

The most common way to get to a shell from any Linux desktop is to open a terminal emulator window. Back in the old UNIX days, before graphical interfaces (or Linux) even existed, a character-based terminal was the only way to access UNIX-like systems. A terminal emulator, as the name implies, emulates one of those old character terminals when you open a terminal emulator window on the desktop. As an alternative, you can open a virtual terminal to get to a shell.

Terminal Windows

In Fedora Linux, a terminal emulator running on the desktop is simply referred to as a Terminal window. Unlike early editions of Red Hat Linux, Fedora and other recent Red Hat Linux systems don't have an icon on the desktop or panel for opening a Terminal window. However, there are still lots of ways to get to one.

Here are a few ways to open the default Terminal window from a KDE (konsole) or GNOME (gnome-terminal) desktop:

  • Ctrl+t- From the KDE desktop, click anywhere on the desktop background, then press Ctrl+t.

  • Desktop menu- Right-click on the GNOME desktop background to see the desktop menu. Then select Open Terminal.

  • Red Hat menu- Click on the Red Hat menu, then select System Tools⇨Terminal.

Figure 6-1 shows two Terminal windows. The top one is the Konsole window (konsole command) and the lower one is the GNOME Terminal window (gnome-terminal command).

Click To expand
Figure 6-1: Konsole (top) and GNOME Terminal (bottom) windows provide shell access on a Linux desktop.

Each of the Terminal windows shown currently has two shell interfaces open. On the Konsole window, the small terminal icons each represent a different shell, while the GNOME Terminal window has a tab representing each shell. The prompt in each window shows the user you are logged in (as root), the name of your local computer (shuttle, in this case), and the name of the current directory (/root and /tmp, respectively).

There are other terminal windows that are less commonly used. The xterm command launches the classic X terminal emulator window. The KDE kterm terminal window offers additional multilingual support. Both xterm and kterm allow many options for changing colors, fonts, and other attributes. However, those features are not selectable from menus on the window, as they are with Konsole and GNOME Terminal windows.

Tips on using Terminal windows are described a bit later. Besides the default bash shell, there are other different shells available (which are also described a bit later in this chapter).

Virtual Terminals

Another way to get to a shell from Linux is by using virtual terminals. When you first boot Linux, six virtual terminals are started, offering you six text-based login screens. If you start Linux in graphical mode (the default runlevel 5), it will typically use a seventh virtual terminal.

You can switch to different virtual terminals from any Linux desktop, using the Ctrl+Alt+function key sequences or one of several virtual-terminal-switching commands (such as chvt or switchto). The number of the virtual terminal corresponds to the function key you use to switch, such as Ctrl+Alt+F1 (virtual terminal 1), Ctrl+Alt+F2 (virtual terminal 2), Ctrl+Alt+F3 (virtual terminal 3), and so on. From any nongraphical virtual terminals (1 to 6), you can drop the Ctrl key. For example, you could use Alt+F7 to get back to your X desktop.

Note

Use the Alt key from the left side of the keyboard.

To use either the chvt or switchto commands to change to another virtual terminal, follow the command with the number of the virtual terminal. For example:

   #  chvt 3

This changes to virtual terminal 3. The chvt and switchto commands are most useful at times when some application may have taken over your function keys.

The programs run in virtual terminals need not be purely text based. Programs that have been implemented using the new curses (ncurses) library routines can offer menu-driven interfaces that run in virtual terminals and Terminal windows. You can use Tab and arrow keys to move around and make selections. Examples include redhat-config-printer-tui and redhat-config-network-tui (the tui stands for text user interface) to see text-based interfaces for printers and network interfaces.

Tuning up a Terminal Window

In this chapter, I'm focusing on the gnome-terminal and konsole terminal emulation windows because these are the two default Terminal windows used with GNOME and KDE desktop environments, respectively. Of course, once a Terminal window is open, you can just type the commands you want and not fuss about it. However, here are a few tips that might make using your Terminal window more pleasant:

  • Multiple shells-You can have multiple shells running in the same Terminal window. Figure 6-1 shows a Konsole window with terminal icons representing multiple shells and a GNOME Terminal window with a tab displayed for each open shell.

    Multiple shells in one Terminal window can provide a convenient way to keep track of a lot of programs running at the same time. You might want separate shells open to compile software, read a man page, tail a log file, and edit a text document, for example. Here's how to open multiple shells in GNOME and KDE Terminal windows:

    • GNOME-With the GNOME Terminal window selected, press Shift+Ctrl+t to open a shell in a new tab. Click the tab you want, to go back and forth between the tabs. Press Shift+Ctrl+n to open a new Terminal window.

      Note

      There are also keyboard shortcuts to move among tabs in the GNOME Terminal window. Use the Alt key plus a number to go to a particular tab (such as Alt+1, Alt+2, or Alt+3 to go to tabs 1, 2, or 3, respectively). To step through tabs, press Ctrl+PageUp to go to the previous tab or Ctrl+PageDown to go to the next tab.

    • KDE-In the KDE Konsole Terminal window, click the plus sign (+) icon in the menu bar. A new terminal icon appears, representing the new shell. Click the different icons to go between these shells.

  • Scrolling settings-If you have ever had an output that you wanted to read scroll past you on the screen, you might be interested in the Scrollback feature. For the GNOME Terminal window, you can edit your profile (click Edit⇨Current Profile, then select the Scrolling tab) to set the number of lines of output that you can scroll back in the Terminal window. The more lines that are saved for scrollback, the more memory in kilobytes is needed to store that information. Setting the scrollback value in lines or kilobytes will automatically change the amount of the other value (the more lines you save, the more memory is required).

    In the Konsole window, you can search back through all scrolled output, to find the text you are looking for. Click Edit⇨Find in History and type the text you are looking for into the search box.

  • Changing font size-If you are having trouble reading some text, you can change the font size in the Terminal window. In the GNOME Terminal, press Shift+Ctrl++ to zoom in to a larger font or Ctrl+- (Ctrl and a minus sign) to zoom out to a smaller font. In Konsole, click Settings⇨Font, then select the font size you want (Normal, Tiny, Small, Medium, Large, or Huge). Or, in Konsole, choose Custom to select a particular font style and size.

  • Changing colors-You can change the default black text on white background in both Konsole and GNOME Terminal windows. In Konsole, click Settings-Schema, then choose a color combination. In GNOME Terminal, click Edit⇨Current Profile, then select the Colors tab. By default, GNOME Terminal will use colors for the Terminal window, based on the current system color scheme. As an alternative, you can select from among several built-in schemes or create your own.

When it comes to performance, a single GNOME Terminal will consume more memory than a single Konsole Terminal. However, the GNOME Terminal will use the same process for each new window and increase memory usage only slightly, while each new Konsole window starts a new process and consumes the same amount of memory again.

To improve performance of GNOME Terminal, reduce the scrollback buffer. Not only do large scrollback buffers consume memory, but a process that is doing a lot of text scrolling in a Terminal window (such as a kernel build or listing a large file) can greatly harm overall performance. Listing a large text file to the screen on a virtual terminal (no X or Terminal window overhead) might take only one-tenth the amount of time to complete. On an xterm, text scrolling will be done in about half the time of that required on a Konsole or GNOME Terminal window.

Features that drain performance in the GNOME Terminal window include using background images or transparent backgrounds (see the Effects tab on the GNOME Terminal Current Profile window). A plain, solid-colored background will yield best performance.

Choosing and using a Shell

While bash is the default shell used in Fedora Linux, it is not the only one available. There are more than a half dozen different types of shells included with Fedora. Each shell includes some different features and different configuration files. Shells that come with Fedora and most Red Hat Linux systems include the following:

  • ash-The ash shell is a copy of the Bourne shell (sh) that was part of the Berkeley UNIX distribution. While ash includes many features that are in the sh shell, it is more lightweight than sh.

  • bash-The bash shell is the default shell used with Fedora and most Linux systems. It was originally created to be compatible with the Bourne shell (sh), but has since added features from C shell (csh) and Korn shell (ksh), as well as some features of its own. Bash configuration files include $HOME/.bashrc (for each user) and /etc/bashrc (for all users).

  • ksh-The Korn shell (ksh) was created at AT&T and became the command predecessor to the Bourne shell (sh). In Linux, the pdksh shell (which is linked to ksh) is the open source version of ksh. When ksh is started as a login shell, shell settings are read from the user's $HOMF/.profile, as well as from the system /etc/profile file.

  • csh-The C shell (csh) was the default shell commonly used with Berkeley UNIX systems (including early UNIX versions from Sun Microsystems). The tcsh shell is the open source version of csh that comes with most Linux systems (although you can call it from csh as well). Configuration settings for the C shell are stored in $HOME/.tcshrc (for each user) and /etc/csh.cshrc and /etc/csh.login files (system wide).

  • sh-The Bourne shell (sh) was the standard shell used in early UNIX systems. In Linux, the sh command actually runs the bash command in sh compatibility mode.

    Note

    The sh-compatibility mode in bash is not completely compatible with sh. If you request a bash feature that is not in sh, the feature will not fail (as it would in sh) if bash is able to provide it. Also, some sh features, such as how braces are handled, are done differently in bash. Start sh with the +B option if you need to use the Linux sh in a way that is completely compatible with the traditional sh shell.

  • zsh-The Z shell (zsh) is similar to the Korn shell, with many enhancements added. Extra features in zsh include spelling correction, programmable command completion, autoloading of shell functions, loadable modules, and an FTP client, to name a few. To add setting to be used by zsh, you can use several files in the user's home directory (.zshrc.zwc, .zshrc, .zshenv, .zproflie, .zlogin, and .zlogout) and in /etc (zshenv, zprofile, zshrc, zlogin, and zlogout).

When you add a user in Fedora Linux, bash is assigned as the default shell. You can change the default shell for a particular user, by running usermod with the -s option. For example:

   # usermod -s /bin/tcsh jjones

This example changes the default shell (the one that starts when you open a Terminal window or virtual terminal) to /bin/tcsh for the user jjones. When you first add a user, you can also indicate that the user be assigned a particular default shell. The following example adds a user named tsmith and assigns the ash shell to this user.

   # useradd -s /bin/ash tsmith

To change the default shell (for example, to csh) for all users that you add in the future, type the following.

   # useradd -D -s /bin/csh

Because bash is the default shell (and used by most people with Linux), for a quick review of how to tune your shell to suit the way you work, I focus on the bash shell.

Tuning a Bash Shell

Each time a bash shell starts up, it reads several configuration files. If there are any options you want to add to bash, you can add them to one of those files. In order, it reads the following files and directories (assuming they exist):

  • /etc/bashrc-System-wide bash configuration file, read by shells started by anyone on the system. This file sets the default umask to 022, so files created from the shell have 644 permissions and directories have 755 permissions). It also sets the terminal to an appropriate value (such as xterm or linux) and the command-line prompt (to [user@host curr_dir], showing the user name, host name, and current directory). You can override any of these values in your personal .bashrc file.

  • /etc/profile.d-This directory contains files that set configuration information for different applications that are needed for each type of shell you run. For example, the colorls.csh and colorls.sh define the colors in which different objects appear when you type an Is from the screen. You can add your own files, or modify existing files if you don't like the way certain features are handled.

  • $HOME/.bashrc-Each home directory can contain its own .bashrc file (notice the leading dot in the filename). The file is the best place to put any personal settings you want to use for your shell environment.

Here are some examples of the kinds of settings you might want to add to your personal .bashrc, system-wide /etc/bashrc, or application-specific /etc/profile.d/ file.

   set -o vi
   set -o notify
   alias rm-"rm -i"
   alias c="tput clear"
   alias la="ls -Al"
   export D="/home/chris/documentation/book1"

Each of the commands just shown is an example of the type of command you can add to a shell configuration file (permanent) or type directly into the shell (temporary). The first two set commands assign the vi command to be used to edit your command lines and history (emacs is the default) and cause the shell to notify you immediately when a background job completes (the default is to tell you at the next command prompt).

The alias commands rm, c, and la set what happens when those commands are run. The rm -i causes you to be prompted before removing each file, tput clear clears your screen, and ls -Al prints a long listing of files, including dot(.) files. The export example is simply used here to set a long path name to a single variable. You could then type cd $D to go to the /home/chris/documentation/book1 directory or the command vi $D/index.html to edit the index.html file from that directory.

There are many other options you might want to set to configure your shell. I recommend that you refer to the bash man page for details on built-in commands and options you can use.

Useful Shell Commands

Again, if you are not familiar with a shell from Linux, you should really get an introductory Linux book to learn the basic commands to use. Those commands should include cat, chgrp, chmod, chown, cd, ls, pwd, mv, rm, cp, mkdir, rmdir, ln, touch, less, and probably a few dozen others, as suits the job you need to do with Linux.

Here are a few commands that you as a system administrator engaged in troubleshooting Linux will find invaluable.

Finding and Sorting Through Files

With most configuration and log files still stored as plain text in Linux, the ability to search, sort, and list information from these files is a critical skill for Linux troubleshooters. The sort, grep, tail, head, and find commands are valuable tools for troubleshooting text files in Linux.

The sort command can be used to sort lines of text in alphabetical order. For example, to alphabetically sort the users on your system, you could type the following.

   # sort /etc/passwd | less

You can also sort by fields in a file, by identifying the field separator and the field you want to sort by. In the following example, the /etc/passwd file is sorted by the contents of the third field (the UID field).

   # sort -t :--key=3 /etc/passwd | less

The grep command is a handy troubleshooting tool because it provides a very powerful means of searching through a large number of text files for particular text. For example, let's say that 1 just removed a computer from the network named duck and want to find all of my configuration files that have a reference to it. The following command suppresses error messages (-s) and recursively (-r) checks all the files beneath the /etc directory for the string duck.

   # grep -s -r duck /etc

Options that I commonly use with grep include -i (ignore case), -n (list the line number on which the term occurs), and -w (list only if term appears as a separate word). I also commonly use regular expressions to more precisely define what I am looking for. For example, ^duck or duck$ would find only those lines in which the word duck appears at the beginning or end of a line, respectively.

A period is another handy character for indicating to match on any character. For example, let's say that you have done such a good job troubleshooting your Linux systems that you have time to do a crossword puzzle. Here's how you could search the dictionary for a five-letter word beginning with l and ending with ty:

   # grep ^l..ty$ /usr/share/dict/words
   lofty
   lusty

A lot of times when you are looking at log files or, for that matter, any file that is getting new information all the time, you might want to watch information being added to that file in real time. The tail command is a great way to do that. I often use the tail command when I am troubleshooting a particular feature (with debugging turned on) and want to watch log messages as they progress. Here's an example:

   # tail -f /var/log/messages

Without the -f option, the default of the tail command is to just list the last ten lines of the selected file. With the -f, this command waits for more lines to be written to the file and lists them as they arrive. Conversely, if you want to see only the first ten lines of a file, you can use the head command.

   # head /var/log/messages

The find command is useful for finding files by their name, rather than by content. After finding the files you want, based on the parameters you choose, you can also act on the files you find, by executing (-exec) another command on the find command line. Here are some examples.

   # find /var/log -exec grep "ip_forward" '{}' \ ; -print

The previous example finds all files in the /var/log directory and its subdirectories. Next, the command line searches (grep) the files that are found for the string "ip_forward" and lists the results. If you are troubleshooting disk space problems, you can use find to look for files over a particular size. For example:

   # find /home -size +1024000k -exec du -h '{}' \ ; -print
   1.1G /home/chris/movie1.avi
   1.3G /home/chris/movie2.avi

In the previous example, find searches the /home directory for files over 1,024,000KB. Then it executes the du -h command. This allows you to see any files that are over about 1GB and print that information in a human-readable output. By following this format you can also substitute any command you want for the du -h to act on the files found. For example, you could remove (rm), move (mv), or change permissions (chmod) of the files found.

There are many other useful options you could use with the find command. To check for a certain type of file, use -type followed by f (file), d (directory), l (symbolic link), b (block special device), c (character special device), or s (socket). The -xdev prevents the find command from searching directories on other file systems (which is useful if you have NFS files systems or CDs mounted, which you don't want to search).

Finding the State of the System

If you are coming on to troubleshoot a system that you didn't set up, it's a good idea to do just a few quick checks to find out basic information about the system. Here are a few commands you can run.

The uname command can tell you the name of your kernel (Linux), name of your computer, the kernel release it is running, the date the kernel was created, the processor type and hardware platform on the computer, and the name of the computer operating system:

   # uname -a
   Linux shuttle.linuxtoys.com 2.4.22-1.2115.nptl #1 Wed Oct 29
        15:42:51 EST 2003 1686 i686 i386 GNU/Linux

To see the amount of disk space available on all your mounted partition, type the df -h command as follows.

   # df -h
   Filesystem       Size    Used    Avail    Use%    Mounted on
   /dev/hda6         59G     52G     4.1G     93%    /
   /dev/hda1         99M    6.3M      88M      7%    /boot
   /dev/hda2        9.7G    7.4G     1.8G     81%    /mnt/backup
   /dev/hda5        4.9G    1.2G     3.5G     25%    /mnt/isos
   none             252M       0     252M      0%    /dev/shm

To see all the partitions available on all the hard disks (including those that are not currently mounted), use the fdisk -l command.

   # fdisk -l

   Disk /dev/hda: 80.0 GB, 80026361856 bytes
   255 heads, 63 sectors/track, 9729 cylinders
   Units = cylinders of 16065 * 512 = 8225280 bytes
      Device   Boot    Start     End     Blocks     Id   System
   /dev/hda1     *         1      13     104391     83   Linux
   /dev/hda2              14    1288   10241437+    83   Linux
   /dev/hda3            1289    1364     610470     82   Linux swap
   /dev/hda4            1365    9729   67191862+     f   Win95 Ext'd (LBA)
   /dev/hda5            1365    2001    5116671     83   7Linux
   /dev/hda6            2002    9729   62075128+    83   Linux

To tell if the computer was booted into graphical mode (runlevel 5), text mode (runlevel 3), or some other run level, type the runlevel command. This example shows that the previous state was N (none) and the current state is 5 (graphical). Since there was no previous run level, the computer most likely booted to runlevel 5.

   # runlevel
   N 5

To take a quick check of your hardware, run the lspci command. This will show you what PCI hardware is installed on your computer.

   # lspci
   00:00.0 Host bridge: Intel Corp. 82845G/GL [Brookdale-G]
        Chipset Host Bridge (rev 03)
   00:01.0 PCI bridge: Intel Corp. 82845G/GL [Brookdale-G]
        Chipset AGP Bridge (rev 03)
   00:ld.0 USB Controller: Intel Corp, 82801DB USB (Hub #1)
        (rev 02)
   00:1d.1 USB Controller: Intel Corp. 82801DB USB (Hub #2)
        (rev 02)
   00:1d.2 USB Controller: Intel Corp. 82801DB USB (Hub #3)
        (rev 02)
   00:1d.7 USB Controller: Intel Corp. 82801DB USB2 (rev 02)
   00:1e.0 PCI bridge: Intel Corp. 82801BA/CA/DB/EB PCI Bridge
        (rev 82)
   00:1f.0 ISA bridge: Intel Corp. 82801DB LPC Interface
        Controller (rev 02)
   00:1f.1 IDE interface: Intel Corp. 82801DB Ultra ATA Storage
        Controller (rev 02)
   00:1f.3 SMBus: Intel Corp. 82801DB/DBM SMBus Controller
        (rev 02)
   00:1f.5 Multimedia audio controller: Intel Corp. 82801DB
        AC'97 Audio Controller (rev 02)
   01:00.0 VGA compatible controller: nVidia Corporation NV34
        [GeForce FX 5200] (rev a1)
   02:06.0 Ethernet controller: Realtek Semiconductor Co., Ltd.
        RTL-8139/8139C/8139C+ (rev 10)
   02:07.0 FireWire (IEEE 1394): VIA Technologies, Inc. IEEE
        1394 Host Controller (rev 46)
   02:08.0 Multimedia video controller: Brooktree Corporation
        Bt878 Video Capture (rev 11)
   02:08.1 Multimedia controller: Brooktree Corporation Bt878
        Audio Capture (rev 11)

As an alternative, you can use the dmesg command. Besides showing your PCI devices, dmesg also shows information about other hardware and drivers as they are detected and accessed by the kernel.

   # dmesg | less
   Linux version 2.4.22-1.2115.nptl
   (bhcompile@daffy.perf.redhat.com) (gcc version 3.2.3 20030422
   (Red Hat Linux 3.2.3-6)) #1 Wed Oct 29 15:42:51 EST 2003
   BIOS-provided physical RAM map:
   BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
   BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
   BIOS-e820: 00000000000f0000 - 0000000000100000 (reserved)
        .
        .
        .
   PCI: PCI BIOS revision 2.10 entry at Oxfb5d0, last bus=2
   PCI: Using configuration type 1
   PCI: Probing PCI hardware
   PCI: Probing PCI hardware (bus 00)
   PCI: Ignoring BARO-3 of IDE controller 00:1f.1
   Transparent bridge - Intel Corp. 82801BA/CA/DB/EB PCI Bridge
   PCI: Using IRQ router PIIX/ICH [8086/24c0] at 00:1f.0
   PCI: Found IRO 11 for device 00:1f.1
   PCI: Sharing IRO 11 with 00:1d.2
   PCI: Sharing IRO 11 with 02:06.0
         .
         .
         .

A recommended technique for debugging problems with hardware detection is to direct the output of dmesg to a file (dmesg > boot.msg), so that the contents of this file can be e-mailed or posted to someone who can help you figure out the problem.

Finding Out About Yourself

You can find out about your current user environment, using some of the following commands.

Using the set command (piped to less), you can step through the environment variables set for shell environment.

   # set | less
   BASH=/bin/bash
   BASH_ENV=/root/.bashrc
   BASH_VERSION='2.05b.0(1)-release'
   COLORS=/etc/DIR-COLORS.xterm
   COLORTERM=gnome-terminal
   COLUMNS=110
   DISPLAY=:0.0
   HISTFILE=/root/.bash_history
   HISTFUESIZE=1000
   HISTSIZE=1000
   HOME=/root
   HOSTNAME=shuttle.linuxtoys.com

The previous code example shows only a small part of the output from the set command. Here you can see information about your bash shell, definitions of how colors are used in terminal windows, the width of your Terminal window (in columns), and your X display number (0.0). It also shows information about your command-line history, your home directory, and host name.

You can type the groups command to see what groups you are in.

   # groups
   chris wheel

You can use the who command to find out about your current session. The output of who am i shows your username, the terminal where you are logged in (a terminal window in this case), the date and time the terminal session started, and (if applicable) the display screen you are currently on (:0.0).

   # who am i
   chris   pts/2 Mar   19 14:30 (:0.0)

To see more information about your user and group accounts, type the id command.

   # id
   uid=500(chris) gid=500(chris) groups=500(chris)
Find Out About Others on Your Computer

If you are troubleshooting a computer that has multiple users with login accounts, you might want to list information about those users. Here are some suggestions on how to do that.

With the lastlog command you can see when a person last logged into the computer and the terminal device they used to login on. Many of the users should be listed as Never logged in because they are administrative logins that don't have valid shell accounts. So a login date listed for lp or bin accounts would probably indicate a break-in.

   # lastlog | less
   Username           Port    From            Latest
   root               pts/3   duck Sun Mar 21 21:24:22 -0800 2004
   bin                                        **Never logged in**
   daemon                                     **Never logged in**
   adm                                        **Never logged in**
   lp                                         **Never logged in**
   sync                                       **Never logged in**
   chris              tty4         Fri Mar 19 15:41:46 -0800 2004
   robby              :0           Fri Jan 23 22:15:03 -0800 2004
   two                tty1         Thu Jan 22 16:24:26 -0800 2004

Using the finger command, you can see who is logged in and actively working at the moment. Notice that jimh has been inactive for 2 days. There are two root logins that have been inactive for over 3 and 6 hours respectively. The other two root logins are currently active. (You can run the pinky command to get similar output.)

   # finger
   Login     Name      Tty     Idle   Login Time   Office Office Phone
   jimh      Jim Hall  tty4     2d    Mar 19 15:41
   root      root      tty2    3:36   Mar 19 14:29
   root      root      *:0            Mar 18 12:10
   root      root      pts/2          Mar 19 14:30 (:0.0)
   root      root      pts/3   6:38   Mar 21 21:24 (duck)

Again, there are hundreds of commands you will want to use over time. I recommend getting a basic text on Linux or UNIX use and administration, to get more information on how to use the shell and shell commands.


Obtaining Root Privilege

The root user is the primary user account used for Linux system administration. Despite file ownership and group assignment, the root user always has the ability to change, add, or delete any file or directory on a Linux system (provided that the device itself is not a read-only device).

Troubleshooting and administering any standard Linux or UNIX systems begins with the root user. While regular user accounts may be able to display information about your system and even modify selected features, the root user is expected (in fact, is the only user allowed) to do most system-wide configuration changes to a Linux system. This section goes through ways of accessing and sharing root privilege in Linux.

Getting Root Access From the Shell

Troubleshooting system-wide features almost always means acting as root user. The most common ways to obtain root privilege from a shell are as follows:

  • Login as root-Whether you login to a computer over the network using ssh (described later in this chapter) or from a login prompt, you can simply login with root as the username and, when prompted, the root user's password. By default, the prompt will end in a pound sign (#) and all subsequent commands run from the login will be run as the root user.

  • su (new shell)-By using the su command while you are logged in as another user, you can effectively become the root user for all subsequent commands from the current shell. It is most common to run the su command followed by a dash (-) as shown here.

    $ su -
Password:
#

    The dash causes the su command to open a new shell for the root user, as a login shell. So all configuration files read during an initial login (such as the /etc/profile file) are read before starting the new shell you just opened with su.

    If you forget to use the dash after the su command, the most common failure you will experience is that your shell will not find common administrative commands that are in /sbin, /usr/sbin, and /usr/local/sbin directories. This is because the /etc/profile file is not examined to include these directories in the root user's PATH. The common annoyance you will experience is that you are changed to the /root directory as your current directory, so you might need to return to the directory you were in if you want to do some work there.

    Note

    The su command, though commonly used to become the root user, can be used to change to any user that has a valid shell account on your system. (You can't su to a user that has /sbin/nologin or /bin/false as its default shell.)

  • su (single command)-If you just want to execute a single command as root user, you can use su with the -c option. Simply follow the su command with -c and the command you want to run (along with any arguments), and then provide the root password when prompted. For example, if you want to create a directory in a location that is restricted to root-only access while you are logged in as another user, you could run the following.

    $ su - -c "mkdir /root/tmp"
Password:

    In this case, the -c tells the su command to run the command in quotes as the root user. After entering the root password you have full root privilege to create, in this case, a directory that requires root privilege to do so. As soon as the command completes, you return to your original shell (which was presumably owned by a non-root user).

  • sudo-As an alternative to logging in as root or using su to get to root, you can configure the /etc/sudoers file to allow selected users to have limited (or unlimited) root access on your Linux system via the sudo command. To change the /etc/sudoers file, run the following command.

    #visudo

    The visudo command locks the /etc/sudoers command while you edit it with the vi command. The following are some ways of configuring the sudoers files, most of which are based on entries that you can activate from within the default sudoers file itself.

    # User privileges
root      ALL=(ALL) ALL
sandy     ALL=/usr/bin/lprm, /usr/sbin/lpc

# Group privileges
%wheel    ALL=(ALL) ALL
%wheel    ALL=(ALL) NOPASSWD: ALL
%users    ALL=/sbin/mount /mnt/cdrom,/sbin/umount /mnt/cdrom
%users    localhost=/sbin/shutdown -h now

    The privilege specification for the root user allows root to have complete access to the Linux system as it relates to the sudo command (ALL). The second line shows the user sandy being given the privilege to remove printer jobs (lprm) and control printers (lpc). This indicates how you can give one person the privilege to do a certain task (such as manage printers), without giving the person full control of the Linux system.

    The first wheel line gives complete access to all users in the wheel group (as listed in the /etc/group file). To gain that access, users in the wheel group have to use the sudo command and (when prompted) enter their own password (not the root user password). The second wheel line gives users in the wheel group the same access as the previous line, except that they can do so without entering a password.

    The two %users lines shown in the example represent what functions those who belong to the users group can do that are typically restricted to root user privilege. Presumably you would add all regular user accounts (by default, UID 500 and higher) to the users line in the /etc/group file. In the first case, these users would be able to mount a CD (/sbin/mount /mnt/cdrom) and unmount a CD (/sbin/umount/mnt/cdrom). In the second case, anyone in the users group on the localhost will be able to shutdown the computer (using the /sbin/shutdown -h now command).

    Caution

    As a Linux troubleshooter, on a multiuser system it may be tempting to provide limited administrative capability to other people on that system. However, be warned that some commands (such as the vieditor) include features that let the user escape to a shell. That will effectively give the user full access to your system and undermine the intention of providing limited access to the user.

Getting Root Access From the GUI

A nice feature of the Red Hat graphical administrative tools that come with Fedora and other Red Hat systems is that they can be run from any user's desktop. If a regular user launches an administrative window, either from the main menu or by running a redhat-config-* command, the user is prompted for the root password before being allowed to continue.

Figure 6-2 shows an example of the Query window prompting for the root password before allowing the user to access the Network Configuration window.

Click To expand
Figure 6-2: Before you can run GUI administration tools, you are prompted to enter the root password.

After you have entered the root password, look at the panel at the bottom of the desktop. A pair of keys should appear, indicating that you continue to be authorized to run applications that require a root password. Click on those keys to choose to keep that authorization, or forget about it.


Examining System Processes

When some software or hardware seems to be broken in Linux, there are some standard tools you can use to monitor what is happening with your system. Temporarily slow performance can be monitored using tools such as top, to see what the processes are doing at the moment. If an application isn't completing properly, you can trace the system calls it is making, by using the strace command.

Tracking Processes with Top

If system performance takes a quicknosedive, top is a good way to check out what is going on. The top command shows the running processes on your system and allows you to sort these processes based across several parameters. Its graphical counterpart in the GNOME environment, called gtop, lets you easily sort information by clicking on columns in the gtop window.

The top command doesn't require a GUI, which makes it convenient for running from a virtual terminal or after logging in over a network using ssh from a Terminal window. To start top from a shell, simply type:

   # top

A list of running processes appears, displaying activity in real time. When you start top, processes are listed in order of current CPU usage (from most to least). By default, the screen is updated every 5 seconds to show the new order of CPU usages by your processes. Press the space bar to update information immediately. Figure 6-3 shows an example of an active top window in Fedora.

Click To expand
Figure 6-3: Use top to sort processes by CPU, memory, or other system usage.

To get a feel for the information you are seeing on the top screen, you can try sorting the data in different ways. With top running and the Terminal window it's running in selected, sort the processes in the top window in the following ways:

N | Display processes numerically, by process ID
A | Display processes by age, with the newest processes being displayed first
P | Display processes by CPU usage (highest to lowest)
M | Display processes by resident memory usage (highest to lowest)
T | Display processes by total CPU time each process has consumed

Sorting this data in various ways can help you uncover problems on your system. Here are a few things you can do with top:

  • Check for memory usage-Press the M key to sort the top output by memory usage. Look under the RSS line to see the actual amount of physical memory being used by each process (such as 40M for 40 megabytes). Under the %MEM column, you can see the percentage of total physical memory the process consumes. Shared memory used by the process is listed under the SHARE column.

    The Mem: line in the upper section of the top screen shows the total memory available on the system, along with how much is used and how much is free. It also shows shared memory and memory used for buffers. Swap information shows the total size of your swap area, along with how much of it is free or currently being used.

    A large amount of swapping will cause a performance drag. Adding more RAM will improve performance in such situations.

  • Check for memory leaks-If an application has a memory leak (the memory it is consuming continues to grow over time), sorting by resident memory usage will point up processes that have grown out of whack with their memory use. Performance hits tend to really occur in cases where the growing process has caused frequent hits on swap space.

    To check whether your system is frequently relying on swap space, try the free command.

       # free -m
                  total used free shared buffers cached
   Mem:           502           389    112        0        42      213
   -/+ buffers/cache:           133    369
   Swap:          596             0    596

    In this example you can see that no swap space is being used, so if there is a memory leak, it probably isn't adversely affecting the system.

    In some cases, processes that may appear to be experiencing memory leaks may not be leaking at all. An example is the gnome-terminal process (which is associated with a Terminal window). Over time, the gnome-terminal process may consume larger amounts of memory. This comes from the fact that each time you open a new gnome-terminal window, the new window relies on the old gnome-terminal process, but simply causes a small increase in the process' memory use.

  • Watch only selected processes-If you want to monitor only a few processes, you can enter the PID for the commands you want to watch on the top command line (up to 20 processes). Here is an example of top being used to monitor three different processes with PIDs of 4043, 4584, and 4353.

    # top -p4043 -p4584 -p4353

  • Check general processor activity-There are a lot of things you can check regarding the general status of the CPU. The first line on the top screen shows the current time, followed by the uptime (how long the system has been running) and the load average. The three load average numbers represent the average number of processes ready to run in the previous 1, 5, and 15 minutes.

    From the second line on the top screen, you can see the total number of processes currently running (since the screen was last refreshed). You can also see the number of processes that are sleeping (waiting for something to happen), running, or stopped. Zombie processes, which are also listed on this line, are actually processes that are dead but remain in the process table because their parent processes have not checked their exit status.

    High load averages might indicate that the CPU is the bottleneck for slow performance. Press P to sort by processor usage to see which processes are making the most demands on the processor (as indicated by the %CPU column). Try some processor-intensive applications, such as 3D-graphics or video or audio compression to check out how much the processor can handle.

    Sleeping and zombie processes are not drawing on the CPU. While some people find zombie processes disconcerting to have around, when exited their resources are released, so they store only a small amount of data (the exit status and process ID of the now defunct process). To see which of your processes are zombies, list the current processes (ps ax) and look for a process with a Z in the STAT line.

  • Give processes more priority-If you are using top as the root user, you can increase or decrease the priority of a running process (referred to as re-nicing the process). Press the r key, then type the PID of the process you want to re-nice. Next, enter the priority number you want to use. Priorities you can enter as root user range from 19 (lowest priority) to -20 (highest priority).

    Caution

    Re-nicing can be dangerous. It can cause some processes to hog the CPU beyond where you would intend, while causing others to be shut out from the CPU. Use this feature at your own risk.

  • Kill processes-While you are on the top window, you can kill (or at least try to kill) any process you like. Press the k key and enter the PID of the process you want to kill. You are asked for the signal to use. By default, it will use the -15 signal (SIGTERM). Sometimes to kill a process, you might need to use -9 (SIGKILL) instead. (SIGKILL cannot be ignored and will kill the process without cleaning up.)

You can simplify the information you see in the top window in several ways. Press the i key to get rid of idle processes and just see the processes that are currently demanding attention from the processor. Normally, top just shows however many processes will fit on the screen. Press the t key to toggle some of the display summary information on and off.

Tracing Down Failed Commands with Strace

Sometimes you run a command and it just hangs or fails outright with no indication of what went wrong. To find out what exactly is going on with the command, you can run the command using strace.

With the strace command, you can watch the system calls being made through the entire run of the command. I most often use strace to see where a command is getting configuration information from and what devices it's trying to open. As each call is made, a listing of that call is sent to standard error. Here's an example.

   # strace /bin/ping anyhost.linuxtoys.net 2>/tmp/ping_out

Here I try to ping a host computer on the network. By running the ping with the strace command and directing standard error (2>) to the /tmp/ping_out file, a ton of processing messages are sent to the ping_out file. Just to get a quick idea of where ping is going to get its information from, I look for all the occurrences of open to see what files it is opening. Here's the result.

   # grep ^ open /tmp/ping.out
   open("/etc/ld.so.cache", 0_RDONLY)       = 3
   open("/lib/libresolv.so.2", 0_RDONLY)    = 3
   open("/lib/tls/libc.so.6", 0_RDONLY)     = 3
   open("/etc/resolv.conf", 0_RDONLY)       = 4
   open("/etc/nsswitch.conf", 0_RDONLY)     = 4
   open("/etc/ld.so.cache", 0_RDONLY)       = 4
   open("/lib/libnss_files.so.2", 0-RDONLY)  = 4
   open("/etc/host.conf", 0_RDONLY)         = 4
   open("/etc/hosts", 0_RDONLY)             = 4
   open("/etc/ld.so.cache", 0_RDONLY)       = 4
   open("/lib/libnss_dns.so.2", 0_RDONLY)   = 4

Besides opening the libraries, the ping command opened several configuration files to look for IP address information. I could see that it opened the resolv.conf, nsswitch.conf and host.conf files to find out where it should look to resolve names to IP addresses. Locally, it checked the hosts file for addresses.

From this output, I would be able to tell if any configuration files ping wanted could not be found or opened. Failed opens usually appear with a -1 return value and a No such file or directory message. Besides opens, you can also look for system calls such as lstat (which check the existence of a file), read (to see calls that read information from files), and signal symbols and strings (to see processes being interrupted).

There are many options to strace that you can read about on the strace man page. You can trace child processes (-f) and print relative timestamps (-r). You can also have it trace only certain system calls. For example to just trace open system calls in the previous example, you could type:

   # strace -e trace=open /bin/ping anyhost.linuxtoys.net

Because GUI administration tools sometimes fail mysteriously, I will often run an strace on them to find out what is wrong. Launching some redhat-config-* commands with strace from a Terminal window will show where Red Hat is saving and accessing the information it is using with the tool.


Going into Rescue Mode

It is possible for something to go wrong, which could cause your system to be unable to boot from the Linux partition on your hard disk. Your master boot record may have become corrupted or your /boot partition may have, for some reason, become inaccessible.

The following procedure describes how to go into rescue mode, using the Fedora boot CD (CD #1 of the installation set). The procedure should work the same for any recent Red Hat Linux system.

  1. Insert the Fedora installation CD #1 into the disk drive.

  2. Reboot your computer. Your computer should begin booting the installation process for the CD and you should see the boot: prompt.

  3. At the boot prompt, type the following.

    boot: linux rescue
    Note

    If you need other boot options (such as nodma or vga=) to get Linux working properly, you can add these after Linux rescue.

    You are asked to choose a language.

  4. Scroll down to select your language and press Enter. You are asked to choose your keyboard type.

  5. Scroll down to choose your keyboard type and press Enter. You are asked if you want to start the network interfaces on your computer.

  6. Choose Yes to start the network interfaces on your computer. If you choose Yes, be ready with configuration information (IP address, netmask, or choose DHCP if that service is available).

    Note

    The network may not be necessary if you are simply fixing your master boot record. However, if the Linux operating system on your hard disk turns out to be unbootable, the network connection can provide a convenient way to copy critical files off of your computer.

    At this point, rescue mode looks for your Fedora or other Red Hat Linux system and tries to mount it on the /mnt/sysimage directory.

  7. If rescue mode finds your Linux system on hard disk, you can:

    • Continue, to have it mount your Linux partition from hard disk on /mnt/sysimage

    • Read-only, to have the file system mounted read-only (to let you copy and use, but not change files)

    • Skip, to not try to mount your Linux partition

      I recommend you select Continue in most cases, so you can get to and possibly change your Linux partition.

  8. If your Linux partition was mounted successfully, you will be told so. Press OK. You will see a shell prompt.

  9. Type ls /mnt/sysimage. You should see the same directories you would normally see from your system's root directory (/).

  10. To change it so that /mnt/sysimage becomes your root directory, type the following:

    # chroot /mnt/sysimage

Now, your system should appear just as it does for a normal reboot. What you do next depends on what went wrong. Here are some possibilities:

  • You want to reinstall your master boot record. If you are using GRUB as your boot loader and you want the MBR to go on the first IDE hard disk on your computer, type the following:

    # grub-install /dev/hda

  • Copy files off. If the system seems to be unbootable, use any commands you are used to for backing up or copying files from the system.

If you have corrected the problem, you should be able to remove the boot CD from the drive and reboot your computer from hard disk.


Detecting and Configuring Hardware

Fedora and Red Hat Linux systems use the kudzu utility to automatically detect when hardware is physically added and removed from your computer, and then configure that hardware as best it can. For most popular hardware, kudzu works quite well. Other times, however, you might need to add the drivers or modules you need to use the hardware yourself.

Using Kudzu to Detect Hardware

Most people never run kudzu manually at all. When you reboot your computer after adding a new network card, storage device, modem, or other piece of hardware, kudzu runs automatically and asks how you would like to configure that hardware. What kudzu does is:

  • Probes your computer hardware (devices connected to the PCI bus, serial ports, parallel ports, usb ports, pcmcia slots, SCSI connections, and so on).

  • Compares the hardware it detects against the system's current hardware listed in the /etc/sysconfig/hwconf file.

  • Prompts you to configure or remove hardware, based on the differences in the two hardware lists.

  • Updates the /etc/sysconfig/hwconf file to reflect the new hardware configuration.

In many cases, kudzu will find the right modules needed by the hardware and configure your system to load them automatically each time your system boots. If you want to use kudzu without rebooting, you should first shut down your graphical desktop interface. You can do that by changing to runlevel 3 (init 3). Then, as root user, simply type kudzu.

Configuring Hardware Modules

If kudzu can't configure your hardware drivers, or if you want to configure the drivers differently, you can use tools from the modutils package to load the modules you need to get your hardware working.

Using modules, instead of having drivers for every possible piece of hardware built into the kernel, allows the kernel to contain a minimal set of drivers. When hardware is added and removed, associated modules can be loaded and unloaded with only a small amount of overhead.

The following is a brief overview of utilities in the modutils package for loading, unloading, and listing information about modules.

Checking Loaded Modules

To see what modules are currently loaded on your system, you can use the lsmod command. Here's an example.

   # lsmod | less
   Module               Size  Used by    Not tainted
   parport_pc          18468   1  (autoclean)
   lp                   8356   0  (autoclean)
   parport             36800   1  (autoclean) [parport_pc lp]
   autofs              12052   0  (autoclean) (unused)
   pcmcia_core         55232   0
   via-rhine           14672   1
   mii                  3992   0  [via-rhine]
   floppy              56348   0  (autoclean)
   sg                  34796   0  (autoclean)
   sr_mod              17016   0  (autoclean)
   ide-scsi            11856   0
   scsi_mod           109384   3  [sg sr_mod ide-scsi]
   ide_cd              34176   0
   cdrom               34048   0  [sr_mod ide-cd]
   keybdev              2624   0  (unused)
   mousedev             5204   0  (unused)
   hid                 23652   0  (unused)
   input                5824   0  [keybdev mousedev hid]
   ehci_hcd            19784   0  (unused)
   usb_uhci            25740   0  (unused)
   usbcore             77984   1  [hid ehci-hcd usb-uhci]
   ext3                70532   3
   jbd                 50796   3  [ext3]

The information for each module includes the module name, the module size (in bytes), and the number of modules using that module. Information in the last column is either unused (if the module is loaded but not being used), autoclean (if the module can be autocleaned using rmmod-a), or the names of modules that depend on this module.

If you are interested in what any of the modules are, you can use the modinfo command. Here is an example.

   # modinfo parport_pc
   filename:    /lib/modules/2.4.22-
                1.2140.nptl/kernel/drivers/parport/parport_pc.o
   description: "PC-style parallel port driver"
   author:      "Phil Blundell, Tim Waugh, others"
   license:     "GPL"
   parm:        io int array (min = 1, max = 16), description
                "Base I/O address (SPP regs)"
   parm:        io-hi int array (min = 1, max = 16), description
                "Base I/O address (ECR)"
   parm:        irq string array (min = 1, max = 16),
                description "IRO line"
   parm:        dma string array (min = 1, max = 16).
                description "DMA channel"

Here you can see that the module is a "PC-style parallel port driver". You can also see parameters that can be passed to the module (parm:). Other options to modinfo let you display partial listings of the information just shown. For example, modinfo - p module lets you just view parameters used by the module.

Loading Modules

To manually load a module (that was either not detected or improperly detected), you can use the modprobe command.

Note

The insmod command can also be used to load modules in Linux. However, insmod loads only the requested module and does not load modules on which the one you want depends. Module dependencies queried by modprobe before loading a module are contained in the /lib/modules/*/modules.dep file.

Supported modules that come with Fedora and other Red Hat Linux systems are typically stored in /lib/modules/*/kernel directory (where you replace the asterisk with the name of your current kernel). There are modules listed as unsupported by Red Hat that are contained in the /lib/modules/*/unsupported directory.

Before just randomly loading the modules, you should know which modules are needed by the hardware you want to use. If you have the kernel-source package installed, you can read descriptions of many of the modules in the /usr/src/linux-*/Documentation directory for your kernel.

To load a module that is included in a Red Hat distribution using the modprobe command, you can simply type modprobe and the module name. For example:

   # modprobe appletalk

The modprobe command will first look in the /etc/modules.conf for anything required of this module. Then it will look in the /lib/modules/*/modules.dep file and load any modules listed as being needed by the one you want. Then it will load the module itself. If there are options you want to pass to the modules, you can also do that when you run modprobe. For example:

   # modprobe pwc size=cif fps=10

The pwc module is used for several different models of Philips Webcams. The options shown here specify the resolution of the image (cif is 352 × 288) and the number of frames per second being used (in this case, 10). I found the descriptions of these particular parameters in the /usr/src/1inux-*/Documentation/usb/philips.txt file.

If you are getting third-party modules, you should be careful that:

  • You know and trust the source of that module.

  • You know whether you are using a tainted module. A tainted module is one that is not distributed under the GPL or other acceptable license. Some vendors distribute modules in binary-only format. Because the Linux community cannot fix problems that might occur with these modules, kernel developers will not respond to problems that occur with these tainted kernels. There are times when you must use a binary-only module to use the hardware at all. (1 personally use binary-only modules for my NVIDIA video card and my Webcam.)

To continue with my Philips Webcam example, there is a pwcx decompressor module available to use with the pwc core driver as a plug-in. Although this module will allow larger images and higher frame rates, it is a binary-only module and is considered tainted. If you decide that you want to load this module anyway (which you do at your own risk), you need to force installation of that module. To temporarily try the module (with the module downloaded and unzipped to the current directory), type the following:

   # insmod --force ./pwcx.o
   Warning: kernel-module version mismatch
            ./pwcx.o was compiled for kernel version 2.4.23
            while this kernel is version 2.4.18-14
   Warning: loading ./pwcx.o will taint the kernel:
   non-GPL license - Proprietary. See
   http://www.smcc.demon.nl/webcam/tainting.html
     See http://www.tux.Org/lkml/#export-tainted for information
            about tainted modules
   Warning: loading ./pwcx.o will taint the kernel: forced load
   Module pwcx loaded, with warnings

As you can see, there are warnings about tainting the kernel, but it loads the module anyway. Follow the links shown in these warnings to see some good descriptions of the issue.

The behavior of modprobe can be modified by adding entries to the /etc/modules.conf file. Some of the more interesting things you can do within this file include setting commands to be run before, after, or instead of loading the requested module. Here are a few examples of entries added to the modules.conf file that make the modules perform better.

   post-install sound-slot-0 /bin/aumix-minimal -f /etc/.aumixrc -L \
           >/dev/null 2>&1 | | :
   post-install pwc /sbin/insmod --force / moddir/pwcx.o >/dev/null 2>&1 | | :

Here are two entries that are run after the modules they are associated with are installed. The first entry runs the aumix-minimal command after the sound-slot-0 modules (representing the first sound card) are loaded. The command reads in the values in the /etc/.aumixrc file to set initial levels for the sound card. The second line continues the example with the pwcx driver. After the pwc module is loaded, the insmod command forces the pwcx.o module to be loaded. (You would replace moddir with the full path to the pwcx.o file.)


Using Network Troubleshooting Tools

If you are responsible for troubleshooting multiple machines, you need efficient ways of connecting to those machines over a network and watching the communications that goes on between them. Tools for connecting to remote computers and troubleshooting them are quite plentiful in Linux. Some of the most basic tools you need are those that let you do remote login (such as ssh), watch network traffic (such as ethereal), and those that let you open sessions and reconnect to them later (such as screen).

Remote Login and Execution with ssh

The ssh command is the preferred command for doing remote logins between Linux and UNIX systems. Unlike older login commands (such as telnet and rlogin), ssh creates a secure, encrypted connection between the two communicating parties. In its most simple form, you can just use ssh with the host name you want to communicate with as an argument.

   # ssh myhost.linuxtoys.net
   root@myhost.linuxtoys.net's password: ******
   Last login: Tue Jun 15 01:21:46 2004 From abc.linuxtoys.net
   [root@myhhost root]#

After entering the root password (represented here by stars, although you won't see anything), you should see a shell prompt. From here, you can just use the shell as though it were your local shell. A few tips about using ssh if you are new to it:

  • The first time you ssh to a machine, if you have not configured host authentication, ssh will remind you of that, to give you the opportunity to reject the ssh session. If you accept the connection, the RSA key fingerprint it gets from the remote machine will be used at each subsequent connection to that machine.

    New Feature

    The RSA key fingerprint is stored in your $HOME/ .ssh/known_hosts file. It is possible that you may fail to connect to a remote host with ssh because the remote host changed something about its network configuration (different IP address or NIC card). You will have to remove the entry for that host from the known_hosts file to be able to ssh to the machine again.

  • Both the login prompt and the title bar of the Terminal window (if you are using the default Terminal window in GNOME) will show the user you are logged in as, the remote host name, and the current directory. Make sure you remember where you are. If you forget that you are working in a remote shell, you could mistakenly reconfigure or reboot a remote server and really make a bunch of people mad.

  • One of the coolest features of ssh that is enabled by default on Red Hat distributions is X11 forwarding. During your ssh session, if you launch an X application from the remote host, instead of ssh trying to open that application on a desktop of the machine where the application was launched, it will open on your local desktop.

  • The X11 forwarding feature is done on the ssh client machine in the /etc/ssh/ssh_config file by the line ForwardX11 yes at the end of the file. If you are hooked on GUI administration tools, this feature lets you launch them remotely from server machines that may not even have an X server running.

The ssh command can also be used for remote execution. If you just want to execute a command and return to the local shell, without having to login to the remote system, you can do that on a single ssh command line. Here is an example:

   # ssh myhost.linuxtoys.net date
   root@myhost.linuxtoys.net's password: ******
   Tue Jun 15 03:11:18 CST 2004
   #

As you can see, after entering the password for root on myhost.linuxtoys.net, the date command ran on the remote host. The output (the current date and time) was displayed on the local shell, the connection was closed, and the local shell prompt was returned.

To debug your ssh connection, you can add the -v option to the command line. This causes a bunch of debugging messages to be output to your screen (standard error). Using this verbose output can help you debug problems you might be having in authenticating to the remote computer or getting local configuration requests to work (such as X11 forwarding).

Monitoring the LAN with Ethereal

Ethereal is a graphical utility that lets you capture and analyze network protocol data from any of the network interfaces configured on your computer. You can watch every packet that is broadcast as it is delivered live, or capture and store the data to analyze later.

You typically start ethereal without options (type ethereal or launch it from the Red Hat menu by clicking Internet⇨More Internet Applications⇨ethereal). From the Ethereal window, click Capture⇨Start, then select the capture options you want, which includes at least the interface you want to listen on. Click OK to start gathering data and click Stop on the Capture window to stop gathering data.

There are lots of ways to configure Ethereal to capture as much or as little data as you want. You can set capture limits to a certain number of packets, kilobytes of data, or seconds. You can wait for your capture to complete before displaying the data or update the list of packets on the screen in real time.

Figure 6-4 shows an example of the Ethereal window, with capture options in the foreground.

Click To expand
Figure 6-4: Select capture options and display network protocol data in the Ethereal window.

On networks with any significant amount of traffic at all, you will probably want to do some filtering of the packets you collect. You can filter on many different fields and save the output to a file. You can also import capture files from many different network protocol analyzers, including snoop, atmsnoop, Surveyer, LANalyzer, Sniffer, Microsoft Network Monitor, and many more.

Keeping a Session Active with Screen

If you want to be able to login to a remote computer and leave an active session going so that you can come back to it later, you can use the screen utility. Using screen, you can detach from an active login session, then login to that computer again at another time from possibly another location and have it resume exactly where you left off.

I got this tip for using screen from Charles Mauch of the Tacoma Linux Users Group. Here's what you do:

  1. Login to a remote computer using ssh.

  2. Open a session with screen by simply typing:

    # screen

    You should now have one screen open with a shell started and ready to do your work. Change to a directory and do something like run a long make process.

  3. To open a second screen, press Ctrl+a, c (hold the Ctrl key, press a and then c). Start another ongoing activity, such as an IRC chat session.

  4. To get back to the first screen, press Ctrl+a, 0 (as in the number zero). Then go back to the second screen by pressing Ctrl+a, 1.

  5. Now, to detach from this screen session press Ctrl+a, d. The [detatched] message indicates that you are now detached from your screen session.

  6. At this point, you can log out from the remote computer completely.

  7. When you are ready to resume your screen session, login to the machine again.

  8. To resume your active screen sessions, type screen-r.

  9. At this point, you can again bounce between the different screens (Ctrl+a, 0 and Ctrl+a, 1) and detach and resume from the session as you please.

Screen gives you the freedom to start a long compile or stay connected to a remote session of some sort, which you might not be able to do if you knew you were going to have to disconnect before the activity is completed. As a Linux troubleshooter, you can leave a sticky debugging problem and know that you can get back right where you were (with your current directory, command-line history, and running commands) exactly where you left them.

Trying Network Troubleshooting Basics

If you are having difficulty communicating from one system to another, here are a few basic commands that most troubleshooters will use first from a Linux system:

  1. Run ping. Running a ping command will tell you immediately if a remote system is up and running. The time shown on each output line of ping will indicate how long a packet takes to make a round trip.

  2. Some servers block the port that ping talks to since it can be used for denial of service attacks. In those cases, you can try using the telnet command to connect directly to a particular port that you know is accepting requests. For example, if the remote computer is a mail server, you can try using telnet to connect to the simple mail transfer protocol (SMTP) port 25. For example:

    # telnet toys.linuxtoys.net 25
lrying 10.0.0.1...
Connected to toys.linuxtoys.net (10.0.0.1),
Escape character is ']'
220 toys.linuxtoys.net ESMTP Postfix
~.

    In this example, we connected to the SMTP port (25) on toys.linuxtoys.net. We saw that we connected to the service from the line 220 toys.linuxtoys.net ESMTP Postfix. Once we knew the server was up, we could just disconnect by typing ~. on a line by itself.

  3. If those and other network commands seem to just hang, it may be that your network connection is not active. You can try starting or restarting the network by typing:

    # service network restart
Shutting down interface efh0:   [ OK ]
Bringing up interface eth0:     [ OK ]

    If your Ethernet interface (eth0) doesn't come up, check for a disconnected cable, possibly a bad Ethernet card, or a misconfigured network interface. Also, check the link lights on both ends of the link.

  4. If you can get on your network, but can't get on the Internet (or other remote network), try to ping the router or machine acting as your gateway to Internet. Then try to ping an address past that gateway.

  5. If you can get out on the network, but you can't reach a particular host computer, try running a traceroute.

    # traceroute linuxtoys.net

    This will give you an idea of where your connection is not going through.

  6. If you are getting on to the network, but are not getting a response from a computer that you know is up, try watching network traffic with ethereal or tcpdump.

In most cases, running the simple set of commands will give you a sense of where your connection to the network stands.


Summary

To begin Linux troubleshooting, you should be familiar with a few basic tools. Mastering the shell and various shell tools is a must for a serious Linux troubleshooter. Choosing and tuning the shell of your choice will make your work more efficient.

Knowing basic commands for finding and sorting files, checking the state of your system, and monitoring users will make it easier to watch over your computer's resources. Learning ways to obtain and use root privilege is necessary to get to critical configuration tasks.

While there are many tools for monitoring your Linux system to prevent or react to trouble, some tools are particularly valuable. The top command lets you display and manage running processes. Features like kudzu, modprobe, modinfo, and Ismod help you configure your computer's hardware. The strace command can let you watch every system call a command makes to track down problems it might be having.

There are some commands that almost every Linux administrator uses to troubleshoot network connections. The ssh command is now the most common way to do remote login among Linux systems. Ethereal and tcpdump can let you watch and analyze network traffic. With screen, you can keep sessions active, even if you have to disconnect from them for a while. Commands like ping, telnet and traceroute can tell you if a remote machine is alive and track down failed network connections.


Chapter 7: X Troubleshooting: Video, Mouse, and Keyboard

A dysfunctional video card can stop new Linux users in their tracks. Getting video card, monitor, and desktop settings tuned just right can take some doing, even for an experienced system administrator. Without a working graphical user interface, a desktop system can end up stuck in command line mode.

This chapter will help you get your video card, monitor, display, mouse, and keyboard all working happily together. It also points you toward resources where you can get further help if you need it.

Beginning with X (Video, Mouse, and Keyboard)

All graphical user interface (GUI) activity in Fedora is based on the X Window System (www.x.org/X11.html)-often simply referred to as X. Likewise, most other Linux and UNIX systems are based on X. The open-source version of X that is included with all versions of Red Hat Linux and Fedora Core 1 is XFree86 (www.xfree86.org). As of Fedora Core 2, the X Window system is being taken directly from the X.org source code.

An interesting aspect of X is that the programs you use (X clients) don't have to run on the same computer as the X display (X server). This presents some very powerful possibilities, such as using low-end computers as displays while actual applications may actually be launched from more powerful computers on the network. (Some people are confused about the fact that the X server runs on the local display and the clients can run from remote displays, and pop up on the local display.)

A casual user will not even deal with any of the X configuration files directly, since the setup of those files is typically done during installation (where the video card can be automatically probed and configured) or using tools such as the Display Settings window (redhat-config-xfree86 command). If all goes well with those tools, the user can simply:

  • Boot to the GUI-During Fedora Core installation, if you successfully configured X, the default run level will be set to 5. This causes the computer to boot to a graphical login screen, so that X is running for each login session.

  • Run startx-If X was not configured, the computer will probably boot to run level 3. Runlevel 3 is a text-only mode. You may have to run the redhat-config-xfree86 or XFree86 commands to configure your video card and monitor. To start the GUI after running those commands, you can run the startx command.

X by itself is not very usable. Run the X command from a text log in and you get a plain gray screen with an "X" as your mouse pointer. There are no menus, windows, colors, panels or anything else you need to use a desktop. There are not even any controls to exit from X (press Ctrl+Alt+Fl to switch back to a virtual terminal and kill the X process).

You need to add a desktop environment, or at least a window manager, to be able to work with an X desktop. After X starts, by default you will see the GNOME desktop in Fedora and Red Hat Linux. GNOME provides the desktop environment: menus, panels, window borders, icons, and definitions of things such as how your keyboard and mouse behave. There are many window managers available for X, but Fedora Core lets you choose among the following:

  • GNOME-The GNOME desktop environment (www.gnome.com) is installed by default with most installation types of Fedora Core. On the surface, GNOME includes components to manage windows, set preferences through a Control Center, launch applications and applets from panels, and manage sessions and files. Underneath, there are libraries for creating applications that can take advantage of GNOME desktop features.

  • KDE-The KDE desktop environment (www.kde.org) provides more applications than its GNOME counterpart. In general, the KDE environment requires more RAM and CPU power than GNOME does to run effectively. However, KDE is also richer in features.

  • TWM-If you want a very simple display manager, without the overhead of panels, menus and everything else that comes with a full-blown desktop environment, you can bypass a full-blown desktop environment and set the window manager to TWM. The TWM desktop manager will run when you start X, with only a simple left-click menu to start applications.

Note

Run the switchdesk command from the desktop to pop up a window that lets you select a different desktop environment or window manager. Restart X (by exiting the desktop and restarting it with the startx command) to use the new desktop.

If Fedora Core was able to detect and configure your desktop environment in such a way that you are happy with your desktop (video, mouse, and keyboard) when it starts up, then you might well be done with this chapter. However, if your desktop, mouse, or keyboard does not start up correctly or require further tuning, read on.


Configuring Video, Mouse, and Keyboard

Your video card, monitor, mouse, and keyboard may be automatically detected and configured during Fedora Core installation. If they are not, there are a few things you can do to make sure they get properly configured. The following sections describe some techniques for getting a stubborn video card working, either during Fedora Core installation or later after the system boots up in text mode.

For even more stubborn configurations, try the Video Troubleshooting Tips and Tips for Particular Video Cards sections.

Configuring Video, Keyboard, Mouse during Installation

One of the first things Fedora Core does after you launch its Anaconda installer is try to probe for your display-related hardware. After you launch the install process, you should see the following messages:

Probing for video card:      video card found
Probing for monitor type:    monitor found
Probing for mouse type:      mouse found
Attempting to start native X server
Waiting for X server to start...log located in /tmp/X.log
1...2...3...4...5.... X Server started successfully.

If the X server doesn't start, you have a few choices, depending on how it fails:

  • If you don't even get far enough to see these messages, you may have problems other than your video card. Refer to Chapter 2 for information on troubleshooting installation.

  • If your video card is not detected and X doesn't start, you might need to run the installation in text mode. To start installation in text mode, type the following from the first installation boot screen:

    linux text

    Continue through the install procedure in text mode. You will have the opportunity to configure your keyboard and mouse, but not your video card and monitor. After installation, your computer will boot Fedora Core to text mode (init 3). You can then go to the section on configuring your video card after installation.

If your video card was detected during installation, you can probably run the install in graphical mode. During that procedure you have the opportunity to configure the following features:

  • Keyboard-Different languages and countries use different keyboard layouts, which can also include special characters not available in other languages. The installation process lets you choose from more than 50 different keyboard configurations.

  • Mouse-You select your mouse configuration based on the way the mouse is connected to your computer. Select USB, PS/2, Bus, or Serial. With a serial mouse, you need to identify which serial port the mouse is connected to (for COM1 use /dev/ttyS0; COM2 use /dev/ttyS1; and so on).

  • Monitor-A list of known monitors and their manufacturers is displayed. When possible, choose the exact model/manufacturer of the monitor you have. Otherwise, you can choose a generic CRT or LCD setting, based on the screen resolution you want to use. Or, you can select Unprobed Monitor and choose the horizontal and vertical sync rates based on documentation that comes with the monitor.

After installation is completed, if your computer boots to a graphical boot screen (after completing the first boot procedure), you can log in and see if the desktop is acceptable to you. However, even if you were able to graphically install, your video card may not have been detected properly. Continue to the next section to check the video card settings and configure it further.

Configuring Video Card after Installation

The Display Setting window is the interface that Red Hat, Inc., recommends for configuring video cards in Fedora Core. It's a good idea to run this window (either by typing redhat-config-xfree86 or clicking on the Red Hat menu⇨System Settings⇨Display). Figure 7-1 shows a shot of the Display Settings window.

Click To expand
Figure 7-1: Choose basic settings for your video card and monitor.

There are three tabs on the Display Settings window: Settings, Hardware, and Dual Head. The settings you configure in this window will end up in the configuration file used when the X server starts up (/etc/X11/XF86Config).

Note

If the Display Settings window fails to run (because it is unable to probe and identify your video hardware), go to the Video Troubleshooting Tips section for other ideas on how to proceed.

Changing Settings

Basic choices on the Settings tab include the screen resolution and color depth. Acceptable resolutions and color depth for your video card should be available from those boxes. You can switch among acceptable resolutions on the fly later from the Screen Resolution window (from the Red Hat menu, click Preferences⇨Screen Resolution).

Changing Monitor Type and Video Card

You can modify more critical settings by clicking on the Hardware tab and selecting Monitor Type or Video Card. You should be able to see the monitor model and video card settings that were configured during Fedora Core installation. If you can't find your exact models of monitor or video card, try the following:

  • Monitor Type-There are Generic CRT and Generic LCD display settings you can choose if your monitor's manufacturer and/or model are not listed. It is often best to choose a generic model rather than select a monitor from the same manufacturer that seems similar.

  • Video Card--If your video card is not listed, you might try VESA driver (generic) for most desktop systems. I've tried that on several systems, using an 800 × 600 resolution, just to get the GUI going. You can search for a better driver later if you like. Many laptop computers will use NeoMagic drivers. For further information on selecting the right driver for your video card, refer to the Tips for Particular Video Cards section later in this chapter.

Setting up Dual Head

By adding another video card to your computer, you can operate your display on two monitors at the same time. With this feature, you can either have the two displays operate as two different desktops or as one virtual desktop, where your desktop area simply expands across the two screens.

From the Display Settings window, select the Dual Head tab. Then click on the Use Dual Head check box. After that, from the information that is un-grayed, configure your second video card as you would your first one, by selecting monitor, video card, resolution, and color depth. Then you can select whether the two screens act as individual desktops or spanning desktops.

Configuring the Keyboard after Installation

To reconfigure your keyboard after installation (presumably because you want to use a different keyboard arrangement for a different language), open the Keyboard window (from the Red Hat menu, click System Settings⇨Keyboard).

Select the keyboard type you want to use. When you click OK, the new keyboard layout immediately takes effect (the system updates the XFConfig file with a new XkbLayout value).

Configuring the Mouse after Installation

Run the redhat-config-mouse command to reconfigure your mouse (if your mouse isn't working or if you changed to a different mouse). Since you are probably not getting around on the desktop if your mouse isn't working, the redhat-config mouse command will start in text mode if no GUI is detected. There are generic settings if you can't find the specific model of your mouse on the list.


Digging into the X Server and XF86Config File

The major components for working with X are the X server (a process that runs in the background, overseeing the GUI) and the XF86Config file. While GUI tools are nice in most cases, if the GUI isn't working at all or if it needs tuning that isn't addressed through a GUI tool; you might need to roll up your sleeves and dig into what the X server and its configuration files are doing.

Note

If you are just looking to change the look and feel of your desktop, select Preferences from the Red Hat menu. From there, you can select Background, Keyboard Shortcuts, Menus and Toolbars, Screen Resolution, Screensaver, Theme or Windows to modify how your desktop looks and behaves.

Understanding the X Server

In Fedora Core, you typically start the X server in one of the two ways: by booting the system to a graphical login screen (which starts X automatically from system runlevel 5) or booting to a text-based login screen and starting X by running the startx command. In either case, there are several major files that the X server keys off of as it's configured in Fedora Core:

  • XF86Config-This is the system-wide configuration file (located in the /etc/X11 directory) that tells the X server the keyboard, mouse, video driver, and other critical information it needs to run properly. This file is used regardless of which user logs into the system from the computer's monitor.

  • desktop-The /etc/sysconfig/desktop file defines the default desktop environment started by X (once X itself starts successfully). By default, the value is GNOME, to start the GNOME desktop environment, but you can change it to KDE to start that desktop environment instead (assuming KDE is installed). Here's how the contents of the desktop file looks by default:

    DESKTOP="GNOME"

    Users who have not specifically selected a desktop environment will get the GNOME desktop when they log in. Changing the line to DESKTOP="KDE" (as root user) changes the system-wide default to KDE.

  • $HOME/.Xdients-default-Users can select their own desktop environment to override the default. To do that, they can simply type the switchdesk command with one of the following options: KDE, GNOME, or TWM. The command to switch to KDE would be as follows:

    $ switchdesk KDE

    The result is that the .Xclients-default file in the user's home directory will be set to execute the KDE environment (exec startkde), and the users will see the selected desktop each time they log in.

Tip

The TWM option starts the TWM window manager (/usr/X11R6/bin/twm). TWM is great if you just want a basic X environment that loads quickly. You don't get the fancy panels, icons, or workspaces. However, you do get an efficient environment if you just want to launch a simple X application and have it perform with little overhead. Once in TWM, click the left mouse button to get to the menu. If you are experienced with window managers, you can probably figure it out from there.

Besides the TWM window manager. Fedora Core also includes the Motif window manager (/usr/XHR6/bin/mwm). If you install other window manager packages, such as WindowMaker, you can launch any of them from the .Xclients-default file as well. (It doesn't work from switchdesk.)

The X server itself is actually called X (run from /usr/X11R6/bin/X). In previous versions of XFree86, there were multiple X servers available for use. Now there is just a single X server (a link to XFree86 in the same directory) that relies on having multiple modules loaded to provide the features needed by the video card. (Ways of adding modules are described in the Module Configuration section.)

You can add options to run with the X server itself. The X command is taken from the /etc/X11/xdm/Xservers file. The default line for the X server(/usr/X11R6/bin/X) that runs on the local system (local), on the first screen (:0) appears as follows in the Xservers file:

   :0 local /usr/X11R6/bin/X

You can add X options to the end of this line to have those options take effect when X is launched. (See different troubleshooting sections later in this chapter for suggestions of options you might want to use with the X server.)

Note

To view options available for your X server, type man XFree86.

Remember that X is a network-ready display server. That means that connections to display applications on the desktop can come from the local computer or (given proper permission) another machine over the network. By default, the X server listens on TCP port 6000 for display :0 (6001 for :1; 6002 for :2, and so on).

Understanding the XF86Config File

When X is launched, it starts up using information contained in the /etc/X11/XF86Config file. In most cases, you will have a single group of settings that bring together one set of screen, mouse, and keyboard definitions. Settings can be adapted, however, to handle multiple video cards and monitors.

The XF86Config file begins with a Server Layout section that identifies the primary screen, mouse, and keyboard used by the server. Here's an example of a ServerLayout for a single-head (one monitor) configuration created by the Display Settings window (redhat-config-xfree86 command).

   Section "ServerLayout"
             Identifier    "single head configuration"
             Screen 0      "Screen0" 0 0
             InputDevice   "Mouse0" "CorePointer"
             InputDevice   "Keyboard0" "CoreKeyboard"
             InputDevice   "DevInputMice" "AlwaysCore"
   EndSection

Other sections describe different aspects of the X server components. Read on to see the kind of information you can expect to find in the XF86Config file.

Caution

Be careful if you decide to edit this file manually. Always make a backup copy. Also, keep in mind that entering bad or poorly formed data into this file can possibly keep the X server from working at all and may make the file unusable by GUI tools, such as the Display Settings window.

Screen Configuration

Most PCs with a single video card and a single monitor have their screen identified as screen 0.0. If you have multiple video cards, you will have multiple screens that can each have its own set of options configured in the XF86Config file. So, in the ServerLayout section near the top of the file, you should see the following line identifying the screen:

   Screen         0           "Screen0"      0    0

Later in the XF86Config file, there is a Screen section where the attributes of that screen are defined. Associated with Screen0 is the video card device (Videocard0) and the monitor (Monitor0). There can be multiple Display subsections that can set multiple screen resolutions (Modes) for different color depths (Depth). Here's how a Screen section might appear:

   Section "Screen"
             Identifier   "Screen0"
             Device       "Videocard0"
             Monitor      "Monitor0"
             DefaultDepth      24
             Subsection "Display"
                     Depth     16
                     Modes     "1024x768" "800x600" "640x480"
             EndSubSection
             Subsection "Display"
                     Depth     24
                     Modes     "1024x768" "800x600" "640x480"
             EndSubSection
   EndSection

In this example, the screen will allow 16- and 24-bit color depths. For each of those depths, screen resolutions of 1024 × 768, 800 × 600, and 640 × 480 can be used.

Note

When you launch an X application, it is sent to the screen based on the value of the DISPLAY variable (type echo $DISPLAY to see that value). Because X lets you run applications on one computer and display them on another, screen addresses can contain host names or IP addresses to point to a display on another computer. For example, the first display on the first screen for a computer named mybox.example.com would be mybox.example.com:0.0.

Mouse Configuration

In the basic ServerLayout section, a single CorePointer must be identified to the X server. This is typically set to the first mouse definition as follows:

   InputDevice        "Mouse0" "CorePointer"

Later that Mouse0 device is defined. For example, here's an InputDevice section defining the options for Mouse0:

   Section "InputDevice"
             Identifier      "Mouse0"
             Driver          "mouse"
             Option          "Protocol" "IMPS/2"
             Option          "Device" "/dev/psaux"
             Option          "ZAxisMapping" "4 5"
   EndSection

In this case, the Protocol used for the mouse is IMPS/2 (IntelliMouse PS/2 protocol). That protocol works with many wheel mice. The protocol could be PS/2 if there is no wheel on the mouse, usb if it is a USB mouse, or Auto, to detect a serial mouse automatically.

The mouse itself is connected to the PS/2 port for the mouse (/dev/psaux) in this example. A serial mouse could use /dev/ttyS0 (for the COM1 port). The ZAxisMapping option is used to set wheel motion for a wheel mouse. (See the "Mouse Troubleshooting Tips" section for information on setting up a left-handed mouse and a multi-wheel mouse.)

Keyboard Configuration

With Keyboard0 as the keyboard identifier, an InputDevice section may look like the following:

   Section "InputDevice"
             Identifier      "Keyboard0"
             Driver          "keyboard"
             Option          "XkbLayout" "us"
   EndSection

This section sets Keyboard0 to use the keyboard driver. The keyboard layout is the English, United States (us) layout, as set by the XkbLayout option. Chances are, that would be the primary option you would want to change for your keyboard if you are using Linux for a different language/keyboard layout. However, there are several other options you could put in an InputDevice section for your keyboard device. Here are a few examples:

             Option "AutoRepeat" 500 30
             Option "XkbRules" xfree86
             Option "XkbModel" pc101

The Option values just shown represent the default values for AutoRepeat, XkbRules, and XKbModel.Autorepeat sets the first value as the delay time before a key that is held down starts repeating (500 milliseconds, as shown here) and the second value defines the number of times the key is repeated per second (30 times is the default here). For XkbRules, you could change xfree86 to xfree98 for Japanese PC-98 platforms. For many keyboards outside the United States, you could change XkbModel from pc101 to pc102. Likewise, a U.S. Microsoft Natural Keyboard can have the XkbModel set to microsoft.

To see descriptions of these and other options you can set for the keyboard driver, type man keyboard.

Monitor Configuration

The Monitor section identifies the attributes of the physical monitor (or monitors) you are using. When possible, this should identify a particular vendor's monitor, to get the best possible chance of matching your monitor's attributes. The following example shows a Monitor section that is configured for a generic 1024 × 768 display:

  Section "Monitor"
            Identifier      "Monitor0"
            VendorName      "Monitor Vendor"
            ModelName       "Monitor 1024x768"
            HorizSync       31.5 - 57.0
            VertRefresh     50.0 - 70.0
            Option          "dpms"
  EndSection

Key elements in this example are the HorizSync, which sets the horizontal sync rate to between 31.5 and 57.0 and VertRefresh, which sets the vertical refresh rate to between 50.0 and 70.0. The dpms option turns on the Display Power Management Signaling extension. This extension allows X screen savers to blank out screens after a selected amount of time or in low-power situations. It also supports standby and suspend modes.

Module Configuration

When the X server starts up it always loads the bitmap font module, and then looks to the Module section of the XF86Config file to see which other modules to load. In particular, the Module section is for loading font modules and extension modules. After those modules are loaded, specific modules from InputDevice and Device sections are loaded (such as the mouse, keyboard, and other driver options set).

The set of modules in the Module section is different for different video card configurations. Here's an example of a Module section:

  Section "Module"
            Load "dbe"
            Load "extmod"
            Load "fbdevhw"
            Load "glx"
            Load "record"
            Load "dri"
            Load "freetype"
            Load "type1"
  EndSection

In general, you shouldn't mess with the modules configured for you automatically when your card is first probed and the XF86Config file is created. However, here is information about some of the modules that might interest you:

  • dbe-Implements double-buffering to improve rendering of animation. The result should be flicker-free animation. Don't remove this module. Type man DBE to read more about it.

  • extmod-Loads some miscellaneous extensions needed to make common features work properly. Don't remove this module.

  • fbdevhw-Loads the Linux framebuffer module used to let X communicate with the framebuffer device in Linux (typically /dev/fb0). Type man fbdev or man fbdevhw to read about options for this module.

  • glx-Adds the GLX extension to your X server. This extension essentially connects together the DRI 3D library and X. Running the glxgears command tells you if the DRI extension is loaded on your X server, gives you information about the GLX module, and shows the frames per second at which your X server can display the gears (for the size window you are using).

    # glxgears -info

  • record-Adds the record extension to capture events that are associated with recording and playing back user actions in X. Do not remove this module.

  • dri-Loads the Direct Rendering Interface (DRI) to provide hardware accelerated 3D graphics. See the DRI section later in this chapter for details.

As noted earlier, the bitmap module is loaded automatically when the X server starts to handle bitmap fonts (.bdf, .pcf, and .snf fonts). The preceding example shows freetype and type1 modules loaded to handle fonts as well. The following list describes those and other font modules that can be loaded into X:

  • freetype-Loads the module that can handle TrueType (.ttf and .ttc fonts), OpenType fonts (.otf and .otc), and Type 1 fonts (.pfa and .pbf).

  • type1-An alternate backend to load .pfa and .pfb Type 1 fonts.

  • xtt-An alternate backend for loading TrueType fonts (.ttf and .ttc).

  • speedo-Loads the module to handle bitstream Speedo fonts (.spd).

Besides the modules mentioned here, there are a variety of other modules you can have loaded for special purposes as well. Libraries for these modules are contained in the /usr/X11R6/lib/modules directory.

Video Configuration

The Device section that identifies the driver to use with your video card is one of the most critical pieces of information in your XF86Config file. The Screen section (described in the Screen Configuration section earlier in the chapter) points to the device identifier (Videocard0 or Card0) that is associated with your video card. Here's an example:

  Section "Device"
            Identifier  "Videocard0"
            Driver      "savage"
            VendorName  "Videocard vendor"
            BoardName   "S3 ProSavage KM133"
  EndSection

Every video driver also has a set of available options. See the Tips for Particular Video Cards section for a procedure on adding and modifying options for the selected video driver.


Setting and Tuning Your Video Driver

While it is important to know the manufacturer and model number of your video card when you set out to troubleshoot it, the key item you need to know to get the X driver working is the video chipset being used on the card. The XFree86 project organizes its support information based on these chipsets.

Note

Information on the latest XFree86 version can be found at www.xfree86.org/current. Follow the Driver Status link to look up information on your particular card. Fedora Core 1 includes XFree86 version 4.3.0-42. The troubleshooting issues described in this section are based on that version. If you upgrade to a later version of XFree86, some of the information in this section may not apply.

While XFree86 version 3.3.6 had several different X servers available (with the one you were using linked to/usr/X11R6/bin/X), in version 4.3.0 there is only one X server (linked to XFree86), which loads the appropriate driver modules when X starts up.

If you have gone through the basic X setup procedures at the beginning of this chapter, you should have a working GUI. (If that is not the case, go to the Video Troubleshooting Tips section for further direction.) The following sections step through procedures for determining which video driver to use and tuning and testing your card.

Determining Your Video Card Driver

Follow this procedure to determine which chipset is used with your video card and to look for ways of tuning that card:

  1. With the X desktop running, start the Hardware Browser from the Red Hat menu by clicking System Tools⇨Hardware Browser (or by typing hwbrowser from a Terminal window).

  2. Click Video Cards in the left column. Information about your video card should appear on the right side of the window. Note the manufacturer and driver that are displayed for the card.

  3. Open a Web browser and go to the Driver Status for XFree86 page for the current release of XFree86 (www.xfree86.org/current/Status.html).

  4. Select the manufacturer's chipset for your card. Driver information for cards made from that chipset is displayed.

  5. Look to see whether your card is shown as supported for the current release of XFree86. If it is, note which driver should be used for that chipset.

    Note

    Check the Fedora Core Release Notes for a list of video cards for which support was added to Fedora after the latest XFree86 release. By default, opening your Web browser in Fedora will show those Release Notes from this address in your file system: file://user/share/doc/HTML/index html. In particular, the radeon driver has been updated to provide support for many new ATI video cards.

  6. Check your /etc/X11/XF86Config file to make sure that the proper video driver is being used. It should have been autodetected and added to this file for you. For example, for my S3 ProSavage KM133 video card, the Device section for Videocard0 shows savage as the driver.

  7. Once you have determined your driver, there are a few ways you can look up the options available for that driver:

    • Check the XF86Config man page-Type man XF86Config or go to www.xfree86.org/current/XF86Corfig.5.html. The Device Section part of the page lists options that can be used with many X video card drivers. Because most of these options should be autodetected, in most cases you will only use them to override the defaults.

    • Check the driver's man page-Each driver has its own specific man page that lists options specific to that driver. From the XFree86 Documentation page (www.xfree86.org/current), select Section 4 under the Manual Pages heading. Then choose your specific driver from the list. Options that are specific to that driver are listed there.

  8. Check the /var/log/XFree86.0.1og file (the 0 in the file name represents the first video card on your computer). This is a critical file for tuning and debugging your video card! As your X server starts up, details of what it is doing are sent to this file. By looking through this file, you can see what modules are being loaded and which options are being set.

    There are hundreds of lines in this file, so I've whittled it down to a handful of lines related to the video driver for my particular video card (in this case, an S1 ProSavage8 KM266 video device):

    (II) LoadModule: "savage"
(II) Loading /usr/X11R6/lib/modules/drivers/savage_drv .o
(II) Module savage: vendor="The XFree86 Project"
     compiled for 4.3.0, module version = 1.1.27
     Module class: XFree86 Video Driver
     ABI class: XFree86 Video Driver, version 0.6
(II) SAVAGE: driver (version 1.1.27mh)   for S3 Savage
     chipsets: Savage4,
Savage3D, Savage3D-MV, Savage2000, Savage/MX-MV, Savage/MX,
Savage/IX-MV, Savage/IX, ProSavage PM133, ProSavage KM133,
Twister PN133, Twister KN133, SuperSavage/MX 128,
SuperSavage/MX 64,
SuperSavage/MX 64C, SuperSavage/IX 128, SuperSavage/IX 128,
SuperSavage/IX 64, SuperSavage/IX 64, SuperSavage/IXC  64,
SuperSavage/IXC 64, ProSavage DDR, ProSavage DDR-K

    Because the driver is set to "savage" in the XF86Config file, the savage driver is loaded from the directory shown. Note the module version (you'll need this if you go to a mailing list to ask questions about X). Following that is a list of Savage chipsets that the driver supports:

    (II) Primary Device is: PCI 01:00:0
(--) Chipset ProSavage found
(**) SAVAGE(0): Depth 16, (--) framebuffer bpp 16
(==) SAVAGE(0): RGB weight 565
(==) SAVAGE(0): Default visual is TrueColor
(**) SAVAGE(0): Option "HWCursor" "false"
(**) SAVAGE(0): Using SW cursor
(--) SAVAGE(0): Chip: id 8d04, "ProSavage DDR-K"
(--) SAVAGE(0): Engine: "ProSavage"
(II) SAVAGE(0): Monitor0: Using hsync range of 31.50-57.00kHz
(II) SAVAGE(0): Monitor0:   Using vrefresh range 50.00-70.00Hz
(II) SAVAGE(0): Not using default mode "1152x768" (width too
     large for virtual size)(--) SAVAGE(0):   Found 13 modes at
     this depth:
     [13c] 1400 X 1050, 60Hz, 75Hz
     [122] 1600 X 1200, 60Hz
(--) SAVAGE(0): Virtual size is 1024x768 (pitch 1024)
(**) SAVAGF(0): *Default mode "1024x768": 75.0 MHz, 56.5 kHz,
     70.1 Hz
(II) SAVAGE(0): Modeline "1024x768" 75.00 1024 1048 1184
     1328 768 771 777 806 -hsync -vsync
(**) SAVAGE(0): *Default mode "800x600": 40.0 MHz, 37.9 kHz,
     60.3 Hz
(==) SAVAGE(0): DPI set to (75, 75)

    The actual chipset found on the device (PCI 01:00:0) was the ProSavage chipset. At this point, X goes about further identifying the driver (id 8d04, ProSavage DDR-K with ProSavage Engine) and setting the options used for the chipset. Some of the options set here are based on default values, while others can be explicitly set in the XF86Config file.

The next section describes how to change options to help debug or improve performance of your video card.

Tuning and Testing Your Video Card

To tune and test the basic operation of your video card, follow this procedure. It will let you change options for your driver, quickly turn the X server on and off, and then run some performance tests to see how well each of the options you use performs.

  1. Close any applications that are open on your desktop.

  2. As root user from a Terminal window, shut down the GUI by typing:

    # init 3

  3. If the desktop is still active, from the Red Hat menu click Log Out and log out of your current session. If you see a login prompt, log in as root user. Otherwise, obtain root privilege (su-) and just continue to work at the shell.

  4. Change your default desktop (if you like) to a simple X plus TWM window manager. This will make it quicker to test your changes. Type the following:

    # Switchdesk TWM

  5. Make a backup copy of your XF86Config file as follows:

    # cd /etc/X11
# cp XF86Config XF86Config.bkup

  6. Using your favorite character-based text editor (typically vi or emacs), open the XF8GConfig file.

    Note

    Most UNIX and Linux experts use vi or emacs as text editors. The joe editor is a bit easier to use, if you don't know vi or emacs. Type joe filename to start. Then press Ctrl+K, H to see the help screen as you work. Use the Ctrl key, followed by a letter in the Help screen to move around, change, save, and exit the file. If you get stuck, press Ctrl+C to get out.

  7. Make changes to your XF86Config file, based on suggestions in the Understanding the XF86Config File section of this chapter.

  8. Type startx to start up the X desktop. Hopefully, X comes up on a blank desktop with the TWM window manager running.

  9. Click on the desktop and open an Xterm window (Terminal window).

  10. Next, you can run some performance tests to see how the changes you made impact your basic X server performance. You can use the x11perf command with various options to check (and recheck) the performance of your X server in different activities. Here's an example:

    # x11perf -repeat 3 -reps 10 -subs 10 100 -circulate
x11perf - X11 performance program, version 1.5
The XFree86 Project, Inc server version 40300000 on :0.0
from toys.example.com
Sat Jan 24 22:00:51 2004
Sync time adjustment is 0.0000 msecs.
100 reps @ 0.0282 msec ( 35400.0/sec): Circulate window (10 kids)
100 reps @ 0.0276 msec ( 36300.0/sec): Circulate window (10 kids)
100 reps @ 0.0283 msec ( 35300.0/sec): Circulate window (10 kids)
300 trep @ 0.0280 msec ( 35700.0/sec): Circulate window (10 kids)
1000 reps @ 0.0336 msec ( 29700.0/sec): Circulate window (100 kids)
1000 reps @ 0.0333 msec ( 30000.0/sec): Circulate window (100 kids)
1000 reps @ 0.0334 msec ( 29900.0/sec): Circulate window (100 kids)
3000 trep @ 0.0335 msec ( 29900.0/sec): Circulate window (100 kids)

    The x11perf command in this example circulates a set of windows from the lowest to the top. Here, the circulate action is repeated three times, first with a group of ten windows circulated ten times (100 repetitions). Then it is repeated with a group of 100 windows, again circulated ten times (1000 times each). At the end of each group the total number of repetitions and average amount of time it took to complete the set of actions are displayed.

    There are many other activities you can run to test the performance of your X server. Here are some examples:

    # xllperf -repeat 2 -reps 1000 -ftext
# xllperf -repeat 2 -reps 500 -copywinwin500
# xllperf -repeat 2 -reps 800 -scroll500
# xllperf -repeat 2 -reps 10 -subs 10 100 -destroy
# xllperf -repeat 2 -reps 10 -subs 10 100 -resize

    With each of these command lines, you can set how many times the action is repeated before noting the time and how many repetitions make up each action. The subs option can be used when you want to set how many subwindows are used in the action (when appropriate).

    There are dozens of options available with x11perf that can be used to test the performance of your X server. Available options are listed on the x11perf man page. The options shown in the examples above are:

    • Send 1000 80-character lines of text to the display (-ftext).

    • Copy a 500 × 500-pixel square from one window to another (-copywinwin500).

    • Vertically scroll a 500 × 500-pixel box 800 times.

    • Destroy a set of windows ten times, first with 10 subwindows, and then with 100 (-destroy).

    • Repeatedly resize sets of 10 windows, followed by sets of 100 windows (-resize).

    Run each of those commands separately from a command line. Or, better yet, put them into a shell script and direct the output to a file. Save the output to a file.

  11. Run glxgears (as described earlier in this chapter) to test the performance of 3D OpenGL too.

  12. Try different options in your XF86Config file. Then repeat the previous few steps (restart the server, re-run the performance tools, and compare the results). Again, read through the troubleshooting tips sections to help you find ways of changing options on your X server.

  13. When you are clone, you can change your desktop environment back to GNOME or KDE. You can also return your system to runlevel 5, so you will see a graphical login. Here's how:

    # switchdesk GNOME
# init  5

Refining Video Settings with xvidtune

Using the xvidtune command, you can refine some of the settings on your video display, and then save those settings to use them permanently. Here's the procedure for using xvidtune:

  1. From a Terminal window on the display you want to adjust, type:

    # xvidtune
Vendor: Monitor Vendor, Model: LCD Panel 1024x768
Num hsync: 1, Num vsync: 1
hsync range 0: 31.50 - 48.50
vsync range 0: 40.00 - 70.00

    Information about your monitor, screen resolution, and horizontal and vertical sync ranges are displayed. Then the xvidtune window opens, as shown in Figure 7-2.

    Note

    A window will appear at this point, warning you that xvidtune can damage your monitor and/or video card. While this is unlikely, you should keep in mind that you use this utility at your own risk.

  2. Select the Auto button, so that any resizing of the screen (up, down, right, left, wider, narrower, shorter, or taller) occurs in real time.

  3. Change the display vertically by clicking Up, Down, Shorter, and Taller buttons.

  4. Change the display horizontally by clicking Left, Right, Wider, and Narrower buttons.

  5. Click the Show button. A line will appear that shows the line to put in your XF86Config file. Here's an example:

    "1024x768" 65.00 1024 1048 1184 1344 768 771 777 806 -hsync -vsync

  6. You can make the settings permanent by copying and pasting the line shown into a Modes line in your XF86Config file.

  7. Click Quit to quit the window when you are done.

Click To expand
Figure 7-2: Adjust video settings with xvidtune.


Video Troubleshooting Tips

Here are some general tips (followed by sections with sets of specific tips) to help you out when you are troubleshooting your video card:

  • How to start troubleshooting a video card-The primary way to troubleshoot your video card when it either isn't working at all or isn't working well is to try starting the server and see what messages it displays. These messages are stored in the /var/log/XFree86.0.log file.

  • Another way to begin troubleshooting X is to have X only probe your display hardware and display some messages, without actually starting the server. You can do this by typing: X-probeonly

  • Find driver-specific options-For each of the X video drivers supported in Fedora Core, you can check the driver's man page for specific options you can use with the driver. If you run the XFree86-configure command to create an initial XF86Config file, that command adds a large number of options to the configuration file for the video driver, with those options commented out. You can simply start uncommenting and adapting the options to try them out with your video driver.

    Another place you can look for driver-specific options is in the /usr/share/hwdata/Cards file. Video cards are identified in that file by name and chipset used in the card.

  • Find current state of X server-Run xdpyinfo|less to see information about the running X server. In particular, you can see the color depth, dimensions (in pixels), and dots per inch. You can also see what extensions are in use (such as DRI for hardware acceleration).

  • Use VESA when in doubt-Most video hardware for the PC available today is supported under XFree86. However, not every new card will be properly detected in Linux. For most desktop systems, if you can't determine which video driver to use, most hardware that isn't supported exactly can use the VESA driver. To use the VESA driver identify "vesa" on the Driver line of the XF86Config file or select VESA from the Display Settings Window when you configure your video card.

  • Improve performance with a low-memory video card-If you reduce the screen size and color depth you can dramatically improve performance. This is particularly true for video cards with less than 16MB of memory where you are trying to run 3D applications.

  • My video card stopped working when I upgraded XFree86-Support for some older video hardware was lost when XFree86 went from version 3.3.6 to version 4 (currently 4.3.0 in Fedora Core 1). Find out how driver support changed between 3.3.6 and 4.3.0 on the Driver Status Page (www.xffree86.org/current/Status.html).

Tips for Getting DRI to Work

Direct Rendering Infrastructure (DRI) allows you to do 3D hardware acceleration on selected video cards. To see if DRI is enabled on your display, type the following:

   # glxinfo | grep direct
   direct rendering: Yes

The Yes shows that DRI is on. If it says No instead, you should check that your video card is on the list of cards supported by DRI (visit the DRI website at dri.sourceforge.net/doc/dri_driver_features.phtml). If DRI should be supported, but appears to be disabled, make sure the following lines appear in your XF86Config file:

   Section "DRI"
                Mode 0666

One limitation of DRI is that it does not run in resolutions greater than 1024 x 768. Trying to run X in a higher resolution on a card that supports DRI, then backing down (using Ctrl+Alt+F1) may cause your X server to crash. The reason is that the higher resolution will consume lots of your video RAM, and not release it when you switch to the lower resolution. When DRI goes to get the memory that it needs, the X server crashes.

If you have trouble getting or keeping the X server running with DRI enabled, try setting a low resolution for the X server (maybe 640 × 480 with 16-bit color depth). If this works, try increasing the resolution (up to 1024 × 7668) and color depth (up to 32 bits) as long as the X server continues to run. If the server doesn't run at the lower resolution, you might have to disable DRI for your card.

Tips for Particular Video Cards (Chipsets)

The following are some tips that may help you through some problems that are specific to certain video cards and chipsets. For configuration information that applies to all video cards, see the XF86Config and XFree86 man pages.

3dfx Chipsets

Even though the maker of 3dfx products, 3dfx Interactive, Inc., is out of business, there are still a lot of video cards around that use 3dfx chipsets. Although the cards are officially unsupported, many Voodoo cards that use 3dfx chips are so inexpensive that you might want to give them a try. Here are a few tips to help you with your 3dfx cards:

  • Identify chipset-To physically identify a 3dfx chipset, try this site: www.plasma-online.de/english/identify/picture/3dfx.html.

  • Video driver-Use the tdfx video driver for Voodoo Banshee, Voodoo3, Voodoo4, and Voodoo5 cards (type man tdfx).

  • Hardware acceleration-DRI is supported on these cards to do hardware acceleration. However, you cannot use resolutions higher than 1024 × 768 at 16-bit color depth.

If you run into problems with a 3dfx card, try the voodoofiles.com website. They have a 3dfx forum where you can get help with your Voodoo 3, 4, 5, Banshee, or other 3dfx cards.

Intel i810 Chipsets

The i810 video chipsets are used on Intel 810 motherboards (i810, i810-dc100, i810e, and i815). For pixel depths up to 24 bits, hardware accelerated 2D drawing is supported. High-resolution video modes are supported up to 1600 × 1200.

The driver name is i810. Since XFree86 version 4.3.0, this graphics chipset has included 2D, 3D (using DRI), and XVideo support. Here are a few tips:

  • Video driver-Use the i810 video driver (type man i810).

  • Hardware acceleration-DRI is supported on these cards to do hardware acceleration. However, you cannot use resolutions higher than 1024 × 768 at 16-bit color depth for i810 and i815 chipsets. Later chipsets allow depths up to 24 bits.

  • Increasing video RAM-The i810 graphics chipsets use a unified memory architecture in which they can use system RAM for video memory. By default, 8MB of system RAM is assigned for graphics. You can increase this number using the VideoRam option in the Device section of XF86Config file for the i810 driver. Here's an example of what that line might look like to use 16MB of RAM for graphics:

       Section "Device"
             Identifier "Intel 810"
             Driver "i810
             VideoRam 16384
             VendorName "Intel 810"
   EndSection

NVIDIA Chipsets

Although a driver for NVIDIA cards is included with XFree86 (the nv driver), it only supports 2D acceleration. That includes support for PCI and AGP video cards that include the following chips: RIVA 128, RIVA TNT, RIVA TNT2, GeForce 256-Quadro, GeForce2-Quadro2, GeForce3Quadro DCC, nForce-nForce2, GeForce4-Quadro4, and GeForce FX-Quadro FX.

To get 3D acceleration on NVIDIA boards, you need to install NVIDIA's own proprietary binary X drivers. Go to the NVIDIA website (www.nvidia.com) and click on the Download Drivers link. From there, follow the link to Linux Drivers to download and install the NVIDIA driver.

Note

It is possible that the redhat-config-xfree86 window will not recognize the proprietary NVIDIA driver. You may need to run XFree86-configure to create your XF86Config file.

ATI Chipsets

The ATI Rage 128-based video cards use the r128 driver, while the ATI Radeon-based video cards use the radeon driver. Although your X server should autodetect this information, you can specify the following values in the Devices section for the driver:

  • VideoRam-Specify in kilobytes how much RAM is available on the card.

  • ChipID-Identify the PCI device ID (run the lspci command to get this ID).

  • IOBase-Identify the physical address of the MMIO register.

  • MemBase-Identify the physical address of the linear framebuffer.

Many options are available to use with the r128 and radeon drivers. Type man r128 or man radeon to view available options for the drivers.

Tips to Fix Video Crashes

  • Broken XF86Config file-It is possible that you, or some tool you used, may have trashed your /etc/X11/XF86Config file. If you are unable to fix the file manually so X can start, you can delete XF86Config (actually, make a copy first, and then delete it). Run redhat-config-xfree86 again to get, hopefully, a sane XF86Config file to start with. If that doesn't work, try typing this:

    # XFree86 -configure
XFree86 -xf86config /root/XF86Config.new

    If the X server starts properly using the XF86Config.new file you just created, you can copy the file to its permanent location and restart X when you are ready.

    # cp /root/XF86Config.new /etc/X11/XF86Config

  • There's nothing I can do to get this card to work!-If the card is not listed as supported, and you can't get it to go, consider buying a new card. There are good, brand-new video cards starting at about $30 that will work well, and some screaming great ones starting at around $100. Xfree86.org recommends S3-based boards and ATI-based boards. I have had good luck with NVIDIA cards, but you need their proprietary drivers to get them to work well.

    The Driver Status for XFree86 page (www.xfree86.org/current/Status.html) is a good place to start. Beware, however, that even some boards that are listed as supported may become unsupported if the manufacturer decides to change the chipset on the board without changing its name or model number.


Mouse Troubleshooting Tips

The X server in Fedora Core supports several different mouse types, depending on how the mouse is connected to the computer: serial, PS/2, bus, and USB mouse types. Two-button, three-button and wheel mouse types are supported. Selecting the appropriate mouse type/connection using the Mouse Configuration window usually does a good job configuring the mouse.

Here are some cases where you might want to go beyond what you can do through the Mouse Configuration (redhat-config-mouse) window:

  • Accessibility-There are accessibility features that let you adjust how your mouse works. To see the accessibility options available for your mouse, from the Red Hat menu click Preferences⇨Accessibility. Then click on the Enable Keyboard Accessibility Features box and choose the Mouse Keys tab. From there, you can adjust maximum pointer speed, the time to accelerate to maximum speed, and the delay between keypress and pointer movement.

  • Remapping mouse buttons-You can remap any of the mouse buttons or wheels using the xmodmap command. For example, the following command switches a mouse from a right-handed to a left-handed mouse:

    # xmodmap -e "pointer = 3 2 1 4 5"

    This command switches the functions of the first and third buttons on a three-button mouse. The 4 and 5 positions define how scrolling is done on a wheel mouse. You can switch the directions that the wheels scroll by switching the numbers 4 and 5. You can also map the actions of multiple wheels on pointing devices that have more than one wheel. (See the xmodmap man page for details.)


Video Troubleshooting Reference

Table 7-1 provides a reference for finding and troubleshooting the X Window System video to work in Linux.

Table 7-1: Video Troubleshooting References
General X Information
XFree86 project | www.xfree86.org-The XFree86 Project is the primary source for information on the open-source version of the X Window System running in Fedora Core
XFree86 documentation | www.xfree86.org74.3.0
XFree86 support | www.xfree86.org/support.html
XFree86 mailing lists | www.xfree86.org/lists.html
XFree86 driver information | Various README files in the /usr/X11R6/lib/X11/doc directory of Fedora Core
/usr/X11R6bin/XF86Config | Primary configuration file for the X (XFree86) server
Current mouse driver information | www.xfree86.org/current/mouse.html
Mouse driver man page (serial mouse) | Type man mouse
X Performance Commands
glxinfo | Displays information about OpenGL rendering in the X server. Use this to check if DRI has been enabled to do hardware acceleration
glxgears | Displays three spinning gears and shows how many frames per second your X server can display the gears spinning
xvidtune | Tunes the height and width of your display, and then saves those settings to your XF96Config file
x11perf | Runs performance tests (based on parameters you choose), and then reports how quickly the X server can complete those tests


Summary

Most modern video cards that come with personal computers today will work with the X window system (XFree86) that comes with Fedora Core. In those cases where Fedora can't properly probe and configure a video card automatically, you can use the tips in this chapter to track down and fix your video card.

This chapter described ways of configuring, troubleshooting, and tuning performance for video cards in Fedora. It also provided pointers to many other resources that are available to help you if you run into trouble with your video card.


Chapter 8: Software Package Troubleshooting

RPM Package Management (RPM) provides the framework in which software is installed, updated, and uninstalled in Fedora Core and Red Hat Enterprise Linux systems. Many other Linux distributions are based on the RPM system, including SUSE, Mandrake, and others. RPM command line and GUI tools can provide the means to query, verify, and otherwise manage the software on your computer.

This chapter describes the basic tools you need to manage software packages in RPM format in Fedora and other Red Hat Linux systems. It also helps you verify and query RPM packages before you install them and troubleshoot problems you may encounter, such as resolving package dependencies and fixing broken RPM databases.

Introduction to RPM

For those who need to set up and maintain Linux systems, software management is a critical issue. You need to strike a balance between having the most up-to-date versions of the software you want and the stability that you need for the systems you maintain. You also need to make sure that your software is both compatible (doesn't break other software) and secure (hasn't been compromised).

By using RPM tools that come with all Red Hat software distributions, you can learn skills for managing and securing your software packages. The primary tools for managing RPM packages in Fedora Core and Red Hat Enterprise Linux include the following:

  • rpm command-This offers options for querying, verifying, installing, upgrading, and removing packages. There are also various options for working with the RPM database.

  • Package Management window-The Package Management window provides a graphical means of viewing, adding, and removing packages that are part of the Linux distribution. Packages are grouped together and dependencies are taken into account when you try to add or delete packages. To get to the Package Management window, from the Red Hat menu, click System Settings⇨Add/Remove Applications.

As a bonus, the skills you'll learn while working with these tools will apply to many different RPM-based Linux distributions.

In this chapter, I focus primarily on using the rpm command to manage and query the software packages on your Linux systems. Because rpm is such a useful tool, its uses are also described in a few other places in this book:

  • Installing, updating, and upgrading-Chapters 1 and 2 cover the basics of installing Linux, then doing subsequent software updates and upgrades. Whether these packages are grabbed from an installation CD or from yum or apt software repositories, for the most part you should end up with all or most of your software installed from RPM packages. Some useful rpm examples are included in these chapters.

  • Detecting intrusions-Chapter 10 describes how to use RPM to see if someone has tampered with the software on your Linux system, based on the contents of your RPM database.

Assuming that you have already installed your basic Linux operating system, this chapter will give you examples of lots of different ways you can manipulate those packages and the RPM database that stores information about them.


Installing, Upgrading, and Deleting Packages

The basic ways of using the rpm command to install, upgrade, and delete packages are covered in many introductory books on Fedora and Red Hat Linux (such as the Red Hat Linux Bible). By way of review, and for troubleshooters who are new to Linux, here are a few basic ways to use the rpm command.

Note

If the package you want is in a yum or apt repository, I recommend you use the appropriate yum or apt tools to install the package. See Chapter 3 for descriptions of yum and apt.

The most common way to install a new package or upgrade an existing package is by using the -U option of rpm. With the file you want to install in the current directory, type the following:

  # rpm -Uhv packagename-1.1.1.i386.rpm

If the package (represented by packagename-1.1.1.i386.rpm) is not yet installed, this command will install it. If it is a later version of an already installed package, the command will also install it (actually, upgrade it). The -h option prints hash marks (for positive feedback as the package installs) and -v gives verbose output (add multiple v's to see even more verbose output, such as -Uhvvv).

Note

Instead of -U you can use the -i option to install a package. If it is a package that is not yet installed on your computer, it should install just fine. If the package (or an earlier version) is already installed, the install will fail and you will have to use -U to install it. For installing kernel packages, always use the -i option.

A great option when you are upgrading a whole lot of packages on an existing system is the freshen (-f) option. Let's say that you download a directory of packages with security updates, but you only want to install new versions of packages that are already installed. With that directory as the current directory, you could type the following:

   # rpm -Fvh *.rpm

Packages that are already installed on your computer, but for which there are later versions in the current directory are installed. All other packages in the directory are not installed. (Keep in mind that packages not installed are skipped silently.)

To delete an installed package, use the -e (erase) option to rpm. For example,

  # rpm -e Wnn6-SDK-devel

Again, the deletion of the package is done silently. So if you want positive feedback, add one or more -v options (for example, rpm -evvv).

There are a lot of options to the rpm command that you can use during basic package installs, upgrades, freshens, and deletions. Again, I'd recommend the Red Hat Linux Bible or the rpm man page for details about rpm options. Here are a few options you can use with basic install, upgrade, freshen, and delete options:

  • Saving space-You can save some space by excluding certain files from being installed from a package. In particular, --excludedocs excludes any files marked as documentation from being installed and --excludepath PATH excludes any files in the specified path from being installed.

  • Verbose-Add a single -v option to make the output more verbose. Add multiple -v options (-vv, -vvv, and so on) to see lots of debugging output.

  • install it anyway-There are several reasons that a package installation may fail, almost all of which can be overridden. On a production machine, you should probably fix the problem before continuing (as in, fix the dependency).

  • However, on a noncritical computer where you just want to check out a package, you can skip dependency checking (--nodeps), not verify any digests (-nodigests), and not verify signatures (--nosignature). You can even skip checking for sufficient disk space (--ignoresize) and install despite having binaries intended for a different computer architecture (--ignorearch).

    Caution

    Dependency, digest, signature, and architecture checks are all put in to make sure that you are installing software that is safe and correct for your environment. Overriding these safeguards can be dangerous. Do so at your own risk.

  • Watch progress-There are several options for watching the progress of a package installation. You can print 50 hash marks (#) as rpm unpacks a package archive using the -h option. Likewise, you can tell rpm to print the percentage at which the file was unpacked with the -percent option.

  • Test it-Before you go ahead and do a real install or uninstall of a package, you can test it using the --test option. I recommend you add one or more -v options to get as much information as possible before running the install or uninstall for real.

Now that you have had a quick review of the basics, the rest of this chapter is devoted to learning how to prevent and track down problems with software packages.


Checking Software Packages

Before you install RPM packages, you may want to run some checks to answer a few questions about those packages. For security purposes, you can make sure that the package was properly signed. You can also find out what dependencies the package has and information about what the package contains. The following sections explore these topics.

Checking Software Package Signatures

Whether you do it to a pile of RPM packages on your hard disk or to packages downloaded from an apt or yum repository, checking the signature of the packages you plan to install on your Linux system is a good idea. Using the rpm command, you can import public keys that are then used to verify the contents of your RPM packages.

The Fedora Core distribution comes with several GNU Privacy Guard (GPG) keys that it uses to verify the packages you install that come directly from the Fedora project. Many software repositories that let you download RPM packages outside of any official Red Hat or Fedora distribution will have their own GPG keys, which you can use to verify that their packages have not been tampered with.

Cross-Reference

For more information on the GNU Privacy Guard project, visit the project's home page at www.gnupg.org.

Checking Fedora/Red Hat Public Keys

If you install Fedora Core from the three install CDs, you can find the PGP public key signatures from Red Hat, Inc., in the top-level directory of each CD. After the install, the keys are stored in the /usr/share/doc/fedora-release-1 directory (for Fedora Core systems) or the /usr/share/doc/redhat-release-* (for Red Hat Linux systems). Files containing the two keys that are installed and used automatically on Fedora Core systems include the following:

  • RPM-GPG-KEY-This file contains the PGP public key used by Red Hat, Inc., to build its own software packages.

  • RPM-GPG-KEY-fedora-This file contains the PGP public key used to sign packages that are specific to the Fedora Project.

The two keys stored in these files are installed in the RPM database during Fedora Core installation so they can be used to check that the packages have been properly signed.

Viewing Installed GPG Keys

GPG keys are stored in your RPM database and can be manipulated using some of the same options you would use to work with regular RPM packages. To see the keys installed in your computer, type the following:

   # rpm -qa gpg-pubkey*
   gpg-pubkey-4F2a6fd2-3f9d9d3b
   gpg-pubkey- db42a60e-37ea5438

The gpg-pubkey-db42a60e-37ea5438 represents the Red Hat, Inc., public key used to sign its packages. The gpg-pubkey-4f2a6fd2-3f9d9d3b represents the Fedora Core public key. To get information about any public key, you can use the rpm command as you would from any RPM package. For example, to see who owns the first public key type

# rpm -qi gpg-pubkey-4f2a6fd2-3f9d9d3b | less
Name        : gpg-pubkey       Relocations: (not relocateable)
Version     : 4f2a6fd2              Vendor: (none)
Release     : 3f9d9d3b          Build Date: Thu 05 Feb 2004 09:44:00 PM PST
Install Date: Thu 05 Feb 2004 09:44:00 PM PST Build Host: localhost
Group       : Public Keys       Source RPM: (none)
Size        : 0                    License: pubkey
Signature   : (none)
Summary     : gpg(Fedora Project <fedora@redhat.com>)
Description:
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: rpm-4.2.1 (beecrypt-3.0.0)

mQGiBD+dnTsRBACwnlz4Ahct0LlVBAsq+RaU82nb5P3bD1YJJpsAce1Ckd2sBU0JD11NUCqH
.
.
.
-----END PGP PUBLIC KEY BLOCK-----

From this output, you can see how the key's name is divided into version and release numbers. In the Summary line, the key is identified as gpg for the Fedora Project. I pipe the output through less here (| less) so that you can page through the text. Press the space bar to page through and type q to quit at the end. I pipe output to less in several examples throughout the chapter.

Importing GPG Keys

If, for some reason, either of the keys (Fedora or Red Hat) is not installed, you can import each to your RPM database as follows:

   # cd /usr/share/doc/fedora-release-*
   # rpm --import RPM-GPG-KEY-fedora
   # rpm --import RPM-GPG-KEY

When you install new or updated RPM packages from Fedora Project or Red Hat, Inc., the keys are automatically checked and you are notified if they don't match any package you try to install. If you ever want to remove a GPG key, you can do so using the -e option of rpm. This is the same command you would use to remove any regular RPM package as well. For example, the following would remove the Fedora public key:

   # rpm -e gpg-pubkey-4f2a6fd2-3f9d9d3b

Checking Third-Party Public Keys

By adding packages to your Fedora or Red Hat Enterprise Linux systems that are not released as part of those distributions, you are assuming more risk for the quality and security of those packages. Getting and using GPG keys to check the signatures on packages you get from third-party software repositories can verify that they haven't been altered since they were created and help insure that the packages come from an identifiable place.

As described in Chapter 3, tools such as yum and apt can be used to download and install RPM packages from the Internet. Getting GPG keys from a download site before you use the software it offers is one way to help check that the software hasn't been compromised. The following procedure describes how to get and import GPG keys to check software you get from the freshrpms.net software repository (a well-known software repository for Fedora and Red Hat Linux software):

  1. Get the freshrpms.net public key by typing the following (if the key file has moved, you might need to go to freshrpms.net and poke around for it):

    #wget http://freshrpms.net/packages/RPM-GPG-KEY.txt

  2. With the RPM-GPG-KEY.txt file in your current directory, type the following to import the freshrpms.net public key into your RPM database:

    #rpm --import RPM-GPG-KEY.txt

  3. Check information for the public key you imported (the next command reflects the public key name currently being used with freshrpms packages).

    #rpm -qi gpg-pubkey-e42d547b |less

    The output should list Matthias Saou in the Summary.

At this point, you should be able to download packages from the freshrpms.net site (either from your web browser or using a download/install tool such as yum) without getting complaints about not having a key to validate them. You can repeat this procedure for any software repository you want to use.

Troubleshooting Software Package Signatures

To check a package before you install it, you can use the checksig option to rpm. Because an RPM package can contain multiple digests and signatures, the checksig option will check each digest and signature to make sure it is correct.

For example, let's say that you copied the fedora-logos RPM package into the current directory. You could check the signature on that package as follows:

   # rpm --checksig fedora-logos-1.1.20-noarch.rpm
   fedora-logos-1.1.20-noarch.rpm: (sha1) dsa sha1 md5 gpg 0K

This output shows that the package uses the Secure Hash Algorithm, version 1 (sha1) header digest, which acts as a fingerprint of the contents of the RPM. It uses a DSA header signature and includes md5 digest and GPG signature. You can tell that this package is all right (was properly created with the given key) by noting the following:

  • Each digest and signature is shown in lower case.

  • There is the word OK at the end (as opposed to NOT OK).

If you want to see more information about the signature checking, you can add a -v or -vv option. Here is an example:

# rpm --checksig -vv fedora-logos-1.1.20-1.noarch.rpm
D: Expected size:       263841 = lead(96)+sigs(344)+pad(0)+data(263401)
D:   Actual size:       263841
D: opening db environment /var/lib/rpm/Packages joinenv
D: read h#    1624 Header sanity check: OK
D: ========== DSA pubkey id b44269d04f2a6fd2
fedora-logos-1.1.20-1.noarch.rpm:
    Header V3 DSA signature: 0K, key ID 4f2a6fd2
    Header SHA1 digest: OK (b712be20fa0b6d3bf89f6a3d0c91f86efce24f0e)
    MD5 digest: 0K (94d9d20ce0265cb1f36d5fb8fa0dfc30)
    V3 DSA signature: 0K, key ID 4f2a6fd2

The output shows that the number of characters expected in the file (263,841) was actually found. You can see that the command connected to the RPM database environment (/var/lib/rpm/Packages joinenv). The header passed a sanity check (OK), and the DSA public key with id number b44269d04f2a6fd2 was found. After this each signature and digest was checked (DSA signature, SHA1 digest, MD5 digest, and DSA signature) and found to be correct (OK).

If the package checks out, and you trust the keys you are using, you can probably go ahead and safely install the package.

The following examples illustrate some cases where the package did not check out:

  • Missing key-If for some reason, the public key associated with the signed package was not installed in your database, you might see the following error message:

    fedora-logos-1.1.20-1.noarch.rpm: (SHAD DSA sha1 md5
     (GPG) NOT OK (MISSING KEYS: GPG#4f2a6fd2)

    In this case, the key 4f2a6fd2 (for Fedora Project packages) is missing. Note that a missing key causes the SHA1, DSA, and GPG notations to display in upper case (indicating a failure). However, the fact that sh1 and md5 are in lower case indicates that the package itself is not corrupted, because the SH1 and MD5 digests were consistent with the contents of the package. Chances are if you install the proper key, the package will pass the check.

  • Corrupted RPM-If the contents of an RPM file have been badly corrupted, all digest and signature indications might appear in upper case or the package might not even be recognized as an RPM file. The following messages reflect these two results:

    fedora-logos-1.1.20-1.noarch.rpm:(SHA1) DSA SHA1 MD5 GPG NOT OK
error: fedora-logos-1.1.20-1.noarch.rpm: not an rpm package

    You won't be able to install this package even if you wanted to. Get a new copy of the package.

  • Modified RPM content-It is possible that the content of an RPM file has been changed, but the headers appear to be valid. In that case, even though the dsa and sh1 appear to be all right, the MD5 and GPG will appear as BAD. Here's what verbose output looks like (rpm --checksig -vv) in that case:

    fedora-logos-1.1.20-1.noarch.rpm:
    Header V3 DSA signature: 0K, key ID 4f2a6fd2
    Header SHA1 digest: 0K (b712be20fa0b6d3bf89f6a3d0c9lf86efce24f0e)
    MD5 digest: BAD Expected(94d9d20ce0265cblf36d5fb8fa0dfc30) !=
(835891bb30ca96d73dc4bce78ddb36d7)
V3 DSA signature: BAD, key ID 4f2a6fd2

    In this case, you should not use the RPM since it was probably tampered with.

Checking Software Package Dependencies

Most RPM packages include built-in information about other components (such as libraries) that they require to work. If you try to install or upgrade a package, and components the package depends on are not already installed, package installation fails (by default). Likewise, if you try to remove a package on which other installed packages depend, it will also fail.

If you install a package from Fedora or Red Hat Linux installation CDs (or if you are installing from a well-maintained yum or apt repository), dependency checks are done automatically when you try to add or remove packages. When you install using yum or apt programs, they ask you if you want to install any dependent package(s) as well. If you say yes, yum or apt will go ahead and install the dependent package(s). Problem solved.

Note

Installing RPMs from a software repository using yum or apt is always preferred over installing individual packages from an FTP or HTTP server. Yum and apt can insure dependencies are met in a way that can't be done by downloading packages individually.

This is a troubleshooting book; however, you might be interested in dealing with those times when there is not a package already installed to fulfill the requirements. In that case you can do the following:

  • Skip dependencies-The rpm command allows you to skip dependency checking using the --nodeps option. While this will get the package installed, it will most likely leave you with a package that doesn't work.

  • Look for the needed RPM package-You might need to do some digging around to find the package that contains the components you need. Then you can install the package that contains the required component so the package you want can be installed.

    To find out what dependencies are not being met, you can just try to install or upgrade the package you want. The missing dependent components will be listed by default and the install will fail. Alternatively, you can ask to explicitly list the dependencies of the package. Once you know what is missing, you can do the following:

    • Search installed RPM packages for the component.

    • Search the full database of RPM packages that are on the full CD set representing Fedora Core, Red Hat Linux 9, or other operating system. (See the Using the RPM Universal Database sidebar for more information.)

    • Check the FTP or HTTP directory where you got the initial package, search the rpmfind.net site for the package you need, or just search the Internet.

Start Sidebar
Using the RPM Universal Database

Each Red Hat Linux distribution (Fedora, Red Hat Linux 9, and Red Hat Enterprise Linux) includes a package containing a database of information that describes all the RPMs in that distribution. In Red Hat Linux install the rpmdb-redhat package; in Fedora install rpmdb-fedora package.

The entire database of RPM information is in the /usr/lib/rpmdb/i386-redhat-linux/redhat directory. You can query that database for dependencies using the rpm --redhat provides and rpm --redhatrequires command. When searching for how to resolve dependencies for a package you want to install, this "universal" database is a good place to look.

End Sidebar

The following procedure takes you through the process of checking a package for dependencies, finding the package needed to fulfill the dependencies, and then getting and installing that package.

  1. Get an RPM package you want to install. For this example, I downloaded the xmame-0.78.1-2.fr.i386 package for Fedora Core 1:

    ftp.freshrpms.net/pub/freshrpms/ayo/fedora/linux/l/i386/RPMS.freshrpms/

  2. Run a test install on the package to see the dependency checking (among other things):

    # rpm --test -Uhvv xmame-0.78.1-2.fr.i386
D: ============== xmame-0.78.1-2.fr.i386.rpm
D: Requires: /bin/sh                                 YES (db files)
D: Requires: /usr/bin/tclsh                          YES (db files)
D: Requires: libc.so.6                               YES (db provides)
D: Requires: libc.so.6(GLIBC_2.0)                    YES (db provides)
D: Requires: libc.so.6(GLIBC_2.1)                    YES (db provides)
D: Requires; libc.so.6(GLIBC_2.3)                    YES (db provides)
D: Requires: libz.so.1                               YES (db provides)
D: Requires: rpmlib(CompressedFileNames) <= 3.0.4-1  YES (rpmlib provides)
D: Requires: rpmlib(PayloadFilesHavePrefix) <= 4.0-1 YES (rpmlib provides)
D: Requires: xmame-bin = 0.78.1                      NO
D: package xmame-0.78.1-2.fr has unsatisfied Requires: xmame-bin = 0.78.1
error: Failed dependencies:
     xmame-bin = 0.78.1. is needed by xmame-0.78.1-2.fr

    As you can see from the output (which I edited down to include just required items), most of the required components needed by the xmame package are already provided by the packages that are already installed. The sh and tclsh files are there, as are several libc libraries and rpmlib features. What is missing is xmame-bin.

  3. You can see if any installed package has the component you need using the query option -whatprovides. (Of course, we already know the component is not in an installed package, since the dependency failed. But we'll just go through the motions anyway.)

    #rpm -q --whatprovides xmame-bin
no package provides xmame-bin

  4. Next, you can search the universal RPM database, which contains all the RPM packages in the entire Fedora Core or Red Hat Enterprise Linux CD set. This will tell us if the component is in any package from your CD set, whether or not the package is installed.

    #rpm --redhatprovides xmame-bin
no package provides xmame-bin

  5. Next, I suggest opening the FTP site where you got the package in an FTP client and seeing if the package needed is there. In my case, I went to the freshrpms.net directory that contained the xmamb package and downloaded the xmame-x11 package to the current directory. I then checked to see what the package provided:

    #rpm -q --provides -p xmame-x11-078.1-2.fr.i386.rpm
xmame-bin = 0.78.1
xmame-SDL = 0.78.1-2.fr

    This package seems to provide the xmame-bin component we need. So we can go ahead and try installing it. If this installs without a problem, we can install our original xmame package. Although in this case it worked, if you could not find a package to meet the dependency in the FTP directory of the original package, continue with the rest of this procedure.

    Note

    Instead of using the name of the package in the current directory to check for the needed component, you could have provided the full URL to the package in the FTP or HTTP directory. It makes you type a long URL, but adds an extra download step.

  6. If the package is still not found, try searching the rpmfind.net site. Go to www.rpmfind.net and search for the needed component. All RPM files that are found that include your search term are displayed. Click on one that is listed for Red Hat Linux 9 or other appropriate distribution.

  7. If you still have not found the component, as a last resort go to www.google.com or other search engine and simply search the Internet for the component.

There may be some cases where you can't match the dependency exactly. For example, the package you want to install may be looking for an exact version of the component and you might only have access to a later version. The later version may well work. In that case, you might want to try the --nodeps option to force the package to install. Here are some of the types of dependent components you may run into:

  • Package-An exact package may be required. In that case, the dependency may specify only the package name or package name and version number. The requirement may also be that the package be less than, greater than, or equal to a certain version.

  • Component-A particular library or binary file may be required. Where that component comes from may not matter, as long as it is provided by some RPM package. In this example, an exact version of xmame-bin was required (xmame-bin = 0.78.1). This represents an xmame binary that can be available from several different RPM files. As long as one is installed, the dependency check will pass.

Cross-Checking Software Package Contents

After your software packages are installed, the RPM database remains an excellent resource for cross-checking original information about a package against the actual installed components on your hard disk. In other words, you can tell if someone has been fooling around with your software.

Chapter 10 is a more complete reference to techniques for verifying if any of your software has been tampered with, including ways of using the rpm command to verify that someone hasn't planted a trojan horse in an executable. A simple way of checking a whole RPM package for files that have been tampered with is the rpm -V command. For example, to check the entire samba package, you could type

  # rpm -V samba

Any files in the samba package that have been changed since they were installed will appear after the command. The next procedure can be used for checking whether a single file has been tampered with.

  1. For the file or command you want to check, find out what package it is from. In this example, I'm checking the startup script for the Samba service:

    #rpm -qf /etc/rc.d/init.d/smb
samba-3.0.0-15

  2. List the files (-ql) contained in the samba software package (from the RPM database of installed software) and dump (--clump) data out about each file. Then sort out (grep) everything except the smb script.

    #rpm -ql --dump samba | grep /etc/rc.d/init.d/smb
/etc/rc.d/init.d/smb 1928 1064589271
99d3f786f9dc199fe0221cce496c5795 0100755 root root 1 0 0 X

    Interesting information associated with the dump includes the following:

    • The original number of characters in the file (1928).

    • The md5sum digest that reflects the contents of the smb file (99d3f786fgdc199fe0221cce496c5795).

    • Permission associated with the file (0755).

    • Ownership of the file (root owner and root group).

  3. Next, check the actual file to see if the contents have been changed. Type the following:

    #Is -1 /etc/rc.d/init.d/smb
-rwxr-xr-x 1 root root 1928 Sep 26 08:14 /etc/rc.d/init.d/smb
#md5sum /etc/rc.d/init.d/smb
99d3f786f9dcl99fe0221cce496c5795 /etc/rc.d/init.d/smb

The long list of the file (ls -l) shows that there are 1928 characters in the file, the owner and group are both root, and file permissions are 0775 (-rwx r - x r - x). All of that matches the dump output shown earlier. Likewise, the md5sum output (99d3f . . . c5795) also matches the dump output for the file.

In this case, information about the installed smb script seemed to match the information about the original script stored in the RPM database. If you believe that the RPM database might have been tampered with as well, you could check against the actual samba RPM file from the CD or other trusted source. For example, with the samba RPM in the current directory, I could type

   # rpm -ql --dump -p samba-3.0.0-15.i386.rpm | grep /etc/rc.d/init.d/smb

Hopefully, you would get the same results as you got from the similar command line you ran against the RPM database.

Cross-Reference

For more information on using the rpm command to verify the contents of a command, library or configuration file, refer to Chapter 10.

Checking Software Package Scripts

Many RPM packages include scripts that run when the package is installed or uninstalled. In particular, when you are upgrading from one version of a package to another and something messes up, you might want to see what those scripts did to your system.

The rpm command has a means of listing the contents of

  • scripts in an RPM package that run during the install or uninstall process and

  • trigger scripts that are kicked off when certain conditions are met.

The following sections describe how to list scripts and trigger scripts in an RPM file.

Check for Scripts

Use the --scripts query option to the rpm command to see scripts that are run before and after installs and uninstalls of a package are done. Here is an example:

   # rpm -q --scripts samba
   postinstall scriptlet (using /bin/sh):
   /sbin/chkconfig --add smb
   preuninstall scriptlet (using /bin/sh):
   if [ $1=0 ]; then
       /sbin/chkconfig --del smb
       rm -rl/var/log/samba/* /var/cache/samba/*
       /sbin/service smb stop >/dev/null 2>&1
   fi
   exit 0
   postuninstall scriptlet (using /bin/sh):
   if [ "$1" -ge "1" ]; then
     /etc/rc.d/init.d/smb condrestart >/dev/null 2>&1
   fi

This example shows the scripts that are included in the samba package. After the samba package is installed (postinstall), the chkconfig command is called to turn on the samba service (- -add smb). Before the package is uninstalled (preuninstall), the samba service is removed from starting again (chkconfig - -del smb) and all log files (/var/log/samba/*) and cache files (/var/cache/samba/*) are removed.

After the package is uninstalled (postuninstall), if the package still exists on the system, the service is restarted. (This is done in cases where you are removing an old version of the package and the new version has been installed and needs to be restarted.)

Scripts that have some level of complexity include those for packages that use the alternatives system and therefore may or may not be activated by default (including sendmail, postfix, and cups). Special scripts that are only used in particular conditions are referred to as trigger scripts.

Check for Trigger Scripts

Some packages have trigger scripts that are kicked off during installation if certain conditions are met. These are particularly useful in upgrades where the software packager wants different installation actions to take place, depending on the current package release you are upgrading from. Here's an example of how to display the trigger scripts included in the sendmail package:

   # rpm -q --triggers sendmail
   triggerpostun scriptlet (using /bin/sh) -- sendmail<8.10.0
   /sbin/chkconfig --add sendmail
   triggerpostun scriptlet (using /bin/sh) -- sendmail<8.11.6-11
   /usr/sbin/alternatives - auto mta

Here you can see that upgrades from pre-8.10.0 versions of sendmail add the sendmail services to be turned on when the computer boots (chkconfig--add sendmail). Pre-8.11.6-11 sendmail versions have the alternatives command set the mail transfer agent (mta) to auto state (which makes the highest priority mail services used as the mta). By default, sendmail is chosen over postfix.

There have been some reports of trigger scripts breaking services. So if a service that was recently upgraded, such as sendmail or initscripts, is broken, you can check for triggers to see if there are trigger scripts that could have caused a problem.


Querying Software Packages

With the rpm query feature, you can display many details about the form and content of the RPM packages in your Linux system. You can then output that information in any way that is useful to you. Querying can be done on individual packages or over the entire RPM database.

Querying Local or Remote Packages

For each of the following query options, if you give the RPM package's base name, rpm will query the database for information about that installed package. If you want to query a package that is not installed, you must use the -p option followed by the package name (in the current directory), the full path to the package name, or a URL to the package. For example, to see the description for a mozilla package in my current directory, I could type

   # rpm -qi -p mozilla-1.4.1-17.i386.rpm

A package may also be contained on a website (HTTP) or FTP server. Here is an example of the rpm command being used to list the contents of an RPM on a remote FTP site. The following example displays all files contained in the apt package from the ftp.freshrpms.net shown (the entire command should be on one line):

   # rpm -ql -p ftp://ftp.freshrpms.net/pub/freshrpms/fedora/linux/1/apt/apt-0.5.lcnc3-
    0.1.fr.i386.rpm

The -p option provides a good way to check out packages on remote systems before you install them. If a username and password are required, they can be added to the command line. In this example, the user is tina and the password is my44hat as follows:

   # rpm -ql -p ftp://tina:my44hat@ftp.example.net/pub/any.rpm

Querying Individual Packages

You can find out a lot of information for any rpm package installed on your system. Here are a few examples:

  • All files in a package-Use the -ql option to display all the files in a package. In the following example, we query a package that is installed on the computer:

    #rpm -ql samba | less
/etc/logrotate.d/samba
/etc/pam.d/samba
/etc/rc.d/init.d/smb
/etc/samba/smbusers

    Here you can page through all the files contained in the samba package.

  • Documentation in a package-Instead of listing all the files (-ql), you can list only documentation files contained in a package using the -qd option. For example,

    #rpm -qd bind-utils
/usr/share/man/man1/dig.1.gz
/usr/share/man/manl/host.l.gz
/usr/share/man/man5/resolver.5.gz
/usr/share/man/man8/nslookup.8.gz
/usr/share/man/man8/nsupdate.8.gz

    This shows that the bind-utils package contains documentation in the form of five man pages (for example, type man dig to see the first one). Besides man pages, packages often include documentation that comes in Texinfo format (type info) and files in /usr/share/doc for the package.

  • Configuration files in a package-You can list configuration files associated with a package with the -qc option. Most packages will keep their configuration files in /etc (or a subdirectory of /etc).

    #rpm -qc samba
/etc/logrotate.d/samba
/etc/pam.d/samba
/etc/rc.d/init.d/smb
/etc/samba/smbusers
/etc/sysconfig/samba

    Sometimes the -qc option will turn up files in locations outside of /etc that you might want to reconfigure. For example, the httpd package lists files in the /var/www/error directory you could change if you want different information displayed by your Apache web server for different error conditions.

  • Changelog for a package-To see changelog entries made to an RPM package, you can use the --changelog query option. If the software developer has been verbose about his updates, this information can help you understand what has been done to the package lately. For example,

    # rpm -q --changelog XFree86 | less
* Mon Oct 20 2003 Mike A. Harris <mharris@redhat.com> 4.3.0-42

- This release is the long awaited answer to the meaning of life, the
  universe and everything.
- Added XFree86-4.3.0-redhat-exec -shield-GNU-stack.patch to make the
  complete Xlree86 build including Mesa et al. exec-shield friendly
  (arjanv, mharris)
- Updated to new XFree86-4.3.0-Mesa-SSE-fixes from-MesaCVS-v2.patch which
  should fix compatibility problems between DRI and 2.6.x kernels which
:

  • Description of a package-A formatted set of information about a package can be displayed using the -qi option. Here's an example:

    # rpm -qi XFree86
Name        : XFree86          Relocations: (not relocateable)
Version     : 4.3.0                 Vendor: Red Hat, Inc.
Release     : 42                Build Date: Fri 24 Oct 2003 03:19:36 PM PDT
Install Date: Tue 11 Nov 2003 04:26:51 PM PST Build Host: porky.redhat.com
Group       : User Interface/X     Source RPM: XFree86-4.3.0-42.src.rpm
Size        : 64230/11                         License: XFree86
Signature   : DSA/SHA1, Tue 28 Oct 2003 04:20:58 PM PST, Key ID
              b44269d04r2a6fd2
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
URL         : http://www.xfree86.org
Summary     : The basic fonts, programs and does for an X workstation. Description:
XFree86 is an open source implementation of the X Window System. It
provides the basic low level functionality which full fledged
graphical user interfaces (GUIs) such as GNOME and KDE are designed upon.

  • Name of a package-Say that you are looking for a package that has the characters "bind" in it, but you can't remember the package's exact name. The -qa option lets you list all packages, from which you can then search for the string of characters that interests you. Here's an example:

    # rpm -qa *bind*
bind-utils-9.2.2.P3-9
kdebindings-devel-3.1.4-1
redhat-config-bind-2.0.0-18
bind-chroot-9.2.2.P3-9
bind-devel-9.2.2.P3-9
kdebindings-3.1.4-1
ypbind-1.12-3
bind-9.2.2.P3-9

Querying Multiple Packages

The content of an RPM package consists of a header, signatures, and the actual files that you are installing from the RPM. Each piece of information in the header is marked with a tag. When you query an RPM package, you are inquiring about the content of one or more of these tags.

The rpm command lets you query the RPM database for tags that you request and then displays the resulting information in a format that you choose. This section provides some useful examples of the rpm query feature for finding and displaying information for multiple RPM files.

You can start out by listing all the query tags that may be available in RPMs on your computer. You do that as follows:

   # rpm --querytags | less

This example lets you page through the list of tags. 1 counted 139 tags available on my Fedora Core 1 system. Any of the tags that interest you can be put in an rpm query command so that information from those tags can be displayed in the format you choose. Here are some examples:

  • To display the name and summary information for every package beginning with the string XFree86, you could type the following:

    # rpm -qp --queryformat "%{NAME} *** %{SUMMARY) \ n" XFree86*
XFree86-base-fonts *** The collection of XFree86 base fonts
XFree86-tools *** Various tools for XFree86
XFree86-IS08859-14-100dpi-fonts *** IS08859-14-100dpi-fonts
XFree86-font-utils *** Utilities required for installing fonts

  • To list the name of every package along with the web address (URL) to the package's website, you could type the following rpm command. This command is piped to the less command, which lets you page through the output.

    # rpm -qp --queryformat "%{NAME} >>> %{URL} \ n" | less
libattr >>> http://acl.bestbits.at/
mktemp >>> http://www.mktemp.org
less >>> http://www.greenwoodsoftware.com/less/
shadow-utils >>> http://shadow.pld.org.pl/
pam >>> http://www.us.kernel.org/pub/linux/libs/pam/index.html
rmt >>> http://dump.sourceforge.net
zip >>> http://www.info-zip.org/pub/infozip/Zip.html
logwatch >>> http://www.logwatch.org/
bind-utils >>> http://www.isc.org/products/BIND/
libxml2-python >>> http://xmlsoft.org/

  • In the following example, the installed RPM database is queried, then the output is in the form of P: followed by the package name, then S: followed by the summary line for the package. Finally, the output is piped to the grep command, which displays only those lines that contain the term image.

    # rpm -qp --queryformat "P: %{NAME} S: %{SUMMARY} \ n" | grep -i image
P: desktop-backgrounds-extra S: Desktop background images.
P: SDL_image-devel S: Development files for the SDL image loading library.
P: gimp S: the GNU Image Manipulation Program.


Verifying Software Packages

Because your RPM database knows all about the packages and files you have installed on your system, you can use that information to compare the original executable commands, configuration files, and other package components against those items as they exist on your hard disk. To do that, you can use the -V option to RPM.

The basic format of the rpm command with the verify option is

   # rpm -V package-name

Replace package.rpm with the package you want to verify. The output from that command will show you any files on your hard disk that are different in any way from how they were originally installed. Here is an example of running the rpm-V command on the initscripts package:

   # rpm -V initscripts
   prelink: /bin/ipcalc: prelinked file size differs
   S.?....T   /bin/ipcalc
   S.5....T c /etc/inittab
   S.5....T c /etc/rc.d/init.d/functions

The component (file or command) that was shown not to match the original is shown on the right. On the left are indications of what is wrong. The first message, prelink: /bin/ipcalc: prelinked file size differs indicates that the ipcalc command is somehow corrupted. The ? indicates that it is too damaged to be trusted.

The two other files shown (inittab and functions) illustrate two configuration files that have been modified (in my case, they were properly modified). Each of the lines starts with eight characters (either dots or some alphanumeric character). When a dot appears in the location, it means the item it represents passes the test. A letter or a question mark indicates that the test doesn't pass. Table 8-1 illustrates what each of the eight slots to start each line represent.

Table 8-1: Output from RPM Verify
S | The original file size differs from the current file size (indicating that someone has changed the content). This might reflect normal changes to configuration files.
M | The permission mode is different. Included in the permission mode are the read/write/execute values, as well as the file type.
5 | The md5sum value, generated from the original contents of the file, is different than it was for the original. This indicates that someone has changed the content of the file.
D | For a device file, the major and/or minor number are mismatched.
L | The readLink path is mismatched.
U | The user ownership is different than it was on the original.
G | The group ownership is different than it was on the original.
T | The time stamp on the file is different than it was on the original.

From the output shown in Table 8-1, you can see for inittab and functions that the file size differs (S), the MD5 sum differs (5), and the time stamp on the file (T) differs as well. For configuration files, changes to the file size, MD5 sum, and time stamps are not unusual. What is unusual is if an executable file has been changed, because that could indicate that someone is trying to replace a real program with one that might be doing some mischief.

To see more details with the output shown, you can add -v or -vv options. This will show each item being checked.

Cross-Reference

For more information on using rpm with the -V option, refer to Chapter 10. Chapter 10 describes how the verify option of rpm can be used as part of a larger scheme for protecting the security of your Linux system.


Repairing a Broken RPM Database

On occasions, the RPM database can break. The result is that when you try to install a package, remove a package, or query the database, your request might just hang.

The problem of corrupted RPM databases seemed to be much worse in Red Hat Linux 8 than it is in Fedora Core 1 or Red Hat Enterprise Linux. However, if your RPM database is showing signs of hanging processes that talk to it, here are a few steps you can go through that should help correct your broken RPM database:

  • Remove the current RPM database by typing

    # rm -f /var/lib/rpm/__ db.*

  • Rebuild the database by typing (use the -vv or just -v depending on how much you like to watch output scroll by)

    # rpm -vv --rebuilddb

This step can take a while to complete.


Troubleshooting Software Package Tips

Following are a bunch of extraneous troubleshooting tips that you might find useful when you are working with RPM packages:

  • Check installed packages over 5 days-Once each day, a list of all installed RPM packages is created and copied to the /var/log/rpmpkgs file. You can refer to that list to see the full package name for each RPM package. The most recent 5 days of package are contained in the /var/log directory: rpmpkgs, rpmpkgs.1, rpmpkgs.2, rpmpkgs.3, and rpmpkgs.4. You can compare these files to see if any packages have been added, removed, or upgraded in the past 5 days.

  • Find packages most recently installed-Some software on your Linux system isn't working properly and you want to see if any software that has been added or upgraded recently might be the problem. Use the --last option to list your RPM packages in the order they were installed (starting from the most recent):

    # rpm -qa --last | less
gpg-pubkey-4f2a6fd2-3f9d9d3b        Fri 06 Feb 2004 10:02:09 PM PST
gpg-pubkey-e42d547b-3960bdf1        Fri 06 Feb 2004 12:49:58 AM PST
acroread-5.06-4                      Wed 04 Feb 2004 11:40:25 AM PST
mozilla-js-debugger-1.4.1-18        Mon 26 Jan 2004 11:33:50 PM PST
mozilla-dom- inspector-1.4.]-18     Mon 26 Jan 2004 11:33:43 PM PST

    From this example, you can se that I installed two GPG public keys on Friday. Before that I installed the Adobe Acrobat reader and upgraded a bunch of mozilla packages. I could then page through the entire 1500 or so packages on by system, in the order they were installed.

  • Find what package a file is in-Locating what package a file, command, or library comes from can help you track down documentation or other components related to that item. The -f query option to rpm lets list the package associated with any file. For example,

    # rpm -qf /etc/alternatives
chkconfig-1.3.9-1

    You see that the alternatives feature is included in the chkconfig package, which includes components to change system services.

  • Change an RPM to a Cpio archive-Convert an RPM package to cpio format by using the rpm2cpio command. Cpio is a standard utility for creating software archives and extracting files from those archives. By converting a package to cpio, you can essentially use the resulting cpio archive to copy the original contents of an RPM package to any Linux or UNIX system, or to extract a single file from an RPM package.


Software Package Troubleshooting Reference

Table 8-2 provides a reference for finding and troubleshooting RPM packages.

Table 8-2: RPM Package Troubleshooting References
General RPM information
RPM project | www.rpm.org
RPM documentation | Maximum RPM (.ps format): www.rpm.org/local/maximum-rpm.ps.gz
RPM HOWTO | www.rpm.org/support/RPM-HOWTO.html
RPM mailing lists | RPM.org mailing list: www.rpm.org/mailing_list/ Usenet newsgroup: linux.redhat.rpm
RPM documentation in Fedora | /usr/share/doc/rpm-* Man pages: rpm, rpm2cpio, rpmbuild, rpmcache, rpmgraph, gendiff, and rpmdeps
RPM configuration and log files /etc/rpm directory | Files that set the locations for RPM components. Generally there's no need to touch these files.
/var/lib/rpm | Location of the database of RPM packages that are currently installed.
/usr/lib/rpmdb/i386-redhat-linux | Location of the database of total RPM packages that are available on the full Fedora or Red Hat Linux distribution (when installed).
/var/log/rpmpkgs | List of all RPM packages on the current system.
RPM commands
rpm | Basic command for adding, removing, upgrading, and querying RPM packages.
redhat-config-packages | Launches the GUI tool to add and remove software packages.
rpm2cpio | Converts an RPM package into a cpio archive file.


Summary

Software in Fedora and other Red Hat Linux systems are packaged in RPM format. To install, upgrade, and delete those packages, you can use the rpm command.

Besides adding and removing software, the rpm command can be used for a variety of other duties, such as querying the RPM database or verifying the validity of the contents of an RPM package. The rpm command can also be used as a security tool to check installed software against software in the original RPM package, to make sure it hasn't been tampered with.


Chapter 9: File System, Disk, and Power Troubleshooting

Journaling file systems, excellent hardware probing, and sensible shutdown procedures have gone a long way toward taking the adventure out of managing your hard disks and the file systems that are stored on them in Linux. There are times, however, when something goes wrong or you want to do more than just sit back and let Linux run the show alone.

You might not be happy just sticking with the defaults when there are some performance gains to be had. Or you might want to get that extra bit of life out of your laptop's battery by better controlling when idle disks are spun down or when nonessential services are turned off. This chapter will help you look for trouble (and opportunities) when it comes to tuning and working with your hard disks, file systems, and battery power.

Starting with Linux File Systems

Much of what was once done manually to set up file systems in Linux can now be done automatically when you install most major Linux distributions. During installation, your hard disks are probed and configured based on:

  • Hard disk type-Linux determines the type of hard disk you are using and tries to set appropriate parameters for operation (data transfer modes, read or write-caching, 16- or 32-bit I/O support, and so on). Linux usually chooses workable hard disk settings, but allows you to change those settings if you like. (Ways of checking and changing hard disk settings are described in the Checking and Tuning Your Hard Disk section later in this chapter.)

  • Partitioning-The Linux installation process can automatically divide your hard disk into appropriate partitions or give you the opportunity to divide your hard disk yourself. During installation, Fedora and other Red Hat Linux systems offer Disk Druid to partition your disk. After your computer is installed, you can use tools such as fdisk to add, delete, and modify partitions.

  • File system types-Each disk partition is assigned a file system type and is structured in such a way that Linux can access the files, directories, devices, and other components that make up the file system. The ext3 file system is the default for Fedora and other Red Hat Linux systems, while Windows systems typically contain FAT, VFAT, or NTFS file systems. There are also a handful of specialty file systems you can use, such as iso9660 (the standard for CD-ROM file systems).

Instead of stepping through how to partition your disk and assign file systems during installation, we will go through different methods of troubleshooting problems that can arise later with your disks and file systems. Assuming that you are starting with an installed Fedora or Red Hat Linux system and you want to fix some problems with it, we're going to start by taking a look at what you have installed on your system.

Checking Partitions and File Systems

To see what partitions and file systems are configured on your computer, with Linux running perform the following steps from a Terminal window:

  1. Run the fdisk command to list information about your hard disk partitions, as follows:

    #fdisk -1
Disk /dev/hda: 6505 MB, 6505/12640 bytes
255 heads, 63 sectors/track, 790 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot    Start    End   Blocks   Id   System
/dev/hda1   *         1     13   104391   83   Linux
/dev/hda2            14    766  6048472+  83   Linux
/dev/hda3           767    790   192780   82   Linux swap

    This example shows a relatively small (6505MB or about 6GB) IDE hard disk (/dev/hda). That disk is partitioned into three parts: /dev/hda1 is an ext3 file system (Id 83) that starts at cylinder 1 and ends at cylinder 13 (104,391 blocks). The asterisk shows that it is a bootable file system. The next partition (/dev/hda2) is a much larger ext3 partition (6,048,472 blocks). The final partition is a swap partition (Id 82) that consists of 192,780 blocks.

    Your fdisk output will almost surely be of a different size. Here are some other ways that it might vary:

    • Disk name-The type of hard disk you have will be reflected in the disk device names you see. The hda stands for the first (1) IDE hard disk (hd). Other IDE hard disks would be named hdb, hdc, and so on. SCSI disks are named /dev/sda, /dev/sdb, and so on. A RAID device would look something like this when displayed from fdisk:

        Device  Boot    Start    End   Blocks   Id   System
/dev/hda1   *         1     13   104391   fd   Linux raid autodetect

    • File system types-In most cases, you will see ext3 (Id 83) and swap (82) file systems assigned to partitions. If you have a system where you can dual boot with Windows, you might also see HPFS/NTFS (Id 7), WIN95 FAT32 (Id b), FAT, or DOS 16-bit (Id 6). To see a complete list of available file system types, you can type the l option within fdisk as follows:

      #  fdisk /dev/hda
Command (m for help): 1
 0 Empty         1c Hidden Win95 FA   70 DiskSecure Mult  bb Boot Wizard hid
 1 FAT12         1e Hidden Win95 FA   75 PC/IX            be Solaris boot
 2 XENIX root    24 NEC DOS           80 Old Minix        c1 DRDOS/sec (FAT-
 3 XENIX usr     39 Plan 9            81 Minix / old Lin  c4 DRDOS/sec (FAT-
 4 FAT16 <32M    3c PartitionMagic    82 Linux swap       c6 DRDOS/sec (FAT-
 5 Extended      40 Venix 80286       83 Linux            c7 Syrinx
 6 FAT16         41 PPC PReP Boot     84 OS/2 hidden C:   da Non-FS data
 7 HPFS/NTFS     42 SFS               85 Linux extended   db CP/M / CTOS / .
 8 AIX           4d ONX4.x            86 NTFS volume set  de Dell Utility
 9 AIX bootabl   4e QNX4.x 2nd part   87 NTFS volume set  df BootIt
 a OS/2 Boota    4f QNX4.x 3rd part   8e Linux LVM        e1 DOS access
 b Win95 FAT32   50 OnTrack DM        93 Amoeba           e3 DOS R/O
 c Win95 FAT32   51 OnTrack DM6 Aux   94 Amoeba BBT       e4 SpeedStor
 e Win95 FAT16   52 CP/M              9f BSD/OS           eb BeOS fs
 f Win95 Ext'd   53 OnTrack DM6 Aux   a0 IBM Thinkpad hi  ee EFI GPT
10 OPUS          54 OnTrackDM6        a5 FreeBSD          ef EFI (FAT-12/16/
11 Hidden FAT1   55 EZ-Drive          a6 OpenBSD          f0 Linux/PA-RISC b
12 Compaq diag   56 Golden Bow        a7 NeXTSTEP         f1 SpeedStor
14 HiddenFAT16   5c Priam Edisk       a8 Darwin UFS       f4 SpeedStor
16 HiddenFAT16   61 SpeedStor         a9 NetBSD           f2 DOS secondary
17 HiddenHPFS/   63 GNU HURD or Sys   ab Darwin boot      fd Linux raid auto
18 AST SmartSl   64 Novell Netware    b7 BSDI fs          fe LANstep
1b HiddenWin95   65 Novell Netware    b8 BSDI swap        ff BBT
Command (m for help): q

    After you exit from fdisk, you can try listing the file systems that are in use or currently available on the system by typing cat/proc/filesystems. Note that this doesn't necessarily mean that a file system of that type is currently mounted. Also, many other file system types may be available as loadable modules. You can find some of those file system type modules in the directories in Fedora: /lib/modules/*/kernel/fs and /lib/modules/*/unsupported/fs.

  2. Check the size of each partition and how it is mounted on the file system with the df command:

    # df -h
Filesystem      Size   Used Avail Use% Mounted on
/dev/hda2       5.7G   3.3G  2.2G  61 % /
/dev/hda1        99M   6.6M   88M    8% /boot
none             30M      0   30M    0% /dev/shm

    Notice here that the /dev/hda2 partition is mounted at the root of the file system (/). The /dev/hda1 partition is mounted on /boot. The swap partition is represented by the /dev/shm device. It is possible that some partitions listed from fdisk are not shown by df. That's because partitions are only mounted if they are configured to mount automatically at boot time, configured to mount when inserted (such as with a CD), or if they were mounted manually later. For example, if you installed Fedora on a hard disk that contains a Windows partition (so you can dual boot), the Windows partition will not automatically be mounted in Fedora.

  3. To see what partitions are being mounted automatically at boot time, display the contents of the /etc/fstab file:

    # cat /etc/fstab
LABEL=/     /              ext3    defaults        1 1
LABEL=/boot /boot          ext3    defaults        1 2
none        /dev/pts       devpts  gid=5,mode=620  0 0
none        /dev/shm       tmpfs   defaults        0 0
none        /proc          proc    defaults        0 0
/dev/hda3   swap           swap    defaults        0 0
/dev/cdrom  /mnt/cdrom     udf,iso9660 noauto,owner,kudzu,ro 0 0
/dev/hdc4   /mnt/zip       auto    noauto,owner,kudzu 0 0
/dev/fd0    /mnt/floppy    auto    noauto,owner,kudzu 0 0

    There are three kinds of file systems you can see in this listing of the /etc/fstab file:

    • Automatically mounted file systems-The /and /boot partitions are automatically mounted by default when the system boots. That's because by default, all ext3 file systems listed in this file are mounted at boot time (as are any file systems that are not listed as noauto). 1 in the fifth field indicates that a file system needs to be dumped. The number 1 in the sixth field indicates that a file system check should be done on the root file system (/) first, 2 in the sixth field for the /boot file system indicates that it should be checked after the root file system.

    • Pseudofile systems-The pts, shm, and proc file systems are actually pseudofile systems (that is, they do not represent real disk partitions).

    • Manually mounted file systems-With a noauto option listed in the fourth field, the devices listed are not automatically mounted at boot time. The noauto option is useful for removable media, such as the CD-ROM, Zip, and floppy disk drives shown in the example.

    • Having an entry in the /etc/fstab file makes it possible for the mount command to mount the removable medium later with simply the device name (mount/dev/cdrom) or mount point (mount/mnt/cdrom) as an argument. The rest of the options are taken from the f stab entry.

  4. If you are using the GNOME desktop (which is the default), by default CD and DVD media are automatically mounted when you insert one of those disks into a drive. To see, and possibly change, this behavior, click Preferences ⇨CD Properties from the Red Hat menu. Figure 9-1 shows an example of the window that appears.

    Click To expand
    Figure 9-1: Set whether CDs are automatically mounted upon insertion.

    As you can see from the top check box, by default a data CD will be mounted (on /mnt/cdrom) if you don't change this preference. This feature (referred to as magicdev) can also otherwise react to the content of the CD or DVD by playing an audio CD or movie DVD or by opening a blank CD so you can write on it (that is a nice feature of nautilus, by the way).

    Another way you can have disks and partitions mounted automatically is with the automount feature. With automount, by changing to a mount point (or subdirectory) of a medium that is configured to automount, the device will be automatically mounted. That way, the medium is only mounted when it is needed.

    Fedora and Red Hat Linux systems offer two different packages for implementing automount. The Autofs automounter (autofs package) is configured in the /etc/auto.master and /etc/auto misc files. It is started by default from the autofs runlevel script. The BSD automounter (am-utils package) is configured in the /etc/amd.conf file and started with the amd runlevel script. Type man autofs or man amd for further information.

Now that you have some idea of how the file systems and partitions are set up on your computer, let's go through some related troubleshooting issues that may come up.

Troubleshooting Partitions and File Systems

Here are some tips for troubleshooting the partitions and file systems on your hard disks.

Computer Won't Boot (No Boot Screen)

If you don't get a GRUB boot screen when you try to boot from hard disk (and are expecting one), it could be that the master boot record is corrupted. You can reload the master boot record by booting from a rescue CD (assuming Linux is installed on the first IDE hard drive). Here's how:

  1. Reboot your computer with the first CD in the installation set for Fedora or Red Hat Linux in the CD drive.

  2. When you see the boot prompt, type:

    # linux rescue

  3. Add information about your language and keyboard as requested.

  4. Choose whether or not to start the network interfaces (you probably don't need to in this case). The rescue process will now look for your Linux installation.

  5. If the rescue mode finds your Linux installation on your hard disk, select OK to continue.

  6. After the computer enters rescue mode, you should see a regular shell prompt (#). Assuming rescue mode mounted your hard disk under /mnt/sysimage , type the following to change your root directory to that of your Linux installation:

    # chroot /mnt/sysimage

  7. To install a new master boot record from the grub.conf configuration on your hard disk, type the following:

    # /sbin/grub-install /dev/hda
Installation finished. No error reported.
This is the contents of the device map /boot/grub/device.map.
Check if this is correct. If any of the lines are incorrect,
fix it and re-run the script 'grub-install.'
#  this device map was generated by anaconda
(fd0)    /dev/fd0
(hd0)    /dev/hda

    Remove the CD and reboot your computer. You should see your original GRUB screen and be able to select which partition and/or kernel to boot.

If you had deleted a partition and simply reinstalling grub to the MBR doesn't work, you may need to do some more work. It's possible that your boot partition is no longer valid, so the boot loader is not able to find your /boot/grub/grub.conf file. Start by booting a rescue CD and running the grub command. Assuming you are booting from the MBR on your first IDE hard disk, type the following (with the first text followed by the tab key):

   grub> root (hd0,   <tab>
    Possible partitions are:
    Partition num: 0, Filesystem type is ext2fs, partition type 0x83
    Partition num: 1, Filesystem type is ext2fs, partition type 0x83
    Partition num: 2, Filesystem type unknown, partition type 0x82

After you press the Tab key, you will see a list of partitions on the first IDE hard disk. Look for the first one that shows up as ext2fs (ext3 file systems appear here as ext2fs). If the first ext2fs that shows up in the list is partition number 0, enter that into your grub command line by typing:

   grub> root (hd0,0)

To make sure that you assigned the proper partition as your boot partition, try typing the following:

   grub> cat /grub/grub.conf

If this lists the contents of your grub.conf file, you have assigned the correct partition as your boot partition.

Recovering a Deleted File

You deleted a file (or a set of files) by mistake and you want it back. Is it possible to get it back? Unfortunately, the answer is probably not. Unlike the earlier ext2, the ext3 file system erases pointers to the deleted file's inode.

If you are using an ext2 file system, however, you might be able to use the debugfs command to find and restore your lost file. With ext2, the blocks are simply marked as being unused, so if the space hasn't been reused yet, the file you need might still be there.

Let's say you deleted a file from your home directory (/home/chris/myfile.txt) and you want it back. If you have a separate /home partition (say, for example, /dev/hda5), you can have anyone using the computer log off for a moment, and then do the following:

  1. Determine the partition containing the file you want to recover. Typing df - h or mount will show your currently mounted partitions.

  2. Unmount the directory containing the file you want to recover. (If the file is on the root partition, you should shut down the computer and go into rescue mode, as described in Chapter 6.) Here's how to unmount the /home partition (if /home were on /dev/hda5, you could use that instead of /home):

    # umount /home

    If the directory fails to unmount, see the Can't Unmount a Directory section later in this chapter.

  3. Run the debugfs command on the home partition. Then from the debugfs prompt, list the deleted files from the directory you are interested in:

    # debugfs /dev/hda5
debugfs 1.35-WIP (07-Dec-2003)
debugfs: ls -ld /home/chris

    Look for the inode number for the file you want to undelete. With an ext3 file system, the entry will look something like this and you will not be able to recover it:

    <    0>    0(1)    0    0    0    myfile.txt

    However, if you see the number greater than zero surrounded by less than and greater than signs, you might be able to recover the file. That is the inode number for the file. For example, if you were to try this on an ext2 file system and if the inode is shown as <115>, you could copy it from the /home/chris directory from the unmounted home partition by typing:

    debugfs: dump <115> /tmp/myfile_dumped.txt

  4. Remount the file system (mount /home) and you are back in business with your recovered file safely in the /tmp directory. Copy it where you want it to go permanently and you are ready to go.

Can't Unmount a Directory

You try to unmount a directory or device representing a partition and you see a message similar to the following:

   unmount: /whatever: device is busy

The most common reason for an unmount to fail is that some process is holding a file or directory open on the partition. In particular, if a directory in the partition is the current directory for any shell, that will cause the device to be busy. To solve the problem, you can often simply close the shell or change the directory so that the current directory is on another partition.

On a multiuser system that has a lot of users logging in and out, it may not be as easy to find out who is making the partition busy. You can try using ps aux to see what is running and kill any run-away process. Or you can go into single-user mode, if it is critical to unmount the directory now. If you can bear to wait a bit, try a lazy unmount:

   # umount -l /whatever

The lazy unmount detaches the file system from the hierarchy. However, it waits until the file system is no longer busy to clean up all references to the file system.

Don't Know What Kind of a File System is on a Device

You can find out what file system type is on a device using the blkid command. For example, to check the first partition on the first IDE hard disk, type:

   # blkid /dev/hda1
   /dev/hda1: LABEL="/boot" UUID="69477377-81c2-4846-bd3e-
   ad5fe8c2caee" SEC_TYPE="ext3" TYPE="ext2"

Checking a File System

If a file system was not cleanly unmounted (for example, when someone kicks out the power cord) or if the time has been reached for a scheduled file system check, an fsck process will run and check (and correct) the file system when you boot the computer. You can check a file system yourself, without rebooting the system, however, if you like. To do that, first unmount the file system.

Note

If you are checking the root file system, you should boot from a rescue disk. You should not do an fsck on a mounted file system, and you cannot unmount the root file system on a running Linux system. You could mount the root file system (/) read-only (add ro to the mount entry in /etc/fstab and reboot) and run fsck on it.
Running fsck

Once the file system is unmounted, run fsck. For example, if you are checking the /boot partition, type the following after it is unmounted:

   # fsck /boot
   fsck 1.34 (25-Jul-2003)
   /boot: clean, 42/26104 files, 11632/104388 blocks
Checking for Bad Blocks

To see if there are bad blocks on the file system, run the badblocks command as follows:

   # badblocks -v /dev/hda1
   Checking blocks 0 to 104391
   Checking for bad blocks (read-only test): done
   Pass completed, 0 bad blocks found.

Repairing a Damaged ext3 File System

You can force a more thorough check of an ext3 file system using the -f option. To see more details about the output, use the -v option as well:

   # e2fsck -fyv /dev/hda1
   e2fsck 1.34 (25-Jul-2003)
   fsck 1.34 (25-Jul-2003)
   Pass 1: Checking inodes, blocks, and sizes
   Pass 2: Checking directory structure
   Pass 3: Checking directory connectivity
   Pass 4: Checking reference counts
   Pass 5: Checking group summary information
   42 inodes used (0%)
          1 non-contiguous inode (2.4%)
            # of inodes with ind/dind/tind blocks: 10/4/0
      11632 blocks used (11%)
          0 bad blocks
          0 large files

   25 regular files
          3 directories
          0 character device files
          0 block device files
          0 fifos
          0 links
          5 symbolic links (5 fast symbolic links)
          0 sockets
   --------
          33 files

This shows a forced, thorough check being done of a small file system. The -y option indicates that if any errors are encountered, and e2fsck would normally ask if you wanted to try to correct each error, it should instead just correct them all without asking for each one individually.

Creating, Modifying, and Deleting File Systems

The fdisk command and related commands such as sfdisk and cfdisk are used to display, modify, delete, and add disk partitions. Once added, file systems can be assigned to the partitions and those file systems can be mounted and used by Linux.

Caution

On a critical system, an inexperienced user shouldn't use fdisk other than for listing the partitions on your system (the -l option). If you do want to use fdisk to change, delete, or add partitions, be aware that a small mistake can completely destroy your data or make it inaccessible. If you are just learning, try this on a machine you don't require. Regardless of your experience, you should make sure that any critical data is backed up.

In this example, I had left about 1GB of free space on my hard disk that I wanted to create into a partition that is mounted on the /mnt/backup directories (so I could use it to make backup copies of my document files). Here is what I did to start the fdisk session, and create the new partition:

   # fdisk /dev/hda
   Command (m for help): n
   Command action
      l   logical (5 or over)
      p   primary partition (1-4)
   p
   Selected partition 3
   First sector (75987450-78165359, default 75987450): <Enter>
   Using default value 75987450
   Last sector or +size or +sizeM or +sizeK
      (75987450-78165359, default 78165359): <Enter>
   Using default value /8165359
   Command (m for help): w

In the previous example, I created a primary partition (partition 3 was available) and just took the defaults. That allowed me to use the rest of the available disk space for the new partition.

After that, I made the partition into an ext2 file system. Here is an example of creating an ext2 file system on the /dev/hda3 partition:

   # mkfs -text2 /dev/hda3
   mke2Fs 1.34 (25-Jul-2003)
   Filesystem label=
   OS type: Linux
   Block size=4096 (log=2)
   Fragment size=4096 (log=2)
   135648 inodes, 2/1096 blocks
   13554 blocks (5.00%) reserved for the super user
   First data block=0
   9 block groups
   32768 blocks per group, 32/68 fragments per group
   15072 inodes per group
   Superblock backups stored on blocks:
           32768, 98304, 163840, 229376
   Writing inode tables: done
   Writing superblocks and filesystem accounting information: done
   This filesystem will be automatically checked every 32 mounts or
   180 days, whichever comes first. Use tune2fs -c or -i to override.

Next 1 added a label to the partition, based on its mount point:

   # e21abel /dev/hda3 /mnt/backup

If I want to make it into a journaling file system (ext3) now, I can do that using the tune2fs command. This action will take about 33MB of disk space to create the journal file needed by ext3. Here's an example:

   # tune2fs -j /dev/hda3
   tune2fs 1.34 (25-Jul-2003)
   Creating journal inode: done
   This filesystem will be automatically checked every 32 mounts
   or 180 days, whichever comes first.
   Use tune2fs -c or -i to override

After that, I created an entry in the /etc/fstab file, so the partition would be automatically mounted each time Linux started (if I hadn't added the journal, the file system type would be ext2 instead of ext3):

   LABEL="/mnt/backup     /mnt/backup    ext3   defaults 1 2

After that I can mount the file system immediately by typing:

   # mount /mnt/backup

At this point, the new partition will be mounted on the /mnt/backup directory each time the system boots.


Checking and Tuning Your Hard Disk

To get the best performance out of your hard disk, you sometimes have to do a bit of tuning. Although Linux will properly detect and configure most hard disks so they will work, sometimes the default settings will be either too conservative (so you get slow performance) or too aggressive (not as likely, but can result in errors).

This section describes how to display information about your hard disk. It then describes different ways of tuning disk options and benchmarking the results of those changes.

Getting Information About Your Disk

You can find out information about your hard disk using hdparm, fdisk, and other commands. To find out some general information, you could start out with hdparm simply followed by the name of the disk device. For example, you would identify the first IDE (ATA) hard disk using /dev/hda in the following command line:

   # hdparm /dev/hda
   /dev/hda:
     multcount    = 16 (on)
     IO_support   = 1 (32-bit)
     unmaskirq    = 1 (on)
     using-dma    = 1 (on)
     keepsettings = 0 (off)
     readonly     = 0 (off)
     readahead    = 8 (on)
     geometry     = 4865/255/63, sectors = 78165360, start = 0

The output shows some basic information about your hard disk. You can also list most of these items separately using options described below:

  • multcount-With multcount set to a number other than 0, IDE block mode is enabled. This allows multiple sectors to be transferred with each I/O interrupt (2, 4, 8, 16, or 32 sectors). Type hdparm-m /dev/hda to list just this line.

  • IO support-IO support set to 1 indicates that the hard disk supports 32-bit data transfers across the bus (usually PCI) to the interface card. From the interface card to the hard drive, data actually only uses a 16-bit connection over the ribbon cable. To take advantage of the 32-bit bus, however, data is packaged together in two 16-bit groups before it is sent to RAM or the CPU. Type hdparm-c /dev/hda to list just this line.

  • unmaskirq-The unmaskirq setting of 1 turns on the ability of the disk's driver to unmask other interrupts while processing a disk interrupt (to improve performance). Type hdparm-u /dev/hda to list just this line.

  • using dma-Describes whether or not DMA is on for the hard drive (1 is on; 0 is off). If the hard drive supports DMA, having DMA on can improve performance substantially. It should improve I/O throughput without increasing CPU use.

  • keepsettings-Shows whether or not DMA (-d), multcount (-m), and unmaskirq (-u) settings are kept after a soft reset. This is typically off, so that a reset will change settings back to a sane state in case of bad -dmu settings. Type hdparm-k/dev/hda to list just this line.

  • readonly-Tells whether or not the hard disk device is set for read-only. If read-only is on (1), no write operations are allowed to the disk. Type hdparm-r/dev/hda to list just this line.

  • readahead-This setting describes whether read-ahead seeks are on and, if so, how many sectors are read ahead. During large sequential reads, performance can improve by reading ahead some number of sectors, in anticipation that they will be needed by the operation requesting the data. The default eight sectors causes the file system to read ahead 4kb of data. Type hdparm-a/dev/hda to list just this line.

  • geometry-Displays information about the disk's geometry. From the output just shown, you can see that the hard disk has 4,865 cylinders, 255 tracks, and 63 sectors per tracks. The total number of addressable sectors is 78,165,360. The starting offset of the device is 0. Type hdparm-g/dev/hda to list just this line.

To display detailed identification information about your hard drive, you can use the hdparm -i command. The information displayed by hdparm-i was detected from the hard drive when you booted the computer. Here is an example:

   # hdparm -i /dev/hda
   /dev/hda:

   Model=WDC WD400JB-00ENA0, FwRev=05.03F05, SerialNo=WD-WCAD13418938
   Config={HardSect NotMFM HdSw>15uSec SpinMotCtl Fixed DTR>5Mbs FmtGapReq}
   RawCHS=16383/16/63, TrkSize=57600, SectSize=600, ECCbytes=40
   BuffType=DualportCache, BuffSize=8192kB, MaxMultSect=16, MultSect=16
   CurCHS=4047/16/255, CurSects=16511760, LBA=yes, LBAsects=78165360
   IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
   PIO modes:  pio0 pio1 pio2 pio3 pio4
   DMA modes:  mdma0 mdma1 mdma2
   UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5
   AdvancedPM=no WriteCache=enabled
   Drive conforms to: device does not report version:

   * signifies the current active mode

The model of this hard drive is a Western Digital WD400JB Caviar 40GB hard drive. If you ever need to contact technical support team about the drive, you can see the firmware revision (FwRev) and serial number (SerialNO) for the drive from this output. Because your settings can change after Linux starts running, you can see current hard drive settings using the -I option as follows:

   # hdparm -I /dev/hda
   /dev/hda:
   ATA device, with non-removable media
     Model Number:       WDC WD400JB-00ENA0
     Serial Number:       WD-WCAD13418938□
     Firmware Revision:   05.03E05
   Standards:
     Supported: 5 4 3 2
     Likely used: 6
   Configuration:
     Logical        max   current
     cylinders      16383 4047
     heads          16    16
     sectors/track  63    255
   --
     CHS current addressable sectors: 16511760
     LBA    user addressable sectors: 78165360
     device size with M = 1024*1024:     38166 MBytes
     device size with M = 1000*1000:     40020 MBytes (40 GB)
   Capabilities:
     LBA, IORDY(can be disabled)
     bytes avail on r/w long: 40 Queue depth: 1
        .
        .
        .

The information just shown was obtained directly from the current drive. So if you have done any disk tuning since the system booted, this output (rather than output from -i) should be used. Near the end of this output is the Commands/features section. This section shows features that are supported by the drive. If the feature is supported, an asterisk appears in the Enabled column. Here's an example of the commands/features output from hdparm-I for a Western Digital WD800JB 80 GB hard drive:

   Commands/features:
     Enabled Supported:
        *    READ BUFFER cmd
        *    WRITE BUFFER cmd
        *    Host  Protected Area feature set
        *    Look-ahead
        *    Write cache
        *    Power Management feature set
             Security Mode feature set
        *    SMART feature set
        *    FLUSH CACHE EXT command
        *    Mandatory FLUSH CACHE command
        *    Device Configuration Overlay feature set
        *    48-bit Address feature set
             Automatic Acoustic Management feature set
             SET MAX security extension
        *    DOWNLOAD MICROCODE cmd
        *    SMART self-test
        *    SMART error logging

Many of the features described in the next section are listed in this section. As you turn these features on and off, you'll notice the asterisk in the Enabled column going on and off when you re-run hdparm-I.

Changing Hard Disk Settings

The hdparm utility is one of the key tools for setting parameters for tuning your hard disk. Before we talk about changing disk settings with hdparm, however, take a look at the first comment in the configuration file (/etc/sysconfig/harddisks) that comes with the hdparm utility:

   WARNING!!!
   The kernel will autodetect the correct settings for most
   drives. Overriding these settings with hdparm can cause data
   corruption and/or data loss.

I think it's a great idea to practice and learn about some of the more potentially destructive features of hdparm on a test computer-or at least on a computer before it is deployed in your business. With that said, however, there are times when you might want to override some conservative defaults to improve the performance of your hard disk.

This section describes some ways of using hdparm to check and tune your disks. It also describes ways of benchmarking your system to check whether the options have improved or affected your disk performance.

Benchmarking Your Changes

A common practice is to check the performance of your hard disk before and after you make changes to it. This gives you some idea of the effect of your changes. Common tools for testing your hard disk performance (as well as the general act of reading and writing data on your system) are the hdparm-Tt and time commands. I also describe the bonnie utility for testing performance, which is not part of Fedora or other Red Hat Linux distributions.

Timing Reads

Using hdparm, you can time cache reads (-T) and device reads (-t).

  • -T option-This option essentially checks the how quickly data can be passed between the processor, memory, and disk cache.

  • -t option-This option reads continuously from the hard disk without reading previously cached data. This gives an indication of the speed at which Linux can read sequential data from the disk.

Here's an example of hdparm-Tt:

   # hdparm -Tt /dev/hda
   /dev/hda:
   Timing buffer-cache reads: 1476 MB in 2.00 seconds = 740.00 MB/sec
   Timing buffered disk reads: 164 MB in 3.01 seconds = 35.18 MB/sec

The output just shown is from a Western Digital Caviar 80MB hard drive (WDC WD800JB). The first line of output shows the number of MB of data (1,476MB) that can be read from cache in about 2 seconds, without touching the hard disk at all. The second line shows the number of MB of data (164MB) that can be read from the hard disk directly in about 3 seconds.

By way of comparison, I ran the same command on a Quantum Bigfoot CY6480A 6GB hard drive:

   # hdparm -Tt /dev/hda
   /dev/hda:
   Timing buffer-cache reads: 128 MB in 2.01 seconds = 63.56 MB/sec
   Timing buffered disk reads: 20 MB in 3.31 seconds =  6.04 MB/sec

You can see that the reads are considerably slower from this circa 1996 hard drive (not a bad command to try if you are looking to buy a used computer).

Here are a few tips for benchmarking with hdparm-Tt to get the best results:

  • Have as few processes as possible active when you do the benchmark. (You can run the top command to see active processes. Type P (which is the default) to sort by CPU usage. Then you can kill any very active processes that you don't need.)

    Caution

    Don't just randomly go killing processes, even processes that are consuming a lot of CPU time. A better idea might be to go into single-user mode, since that should provide you with a system with very little activity.

  • Run it a few times, to be sure a momentary hiccup doesn't skew your results.

  • Have some free memory available. (If you are using top, press M to sort by memory usage and kill any process that is not required to free some memory.)

Timing Writes

To test options that change how writing is done to a disk (such as write-caching), you can simply time a large write operation with the options on or off. Here is an example where I used the time command to see how long it would take to copy a large file (in this case, a 637MB iso image) from one partition to another:

   # time cp severn-i386-disc1.iso /root
   real     0m34.295s
   user     0m0.110s
   sys      0m7.860s
Note

Because the I/O speed depends to some extent on the location of the data on the platter, you will probably get different results from running the exact same command several times. By running it a few times, however, you can get an idea of the average amount of time the command takes.

The example just shown copies the iso image from the current directory to the /root directory. The time command shows that the operation took 34.295 seconds in real time. The user and sys lines show the amount of CPU time that was consumed by user-space calls and from kernel-space calls, respectively. Next, I turned off the write-caching feature (hdparm -W0), removed the iso file from /root, and ran the command again:

   # time cp severn-i386-discl.iso /root
   real  1m59.077s
   user  0m0.60s
   sys   0m8.900s

You can see that the copy took about four times as long (1 minute, 59.077 seconds) to complete. So, if you are willing to risk possible data loss in case power goes out, you can see that the time command shows significant improvement in completing a large copy operation with write-caching on in this case. (Of course, if you are that worried about data loss due to power going out, you should get a UPS.)

Instead of a plain copy, you can use the dd command to take a bit more control over the writes you test. With dd, you can precisely specify the block size and the number of blocks being copied. Using /dev/zero as the input file, you can copy as many "\0" characters to the output file as you like. Here's an example:

   # time dd if-/dev/zero of=/tmp/my-file bs=8k count=65536
   65536+0 records in
   65536+0 records out
   real    1m7.649s
   user    0m0.030s
   sys     0m3.780s

This created a 513MB /tmp/my_file. I can adjust the blocksize from 8k or the count from 65536 to test the disk performance based on different amounts and sizes of data.

Checking Reads, Writes, and CPU with Bonnie

Bonnie is a nice tool for checking performance of your hard disk, CPU, and other components in ways that emulate how the system is actually used. Using bonnie, you can check the actual system calls that are being made by the kernel to different devices as well as library calls to the basic libc library. The following are the types of performance activities you can check with bonnie:

  • Programs that read and write single characters-Bonnie can test writes of single characters (putc call) and (getc call) to hard disks. The potential bottleneck for these operations is the time it takes to make libc library calls on your computer.

  • Programs that read and write blocks of data-Bonnie can test writes (write call) and reads (read call) of blocks of data. Because performance of these activities may reflect buffering (that is, data that is in memory but not being currently read or written to hard disk), you might want to add some options to insure that you are testing disk performance and not just buffering. With the -y option to bonni e, buffered data is synced to hard disk after each write.

  • Database applications-Bonnie can test activities that are similar to the way applications access data in a database. The block in/out rewrite test that bonnie does interacts with data on a hard disk by reading some data, changing several bytes, writing the data back to disk, and rereading the data. This is a kind of overall test of how well your system might handle database access.

  • Busy system-By requesting more data than the system can handle (using the lseek call), bonnie can emulate how an overloaded system will perform. This tests how fast the hardware is at randomly accessing data from the hard disk.

The Bonnie project (www.textuality.com/bonnie) contains descriptions of bonnie and links where you can get the software (it's not included in any Red Hat distribution). I found a SuSE RPM version at www.garloff.de/kurt/linux/bonnie, which had a few nice enhancements. You can build from the source RPM you get there or use the SuSE i386 binary (which seems to work fine on Fedora and other Red Hat systems). You can also get that package from rpmfind.net (until they change the release number) by typing the following (all on one line):

   # wget rpmfind.net/linux/SuSE-Linux/i386/8.2/suse/i586/bonnie-1.4-
   124.i586.rpm

It is a good idea to run bonnie with amount of data larger than the default (100MB by default). Likewise, for a better test of hard disk performance, adding the -y option will work the disk harder by syncing all writes to disk. I ran the following bonnie command with write-caching on (hdparm-W1/dev/hda) or you could drop -y if write-cashing is off:

   # bonnie -s 300 -y
   Bonnie 1.4: File './Bonnie.18177', size: 314572800, volumes: 1
   Writing with putc()...         done: 20386 kB/s 89.0 %CPU
   Rewriting...                   done: 15913 kB/s 25.1 %CPU
   Writing intelligently...       done: 34819 kB/s 31.8 %CPU
   Reading with getc()...         done: 20819 kB/s 95.6 %CPU
   Reading intelligently...       done: 36583 kB/s 31.3 %CPU
   Seeker 1...Seeker 2...Seeker 3...start 'em...done...done...done...
              ---Sequential Output (sync)----- ---Sequential Input- -Rnd Seek-
              -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- -04k (03)
   Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
   toy 1* 300 20386 89.0 34819 31.8 15913 25.1 20819 95.6 36583 31.3 257.2 1.7

With the command just shown, 300MB of data (- s 300) is used for each operation and all writes are synced to disk (-y). Assuming a system with little or no extra activity, the CPU numbers above show cases where character-at-a-time writes (putc) and reads (getc) proceed about as fast as the processor can handle (89.0% and 95.6%, respectively). The hard disk, rather than the CPU, is the bottleneck for block writing and reading (31.8% and 31.3%, respectively), as well as rewriting (25.1%).

After that, I turned off write cache (hdparm-W0/dev/hda) and ran bonnie again.

   # bonnie -s 300 -y
   Bonnie 1.4: File './Bonnie.18151', size: 314572800, volumes: 1
   Writing with putc()...         done: 5950 kB/s 26.0 %CPU
   Rewriting...                   done: 5070 kB/s 8.0 %CPU
   Writing intelligently...       done: 6179 kB/s 5.6 %CPU
   Reading with getc()...         done: 20820 kB/s 96.0 %CPU
   Reading intelligently...       done: 33795 kB/s 29.7 %CPU
   Seeker 1...Seeker 2...Seeker 3...start 'em...done...done ...done ...
   ---Sequential Output (sync)----- ---Sequential Input- -Rnd Seek-
             -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- -04k (03)
   Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU
   toy 1* 300 5950  26.0 6179  5.6  5070   8.0 20820 96.0 33795 29.7 226.2 1.8

As you might guess, writing to disk in particular slowed down quite a lot. Writing operations were three to five times faster with write cache enabled. For more information on options to use with bonnie, see the bonnie man page or the bonnie README file (in /usr/share/doc/ packages/bonnie).

Enabling Disk Write-Back and Read-Ahead Caches

You can improve overall write performance to disks by enabling disk write-back caches. Likewise, tuning the disk to do read-ahead caches can improve disk read performance when large amounts of sequential data are requested from the hard disk.

Turning on Write-Back Cache

With the write-back option on, a write operation can appear to be done after data is written to cache, but before it is actually moved to the hard disk. The driver can move on to get more data, and delays that come from slow seek times and disk revolutions can be made up over time.

Risks to enabling write-back caches include:

  • Power failures-If someone kicks out the power cord or if there is a power surge, it is possible that data the driver has been told was written to hard disk is not yet written on the physical hard disk. This can lead to data corruption.

  • Data flushed out of order-It's possible that a second data block can be flushed to disk before the first block arrives, which might also cause data corruption.

  • Data might become inconsistent-In instances where data must always be consistent, write-back caching should be turned off. In RAID systems where write stripes can span multiple disks, it is more likely for inconsistencies to arise if write-back caching is on.

Most modern hard drives have write cashing enabled by default. In fact, in many cases, the driver will not let you disable it. However, with some older hard drives, you can risk data corruption by turning this feature on. In fact, the hdparm command labels this feature as DANGEROUS.

If this feature is not already on (and you have assessed the risk of data loss as acceptable), you can turn on write-caching using the following command:

   # hdparm -W1 /dev/hda
   /dev/hda:
   setting drive write-caching to 1 (on)

When I used the time command to time a large file copy (see the Timing Writes section), there was significant real-time improvements (about four times faster) with write-caching on (-W1).

Turning on Read-Ahead Cache

With the read-ahead cache option turned on (which is the case by default in most cases), the hard disk's driver will read ahead a specified amount of data with the assumption that the read action will need more data. If the feature is turned on, you should see the following near the end of hdparm -I output:

   Commands/features:
     Enabled Supported:
        *    Look-ahead

You use the -A option to turn on (1) or off (0) the read-ahead feature. The following shows the feature being turned on:

   # hdparm -A1 /dev/hda
   /dev/hda:
   setting drive read-lookahead to 1 (on)

With read-ahead on, you can try different values to set how far ahead data should be read. You do that with the -a option. For example, to set the read-ahead value to eight sectors (4KB) you would type

   # hdparm -a8 /dev/hda
   /dev/hda:
    setting fs readahead to 8
    readahead    = 8 (on)

With a slow, older hard disk, it can help to have a higher read-ahead value (to a point). The following examples show how read performance differed when I changed the read-ahead values with my old Quantum Bigfoot hard drive (described earlier).

Note

Notice that changing the read-ahead value doesn't have any impact on the buffer-cache test. That's because the buffer-cache reads test only gets data from cache and doesn't touch the hard disk.

The following shows read performance with read ahead turned off (-A0). I had similar results when I set read ahead to 8 (-a8).

# hdparm -a8 /dev/hda
# hdparm -Tt /dev/hda
/dev/hda:
 Timing buffer-cache reads: 128 MB in 2.00 seconds = 63.98 MB/sec
 Timing buffered disk reads: 10 MB in 3.60 seconds = 2.77 MB/sec

The next example shows how performance improved with read ahead set to 128 (I had similar results on setting it to 256):

# hdparm -a128 /dev/hda
# hdparm -Tt /dev/hda
/dev/hda:
 Timing buffer-cache reads:  128 MB in 2.01 seconds = 63.69 MB/sec
 Timing buffered disk reads:  22 MB in 3.24 seconds = 6.79 MB/sec

Setting read ahead to 64 resulted in improved performance, although not quite as much as setting it to 128 or 256:

# hdparm -a64 /dev/hda
# hdparm -Tt /dev/hda
/dev/hda:
 Timing buffer-cache reads:   128 MB in 2.01 seconds = 63.56 MB/sec
 Timing buffered disk reads:  20 MB in 3.25 seconds = 6.16 MB/sec

Turning on DMA

Hard disks typically support several different modes for transferring data, direct memory Access (DMA) is enabled on most hard disks today by default. In most cases, DMA will be turned on and the best possible DMA mode will be enabled. Older disks that don't operate in DMA mode can operate in the slower PIO mode. Here are descriptions of popular transfer modes for IDE/ATA hard disks:

  • Programmed I/O (PIO)-This is a fairly safe transfer mode, but requires a lot of CPU overhead and results in slower data transfers. Available modes include pio0, pio1, pio2, pio3, and pio4.

  • Multiword DMA (MDMA)-Multiword DMA allows the hard disk to communicate directly with system memory, bypassing the CPU. It does this using a feature called bus mastering, which lets the hard disk device control the PCI bus. Available modes include mdma0, mdmal, and mdma2.

  • Ultra DMA (UDMA)-UDMA refers to certain DMA modes that operate at a much higher rate of speed than multiword DMA rates. Available UDMA modes include udma2 (ATA/33), udma4 (ATA/66), udma5 (ATA/100), and udma6 (ATA/133).

Caution

Use caution when you are changing DMA settings. Linux will let you switch to an unsupported DMA mode using hdparm, which could result in data corruption. Be sure to back up any important data before trying different DMA modes.

To enable UDMA, your computer's BIOS must support DMA and the I/O controller and the hard drive itself must be capable of UDMA transfers. For modes greater than udma2, your hard disk must be connected with a 40-pin, 80-conductor cable. The hard disk should report which DMA modes it supports. Here's an example of the portion of hdpa rm- i output that describes supported DMA modes:

   # hdparm -i /dev/hda
      .
      .
      .
   PIO modes:  pio0 pio1 pio2 pio3 pio4
   DMA modes:  mdma0 mdma1 mdma2
   UDMA modes: udma0 udma1 udma2 udma3 udma4 *udma5

The asterisk next to udma5 shows that it is the mode that was activated during boot time. To see the current setting (in case you changed it), use the hdparm -I option. To turn on DMA mode, you can use the hdparm-d option and to switch DMA modes you use the -X option. For example, to turn on DMA and set the current mode to mdma2 for the first IDE hard disk, you could type the following:

   # hdparm -d1 -X34 /dev/hda
   /dev/hda:
    setting using_dma to 1 (on)
    setting xfermode to 34 (multiword DMA mode2)
   using_dma           = 1 (on)

Table 9-1 shows the different -X values you could use to set different DMA and PIO modes.

Table 9-1: DMA and PIO Transfer Modes
Mode | Option to hdparm | Theoretical Maximum Transfer Rate
pio0 | -X08 | 3.3MB per second
pio1 | -X09 | 5.2MB per second
pio2 | -X10 | 8.3MB per second
pio3 | -X11 | 11.1MB per second
pio4 | -X12 | 16.7MB per second
mdma0 | -X32 | 4.2MB per second
mdma1 | -X33 | 13.3MB per second
mdma0 | -X34 | 16.6MB per second
udma0 | -X64 | 16.7MB per second
udma1 | -X65 | 25MB per second
udma2 | -X66 | 33.3MB per second
udma3 | -X67 | 44.4MB per second
udma4 | -X68 | 66.7MB per second
udma5 | -X69 | 100MB per second

Again, most times Linux will probe the best available DMA settings for your hard drive and configure it correctly. You can change them as just described, however, if you feel that the settings were either too conservative or too aggressive. After you change these modes, try running some benchmarks described earlier to choose the mode that is working best.

Trying Other Disk Tuning Options

Here are a few other parameters you can tune with hdparm:

  • Block mode transfers-Turn on multisector I/O mode (also called IDE Block Mode) to allow the driver to transfer more than one sector for each I/O interrupt. The -m0 option turns off this feature. A number other than zero sets the amount of data transferred in each interrupt to that number (-m2, -m4, -m8, or -m16). This is often set to 16 for 16 sector transfers. (See the MaxMultSect setting from the hdparm- i output to see the maximum number you can set multcount to.)

  • Hard disks with smaller drive buffers may perform better with smaller sector counts. This includes some WD Caviar drives. The following is an example of this option being set to 16:

    # hdparm -m16 /dev/hda
/dev/hda:
 setting multcount to 16
 multcount    = (on)

  • Keep settings over reset-You got your disk tuned just the way you want it. So you want to preserve all your settings as long as your machine stays up. The problem is that once the drive is reset, your drive goes back to all the default options. To override that behavior, you can use the hdparm options - k1 (to preserve -dma options) or -K1 (to preserve -APSWXZ options).

    Note

    The -K1 option only preserves settings over a reset of the disk drive controller. To make the changes permanent across a computer reboot, see the Making hdparm Changes Permanent section.

  • Set options quietly-When you set options in a configuration file or any other time that you don't want any output, you can add the - q option in front of each option. For example, hdparm_qd1-qX69/dev/hda will suppress output when turning on DMA support and udma5 mode.

Making hdparm Changes Permanent

Once you find hdparm settings you like, you can make them permanent in Fedora and other Red Hat Linux systems by adding them to the /etc/sysconfig/harddisks file. That way, the settings are used each time you boot your computer. You can set some options by simply uncommenting appropriate lines from the hard disk file. For example:

   USE_DMA=1
   MULTIPLE_10=16
   EIDE_32BIT=3
   LOOKAHEAD=1

By uncommenting those lines (removing the "#" character) you can turn on DMA, enable I/O transfer in multiple sectors (16 as set here), enable 32-bit I/O support, and turn on the read-ahead feature, respectively. As an alternative, you can add hdparm parameters directly to the EXTRA_PARAMS line at the end of this file. For example:

   EXTRA_PARAMS="-d1 -X67"

In this example, DMA is turned on (-d1) and the mode is set to udma3 (-X67).


Managing Power Settings for Laptops

Laptop computers need to be able to efficiently manage battery power to be effective tools. Besides settings used for manipulation in your laptop's BIOS, there are several utilities that come with Fedora Core and other Linux systems that can be used to monitor power use on your laptop and respond to power events that occur, such as:

  • Low battery power-The computer can send warnings, display low-battery icons, or even suspend hardware activity.

  • Inactivity-If there has been no activity on the computer for a set amount of time, you can extend battery life by spinning down disks and turning off services (such as audio and network services).

By detecting whether your laptop is running on AC power or battery, your computer can go into different levels of power savings for each of those states. Tools for managing and troubleshooting power problems on laptop computers in Fedora include:

  • Advanced Power Management (APM)-The APM software in Fedora consists of the a pmd daemon (which monitors battery use) and several utilities for displaying and reacting to battery activity. These features are contained in the apmd software package.

  • Advanced Configuration and Power Interface (ACPI)-The acpid event daemon monitors power management events and allows you to set up rules to react to those events. Those rules can do things such as turn off hardware components or power down the system. ACPI is meant to replace APM power management with more generic tools. At the moment, however, ACPI has not yet lived up to its potential.

  • hdparm command-The hdparm command can be used to turn power management features on and off. There are options to do things such as spin down your hard drive or put it in a standby state.

With ACPI, the operating system and not the BIOS is responsible for handling power management. Although ACPI will allow greater power and flexibility in managing power issues, APM currently has more user-level features built around it. With ACPI, you currently have to tell it yourself what power events to look for and how to respond to those events.

Note

If you want to have a visual representation of your battery power on your GNOME desktop, you can add a Battery Charge Monitor applet to your panel. Right-click on the panel, select Add to Panel ⇨ Utility ⇨ Battery Charge Monitor. By default, the applet will change colors from green (charged) to yellow (40%), orange (25%), and red (15%) to show battery levels. Figure 9-2 shows the Battery Charge Monitor icon for a computer currently running on AC power.

Click To expand
Figure 9-2: Watch your battery charge from an icon in the GNOME panel.

Managing Power with APM

APM is configured in Fedora to detect certain power events and respond to them. It does this by running a daemon process (/usr/sbin/apmd) that listens for events, and in turn responds to those events, based on settings in a configuration file that comes with the service (/etc/sysconfig/apmd).

To configure APM to run (which it already may be, but it won't hurt to do it again), type the following:

   # chkconfig apmd on
   # service apmd restart
Note

Both apmd and acpid should not be trying to control power services at the same time. So, turn off the service you don't want by changing on and restart to off and stop for that service (you'd turn off acpid in this case).

If APM starts successfully, you should see messages in the /var/log/messages file that show the service starting and listing the current battery charge. By default, the a pmd daemon runs based on settings in the /etc/sysconfig/apmd file. You can list the running process to see what options apmd is running by default:

# ps ax |  grep apmd
/usr/sbin/apmd -p 10 -w 5 -W -P /etc/sysconfig/apm-scripts/apmscript

Based on these settings, here is how the apmd daemon behaves in Fedora Core:

  • Logs battery power changes-If the battery power changes by 10% or more, a message is sent to the system log file (/var/log/messages by default). This based on the -p 10 option.

  • Logs low battery-When the battery power falls below 5% and is not being charged, a warning message is sent to the system log file. This is based on the -w 5 option.

  • Warns users-If battery power falls below the value set by the -w option, a warning message is sent to all users currently logged into the computer (using the wall command). This is based on the -W option.

  • Uses apmscript to respond to events-When power events occur, apmd forwards those events to a script that is set up to respond to those events. As shown in the command line above, the script that is run is the /etc/sysconfig/apm-scripts/apmscript shell script (based on that file being used as an option to - P).

Besides the low-battery and power change settings just described, most of the responses to power events are handled within the apmscript file just mentioned. I recommend you review the contents of that file if you are interested in seeing what power management is being done on your computer.

If you want to add your own responses to power events in Fedora Core, you can create your own script. By default, the existing apmscript points to the file /etc/sysconfig/apm-scripts/apmcontinue. The apmscript file currently handles the following events:

  • suspend-If your computer goes into a suspend or standby state, the apmscript file stops all sound programs, suspends operation of all hard drives, stops any NFS connections, shuts down network connections, and turns off pcmcia card services.

  • resume-In resume state, apmscript tries to restore everything that was stopped during the suspend actions.

  • change power-If the power source changes, the script changes the laptop to operate with better performance on AC power and more efficient battery use if the laptop is running off the battery.

  • change battery-If the power of battery is low, this event gives apm the opportunity to stop services and go into an extreme powersaving mode.

  • start-When the computer starts up, apmscript doesn't do anything special except checking whether you have any events you entered yourself to be run at startup in the apmcontinue file.

  • stop-Likewise, when the computer shuts down, apmscript just checks whether you have any events you entered yourself to be run at shutdown time in the apmcontinue file.

All other events are directed to the apmcontinue file (which doesn't exist by default), giving you the opportunity to respond to those events any way you choose. Here you have the option to create a/etc/sysconfig/apm-scripts/apmcontinue script that can be set up to respond to events in the same way as the apmscript file does.

If you want to just poke around or make any real-time changes to your APM service, you can use the apm command. With no options, the apm command reports whether power is plugged in and the status of your battery charging:

   # apm
   AC on-line, battery charging: 100%

You can also put your laptop in suspend mode (apm-s) or standby mode (apm-S). If you are running on AC power, you can tell the laptop to ignore any suspend and standby events generated by APM (apm-i).

Managing Power with ACPI

Like APM, ACPI operates by having a daemon listen for power events, and then pass those events to a program that can respond to them. In this case, the acpid daemon runs by starting the acpid service as follows:

   # chkconfig acpid on
   # service acpid restart
Note

Again, remember to turn off the power management service you don't want. In this case, turn off apmd (chkconfig apmd off and service apmd stop).

ACPI will only work on Linux systems that have ACPI enabled in the kernel. You may have to enable ACPI by force by adding ACPI=force as a boot option. You also need to enable ACPI in the BIOS. If ACPI is working, you should see a /proc/acpi/event file that contains ACPI events.

The acpid daemon monitors each event it receives from /proc/acpi/events and tries to match it against entries in files in the /etc/acpi/events directory. By default, only one file exists: sample.conf. The contents of that file show a simple form of an ACPI event and action:

   # This is a sample ACPID configuration
   event=button/power.*
   action=/sbin/shutdown -h now

In this example, when a button/power.* event is received, the shutdown command is run immediately (- h now). To configure your own actions to ACPI events, you can add your own files (owned by the root user) to the /etc/acpi/events directory that contains entries in the same format. (Type man acpid to see the exact format of these configuration files.) When the event you listed occurs, any command you listed on the action= line is run as root user.

To find the names of events, type cat/proc/acpi/event. To track acpid activities, you should view the contents of the /var/log/acpid file. Try changing power states a few times (unplug, plug in again, press suspend, and so on) and see how each of these events are logged into the log file.

Changing Power Settings with hdparm

If you are using a laptop computer, there are options to hdparm you can use to control power consumption. Some of these options allow you to take immediate actions (such as putting a hard drive into sleep mode), while others set actions to occur after a certain period of time (such as spinning down the hard disk after a period of inactivity).

To see the IDE power mode status for your computer, type:

   # hdparm -C /dev/hda
   /dev/hda:
   drive state is: active/idle

This shows the power mode set for normal operation. If the drive doesn't support this feature, the drive state is listed as unknown.

One of the most widely used ways of gaining power savings is to set the spindown mode (- S) to a low value. Some believe that even the minimum of a spindown after 5 seconds of idle time can improve overall battery life without drastically affecting performance. Spindown is set in 5-second increments using the -S option of hdparm and you can set the value from 0 (off) to 240 (20 minutes). So, for example, the following value of 3 tells the disk drive to go into standby mode (spindown) after 15 seconds:

   # hdparm -S3 /dev/hda
   /dev/hda:
   sotting standby to 3 (15 seconds)
Note

You can also set the disk to spindown at increments higher than 20 minutes. Values of -S from 41 t0 251 each signify 30-minute increments, so 241 sets the spindown value to 30 minutes, while 251 sets it to 5.5 hours.

You can send a hard disk immediately into standby mode using the -y option as follows:

   # hdparm -y /dev/hda
   /dev/hda:
   issuing standby command

Alternatively, you can enter sleep mode (the lowest power mode) using the -Y option:

   # hdparm -Y /dev/hda
   /dev/hda:
   issuing sleep command

The next time the hard drive is needed, Linux will automatically reset to start the disk drive again.


Disk, File System, and Power Troubleshooting References

Table 9-2 provides some references for troubleshooting your hard disks, file systems, and power management features.

Table 9-2: Troubleshooting References
File System Troubleshooting
NTFS File system in Linux | http://linux-ntfs.sourceforge.net/rpm/index.html
File systems HOWTO | www.tldp.org/HOWTO/Filesystems-HOWTO.html
The Software-RAID HOWTO | www.tldp.org/HOWTO/Software-RAID-HOWTO.html
Linux System Administrator's Guide, Chapter 6. Using Disks and Other Storage Media | www.tldp.org/LDP/sag/html/c973.html
Large Disk HOWTO | www.tldp.org/HOWTO/Large-Disk-HOWTO.html
Linux Partition HOWTO | www.tldp.org/HOWTO/Partition/index.html
Disk Performance Troubleshooting
Benchmarking disk I/O (bonnie) | www.tux.org/bench/Disk-IO/bonnie.tar.gz
PC Guide to Hard Disk Drives | www.pcguide.com/ref/hdd(Hard drive general reference)
Power Management Troubleshooting
Advanced Configuration and Power Interface (ACPI) HOWTO | www.tldp.org/HOWTO/ACPI-HOWTO/
ACPI Specifications | www.acpi.info
ACPI Project | Home page: http://acpi.sourceforge.net Documentation: http://acpi.sourceforge.net/documentation Mailing list: http://lists.sourceforge.net/lists/listinfo/acpi-devel
TuxMobil: Power | http://tuxmobil.org/apm_linux.html
Management with Linux | Good overview of Linux power management. The whole site is a great general resource for Linux laptop computers, including information on power management issues.


Summary

In the past few years, the ability of Linux to detect and configure hard disks has greatly improved. In most cases, you don't have to do anything to have Linux properly probe your hard disk during installation, choose good settings (transfer rates, read/write-caching, and so on), and even select and create usable partitioning.

However, if something goes wrong with your disk or file systems, it's nice to be able to understand what is going on and possibly even fix up a few things. Using tools such as hdparm, you can tune your hard disk behavior. With utilities such as fdisk and mkfs, you can create and modify your own partitions and file systems. With a little bit of care (and, hopefully, the security of good data backups), you can tune your disks and file systems to peak efficiency.


Part IV: Troubleshooting the Network

Chapter List

Chapter 10: Detecting and Responding to Intrusions
Chapter 11: Firewall Troubleshooting
Chapter 12: Troubleshooting BIND9 and DNS
Chapter 13: Modem TroubleshootIng


Chapter 10: Detecting and Responding to Intrusions

Highlights

In years past, the term intrusion detection had a general meaning: the methods by which an administrator learned about system intrusions, or about attempts to intrude, on a given system. As in most technological areas, intrusion detection has evolved and specialized. The security industry has grown to include a number of disciplines and subspecialties, each with its own cadre of professionals.

Why worry about intrusion detection if you have a good firewall? Just remember: no lock is unpickable. Firewalls have holes so that services can run (web, mail, and so on). Where there's a hole, there's a way. Furthermore, most security experts will tell you that security is not a destination, but a journey. Even if you have outstanding security policies in place, rock-hard firewalls, and a completely trustworthy user and administrator base, you still need to watch your system like a hawk to make sure that everything remains safe.

In this chapter, we explain the basic concepts of intrusion detection and response. We also show you how to use common Linux tools and shell scripts to maintain the security of your system, keep a watchful (and automated) eye on your system, and offer some solutions to common intrusion detection problems.


Intrusion Detection and Response: An Overview

Modern intrusion detection covers a wide range of systems, functions, and tools. Some of these tools simply detect, log, or report intrusion attempts. Others respond to such attempts proactively. It's easiest to understand security as a multilayered issue. Firewalls and network intrusion detection systems (NIDS) make up the outer shells of a comprehensive security solution. In this chapter, we take a host-centric view of security, and thus view these external forms of intrusion detection as the outer layer of defense.

Once past these outer layers, you'll find other security tools for the inner layers. These host-level ID tools, or HIDS, monitor local user, file, and log activity. Some HIDS are host-network-based, but many are not network-based at all. These programs watch for signs of user escalation violations, deviations of regular activity from baseline comparisons, and log or file monitoring, among other functions.

Note

Network-based intrusion detection tools watch for attack precursors (scans), attempts, and related network signatures. Generally, these tools are referred to as intrusion detection systems, or IDS. There are many subcategories of IDS, such as network-node-based IDS (NNIDS) and network-based IDS (NIDS), as well as host-based IDS (HIDS). In addition to these, you'll also find passive and reactive xlDS systems. Reactive systems can be tied into firewalls and are called intrusion prevention systems, or IPS. Learn more about these security tools at www.securityfocus.com/infocus/1733.

Even though it's common to think that the greatest threat to a system is external, remember that valid local user accounts should also be considered untrusted or potentially hostile. System attacks that succeed usually install a rootkit of some sort, software that compromises your system and leaves a back door daemon installed and running. Thus, to add to your layers of security, you should also consider local file-system-level forms of intrusion detection in the form of file alteration and system baseline scan comparisons, as well as system to watch for signs of trojans, worms, and new, related cracking tools. These defense tools can include both native and third-party tools or suites. Examples of these would include cracking and rootkit detection tools, automated systems that keep an eye on local user access, accounts, user and system files, and common security auditing tools (which can be maliciously used to serve the cracker's needs).

Note

This chapter is, by necessity, a mere introduction to the basics of host-based intrusion detection for Linux systems. We show you the basics of file alteration monitoring, a useful aspect of host-based intrusion detection, as well as offer pointers to other tools that may be useful in your installation. However, if you are responsible for security, take advantage of the many other books and resources on these topics. Don't take our word for it-do your own homework. You'll find a list of references and resources at the end of this chapter.