8SEpFk8z

1LINUX
2SERVER
3HACKS
4Rob Flickenger
5Beijing • Cambridge • Farnham • Köln • Paris • Sebastopol • Taipei • Tokyo
6Linux Server Hacks
7by Rob Flickenger
8Copyright © 2003 O’Reilly Media, Inc. All rights reserved.
9Printed in the United States of America.
10Publishedby O’ReillyMedia,Inc.,1005GravensteinHighwayNorth, Sebastopol,CA95472.
11O’ReillyMedia,Inc.booksmaybepurchasedforeducational,business,orsalespromotional
12use. Online editions are also available for most titles (safari.oreilly.com). For more
13information, contact our corporate/institutional sales department: (800) 998-9938 or
14corporate@oreilly.com.
15Editor: Dale Dougherty
16Series Editor: Rael Dornfest
17Executive Editor: Dale Dougherty
18Production Editor: Sarah Sherman
19Cover Designer: Edie Freedman
20Interior Designer: David Futato
21Printing History:
22January 2003: First Edition.
23Nutshell Handbook, the Nutshell Handboook logo, and the O’Reilly logo are registered
24trademarks of O’Reilly Media, Inc. The association between the image of an ax and the topic
25of Linux servers is a trademark of O’Reilly Media, Inc.
26The trademarks “Hacks Books” and “The Hacks Series,” and related trade dress, are owned
27by O’Reilly Media, Inc., in the United States and other countries, and may not be used
28without written permission.
29Many of the designations used by manufacturers and sellers to distinguish their products are
30claimed as trademarks. Where those designations appear in this book, and O’Reilly Media,
31Inc. was aware of a trademark claim, the designations have been printed in caps or initial
32caps. All other trademarks are property of their respective owners.
33While every precaution has been taken in the preparation of this book, the publisher and
34authorassumenoresponsibilityforerrorsoromissions,orfordamagesresultingfromtheuse
35of the information contained herein.
36ISBN13: 978-0-596-00461-3
37This book uses RepKover â„¢ , a durable and flexible lay-flat binding.
38[C] [4/08]
39D ownload from Wow! eBook <www.wowebook.com>
40iii
41Contents
42Credits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
43How to Become a Hacker . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
44Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
45Chapter 1. Server Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
461.Removing Unnecessary Services 3
472.Forgoing the Console Login 6
483.Common Boot Parameters 7
494.Creating a Persistent Daemon with init 9
505.n>&m: Swap Standard Output and Standard Error 10
516. Building Complex Command Lines 12
527. Working with Tricky Files in xargs 15
538. Immutable Files in ext2/ext3 17
549. Speeding Up Compiles 19
5510. At Home in Your Shell Environment 20
5611.Finding and Eliminating setuid/setgid Binaries 23
5712.Make sudo Work Harder 25
5813.Using a Makefile to Automate Admin Tasks 27
5914.Brute Forcing Your New Domain Name 29
6015.Playing Hunt the Disk Hog 30
6116.Fun with /proc 31
6217.Manipulating Processes Symbolically with procps 34
6318.Managing System Resources per Process 36
6419.Cleaning Up after Ex-Users 38
65iv | Contents
6620.Eliminating Unnecessary Drivers from the Kernel 40
6721.Using Large Amounts of RAM 42
6822.hdparm: Fine Tune IDE Drive Parameters 43
69Chapter 2. Revision Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
7023. Getting Started with RCS 48
7124. Checking Out a Previous Revision in RCS 49
7225. Tracking Changes with rcs2log 50
7326. Getting Started with CVS 52
7427. CVS: Checking Out a Module 54
7528. CVS: Updating Your Working Copy 55
7629. CVS: Using Tags 56
7730. CVS: Making Changes to a Module 57
7831. CVS: Merging Files 58
7932. CVS: Adding and Removing Files and Directories 58
8033. CVS: Branching Development 59
8134. CVS: Watching and Locking Files 60
8235. CVS: Keeping CVS Secure 60
8336. CVS: Anonymous Repositories 62
84Chapter 3. Backups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
8537.Backing Up with tar over ssh 65
8638.Using rsync over ssh 66
8739.Archiving with Pax 67
8840.Backing Up Your Boot Sector 72
8941.Keeping Parts of Filesystems in sync with rsync 74
9042.Automated Snapshot-Style Incremental Backups with rsync 79
9143.Working with ISOs and CDR/CDRWs 84
9244.Burning a CD Without Creating an ISO File 86
93Chapter 4. Networking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
9445.Creating a Firewall from the Command Line of any Server 88
9546.Simple IP Masquerading 91
9647.iptables Tips & Tricks 92
9748.Forwarding TCP Ports to Arbitrary Machines 94
9849.Using Custom Chains in iptables 96
99Contents | v
10050.Tunneling: IPIP Encapsulation 97
10151.Tunneling: GRE Encapsulation 99
10252.Using vtun over ssh to Circumvent NAT 101
10353. Automatic vtund.conf Generator 106
104Chapter 5. Monitoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
10554. Steering syslog 111
10655. Watching Jobs with watch 114
10756. What’s Holding That Port Open? 115
10857. Checking On Open Files and Sockets with lsof 116
10958. Monitor System Resources with top 119
11059. Constant Load Average Display in the Titlebar 120
11160. Network Monitoring with ngrep 121
11261. Scanning Your Own Machines with nmap 123
11362. Disk Age Analysis 125
11463. Cheap IP Takeover 127
11564. Running ntop for Real-Time Network Stats 129
11665. Monitoring Web Traffic in Real Time with httptop 132
117Chapter 6. SSH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
11866. Quick Logins with ssh Client Keys 139
11967. Turbo-mode ssh Logins 141
12068. Using ssh-Agent Effectively 142
12169. Running the ssh-Agent in a GUI 144
12270. X over ssh 145
12371. Forwarding Ports over ssh 146
124Chapter 7. Scripting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
12572. Get Settled in Quickly with movein.sh 149
12673. Global Search and Replace with Perl 151
12774. Mincing Your Data into Arbitrary Chunks (in bash) 153
12875. Colorized Log Analysis in Your Terminal 155
129Chapter 8. Information Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
13076. Running BIND in a chroot Jail 158
13177. Views in BIND 9 160
13278. Setting Up Caching DNS with Authority for Local Domains 165
133vi | Contents
13479.Distributing Server Load with Round-Robin DNS 167
13580. Running Your Own Top-Level Domain 168
13681. Monitoring MySQL Health with mtop 169
13782. Setting Up Replication in MySQL 172
13883. Restoring a Single Table from a Large MySQL Dump 175
13984. MySQL Server Tuning 175
14085. Using proftpd with a mysql Authentication Source 178
14186. Optimizing glibc, linuxthreads, and the Kernel
142for a Super MySQL Server 180
14387. Apache Toolbox 182
14488. Display the Full Filename in Indexes 185
14589. Quick Configuration Changes with IfDefine 186
14690. Simplistic Ad Referral Tracking 188
14791. Mimicking FTP Servers with Apache 191
14892. Rotate and compress Apache Server Logs 193
14993. Generating an SSL cert and Certificate Signing Request 194
15094. Creating Your Own CA 196
15195. Distributing Your CA to Client Browsers 199
15296. Serving multiple sites with the same DocumentRoot 201
15397. Delivering Content Based on the Query String
154Using mod_rewrite 203
15598. Using mod_proxy on Apache for Speed 204
15699. Distributing Load with Apache RewriteMap 206
157100. Ultrahosting: Mass Web Site Hosting
158with Wildcards, Proxy, and Rewrite 208
159Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
160D ownload from Wow! eBook <www.wowebook.com>
161vii
1620
163Credits
164About the Author
165Rob Flickenger authored the majority of hacks in this book. Rob has worked
166with Linux since Slackware 3.5. He was previously the system administrator
167of the O’Reilly Network (an all-Linux shop, naturally) and is the author of
168Building Wireless Community Networks, also by O’Reilly.
169Contributors
170The following people contributed their hacks, writing, and inspiration to
171this book:
172• Rael Dornfest (“Apache Toolbox” [Hack #75] ) is a maven at the O’Reilly &
173Associates focusing on technologies just beyond the pale. He assesses,
174experiments, programs, and writes for the O’Reilly network and
175O’Reilly publications.
176• Schuyler Erle (contributed code for httptop, mysql-table-restore, bal-
177ance-push, find-whois, and vtundgen) is, by day, a mild-mannered
178Internet systems developer for O’Reilly & Associates. By night, he cru-
179sades for justice and freedom as a free software hacker and community
180networking activist.
181• Kevin Hemenway (“Quick Configuration Changes with IfDefine” [Hack
182#75] , “Simplistic Ad Referral Tracking” [Hack #75] , “Mimicking FTP Serv-
183ers with Apache” [Hack #75] ), better known as Morbus Iff, is the creator of
184disobey.com, which bills itself as “content for the discontented.” Pub-
185lisher and developer of more home cooking than you could ever imag-
186ine, he’d love to give you a Fry Pan of Intellect upside the head. Politely,
187of course. And with love.
188viii | Credits
189• Seann Herdejurgen (“iptables Tips & Tricks” [Hack #44] , “Disk Age Anal-
190ysis” [Hack #53] ) has been working with Unix since 1987 and now archi-
191tects high availability solutions as a senior systems engineer with D-
192Tech corporation in Dallas, Texas. He holds a master’s degree in com-
193puter science from Texas A&M University. He may be reached at:
194http://seann.herdejurgen.com/.
195• Dru Lavigne (“Archiving with Pax” [Hack #36] ) is an instructor at a private
196technical college in Kingston, ON where she teaches the fundamentals
197of TCP/IP networking, routing, and security. Her current hobbies
198include matching every TCP and UDP port number to its associated
199application(s) and reading her way through all of the RFCs.
200• Cricket Liu (“Views in BIND 9” [Hack #75] ) matriculated at the University
201of California’s Berkeley campus, that great bastion of free speech, unen-
202cumbered Unix, and cheap pizza. He worked for a year as Director of
203DNS Product Management for VeriSign Global Registry Services, and is
204a co-author of DNS and BIND, also published by O’Reilly & Associates.
205• Mike Rubel (“Automated Snapshot-Style Incremental Backups with
206rsync” [Hack #36] , http://www.mikerubel.org) studied mechanical engineer-
207ing at Rutgers University (B.S. 1998) and aeronautics at Caltech (M.S.
2081999), where he is now a graduate student. He has enjoyed using Linux
209and GNU software for several years in the course of his numerical meth-
210ods research.
211• Jennifer Vesperman (all of the CVS pieces except “CVS: Anonymous
212Repositories” [Hack #22] were adapted from her online CVS pieces for
213O’ReillyNet) contributes to open source as a user, writer, and occasional
214programmer. Her coding experience ranges from the hardware interface
215to an HDLC card to the human interface of Java GUIs. Jenn is the cur-
216rent coordinator and co-sysadmin for Linuxchix.org.
217Acknowledgments
218I would like to thank my family and friends for their support and encourage-
219ment. Thanks especially to my dad for showing me “proper troubleshooting
220technique” at such an early age and inevitably setting me on the path of the
221Hacker before I had even seen a computer.
222Of course, this book would be nothing without the excellent contributions
223of all the phenomenally talented hackers contained herein. But of course,
224our hacks are built by hacking the shoulders of giants (to horribly mix a
225metaphor), and it is my sincere hope that you will in turn take what you
226learn here and go one better, and most importantly, tell everyone just how
227you did it.
228ix
229F O R E W O R D
230How to Become a Hacker
231The Jargon File contains a bunch of definitions of the term “hacker,” most
232having to do with technical adeptness and a delight in solving problems and
233overcoming limits. If you want to know how to become a hacker, though,
234only two are really relevant.
235There is a community, a shared culture, of expert programmers and net-
236working wizards that traces its history back through decades to the first
237time-sharing minicomputers and the earliest ARPAnet experiments. The
238members of this culture originated the term “hacker.” Hackers built the
239Internet. Hackers made the Unix operating system what it is today. Hackers
240run Usenet. Hackers make the Web work. If you are part of this culture, if
241you have contributed to it and other people in it know who you are and call
242you a hacker, you’re a hacker.
243The hacker mind-set is not confined to this software-hacker culture. There
244are people who apply the hacker attitude to other things, like electronics or
245music—actually, you can find it at the highest levels of any science or art.
246Software hackers recognize these kindred spirits elsewhere and may call
247them “hackers” too—and some claim that the hacker nature is really inde-
248pendent of the particular medium the hacker works in. But in the rest of this
249document, we will focus on the skills and attitudes of software hackers, and
250the traditions of the shared culture that originated the term “hacker.”
251There is another group of people who loudly call themselves hackers, but
252aren’t. These are people (mainly adolescent males) who get a kick out of
253breaking into computers and breaking the phone system. Real hackers call
254these people “crackers” and want nothing to do with them. Real hackers
255mostly think crackers are lazy, irresponsible, and not very bright—being
256able to break security doesn’t make you a hacker any more than being able
257to hotwire cars makes you an automotive engineer. Unfortunately, many
258journalists and writers have been fooled into using the word “hacker” to
259describe crackers; this irritates real hackers no end.
260x | How to Become a Hacker
261The basic difference is this: hackers build things, crackers break them.
262If you want to be a hacker, keep reading. If you want to be a cracker, go read
263the alt.2600 newsgroup and get ready to do five to ten in the slammer after
264finding out you aren’t as smart as you think you are. And that’s all I’m going
265to say about crackers.
266The Hacker Attitude
267Hackers solve problems and build things, and they believe in freedom and
268voluntary mutual help. To be accepted as a hacker, you have to behave as
269though you have this kind of attitude yourself. And to behave as though you
270have the attitude, you have to really believe the attitude.
271But if you think of cultivating hacker attitudes as just a way to gain accep-
272tance in the culture, you’ll miss the point. Becoming the kind of person who
273believes these things is important for you—for helping you learn and keep-
274ing you motivated. As with all creative arts, the most effective way to
275become a master is to imitate the mind-set of masters—not just intellectu-
276ally but emotionally as well.
277Or, as the following modern Zen poem has it:
278To follow the path:
279look to the master,
280follow the master,
281walk with the master,
282see through the master,
283become the master.
284So if you want to be a hacker, repeat the following things until you believe
285them.
2861. The world is full of fascinating problems waiting to be
287solved.
288Being a hacker is a lot of fun, but it’s a kind of fun that takes a lot of effort.
289The effort takes motivation. Successful athletes get their motivation from a
290kind of physical delight in making their bodies perform, pushing themselves
291past their own physical limits. Similarly, to be a hacker you have to get a
292basic thrill from solving problems, sharpening your skills, and exercising
293your intelligence.
294If you aren’t the kind of person that feels this way naturally, you’ll need to
295become one in order to make it as a hacker. Otherwise you’ll find your hack-
296ing energy is zapped by distractions like sex, money, and social approval.
297How to Become a Hacker | xi
298(You also have to develop a kind of faith in your own learning capacity—a
299belief that even though you may not know all of what you need to solve a
300problem, if you tackle just a piece of it and learn from that, you’ll learn
301enough to solve the next piece—and so on, until you’re done.)
3022. No problem should ever have to be solved twice.
303Creative brains are a valuable, limited resource. They shouldn’t be wasted
304on re-inventing the wheel when there are so many fascinating new problems
305waiting out there.
306To behave like a hacker, you have to believe that the thinking time of other
307hackers is precious—so much so that it’s almost a moral duty for you to
308share information, solve problems, and then give the solutions away just so
309other hackers can solve new problems instead of having to perpetually re-
310address old ones.
311(You don’t have to believe that you’re obligated to give all your creative
312product away, though the hackers that do are the ones that get most respect
313from other hackers. It’s consistent with hacker values to sell enough of it to
314keep you in food and rent and computers. It’s fine to use your hacking skills
315to support a family or even get rich, as long as you don’t forget your loyalty
316to your art and your fellow hackers while doing it.)
3173. Boredom and drudgery are evil.
318Hackers (and creative people in general) should never be bored or have to
319drudge at stupid repetitive work, because when this happens it means they
320aren’t doing what only they can do—solve new problems. This wastefulness
321hurts everybody. Therefore boredom and drudgery are not just unpleasant
322but actually evil.
323To behave like a hacker, you have to believe this enough to want to auto-
324mate away the boring bits as much as possible, not just for yourself but for
325everybody else (especially other hackers).
326(There is one apparent exception to this. Hackers will sometimes do things
327that may seem repetitive or boring to an observer as a mind-clearing exer-
328cise, or in order to acquire a skill or have some particular kind of experience
329you can’t have otherwise. But this is by choice—nobody who can think
330should ever be forced into a situation that bores them.)
3314. Freedom is good.
332Hackers are naturally anti-authoritarian. Anyone who can give you orders
333can stop you from solving whatever problem you’re being fascinated
334xii | How to Become a Hacker
335by—and, given the way authoritarian minds work, will generally find
336some appallingly stupid reason to do so. So the authoritarian attitude has
337to be fought wherever you find it, lest it smother you and other hackers.
338(This isn’t the same as fighting all authority. Children need to be guided and
339criminals restrained. A hacker may agree to accept some kinds of authority
340in order to get something he wants more than the time he spends following
341orders. But that’s a limited, conscious bargain; the kind of personal surren-
342der authoritarians want is not on offer.)
343Authoritarians thrive on censorship and secrecy. And they distrust volun-
344tary cooperation and information-sharing—they only like “cooperation”
345that they control. So to behave like a hacker, you have to develop an instinc-
346tive hostility to censorship, secrecy, and the use of force or deception to
347compel responsible adults. And you have to be willing to act on that belief.
3485. Attitude is no substitute for competence.
349To be a hacker, you have to develop some of these attitudes. But copping an
350attitude alone won’t make you a hacker, any more than it will make you a
351champion athlete or a rock star. Becoming a hacker will take intelligence,
352practice, dedication, and hard work.
353Therefore, you have to learn to distrust attitude and respect competence of
354every kind. Hackers won’t let posers waste their time, but they worship
355competence—especially competence at hacking, but competence at any-
356thing is good. Competence at demanding skills that few can master is espe-
357cially good, and competence at demanding skills that involve mental
358acuteness, craft, and concentration is best.
359If you revere competence, you’ll enjoy developing it in yourself—the hard
360work and dedication will become a kind of intense play rather than drudg-
361ery. That attitude is vital to becoming a hacker.
362The complete essay can be found online at http://www.catb.org/~esr/faqs/
363hacker-howto.html and in an appendix to the “The Cathedral and the
364Bazaar" book (O’Reilly.)
365—Eric S. Raymond
366Eric S. Raymond is the author of the New Hacker’s Dictionary, based on the Jargon
367File, and the famous “Cathedral and the Bazaar” essay that served as a catalyst for
368the Open Source movement. The text in this Foreword is an excerpt from his 1996
369essay, “What is a hacker?” Raymond argues that hackers are ingenious at solving
370interesting problems, an idea that is the cornerstone of O’Reilly’s Hacks series.
371xiii
372Preface
373A hacker does for love what others
374would not do for money.
375—/usr/games/fortune
376The word hack has many connotations. A “good hack” makes the best of the
377situation of the moment, using whatever resources are at hand. An “ugly
378hack” approaches the situation in the most obscure and least understand-
379able way, although many “good hacks” may also appear unintelligible to the
380uninitiated.
381The effectiveness of a hack is generally measured by its ability to solve a par-
382ticular technical problem, inversely proportional to the amount of human
383effort involved in getting the hack running. Some hacks are scalable and
384some are even sustainable. The longest running and most generally accepted
385hacks become standards and cause many more hacks to be invented. A good
386hack lasts until a better hack comes along.
387A hack reveals the interface between the abstract and wonderfully complex
388mind of the designer, and the indisputable and vulgar experience of human
389needs. Sometimes, hacks may be ugly and only exist because someone had an
390itch that needed scratching. To the engineer, a hack is the ultimate expression
391of the Do-It-Yourself sentiment: no one understands how a hack came to be
392better than the person who felt compelled to solve the problem in the first
393place. If a person with a bent for problem solving thinks a given hack is ugly,
394then they are almost always irresistibly motivated to go one better—and hack
395the hack, something that we encourage the readers of this book to do.
396In the end, even the most capable server, with the most RAM and running
397the fastest (and most free) operating system on the planet, is still just a fancy
398back-scratcher fixing the itch of the moment, until a better, faster and
399cheaper back-scratcher is required.
400D ownload from Wow! eBook <www.wowebook.com>
401xiv | Preface
402Where does all of this pseudo-philosophical rambling get you? Hopefully,
403this background will give you some idea of the mindset that prompted the
404compiling of this collection of solutions that we call Linux Server Hacks.
405Some are short and simple, while some are quite complex. All of these hacks
406are designed to solve a particular technical problem that the designer simply
407couldn’t let go without “scratching.” I hope that some of them will be
408directly applicable to an “itch” or two that you may have felt yourself as a
409new or experienced administrator of Linux servers.
410How This Book Is Organized
411A competent sysadmin must be a jack-of-all-trades. To be truly effective,
412you’ll need to be able to handle every problem the system throws at you,
413from power on to halt. To assist you in the time in between, I present this
414collection of time-saving and novel approaches to daily administrative tasks.
415• Server Basics begins by looking at some of the most common sorts of
416tasks that admins encounter: manipulating the boot process, effectively
417working with the command line, automating common tasks, watching
418(and regulating) how system resources are used, and tuning various
419pieces of the Linux kernel to make everything run more efficiently. This
420isn’t an introduction to system administration but a look at some very
421effective and non-obvious techniques that even seasoned sysadmins may
422have overlooked.
423• Revision Control gives a crash-course in using two fundamental revision
424control systems, RCS and CVS. Being able to recall arbitrary previous
425revisions of configuration files, source code, and documentation is a
426critical ability that can save your job. Too many professional admins are
427lacking in revision control basics (preferring instead to make the inevita-
428ble, but unsupportable .old or .orig backup). This section will get you
429up and running quickly, giving you commands and instructions that are
430succinct and to the point.
431• The next section, Backups, looks at quick and easy methods for keeping
432spare copies of your data. I pay particular attention to network back-
433ups, rsync, and working with ISOs. I’ll demonstrate some of the enor-
434mous flexibility of standard system backup tools and even present one
435way of implementing regular “snapshot” revisions of a filesystem (with-
436out requiring huge amounts of storage).
437• Networking is my favorite section of this entire book. The focus isn’t on
438basic functionality and routing, but instead looks at some obscure but
439insanely useful techniques for making networks behave in unexpected
440ways. I’ll set up various kinds of IP tunnels (both encrypted and
441Preface | xv
442otherwise), work with NAT, and show some advanced features that
443allow for interesting behavior based on all kinds of parameters. Did you
444ever want to decide what to do with a packet based on its data con-
445tents? Take a look at this section.
446•Monitoring is an eclectic mix of tips and tools for finding out exactly
447what your server is up to. It looks at some standard (and some abso-
448lutely required “optional”) packages that will tell you volumes about
449who is using what, when, and how on your network. It also looks at a
450couple of ways to mitigate inevitable service failures and even help
451detect when naughty people attempt to do not-so-nice things to your
452network.
453•Truly a font of hackery unto itself, the SSH section describes all sorts of
454nifty uses for ssh, the cryptographically strong (and wonderfully flexi-
455ble) networking tool. There are a couple of versions of ssh available for
456Linux, and while many of the examples will work in all versions, they
457are all tested and known to work with OpenSSH v3.4p1.
458•Scripting provides a short digression by looking at a couple of odds and
459ends that simply couldn’t fit on a single command line. These hacks will
460save you time and will hopefully serve as examples of how to do some
461nifty things in shell and Perl.
462•Information Services presents three major applications for Linux: BIND 9,
463MySQL, and Apache. This section assumes that you’re well beyond basic
464installation of these packages, and are looking for ways to make them
465deliver their services faster and more efficiently, without having to do a lot
466of work yourself. You will see methods for getting your server running
467quickly, helping it scale to very large installations and behave in all sorts
468of slick ways that save a lot of configuration and maintenance time.
469How to Use This Book
470You may find it useful to read this book from cover to cover, as the hacks do
471build on each other a bit from beginning to end. However, each hack is
472designed to stand on its own as a particular example of one way to accom-
473plish a particular task. To that end, I have grouped together hacks that fit a
474particular theme into sections, but I do cross-reference quite a bit between
475hacks from different sections (and also to more definitive resources on the
476subject). Don’t consider a given section as a cut-and-dried chapter with rig-
477idly defined subject boundaries but more as a convenient way of collecting
478similar (and yet independent) hacks. You may want to read this book much
479like the way most people browse web pages online: follow whatever inter-
480ests you, and if you get lost, follow the links within the piece to find more
481information.
482xvi | Preface
483Conventions Used in This Book
484The following is a list of the typographical conventions used in this book:
485Italic
486Used to indicate new terms, URLs, filenames, file extensions, directo-
487ries, commands and options, and program names.
488Constant Width
489Used to show code examples, the contents of files, or the output from
490commands.
491Constant Width Bold
492Used in examples and tables to show commands or other text that
493should be typed literally.
494Constant Width Italic
495Used in examples and tables to show text that should be replaced with
496user-supplied values.
497The thermometer icons, found next to each hack, indicate the relative com-
498plexity of the hack:
499How to Contact Us
500We have tested and verified the information in this book to the best of our
501ability, but you may find that features have changed (or even that we have
502made mistakes!). Please let us know about any errors, inaccuracies, bugs, mis-
503leading or confusing statements, and typos that you find in this book.
504You can write to us at:
505O’Reilly & Associates, Inc.
5061005 Gravenstein Hwy N.
507Sebastopol, CA 95472
508(800) 998-9938 (in the U.S. or Canada)
509(707) 829-0515 (international/local)
510(707) 829-0104 (fax)
511To ask technical questions or to comment on the book, send email to:
512bookquestions@oreilly.com
513Visit the web page for Linux Server Hacks to find additional support infor-
514mation, including examples and errata. You can find this page at:
515http://www.oreilly.com/catalog/linuxsvrhack
516beginner moderate expert
517Preface | xvii
518For more information about this book and others, see the O’Reilly web site:
519http://www.oreilly.com
520Gotta Hack?
521Got a good hack you’d like to share with others? Go to the O’Reilly Hacks
522web site at:
523http://hacks.oreilly.com
524You’ll find book-related resources, sample hacks and new hacks contrib-
525uted by users. You’ll find information about additional books in the Hacks
526series.
527D ownload from Wow! eBook <www.wowebook.com>
5281
529Chapter 1
530C H A P T E R O N E
531Server Basics
532Hacks #1–22
533A running Linux system is a complex interaction of hardware and software
534where invisible daemons do the user’s bidding, carrying out arcane tasks to
535the beat of the drum of the uncompromising task master called the Linux
536kernel.
537A Linux system can be configured to perform many different kinds of tasks.
538When running as a desktop machine, the visible portion of Linux spends
539much of its time controlling a graphical display, painting windows on the
540screen, and responding to the user’s every gesture and command. It must
541generally be a very flexible (and entertaining) system, where good respon-
542siveness and interactivity are the critical goals.
543On the other hand, a Linux server generally is designed to perform a couple
544of tasks, nearly always involving the squeezing of information down a net-
545work connection as quickly as possible. While pretty screen savers and GUI
546features may be critical to a successful desktop system, the successful Linux
547server is a high performance appliance that provides access to information as
548quickly and efficiently as possible. It pulls that information from some sort
549of storage (like the filesystem, a database, or somewhere else on the net-
550work) and delivers that information over the network to whomever
551requested it, be it a human being connected to a web server, a user sitting in
552a shell, or over a port to another server entirely.
553It is under these circumstances that a system administrator finds their
554responsibilities lying somewhere between deity and janitor. Ultimately, the
555sysadmin’s job is to provide access to system resources as quickly (and equi-
556tably) as possible. This job involves both the ability to design new systems
557(that may or may not be rooted in solutions that already exist) and the tal-
558ent (and the stomach) for cleaning up after people who use that system
559without any concept of what “resource management” really means.
5602 | Server Basics
561The most successful sysadmins remove themselves from the path of access
562to system resources and let the machines do all of the work. As a user, you
563know that your sysadmin is effective when you have the tools that you need
564to get the job done and you never need to ask your sysadmin for anything.
565To pull off (that is, to hack) this impossible sounding task requires that the
566sysadmin anticipate what the users’ needs will be and make efficient use of
567the resources that are available.
568To begin with, I’ll present ways to optimize Linux to perform only the work
569that is required to get the job done and not waste cycles doing work that
570you’re not interested in doing. You’ll see some examples of how to get the
571system to do more of the work of maintaining itself and how to make use of
572some of the more obscure features of the system to make your job easier.
573Parts of this section (particularly Command Line and Resource Manage-
574ment) include techniques that you may find yourself using every day to help
575build a picture of how people are using your system and ways that you
576might improve it.
577These hacks assume that you are already familiar with Linux. In particular,
578you should already have root on a running Linux system available with
579which to experiment and should be comfortable with working on the sys-
580tem from the command line. You should also have a good working knowl-
581edge of networks and standard network services. While I hope that you will
582find these hacks informative, they are certainly not a good introduction to
583Linux system administration. For in-depth discussion on good administra-
584tive techniques, I highly recommend the Linux Network Administrator’s
585Guide and Essential System Administration, both by O’Reilly and Associates.
586The hacks in this chapter are grouped together into the following five cate-
587gories: Boot Time, Command Line, Automation, Resource Management,
588and Kernel Tuning.
589Boot Time
5901. Fine tune your server to provide only the services you really want to
591serve
5922. Forgoing the Console Login
5933. Common Boot Parameters
5944. Creating a Persistent Daemon with init
595Command Line
5965. n>&m: Swap Standard Output and Standard Error
5976. Building Complex Command Lines
5987. Working with Tricky Files in xargs
599Removing Unnecessary Services #1
600Server Basics | 3
601HACK
6028. Immutable Files in ext2/ext3
6039. Speeding Up Compiles
604Automation
60510. At home in your shell environments
60611. Finding and eliminating setuid/setgid binaries
60712. Make sudo work harder for you
60813. Using a Makefile to automate admin tasks
60914. Brute forcing your new domain name
610Resource Management
61115. Playing Hunt the Disk Hog
61216. Fun with /proc
61317. Manipulating processes symbolically with procps
61418. Managing system resources per process
61519. Cleaning up after ex-users
616Kernel Tuning
61720. Eliminating unnecessary drivers from the kernel
61821. Using large amounts of RAM
61922. hdparm: fine tune IDE drive parameters
620H A C K
621#1
622Removing Unnecessary Services Hack #1
623Fine tune your server to provide only the services you really want to serve
624When you build a server, you are creating a system that should perform its
625intended function as quickly and efficiently as possible. Just as a paint mixer
626has no real business being included as an espresso machine attachment,
627extraneous services can take up resources and, in some cases, cause a real
628mess that is completely unrelated to what you wanted the server to do in the
629first place. This is not to say that Linux is incapable of serving as both a top-
630notch paint mixer and making a good cup of coffee simultaneously—just be
631sure that this is exactly what you intend before turning your server loose on
632the world (or rather, turning the world loose on your server).
633When building a server, you should continually ask yourself: what do I
634really need this machine to do? Do I really need FTP services on my web
635server? Should NFS be running on my DNS server, even if no shares are
636exported? Do I need the automounter to run if I mount all of my volumes
637statically?
6384 | Server Basics
639#1 Removing Unnecessary Services
640HACK
641To get an idea of what your server is up to, simply run a ps ax. If nobody is
642logged in, this will generally tell you what your server is currently running.
643You should also see what programs for which your inetd is accepting con-
644nections, with either a grep -v ^# /etc/inetd.conf or (more to the point)
645netstat -lp . The first command will show all uncommented lines in your
646inetd.conf, while the second (when run as root) will show all of the sockets
647that are in the LISTEN state, and the programs that are listening on each
648port. Ideally, you should be able to reduce the output of a ps ax to a page of
649information or less (barring preforking servers like httpd, of course).
650Here are some notorious (and typically unnecessary) services that are
651enabled by default in many distributions:
652portmap, rpc.mountd, rpc.nfsd
653These are all part of the NFS subsystem. Are you running an NFS
654server? Do you need to mount remote NFS shares? Unless you answered
655yes to either of these questions, you don’t need these daemons running.
656Reclaim the resources that they’re taking up and eliminate the potential
657security risk.
658smbd and nmbd
659These are the Samba daemons. Do you need to export SMB shares to
660Windows boxes (or other machines)? If not, then these processes can be
661safely killed.
662automount
663The automounter can be handy to bring up network (or local) filesys-
664tems on demand, eliminating the need for root privileges when access-
665ing them. This is especially handy on client desktop machines, where a
666user needs to use removable media (such as CDs or floppies) or to
667access network resources. But on a dedicated server, the automounter is
668probably unnecessary. Unless your machine is providing console access
669or remote network shares, you can kill the automounter (and set up all
670of your mounts statically, in /etc/fstab).
671named
672Are you running a name server? You don’t need named running if you
673only need to resolve network names; that’s what /etc/resolv.conf and the
674bind libraries are for. Unless you’re running name services for other
675machines, or are running a caching DNS server (see “Setting Up Caching
676DNS with Authority for Local Domains” [Hack #78] ), then named isn’t
677needed.
678Removing Unnecessary Services #1
679Server Basics | 5
680HACK
681lpd
682Do you ever print to this machine? Chances are, if it’s serving Internet
683resources, it shouldn’t be accepting print requests anyway. Remove the
684printer daemon if you aren’t planning on using it.
685inetd
686Do you really need to run any services from inetd? If you have ssh run-
687ning in standalone mode, and are only running standalone daemons
688(such as Apache, BIND, MySQL, or ProFTPD) then inetd may be
689superfluous. In the very least, review which services are being accepted
690with the grep command grep -v ^# /etc/inetd.conf . If you find that
691every service can be safely commented out, then why run the daemon?
692Remove it from the boot process (either by removing it from the sys-
693tem rc’s or with a simple chmod -x /usr/sbin/inetd ).
694telnet, rlogin, rexec, ftp
695The remote login, execution, and file transfer functionality of these vener-
696able daemons has largely been supplanted by ssh and scp, their crypto-
697graphically secure and tremendously flexible counterparts. Unless you
698have a really good reason to keep them around, it’s a good idea to elimi-
699nate support for these on your system. If you really need to support ftp
700connections, you might try the mod_sql plugin for proftpd (see “Using
701proftpd with a mysql Authentication Source” [Hack #85] ).
702finger, comsat, chargen, echo, identd
703The finger and comsat services made sense in the days of an open Inter-
704net, where users were curious but generally well-intentioned. In these
705days of stealth portscans and remote buffer overflow exploits, running
706extraneous services that give away information about your server is gen-
707erally considered a bad idea. The chargen and echo ports were once
708good for testing network connectivity, but are now too inviting for a
709random miscreant to fiddle with (and perhaps connect to each other to
710drive up server load quickly and inexpensively).
711Finally, the identd service was once a meaningful and important source of
712information, providing remote servers with an idea of which users were
713connecting to their machines. Unfortunately, in these days of local root
714exploits and desktop Linux machines, installing an identd that (perish the
715thought!) actually lies about who is connected has become so common
716that most sites ignore the author information anyway. Since identd is a
717notoriously shaky source of information, why leave it enabled at all?
718To eliminate unnecessary services, first shut them down (either by running
719service stop in /etc/rc.d/init.d/, removing them from /etc/inetd.conf, or by
720killing them manually). Then to be sure that they don’t start again the next
7216 | Server Basics
722#2 Forgoing the Console Login
723HACK
724time the machine reboots, remove their entry from /etc/rc.d/*. Once you have
725your system trimmed down to only the services you intend to serve, reboot
726the machine and check the process table again.
727If you absolutely need to run insecure services on your machine, then you
728should use tcp wrappers or local firewalling to limit access to only the
729machines that absolutely need it.
730See also:
731• “Creating a Firewall from the Command Line of any Server” [Hack #45]
732• “Setting Up Caching DNS with Authority for Local Domains” [Hack #78]
733• “Using proftpd with a mysql Authentication Source” [Hack #85]
734H A C K
735#2
736Forgoing the Console Login Hack #2
737All of the access, none of the passwords
738It will happen to you one day. You’ll need to work on a machine for a friend
739or client who has “misplaced” the root password on which you don’t have
740an account.
741If you have console access and don’t mind rebooting, traditional wisdom
742beckons you to boot up in single user mode. Naturally, after hitting Control-
743Alt-Delete, you simply wait for it to POST and then pass the parameter
744single to the booting kernel. For example, from the LILO prompt:
745LILO:linux single
746On many systems, this will happily present you with a root shell. But on
747some systems (notably RedHat), you’ll run into the dreaded emergency
748prompt:
749Give root password for maintenance
750(or type Control-D for normal startup)
751If you knew the root password, you wouldn’t be here! If you’re lucky, the
752init script will actually let you hit ^C at this stage and will drop you to a
753root prompt. But most init processes are “smarter” than that, and trap ^C.
754What to do? Of course, you could always boot from a rescue disk and reset
755the password, but suppose you don’t have one handy (or that the machine
756doesn’t have a CD-ROM drive).
757All is not lost! Rather than risk running into the above mess, let’s modify the
758system with extreme prejudice, right from the start. Again, from the LILO
759prompt:
760LILO:linux init=/bin/bash
761Common Boot Parameters #3
762Server Basics | 7
763HACK
764What does this do? Rather than start /sbin/init and proceed with the usual /etc/
765rc.d/* procedure, we’re telling the kernel to simply give us a shell. No pass-
766words, no filesystem checks (and for that matter, not much of a starting envi-
767ronment!) but a very quick, shiny new root prompt.
768Unfortunately, that’s not quite enough to be able to repair your system. The
769root filesystem will be mounted read-only (since it never got a chance to be
770checked and remounted read/write). Also, networking will be down, and
771none of the usual system daemons will be running. You don’t want to do
772anything more complicated than resetting a password (or tweaking a file or
773two) at a prompt like this. Above all: don’t hit ^D or type Exit! Your little
774shell (plus the kernel) constitutes the entire running Linux system at the
775moment. So, how can you manipulate the filesystem in this situation, if it is
776mounted read-only? Try this:
777#mount -o remount,rw /
778That will force the root filesystem to be remounted read-write. You can now
779type passwd to change the root password (and if the original admin lost the
780password, consider the ramifications of giving them access to the new one. If
781you were the original admin, consider writing it in invisible ink on a post-it
782note and sticking it to your screen, or stitching it into your underwear, or
783maybe even taking up another hobby).
784Once the password is reset, DO NOT REBOOT. Since there is no init run-
785ning, there is no process in place for safely taking the system down. The
786quickest way to shutdown safely is to remount root again:
787#mount -o remount,ro /
788With the root partition readonly, you can confidently hit the Reset button,
789bring it up in single-user mode, and begin your actual work.
790H A C K
791#3
792Common Boot Parameters Hack #3
793Manipulate kernel parameters at boot time
794As we saw in “Forgoing the Console Login” [Hack #2] , it is possible to pass
795parameters to the kernel at the LILO prompt allowing you to change the
796program that is first called when the system boots. Changing init (with the
797init=/bin/bash line) is just one of many useful options that can be set at
798boot time. Here are more common boot parameters:
799single
800Boots up in single user mode.
801root=
802Changes the device that is mounted as /. For example:
803root=/dev/sdc4
8048 | Server Basics
805#3 Common Boot Parameters
806HACK
807will boot from the fourth partition on the third scsi disk (instead of
808whatever your boot loader has defined as the default).
809hdX=
810Adjusts IDE drive geometry. This is useful if your BIOS reports incor-
811rect information:
812hda=3649,255,63 hdd=cdrom
813This defines the master/primary IDE drive as a 30GB hard drive in LBA
814mode, and the slave/secondary IDE drive as a CD-ROM.
815console=
816Defines a serial port console on kernels with serial console support. For
817example:
818console=ttyS0,19200n81
819Here we’re directing the kernel to log boot messages to ttyS0 (the first
820serial port), at 19200 baud, no parity, 8 data bits, 1 stop bit. Note that
821to get an actual serial console (that you can log in on), you’ll need to
822add a line to /etc/inittab that looks something like this:
823s1:12345:respawn:/sbin/agetty 19200 ttyS0 vt100
824nosmp
825Disables SMP on a kernel so enabled. This can help if you suspect ker-
826nel trouble on a multiprocessor system.
827mem=
828Defines the total amount of available system memory. See “Using Large
829Amounts of RAM” [Hack #21] .
830ro
831Mounts the / partition read-only (this is typically the default, and is
832remounted read-write after fsck runs).
833rw
834Mounts the / partition read-write. This is generally a bad idea, unless
835you’re also running the init hack. Pass your init line along with rw, like
836this:
837init=/bin/bash rw
838to eliminate the need for all of that silly mount -o remount,rw / stuff in
839“Forgoing the Console Login” [Hack #2] . Congratulations, now you’ve
840hacked a hack.
841You can also pass parameters for SCSI controllers, IDE devices, sound cards,
842and just about any other device driver. Every driver is different, and typi-
843cally allows for setting IRQs, base addresses, parity, speeds, options for
844auto-probing, and more. Consult your online documentation for the excru-
845ciating details.
846Creating a Persistent Daemon with init #4
847Server Basics | 9
848HACK
849See also:
850• man bootparam
851• /usr/src/linux/Documentation/*
852H A C K
853#4
854Creating a Persistent Daemon with init Hack #4
855Make sure that your process stays up, no matter what
856There are a number of scripts that will automatically restart a process if it
857exits unexpectedly. Perhaps the simplest is something like:
858$while : ; do echo "Run some code here..."; sleep 1; done
859If you run a foreground process in place of that echo line, then the process is
860always guaranteed to be running (or, at least, it will try to run). The : sim-
861ply makes the while always execute (and is more efficient than running /bin/
862true, as it doesn’t have to spawn an external command on each iteration).
863Definitely do not run a background process in place of the echo, unless you
864enjoy filling up your process table (as the while will then spawn your com-
865mand as many times as it can, one every second). But as far as cool hacks go,
866the while approach is fairly lacking in functionality.
867What happens if your command runs into an abnormal condition? If it exits
868immediately, then it will retry every second, without giving any indication
869that there is a problem (unless the process has its own logging system or
870uses syslog). It might make sense to have something watch the process, and
871stop trying to respawn it if it returns too quickly after a few tries.
872There is a utility already present on every Linux system that will do this
873automatically for you: init. The same program that brings up the system and
874sets up your terminals is perfectly suited for making sure that programs are
875always running. In fact, that is its primary job.
876You can add arbitrary lines to /etc/inittab specifying programs you’d like init
877to watch for you:
878zz:12345:respawn:/usr/local/sbin/my_daemon
879The inittab line consists of an arbitrary (but unique) two character identifica-
880tion string (in this case, zz ), followed by the runlevels that this program
881should be run in, then the respawn keyword, and finally the full path to the
882command. In the above example, as long as my_daemon is configured to run
883in the foreground, init will respawn another copy whenever it exits. After
884making changes to inittab, be sure to send a HUP to init so it will reload its
885configuration. One quick way to do this is:
886#kill -HUP 1
887D ownload from Wow! eBook <www.wowebook.com>
88810 | Server Basics
889#5 n>&m: Swap Standard Output and Standard Error
890HACK
891If the command respawns too quickly, then init will postpone execution for
892a while, to keep it from tying up too many resources. For example:
893zz:12345:respawn:/bin/touch /tmp/timestamp
894This will cause the file /tmp/timestamp to be touched several times a sec-
895ond, until init decides that enough is enough. You should see this message
896in /var/log/messages almost immediately:
897Sep 8 11:28:23 catlin init: Id "zz" respawning too fast: disabled for 5
898minutes
899In five minutes, init will try to run the command again, and if it is still
900respawning too quickly, it will disable it again.
901Obviously, this method is fine for commands that need to run as root, but
902what if you want your auto-respawning process to run as some other user?
903That’s no problem: use sudo:
904zz:12345:respawn:/usr/bin/sudo -u rob /bin/touch /tmp/timestamp
905Now that touch will run as rob, not as root. If you’re trying these com-
906mands as you read this, be sure to remove the existing /tmp/timestamp
907before trying this sudo line. After sending a HUP to init, take a look at the
908timestamp file:
909rob@catlin:~#ls -al /tmp/timestamp
910-rw-r--r-- 1 rob users 0 Sep 8 11:28 /tmp/timestamp
911The two drawbacks to using init to run arbitrary daemons are that you need
912to comment out the line in inittab if you need to bring the daemon down
913(since it will just respawn if you kill it) and that only root can add entries to
914inittab. But for keeping a process running that simply must stay up no mat-
915ter what, init does a great job.
916See also:
917• “Make sudo Work Harder” [Hack #12]
918H A C K
919#5
920n>&m: Swap Standard Output and
921Standard Error Hack #5
922Direct standard out and standard error to wherever you need them to go
923By default, a command’s standard error goes to your terminal. The standard
924output goes to the terminal or is redirected somewhere (to a file, down a
925pipe, into backquotes).
926Sometimes you want the opposite. For instance, you may need to send a com-
927mand’s standard output to the screen and grab the error messages (standard
928error) with backquotes. Or, you might want to send a command’s standard
929n>&m: Swap Standard Output and Standard Error #5
930Server Basics | 11
931HACK
932output to a file and the standard error down a pipe to an error-processing
933command. Here’s how to do that in the Bourne shell. (The C shell can’t do
934this.)
935File descriptors 0, 1, and 2 are the standard input, standard output, and
936standard error, respectively. Without redirection, they’re all associated with
937the terminal file /dev/tty. It’s easy to redirect any descriptor to any file—if
938you know the filename. For instance, to redirect file descriptor to errfile,
939type:
940$command 2> errfile
941You know that a pipe and backquotes also redirect the standard output:
942$command | ...\
943$var=`command`
944But there’s no filename associated with the pipe or backquotes, so you can’t
945use the 2> redirection. You need to rearrange the file descriptors without
946knowing the file (or whatever) that they’re associated with. Here’s how.
947Let’s start slowly. By sending both standard output and standard error to
948the pipe or backquotes, the Bourne shell operator n>&m rearranges the files
949and file descriptors. It says “make file descriptor n point to the same file as
950file descriptor m.” Let’s use that operator on the previous example. We’ll
951send standard error to the same place standard output is going:
952$command 2>&1 | ...
953$var=`command 2>&1`
954In both those examples, 2>&1 means “send standard error (file descriptor 2)
955to the same place standard output (file descriptor 1) is going.” Simple, eh?
956You can use more than one of those n>&m operators. The shell reads them
957left-to-right before it executes the command.
958“Oh!” you might say, “To swap standard output and standard error—make
959stderr go down a pipe and stdout go to the screen—I could do this!”
960$command 2>&1 1>&2 | ...(wrong...)
961Sorry, Charlie. When the shell sees 2>&1 1>&2, the shell first does 2>&1.
962You’ve seen that before—it makes file descriptor 2 (stderr) go the same place
963as file descriptor 1 (stdout). Then, the shell does 1>&2. It makes stdout (1)
964go the same place as stderr (2), but stderr is already going the same place as
965stdout, down the pipe.
966This is one place that the other file descriptors, 3 through 9, come in handy.
967They normally aren’t used. You can use one of them as a “holding place” to
968remember where another file descriptor “pointed.” For example, one way to
969read the operator 3>&2 is “make 3 point the same place as 2”. After you use
97012 | Server Basics
971#6 Building Complex Command Lines
972HACK
9733>&2 to grab the location of 2, you can make 2 point somewhere else.
974Then, make 1 point to where 2 used to (where 3 points now).
975The command line you want is one of these:
976$command 3>&2 2>&1 1>&3 | ...
977$var=`command 3>&2 2>&1 1>&3`
978Open files are automatically closed when a process exits. But it’s safer to
979close the files yourself as soon as you’re done with them. That way, if you
980forget and use the same descriptor later for something else (for instance, use
981F.D. 3 to redirect some other command, or a subprocess uses F.D. 3), you
982won’t run into conflicts. Use m<&- to close input file descriptor m and m>
983&- to close output file descriptor m. If you need to close standard input, use
984<&- ; >&- will close standard output.
985H A C K
986#6
987Building Complex Command Lines Hack #6
988Build simple commands into full-fledged paragraphs
989for complex (but meaningful) reports
990Studying Linux (or indeed any Unix) is much like studying a foreign lan-
991guage. At some magical point in the course of one’s studies, halting monosyl-
992labic mutterings begin to meld together into coherent, often used phrases.
993Eventually, one finds himself pouring out entire sentences and paragraphs of
994the Unix Mother Tongue, with one’s mind entirely on the problem at hand
995(and not on the syntax of any particular command). But just as high school
996foreign language students spend much of their time asking for directions to
997the toilet and figuring out just what the dative case really is, the path to Linux
998command-line fluency must begin with the first timidly spoken magic words.
999Your shell is very forgiving, and will patiently (and repeatedly) listen to your
1000every utterance, until you get it just right. Any command can serve as the
1001input for any other, making for some very interesting Unix “sentences.”
1002When armed with the handy (and probably over-used) up arrow, it is possi-
1003ble to chain together commands with slight tweaks over many tries to
1004achieve some very complex behavior.
1005For example, suppose that you’re given the task of finding out why a web
1006server is throwing a bunch of errors over time. If you type less error_log ,
1007you see that there are many “soft errors” relating to missing (or badly
1008linked) graphics:
1009[Tue Aug 27 00:22:38 2002] [error] [client 17.136.12.171] File does not
1010exist: /htdocs/images/spacer.gif
1011[Tue Aug 27 00:31:14 2002] [error] [client 95.168.19.34] File does not
1012exist: /htdocs/image/trans.gif
1013Building Complex Command Lines #6
1014Server Basics | 13
1015HACK
1016[Tue Aug 27 00:36:57 2002] [error] [client 2.188.2.75] File does not exist:
1017/htdocs/images/linux/arrows-linux-back.gif
1018[Tue Aug 27 00:40:37 2002] [error] [client 2.188.2.75] File does not exist:
1019/htdocs/images/linux/arrows-linux-back.gif
1020[Tue Aug 27 00:41:43 2002] [error] [client 6.93.4.85] File does not exist: /
1021htdocs/images/linux/hub-linux.jpg
1022[Tue Aug 27 00:41:44 2002] [error] [client 6.93.4.85] File does not exist: /
1023htdocs/images/xml/hub-xml.jpg
1024[Tue Aug 27 00:42:13 2002] [error] [client 6.93.4.85] File does not exist: /
1025htdocs/images/linux/hub-linux.jpg
1026[Tue Aug 27 00:42:13 2002] [error] [client 6.93.4.85] File does not exist: /
1027htdocs/images/xml/hub-xml.jpg
1028and so on. Running a logging package (like analog) reports exactly how
1029many errors you have seen in a day but few other details (which is how you
1030were probably alerted to the problem in the first place). Looking at the log-
1031file directly gives you every excruciating detail but is entirely too much infor-
1032mation to process effectively.
1033Let’s start simple. Are there any errors other than missing files? First we’ll
1034need to know how many errors we’ve had today:
1035$wc -l error_log
10361265 error_log
1037And how many were due to File does not exist errors?
1038$grep "File does not exist:" error_log | wc -l
10391265 error_log
1040That’s a good start. At least we know that we’re not seeing permission prob-
1041lems or errors in anything that generates dynamic content (like cgi scripts.) If
1042every error is due to missing files (or typos in our html that point to the
1043wrong file) then it’s probably not a big problem. Let’s generate a list of the
1044filenames of all bad requests. Hit the up arrow and delete that wc -l :
1045$ grep "File does not exist:" error_log |awk '{print $13}' | less
1046That’s the sort of thing that we want (the 13th field, just the filename), but
1047hang on a second. The same couple of files are repeated many, many times.
1048Sure, we could email this to the web team (all whopping 1265 lines of it),
1049but I’m sure they wouldn’t appreciate the extraneous spam. Printing each
1050file exactly once is easy:
1051$ grep "File does not exist:" error_log | awk '{print $13}' |sort | uniq |
1052less
1053This is much more reasonable (substitute a wc -l for that less to see just
1054how many unique files have been listed as missing). But that still doesn’t
1055really solve the problem. Maybe one of those files was requested once, but
1056another was requested several hundred times. Naturally, if there is a link
1057somewhere with a typo in it, we would see many requests for the same
105814 | Server Basics
1059#6 Building Complex Command Lines
1060HACK
1061“missing” file. But the previous line doesn’t give any indication of which
1062files are requested most. This isn’t a problem for bash; let’s try out a com-
1063mand line for loop.
1064$for x in `grep "File does not exist" error_log | awk '{print $13}' | sort
1065| uniq`; do \
1066echo -n "$x : "; grep $x error_log | wc -l; done
1067We need those backticks (`) to actually execute our entire command from
1068the previous example and feed the output of it to a for loop. On each itera-
1069tion through the loop, the $x variable is set to the next line of output of our
1070original command (that is, the next unique filename reported as missing).
1071We then grep for that filename in the error_log, and count how many times
1072we see it. The echo at the end just prints it in a somewhat nice report format.
1073I call it a somewhat nice report because not only is it full of single hit errors
1074(which we probably don’t care about), the output is very jagged, and it isn’t
1075even sorted! Let’s sort it numerically, with the biggest hits at the top, num-
1076bers on the left, and only show the top 20 most requested “missing” files:
1077$ for x in `grep "File does not exist" error_log | awk '{print $13}' | sort
1078| uniq`; do \
1079grep $x error_log | wc -l| tr -d '\n'; echo " : $x"; done| sort +2 -rn |
1080head -20
1081That’s much better and not even much more typing than the last try. We
1082need the tr to eliminate the trailing newline at the end of wc ’s output (why it
1083doesn’t have a switch to do this, I’ll never know). Your output should look
1084something like this:
1085595 : /htdocs/images/pixel-onlamp.gif.gif
1086156 : /htdocs/image/trans.gif
1087139 : /htdocs/images/linux/arrows-linux-back.gif
108868 : /htdocs/pub/a/onjava/javacook/images/spacer.gif
108950 : /htdocs/javascript/2001/03/23/examples/target.gif
1090From this report, it’s very simple to see that almost half of our errors are due
1091to a typo on a popular web page somewhere (note the repeated .gif.gif in
1092the first line). The second is probably also a typo (should be images/ , not
1093image/ ). The rest are for the web team to figure out:
1094$( echo "Here's a report of the top 20 'missing' files in the error_log.";
1095echo; \
1096for x in `grep "File does not exist" error_log | awk '{print $13}' | sort |
1097uniq`; do \
1098grep $x error_log | wc -l | tr -d '\n'; echo " : $x"; done | sort +2 -rn |
1099head -20)\
1100| mail -s "Missing file report" webmaster@oreillynet.com
1101and maybe one hardcopy for the weekly development meeting:
1102Working with Tricky Files in xargs #7
1103Server Basics | 15
1104HACK
1105$ for x in `grep "File does not exist" error_log | awk '{print $13}' | sort
1106| uniq`; do \
1107grep $x error_log | wc -l | tr -d '\n'; echo " : $x"; done | sort +2 -rn |
1108head -20 \
1109| enscript
1110Hacking the Hack
1111Once you get used to chunking groups of commands together, you can
1112chain their outputs together indefinitely, creating any sort of report you like
1113out of a live data stream. Naturally, if you find yourself doing a particular
1114task regularly, you might want to consider turning it into a shell script of its
1115own (or even reimplementing it in Perl or Python for efficiency’s sake, as
1116every | in a command means that you’ve spawned yet another program). On
1117a modern (and unloaded) machine, you’ll hardly notice the difference, but
1118it’s considered good form to clean up the solution once you’ve hacked it out.
1119And on the command line, there’s plenty of room to hack.
1120H A C K
1121#7
1122Working with Tricky Files in xargs Hack #7
1123Deal with many files containing spaces or other strange characters
1124When you have a number of files containing spaces, parentheses, and other
1125“forbidden” characters, dealing with them can be daunting. This is a prob-
1126lem that seems to come up frequently, with the recent explosive popularity
1127of digital music. Luckily, tab completion in bash makes it simple to handle
1128one file at a time. For example:
1129rob@catlin:~/Music$ls
1130Hallucinogen - The Lone Deranger
1131Misc - Pure Disco
1132rob@catlin:~/Music$rm -rf Misc[TAB]
1133rob@catlin:~/Music$rm -rf Misc\ -\ Pure\ Disco/
1134Hitting the Tab key for [TAB] above replaces the command line with the line
1135below it, properly escaping any special characters contained in the file.
1136That’s fine for one file at a time, but what if we want to do a massive trans-
1137formation (say, renaming a bunch of mp3s to include an album name)? Take
1138a look at this:
1139rob@catlin:~/Music$cd Hall[TAB]
1140rob@catlin:~/Music$cd Hallucinogen\ -\ The\ Lone\ Deranger/
1141rob@catlin:~/Music/Hallucinogen - The Lone Deranger$ls
1142Hallucinogen - 01 - Demention.mp3
1143Hallucinogen - 02 - Snakey Shaker.mp3
1144Hallucinogen - 03 - Trancespotter.mp3
1145Hallucinogen - 04 - Horrorgram.mp3
1146Hallucinogen - 05 - Snarling (Remix).mp3
1147Hallucinogen - 06 - Gamma Goblins Pt. 2.mp3
1148Hallucinogen - 07 - Deranger.mp3
114916 | Server Basics
1150#7 Working with Tricky Files in xargs
1151HACK
1152Hallucinogen - 08 - Jiggle of the Sphinx.mp3
1153rob@catlin:~/Music/Hallucinogen - The Lone Deranger$
1154When attempting to manipulate many files at once, things get tricky. Many
1155system utilities break on whitespace (yielding many more chunks than you
1156intended) and will completely fall apart if you throw a ) or a { at them. What
1157we need is a delimiter that is guaranteed never to show up in a filename, and
1158break on that instead.
1159Fortunately, the xargs utility will break on NULL characters, if you ask it to
1160nicely. Take a look at this script:
1161Listing: albumize
1162#!/bin/sh
1163if [ -z "$ALBUM" ]; then
1164echo 'You must set the ALBUM name first (eg. export ALBUM="Greatest Hits")'
1165exit 1
1166fi
1167for x in *; do
1168echo -n $x; echo -ne '\000'
1169echo -n `echo $x|cut -f 1 -d '-'`
1170echo -n " - $ALBUM - "
1171echo -n `echo $x|cut -f 2- -d '-'`; echo -ne '\000'
1172done | xargs -0 -n2 mv
1173We’re actually doing two tricky things here. First, we’re building a list con-
1174sisting of the original filename followed by the name to which we’d like to
1175mv it, separated by NULL characters, for all files in the current directory.
1176We then feed that entire list to an xargs with two switches: -0 tells it to
1177break on NULLs (instead of newlines or whitespace), and -n2 tells it to take
1178two arguments at a time on each pass, and feed them to our command (mv).
1179Save the script as ~/bin/albumize. Before you run it, set the $ALBUM envi-
1180ronment variable to the name that you’d like injected into the filename just
1181after the first - . Here’s a trial run:
1182rob@catlin:~/Music/Hallucinogen - The Lone Deranger$export ALBUM="The Lone
1183Deranger"
1184rob@catlin:~/Music/Hallucinogen - The Lone Deranger$albumize
1185rob@catlin:~/Music/Hallucinogen - The Lone Deranger$ls
1186Hallucinogen - The Lone Deranger - 01 - Demention.mp3
1187Hallucinogen - The Lone Deranger - 02 - Snakey Shaker.mp3
1188Hallucinogen - The Lone Deranger - 03 - Trancespotter.mp3
1189Hallucinogen - The Lone Deranger - 04 - Horrorgram.mp3
1190Hallucinogen - The Lone Deranger - 05 - Snarling (Remix).mp3
1191Hallucinogen - The Lone Deranger - 06 - Gamma Goblins Pt. 2.mp3
1192Hallucinogen - The Lone Deranger - 07 - Deranger.mp3
1193Hallucinogen - The Lone Deranger - 08 - Jiggle of the Sphinx.mp3
1194rob@catlin:~/Music/Hallucinogen - The Lone Deranger$
1195Immutable Files in ext2/ext3 #8
1196Server Basics | 17
1197HACK
1198What if you would like to remove the album name again? Try this one, and
1199call it ~/bin/dealbumize:
1200#!/bin/sh
1201for x in *; do
1202echo -n $x; echo -ne '\000'
1203echo -n `echo $x|cut -f 1 -d '-'`; echo -n ' - '
1204echo -n `echo $x|cut -f 3- -d '-'`; echo -ne '\000'
1205done | xargs -0 -n2 mv
1206and simply run it (no $ALBUM required):
1207rob@catlin:~/Music/Hallucinogen - The Lone Deranger$dealbumize
1208rob@catlin:~/Music/Hallucinogen - The Lone Deranger$ls
1209Hallucinogen - 01 - Demention.mp3
1210Hallucinogen - 02 - Snakey Shaker.mp3
1211Hallucinogen - 03 - Trancespotter.mp3
1212Hallucinogen - 04 - Horrorgram.mp3
1213Hallucinogen - 05 - Snarling (Remix).mp3
1214Hallucinogen - 06 - Gamma Goblins Pt. 2.mp3
1215Hallucinogen - 07 - Deranger.mp3
1216Hallucinogen - 08 - Jiggle of the Sphinx.mp3
1217rob@catlin:~/Music/Hallucinogen - The Lone Deranger$
1218The -0 switch is also popular to team up with the -print0 option of find
1219(which, naturally, prints matching filenames separated by NULLs instead of
1220newlines). With find and xargs on a pipeline, you can do anything you like
1221to any number of files, without ever running into the dreaded Argument list
1222too long error:
1223rob@catlin:~/Pit of too many files$ls *
1224bash: /bin/ls: Argument list too long
1225A find/xargs combo makes quick work of these files, no matter what they’re
1226called:
1227rob@catlin:/Pit of too many files$find -type f -print0 | xargs -0 ls
1228To delete them, just replace that trailing ls with an rm, and away you go.
1229H A C K
1230#8
1231Immutable Files in ext2/ext3 Hack #8
1232Create files that even root can’t manipulate
1233Here’s a puzzle. Suppose we’re cleaning up /tmp, and run into some trouble:
1234root@catlin:/tmp#rm -rf junk/
1235rm: cannot unlink `junk/stubborn.txt': Operation not permitted
1236rm: cannot remove directory `junk': Directory not empty
1237root@catlin:/tmp#cd junk/
1238root@catlin:/tmp/junk#ls -al
1239total 40
1240drwxr-xr-x 2 root root 4096 Sep 4 14:45 ./
1241drwxrwxrwt 13 root root 4096 Sep 4 14:45 ../
124218 | Server Basics
1243#8 Immutable Files in ext2/ext3
1244HACK
1245-rw-r--r-- 1 root root 29798 Sep 4 14:43 stubborn.txt
1246root@catlin:/tmp/junk#rm ./stubborn.txt
1247rm: remove write-protected file `./stubborn.txt'?y
1248rm: cannot unlink `./stubborn.txt': Operation not permitted
1249What’s going on? Are we root or aren’t we? Let’s try emptying the file
1250instead of deleting it:
1251root@catlin:/tmp/junk#cp /dev/null stubborn.txt
1252cp: cannot create regular file `stubborn.txt': Permission denied
1253root@catlin:/tmp/junk#> stubborn.txt
1254bash: stubborn.txt: Permission denied
1255Well, /tmp certainly isn’t mounted read-only. What is going on?
1256In the ext2 and ext3 filesystems, there are a number of additional file
1257attributes that are available beyond the standard bits accessible through
1258chmod. If you haven’t seen it already, take a look at the manpages for chattr
1259and its companion, lsattr.
1260One of the very useful new attributes is -i, the immutable flag. With this bit
1261set, attempts to unlink, rename, overwrite, or append to the file are forbid-
1262den. Even making a hard link is denied (so you can’t make a hard link, then
1263edit the link). And having root privileges makes no difference when immuta-
1264ble is in effect:
1265root@catlin:/tmp/junk#ln stubborn.txt another.txt
1266ln: creating hard link `another.txt' to `stubborn.txt': Operation not
1267permitted
1268To view the supplementary ext flags that are in force on a file, use lsattr:
1269root@catlin:/tmp/junk#lsattr
1270---i--------- ./stubborn.txt
1271and to set flags a la chmod, use chattr:
1272root@catlin:/tmp/junk#chattr -i stubborn.txt
1273root@catlin:/tmp/junk#rm stubborn.txt
1274root@catlin:/tmp/junk#
1275This could be terribly useful for adding an extra security step on files you
1276know you’ll never want to change (say, /etc/rc.d/* or various configuration
1277files.) While little will help you on a box that has been r00ted, immutable
1278files probably aren’t vulnerable to simple overwrite attacks from other pro-
1279cesses, even if they are owned by root.
1280There are hooks for adding compression, security deletes, undeletability, syn-
1281chronous writes, and a couple of other useful attributes. As of this writing,
1282many of the additional attributes aren’t implemented yet, but keep watching
1283for new developments on the ext filesystem.
1284Speeding Up Compiles #9
1285Server Basics | 19
1286HACK
1287H A C K
1288#9
1289Speeding Up Compiles Hack #9
1290Make sure you’re keeping all processors busy with parallel builds
1291If you’re running a multiprocessor system (SMP) with a moderate amount of
1292RAM, you can usually see significant benefits by performing a parallel make
1293when building code. Compared to doing serial builds when running make
1294(as is the default), a parallel build is a vast improvement.
1295To tell make to allow more than one child at a time while building, use
1296the -j switch:
1297rob@mouse:~/linux$make -j4; make -j4 modules
1298Some projects aren’t designed to handle parallel builds and can get con-
1299fused if parts of the project are built before their parent dependencies have
1300completed. If you run into build errors, it is safest to just start from scratch
1301this time without the -j switch.
1302By way of comparison, here are some sample timings. They were performed
1303on an otherwise unloaded dual PIII/600 with 1GB RAM. Each time I built a
1304bzImage for Linux 2.4.19 (redirecting STDOUT to /dev/null), and removed
1305the source tree before starting the next test.
1306time make bzImage:
1307real 7m1.640s
1308user 6m44.710s
1309sys 0m25.260s
1310time make -j2 bzImage:
1311real 3m43.126s
1312user 6m48.080s
1313sys 0m26.420s
1314time make -j4 bzImage:
1315real 3m37.687s
1316user 6m44.980s
1317sys 0m26.350s
1318time make -j10 bzImage:
1319real 3m46.060s
1320user 6m53.970s
1321sys 0m27.240s
1322As you can see, there is a significant improvement just by adding the -j2
1323switch. We dropped from 7 minutes to 3 minutes and 43 seconds of actual
1324time. Increasing to -j4 saved us about five more seconds, but jumping all the
1325way to -j10 actually hurt performance by a few seconds. Notice how user and
132620 | Server Basics
1327#10 At Home in Your Shell Environment
1328HACK
1329system seconds are virtually the same across all four runs. In the end, you need
1330to shovel the same sized pile of bits, but -j on a multi-processor machine sim-
1331ply lets you spread it around to more people with shovels.
1332Of course, bits all eventually end up in the bit bucket anyway. But hey, if
1333nothing else, performance timings are a great way to keep your cage warm.
1334H A C K
1335#10
1336At Home in Your Shell Environment Hack #10
1337Make bash more comfortable through environment variables
1338Consulting a manpage for bash can be a daunting read, especially if you’re
1339not precisely sure what you’re looking for. But when you have the time to
1340devote to it, the manpage for bash is well worth the read. This is a shell just
1341oozing with all sorts of arcane (but wonderfully useful) features, most of
1342which are simply disabled by default.
1343Let’s start by looking at some useful environment variables, and some use-
1344ful values to which to set them:
1345export PS1=`echo -ne "\033[0;34m\u@\h:\033[0;36m\w\033[0;34m\$\033[0;37m "`
1346As you probably know, the PS1 variable sets the default system prompt, and
1347automatically interprets escape sequences such as \u (for username) and \w
1348(for the current working directory.) As you may not know, it is possible to
1349encode ANSI escape sequences in your shell prompt, to give your prompt a
1350colorized appearance. We wrap the whole string in backticks (`) in order to
1351get echo to generate the magic ASCII escape character. This is executed
1352once, and the result is stored in PS1. Let’s look at that line again, with bold-
1353face around everything that isn’t an ANSI code:
1354export PS1=`echo -ne "\033[0;34m\u@\h:\033[0;36m\w\033[0;34m\$\033[0;37m "`
1355You should recognize the familiar \u@\h:\w\$prompt that we’ve all grown to
1356know and love. By changing the numbers just after each semicolon, you can
1357set the colors of each part of the prompt to your heart’s content.
1358Along the same lines, here’s a handy command that is run just before bash
1359gives you a prompt:
1360export PROMPT_COMMAND='echo -ne "\033]0;${USER}@${HOSTNAME}: ${PWD}\007"'
1361(We don’t need backticks for this one, as bash is expecting it to contain an
1362actual command, not a string.) This time, the escape sequence is the magic
1363string thatmanipulates the titlebar on most terminal windows (such as
1364xterm, rxvt, eterm, gnometerm, etc.). Anything after the semicolon and
1365before the \007 gets printed to your titlebar every time you get a new
1366prompt. In this case, we’re displaying your username, the host you’re logged
1367into, and the current working directory. This is quite handy for being able to
1368D ownload from Wow! eBook <www.wowebook.com>
1369At Home in Your Shell Environment #10
1370Server Basics | 21
1371HACK
1372tell at a glance (or even while within vim) to which machine you’re logged in,
1373and to what directory you’re about to save your file. See “Constant Load
1374Average Display in the Titlebar” [Hack #59] if you’d like to update your titlebar
1375in real time instead of at every new bash prompt.
1376Have you ever accidentally hit ^D too many times in a row, only to find
1377yourself logged out? You can tell bash to ignore as many consecutive ^D
1378hits as you like:
1379export IGNOREEOF=2
1380This makes bash follow the Snark rule (“What I tell you three times is true”)
1381and only log you out if you hit ^D three times in a row. If that’s too few for
1382you, feel free to set it to 101 and bash will obligingly keep count for you.
1383Having a directory just off of your home that lies in your path can be
1384extremely useful (for keeping scripts, symlinks, and other random pieces of
1385code.) A traditional place to keep this directory is in bin underneath your
1386home directory. If you use the ~ expansion facility in bash, like this:
1387export PATH=$PATH:~/bin
1388then the path will always be set properly, even if your home directory ever
1389gets moved (or if you decide you want to use this same line on multiple
1390machines with potentially different home directories—as in movein.sh). See
1391“Get Settled in Quickly with movein.sh” [Hack #72] .
1392Did you know that just as commands are searched for in the PATH variable
1393(and manpages are searched for in the MANPATH variable), directories are
1394likewise searched for in the CDPATH variable every time you issue a cd? By
1395default, it is only set to “.”, but can be set to anything you like:
1396export CDPATH=.:~
1397This will make cd search not only the current directory, but also your home
1398directory for the directory you try to change to. For example:
1399rob@caligula:~$ls
1400bin/ devel/ incoming/ mail/ test/ stuff.txt
1401rob@caligula:~$cd /usr/local/bin
1402rob@caligula:/usr/local/bin$cd mail
1403bash: cd: mail: No such file or directory
1404rob@caligula:/usr/local/bin$export CDPATH=.:~
1405rob@caligula:/usr/local/bin$cd mail
1406/home/rob/mail
1407rob@caligula:~/mail$
1408You can put as many paths as you like to search for in CDPATH, separating
1409each with a : (just as with the PATH and MANPATH variables.)
141022 | Server Basics
1411#10 At Home in Your Shell Environment
1412HACK
1413We all know about the up arrow and the history command. But what hap-
1414pens if you accidentally type something sensitive on the command line? Sup-
1415pose you slip while typing and accidentally type a password where you
1416meant to type a command. This accident will faithfully get recorded to your
1417~/.bash_history file when you logout, where another unscrupulous user
1418might happen to find it. Editing your .bash_history manually won’t fix the
1419problem, as the file gets rewritten each time you log out.
1420To clear out your history quickly, try this from the command line:
1421export HISTSIZE=0
1422This completely clears out the current bash history and will write an empty
1423.bash_history on logout. When you log back in, your history will start over
1424from scratch but will otherwise work just as before. From now on, try to be
1425more careful!
1426Do you have a problem with people logging into a machine, then discon-
1427necting their laptop and going home without logging back out again? If
1428you’ve ever run a w and seen a bunch of idle users who have been logged in
1429for several days, try setting this in their environment:
1430export TMOUT=600
1431The TMOUT variable specifies the number of seconds that bash will wait for
1432input on a command line before logging the user out automatically. This
1433won’t help if your users are sitting in a vi window but will alleviate the prob-
1434lem of users just sitting at an idle shell. Ten minutes might be a little short
1435for some users, but kindly remind them that if they don’t like the system
1436default, they are free to reset the variable themselves.
1437This brings up an interesting point: exactly where do you go to make any of
1438these environment changes permanent? There are several files that bash con-
1439sults when starting up, depending on whether the shell was called at login or
1440from within another shell.
1441From bash(1), on login shells:
1442...it first reads and executes commands from the file /etc/profile, if that file
1443exists. After reading that file, it looks for ~/.bash_profile, ~/.bash_login, and
1444~/.profile, in that order, and reads and executes commands from the first one
1445that exists and is readable.... When a login shell exits, bash reads and exe-
1446cutes commands from the file ~/.bash_logout, if it exists.
1447For all other shells:
1448bash reads and executes commands from ~/.bashrc, if that file exists.
1449For the full definition of what constitutes a login shell (and for a whole
1450bunch of information about the environment you work in every day), con-
1451sult bash(1).
1452Finding and Eliminating setuid/setgid Binaries #11
1453Server Basics | 23
1454HACK
1455H A C K
1456#11
1457FindingandEliminatingsetuid/setgidBinaries Hack #11
1458Eliminate potential root exploits before they have a chance to happen
1459While running Linux as a server, one guiding principle that has served me
1460well is to continually ask, “what am I trying to achieve?” Does it make sense
1461for a web server to have the printing subsystem installed? Should a system
1462with no console have gpm installed? Usually, extra software packages just
1463take up unnecessary disk space, but in the case of setuid or setgid binaries,
1464the situation could be far worse.
1465While distribution maintainers work very hard to ensure that all known
1466exploits for setuid and setgid binaries have been removed, it seems that a
1467few new unexpected exploits come out every month or two. Especially if
1468your server has more shell users than yourself, you should regularly audit
1469the setuid and setgid binaries on your system. Chances are you’ll be sur-
1470prised at just how many you’ll find.
1471Here’s one command for finding all the files with a setuid or setgid bit set:
1472root@catlin:~#find / -perm +6000 -type f -exec ls -ld {} \; > setuid.txt &
1473This will create a file called setuid.txt that contains the details of all of the
1474matching files present on your system. It is a very good idea to look through
1475this list, and remove the s bits of any tools that you don’t use.
1476Let’s look through what we might find on a typical system:
1477-rws--x--x 1 root bin 35248 May 30 2001 /usr/bin/at
1478-rws--x--x 1 root bin 10592 May 30 2001 /usr/bin/crontab
1479Not much surprise here. at and crontab need root privileges in order to
1480change to the user that requested the at job or cron job. If you’re paranoid,
1481and you don’t use these facilities, then you could remove the setuid bits
1482with:
1483#chmod a-s /usr/bin/{at,crontab}
1484Generally speaking, it’s a bad idea to disable cron (as so many systems
1485depend on timed job execution). But when was the last time you used at?
1486Do your users even know what it’s for? Personally, I find at a nice shortcut
1487to setting up a full-blown cron job, and wouldn’t like to part with it. But if
1488there is no call for it on your particular system, you should consider
1489defanging it. With the setuid bit removed, the commands will no longer be
1490available to regular users but will still work fine as root.
1491-rws--x--x 1 root bin 11244 Apr 15 2001 /usr/bin/disable-paste
1492This is part of the gpm package (a mouse driver for the Linux console). Do
1493you have a mouse attached to the console of this machine? Do you use it in
149424 | Server Basics
1495#11 Finding and Eliminating setuid/setgid Binaries
1496HACK
1497text mode? If not, then why leave a setuid root binary in place that will never
1498even be called?
1499-r-s--s--x 1 root lp 14632 Jun 18 2001 /usr/bin/lpq
1500-r-s--s--x 1 root lp 15788 Jun 18 2001 /usr/bin/lpr
1501-r-s--s--x 1 root lp 15456 Jun 18 2001 /usr/bin/lprm
1502-r-xr-s--x 1 root lp 23772 Jun 18 2001 /usr/sbin/lpc
1503These are all part of the printing subsystem. Does this machine actually use
1504lp to print?
1505-rws--x--x 1 root bin 33760 Jun 18 2000 /usr/bin/chage
1506-rws--x--x 1 root bin 29572 Jun 18 2000 /usr/bin/chfn
1507-rws--x--x 1 root bin 27188 Jun 18 2000 /usr/bin/chsh
1508-rws--x--x 1 root bin 35620 Jun 18 2000 /usr/bin/passwd
1509These are all necessary for users to be able to set their passwords, shell, and
1510finger information. Does your site use finger information at all? If, like most
1511sites, you have a roster somewhere else (probably on the web) that isn’t kept
1512in sync with user’s GECOS field, then this information is generally useless
1513(except for the user’s “real” name, which is still used in some email clients).
1514Do you really need to allow users to change this information on their own,
1515without admin intervention?
1516-r-xr-sr-x 1 root tty 9768 Jun 21 2001 /usr/bin/wall
1517-r-xr-sr-x 1 root tty 8504 Jun 21 2001 /usr/bin/write
1518Both wall and write need to be setgid tty to write to other user’s terminals.
1519This is generally a safe operation, but can be abused by miscreants who like
1520to write bad data (or lots of bad data) to other user’s terminals. If you don’t
1521need to provide this functionality to other users, why not disable the setgid
1522bit? If you do, root will still be able to send walls and writes (for example,
1523when a message is sent by shutdown when rebooting the system).
1524-rwsr-xr-x 1 root bin 14204 Jun 3 2001 /usr/bin/rcp
1525-rwsr-xr-x 1 root bin 10524 Jun 3 2001 /usr/bin/rlogin
1526-r-sr-xr-x 1 root bin 7956 Jun 3 2001 /usr/bin/rsh
1527The r* commands are left from an age (perhaps not so long ago), before the
1528days of ssh. Do you need to provide the r commands to your users? Is there
1529anything that ssh and scp can’t do that you absolutely need rsh and rcp for?
1530More than likely, once you’ve worked with ssh for a while, you’ll never miss
1531the r commands, and can safely remove the potential ticking time bomb of a
1532disused setuid rsh installation. If you’re looking for interesting things to do
1533with ssh, check out any of the ssh hacks elsewhere in this book.
1534-r-sr-xr-x 1 root bin 10200 Jun 3 2001 /usr/bin/traceroute
1535-r-sr-xr-x 1 root bin 15004 Jun 3 2001 /bin/ping
1536The traceroute and ping commands need the setuid root bit to be able to cre-
1537ate ICMP packets. If you want your users to be able to run these network
1538Make sudo Work Harder #12
1539Server Basics | 25
1540HACK
1541diagnostic tools, then they’ll need to be setuid root. Otherwise, remove set-
1542uid and then only root will be able to run ping and traceroute.
1543-r-sr-sr-- 1 uucp uucp 83344 Feb 10 2001 /usr/bin/uucp
1544-r-sr-sr-- 1 uucp uucp 36172 Feb 10 2001 /usr/bin/uuname
1545-r-sr-sr-- 1 uucp uucp 93532 Feb 10 2001 /usr/bin/uustat
1546-r-sr-sr-- 1 uucp uucp 85348 Feb 10 2001 /usr/bin/uux
1547-r-sr-sr-- 1 uucp uucp 65492 Feb 10 2001 /usr/lib/uucp/uuchk
1548-r-sr-sr-- 1 uucp uucp 213832 Feb 10 2001 /usr/lib/uucp/uucico
1549-r-sr-sr-- 1 uucp uucp 70748 Feb 10 2001 /usr/lib/uucp/uuconv
1550-r-sr-sr-- 1 uucp uucp 315 Nov 22 1995 /usr/lib/uucp/uusched
1551-r-sr-sr-- 1 uucp uucp 95420 Feb 10 2001 /usr/lib/uucp/uuxqt
1552When was the last time you connected to another machine with UUCP?
1553Have you ever set up the UUCP system? I have been a network admin for
1554ten years, and in that time, I have never come across a live UUCP installa-
1555tion. That’s not to say that UUCP isn’t useful, just that in these days of per-
1556manently connected TCP/IP networks, UUCP is becoming extremely
1557uncommon. If you’re not using UUCP, then leaving setuid and setgid bina-
1558ries online to support it doesn’t make much sense.
1559Do any of the binaries in the examples above have potential root (or other
1560privilege elevation) exploits? I have no idea. But I do know that by removing
1561unnecessary privileges, I minimize my exposure to the possibility that an
1562exploit might be run on this system if one is discovered.
1563I have tried to present this hack as a process, not as a solution. This list is
1564certainly by no means definitive. As you build a server, always keep in mind
1565what the intended use is, and build the system accordingly. Whenever possi-
1566ble, remove privileges (or even entire packages) that provide functionality
1567that you simply don’t need. Consult the manpage if you ever wonder about
1568what a particular binary is supposed to be doing, and why it is installed set-
1569uid (and when the manpage fails you, remember to use the source).
1570H A C K
1571#12
1572Make sudo Work Harder Hack #12
1573Use sudo to let other users do your evil bidding, without giving away the
1574machine
1575The sudo utility can help you delegate some system responsibilities to other
1576people, without giving away full root access. It is a setuid root binary that
1577executes commands on an authorized user’s behalf, after they have entered
1578their current password.
1579As root, run /usr/sbin/visudo to edit the list of users who can call sudo. The
1580default sudo list looks something like this:
1581root ALL=(ALL) ALL
158226 | Server Basics
1583#12 Make sudo Work Harder
1584HACK
1585Unfortunately, many system admins tend to use this entry as a template and
1586grant unrestricted root access to all other admins unilaterally:
1587root ALL=(ALL) ALL
1588rob ALL=(ALL) ALL
1589jim ALL=(ALL) ALL
1590david ALL=(ALL) ALL
1591While this may allow you to give out root access without giving away the
1592root password, it is really only a useful method when all of the sudo users
1593can be completely trusted. When properly configured, the sudo utility allows
1594for tremendous flexibility for granting access to any number of commands,
1595run as any arbitrary uid.
1596The syntax of the sudo line is:
1597user machine=(effective user)command
1598The first column specifies the sudo user. The next column defines the hosts
1599in which this sudo entry is valid. This allows you to easily use a single sudo
1600configuration across multiple machines.
1601For example, suppose you have a developer who needs root access on a
1602development machine, but not on any other server:
1603peter beta.oreillynet.com=(ALL) ALL
1604The next column (in parentheses) specifies the effective user that may run
1605the commands. This is very handy for allowing users to execute code as
1606users other than root:
1607peter lists.oreillynet.com=(mailman) ALL
1608Finally, the last column specifies all the commands that this user may run:
1609david ns.oreillynet.com=(bind) /usr/sbin/rndc,/usr/sbin/named
1610If you find yourself specifying large lists of commands (or, for that matter,
1611users or machines), then take advantage of sudo’s Alias syntax. An Alias can
1612be used in place of its respective entry on any line of the sudo configuration:
1613User_Alias ADMINS=rob,jim,david
1614User_Alias WEBMASTERS=peter,nancy
1615Runas_Alias DAEMONS=bind,www,smmsp,ircd
1616Host_Alias WEBSERVERS=www.oreillynet.com,www.oreilly.com,www.perl.com
1617Cmnd_Alias PROCS=/bin/kill,/bin/killall,/usr/bin/skill,/usr/bin/top
1618Cmnd_Alias APACHE=/usr/local/apache/bin/apachectl
1619WEBMASTERS WEBSERVERS=(www) APACHE
1620ADMINS ALL=(DAEMONS) ALL
1621Using a Makefile to Automate Admin Tasks #13
1622Server Basics | 27
1623HACK
1624It is also possible to specify system groups in place of the user specification
1625to allow any user who belongs to that group to execute commands. Just
1626preface the group with a %, like this:
1627%wwwadmin WEBSERVERS=(www) APACHE
1628Now any user who is part of the wwwadmin group can execute apachectl as
1629the www user on any of the web server machines.
1630One very useful feature is the NOPASSWD: flag. When present, the user
1631won’t have to enter his password before executing the command:
1632rob ALL=(ALL) NOPASSWD: PROCS
1633This will allow the user rob to execute kill, killall, skill, and top on any
1634machine, as any user, without entering a password.
1635Finally, sudo can be a handy alternative to su for running commands at star-
1636tup out of the system rc files:
1637(cd /usr/local/mysql; sudo -u mysql ./bin/safe_mysqld &)
1638sudo -u www /usr/local/apache/bin/apachectl start
1639For that to work at boot time, you’ll need the default line root ALL=(ALL)
1640ALL to be present.
1641Use sudo with the usual caveats that apply to setuid binaries. Particularly if
1642you allow sudo to execute interactive commands (like editors) or any sort of
1643compiler or interpreter, you should assume that it is possible that the sudo
1644user will be able to execute arbitrary commands as the effective user. Still,
1645under most circumstances this isn’t a problem and is certainly preferable to
1646giving away undue access to root privileges.
1647H A C K
1648#13
1649Using a Makefile to Automate Admin Tasks Hack #13
1650Makefiles make everything (not just gcc) faster and easier
1651You probably know the make command from building projects (probably
1652involving gcc) from source. But not many people also realize that since it
1653keeps track of file modification times, it can be a handy tool for making all
1654sorts of updates whenever arbitrary files are updated.
1655Here’s a Makefile that is used to maintain sendmail configuration files.
1656Listing: Makefile.mail
1657M4= m4
1658CFDIR= /usr/src/sendmail-8.12.5/cf
1659CHMOD= chmod
1660ROMODE= 444
166128 | Server Basics
1662#13 Using a Makefile to Automate Admin Tasks
1663HACK
1664RM= rm -f
1665.SUFFIXES: .mc .cf
1666all: virtusers.db aliases.db access.db sendmail.cf
1667access.db: access.txt
1668makemap -v hash access < access.txt
1669aliases.db: aliases
1670newaliases
1671virtusers.db: virtusers.txt
1672makemap -v hash virtusers < virtusers.txt
1673.mc.cf:
1674$(RM) $@
1675$(M4) ${CFDIR}/m4/cf.m4 $*.mc > $@ || ( $(RM) $@ && exit 1 )
1676$(CHMOD) $(ROMODE) $@
1677With this installed as /etc/mail/Makefile, you’ll never have to remember to
1678run newaliases when editing your sendmail aliases file, or the syntax of
1679that makemap command when you update virtual domain or access con-
1680trol settings. And best of all, when you update your master mc
1681configuration file (you are using mc and not editing the sendmail.cf by
1682hand, right?) then it will build your new .cf file for you—all by simply typ-
1683ing make . Since make keeps track of files that have been recently updated, it
1684takes care of rebuilding only what needs to be rebuilt.
1685Here’s another example, used to push Apache configuration files to
1686another server (say, in a round-robin Apache setup, which you can learn
1687more about in “Distributing Load with Apache RewriteMap” [Hack #99] . Just
1688put this in your /usr/local/apache/conf directory:
1689Listing: Makefile.push
1690#
1691# Makefile to push *.conf to the slave, as needed.
1692#
1693SLAVE= www2.oreillynet.com
1694APACHE= /usr/local/apache
1695RM= /bin/rm
1696TOUCH= /bin/touch
1697SSH= /usr/local/bin/ssh
1698SCP= /usr/local/bin/scp
1699.SUFFIXES: .conf .ts
1700all: test restart sites.ts globals.ts httpd.ts
1701Brute Forcing Your New Domain Name #14
1702Server Basics | 29
1703HACK
1704configtest: test
1705test:
1706@echo -n "Testing Apache configuration: "
1707@$(APACHE)/bin/apachectl configtest
1708restart:
1709$(APACHE)/bin/apachectl restart
1710.conf.ts:
1711@$(RM) -f $@
1712@$(SCP) $*.conf $(SLAVE):$(APACHE)/conf
1713@$(SSH) $(SLAVE) $(APACHE)/bin/apachectl restart
1714@$(TOUCH) $@
1715This example is a little trickier because we’re not actually building any new
1716files, so it’s difficult for make to tell if any of the configuration files have actu-
1717ally changed. We fake out make by creating empty .ts files (short for TimeS-
1718tamp) that only serve to hold the time and date of the last update. If the real
1719files (httpd.conf, sites.conf, or globals.conf) have changed, then we first run an
1720apachectl configtest to verify that we have a good configuration. If all goes well,
1721it will then restart the local Apache, copy the newly changed files over to the
1722slave server, then restart Apache on the slave server. Finally, we touch the rele-
1723vant .ts files so we won’t process them again until the .conf files change.
1724This saves a lot of typing and is significantly quicker (and safer) than doing
1725it all by hand on each update.
1726H A C K
1727#14
1728Brute Forcing Your New Domain Name Hack #14
1729Find exactly the domain you’d like to register, whatever it turns out to be
1730There are many tools available online that will assist in performing whois
1731queries for you to determine if your favorite domain name is still available,
1732and if not, who has registered it. These tools are usually web based and
1733allow you to submit a few queries at a time (and frequently suggest several
1734inane alternatives if your first choice is taken).
1735If you’re not so much interested in a particular name as in finding one that
1736matches a pattern, why not let the command line do the work for you? Sup-
1737pose you wanted to find a list of all words that end in the letters “st”:
1738cat /usr/share/dict/words | grep 'st$' | sed 's/st$/.st/' | \
1739while read i; do \
1740(whois $i | grep -q '^No entries found') && echo $i; sleep 60; \
1741done | tee list_of_st_domains.txt
1742This will obligingly supply you with a visual running tab of all available
1743words that haven’t yet been registered to el Republica Democratica de Sào
174430 | Server Basics
1745#15 Playing Hunt the Disk Hog
1746HACK
1747Tomé e Príncipe (the domain registrar for the st TLD). This example
1748searches the system dictionary and tries to find the whois record for each
1749matching word, one at a time, every 60 seconds. It saves any nonexistent
1750records to a file called list_of_st_domains.txt, and shows you its progress as
1751it runs. Replace that st with any two letter TLD (like us or to) to brute force
1752the namespace of any TLD you like.
1753Some feel that the domain name land grab is almost turning the Internet into
1754a corporate ghetto, but I don’t subscribe to that idea. I actually find the
1755whole situation quite humorous.
1756H A C K
1757#15
1758Playing Hunt the Disk Hog Hack #15
1759Browse your filesystem for heavy usage quickly with a handy alias
1760It always seems to happen late on a Saturday night. You’re getting paged
1761because a partition on one of the servers (probably the mail server) is dan-
1762gerously close to full.
1763Obviously, running a df will show what’s left:
1764rob@magic:~$df
1765Filesystem 1k-blocks Used Available Use% Mounted on
1766/dev/sda1 7040696 1813680 4863600 27% /
1767/dev/sda2 17496684 13197760 3410132 79% /home
1768/dev/sdb1 8388608 8360723 27885 100% /var/spool/mail
1769But you already knew that the mail spool was full (hence, the page that took
1770you away from an otherwise pleasant, non-mailserver related evening). How
1771can you quickly find out who’s hogging all of the space?
1772Here’s a one-liner that’s handy to have in your .profile:
1773alias ducks='du -cks * |sort -rn |head -11'
1774Once this alias is in place, running ducks in any directory will show you the
1775total in use, followed by the top 10 disk hogs, in descending order. It
1776recurses subdirectories, which is very handy (but can take a long time to run
1777on a heavily loaded server, or in a directory with many subdirectories and
1778files in it). Let’s get to the bottom of this:
1779rob@magic:~$cd /var/spool/mail
1780rob@magic:/var/spool/mail$ducks
17818388608 total
17821537216 rob
178355120 phil
178448800 raw
178543175 hagbard
178636804 mal
178730439 eris
1788Fun with /proc #16
1789Server Basics | 31
1790HACK
179130212 ferris
179226042 nick
179322464 rachael
179422412 valis
1795Oops! It looks like my mail spool runneth over. Boy, I have orders of
1796magnitude more mail than any other user. I’d better do something about
1797that, such as appropriate new hardware and upgrade the /var/spool/mail
1798partition. ;)
1799As this command recurses subdirectories, it’s also good for running a peri-
1800odic report on home directory usage:
1801root@magic:/home#ducks
1802[ several seconds later ]
180313197880 total
18042266480 ferris
18051877064 valis
18061692660 hagbard
18071338992 raw
18081137024 nick
18091001576 rob
1810925620 phil
1811870552 shared
1812607740 mal
1813564628 eris
1814For running simple spot checks while looking for disk hogs, ducks can save
1815many keystrokes (although if we called it something like ds, it would save
1816even more, but wouldn’t be nearly as funny).
1817H A C K
1818#16
1819Fun with /proc Hack #16
1820Directly view the kernel’s running process table and system variables
1821The /proc filesystem contains a representation of the kernel’s live process table.
1822By manipulating files and directories in /proc, you can learn about (and fiddle
1823with) all sorts of parameters in the running system. Be warned that poking
1824around under /proc as root can be extraordinarily dangerous, as root has the
1825power to overwrite virtually anything in the process table. One slip of a redi-
1826rector, and Linux will obligingly blow away your entire kernel memory, with-
1827out so much as a “so long and thanks for all the kcore.”
1828Here are some examples of interesting things to do with /proc. In these
1829examples, we’ll assume that you’re using a recent kernel (about 2.4.18 or so)
1830and that you are logged in as root. Unless you’re root, you will only be able
1831to view and modify processes owned by your uid.
183232 | Server Basics
1833#16 Fun with /proc
1834HACK
1835First, let’s take a look at a lightly loaded machine:
1836root@catlin:/proc#ls
18371/ 204/ 227/ 37/ bus/ hermes/ loadavg scsi/ version
18381039/ 212/ 228/ 4/ cmdline ide/ locks self@
18391064/ 217/ 229/ 5/ cpuinfo interrupts meminfo slabinfo
18401078/ 220/ 230/ 6/ devices iomem misc stat
1841194/ 222/ 231/ 698/ dma ioports modules swaps
1842197/ 223/ 232/ 7/ driver/ irq/ mounts sys/
18432/ 224/ 233/ 826/ execdomains kcore net/ sysvipc/
1844200/ 225/ 254/ 827/ filesystems kmsg partitions tty/
1845202/ 226/ 3/ apm fs/ ksyms pci uptime
1846The directories consisting of numbers contain information about every pro-
1847cess running on the system. The number corresponds to the PID. The rest of
1848the files and directories correspond to drivers, counters, and many other
1849internals of the running kernel. The interface operates just as any other file
1850or device in the system, by reading from and writing to each entry as if it
1851were a file. Suppose you want to find out which kernel is currently booted:
1852root@catlin:/proc#cat version
1853Linux version 2.4.18 (root@catlin) (gcc version 2.95.3 20010315 (release))
1854#2 Sat Jun 22 19:01:17 PDT 2002
1855Naturally, you could find much of that out by simply running uname -a,
1856but this way cuts out the middle man (and actually does what uname does
1857internally).
1858Interested in how much RAM we have installed? Take a look at kcore:
1859root@catlin:/proc#ls -l kcore
1860-r-------- 1 root root 201330688 Aug 28 21:39 kcore
1861Looks like we have 192MB installed in this machine (201330688/1024/
18621024 == 192, more or less). Notice the restricted file permissions? That is
1863the system’s defense against anyone attempting to read the memory
1864directly. Of course, you’re root and can do whatever you like. There’s
1865nothing preventing you from running grep or strings on kcore and looking
1866for interesting tidbits. This is the system memory, and things get cached in
1867there until overwritten (making it possible to hunt down accidentally lost
1868data or track down naughty people doing things they oughtn’t). Grovel-
1869ling over kcore is actually not much fun and is typically only a method
1870used by the very desperate (or very bored).
1871Some other notable status files:
1872root@catlin:/proc#cat interrupts
1873CPU0
18740: 34302408 XT-PIC timer
18751: 2 XT-PIC keyboard
18762: 0 XT-PIC cascade
18773: 289891 XT-PIC orinoco_cs
1878D ownload from Wow! eBook <www.wowebook.com>
1879Fun with /proc #16
1880Server Basics | 33
1881HACK
18828: 1 XT-PIC rtc
18839: 13933886 XT-PIC eth0
188410: 25581 XT-PIC BusLogic BT-958
188514: 301982 XT-PIC ide0
1886NMI: 0
1887ERR: 0
1888These are the system counters for every interrupt that has ever been called,
1889its number, and the driver that called it.
1890root@catlin:/proc#cat partitions
1891major minor #blocks name
18928 0 8971292 sda
18938 1 8707198 sda1
18948 2 257040 sda2
18953 64 29316672 hdb
18963 65 29310561 hdb1
1897This is a list of all of the hard disk partitions (and devices) that were discov-
1898ered at boot, along with their respective sizes. Here we have a 9GB SCSI
1899disk, and a 30GB IDE disk. All available partitions are represented here,
1900regardless of whether they are currently mounted (making it a handy refer-
1901ence to see if there are any unmounted disks on the system).
1902Let’s leave the system parameters and take a look at the structure of an indi-
1903vidual process.
1904root@catlin:/proc#cd 1
1905root@catlin:/proc/1#ls -l
1906total 0
1907-r--r--r-- 1 root root 0 Aug 28 22:05 cmdline
1908lrwxrwxrwx 1 root root 0 Aug 28 22:05 cwd -> //
1909-r-------- 1 root root 0 Aug 28 22:05 environ
1910lrwxrwxrwx 1 root root 0 Aug 28 22:05 exe -> /sbin/init*
1911dr-x------ 2 root root 0 Aug 28 22:05 fd/
1912-r--r--r-- 1 root root 0 Aug 28 22:05 maps
1913-rw------- 1 root root 0 Aug 28 22:05 mem
1914lrwxrwxrwx 1 root root 0 Aug 28 22:05 root -> //
1915-r--r--r-- 1 root root 0 Aug 28 22:05 stat
1916-r--r--r-- 1 root root 0 Aug 28 22:05 statm
1917-r--r--r-- 1 root root 0 Aug 28 22:05 status
1918There are three interesting symlinks in this directory. cwd points to the cur-
1919rent working directory of this process (you can, for example, cd /proc/3852/
1920cwd to land in the directory from which process ID 3852 was run.) The exe
1921link points to full path to the binary that was called, and root points to the
1922notion of the root directory that this process has. The root link will almost
1923always be /, unless it has executed a chroot.
1924The cmdline and environ files contain the command line as it was originally
1925called and the processes complete environment. These are separated by NULL
1926characters, so to see them in a more human readable form, try this:
192734 | Server Basics
1928#17 Manipulating Processes Symbolically with procps
1929HACK
1930root@catlin:/proc/1#cat environ |tr '\0' '\n'
1931HOME=/
1932TERM=linux
1933BOOT_IMAGE=catlin
1934(or for a better example, try this on /proc/self/environ, the environment of
1935the currently running process):
1936root@catlin:/proc/1#cat /proc/self/environ |tr '\0' '\n'
1937PWD=/proc/1
1938HOSTNAME=catlin.nocat.net
1939MOZILLA_HOME=/usr/lib/netscape
1940ignoreeof=10
1941LS_OPTIONS= --color=auto -F -b -T 0
1942MANPATH=/usr/local/man:/usr/man:/usr/X11R6/man
1943LESSOPEN=|lesspipe.sh %s
1944PS1=\u@\h:\w\$
1945PS2=>
1946...
1947and so on. This can be tremendously handy for use in shell scripts (or other
1948programs) where you need specific information about running processes.
1949Just use it with care, and remember that unprivileged users can usually only
1950access information about their own processes.
1951In closing, here’s a practical example of one use for /proc. By checking the
1952output of ps against the running process table, you can see if the two agree:
1953#ls -d /proc/* |grep [0-9]|wc -l; ps ax |wc -l
1954This will give you a quick spot check of the number of running processes
1955versus the number that ps actually reports. Many rootkits install a hacked ps
1956that allows a miscreant to hide processes (by simply not displaying them
1957when ps runs). You may hide from ps, but it’s much more difficult to hide
1958from /proc. If the second number is considerably larger than the first (partic-
1959ularly if you run it several times in a row) then you might want to consider
1960taking your box offline for further inspection. Quickly.
1961H A C K
1962#17
1963Manipulating Processes Symbolically
1964with procps Hack #17
1965Signal and renice processes by name, terminal, or username (instead of PID)
1966If you often find yourself running a ps awux |grep something just to find the
1967PID of a job you’d like to kill, then you should take a look at some of the
1968more modern process manipulation packages.
1969Probably the best known process tools package is procps, the same package
1970that includes the Linux version of the ubiquitous top command. The top tool
1971is so tremendously useful and flexible that it deserves its own discussion.
1972Learn more about it in “Monitor System Resources with top” [Hack #58] .
1973Manipulating Processes Symbolically with procps #17
1974Server Basics | 35
1975HACK
1976Among the other nifty utilities included in procps: skill lets you send signals
1977to processes by name, terminal, username, or PID, and snice does the same
1978but renices processes instead of sending them signals.
1979For example, to freeze the user on terminal pts/2:
1980#skill -STOP pts/2
1981To release them from the grip of sleeping death, try this:
1982#skill -CONT pts/2
1983Or to renice all of luser’s processes to 5:
1984#snice +5 luser
1985pkill is similar to skill, but with more formal parameters. Rather than
1986attempting to guess whether you are referring to a username, process name,
1987or terminal, you specify them explicitly with switches. For example, these
1988two commands do the same thing:
1989#skill -KILL rob bash
1990#pkill -KILL -u rob bash
1991pkill may take slightly more typing, but is guaranteed to be unambiguous
1992(assuming that you happened to have a user and a process with the same
1993name, for example).
1994pgrep works just like pkill, but instead of sending a signal to each process, it
1995simply prints the matching PID(s) on STDOUT:
1996$pgrep httpd
19973211
19983212
19993213
20003214
20013215
20023216
2003Finally, vmstat gives you a nice, easily parsed pinpoint measurement of vir-
2004tual memory and cpu statistics:
2005$vmstat
2006procs memory swap io system cpu
2007r b w swpd free buff cache si so bi bo in cs us sy id
20080 0 0 5676 6716 35804 58940 0 0 9 9 7 9 0 0 29
2009If you’d like to watch how usage changes over time, give it a number on the
2010command line. The number represents the number of seconds to pause
2011before displaying the results of another measurement.
2012Learning how to use the lesser known procps utilities can save you lots of
2013typing, not to mention wincing. Use skill or pkill to avoid accidentally
2014mistyping a PID argument to kill, and suddenly bringing sshd (and the wrath
2015of many angry users) down upon your unfortunate sysadmin head.
201636 | Server Basics
2017#18 Managing System Resources per Process
2018HACK
2019See also:
2020• ftp://people.redhat.com/johnsonm/procps/
2021• “Monitor System Resources with top” [Hack #58]
2022• manpages for pkill, pgrep, skill, snice, and vmstat
2023H A C K
2024#18
2025Managing System Resources per Process Hack #18
2026Prevent user processes from running away with all system resources
2027Whether intentionally or accidentally, it is entirely possible for a single user
2028to use up all system resources, leading to poor performance or outright sys-
2029tem failure. One frequently overlooked way to deal with resource hogs is to
2030use the ulimit functionality of bash.
2031To prevent a process (or any of its children) from creating enormous files,
2032try specifying a ulimit -f (with the maximum file size specified in kilobytes).
2033rob@catlin:/tmp$ulimit -f 100
2034rob@catlin:/tmp$yes 'Spam spam spam spam SPAM!' > spam.txt
2035File size limit exceeded
2036rob@catlin:/tmp$ls -l spam.txt
2037-rw-r--r-- 1 rob users 102400 Sep 4 17:05 spam.txt
2038rob@catlin:/tmp$
2039Users can decrease their own limits, but not increase them (as with nice and
2040renice). This means that ulimits set in /etc/profile cannot be increased later
2041by users other than root:
2042rob@catlin:/tmp$ulimit -f unlimited
2043bash: ulimit: cannot modify limit: Operation not permitted
2044Note that nothing is preventing a user from creating many files, all as big as
2045their ulimit allows. Users with this particular temperament should be
2046escorted to a back room and introduced to your favorite LART. Or alter-
2047nately, you could look into introducing disk quotas (although this is usually
2048less than fun to implement, if a simple stern talking to will fix the problem).
2049Likewise, ulimit can limit the maximum number of children that a single
2050user can spawn:
2051rob@catlin:~$cat > lots-o-procs
2052#!/bin/bash
2053export RUN=$((RUN + 1))
2054echo $RUN...
2055$0
2056^D
2057rob@catlin:~$ulimit -u 10
2058rob@catlin:~$./lots-o-procs
20591...
20602...
2061Managing System Resources per Process #18
2062Server Basics | 37
2063HACK
20643...
20654...
20665...
20676...
20687...
20698...
20709...
2071./lots-o-procs: fork: Resource temporarily unavailable
2072rob@catlin:~$
2073This limits the number of processes for a single user across all terminals (and
2074back grounded processes). It has to be this way, because once a process is
2075forked, it disassociates itself from the controlling terminal. (And how would
2076you count it against a given subshell then?)
2077One other very useful ulimit option is -v, maximum virtual memory size.
2078Once this ceiling is reached, processes will exit with Segmentation fault
2079(which isn’t ideal, but will keep the system from crashing as it runs out of
2080RAM and swap). If you have a particularly badly behaving process that
2081shows significant bloat (like Apache + mod_perl and poorly written CGI
2082code, for example) you could set a ulimit to act as an “emergency brake”
2083while debugging the real source of the trouble. Again, specify the limit in
2084kilobytes.
2085To see all available ulimit settings, use -a:
2086rob@catlin:~$ulimit -a
2087core file size (blocks) 0
2088data seg size (kbytes) unlimited
2089file size (blocks) unlimited
2090max locked memory (kbytes) unlimited
2091max memory size (kbytes) unlimited
2092open files 1024
2093pipe size (512 bytes) 8
2094stack size (kbytes) 8192
2095cpu time (seconds) unlimited
2096max user processes 1536
2097virtual memory (kbytes) unlimited
2098You can see that before setting system-wide hard limits, user processes can
2099grow to be quite large. In tcsh, the analogous command you’re after is limit:
2100rob@catlin:~>limit
2101cputime unlimited
2102filesize unlimited
2103datasize unlimited
2104stacksize 8192 kbytes
2105coredumpsize 0 kbytes
2106memoryuse unlimited
2107descriptors 1024
2108memorylocked unlimited
2109maxproc 1536
2110openfiles 1024
211138 | Server Basics
2112#19 Cleaning Up after Ex-Users
2113HACK
2114Setting system resource limits may sound draconian but is a much better
2115alternative to the downward spiral of a user process gone amok.
2116See also:
2117• http://www.tuxedo.org/~esr/jargon/html/entry/LART.html
2118H A C K
2119#19
2120Cleaning Up after Ex-Users Hack #19
2121Make sure you close the door all the way when a user takes their leave
2122It happens. Plans change, companies shift focus, and people move on. At
2123some point, every admin has had to clean up shell access after someone has
2124left the company, happily or otherwise.
2125I am personally of the opinion that if one doesn’t trust one’s users from the
2126beginning, then they will one day prove themselves untrustworthy. (Of
2127course, I’ve never had to admin an open shell server at an ISP, either.) At any
2128rate, building trust with your users from the beginning will go a long way
2129toward being able to sleep at night later on.
2130When you do have to lock old accounts up, it’s best to proceed strategi-
2131cally. Don’t assume that just because you ran a passwd -l that the user in
2132question can’t regain access to your machine. Let’s assume that we’re lock-
2133ing up after an account called luser. Here are some obvious (and some not so
2134obvious) things to check on in the course of cleaning up:
2135passwd -l luser
2136Obviously, locking the user’s password is a good first step.
2137chsh -s /bin/true luser
2138This is another popular step, changing the user’s login shell to something
2139that exits immediately. This generally prevents a user from gaining shell
2140access to the server. But be warned, if sshd is running on this box, and you
2141allow remote RSA or DSA key authentication, then luser can still forward
2142ports to any machine that your server can reach! With a command like this:
2143luser@evil:~$ssh -f -N -L8000:private.intranet.server.com:80 old.server.com
2144luser has just forwarded his local port 8000 to your internal intranet server’s
2145http port. This is allowed since luser isn’t using a password (he is using an
2146RSA key) and isn’t attempting to execute a program on old.server.com
2147(since he specified the -N switch).
2148Obviously, you should remove ~luser/.ssh/authorized_keys* and prevent luser
2149from using his ssh key in the first place. Likewise, look for either of these files:
2150~luser/.shosts
2151~luser/.rhosts
2152Cleaning Up after Ex-Users #19
2153Server Basics | 39
2154HACK
2155This usually isn’t a problem unless you’re running rsh, or have enabled this
2156functionality in ssh. But you never know if a future admin will decide to
2157enable .shosts or .rhosts functionality, so it’s better to remove them now, if
2158they exist.
2159Did luser have any sudo privileges? Check visudo to be sure.
2160How about cron jobs or at jobs?
2161crontab -u luser -e
2162atq
2163For that matter, is luser running any jobs right now?
2164ps awux |grep -i ^luser
2165or as in “Manipulating Processes Symbolically with procps” [Hack #17] , you
2166can use:
2167skill -KILL luser
2168Could luser execute cgi programs from his home directory (or somewhere
2169else)?
2170find ~luser/public_html/ -perm +111
2171What about PHP or other embedded scripting languages?
2172find ~luser ~public_html/ -name '*.php*'
2173Does luser have any email forwarding set up? Forwarders can frequently be
2174made to execute arbitrary programs.
2175less ~luser/.forward
2176grep -C luser /etc/mail/aliases
2177Finally, does luser own any files in strange places?
2178find / -user luser > ~root/luser-files.report
2179One safe (and quick) way of ensuring that all of luser’s personal configura-
2180tion files are invalidated is to mv /home/luser /home/luser.removed . This will
2181keep the contents of luser’s home directory intact, without worrying about
2182having missed other possible points of entry. Note that this will break a
2183legitimate .forward file (and also ~luser/public_html and any other publicly
2184accessible data that luser might have kept in his home directory), so if you
2185go this route be sure to take that into account (say, by adding an appropri-
2186ate entry to the system aliases file, and moving a cleaned version of public_
2187html/ back to /home/luser/public_html/.
2188Look at configuration files for any system software to which luser had
2189access, particularly services that run as privileged users. Even something as
2190innocuous as a user-supplied Apache configuration file could be used to pro-
2191vide shell access later.
219240 | Server Basics
2193#20 Eliminating Unnecessary Drivers from the Kernel
2194HACK
2195This list is by no means exhaustive but is meant to demonstrate that there is
2196a lot more to revoking access than simply locking a user’s password. If this
2197user ever had root access, all bets are off. Access could later be granted by
2198anything from a Trojan system binary to an invisible kernel module to hav-
2199ing simply changed the root password.
2200Get to know your shell users long before the day you have to say goodbye.
2201This job is difficult enough without having to worry about whom you can
2202trust.
2203H A C K
2204#20
2205Eliminating Unnecessary Drivers from the
2206Kernel Hack #20
2207Keep your kernel optimized for quick booting and long-term stability
2208Linux will run on an enormous variety of computer hardware. There is sup-
2209port for all manner of hardware, from critical components (such as hard disk
2210drives and RAM) to more exotic devices (such as USB scanners and video
2211capture boards). The kernel that ships with most Linux distributions aims to
2212be complete (and safe) at the expense of possibly being less efficient than it
2213could be, by including support for as many devices as possible.
2214As your machine boots, take a look at the messages the kernel produces.
2215You may find it probing for all sorts of hardware (particularly SCSI control-
2216lers and Ethernet cards) that you don’t actually have installed. If your distri-
2217bution hides the kernel boot messages, try the dmesg command (probably
2218piped through less) to see what kernel messages were generated at boot.
2219To make your kernel boot quickly and at the same time eliminate the possi-
2220bility that an unnecessary device driver might be causing problems with your
2221installed hardware, you should trim down the drivers that the kernel
2222attempts to load to fit your hardware.
2223There are two schools of thought on managing kernel drivers. Some people
2224prefer to build a kernel with all of the functionality they need built into it,
2225without using loadable kernel modules. Others prefer to build a more light-
2226weight kernel and load the drivers they need when the system boots. Of
2227course, both methods have their advantages and disadvantages.
2228For example, the monolithic kernel (without loadable modules) is guaran-
2229teed to boot, even if something happens to the drivers under /lib/modules.
2230Some admins even prefer to build a kernel with no loadable module support
2231at all, to discourage the possibility of Trojan horse device drivers being
2232loaded by a random miscreant down the road. On the other hand, if a new
2233piece of hardware is added to the system, then you will need to rebuild your
2234kernel to accommodate it.
2235Eliminating Unnecessary Drivers from the Kernel #20
2236Server Basics | 41
2237HACK
2238If you use loadable modules, then you have enormous flexibility in how you
2239load device drivers. You can alter the order that drivers load and even pass
2240parameters to various modules as you load the driver, all without rebooting.
2241The downside is that good copies of the modules must exist under /lib/
2242modules, or else the system can’t load its drivers. When you build a new ker-
2243nel, you’ll need to remember to run a make modules_install , and to keep
2244your kernel module utilities (like modprobe and lsmod) up to date. Building
2245a kernel with loadable modules can also help if your monolithic kernel is too
2246large to boot (by loading extra device drivers after the kernel boots and the
2247filesystems are mounted).
2248Regardless of the method that you choose for your system, you’ll need to
2249build a kernel with the minimal functionality that is required to boot. This
2250includes drivers for the IDE, SCSI, or other bus (maybe floppy disk or even
2251network card?) that your machine boots from. You will also need support
2252for the filesystem that your root partition is installed on (likely ext2, ext3, or
2253reiserfs). Make sure that you build a kernel with support for the amount of
2254RAM that your machine has installed (see “Using Large Amounts of RAM”
2255[Hack #21] ). Select a processor that matches your hardware to be sure that all
2256appropriate optimizations are turned on. Be sure to build an SMP kernel if
2257you have more than one processor.
2258To build a customized kernel, unpack the kernel sources somewhere that
2259has enough room (say, in /usr/local/src/linux). Run a make menuconfig (or, if
2260you’re running X, make xconfig ). Select your drivers carefully (hitting Y for
2261built-in drivers, and M for loadable modules.) Remember that you’re build-
2262ing a kernel for a server and don’t include extraneous drivers. Does your
2263server really need sound support, even if it is built into your motherboard?
2264What about USB? Unless you have a specific use for a particular piece of
2265hardware that is directly related to its job as a server, don’t bother installing
2266a driver for it. You should also consider disabling unused hardware in the
2267BIOS, wherever possible, to conserve system resources and reduce the possi-
2268bility of hardware resource conflicts.
2269After you’re finished selecting which bits of the kernel to build, it will save
2270your configuration to a file called .config at the top of the kernel source tree.
2271Save this file for later, because it will make upgrading your kernel much eas-
2272ier down the road (by copying it to the top of the new kernel tree, and run-
2273ning make oldconfig ). Now build the new kernel (and modules, if
2274applicable), install it, and give it a try. Be sure to try out all of your devices
2275after installing a new kernel, and watch your boot messages carefully. Track
2276down any unexpected warnings or errors now, before they cause trouble at
2277some future date.
227842 | Server Basics
2279#21 Using Large Amounts of RAM
2280HACK
2281Finally, if you’re doing kernel work remotely, don’t forget to build in sup-
2282port for your network devices! There’s nothing quite like the pain of realiz-
2283ing that the network drivers aren’t built just after you issue a shutdown -r
2284now . Hopefully that console is accessible by someone on call (who will dis-
2285cretely boot your old kernel for you, without telling too many of your
2286friends).
2287Building a kernel well-tailored to your hardware can be challenging, but is
2288necessary to make your server run as efficiently as it can. Don’t be discour-
2289aged if it takes a couple of tries to build the perfect Linux kernel, the effort is
2290well worth it.
2291See also:
2292• Running Linux, Fourth Edition (O’Reilly)
2293• “Using Large Amounts of RAM” [Hack #21]
2294H A C K
2295#21
2296Using Large Amounts of RAM Hack #21
2297Be sure that Linux is using all of your available system RAM
2298Linux is capable of addressing up to 64 GB of physical RAM on x86 sys-
2299tems. But if you want to accommodate more than 960 MB RAM, you’ll have
2300to let the system know about it.
2301First of all, your Linux kernel must be configured to support the additional
2302RAM. Typically, the default kernel configuration will address up to 960 MB
2303RAM. If you install more than that in a machine, it will simply be ignored.
2304(The common complaint is that you’ve just installed 1 GB, and yet a ‘free’
2305only reports 960MB, even though it counts to 1024 MB at post time.)
2306The way that the kernel addresses its available system memory is dictated
2307by the High Memory Support setting (a.k.a. the CONFIG_NOHIGH-
2308MEM define.) Depending on the amount of RAM you intend to use, set it
2309accordingly:
2310up to 960MB:off
2311up to 4GB:4GB
2312more than 4GB:64GB
2313Be warned that selecting 64 GB requires a processor capable of using Intel
2314Physical Address Extension (PAE) mode. According to the kernel notes, all
2315Intel processors since the Pentium Pro support PAE, but this setting won’t
2316work on older processors (and the kernel will refuse to boot, which is one
2317reason that it isn’t on by default). Make your selection and rebuild your ker-
2318nel, as in “Eliminating Unnecessary Drivers from the Kernel” [Hack #20] .
2319hdparm: Fine Tune IDE Drive Parameters #22
2320Server Basics | 43
2321HACK
2322Once the kernel is built and installed, you may have to tell your boot loader
2323how much RAM is installed, so it can inform the kernel at boot time (as not
2324every BIOS is accurate in reporting the total system RAM at boot.) To do
2325this, add the mem= kernel parameter in your bootloader configuration. For
2326example, suppose we have a machine with 2GB RAM installed.
2327If you’re using Lilo, add this line to /etc/lilo.conf:
2328append="mem=2048M"
2329If you’re using Grub, try this in your /etc/grub.conf:
2330kernel /boot/vmlinuz-2.4.19 mem=2048M
2331If you’re running loadlin, just pass it on the loadlin line:
2332c:\loadlin c:\kernel\vmlinuz root=/dev/hda3 ro mem=2048M
2333Although, if you’re running loadlin, why are you reading a book on Linux
2334Server Hacks? ;)
2335H A C K
2336#22
2337hdparm: Fine Tune IDE Drive Parameters Hack #22
2338Get the best possible performance from your IDE hardware
2339If you’re running a Linux system with at least one (E)IDE hard drive, and
2340you’ve never heard of hdparm, read on.
2341By default, most Linux distributions use the default kernel parameters when
2342accessing your IDE controller and drives. These settings are very conserva-
2343tive and are designed to protect your data at all costs. But as many have
2344come to discover, safe almost never equals fast. And with large volume data
2345processing applications, there is no such thing as “fast enough.”
2346If you want to get the most performance out of your IDE hardware, take a
2347look at the hdparm(8) command. It will not only tell you how your drives are
2348currently performing, but will let you tweak them to your heart’s content.
2349It is worth pointing out that under some circumstances, these commands
2350CAN CAUSE UNEXPECTED DATA CORRUPTION! Use them at your
2351own risk! At the very least, back up your box and bring it down to single-
2352user mode before proceeding.
2353Let’s begin. Now that we’re in single user mode (which we discussed in
2354“Forgoing the Console Login” [Hack #2] ), let’s find out how well the primary
2355drive is currently performing:
2356hdparm -Tt /dev/hda
2357You should see something like:
2358/dev/hda:
2359Timing buffer-cache reads: 128 MB in 1.34 seconds =95.52 MB/sec
2360Timing buffered disk reads: 64 MB in 17.86 seconds = 3.58 MB/sec
236144 | Server Basics
2362#22 hdparm: Fine Tune IDE Drive Parameters
2363HACK
2364What does this tell us? The -T means to test the cache system (i.e., the mem-
2365ory, CPU, and buffer cache). The -t means to report stats on the disk in
2366question, reading data not in the cache. The two together, run a couple of
2367times in a row in single-user mode, will give you an idea of the performance
2368of your disk I/O system. (These are actual numbers from a PII/350/128M
2369Ram/EIDE HD; your numbers will vary.)
2370But even with varying numbers, 3.58 MB/sec is pathetic for the above hard-
2371ware. I thought the ad for the HD said something about 66 MB per sec-
2372ond!!?!? What gives?
2373Let’s find out more about how Linux is addressing this drive:
2374#hdparm /dev/hda
2375/dev/hda:
2376multcount = 0 (off)
2377I/O support = 0 (default 16-bit)
2378unmaskirq = 0 (off)
2379using_dma = 0 (off)
2380keepsettings = 0 (off)
2381nowerr = 0 (off)
2382readonly = 0 (off)
2383readahead = 8 (on)
2384geometry = 1870/255/63, sectors = 30043440, start = 0
2385These are the defaults. Nice, safe, but not necessarily optimal. What’s all
2386this about 16-bit mode? I thought that went out with the 386!
2387These settings are virtually guaranteed to work on any hardware you might
2388throw at it. But since we know we’re throwing something more than a dusty,
23898-year-old, 16-bit multi-IO card at it, let’s talk about the interesting options:
2390multcount
2391Short for multiple sector count. This controls how many sectors are
2392fetched from the disk in a single I/O interrupt. Almost all modern IDE
2393drives support this. The manpage claims:
2394When this feature is enabled, it typically reduces operating system overhead
2395for disk I/O by 30-50%. On many systems, it also provides increased data
2396throughput of anywhere from 5% to 50%.
2397I/O support
2398This is a big one. This flag controls how data is passed from the PCI bus
2399to the controller. Almost all modern controller chipsets support mode 3,
2400or 32-bit mode w/sync. Some even support 32-bit async. Turning this
2401on will almost certainly double your throughput (see below).
2402unmaskirq
2403Turning this on will allow Linux to unmask other interrupts while pro-
2404cessing a disk interrupt. What does that mean? It lets Linux attend to
2405hdparm: Fine Tune IDE Drive Parameters #22
2406Server Basics | 45
2407HACK
2408other interrupt-related tasks (i.e., network traffic) while waiting for your
2409disk to return with the data it asked for. It should improve overall sys-
2410tem response time, but be warned: not all hardware configurations will
2411be able to handle it. See the manpage.
2412using_dma
2413DMA can be a tricky business. If you can get your controller and drive
2414using a DMA mode, do it. However, I have seen more than one machine
2415hang while playing with this option. Again, see the manpage.
2416Let’s try out some turbo settings:
2417#hdparm -c3 -m16 /dev/hda
2418/dev/hda:
2419setting 32-bit I/O support flag to 3
2420setting multcount to 16
2421multcount = 16 (on)
2422I/O support = 3 (32-bit w/sync)
2423Great! 32-bit sounds nice. And some multi-reads might work. Let’s re-run
2424the benchmark:
2425#hdparm -tT /dev/hda
2426/dev/hda:
2427Timing buffer-cache reads: 128 MB in 1.41 seconds =90.78 MB/sec
2428Timing buffered disk reads: 64 MB in 9.84 seconds = 6.50 MB/sec
2429Hmm, almost double the disk throughput without really trying! Incredible.
2430But wait, there’s more: we’re still not unmasking interrupts, using DMA, or
2431even a using decent PIO mode! Of course, enabling these gets riskier. The
2432manpage mentions trying Multiword DMA mode2, so let’s try this:
2433#hdparm -X34 -d1 -u1 /dev/hda
2434Unfortunately this seems to be unsupported on this particular box (it hung
2435like an NT box running a Java application) So, after rebooting it (again in
2436single-user mode), I went with this:
2437#hdparm -X66 -d1 -u1 -m16 -c3 /dev/hda
2438/dev/hda:
2439setting 32-bit I/O support flag to 3
2440setting multcount to 16
2441setting unmaskirq to 1 (on)
2442setting using_dma to 1 (on)
2443setting xfermode to 66 (UltraDMA mode2)
2444multcount = 16 (on)
2445I/O support = 3 (32-bit w/sync)
2446unmaskirq = 1 (on)
2447using_dma = 1 (on)
2448D ownload from Wow! eBook <www.wowebook.com>
244946 | Server Basics
2450#22 hdparm: Fine Tune IDE Drive Parameters
2451HACK
2452And then checked:
2453#hdparm -tT /dev/hda
2454/dev/hda:
2455Timing buffer-cache reads: 128 MB in 1.43 seconds =89.51 MB/sec
2456Timing buffered disk reads: 64 MB in 3.18 seconds =20.13 MB/sec
245720.13 MB/sec. A far cry from the miniscule 3.58 with which we started.
2458Did you notice how we specified the -m16 and -c3 switch again? That’s
2459because it doesn’t remember your hdparm settings between reboots. Be sure
2460to add the above line to your /etc/rc.d/* scripts once you’re sure the system is
2461stable (and preferably after your fsck runs; running an extensive filesystem
2462check with your controller in a flaky mode may be a good way to generate
2463vast quantities of entropy, but it’s no way to administer a system. At least
2464not with a straight face).
2465If you can’t find hdparm on your system (usually in /sbin or /usr/sbin), get it
2466from the source at http://metalab.unc.edu/pub/Linux/system/hardware/.
246747
2468Chapter 2
2469C H A P T E R T W O
2470Revision Control
2471Hacks #23–36
2472If you take administration seriously, you will likely spend untold hours fine-
2473tuning your system to behave just so. Unfortunately, it isn’t always possible
2474to define when a given application is “working” and when it is “broken”;
2475usually functionality is much more finely graded, somewhere in the grey
2476area between up and down. Even making a small configuration change can
2477bring about subtle effects that aren’t seen for days (or weeks) down the road.
2478This is where Revision Control can be a powerful ally. Beyond providing
2479simple backups, any revision control package worth its weight in bits will
2480keep track of the changes in any file, when it was changed, who changed it,
2481and why the change was made. This information can be of untold value
2482when working on a complex system, particularly if more than one person is
2483responsible for its upkeep. By keeping critical configuration files in systems
2484such as RCS or CVS, you will be able to “roll back” to any previous revision
2485of a file, and analyze the differences between what was present then and the
2486version you have online right now.
2487In this section, the goal is to provide the specific syntax you need to make
2488effective use of RCS and CVS, with useful examples of how you might use
2489them. While RCS is extremely handy for maintaining file revisions on a single
2490machine, CVS offers extensive archiving and merging capabilities, keeping a
2491central repository somewhere on the network. Understandably, CVS is a
2492much more complicated tool than RCS. Hacks 23 through 25 offer a crash
2493course in RCS, while 26 through 36 will get you up to speed quickly on CVS,
2494from simple commands to setting up your own anonymous CVS repository.
2495Naturally, this chapter assumes that you’re already comfortable working
2496with files from the command line, even if you haven’t worked withRCS or
2497CVS before. If you’re looking for in-depth information about CVS, take a
2498look at the excellent book Open Source Development with CVS by Karl Fogel
2499(CoriolisOpen Press).
250048 | Revision Control
2501#23 Getting Started with RCS
2502HACK
2503H A C K
2504#23
2505Getting Started with RCS Hack #23
2506Let RCS manage your system files, and keep a revision history
2507RCS is a revision control system, useful for keeping multiple revisions of con-
2508figuration files, shell scripts, and any other text files you have lying around.
2509Unlike CVS, there is no functionality for using RCS with remote machines; in
2510RCS, everything is intended to be kept in the local filesystem.
2511RCS keeps all its revisions in a directory called RCS/ (under the directory
2512you’re currently in). To start a new RCS repository, just make the directory:
2513root@catlin:/etc#mkdir RCS
2514Now, we’ll need to initialize a file into the new RCS repository before work-
2515ing with it. Let’s work with syslog.conf:
2516root@catlin:/etc#ci -i syslog.conf
2517RCS/syslog.conf,v <-- syslog.conf
2518enter description, terminated with single '.' or end of file:
2519NOTE: This is NOT the log message!
2520>>Here's the syslog.conf for Catlin
2521initial revision: 1.1
2522done
2523The initial ci -i does a “check in and initialize,” to give RCS a good first copy
2524of the file. RCS prompts for an initial description, and then removes the file
2525from the current directory. You probably don’t want to leave it like that for
2526too long, so after you’ve initialized the file in RCS, check it out:
2527root@catlin:/etc#co syslog.conf
2528RCS/syslog.conf,v --> syslog.conf
2529revision 1.1
2530done
2531That copies the file back into the current directory. But we’re not ready to
2532start working on it just yet. First, let’s check it out with a lock to prevent
2533other people from making updates while we work on it.
2534root@catlin:/etc#co -l syslog.conf
2535RCS/syslog.conf,v --> syslog.conf
2536revision 1.1 (locked)
2537done
2538Now you can edit the file to your heart’s content. Once you’re finished with
2539it, check it back in and unlock it with ci -u:
2540root@catlin:/etc#ci -u syslog.conf
2541syslog.conf,v <-- syslog.conf
2542new revision: 1.3; previous revision: 1.2
2543enter log message, terminated with single '.' or end of file:
2544>>Added remote logging to the new security log host
2545>>.
2546done
2547Checking Out a Previous Revision in RCS #24
2548Revision Control | 49
2549HACK
2550Be sure to use meaningful log entries, because you won’t remember what
2551you meant by “made a couple of edits” in six months, when your code starts
2552doing strange things.
2553If you have trouble checking anything in or out, you’ve probably forgotten
2554the -l (on co) or -u (on ci) switch at some point. You can usually start over
2555by making a backup copy, then checking the file back out again, and copy-
2556ing it back, assuming that you’ve just edited syslog.conf:
2557root@catlin:/etc#ci -u syslog.conf
2558syslog.conf,v <-- syslog.conf
2559ci: syslog.conf,v: no lock set by root
2560root@catlin:/etc#
2561Uh-oh. Better back it up and start again from scratch:
2562root@catlin:/etc#cp syslog.conf syslog.conf.backup
2563root@catlin:/etc#co -l syslog.conf
2564syslog.conf,v --> syslog.conf
2565revision 1.4 (locked)
2566done
2567root@catlin:/etc#cp syslog.conf.backup syslog.conf
2568root@catlin:/etc#ci -u syslog.conf
2569syslog.conf,v <-- syslog.conf
2570new revision: 1.5; previous revision: 1.4
2571enter log message, terminated with single '.' or end of file:
2572>>commented out the FIFO line
2573>>.
2574done
2575We’ll see some more fun things to do with RCS in the next couple of hacks.
2576H A C K
2577#24
2578Checking Out a Previous Revision in RCS Hack #24
2579Save yourself with RCS revisions
2580Inevitably, you will break things so horribly that you have no hope of get-
2581ting them back again. Suppose you’ve just done a complicated edit to your
2582httpd.conf, and you get the error message that strikes fear (and occasionally
2583heartburn) in the heart of admins everywhere:
2584root@catlin:/usr/local/apache/conf#../bin/apachectl restart
2585apachectl restart: httpd not running, trying to start
2586apachectl restart: httpd could not be started
2587Oh, dear. What have you done? With the web server down, people aren’t
2588even getting your friendly 404 page, they’re seeing Connection Refused
2589errors. You could dig into httpd.conf desperately with vi to figure out exactly
2590what you (or perhaps the admin before you) did to deserve this.
2591But if you’re using RCS, things aren’t so bleak. To see what has changed
2592between your last checkout and this one, use rcsdiff:
259350 | Revision Control
2594#25 Tracking Changes with rcs2log
2595HACK
2596root@catlin:/usr/local/apache/conf#rcsdiff httpd.conf
2597===================================================================
2598RCS file: RCS/httpd.conf,v
2599retrieving revision 1.1
2600diff -r1.1 httpd.conf
2601458c458
2602< ErrorLog /usr/local/apache/logs/error_log
2603---
2604> ErrorLog :wq/usr/local/apache/logs/error_log
2605There we are. Evidently, a :wq accidentally got inserted at line 458. Ah, must
2606have missed that ESC key. Make your fix, start Apache, and check it back in.
2607But what if the changes were more extensive than that? Suppose you make a
2608big change to a configuration file on Tuesday morning, and gremlins start to
2609manifest themselves around Thursday afternoon. How can you get back to a
2610known good copy?
2611Simple: check out a particular revision using the -r switch:
2612root@catlin:/etc#co -l -r1.2 syslog.conf
2613syslog.conf,v --> syslog.conf
2614revision 1.2
2615done
2616Presto, you’re back to the state you were in when you checked the file in at
2617rev 1.2. Remember with RCS, it makes sense to save early and often.
2618H A C K
2619#25
2620Tracking Changes with rcs2log Hack #25
2621See at a glance who’s been editing your files, and why
2622RCS really shows its strength when more than one person needs to edit a
2623particular file. Having multiple admins can sometimes lead to a blame game
2624when things aren’t working properly. In these cases, it’s obvious that some-
2625body changed something, but nobody will admit to it. If you keep track of
2626changes in RCS, then it’s simple to tell who’s been changing what:
2627root@www:/usr/local/apache/htdocs#rcs2log index.html
26282002-08-14 rob <rob@mouse>
2629* index.html: meeting announcement
26302002-07-30 rob <rob@mouse>
2631* index.html: gre.tunnel announcement
26322002-07-12 sderle <sderle@mouse>
2633* index.html: added Marin and shuffled around a bit.
26342002-07-10 rob <rob@mouse>
2635Tracking Changes with rcs2log #25
2636Revision Control | 51
2637HACK
2638* index.html: meeting announcement + v0.81
26392002-07-01 jim <jim@mouse>
2640* index.html: *** empty log message ***
26412002-06-20 rob <rob@mouse>
2642* index.html: meeting reminder
2643Hmm, what’s that empty log message doing there? It might be worth talk-
2644ing to Jim (and possibly doing an rcsdiff between it and the previous revi-
2645sion). But how can you tell which versions correspond to which log entries?
2646Try the -v switch on rcs2log:
2647root@www:/usr/local/apache/htdocs#rcs2log -v index.html
26482002-08-14 rob <rob@mouse>
2649* index.html 1.54: meeting announcement
26502002-07-30 rob <rob@mouse>
2651* index.html 1.53: gre.tunnel announcement
26522002-07-12 sderle <sderle@mouse>
2653* index.html 1.52: added Marin and shuffled around a bit.
26542002-07-10 rob <rob@mouse>
2655* index.html 1.51: meeting announcement + v0.81
26562002-07-01 jim <jim@mouse>
2657* index.html 1.50: *** empty log message ***
26582002-06-20 rob <rob@mouse>
2659* index.html 1.49: meeting reminder
2660Now do a diff between whichever revisions you like:
2661root@www:/usr/local/apache/htdocs#rcsdiff -r1.49 -r1.50 index.html
2662===================================================================
2663RCS file: RCS/index.html,v
2664retrieving revision 1.49
2665retrieving revision 1.50
2666diff -r1.49 -r1.50
2667199a200,202
2668> <dt><a href="http://labs.google.com/">Google Labs</a></dt>
2669> <dd>Take a look at the strange and wondrous things that Google is up to...
2670</dd>
2671>
267252 | Revision Control
2673#26 Getting Started with CVS
2674HACK
2675So it was just a link addition. Still, it might be easy to just hit that ^D when
2676prompted, but it’s generally considered bad form to not make any log entry.
2677At any rate, once you master the simple ci, co, rcsdiff, and rcs2log com-
2678mands, you have some very flexible revision control tools at your disposal.
2679Use them well, and they’ll probably save your life one day. Or in the very
2680least, they’ll save the next best thing: your data.
2681H A C K
2682#26
2683Getting Started with CVS Hack #26
2684The ins and outs of the Concurrent Versioning System
2685Concurrent Versioning System (CVS) is a system for managing simulta-
2686neous development of files. It is commonly used in large programming
2687projects and is also useful to system administrators, technical writers, and
2688anyone who needs to manage files.
2689CVS stores files in a central repository. It is set up to be accessible to all
2690users of the files, using standard Unix permissions. Commands are given to
2691“check out” a copy of a file for development and “commit” changes back to
2692the repository. It also scans the files as they are moved to and from the
2693repository, to prevent one person’s work from overwriting another’s.
2694This system ensures that a history of the file is retained, which is extremely
2695useful when the boss decides he wants a feature you trashed months ago. It
2696also ensures that backing up the repository is enough to backup a project
2697(providing all necessary files are kept in repository).
2698Typical Uses
2699CVS is designed for developers, either individually or in teams. For individu-
2700als, CVS provides a repository from which you can work from home, the
2701office, or the client site without having to haul disks around. It also pro-
2702vides version control, allowing rollbacks without loss of data. For teams, it
2703also keeps a record of who changed which lines of a file and prevents direct
2704overwriting of each other’s work.
2705System administrators can keep configuration files in CVS. You can make a
2706change, cvs commit it, test it. If it fails, roll back the change, even if you only
2707discover the failure six months down the track.
2708Administrators can keep a CVS tree of the configurations for server farms.
2709Adding a new server? Just use cvs to checkout the config tree for that type
2710of server. Committing all changes also helps you keep track of who did
2711what, when.
2712In this section, we’ll take a look at what you need to get CVS up and run-
2713ning quickly, both as a client and as a server.
2714Getting Started with CVS #26
2715Revision Control | 53
2716HACK
2717Creating a Repository
2718The repository needs to be hosted on a machine with sufficient disk space to
2719store all your files and all the data for the changes you expect. As a rule of
2720thumb, put the repository on a partition with enough room for three times
2721the expected final size of the module. Then use the “Scotty principle” and
2722double your estimate—the project will expand. If you intend to store binary
2723files, multiply by ten. After your first project, you’ll have a feel for how
2724much space to allow.
2725First, ensure that all CVS users have valid user accounts and can access the
2726repository machine from all the machines they intend to use.
2727Now create your repository root directory. Repositories are often stored in
2728/home/cvsroot or /usr/local/cvsroot. Use cvs init to set up the directory as a
2729CVS repository.
2730cvs -d /home/cvsroot init
2731Debian Linux has a script, cvs-makerepos, that will build a repository based
2732on pre-existing Debian configuration scripts. See man cvs-makerepos for
2733more information and man cvsconfig for an automated system for configur-
2734ing a Debian CVS repository.
2735In general, most CVS servers tend to use a single repository, with multi-
2736ple Modules contained within it. For example, your repository might live
2737in /home/cvsroot, but could contain many projects (such as tools, email,
2738dns, myproj, etc.).
2739Importing a New Module
2740Before loading your project into CVS, consider its structure. Moving or
2741renaming files and directories can damage the CVS record of the files’ his-
2742tory. Deleting a directory can cost you the record of all its files and subdirec-
2743tories. For this reason, CVS has no facility for moving or renaming files and
2744directories, or removing directories.
2745Make your initial directory structure—even if it’s just one directory. Add any
2746initial files you want. From within the root directory of your project, use the
2747command cvs -d repository import nameofmodule vendortag releasetag .
2748For most cases, you will not need to know about vendor tags and release
2749tags. CVS requires them to be present, but you can simply use the name of
2750the module as the vendor tag, and the current version as the release tag.
2751/home/jenn$cd example
2752/home/jenn/example$cvs -d /home/cvsroot import example example_project ver_
27530-1
2754D ownload from Wow! eBook <www.wowebook.com>
275554 | Revision Control
2756#27 CVS: Checking Out a Module
2757HACK
2758Environment Variables
2759If you are only working with one repository, and would like to save yourself
2760some typing, set the $CVSROOT environment variable to the repository’s
2761full path, like this:
2762export CVSROOT=/home/cvsroot/
2763With $CVSROOT set, you can now omit the -d repository portion of any
2764CVS command. If you need to temporarily work with another repository,
2765either reset your $CVSROOT, or use the -d switch to override your existing
2766$CVSROOT setting.
2767If the CVS repository resides on a different machine, then you need to tell
2768CVS how to access it. For remote repositories, $CVSROOT is set to :method:
2769user@host:path . It should look something like :ext:jenn@cvs.example.com.
2770au:/home/cvs .
2771By default, CVS uses rsh to access a repository on a remote machine. While
2772this may have been a very logical transport choice a few years ago, running
2773rsh on a production server is now generally considered a Very Bad Idea. For-
2774tunately, the ext access method tells CVS to look in the $CVS_RSH environ-
2775ment variable, and run that program to connect to the machine that contains
2776the repository. Most people set the $CVS_RSH variable to ssh, and set up ssh
2777client keys (as in “Quick Logins with ssh Client Keys” [Hack #66] ) on the reposi-
2778tory machine. This greatly simplifies working with your repository, as you
2779won’t have to type in a password every time you make an update.
2780Incidentally, the other common access method is pserver, and these days is
2781generally used for accessing anonymous public CVS repositories. For an
2782example of how to work with anonymous CVS (and set up your own public
2783repository), see “Quick Logins with ssh Client Keys” [Hack #66] .
2784See Also:
2785• CVS in a Nutshell (O’Reilly)
2786• Anonymous CVS
2787• Quick logins with ssh keys
2788H A C K
2789#27
2790CVS: Checking Out a Module Hack #27
2791How to get a working copy of a CVS Module
2792CVS stores the files in a central repository, but users work from a working
2793copy of a file.
2794CVS: Updating Your Working Copy #28
2795Revision Control | 55
2796HACK
2797Make a directory to do your work in (I tend to use ~/cvs), then cd into that
2798directory. The checkout syntax is cvs checkout module To checkout a mod-
2799ule called example, try cvs checkout example .
2800The checkout will put a copy of that module’s files and subdirectories into
2801your cvs directory.
2802cvs$ls
2803example
2804cvs$cd example; ls
2805CVS src
2806cvs/example$cd CVS; ls
2807Entries Repository Root
2808The cvs directory is a special directory that CVS uses for its own purposes.
2809CVS/Entries lists files and subdirectories CVS knows about. CVS/Reposi-
2810tory contains the path to the corresponding directory in the repository.
2811CVS/Root contains the path to the repository, so you won’t need to use
2812the -d repository-path option again for these files.
2813Note that CVS/Root overrides the $CVSROOT environment variable, so if
2814you change the repository, you should check out the module again. Alter-
2815nately, if you’re in the middle of a big edit when you realize that your reposi-
2816tory needs to be changed, try this perl one-liner, as shown in “Global Search
2817and Replace with Perl” [Hack #73] :
2818cvs/src$ls
2819CVS Makefile sample.h sample.c
2820The src directory contains the source files for the example project. sample.c,
2821sample.h, and Makefile are ordinary files in the working copy. In the reposi-
2822tory, they are stored in a format that tracks the changes.
2823H A C K
2824#28
2825CVS: Updating Your Working Copy Hack #28
2826Receiving recent changes in your CVS Module
2827Every day before you start work, and any time someone else may have made
2828and committed changes, cd into your working directory and run cvs update.
2829This checks your working copies against the repository files and imports any
2830changed files for you. cvs update -d also gives you any new directories.
2831Update reports on the status of each file as it checks it:
2832U File updated successfully
2833A File added but not yet committed (need to run a cvs commit)
2834R File removed but not yet committed (need to run a cvs commit)
2835M File modified in your working directory; the file in the repository was
2836changed and your working directory file was older than the last time
283756 | Revision Control
2838#29 CVS: Using Tags
2839HACK
2840CVS checked it or the repository had changes that the system could
2841safely merge
2842C Conflict between the repository copy, and your copy, which requires
2843human intervention
2844? File the file is in your working directory but not the repository; CVS
2845doesn’t know what to do with it
2846H A C K
2847#29
2848CVS: Using Tags Hack #29
2849Tagging a Module revision in CVS
2850Tags assign a symbolic name to a revision of a file or a set of files. cvs tag
2851tags the repository versions of all the files in the current working directory
2852and its subdirectories.
2853The cvs tag command is based on a timestamp. It assigns the symbolic name
2854to the version of the file or directory that is closest to and older than the
2855timestamp; it does not look at the working directory.
2856Note that CVS doesn’t allow the ‘.’ character in tags. Specify the filename
2857after the tag name, as in the following example:
2858cvs tag tagname filename
2859cvs tag tagname
2860Otherwise, the command tags the repository versions of all files in the cur-
2861rent working directory:
2862cvs tag -c tagname
2863Use the -c option to cause the process to abort if the repository copy differs
2864from the working copy:
2865cvs/example$cvs tag release-1-0 src/sample.c
2866cvs/example/src$cvs tag release-1-0
2867cvs/example/src$cvs tag -c release-1-0
2868To retrieve a tagged version, use the -r flag to checkout or update. If you
2869checkout or update in your usual working directory, the tagged version will
2870overwrite the existing files, as in:
2871cvs checkout -r tagname
2872cvs update -r tagname
2873Here’s an example:
2874cvs$mkdir example-rel-1.0
2875cvs/example-rel-1.0$cvs checkout -r release-1-0
2876or:
2877cvs/example$cvs update -r release-1-0
2878CVS: Making Changes to a Module #30
2879Revision Control | 57
2880HACK
2881H A C K
2882#30
2883CVS: Making Changes to a Module Hack #30
2884Checking in your changes in CVS
2885Once your files are checked out, edit them and compile them normally.
2886Apply the updates to the repository with cvs commit. This command needs
2887to be run higher in the hierarchy than all the files you have changed—you
2888can always run it from the base of your working copy.
2889You can also cvs commit filename , which will commit a single file or recur-
2890sively commit a directory.
2891Different project teams have different opinions on how often to do a cvs
2892commit. Good rules of thumb include “every time you have a clean com-
2893pile,” and “every day before lunch and before you leave.”
2894cvs/example$cvs commit
2895cvs commit: Examining .
2896cvs commit: Examining src
2897jenn@cvs.sample.com.au's password:
2898CVS examines each directory and subdirectory below the current working
2899directory. Any file that CVS knows will be checked for changes. If your cvs
2900repository is not on the local machine, CVS will ask for a password for the
2901remote machine, unless you have already set up your ssh host keys (as
2902shown in “Quick Logins with ssh Client Keys” [Hack #66] ) and have elimi-
2903nated the need for passwords.
2904CVS then opens whichever editor is the default in your environment—based
2905on the $CVSEDITOR or $EDITOR environment variables. Add change-
2906notes for the appropriate files, as in:
2907CVS:------------------------------------------------------------------
2908CVS: Enter Log. Lines beginning with 'CVS:' are removed automatically
2909CVS:
2910CVS: Committing in .
2911CVS:
2912CVS: Modified Files:
2913CVS: example/src/sample.h example/src/sample.c
2914CVS:------------------------------------------------------------------
2915I strongly recommend meaningful change-notes—if you’re trying to do a
2916rollback and all you have are messages that say “fixed a few bugs,” you’ll
2917not know which version to roll back to without using cvs diff.
2918If there is a potential conflict, cvs commit fails. Correct this by running a cvs
2919update on the repository—CVS will attempt to merge the files, and will ask
2920for human help if it cannot do this without losing data:
2921cvs server: Up-to-date check failed for 'cvs_intro.html'
2922cvs [server aborted]: correct above errors first!
2923cvs commit: saving log message in /tmp/cvst7onmJ
292458 | Revision Control
2925#31 CVS: Merging Files
2926HACK
2927H A C K
2928#31
2929CVS: Merging Files Hack #31
2930Resolving update conflicts in CVS
2931If CVS can’t merge a modified file successfully with the copy in the reposi-
2932tory, it announces the conflict in the output of cvs update. The original file is
2933stored in .#file.version in the file’s working directory, and the results of the
2934merge are stored as the original filename:
2935cvs/example$cvs update
2936jenn@cvs.example.com.au's password:
2937cvs server: Updating .
2938RCS file: /home/cvs/example/sample.c,v
2939retrieving revision 1.3
2940retrieving revision 1.4
2941Merging differences between 1.3 and 1.4 into sample.c
2942rcsmerge: warning: conflicts during merge
2943cvs server: conflicts found in sample.c
2944C sample.c
2945CVS writes the merge with the conflicting lines surrounded by CVS tags.
2946CVS can’t automatically merge conflicts where the same line is changed in
2947both versions of a file:
2948<<<<<<< sample.c
2949Deliberately creating a conflict.
2950=======
2951Let's make a conflict.
2952>>>>>>> 1.4
2953H A C K
2954#32
2955CVS: Adding and Removing Files and
2956Directories Hack #32
2957Adding and removing files in your Module
2958Files that CVS doesn’t know what to do with are reported with a question
2959mark after the commit process and during a cvs update. They need to be
2960added to the repository before CVS will recognize them.
2961Use cvs add filename to mark a new file for inclusion. CVS doesn’t put the
2962file in the repository until you do a cvs commit.
2963Directories are added with the same command. Files within a directory can’t
2964be added until the directory is added.
2965Removing Files
2966To mark a file for removal from the working copies, use cvs remove filename .
2967Before CVS will remove a file from the repository, you have to actually delete
2968it from the filesystem. CVS doesn’t actually remove the file entirely, it puts the
2969file in a special subdirectory in the repository called Attic.
2970CVS: Branching Development #33
2971Revision Control | 59
2972HACK
2973Removing Directories
2974Directories cannot be removed from the repository using CVS commands.
2975If a directory is no longer required, empty the directory with cvs remove,
2976and use cvs update -P and cvs checkout -P when retrieving a working
2977copy. The -P flag ensures that empty directories are not retrieved.
2978If you must, you can remove a directory by using rmdir on the repository.
2979Do this on a copy of the repository first and check that you aren’t breaking
2980anything: If the directory in the repository has an Attic subdirectory, you
2981will lose archived copies of files formerly stored there.
2982If you remove a directory from the repository, you should have all your users
2983remove their existing working copies, and check out fresh copies of the
2984module.
2985H A C K
2986#33
2987CVS: Branching Development Hack #33
2988Setting up a development branch in CVS
2989If you need to fix a bug in an older version of your code without changing
2990current code, or modify a configuration set for staging servers without modi-
2991fying the set for your production servers, you might need to branch your
2992modules. A branch allows storage and retrieval of a variation of the main
2993module, without affecting the main module. Changes on the branch can be
2994merged in later. To make a branch, use cvs tag -b branchtag , as in:
2995cvs/example$cvs tag -b release-1-0-patches
2996Retrieve a branch using either checkout or update. Checkout will create a
2997new directory for the branch, and update will overwrite your current work-
2998ing directory with the branch.
2999cvs checkout -r branchtag
3000cvs update -r branchtag
3001Example:
3002cvs/example-rel-1.0$cvs checkout -r release-1-0-patches
3003cvs/example$cvs update -r release-1-0-patches
3004Branches can be merged back into the main trunk, using the conflict resolu-
3005tion system invoked by cvs update and cvs commit.
3006cvs checkout module
3007cvs update -j branchtag
3008Example:
3009/tmp/example$cvs checkout example
3010/tmp/example$cvs update -j release-1-0-patches
301160 | Revision Control
3012#34 CVS: Watching and Locking Files
3013HACK
3014Or in a single command:
3015cvs checkout -j branchtag module
3016The following example:
3017/tmp/example$cvs checkout -j release-1-0-patches example
3018resolves any conflicts the system reports, then use cvs commit.
3019H A C K
3020#34
3021CVS: Watching and Locking Files Hack #34
3022Setting up an email watch on files in CVS
3023Unlike many versioning systems, CVS doesn’t have file locking—it doesn’t
3024prevent simultaneous editing of files. However, you can set files to be
3025watched, so CVS will mail watchers when a file is being edited. If files are
3026being watched, developers need to use cvs edit and cvs unedit to release a file
3027for editing. Unwatched files can be edited without notifying CVS in any
3028way.
3029To set up files for being watched, use:
3030cvs watch on (files)
3031cvs watch off (files)
3032To set yourself as a watcher, use:
3033cvs watch add (files)
3034cvs watch remove (files)
3035or:
3036cvs watch add -a edit|unedit|commit|all (files)
3037cvs watch remove -a edit|unedit|commit|all (files)
3038The special CVS file notify determines what occurs when a watched file is
3039changed. It defaults to sending mail to the user’s username on the CVS
3040server. If your users have other addresses, set up the file users in the reposi-
3041tory’s CVSROOT directory. Entries should be in the format user:email , one
3042to a line.
3043jenn:jenn@cvs.example.com.au
3044H A C K
3045#35
3046CVS: Keeping CVS Secure Hack #35
3047Protecting your users and your code base in CVS
3048Remote Repositories
3049If the repository is on the local machine, both access and security are fairly
3050straightforward. You can set the $CVSROOT environment variable to the root
3051directory of the CVS repository, or call checkout with the -d directory option.
3052CVS: Keeping CVS Secure #35
3053Revision Control | 61
3054HACK
3055If the repository is on a remote machine, it is necessary to tell CVS which
3056machine it is on, and what method will be used to access the machine. There
3057are several methods available, but for security and simplicity, I prefer SSH.
3058The syntax for defining a remote $CVSROOT is :method:[[user]:
3059[password]@]hostname[:[port]]:/path/to/repository . For example:
3060:ext:jenn@cvs.example.com.au:/usr/local/cvsroot
3061(Note that info cvs disagrees slightly with what my copy of CVS actually
3062does. I have included the syntax that works for me—a colon between the
3063host and the path. Use the syntax that works on your system.)
3064To use SSH, we use the ext method. This method uses an external-to-CVS rsh
3065or rsh-compatible program to communicate with the CVS server. To tell CVS
3066to use SSH instead of rsh, set the environment variable $CVS_RSH to SSH.
3067Ensure that SSH is set up on the server and on all clients, that SSH keys are
3068generated and that users have usernames and passwords on both machines. If
3069the usernames are the same, the user@ part of the CVSROOT string is not
3070necessary. If a standard SSH port is used, the port is not necessary.
3071cvs -d :ext:cvs.example.com.au:/usr/local/cvsroot checkout sample<code>
3072Permissions
3073The files in the repository are all read-only. Permissions to those files
3074shouldn’t be changed. To control access, use the directory permissions. Most
3075administrators make a group for the people who should have access to the
3076module, and ensure that the group has write access for the directory.
3077If using a remote repository, set the root directory of the module setgid to
3078ensure that all directories beneath it are made with the correct permissions.
3079If using a local repository, $CVSUMASK can be set to control the permis-
3080sions of files and directories in the repository.
3081Developer Machines
3082Securing the project involves securing the repository and securing all
3083checked out copies—typically your developer’s machines. It’s not enough to
3084ensure that the repository is safe and all transmissions are properly
3085encrypted if someone can walk into your developer’s office on his day off
3086and burn a CD of your code. Maintain the usual physical and Net-based
3087security foryour development machines, prototype and demonstration cop-
3088ies, and any other places the code gets checked out to.
308962 | Revision Control
3090#36 CVS: Anonymous Repositories
3091HACK
3092H A C K
3093#36
3094CVS: Anonymous Repositories Hack #36
3095Create your own read-only anonymous CVS repository
3096Creating an Anonymous Repository
3097The pserver access method allows users to login to remote repositories by
3098supplying a username and password that is checked against a password file
3099that CVS maintains, or against the system’s /etc/passwd. The pserver method
3100unfortunately passes all credentials in the clear, so at best an intervening
3101snooper might capture a login to your CVS repository, or at worst could
3102compromise the entire machine. For this reason, most people use ssh as
3103their transport when using remote repositories. Please see “CVS: Keeping
3104CVS Secure” [Hack #35] , for further details.
3105Obviously, if you want to provide read-only access to your source tree to the
3106public, using ssh as your transport would be unnecessary (and impractical).
3107This is where pserver shows its real usefulness: allowing easy anonymous
3108repository access.
3109Before we get anonymous CVS running, first we’ll need to set the repository
3110machine up to use the traditional pserver method.
3111Installing pserver
3112As we’ll be using pserver for anonymous CVS access, we’ll need to create a
3113user that has no permissions to write to anything in your repository. Create
3114a user called anonymous (or if 9-letter usernames bother you, cvsanon is
3115another common choice.) Set its shell to /bin/true, its home directory to
3116something innocuous (like /var/empty), put it in its own group, and lock its
3117password (a passwd -l is a quick way to do that.) This user will never login;
3118it’s just a placeholder account for CVS to setuid to later.
3119Next we’ll create a password file for CVS. Put the following in a file called
3120CVSROOT/passwd, under your repository directory:
3121anonymous:23MLN3ne5kvBM
3122If you created a user account called cvsanon, use this line instead:
3123anonymous:23MLN3ne5kvBM:cvsanon
3124In the CVS passwd file, the left hand entry is the CVS login name, followed
3125by the encrypted password, and finally ending in an optional system login
3126name to map the CVS login to. If it is omitted, CVS will look up the first
3127entry in the system’s password file and use that. The encrypted string is the
3128word anonymous.
3129CVS: Anonymous Repositories #36
3130Revision Control | 63
3131HACK
3132To be absolutely sure that the anonymous user can’t ever make changes to
3133the repository, add the line anonymous to the CVSROOT/readers file under
3134your repository. This file flags any users contained with in it (one per line) as
3135read-only users.
3136Now we want to tell CVS to never accept regular system users under the
3137pserver method (to prevent wayward users from habitually using their sys-
3138tem logins with pserver.) This is set up in CVSROOT/config, under your
3139repository directory. Uncomment the line that says SystemAuth=no and then
3140only users specified in CVSROOT/passwd can login using pserver. Note that
3141this will have no effect on CVS users that use ext and ssh; they still use sim-
3142ple filesystem permissions for access control, and never consult the CVS
3143passwd file.
3144Finally, we can tell the system to actually accept pserver connections. CVS
3145doesn’t ever run as a daemon; it expects to be launched from inetd. It runs
3146on port 2401, so add the following line to your /etc/services:
3147pserver 2401/tcp
3148And add this entry to /etc/inetd.conf:
3149pserver stream tcp nowait root /usr/bin/cvs cvs --allow-root=/usr/local/
3150cvsroot pserver
3151Substitute your repository’s directory for /usr/local/cvsroot. Now just skill
3152-HUP inetd (and see “Getting Started with RCS” [Hack #23] if you don’t have
3153skill installed) and away you go.
3154Using a Remote pserver
3155To test your new anonymous repository, first set your $CVSROOT environ-
3156ment variable to this:
3157:pserver:anonymous@your.machine.here:/usr/local/cvsroot
3158Before you can do a cvs checkout, you’ll first need to login to pserver with
3159cvs login . When prompted for a password, enter anonymous. Now proceed
3160with cvs checkout module , and you should see an update as normal. Verify
3161that you can’t perform a cvs checkin, and you’re ready to publish your anon-
3162ymous CVS details to the world.
316364
3164Chapter 3
3165C H A P T E R T H R E E
3166Backups
3167Hacks #37–44
3168Be assured that one day, the unthinkable will happen. Hard drives wear.
3169Tapes stretch and break. You might even accidentally pass the wrong switch
3170to a command (or be in the wrong directory, or on the wrong machine) and
3171suddenly find yourself cursing at having hit the Enter key, as your data evap-
3172orates into the ether. It is at these moments that you learn the value of a
3173properly designed and executed backup solution.
3174While we can get you started with some interesting approaches to backing
3175up your data, this section is by no means comprehensive. As a systems
3176administrator, you should design a backup plan that fits the needs of your
3177organization, implement the plan, and continually revise it as necessary as
3178time goes on. Definitely do not assume that just because you set up a cron
3179job to run a backup that the backups themselves are good (or even that the
3180job itself is running properly). Watch your logs. Buy new backup media
3181before the old media develops problems. Make it a policy to test your back-
3182ups by doing a full restore from backup as often as you deem necessary to
3183sleep well at night. And even then, consider keeping another copy of your
3184data, off-site if need be. For an extremely detailed look at everything that
3185goes into building a secure data backup policy, be sure to check out Unix
3186Backups & Recovery by W. Curtis Preston (O’Reilly).
3187In this section, we’ll take a look at some methods for keeping copies of your
3188data ready for when that day of reckoning comes. Every installation is differ-
3189ent, and each site needs its own degree of backup functionality (ranging
3190from weekly differentials to 5-minute snapshots). While we won’t try to
3191define the perfect backup policy for your site, we will see some tools you can
3192use to develop your own.
3193Backing Up with tar over ssh #37
3194Backups | 65
3195HACK
3196H A C K
3197#37
3198Backing Up with tar over ssh Hack #37
3199Copy arbitrary bits of the filesystem between servers using ssh and tar
3200Shuffling files between servers is simple with scp:
3201root@inky:~#scp some-archive.tgz blinky:/
3202Or even copying many files at once:
3203root@pinky:~/tmp#scp clyde:/usr/local/etc/* .
3204But scp isn’t designed to traverse subdirectories and preserve ownership and
3205permissions. Fortunately, tar is one of the very early (and IMHO, most bril-
3206liant) design decisions in ssh to make it behave exactly as any other stan-
3207dard Unix command. When it is used to execute commands without an
3208interactive login session, ssh simply accepts data on STDIN and prints the
3209results to STDOUT. Think of any pipeline involving ssh as an easy portal to
3210the machine you’re connecting to. For example, suppose you want to
3211backup all of the home directories on one server to an archive on another:
3212root@inky~#tar zcvf - /home | ssh pinky "cat > inky-homes.tgz"
3213Or even write a compressed archive directly to a tape drive on the remote
3214machine:
3215root@blinky~#tar zcvf - /var/named/data | ssh clyde "cat > /dev/tape"
3216Suppose you wanted to just make a copy of a directory structure from one
3217machine directly into the filesystem of another. In this example, we have a
3218working Apache on the local machine but a broken copy on the remote side.
3219Let’s get the two in sync:
3220root@clyde:~#cd /usr/local
3221root@clyde:/usr/local#tar zcf - apache/ \
3222| ssh pacman "cd /usr/local; mv apache apache.bak; tar zpxvf -"
3223This moves /usr/local/apache/ on pacman to /usr/local/apache.bak/, then cre-
3224ates an exact copy of /usr/local/apache/ from clyde, preserving permissions and
3225the entire directory structure. You can experiment with using compression on
3226both ends or not (with the z flag to tar), as performance will depend on the
3227processing speed of both machines, the speed (and utilization) of the net-
3228work, and whether you’re already using compression in ssh.
3229Finally, let’s assume that you have a large archive on the local machine and
3230want to restore it to the remote side without having to copy it there first
3231(suppose it’s really huge, and you have enough space for the extracted copy,
3232but not enough for a copy of the archive as well):
3233root@blinky~#ssh pinky "cd /usr/local/pacland; tar zpvxf -" \
3234< really-big-archive.tgz
323566 | Backups
3236#38 Using rsync over ssh
3237HACK
3238Or alternately, from the other direction:
3239root@pinky:/usr/local/pacland#ssh blinky "cat really-big-archive.tgz" \
3240| tar zpvxf -
3241If you encounter problems with archives created or extracted on the remote
3242end, check to make sure nothing is written to the terminal in your ~/.bashrc
3243on the remote machine. If you like to run /usr/games/fortune or some other
3244program that writes to your terminal, it’s a better idea to keep it in ~/.bash_
3245profile or ~/.bash_login than in ~/.bashrc, because you’re only interested in
3246seeing what fortune has to say when there is an actual human being logging
3247in and definitely not when remote commands are executed as part of a pipe-
3248line. You can still set environment variables or run any other command you
3249like in ~/.bashrc, as long as those commands are guaranteed never to print
3250anything to STDOUT or STDERR.
3251Using ssh keys to eliminate the need for passwords makes slinging around
3252arbitrary chunks of the filesystem even easier (and easily scriptable in cron, if
3253you’re so inclined). See the hacks listed below for an example of how to make
3254it very easy (and secure) to connect to any of your servers with ssh.
3255See also:
3256• “Quick Logins with ssh Client Keys” [Hack #66]
3257• “Turbo-mode ssh Logins” (#67)
3258• “Using ssh-Agent Effectively” [Hack #68]
3259H A C K
3260#38
3261Using rsync over ssh Hack #38
3262Keep large directory structures in sync quickly with rsync
3263While tar over ssh is ideal for making remote copies of parts of a filesys-
3264tem, rsync is even better suited for keeping the filesystem in sync between
3265two machines. Typically, tar is used for the initial copy, and rsync is used
3266to pick up whatever has changed since the last copy. This is because tar
3267tends to be faster than rsync when none of the destination files exist, but
3268rsync is much faster than tar when there are only a few differences between
3269the two filesystems.
3270To run an rsync over ssh, pass it the -e switch, like this:
3271root@rover:~#rsync -ave ssh greendome:/home/ftp/pub/ /home/ftp/pub/
3272Notice the trailing / on the file spec from the source side (on greendome.)
3273On the source specification, a trailing / tells rsync to copy the contents of the
3274directory, but not the directory itself. To include the directory as the top
3275level of whatever is being copied, leave off the /:
3276root@village:~#rsync -ave ssh bcnu:/home/six .
3277Archiving with Pax #39
3278Backups | 67
3279HACK
3280This will keep a copy of the ~root/six/ directory on village in sync with what-
3281ever is present on bcnu:/home/six/.
3282By default, rsync will only copy files and directories, but not remove them
3283from the destination copy when they are removed from the source. To keep
3284the copies exact, include the --delete flag:
3285six@jammer:~/public_html#rsync -ave ssh --delete greendome:~one/reports .
3286Now when old reports are removed from ~one/reports/ on greendome,
3287they’re also removed from ~six/public_html/reports/ on jammer, every time
3288this command is run. If you run a command like this in cron, leave off the v
3289switch. This will keep the output quiet (unless rsync has a problem running,
3290in which case you’ll receive an email with the error output).
3291Using ssh as your transport for rsync traffic has the advantage of encrypting
3292the data over the network and also takes advantage of any trust relation-
3293ships you already have established using ssh client keys. For keeping large,
3294complex directory structures in sync between two machines (especially
3295when there are only a few differences between them), rsync is a very handy
3296(and fast) tool to have at your disposal.
3297See also:
3298• man rsync
3299• “Quick Logins with ssh Client Keys” [Hack #66]
3300• “Using ssh-Agent Effectively” [Hack #68]
3301• “Automated Snapshot-Style Incremental Backups with rsync” [Hack #42]
3302H A C K
3303#39
3304Archiving with Pax Hack #39
3305Make easy, portable archives using pax
3306pax stands for “portable archive exchange,” as it was designed specifically to
3307allow portability between different versions of Unix. There’s also a bit of
3308wry humor in the name, as pax attempts to bring some “peace” to the long-
3309standing battle over which is better: tar or cpio. The pax utility can be used
3310to create either type of archive, and during a restore, it automatically detects
3311the type of archive for you. We’ll start with some examples of basic pax
3312usage, then move on to some fancier stuff.
3313Creating Archives
3314To back up the contents of your home directory, invoke write mode using
3315the w switch:
3316cd
3317pax -wf home.pax .
3318D ownload from Wow! eBook <www.wowebook.com>
331968 | Backups
3320#39 Archiving with Pax
3321HACK
3322In this example, I went to my home directory with cd, then told pax to write
3323( w ) to a file ( f ) named home.pax the contents of the current directory ( . ).
3324When you use pax, it’s very important to remember to include that f switch
3325to indicate the name of the archive you’d like to create. If you forget the f,
3326weird characters will be sent to your screen, accompanied by horrible,
3327pained noises. Also, if you want to watch as pax does its thing, simply add
3328the v, or verbose, switch to the switch portion of the command.
3329To see what type of file you’ve just created, use the file command:
3330file home.pax
3331home.pax: POSIX tar archive
3332To see the contents of that archive, tell pax which archive file you’d like to
3333view, using the f switch:
3334pax -f home.pax |more
3335Since my archive file is rather large, I piped this output to the more command
3336so I could read the contents of the archive one page at a time. If you also
3337include the v switch, you’ll get ls -l–type output of the archive contents. Again,
3338don’t forget to specify the name of the archive with the f switch, or nothing
3339will happen, except that you’ll lose your prompt until you press Ctrl-c.
3340Expanding Archives
3341To restore (or use read mode on) an archive, first cd into the destination
3342directory, then use the r switch. For example, I’ll restore the backup named
3343home.pax into the test subdirectory of my home directory:
3344cd test
3345pax -rvf ~/home.pax
3346The pax utility can also restore tar and cpio archives. It is able to automati-
3347cally detect the correct format for you; however, you should use the file util-
3348ity before attempting the restore to determine whether or not the archive is
3349compressed. If it is, you’ll need to include the z switch.
3350As an example, I have a file called backup.old located in my home directory
3351(~). I’ll first use the file utility:
3352file backup.old
3353backup: gzip compressed data, deflated, last modified:
3354Sat Aug 17 14:21:12 2002, os: Unix
3355Since this backup is compressed, I’ll use this command to restore it to the
3356test directory:
3357cd test
3358pax -rvzf ~/backup.old
3359Archiving with Pax #39
3360Backups | 69
3361HACK
3362I have another file in my home directory called backup:
3363file ~/backup
3364backup: cpio archive
3365This file isn’t compressed, so i’ll restore it, like so:
3366pax -rvf ~/backup
3367The fact that the first backup happened to be a tar archive and the second a
3368cpio archive didn’t confuse pax; however, I would have received some
3369strange error messages if I had forgotten to inform pax that the first archive
3370was compressed.
3371Interactive Restores
3372You can do some pretty funky things when restoring with pax. For exam-
3373ple, you can do an interactive rename/restore by including the i switch. Issu-
3374ing the following command:
3375pax -rif ~/backup
3376will start an interactive restore of the archive named backup into the current
3377directory. In interactive mode, pax will display the name of each file, one at
3378a time, and prompt you to either rename it as it’s restored, restore it with the
3379original name, or to skip it and not restore it:
3380ATTENTION: pax interactive file rename operation.
3381drwxr-xr-x Aug 17 15:08 .
3382Input new name, or a "." to keep the old name, or a "return" to skip this
3383file.
3384Input >
3385Skipping file.
3386Here, I pressed Enter as I didn’t want to change the name of “ . ” or the cur-
3387rent directory:
3388ATTENTION: pax interactive file rename operation.
3389drwxr-xr-x Jul 26 16:10 file1
3390input new name, or a "." to keep the old name, or a "return" to skip this
3391file.
3392input >old
3393Processing continues, name changed to: old
3394ATTENTION: pax interactive file rename operation.
3395-rw-r--r-- Jun 11 00:20 file2
3396input new name, or a "." to keep the old name, or a "return" to skip this
3397file.
3398input > .
3399Processing continues, name unchanged.
3400You’ll note that I changed the name of file1 to old and kept file2 as is. A list-
3401ing of the restored directory will show two files: one named old and one
3402named file2.
340370 | Backups
3404#39 Archiving with Pax
3405HACK
3406Recursively Copy a Directory
3407One of the most powerful features of pax is that it is able to very quickly
3408copy a complete directory structure to another portion of your hard drive,
3409using copy mode. To use copy mode:
34101. cd into the source directory
34112. Ensure the destination directory exists; if it doesn’t, use mkdir to create it
34123. Issue this command: pax -rw . destination_directory
3413Note that you don’t include the f switch in copy mode, as an archive file
3414doesn’t get created. Instead, the old directory structure is directly recreated
3415into the new directory structure. This can be much easier to remember than
3416the tar equivalent:
3417tar cf - . | (cddestination_directory; tar vpxf -)
3418Also note that you never want to do this:
3419cd
3420mkdir test
3421pax -rw . test
3422In the previous example, I cd’d into my home directory, made a subdirec-
3423tory named test, then invoked copy mode. In doing so, I ended up in an end-
3424less loop of test subdirectories, each containing the contents of my home
3425directory. If I hadn’t interrupted this cycle with a Ctrl-c, pax would have
3426continued ad infinitum, where infinitum is defined as the point where I run
3427out of disk space. That’s what this section of man pax refers to.
3428Warning: the destination directory must not be one of the file operands or a
3429member of a file hierarchy rooted at one of the file operands. The result of a
3430copy under these conditions is unpredictable.
3431However, this works beautifully and almost instantaneously:
3432su
3433Password:
3434cd ~user1/big_project
3435mkdir ~user2/big_project
3436chown user2 ~user2/big_project
3437pax -rw . ~user2/big_project
3438Voila, the entire big_project directory structure is now also in the second
3439user’s home directory. When using copy mode, you’ll have to become the
3440superuser as you’ll be copying out of your home directory, so you can avoid
3441the endless loop situation. If you have to make the new directory, it will be
3442owned by root; if need be, use the chown command like I did to ensure that
3443it has the desired ownership before doing the copy operation. You’ll also
3444want to take a look at man pax first to see how you want to handle the per-
3445missions of the copied directory structure.
3446Archiving with Pax #39
3447Backups | 71
3448HACK
3449It is also possible to interactively copy a directory structure by including the i
3450switch:
3451pax -rwi . ~user2/big_project
3452Similarly to the previous interactive example, pax will display each file-
3453name, one at a time, so you can decide which files to copy over and which
3454files to rename as you do so.
3455Incremental Backups
3456Now, let’s do something useful with the pax command. I’ll demonstrate
3457how to create an incremental backup system. In this example, the user
3458“genisis” would like to back up any changes she made to her home direc-
3459tory on a daily basis.
3460First, I’ll become the superuser to create a directory to hold the backups:
3461su
3462Password:
3463mkdir /usr/backups
3464I’ll then create a subdirectory and give the user “genisis” ownership of that
3465subdirectory:
3466mkdir /usr/backups/genisis
3467chown genisis /usr/backups/genisis
3468I’ll then leave the superuser account and as the user “genisis,” cd into my
3469home directory:
3470exit
3471cd
3472I’ll then do a full backup of my home directory and save it to an archive file
3473called Monday:
3474pax -wvf /usr/backups/genisis/Monday .
3475Now that I have a full backup, I can take daily incremental backups to just
3476back up each day’s changes. So when I’m finished with my work on Tues-
3477day, I’ll issue this command:
3478pax -wv -T 0000 -f /usr/backups/genisis/Tuesday .
3479Notice that I included the time switch ( -T ) and specified a time of midnight
3480( 0000 ). This tells pax to only back up the files that have changed since mid-
3481night, so it will catch all of the files that changed today. On Wednesday, I’ll
3482repeat that command but will change the archive name to Wednesday.
3483If you have the disk space and want to keep backups for longer than a week,
3484modify your archive names to something like: Aug01, Aug02, etc. It’s still a
3485good idea to do a full backup once a week, followed by incremental back-
3486ups the other days of that week. If disk space is an issue, include the z switch
348772 | Backups
3488#40 Backing Up Your Boot Sector
3489HACK
3490so the backups will be compressed. Also note that the T switch can be much
3491pickier than I’ve demonstrated; see man pax for the details.
3492Skipping Files on Restore
3493To restore all of the files except file3, use this command:
3494pax -rvf ~/backup -c './file3'
3495The c switch is the exception switch. Note that your exception pattern (in
3496my case, file3) needs to be enclosed in single quotes (the key next to your
3497Enter key). Either use the literal pattern like I did (to pax, this file is known
3498as ./file3, not file3) or use a wildcard, like so:
3499pax -rvf ~/backup -c '*file3'
3500If you use a wildcard (*) at the beginning of your pattern as in the above
3501example, you will exclude all files that end with “file3”—for example: file3,
3502myfile3, thatfile3.
3503You can also specify which file to restore by using the n, or pattern match-
3504ing, switch. The following will just restore file2:
3505pax -rvf ~/backup -n './file2'
3506The n switch differs from the c switch in that it will only restore the first file
3507that matches the pattern. This means that this command will not restore
3508file3, myfile3, and thatfile3:
3509pax -rvf ~/backup -n '*file3'
3510Since file3 is the first file to match the expression, it will be the only file that
3511will be restored.
3512The c and n switches are also useful when creating an archive; use them to
3513specify which file you’d like to back up, or which file(s) you don’t want to
3514back up.
3515See also:
3516• Original article: http://www.onlamp.com/pub/a/bsd/2002/08/22/freeBSD_
3517Basics.html
3518• pax source code (part of the ast-open package), http://www.research.att.
3519com/sw/download/
3520H A C K
3521#40
3522Backing Up Your Boot Sector Hack #40
3523Keep a copy of your boot sector packed away for a rainy day
3524Installing a boot loader (such as LILO) can be trickier than one might like.
3525Particularly when using IDE hardware, it’s easy to get yourself into trouble
3526Backing Up Your Boot Sector #40
3527Backups | 73
3528HACK
3529and work a system into a state that makes it impossible to boot without a
3530rescue disk.
3531One common mistake when using IDE hardware is to install a kernel on a
3532partition that extends beyond the 1024th cylinder. The symptom is very
3533strange, because a machine will boot fine at first, but installing a kernel later
3534(after the machine has been used for some time) makes LILO throw an error
3535and will refuse to come up on the next boot. This can be a very confusing
3536symptom, since “it used to work.” Most likely, the kernel that was installed
3537when the system was built the first time happens to reside on a space on
3538disk before cylinder 1024. After the system software and user data are
3539installed, the disk begins to fill up. When the disk contains about 500 MB
3540(or 1GB on some BIOS) of data, any new kernels will necessarily lie (at least
3541in part) beyond cylinder 1024—which is inaccessible to the BIOS at boot
3542time.
3543Modern versions of LILO will refuse to install such a kernel, but some
3544older versions simply throw a warning, and install anyway. One way to
3545deal with this is to make a small (say, 10 MB) partition at the beginning of
3546your install process (as /dev/hda1, the first partition on the disk), and
3547mount it under /boot. Now when you install new kernels, always copy
3548them to /boot, and they are guaranteed to work with your BIOS, as they
3549will necessarily be contained well before the 1024 cylinder limit.
3550At any rate, installing the boot loader shouldn’t be taken lightly. Once you
3551have your boot loader up and working as you want it, you should consider
3552making a backup of the entire boot sector of your boot drive.
3553For IDE:
3554dd if=/dev/hda of=bootsector.bin bs=512 count=1
3555For SCSI:
3556dd if=/dev/sda of=bootsector.bin bs=512 count=1
3557Alternately, you can copy the boot sector directly to a floppy disk:
3558dd if=/dev/hda of=/dev/fd0 bs=512 count=1
3559Be extremely careful when specifying your if and of statements, as confusing
3560them could obliterate your boot sector. It’s probably a better idea to create a file
3561and copy it to removable media as you would any other file. Trusting a floppy
3562disk with critical system data may be a fun party trick, but after your first (or
3563thousandth) floppy fails at a critical moment, you’ll wish you had another copy.
3564Now if you ever end up with a munged boot sector, you can restore it
3565quickly (after booting from rescue media) like this:
3566dd if=bootsector.bin of=/dev/hda
356774 | Backups
3568#41 Keeping Parts of Filesystems in sync with rsync
3569HACK
3570Or if you backed it up to floppy directly, against medical advice:
3571dd if=/dev/fd0 of=/dev/hda bs=512 count=1
3572Naturally, substitute sda for hda if you’re using SCSI.
3573See also:
3574• Lilo’s README (or other format of documentation under doc/ from the
3575lilo distribution)
3576H A C K
3577#41
3578Keeping Parts of Filesystems in sync with
3579rsync Hack #41
3580Use rsync over ssh to mirror exactly what you want, to any number of servers
3581For O’Reilly’s web publishing system, we built a web “cluster” that spreads
3582the work of one machine across several, increasing both performance and
3583reliability. But how do you keep the actual data in the filesystem of each
3584server in sync with each other?
3585One method is to use NFS to provide a common filesystem for your data.
3586While making it simple to update content across all of the servers simulta-
3587neously, NFS has some significant drawbacks. File locking on NFS is notori-
3588ously tricky. Performance tuning on NFS is generally considered something
3589of a black art. (When it works, it’s very good, and when it doesn’t...that’s
3590why we’re on call, right?) But probably the biggest drawback to a mono-
3591lithic NFS server is that it introduces a single point of failure for your entire
3592enterprise. If the NFS server becomes unavailable (or overloaded) for any
3593reason, all of the machines dependent on it will suffer.
3594One alternative approach is to asynchronously update servers using a tool
3595such as rsync. With the cost of disk space at an all-time low, having a locally
3596cached copy of your data makes sense: not only will the single point of fail-
3597ure be eliminated, but the files will be served much faster from the local disk
3598than they could be from network storage.
3599It is very straightforward to add an rsync job to cron on each of the web serv-
3600ers, containing something like this:
3601rsync -ae ssh master.machine.com:/usr/local/apache/htdocs/ \
3602/usr/local/apache/htdocs/
3603Assuming that you have your ssh keys set up in advance (see “Quick Logins
3604with ssh Client Keys” [Hack #66] ), this will update the local Apache document
3605root with the current copy on master.machine.com over an encrypted ssh ses-
3606sion. As long as updates are always made to master.machine.com, they will
3607be copied faithfully to each of your other servers on the next pass.
3608Keeping Parts of Filesystems in sync with rsync #41
3609Backups | 75
3610HACK
3611The biggest drawback to this approach is that updates must happen asyn-
3612chronously. If this job runs every five minutes, then the copy will be at most
3613five minutes old (and about three minutes old, on average). As long as your
3614application can tolerate the “be there in a minute” nature of rsync’ing large
3615file structures, this method can buy you huge wins in terms of actual server
3616performance and reliability. One strong caveat to using a bare rsync in cron
3617is that you must take care that the job finishes before the next one is run.
3618Otherwise, if a server gets particularly busy, it can enter a downward spiral
3619where launching a new rsync drives up the load, which makes the running
3620rsync take longer, which means that it is still running when the next one
3621runs, which drives up the load.
3622Using rsync has a couple of advantages over using tar for this application.
3623Using tar over ssh (see “Backing Up with tar over ssh” [Hack #37] ) is generally
3624quicker when making a full copy of a directory tree from scratch, but rsync
3625is much faster when only a few files have changed on the master copy. It
3626runs an analysis pass first to determine which files have been updated and
3627then only transfers the changes.
3628But suppose that our web cluster has more complicated requirements. Assume
3629that we are attempting to divide the load (see “Distributing Load with Apache
3630RewriteMap” [Hack #99] ) of serving pages into two groups: Application Servers
3631(which run a heavy mod_perl-enabled Apache, driving our content manage-
3632ment system) and lightweight front-end Apache servers, which only serve cop-
3633ies of static data (such as images and file archives). In this case, it would be a
3634waste of space to copy the entire document tree to all of the servers, since the
3635front-end machines will only ever serve a few particular types of files.
3636Using the exclude-from feature of rsync, you can supply a file that specifies
3637what parts of the filesystem rsync will consider when syncing. Create a file that
3638looks something like this in /usr/local/etc/balance.front for your static servers:
3639### Stuff we definitely don't want to mirror.
3640#
3641- logs/
3642### the entire document root
3643#
3644+ /usr/
3645+ /usr/local/
3646+ /usr/local/apache/
3647+ /usr/local/apache/htdocs/
3648+ /usr/local/apache/htdocs/**
3649### user public_html directories
3650#
3651+ /home/
3652+ /home/*/
3653+ /home/*/public_html/
365476 | Backups
3655#41 Keeping Parts of Filesystems in sync with rsync
3656HACK
3657+ /home/*/public_html/**
3658# Images, archives, etc.
3659#
3660+ *.jpg
3661+ *.gif
3662+ *.png
3663+ *.pdf
3664+ *.mp3
3665+ *.zip
3666+ *.tgz
3667+ *.gz
3668# Exclude everything else.
3669#
3670- *
3671And create a similar file for syncing to your Application Servers:
3672### Stuff we definitely don't want to mirror.
3673#
3674- logs/
3675- *.tmp
3676- *.swp
3677### the entire document root
3678#
3679+ /usr/
3680+ /usr/local/
3681+ /usr/local/apache/
3682+ /usr/local/apache/htdocs/
3683+ /usr/local/apache/htdocs/**
3684# Exclude everything else.
3685#
3686- *
3687Now, create a list of each type of server (front and back), one machine per
3688line. Call the files /usr/local/etc/servers.front and /usr/local/etc/servers.back,
3689respectively.
3690For example, put this in servers.front:
3691tiberius
3692caligula
3693and this in servers.back:
3694augustus
3695claudius
3696germanicus
3697posthumous
3698castor
3699Finally, rather than calling rsync directly from cron on each of your web
3700servers, try this bit of shell code in cron on the authoritative master machine.
3701Keeping Parts of Filesystems in sync with rsync #41
3702Backups | 77
3703HACK
3704Listing: Balance-push.sh
3705#!/bin/bash
3706#
3707# balance-push - Push content from the master server (localhost)
3708# to multiple front- and back-end servers, in parallel.
3709#
3710# $FRONT_END lists the servers that receive the front-end (e.g. static
3711content) updates.
3712#
3713FRONT_END=$(cat /usr/local/etc/servers.front)
3714# $BACK_END lists the hosts that receive the full back-end (e.g. everything)
3715updates.
3716#
3717BACK_END=$(cat /usr/local/etc/servers.back)
3718# $TARGET specifies the filesystem root on the remote host to push to.
3719# Normally, you want this to be /, unless you're doing testing.
3720#
3721TARGET=/
3722# $EXCLUDE specifies the prefix of the per-mode rsync exclude files.
3723# For example, if your exclude files are /usr/local/etc/balance.front and
3724# /usr/local/etc/balance.back, set this to "/usr/local/etc/balance". The
3725# per-mode extensions will be added.
3726#
3727EXCLUDE=/usr/local/etc/balance
3728# $LOCK_DIR specifies a path to put the lock files in.
3729#
3730LOCK_DIR=/var/tmp
3731######## Ignore the shell functions behind the curtain. ########
3732PATH=/bin:/usr/bin:/usr/local/bin
3733lock () {
3734local lockfile="$LOCK_DIR/balance.$1.lock"
3735if [ -f $lockfile ]; then
3736if kill -0 $(cat $lockfile); then
3737echo "$0 appears to be already running on $1."
3738echo "Please check $lockfile if you think this is in error."
3739exit 1
3740else
3741echo "$0 appears to have completed for $1 without cleaning up its lockfile."
3742fi
3743fi
3744echo $$ > $lockfile
3745}
374678 | Backups
3747#41 Keeping Parts of Filesystems in sync with rsync
3748HACK
3749unlock () {
3750rm -f $LOCK_DIR/balance.$1.lock
3751}
3752push_files () {
3753local mode=$1 host=$2
3754if [ ! "$mode" -o ! -r "$EXCLUDE.$mode" ]; then
3755echo "$0 $$: mode unset for $host!"
3756return
3757fi
3758if [ ! "$host" ]; then
3759echo "$0 $$: host unset for push $mode!"
3760return
3761fi
3762lock $host
3763rsync --archive --rsh=ssh --delete --ignore-errors --whole-file \
3764--exclude-from="$EXCLUDE.$mode" / ${host}:${TARGET}
3765unlock $host
3766}
3767push_tier () {
3768local mode=$1 host_list=$2
3769for host in $host_list; do
3770$SHELL -c "push_files $mode $host" &
3771done
3772}
3773export -f lock unlock push_files
3774export TARGET EXCLUDE LOCK_DIR PATH
3775[ "$FRONT_END" ] && push_tier front "$FRONT_END"
3776[ "$BACK_END" ] && push_tier back "$BACK_END"
3777#
3778# Fin.
3779#
3780This script (call it balance-push) will manage your rsync jobs for you, ensur-
3781ing that servers don’t “lap” themselves on each run. It will push the proper
3782files to each group, depending on whatever you specify in the files in /usr/
3783local/etc/. If it finds a server that hasn’t finished its last run, it will continue
3784to skip the server until it has finished (and issue you an email to that effect
3785via cron’s MAILTO feature).
3786The load on your master will likely go up, depending on how many machines
3787you’re syncing to (as each server requires both an rsync and an ssh session.)
3788Automated Snapshot-Style Incremental Backups with rsync #42
3789Backups | 79
3790HACK
3791But practically speaking, on a production network serving millions of hits a
3792day, the load introduced to each individual web server is negligible.
3793H A C K
3794#42
3795Automated Snapshot-Style Incremental
3796Backups with rsync Hack #42
3797Use rsync to create fast, small, and safe snapshots of your filesystem
3798Here is a method for generating automatic rotating snapshot-style backups
3799on a Linux server. Snapshot backups are a feature of some high-end indus-
3800trial strength file servers; they create the illusion of multiple full (up to own-
3801ership/permission) backups per day without the space or processing
3802overhead. All of the snapshots are read-only and are accessible directly by
3803users as special system directories.
3804Since making a full copy of a large filesystem can be a time-consuming and
3805expensive process, it is common to make full backups only once a week or
3806once a month, and store only changes on the other days. This technique is
3807called making “incremental” backups, and is supported by the venerable old
3808dump and tar utilities, along with many others.
3809The standard GNU fileutils cp command comes with a -l flag that causes it to
3810create (hard) links instead of copies (it doesn’t hard-link directories, though,
3811which is good; you might want to think about why that is). Another handy
3812switch for the cp command is -a (archive), which causes it to recurse through
3813directories and preserve file owners, timestamps, and access permissions.
3814Together, the combination cp -al makes what appears to be a full copy of a
3815directory tree but is really just an illusion that takes almost no space. If we
3816restrict operations on the copy to adding or removing (unlinking) files—i.e.,
3817never changing one in place—then the illusion of a full copy is complete. To
3818the end-user, the only differences are that the illusion-copy takes almost no
3819disk space and almost no time to generate.
3820We can combine rsync and cp -al to create what appear to be multiple full
3821backups of a filesystem without taking multiple disks’ worth of space, as in:
3822rm -rf backup.3
3823mv backup.2 backup.3
3824mv backup.1 backup.2
3825cp -al backup.0 backup.1
3826rsync -a --delete source_directory/ backup.0/
3827If the above commands are run once every day, then backup.0 , backup.1 ,
3828backup.2 , and backup.3 will appear to each be a full backup of source_
3829directory/ as it appeared today, yesterday, two days ago, and three days ago,
3830respectively—complete, except that permissions and ownerships in old
383180 | Backups
3832#42 Automated Snapshot-Style Incremental Backups with rsync
3833HACK
3834snapshots will get their most recent values (thanks to J.W. Schultz for point-
3835ing this out). In reality, the extra storage will be equal to the current size of
3836source_directory/ plus the total size of the changes over the last three days—
3837exactly the same space that a full plus daily incremental backup with dump
3838or tar would have taken.
3839This method is much better for network-based backups, since it’s only nec-
3840essary to do a full backup once, instead of once per week. Thereafter, only
3841the changes need to be copied. Unfortunately, you can’t rsync to a tape;
3842you’ll still need dump or tar for that.
3843If you have a spare machine, even a very low-end one, you can turn it into a
3844dedicated backup server. Make it standalone, and keep it in a physically sep-
3845arate place—another room or even another building. Disable every single
3846remote service on the backup server, and connect it only to a dedicated net-
3847work interface on the source machine.
3848You can then perform backups from this machine using rsync over ssh (see
3849“Keeping Parts of Filesystems in sync with rsync” [Hack #41] ), and export the
3850backups back to the original machine via read-only NFS. Then users can get
3851to the snapshots themselves (without needing sysadmin intervention) and
3852can’t possibly change or delete them.
3853I’d consider this “pretty good” protection, but if you’re (wisely) paranoid, or
3854your job is on the line, build two backup servers. Then you can make sure
3855that at least one of them is always offline.
3856Extensions: Hourly, Daily, and Weekly Snapshots
3857With a little bit of tweaking, you can make multiple-level rotating snap-
3858shots. On my system, for example, I keep the last four “hourly” snapshots
3859(which are taken every four hours) as well as the last three “daily” snap-
3860shots (which are taken at midnight every day). You might also want to keep
3861weekly or even monthly snapshots too, depending upon your needs and
3862your available space.
3863I keep one script that runs every four hours to make and rotate hourly snap-
3864shots, and another script that runs once a day rotate the daily snapshots.
3865There is no need to use rsync for the higher-level snapshots; just cp -al from
3866the appropriate hourly one.
3867To make the automatic snapshots happen, I have added the following lines
3868to root’s crontab:
38690 */4 * * * /usr/local/bin/make_snapshot.sh
38700 13 * * * /usr/local/bin/daily_snapshot_rotate.sh
3871D ownload from Wow! eBook <www.wowebook.com>
3872Automated Snapshot-Style Incremental Backups with rsync #42
3873Backups | 81
3874HACK
3875They cause make_snapshot.sh to be run every four hours on the hour and
3876daily_snapshot_rotate.sh to be run every day at 13:00 (that is, 1:00 PM).
3877Those scripts are included below.
3878Listing: make_snapshot.sh
3879#!/bin/bash
3880# ----------------------------------------------------------------------
3881# mikes handy rotating-filesystem-snapshot utility
3882# ----------------------------------------------------------------------
3883# RCS info: $Id$
3884# ----------------------------------------------------------------------
3885# this needs to be a lot more general, but the basic idea is it makes
3886# rotating backup-snapshots of /home whenever called
3887# ----------------------------------------------------------------------
3888# ------------- system commands used by this script --------------------
3889ID=/usr/bin/id;
3890ECHO=/bin/echo;
3891MOUNT=/bin/mount;
3892RM=/bin/rm;
3893MV=/bin/mv;
3894CP=/bin/cp;
3895TOUCH=/bin/touch;
3896RSYNC=/usr/bin/rsync;
3897# ------------- file locations -----------------------------------------
3898MOUNT_DEVICE=/dev/hdb1;
3899SNAPSHOT_Rw=/root/snapshot;
3900EXCLUDES=/usr/local/etc/backup_exclude;
3901# ------------- the script itself --------------------------------------
3902# make sure we're running as root
3903if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root. Exiting...";
3904exit; } fi
3905# attempt to remount the Rw mount point as Rw; else abort
3906$MOUNT -o remount,rw $MOUNT_DEVICE $SNAPSHOT_Rw ;
3907if (( $? )); then
3908{
3909$ECHO "snapshot: could not remount $SNAPSHOT_Rw readwrite";
3910exit;
3911}
3912fi;
391382 | Backups
3914#42 Automated Snapshot-Style Incremental Backups with rsync
3915HACK
3916# rotating snapshots of /home (fixme: this should be more general)
3917# step 1: delete the oldest snapshot, if it exists:
3918if [ -d $SNAPSHOT_Rw/home/hourly.3 ] ; then \
3919$RM -rf $SNAPSHOT_Rw/home/hourly.3 ; \
3920fi ;
3921# step 2: shift the middle snapshots(s) back by one, if they exist
3922if [ -d $SNAPSHOT_Rw/home/hourly.2 ] ; then \
3923$MV $SNAPSHOT_Rw/home/hourly.2 $SNAPSHOT_Rw/home/hourly.3 ; \
3924fi;
3925if [ -d $SNAPSHOT_Rw/home/hourly.1 ] ; then \
3926$MV $SNAPSHOT_Rw/home/hourly.1 $SNAPSHOT_Rw/home/hourly.2 ; \
3927fi;
3928# step 3: make a hard-link-only (except for dirs) copy of the latest
3929snapshot,
3930# if that exists
3931if [ -d $SNAPSHOT_Rw/home/hourly.0 ] ; then \
3932$CP -al $SNAPSHOT_Rw/home/hourly.0 $SNAPSHOT_Rw/home/hourly.1 ; \
3933fi;
3934# step 4: rsync from the system into the latest snapshot (notice that
3935# rsync behaves like cp --remove-destination by default, so the destination
3936# is unlinked first. If it were not so, this would copy over the other
3937# snapshot(s) too!
3938$RSYNC \
3939-va --delete --delete-excluded \
3940--exclude-from="$EXCLUDES" \
3941/home/ $SNAPSHOT_Rw/home/hourly.0 ;
3942# step 5: update the mtime of hourly.0 to reflect the snapshot time
3943$TOUCH $SNAPSHOT_Rw/home/hourly.0 ;
3944# and thats it for home.
3945# now remount the Rw snapshot mountpoint as readonly
3946$MOUNT -o remount,ro $MOUNT_DEVICE $SNAPSHOT_Rw ;
3947if (( $? )); then
3948{
3949$ECHO "snapshot: could not remount $SNAPSHOT_Rw readonly";
3950exit;
3951} fi;
3952If you notice above, I have added an excludes list to the rsync call. This is
3953just to prevent the system from backing up garbage like web browser caches,
3954which change frequently (so they’d take up space in every snapshot) but
3955would be no loss if they were accidentally destroyed.
3956Automated Snapshot-Style Incremental Backups with rsync #42
3957Backups | 83
3958HACK
3959Listing: Daily_snapshot_rotate.sh
3960#!/bin/bash
3961# ----------------------------------------------------------------------
3962# mikes handy rotating-filesystem-snapshot utility: daily snapshots
3963# ----------------------------------------------------------------------
3964# RCS info: $Id: daily_snapshot_rotate.sh,v 1.2 2002/03/25 21:53:27 mrubel
3965Exp $
3966# ----------------------------------------------------------------------
3967# intended to be run daily as a cron job when hourly.3 contains the
3968# midnight (or whenever you want) snapshot; say, 13:00 for 4-hour snapshots.
3969# ----------------------------------------------------------------------
3970# ------------- system commands used by this script --------------------
3971ID=/usr/bin/id;
3972ECHO=/bin/echo;
3973MOUNT=/bin/mount;
3974RM=/bin/rm;
3975MV=/bin/mv;
3976cp=/bin/cp;
3977# ------------- file locations -----------------------------------------
3978MOUNT_DEVICE=/dev/hdb1;
3979SNAPSHOT_Rw=/root/snapshot;
3980# ------------- the script itself --------------------------------------
3981# make sure we're running as root
3982if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root. Exiting...";
3983exit; } fi
3984# attempt to remount the Rw mount point as Rw; else abort
3985$MOUNT -o remount,rw $MOUNT_DEVICE $SNAPSHOT_Rw ;
3986if (( $? )); then
3987{
3988$ECHO "snapshot: could not remount $SNAPSHOT_Rw readwrite";
3989exit;
3990}
3991fi;
3992# step 1: delete the oldest snapshot, if it exists:
3993if [ -d $SNAPSHOT_Rw/home/daily.2 ] ; then \
3994$RM -rf $SNAPSHOT_Rw/home/daily.2 ; \
3995fi ;
3996# step 2: shift the middle snapshots(s) back by one, if they exist
3997if [ -d $SNAPSHOT_Rw/home/daily.1 ] ; then \
3998$MV $SNAPSHOT_Rw/home/daily.1 $SNAPSHOT_Rw/home/daily.2 ; \
3999fi;
4000if [ -d $SNAPSHOT_Rw/home/daily.0 ] ; then \
400184 | Backups
4002#43 Working with ISOs and CDR/CDRWs
4003HACK
4004$MV $SNAPSHOT_Rw/home/daily.0 $SNAPSHOT_Rw/home/daily.1; \
4005fi;
4006# step 3: make a hard-link-only (except for dirs) copy of
4007# hourly.3, assuming that exists, into daily.0
4008if [ -d $SNAPSHOT_Rw/home/hourly.3 ] ; then \
4009$cp -al $SNAPSHOT_Rw/home/hourly.3 $SNAPSHOT_Rw/home/daily.0 ; \
4010fi;
4011# note: do *not* update the mtime of daily.0; it will reflect
4012# when hourly.3 was made, which should be correct.
4013# now remount the Rw snapshot mountpoint as readonly
4014$MOUNT -o remount,ro $MOUNT_DEVICE $SNAPSHOT_Rw ;
4015if (( $? )); then
4016{
4017$ECHO "snapshot: could not remount $SNAPSHOT_Rw readonly";
4018exit;
4019} fi;
4020Sample Output of ls -l /snapshot/home
4021total 28
4022drwxr-xr-x 12 root root 4096 Mar 28 00:00 daily.0
4023drwxr-xr-x 12 root root 4096 Mar 27 00:00 daily.1
4024drwxr-xr-x 12 root root 4096 Mar 26 00:00 daily.2
4025drwxr-xr-x 12 root root 4096 Mar 28 16:00 hourly.0
4026drwxr-xr-x 12 root root 4096 Mar 28 12:00 hourly.1
4027drwxr-xr-x 12 root root 4096 Mar 28 08:00 hourly.2
4028drwxr-xr-x 12 root root 4096 Mar 28 04:00 hourly.3
4029Notice that the contents of each of the subdirectories of /snapshot/home/ is a
4030complete image of /home at the time the snapshot was made. Despite the w
4031in the directory access permissions, no one—not even root—can write to
4032this directory; it’s mounted read-only.
4033See also:
4034• “Keeping Parts of Filesystems in sync with rsync” [Hack #41]
4035• See the version at http://www.mikerubel.org/computers/rsync_snapshots/)
4036H A C K
4037#43
4038Working with ISOs and CDR/CDRWs Hack #43
4039The command line makes working with ISOs a snap
4040There are a number of graphical CD-ROM utilities for Linux. Most of these
4041are simply front-ends that call command line tools to do the actual work of
4042building ISOs and burning disks. And graphical tools aren’t much help if
4043you’re working on servers that don’t have a console attached, unless you are
4044Working with ISOs and CDR/CDRWs #43
4045Backups | 85
4046HACK
4047running X remotely, perhaps over ssh (as in “X over ssh” [Hack #70] ). With the
4048mkisofs and cdrecord utilities installed, working with ISO images from the
4049command line is very straightforward.
4050If you’ve never heard of an ISO, it’s slang for an image of an ISO9660 filesys-
4051tem. The ISO9660 format (along with a couple of common extensions) is
4052the common format of data CD-ROMs.
4053To make an ISO image to prepare for burning in a CD burner, use mkisofs:
4054mkisofs -r /home/rob/ > /tmp/rob-home.iso
4055The -r tells mkisofs to build Rock Ridge extensions into the resulting image
4056file. This means that long filenames and file permissions will be preserved
4057when the disk is mounted on systems that support Rock Ridge. Linux has
4058great support for RR, and using -r generally makes your life much easier when
4059building disks designed to be used with Linux. Anyone can make an ISO of
4060files they have read access to; you don’t have to be root to run mkisofs. But for
4061the rest of the commands in this hack, you’ll likely need root privileges.
4062Note that mkisofs stores the contents of the directories you specify on the
4063command line, not the directories themselves. In the above example, the
4064ISO will have /home/rob/ mapped to its /, with the contents of /home/rob/
4065filling the root of the CD.
4066If you need to make an ISO from an existing data CD, try this:
4067#dd if=/dev/cdrom of=image.iso
4068This will create an ISO image of the CD, in all glorious 650+MB, so be sure
4069you have plenty of space before trying this. You may be able to make this go
4070faster by manipulating the bs parameter of dd:
4071#dd if=/dev/cdrom of=image.iso bs=10k
4072The optimal setting is usually dependent on your drive and IDE controller,
4073so try a couple of values to see what works best on your system.
4074To mount an ISO that you’ve made (either with mkisofs or straight from dd),
4075try this:
4076#mkdir /mnt/iso
4077#mount -o loop,ro -t iso9660 ./image.iso /mnt/iso
4078If the mount command complains about Could not find any loop device,
4079you might need to load the loopback driver:
4080#modprobe loop
4081Once the ISO is mounted, you should be able to cd /mnt/iso and look
4082around at the filesystem contained in the image. If you find problems with
4083it, simply umount /mnt/iso , remove the image file, and try again.
408486 | Backups
4085#44 Burning a CD Without Creating an ISO File
4086HACK
4087When you’re ready to burn an ISO to CDR, try something like this:
4088#cdrecord -v speed=12 dev=0,0,0 -data image.iso
4089You should specify the writing speed of your burner (or slower) in the
4090speed= option. If you need to erase a CDRW before burning the ISO, try
4091passing the blank= parameter:
4092#cdrecord -v speed=12 dev=0,0,0 blank=fast -data image.iso
4093Getting a burner running under Linux isn’t as hard as it used to be, thanks
4094to the ide-scsi driver. This is a kernel module that makes IDE (and other) CD
4095burners appear to be SCSI burners, which are much easier to program.
4096See also:
4097• For help on getting your burner running under Linux, check out the CD
4098Writing HOWTO available at The Linux Documentation Project at
4099http://www.tldp.org/HOWTO/CD-Writing-HOWTO.html
4100H A C K
4101#44
4102Burning a CD Without Creating an ISO File Hack #44
4103Create a CD from another CD, the live filesystem, or even an http download
4104The safest method for making a copy of a CD is to first make an ISO and
4105then burn the ISO (as in “Working with ISOs and CDR/CDRWs” [Hack #43] ).
4106But sometimes you don’t have the space (or time) for the interim step of
4107making a copy.
4108If you have a fast enough machine, you can usually burn straight from one CD
4109to another. This usually works best when the source CD and the burner are on
4110separate device chains (like primary and secondary IDE, or IDE and SCSI).
4111To make a real-time copy of a CD, give this a try:
4112#dd if=/dev/hdb | cdrecord -v speed=12 dev=0,0,0 fs=8m -data -
4113The - argument to cdrecord means that the data track should be read from
4114STDIN instead of from a file. The dd line is feeding the cdrecord pipe with a
4115copy of the CD in the slave drive on the primary IDE chain (hdb). The fs=8m
4116parameter makes the write FIFO a bit bigger, to help offset any momentary
4117pipeline hiccups. As long as your bus is up to the task (and your machine
4118isn’t otherwise too occupied) then this method will work fine.
4119Likewise, there is no real need to make an ISO before burning a copy of data
4120from the filesystem. Give this a try:
4121#mkisofs -r /home/ftp/ | cdrecord -v speed=12 dev=0,0,0 fs=8m -data -
4122Burning a CD Without Creating an ISO File #44
4123Backups | 87
4124HACK
4125Like the dd above, mkisofs writes to STDOUT by default. This is then fed to
4126STDIN of the cdrecord process, burning the ISO as it is created, in real time.
4127This saves the need to keep a copy of the ISO file lying around your filesys-
4128tem. Be warned, if your data source is greater than the size of your CDR
4129(somewhere between 650 and 700MB) then you’ll end up with a coaster but
4130not a usable disk.
4131Get an idea of how much space you’ll need with du first:
4132#du -hs /home/ftp/
4133412M /home/ftp
4134Perfect. A 412M ISO will fit nicely.
4135Anything that can print ISO data to STDOUT is a candidate for the left-
4136hand side of a pipeline. How about doing a real-time network burn?
4137root@catlin:~#mkisofs -r backup/ \
4138| ssh florian "cdrecord -v speed=12 dev=0,0,0 fs=8m -data -"
4139Or copying a local CD to a remote burner over the network?
4140root@catlin:~#dd if=/dev/cdrom \
4141| ssh florian "cdrecord -v speed=12 dev=0,0,0 fs=8m -data -"
4142Or even downloading an ISO and burning it, all in one pass?
4143#curl http://my.server.com/slackware-8.1-install.iso \
4144| cdrecord -v speed=0 dev=0,0,0 fs=8m -data -
4145I wouldn’t recommend attempting to do a network burn over wireless;
4146you’ll want a nice, solid 100Mbps Ethernet cable for this job. And the down-
4147load example might look silly, but it does illustrate the power of the Unix
4148pipeline: any program can be plugged into nearly any other, to make things
4149happen that the designer of either program likely hadn’t ever thought.
415088
4151Chapter 4
4152C H A P T E R F O U R
4153Networking
4154Hacks #45–53
4155There was once a time when a network admin was a person who spent all of
4156his time trying to figure out how to make machines talk to each other over a
4157network. It seems that lately, much of a network admins’ time is spent trying
4158to figure out how to restrict access to their machines via the network, thus
4159keeping out undesirables while still allowing legitimate traffic to pass through.
4160Fortunately, the netfilter firewall in Linux provides a very flexible interface
4161to the kernel’s networking decisions. Using the iptables command, you can
4162create firewall rules that let you create a rich and very flexible access policy.
4163It can not only match packets based on port, interface and MAC addresses,
4164but also on data contained within the packet and even by the rate that pack-
4165ets are received. This information can be used to help weed out all sorts of
4166attempted attacks, from port floods to virii.
4167But locking users out isn’t nearly as much fun as connecting users together.
4168After all the whole point of a computer network is to allow people to com-
4169municate with each other! We’ll take a look at some more unusual methods
4170for controlling the flow of network traffic, from the remote port forwarding
4171to various forms of IP tunnelling. By the time we’ve explored IP encapsula-
4172tion and user space tunnels like vtun, we’ll see how it is possible to build
4173networks on top of the Internet that behave in all sorts of unexpected and
4174surprisingly useful ways.
4175H A C K
4176#45
4177Creating a Firewall from the Command Line of
4178any Server Hack #45
4179You don’t have to have a dedicated firewall to benefit from using iptables
4180The netfilter firewall (available in Linux 2.4 and later) allows for very flexi-
4181ble firewall manipulation from the command line. Using iptables can take a
4182while to get used to, but it allows for a very expressive syntax that lets you
4183create complex (and hopefully useful ;) firewall rules.
4184Creating a Firewall from the Command Line of any Server #45
4185Networking | 89
4186HACK
4187Even if your machine isn’t a “real” firewall (that is, it only has one network
4188interface and isn’t protecting other machines) the filter functionality can be
4189very useful. Suppose you want to allow telnet access to this machine (just in
4190case something happens to ssh or its libraries) but don’t want to permit it from
4191just anywhere on the Net. You could use a tcpwrapper (by populating /etc/
4192hosts.allow and /etc/hosts.deny, and setting up /etc/inetd.conf appropriately).
4193Or, you could use iptables with a line like this:
4194iptables -A INPUT -t filter -s ! 208.201.239.36 -p tcp --dport 23 -j DROP
4195Generally, most people want to permit unrestricted access from trusted
4196hosts, block all access from known problem hosts, and allow something in
4197between for everyone else. Here is one method for using a whitelist, black-
4198list, and restricted port policy simultaneously.
4199#!/bin/sh
4200#
4201# A simple firewall initialization script
4202#
4203WHITELIST=/usr/local/etc/whitelist.txt
4204BLACKLIST=/usr/local/etc/blacklist.txt
4205ALLOWED="22 25 80 443"
4206#
4207# Drop all existing filter rules
4208#
4209iptables -F
4210#
4211# First, run through $WHITELIST, accepting all traffic from the hosts and
4212networks
4213# contained therein.
4214#
4215for x in `grep -v ^# $WHITELIST | awk '{print $1}'`; do
4216echo "Permitting $x..."
4217iptables -A INPUT -t filter -s $x -j ACCEPT
4218done
4219#
4220# Now run through $BLACKLIST, dropping all traffic from the hosts and
4221networks
4222# contained therein.
4223#
4224for x in `grep -v ^# $BLACKLIST | awk '{print $1}'`; do
4225echo "Blocking $x..."
4226iptables -A INPUT -t filter -s $x -j DROP
4227done
4228#
4229# Next, the permitted ports: What will we accept from hosts not appearing
4230# on the blacklist?
4231#
4232D ownload from Wow! eBook <www.wowebook.com>
423390 | Networking
4234#45 Creating a Firewall from the Command Line of any Server
4235HACK
4236for port in $ALLOWED; do
4237echo "Accepting port $port..."
4238iptables -A INPUT -t filter -p tcp --dport $port -j ACCEPT
4239done
4240#
4241# Finally, unless it's mentioned above, and it's an inbound startup request,
4242# just drop it.
4243#
4244iptables -A INPUT -t filter -p tcp --syn -j DROP
4245Be sure to specify all of the ports you’d like to include in the $ALLOWED
4246variable at the top of the script. If you forget to include 22, you won’t be
4247able to ssh into the box!
4248The /usr/local/etc/blacklist.txt file is populated with IP addresses, host
4249names, and networks like this:
42501.2.3.4 # Portscanned on 8/15/02
42517.8.9.0/24 # Who knows what evil lurks therein
4252r00tb0y.script-kiddie.coop # $0 s0rR33 u 31337 h4x0r!
4253Likewise, /usr/local/etc/whitelist.txt contains the “good guys” that should be
4254permitted no matter what the other rules specify:
425511.22.33.44 # My workstation
4256208.201.239.0/26 # the local network
4257Since we’re only grabbing lines that don’t start with #, you can comment
4258out an entire line if you need to. The next time you run the script, any com-
4259mented entries will be ignored. We run an iptables -F at the beginning to
4260flush all existing filter entries, so you can simply run the script again when
4261you make changes to blacklist.txt, whitelist.txt, or the ports specified in
4262$ALLOWED.
4263Also note that this script only allows for TCP connections. If you need to
4264also support UDP, ICMP, or some other protocol, run another pass just like
4265the $ALLOWED for loop, but include your additional ports and protocols
4266(passing -p udp or -p icmp to iptables, for example).
4267Be careful about using whitelists. Any IPs or networks appearing on this list
4268will be permitted to access all ports on your machine. In some circum-
4269stances, a clever miscreant may be able to send forged packets apparently
4270originating from one of those IPs, if they can find out ahead of time (or logi-
4271cally deduce) what IPs appear on your whitelist. This kind of attack is diffi-
4272cult to perform, but it is possible. If you are particularly paranoid, you might
4273only allow whitelist addresses from networks that aren’t routable over the
4274Internet but are used on your internal network.
4275Simple IP Masquerading #46
4276Networking | 91
4277HACK
4278It is extremely useful to have console access while working with new fire-
4279wall rules (you can’t lock yourself out of the console with iptables!) If you
4280get confused about where you are when working with iptables, remember
4281that you can always list out all rules with iptables -L, and start over by issu-
4282ing iptables -F. If iptables -L seems to hang, try iptables -L -n to show the
4283rules without doing any DNS resolution—your rules might accidentally be
4284prohibiting DNS requests.
4285You can do a lot with simple filtering, but there’s much more to iptables
4286than just the filter target.
4287See also:
4288• Netfilter HOWTO at www.netfilter.org
4289• “iptables Tips & Tricks” [Hack #47]
4290H A C K
4291#46
4292Simple IP Masquerading Hack #46
4293Set up NAT on your gateway in ten seconds
4294If you have a private network that needs to share an Internet connection
4295with one IP address, you’ll want to use IP Masquerading on your gateway
4296machine. Luckily, with iptables this is a simple two-liner:
4297#echo "1" > /proc/sys/net/ipv4/ip_forward
4298#iptables -t nat -A POSTROUTING -o $EXT_IFACE -j MASQUERADE
4299where $EXT_IFACE is the outside interface of your gateway. Now any
4300machines that reside on a network on any of the other interfaces in your
4301gateway will be able to “get out” to the Internet. As far as the Net is con-
4302cerned, all traffic originates from your gateway’s external IP address.
4303There was a time when one had to worry about miscreants on the external
4304network sending forged packets to your gateway, claiming to be originating
4305from the internal network. These packets would obligingly be masqueraded
4306by the kernel, and leave your network as if they were legitimate traffic. This
4307made it possible for anyone to launch attacks that apparently originated
4308from your network, making very bad times for the hapless gateway owner.
4309I say that this was a problem, because recent kernels give you a free firewall
4310rule to deal with exactly this problem, called rp_filter. With rp_filter
4311enabled, if a packet that arrives on an interface has a source address that
4312doesn’t match the corresponding routing table entry, it is dropped. This
4313effectively prevents IP spoofing and allows simple (and safe) masquerading
4314with the example above.
431592 | Networking
4316#47 iptables Tips & Tricks
4317HACK
4318In the very unlikely event that rp_filter is causing problems for you, you can
4319deactivate it very easily:
4320#echo "0" > /proc/sys/net/ipv4/conf/all/rp_filter
4321See also:
4322• “iptables Tips & Tricks” [Hack #47]
4323H A C K
4324#47
4325iptables Tips & Tricks Hack #47
4326Make your firewall do far more than filter packets with iptables
4327iptables is the next generation of firewall software for the netfilter project. It
4328provides all of the functionality of its predecessor, ipchains, in addition to
4329support for stateful firewalling. iptables also supports a framework for
4330extending its capabilities with loadable modules. Here are a few tricks you
4331can use with the base distribution of iptables, as well as some of the extensi-
4332ble modules available for iptables.
4333For these examples, we’ll assume that the following environment variables
4334are already set:
4335$EXT_IFACE
4336The external (public) interface of the firewall
4337$INT_IFACE
4338The inside (private) interface of the firewall
4339$DEST_IP
4340The ultimate requested destination of this packet
4341You can use iptables to limit new inbound TCP packets to prevent a Denial
4342of Service attack. This is accomplished with the following rules:
4343# Create syn-flood chain for detecting Denial of Service attacks
4344iptables -t nat -N syn-flood
4345# Limit 12 connections per second (burst to 24)
4346iptables -t nat -A syn-flood -m limit --limit 12/s --limit-burst 24 \
4347-j RETURN
4348iptables -t nat -A syn-flood -j DROP
4349# Check for DoS attack
4350iptables -t nat -A PREROUTING -i $EXT_IFACE -d $DEST_IP -p tcp --syn \
4351-j syn-flood
4352These rules limit new inbound TCP connections (packets with SYN bit set)
4353to 12 per second after 24 connections per second have been seen.
4354Using iptables, a transparent Squid proxy can be set up. This will transpar-
4355ently cache and log all outbound HTTP requests to the Internet. It requires
4356iptables Tips & Tricks #47
4357Networking | 93
4358HACK
4359no modification to the user’s browser and is useful for blocking unwanted
4360content. This is accomplished with the following iptables rule at the top of
4361the PREROUTING chain:
4362# Setup transparent Squid proxy for internal network
4363#
4364# For details on setting up Squid, see:
4365# http://www.linuxdoc.org/HOWTO/mini/TransparentProxy.html
4366#
4367iptables -t nat -A PREROUTING -i $INT_IFACE -p tcp --dport 80 \
4368-j REDIRECT --to-port 3128
4369This rule redirects outgoing requests on TCP port 80 to a Squid proxy run-
4370ning on TCP port 3128 on the firewall.
4371Arbitrary TCP flags can be matched with iptables. This means you can block
4372XMAS-tree (all flags set) and NULL packets with the following rules:
4373# DROP XMAS & NULL TCP packets
4374iptables -t nat -A PREROUTING -p tcp --tcp-flags ALL ALL -j DROP
4375iptables -t nat -A PREROUTING -p tcp --tcp-flags ALL NONE -j DROP
4376Advanced iptables Features
4377iptables has introduced several advanced firewall features that are available by
4378patching the Linux kernel. These patches can be obtained from http://www.
4379netfilter.org/ by downloading the patch-o-matic version corresponding to the
4380iptables version you are using. Patch-o-matic patches are iptables patches that
4381are not yet available in the mainstream Linux kernel. Some of the patches are
4382experimental and should be used with caution.
4383Using the experimental netfilter psd patch, iptables can detect and block
4384inbound port scans with the following rule:
4385# DROP inbound port scans
4386iptables -t nat -A PREROUTING -i $EXT_IFACE -d $DEST_IP -m psd -j DROP
4387Using the experimental netfilter iplimit patch, iptables can limit the number
4388of connections received from a particular IP address with the following rule:
4389# DROP packets from hosts with more than 16 active connections
4390iptables -t nat -A PREROUTING -i $EXT_IFACE -p tcp --syn -d $DEST_IP -m
4391iplimit --iplimit-above 16 -j DROP
4392One of the most powerful netfilter patches allows you to match packets
4393based on their content. The experimental string-matching patch allows you
4394to filter out packets that match a certain string. This is helpful to filter out
4395the CodeRed or Nimda viruses before they hit your web server. The follow-
4396ing rules achieve this:
4397# DROP HTTP packets related to CodeRed and Nimda viruses silently
4398iptables -t filter -A INPUT -i $EXT_IFACE -p tcp -d $DEST_IP --dport http \
4399-m string --string "/default.ida?" -j DROP
440094 | Networking
4401#48 Forwarding TCP Ports to Arbitrary Machines
4402HACK
4403iptables -t filter -A INPUT -i $EXT_IFACE -p tcp -d $DEST_IP --dport http \
4404-m string --string ".exe?/c+dir" -j DROP
4405iptables -t filter -A INPUT -i $EXT_IFACE -p tcp -d $DEST_IP --dport http \
4406-m string --string ".exe?/c+tftp" -j DROP
4407Port forwarding is now native to iptables. The nat table uses a feature called
4408Destination NAT in the PREROUTING chain to accomplish this. The fol-
4409lowing rule can be used to port forward HTTP requests to a system (10.0.0.3)
4410on the internal network:
4411# Use DNAT to port forward http
4412iptables -t nat -A PREROUTING ! -i $INT_IFACE -p tcp --destination-port \
441380 -j DNAT --to 10.0.0.3:80
4414You can also port forward UDP packets. If you port forward traffic for a par-
4415ticular port, you do not need to have a corresponding rule in the INPUT
4416chain to accept inbound connections on that port. This will only work if the
4417destination is on a network on a locally attached interface (that is, not to
4418destinations on foreign networks). Take a look at tools like rinetd (“For-
4419warding TCP Ports to Arbitrary Machines” [Hack #48] ) or nportredird if you
4420need traffic to forward to remote networks.
4421If you port forward your HTTP requests to an internal host, you can filter
4422out the CodeRed virus in the FORWARD chain with this rule:
4423iptables -t filter -A FORWARD -p tcp --dport http \
4424-m string --string "/default.ida?" -j DROP
4425Using iptables can be challenging at first, but its flexibility makes it a tre-
4426mendously useful tool. If you ever get stuck while developing your rule set
4427(and you will), remember that your two best friends are iptables -L -n and
4428tcpdump (maybe followed by a quick session with ethereal).
4429See also:
4430• “Forwarding TCP Ports to Arbitrary Machines” [Hack #48]
4431• The Linux Firewall HOWTO
4432H A C K
4433#48
4434Forwarding TCP Ports to Arbitrary Machines Hack #48
4435Make non-local services appear to come from local ports
4436As we saw in “iptables Tips & Tricks” [Hack #47] , it is simple to forward TCP
4437and UDP ports from a firewall to internal hosts using iptables. But what if
4438you need to forward traffic from arbitrary addresses to a machine that isn’t
4439even on your network? Try an application layer port forwarder, like rinetd.
4440Forwarding TCP Ports to Arbitrary Machines #48
4441Networking | 95
4442HACK
4443This simple bit of code is a couple of years old but is small, efficient, and
4444perfect for just this sort of problem. Unpack the archive and simply run
4445make, and you’ll be presented with a tiny rinetd binary that will let you
4446forward TCP ports to your heart’s content. Unfortunately, UDP ports aren’t
4447supported by rinetd.
4448The configuration file is dead simple:
4449[Source Address] [Source Port] [Destination Address] [Destination Port]
4450Each port to be forwarded is specified on a separate line. The source and
4451destination addresses can be either host names or IP addresses, and an IP
4452address of 0.0.0.0 binds rinetd to every available local IP:
44530.0.0.0 80 some.othersite.gov 80
4454216.218.203.211 25 123.45.67.89 25
44550.0.0.0 5353 my.shellserver.us 22
4456Save the file to /etc/rinetd.conf, and copy rinetd to somewhere handy (like /usr/
4457local/sbin/, for example.) Then start it by simply running rinetd.
4458The first example forwards all web traffic destined for any local address to
4459some.othersite.gov. Note that this will only work if there isn’t another pro-
4460cess (like Apache) already bound to local port 80.
4461The next forwards inbound SMTP traffic destined for 216.218.203.211 to
4462the mail server at 123.45.67.89 (but doesn’t interfere with any SMTP agents
4463bound to other local IPs). The final example will forward any inbound traf-
4464fic on port 5353 to the ssh server on my.shellserver.us. These all work with-
4465out NAT or any special kernel configuration. Simply run rinetd, and it
4466daemonizes and starts listening on the ports you have specified.
4467This utility can really help ease the transition when renumbering or physi-
4468cally relocating servers, as services can appear to remain up on the original
4469IP (even though they are actually coming from another network entirely).
4470rinetd doesn’t even need to run as root, if you’re only binding to ports higher
4471than 1024. There are also extensive options for providing access control and
4472keeping logs. This tiny tool is well worth having handy for when TCP port
4473indirection is called.
4474See also:
4475• http://www.boutell.com/rinetd/
4476• “iptables Tips & Tricks” [Hack #47]
447796 | Networking
4478#49 Using Custom Chains in iptables
4479HACK
4480H A C K
4481#49
4482Using Custom Chains in iptables Hack #49
4483Keep your firewall rules under control with custom chains
4484By default, the iptables filter table consists of three chains: INPUT, FOR-
4485WARD, and OUTPUT. You can add as many custom chains as you like to
4486help simplify managing large rule sets. Custom chains behave just as built-in
4487chains, introducing logic that must be passed before the ultimate fate of a
4488packet is determined.
4489To create a new chain, use the -N switch:
4490root@mouse:~#iptables -N fun-filter
4491You can see which chains are defined at any time with the standard -L
4492switch:
4493root@mouse:~#iptables -L
4494Chain INPUT (policy ACCEPT)
4495target prot opt source destination
4496Chain FORWARD (policy ACCEPT)
4497target prot opt source destination
4498Chain OUTPUT (policy ACCEPT)
4499target prot opt source destination
4500Chain fun-filter (0 references)
4501target prot opt source destination
4502In order to make use of your custom chain, you’ll have to jump to it from
4503somewhere. Let’s add a jump to the fun-filter chain we’ve just created
4504straight from the INPUT chain:
4505root@mouse:~#iptables -t filter -A INPUT -j fun-filter
4506Now your custom chain can grow to any sort of complexity you like. For
4507example, you may want to match packets based on the source MAC
4508address:
4509root@mouse:~#iptables -A fun-filter -m mac --mac-source 11:22:33:aa:bb:cc \
4510-j ACCEPT
4511root@mouse:~#iptables -A fun-filter -m mac --mac-source de:ad:be:ef:00:42 \
4512-j ACCEPT
4513root@mouse:~#iptables -A fun-filter -m mac --mac-source 00:22:44:fa:ca:de
4514-j REJECT --reject-with icmp-host-unreachable
4515root@mouse:~#iptables -A fun-filter -j RETURN
4516The RETURN jump at the end of the table makes processing resume back in
4517the chain that called this one (in this case, back in the INPUT chain). Again,
4518show what all of your tables look like with the -L switch:
4519Tunneling: IPIP Encapsulation #50
4520Networking | 97
4521HACK
4522root@mouse:~#iptables -L
4523Chain INPUT (policy ACCEPT)
4524target prot opt source destination
4525fun-filter all -- anywhere anywhere
4526Chain FORWARD (policy ACCEPT)
4527target prot opt source destination
4528Chain OUTPUT (policy ACCEPT)
4529target prot opt source destination
4530Chain fun-filter (0 references)
4531target prot opt source destination
4532ACCEPT all -- anywhere anywhere MAC 11:22:33:AA:BB:CC
4533ACCEPT all -- anywhere anywhere MAC DE:AD:BE:EF:00:42
4534REJECT all -- anywhere anywhere MAC 00:22:44:FA:CA:DE reject-with icmp-host-
4535unreachable
4536RETURN all -- anywhere anywhere
4537You can jump into any number of custom defined chains and even jump
4538between them. This helps to isolate rules that you’re developing from the
4539standard system policy rules, and enable and disable them easily. If you
4540want to stop using your custom chain temporarily, you can simply delete the
4541jump from the INPUT chain (rather than flushing the entire custom chain):
4542root@mouse:~#iptables -t filter -D INPUT -j fun-filter
4543If you decide to delete your custom chain, use -X:
4544root@mouse:~#iptables -X fun-filter
4545Note that there can be no references to your custom chain if you try to
4546delete it; use -F to flush the chain first if there are still rules referring to your
4547chain.
4548When properly managed, even the most complex iptables rulesets can be
4549easily read, if you use intuitively named custom chains.
4550See also:
4551• “iptables Tips & Tricks” [Hack #47]
4552H A C K
4553#50
4554Tunneling: IPIP Encapsulation Hack #50
4555IP tunneling with the Linux IPIP driver
4556If you have never worked with IP tunneling before, you might want to take a
4557look at the Advanced Router HOWTO before continuing. Essentially, an IP
4558tunnel is much like a VPN, except that not every IP tunnel involves encryp-
4559tion. A machine that is “tunneled” into another network has a virtual inter-
4560face configured with an IP address that isn’t local, but exists on a remote
456198 | Networking
4562#50 Tunneling: IPIP Encapsulation
4563HACK
4564network. Usually, all (or most) network traffic is routed down this tunnel, so
4565remote clients appear to exist on the network services, or more generally, to
4566connect to any two private networks together using the Internet to carry the
4567tunnel traffic.
4568If you want to perform simple IP-within-IP tunneling between two
4569machines, you might want to try IPIP. It is probably the simplest tunnel pro-
4570tocol available and will also work with *BSD, Solaris, and even Windows.
4571Note that IPIP is simply a tunneling protocol and does not involve any sort
4572of encryption. It is also only capable of tunneling unicast packets; if you
4573need to tunnel multicast traffic, take a look at GRE tunneling in “Tunnel-
4574ing: GRE Encapsulation” [Hack #51] .
4575Before we rush right into our first tunnel, you’ll need a copy of the advanced
4576routing tools (specifically the ip utility). You can get the latest authoritative
4577copy at ftp://ftp.inr.ac.ru/ip-routing/. Be warned, the advanced routing tools
4578aren’t especially friendly, but they allow you to manipulate nearly any facet
4579of the Linux networking engine.
4580Assume that you have two private networks (10.42.1.0/24 and 10.42.2.0/24)
4581and that these networks both have direct Internet connectively via a Linux
4582router at each network. The “real” IP address of the first network router is
4583240.101.83.2, and the “real” IP of the second router is 251.4.92.217. This isn’t
4584very difficult, so let’s jump right in.
4585First, load the kernel module on both routers:
4586#modprobe ipip
4587Next, on the first network’s router (on the 10.42.1.0/24 network), do the
4588following:
4589#ip tunnel add mytun mode ipip remote 251.4.92.217 \
4590local 240.101.83.2 ttl 255
4591#ifconfig mytun 10.42.1.1
4592#route add -net 10.42.2.0/24 dev mytun
4593And on the second network’s router (on the 10.42.2.0/24), reciprocate:
4594#ip tunnel add mytun mode ipip remote 240.101.83.2 \
4595local 251.4.92.217 ttl 255
4596# ifconfig tun10 10.42.2.1
4597#route add -net 10.42.1.0/24 dev mytun
4598Naturally, you can give the interface a more meaningful name than
4599mytun if you like. From the first network’s router, you should be able to
4600ping 10.42.2.1, and from the second network router, you should be able
4601to ping 10.42.1.1. Likewise, every machine on the 10.42.1.0/24 network
4602should be able to route to every machine on the 10.42.2.0/24 network,
4603just as if the Interent weren’t even there.
4604Tunneling: GRE Encapsulation #51
4605Networking | 99
4606HACK
4607If you’re running a Linux 2.2x kernel, you’re in luck: here’s a shortcut that
4608you can use to avoid having to use the Advanced Router tools package at all.
4609After loading the module, try these commands instead:
4610#ifconfig tun10 10.42.1.1 pointopoint 251.4.92.217
4611#route add -net 10.42.2.0/24 dev tun10
4612And on the second network’s router (on the 10.42.2.0/24):
4613#ifconfig tun10 10.42.2.1 pointopoint 240.101.83.2
4614#route add -net 10.42.1.0/24 dev tun10
4615That’s all there is to it.
4616If you can ping the opposite router but other machines on the network don’t
4617seem to be able to pass traffic beyond the router, make sure that both rout-
4618ers are configured to forward packets between interfaces:
4619#echo "1" > /proc/sys/net/ipv4/ip_forward
4620If you need to reach networks beyond 10.42.1.0 and 10.42.2.0, simply add
4621additional route add -net... lines. There is no configuration needed on any
4622of your network hosts, as long as they have a default route to their respec-
4623tive router (which they definitely should, since it is their router, after all).
4624To bring the tunnel down: On both routers, bring down the interface and
4625delete it, if you like:
4626#ifconfig mytun down
4627#ip tunnel del mytun
4628(or, in Linux 2.2):
4629#ifconfig tun10 down
4630The kernel will very politely clean up your routing table for you when the
4631interface goes away.
4632See also:
4633• Advanced Routing HOWTO, http://www.tldp.org/HOWTO/
4634Adv-Routing-HOWTO/
4635• Advanced Routing Tools (iproute2), ftp://ftp.inr.ac.ru/ip-routing/
4636H A C K
4637#51
4638Tunneling: GRE Encapsulation Hack #51
4639IP tunnels with Generic Routing Encapsulation
4640GRE stands for Generic Routing Encapsulation. Like IPIP tunneling (see
4641“Tunneling: IPIP Encapsulation” [Hack #50] ), GRE is an unencrypted encapsula-
4642tion protocol. The main advantages of using GRE instead of IPIP are it sup-
4643ports multicast packets, and it will interoperate with Cisco routers.
4644D ownload from Wow! eBook <www.wowebook.com>
4645100 | Networking
4646#51 Tunneling: GRE Encapsulation
4647HACK
4648Just as with the IPIP tunneling hack, we’ll assume that you have two private
4649networks (10.42.1.0/24 and 10.42.2.0/24) and that these networks both
4650have direct Internet connectivity via a Linux router at each network. The
4651“real” IP address of the first network router is 240.101.83.2, and the “real”
4652IP of the second router is 251.4.92.217.
4653Again, as with IPIP tunneling (“Tunneling: IPIP Encapsulation” [Hack #50] ),
4654you will need a copy of the advanced routing tools package (there is no
4655shortcut for GRE tunnels in Linux 2.2 that I’ve been able to find). Once you
4656have the iproute2 package installed, we’ll begin by loading the GRE kernel
4657module on both routers:
4658#modprobe ip_gre
4659On the first network’s router, set up a new tunnel device:
4660#ip tunnel add gre0 mode gre remote 251.4.92.217 local 240.101.83.2 ttl 255
4661#ip addr add 10.42.1.254 dev gre0
4662#ip link set gre0 up
4663Note that you can call the device anything you like; gre0 is just an example.
4664Also, that 10.42.1.254 address can be any available address on the first net-
4665work, but shouldn’t be 10.42.1.1 (the IP already bound to its internal inter-
4666face). Now, add your network routes via the new tunnel interface:
4667#ip route add 10.42.2.0/24 dev gre0
4668The first network is finished. Now for the second:
4669#ip tunnel add gre0 mode gre remote 240.101.83.2 local 251.4.92.217 ttl 255
4670# ip addr add 10.42.2.254 dev gre0
4671# ip link set gre0 up
4672# ip route add 10.42.1.0/24 dev gre0
4673Again, the 10.42.2.254 address can be any available address on the second
4674network. Feel free to add as many ip route add ... dev gre0 commands as
4675you need.
4676That’s it! You should now be able to pass packets between the two net-
4677works as if the Internet didn’t exist. A traceroute from the first network
4678should show just a couple of hops to any host in the second network
4679(although you’ll probably notice a fair bit of latency when crossing the 10.
468042.2.254 hop, unless you’re really well connected). If you’re having trouble,
4681check the notes in the IPIP example and don’t panic. Your best friend when
4682debugging new network configurations is probably a packet sniffer like
4683tcpdump or ethereal. Running a tcpdump 'proto \icmp' on both routers
4684while pinging will give you a very detailed overview of what’s going on.
4685To bring the tunnel down, run this on both routers:
4686# iplink set gre0 down
4687# iptunnel del gre0
4688Using vtun over ssh to Circumvent NAT #52
4689Networking | 101
4690HACK
4691See also:
4692• Advanced Routing HOWTO, http://www.tldp.org/HOWTO/
4693Adv-Routing-HOWTO/
4694• Advanced Routing Tools (iproute2), ftp://ftp.inr.ac.ru/ip-routing/
4695H A C K
4696#52
4697Using vtun over ssh to Circumvent NAT Hack #52
4698Connect two networks together using vtun and a single ssh connection
4699vtun is a user space tunnel server, allowing entire networks to be tunneled to
4700each other using the tun universal tunnel kernel driver. Connections can be
4701made directly over IP or even over PPP or serial. this technique can be partic-
4702ularly useful when general network access is restricted by an intervening fire-
4703wall, as all IP traffic will be encryted and forwarded over a single TCP port
4704(that is, over a single ssh connection).
4705The procedure described below will allow a host with a private IP address
4706(10.42.4.6) to bring up a new tunnel interface with a real, live routed IP
4707address (208.201.239.33) that works as expected, as if the private network
4708weren’t even there. We’ll do this by bringing up the tunnel, dropping the
4709default route, then adding a new default route via the other end of the tunnel.
4710To begin with, here is the (pre-tunneled) configuration of the network.
4711root@client:~#ifconfig eth2
4712eth2 Link encap:Ethernet HWaddr 00:02:2D:2A:27:EA
4713inet addr:10.42.3.2 Bcast:10.42.3.63 Mask:255.255.255.192
4714UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
4715RX packets:662 errors:0 dropped:0 overruns:0 frame:0
4716TX packets:733 errors:0 dropped:0 overruns:0 carrier:0
4717collisions:0 txqueuelen:100
4718RX bytes:105616 (103.1 Kb) TX bytes:74259 (72.5 Kb)
4719Interrupt:3 Base address:0x100
4720root@client:~#route
4721Kernel IP routing table
4722Destination Gateway Genmask Flags Metric Ref Use Iface
472310.42.3.0 * 255.255.255.192 U 0 0 0 eth2
4724loopback * 255.0.0.0 U 0 0 0 lo
4725default 10.42.3.1 0.0.0.0 UG 0 0 0 eth2
4726As you can see, our local network is 10.42.3.0/26, our IP is 10.42.3.2, and our
4727default gateway is 10.42.3.1. This gateway provides network address transla-
4728tion (NAT) to the internet. Here’s what our path looks like to yahoo.com:
4729root@client:~#traceroute -n yahoo.com
4730traceroute to yahoo.com (64.58.79.230), 30 hops max, 40 byte packets
47311 10.42.3.1 2.848 ms 2.304 ms 2.915 ms
4732102 | Networking
4733#52 Using vtun over ssh to Circumvent NAT
4734HACK
47352 209.204.179.1 16.654 ms 16.052 ms 19.224 ms
47363 208.201.224.194 20.112 ms 20.863 ms 18.238 ms
47374 208.201.224.5 213.466 ms 338.259 ms 357.7 ms
47385 206.24.221.217 20.743 ms 23.504 ms 24.192 ms
47396 206.24.210.62 22.379 ms 30.948 ms 54.475 ms
47407 206.24.226.104 94.263 ms 94.192 ms 91.825 ms
47418 206.24.238.61 97.107 ms 91.005 ms 91.133 ms
47429 206.24.238.26 95.443 ms 98.846 ms 100.055 ms
474310 216.109.66.7 92.133 ms 97.419 ms 94.22 ms
474411 216.33.98.19 99.491 ms 94.661 ms 100.002 ms
474512 216.35.210.126 97.945 ms 93.608 ms 95.347 ms
474613 64.58.77.41 98.607 ms 99.588 ms 97.816 ms
4747In this example, we’ll be connecting to a tunnel server on the Internet at
4748208.201.239.5. It has two spare live IP addresses (208.201.239.32 and 208.
4749201.239.33) to be used for tunneling. We’ll refer to that machine as the
4750server, and our local machine as the client.
4751Now, let’s get the tunnel running. To begin with, load the tun driver on both
4752machines:
4753#modprobe tun
4754It is worth noting that the tun driver will sometimes fail if the kernel version
4755on the server and client don’t match. For best results, use a recent kernel
4756(and the same version, e.g., 2.4.19) on both machines.
4757On the server machine, install this file to /usr/local/etc/vtund.conf :
4758options {
4759port 5000;
4760ifconfig /sbin/ifconfig;
4761route /sbin/route;
4762syslog auth;
4763}
4764default {
4765compress no;
4766speed 0;
4767}
4768home {
4769type tun;
4770proto tcp;
4771stat yes;
4772keepalive yes;
4773pass sHHH; # Password is REQUIRED.
4774up {
4775ifconfig "%% 208.201.239.32 pointopoint 208.201.239.33";
4776program /sbin/arp "-Ds 208.201.239.33 %% pub";
4777program /sbin/arp "-Ds 208.201.239.33 eth0 pub";
4778Using vtun over ssh to Circumvent NAT #52
4779Networking | 103
4780HACK
4781route "add -net 10.42.0.0/16 gw 208.201.239.33";
4782};
4783down {
4784program /sbin/arp "-d 208.201.239.33 -i %%";
4785program /sbin/arp "-d 208.201.239.33 -i eth0";
4786route "del -net 10.42.0.0/16 gw 208.201.239.33";
4787};
4788}
4789and launch the vtund server with this command:
4790root@server:~#vtund -s
4791Now, you’ll need a vtund.conf for the client side. Try this one, again in /usr/
4792local/etc/vtund.conf:
4793options {
4794port 5000;
4795ifconfig /sbin/ifconfig;
4796route /sbin/route;
4797}
4798default {
4799compress no;
4800speed 0;
4801}
4802home {
4803type tun;
4804proto tcp;
4805keepalive yes;
4806pass sHHH; # Password is REQUIRED.
4807up {
4808ifconfig "%% 208.201.239.33 pointopoint 208.201.239.32 arp";
4809route "add 208.201.239.5 gw 10.42.3.1";
4810route "del default";
4811route "add default gw 208.201.239.32";
4812};
4813down {
4814route "del default";
4815route "del 208.201.239.5 gw 10.42.3.1";
4816route "add default gw 10.42.3.1";
4817};
4818}
4819Finally, run this command on the client:
4820root@client:~#vtund -p home server
4821104 | Networking
4822#52 Using vtun over ssh to Circumvent NAT
4823HACK
4824Presto! You now not only have a tunnel up between client and server, but
4825have added a new default route via the other end of the tunnel. Take a look
4826at what happens when we traceroute to yahoo.com with the tunnel in place:
4827root@client:~#traceroute -n yahoo.com
4828traceroute to yahoo.com (64.58.79.230), 30 hops max, 40 byte packets
48291 208.201.239.32 24.368 ms 28.019 ms 19.114 ms
48302 208.201.239.1 21.677 ms 22.644 ms 23.489 ms
48313 208.201.224.194 20.41 ms 22.997 ms 23.788 ms
48324 208.201.224.5 26.496 ms 23.8 ms 25.752 ms
48335 206.24.221.217 26.174 ms 28.077 ms 26.344 ms
48346 206.24.210.62 26.484 ms 27.851 ms 25.015 ms
48357 206.24.226.103 104.22 ms 114.278 ms 108.575 ms
48368 206.24.238.57 99.978 ms 99.028 ms 100.976 ms
48379 206.24.238.26 103.749 ms 101.416 ms 101.09 ms
483810 216.109.66.132 102.426 ms 104.222 ms 98.675 ms
483911 216.33.98.19 99.985 ms 99.618 ms 103.827 ms
484012 216.35.210.126 104.075 ms 103.247 ms 106.398 ms
484113 64.58.77.41 107.219 ms 106.285 ms 101.169 ms
4842This means that any server processes running on client are now fully avail-
4843able to the Internet, at IP address 208.201.239.33. This has happened all
4844without making a single change (e.g., port forwarding) on the gateway 10.
484542.3.1.
4846Here’s what the new tunnel interface looks like on the client:
4847root@client:~#ifconfig tun0
4848tun0 Link encap:Point-to-Point Protocol
4849inet addr:208.201.239.33 P-t-P:208.201.239.32 Mask:255.255.255.255
4850UP POINTOPOINT RUNNING MULTICAST MTU:1500 Metric:1
4851RX packets:39 errors:0 dropped:0 overruns:0 frame:0
4852TX packets:39 errors:0 dropped:0 overruns:0 carrier:0
4853collisions:0 txqueuelen:10
4854RX bytes:2220 (2.1 Kb) TX bytes:1560 (1.5 Kb)
4855and here’s the updated routing table. Note that we still need to keep a host
4856route to the tunnel server’s IP address via our old default gateway, other-
4857wise the tunnel traffic couldn’t get out:
4858root@client:~#route
4859Kernel IP routing table
4860Destination Gateway Genmask Flags Metric Ref Use Iface
4861208.201.239.5 10.42.3.1 255.255.255.255 UGH 0 0 0 eth2
4862208.201.239.32 * 255.255.255.255 UH 0 0 0 tun0
486310.42.3.0 * 255.255.255.192 U 0 0 0 eth2
486410.42.4.0 * 255.255.255.192 U 0 0 0 eth0
4865loopback * 255.0.0.0 U 0 0 0 lo
4866default 208.201.239.32 0.0.0.0 UG 0 0 0 tun0
4867To bring down the tunnel, simply kill the vtund process on the client. This
4868will restore all network settings back to their original state.
4869Using vtun over ssh to Circumvent NAT #52
4870Networking | 105
4871HACK
4872This method works fine, if you trust vtun to use strong encryption and to be
4873free from remote exploits. Personally, I don’t think you can be too paranoid
4874when it comes to machines connected to the Internet. To use vtun over ssh
4875(and therefore rely on the strong authentication and encryption that ssh pro-
4876vides) simply forward port 5000 on client to the same port on server. Give
4877this a try:
4878root@client:~#ssh -f -N -c blowfish -C -L5000:localhost:5000 server
4879root@client:~#vtund -p home localhost
4880root@client:~#traceroute -n yahoo.com
4881traceroute to yahoo.com (64.58.79.230), 30 hops max, 40 byte packets
48821 208.201.239.32 24.715 ms 31.713 ms 29.519 ms
48832 208.201.239.1 28.389 ms 36.247 ms 28.879 ms
48843 208.201.224.194 48.777 ms 28.602 ms 44.024 ms
48854 208.201.224.5 38.788 ms 35.608 ms 35.72 ms
48865 206.24.221.217 37.729 ms 38.821 ms 43.489 ms
48876 206.24.210.62 39.577 ms 43.784 ms 34.711 ms
48887 206.24.226.103 110.761 ms 111.246 ms 117.15 ms
48898 206.24.238.57 112.569 ms 113.2 ms 111.773 ms
48909 206.24.238.26 111.466 ms 123.051 ms 118.58 ms
489110 216.109.66.132 113.79 ms 119.143 ms 109.934 ms
489211 216.33.98.19 111.948 ms 117.959 ms 122.269 ms
489312 216.35.210.126 113.472 ms 111.129 ms 118.079 ms
489413 64.58.77.41 110.923 ms 110.733 ms 115.22 ms
4895In order to discourage connections to vtund on port 5000 of the server, add
4896a netfilter rule to drop connections from the outside world:
4897root@server:~#iptables -A INPUT -t filter -i eth0 -p tcp --dport 5000 -j
4898DROP
4899This will allow local connections to get through (since they use loopback), and
4900therefore require an ssh tunnel to server before accepting a connection.
4901As you can see, this can be an extremely handy tool to have around. In addi-
4902tion to giving live IP addresses to machines behind a NAT, you can effec-
4903tively connect any two networks together if you can obtain a single ssh
4904connection between them (originating from either direction).
4905If your head is swimming from the vtund.conf configuration previously, or if
4906you’re terminally lazy and don’t want to figure out what to change when set-
4907ting up your own client’s vtund.conf, take a look at the Automatic vtund.conf
4908generator, in “Automatic vtund.conf Generator” [Hack #53] .
4909Notes:
4910• The session name (home in the above example) must match on the cli-
4911ent AND server sides, or you’ll get an ambiguous “server disconnected”
4912message.
4913• The same goes for the password field in the vtund.conf. It must be present
4914AND match on both sides, or the connection won’t work.
4915106 | Networking
4916#53 Automatic vtund.conf Generator
4917HACK
4918• If you’re having trouble connecting, make sure you’re using the same
4919kernel version on both sides, and that the server is up and running (try
4920telnet server 5000 from the client side to verify that the server is
4921happy).
4922• Try the direct method first, then get ssh working once you are happy
4923with your vtund.conf settings.
4924• If you’re still having trouble, check /etc/syslog.conf to see where your
4925auth facility messages are going, and watch that log on both the client
4926and server when trying to connect.
4927See also:
4928• vtun’s homepage: http://vtun.sourceforge.net/
4929• /usr/src/linux/Documentation/networking/tuntap.txt
4930• man vtund; man vtund.conf
4931• “Automatic vtund.conf Generator” [Hack #53]
4932• “Creating a Firewall from the Command Line of any Server” [Hack #45]
4933• “Forwarding Ports over ssh” [Hack #71]
4934H A C K
4935#53
4936Automatic vtund.conf Generator Hack #53
4937Generate a vtund.conf on the fly to match changing network conditions
4938If you’ve just come from “Tunneling: GRE Encapsulation” [Hack #51] , this script
4939will generate a working vtund.conf for the client side automatically.
4940If you haven’t read “Tunneling: GRE Encapsulation” [Hack #51] (or if you’ve
4941never used vtun), then go back and read it before attempting to grok this bit
4942of Perl. Essentially, it attempts to take the guesswork out of changing the
4943routing table around on the client side by auto-detecting the default gate-
4944way and building the vtund.conf accordingly.
4945To configure the script, take a look at the Configuration section. The first
4946line of $Config contains the addresses, port, and secret that we used in
4947“Tunneling: GRE Encapsulation” [Hack #51] . The second is there simply as an
4948example of how to add more.
4949To run the script, either call it as vtundconf home, or set $TunnelName to
4950the one to which you want to default. Or better yet, make symlinks to the
4951script like this:
4952#ln -s vtundconf home
4953#ln -s vtundconf tunnel2
4954Then generate the appropriate vtund.conf by calling the symlink directly:
4955#vtundconf home > /usr/local/etc/vtund.conf
4956Automatic vtund.conf Generator #53
4957Networking | 107
4958HACK
4959You might be wondering why anyone would go to all of the trouble to make
4960a script to generate a vtund.conf in the first place. Once you get the settings
4961right, you’ll never have to change them, right?
4962Well, usually that is the case. But consider the case of a Linux laptop that
4963uses many different networks in the course of the day (say a DSL line at
4964home, Ethernet at work, and maybe a wireless connection at the local coffee
4965shop). By running vtund.conf once at each location, you will have a working
4966configuration instantly, even if your IP and gateway is assigned by DHCP.
4967This makes it very easy to get up and running quickly with a live, routable IP
4968address, regardless of the local network topology.
4969Incidentally, vtund and vtund.conf currently runs great on Linux, FreeBSD,
4970OS X, Solaris, and a few others.
4971Listing: vtundconf
4972#!/usr/bin/perl -w
4973# vtund wrapper in need of a better name.
4974#
4975# (c)2002 Schuyler Erle & Rob Flickenger
4976#
4977################ CONFIGURATION
4978# If TunnelName is blank, the wrapper will look at @ARGV or $0.
4979#
4980# Config is TunnelName, LocalIP, RemoteIP, TunnelHost, TunnelPort, Secret
4981#
4982my $TunnelName = "";
4983my $Config = q{
4984home 208.201.239.33 208.201.239.32 208.201.239.5 5000 sHHH
4985tunnel2 10.0.1.100 10.0.1.1 192.168.1.4 6001 foobar
4986};
4987################ MAIN PROGRAM BEGINS HERE
4988use POSIX 'tmpnam';
4989use IO::File;
4990use File::Basename;
4991use strict;
4992# Where to find things...
4993#
4994$ENV{PATH} = "/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin";
4995my $IP_Match = '((?:\d{1,3}\.){3}\d{1,3})'; # match xxx.xxx.xxx.xxx
4996my $Ifconfig = "ifconfig -a";
4997my $Netstat = "netstat -rn";
4998my $Vtund = "/bin/echo";
4999my $Debug = 1;
5000108 | Networking
5001#53 Automatic vtund.conf Generator
5002HACK
5003# Load the template from the data section.
5004#
5005my $template = join( "", <DATA> );
5006# Open a temp file -- adapted from Perl Cookbook, 1st Ed., sec. 7.5.
5007#
5008my ( $file, $name ) = ("", "");
5009$name = tmpnam() until $file = IO::File->new( $name, O_RDWR|O_CREAT|O_EXCL
5010);
5011END { unlink( $name ) or warn "Can't remove temporary file $name!\n"; }
5012# If no TunnelName is specified, use the first thing on the command line,
5013# or if there isn't one, the basename of the script.
5014# This allows users to symlink different tunnel names to the same script.
5015#
5016$TunnelName ||= shift(@ARGV) || basename($0);
5017die "Can't determine tunnel config to use!\n" unless $TunnelName;
5018# Parse config.
5019#
5020my ($LocalIP, $RemoteIP, $TunnelHost, $TunnelPort, $Secret);
5021for (split(/\r*\n+/, $Config)) {
5022my ($conf, @vars) = grep( $_ ne "", split( /\s+/ ));
5023next if not $conf or $conf =~ /^\s*#/o; # skip blank lines, comments
5024if ($conf eq $TunnelName) {
5025($LocalIP, $RemoteIP, $TunnelHost, $TunnelPort, $Secret) = @vars;
5026last;
5027}
5028}
5029die "Can't determine configuration for TunnelName '$TunnelName'!\n"
5030unless $RemoteIP and $TunnelHost and $TunnelPort;
5031# Find the default gateway.
5032#
5033my ( $GatewayIP, $ExternalDevice );
5034for (qx{ $Netstat }) {
5035# In both Linux and BSD, the gateway is the next thing on the line,
5036# and the interface is the last.
5037#
5038if ( /^(?:0.0.0.0|default)\s+(\S+)\s+.*?(\S+)\s*$/o ) {
5039$GatewayIP = $1;
5040$ExternalDevice = $2;
5041last;
5042}
5043}
5044die "Can't determine default gateway!\n" unless $GatewayIP and
5045$ExternalDevice;
5046# Figure out the LocalIP and LocalNetwork.
5047#
5048D ownload from Wow! eBook <www.wowebook.com>
5049Automatic vtund.conf Generator #53
5050Networking | 109
5051HACK
5052my ( $LocalNetwork );
5053my ( $iface, $addr, $up, $network, $mask ) = "";
5054sub compute_netmask {
5055($addr, $mask) = @_;
5056# We have to mask $addr with $mask because linux /sbin/route
5057# complains if the network address doesn't match the netmask.
5058#
5059my @ip = split( /\./, $addr );
5060my @mask = split( /\./, $mask );
5061$ip[$_] = ($ip[$_] + 0) & ($mask[$_] + 0) for (0..$#ip);
5062$addr = join(".", @ip);
5063return $addr;
5064}
5065for (qx{ $Ifconfig }) {
5066last unless defined $_;
5067# If we got a new device, stash the previous one (if any).
5068if ( /^([^\s:]+)/o ) {
5069if ( $iface eq $ExternalDevice and $network and $up ) {
5070$LocalNetwork = $network;
5071last;
5072}
5073$iface = $1;
5074$up = 0;
5075}
5076# Get the network mask for the current interface.
5077if ( /addr:$IP_Match.*?mask:$IP_Match/io ) {
5078# Linux style ifconfig.
5079compute_netmask($1, $2);
5080$network = "$addr netmask $mask";
5081} elsif ( /inet $IP_Match.*?mask 0x([a-f0-9]{8})/io ) {
5082# BSD style ifconfig.
5083($addr, $mask) = ($1, $2);
5084$mask = join(".", map( hex $_, $mask =~ /(..)/gs ));
5085compute_netmask($addr, $mask);
5086$network = "$addr/$mask";
5087}
5088# Ignore interfaces that are loopback devices or aren't up.
5089$iface = "" if /\bLOOPBACK\b/o;
5090$up++ if /\bUP\b/o;
5091}
5092die "Can't determine local IP address!\n" unless $LocalIP and $LocalNetwork;
5093# Set OS dependent variables.
5094#
5095my ( $GW, $NET, $PTP );
5096if ( $^O eq "linux" ) {
5097$GW = "gw"; $PTP = "pointopoint"; $NET = "-net";
5098} else {
5099110 | Networking
5100#53 Automatic vtund.conf Generator
5101HACK
5102$GW = $PTP = $NET = "";
5103}
5104# Parse the config template.
5105#
5106$template =~ s/(\$\w+)/$1/gee;
5107# Write the temp file and execute vtund.
5108#
5109if ($Debug) {
5110print $template;
5111} else {
5112print $file $template;
5113close $file;
5114system("$Vtund $name");
5115}
5116__DATA__
5117options {
5118port $TunnelPort;
5119ifconfig /sbin/ifconfig;
5120route /sbin/route;
5121}
5122default {
5123compress no;
5124speed 0;
5125}
5126$TunnelName { # 'mytunnel' should really be `basename $0` or some such
5127# for automagic config selection
5128type tun;
5129proto tcp;
5130keepalive yes;
5131pass $Secret;
5132up {
5133ifconfig "%% $LocalIP $PTP $RemoteIP arp";
5134route "add $TunnelHost $GW $GatewayIP";
5135route "delete default";
5136route "add default $GW $RemoteIP";
5137route "add $NET $LocalNetwork $GW $GatewayIP";
5138};
5139down {
5140ifconfig "%% down";
5141route "delete default";
5142route "delete $TunnelHost $GW $GatewayIP";
5143route "delete $NET $LocalNetwork";
5144route "add default $GW $GatewayIP";
5145};
5146}
5147111
5148Chapter 5
5149C H A P T E R F I V E
5150Monitoring
5151Hacks #54–65
5152It is difficult to know how to tune a running system if you have no idea how a
5153system “normally” runs. By asking careful questions of the system (and inter-
5154preting the answers correctly), you can avoid poking around at variables in
5155the dark and make effective changes exactly where they need to be made.
5156This is where log files can be your best friend. Treat them well, and pay
5157them the attention they’re due; you will learn volumes about how your sys-
5158tem is being used. But if you simply let them fill up your disks, they can be a
5159source of much confusion. With the proper tools and techniques, your sys-
5160tem logs will be concise and detailed enough to tell you exactly what you
5161need to know.
5162But sometime, the information you’re after doesn’t get logged anywhere, but
5163is expressed in the running Linux system, either as a pinpoint check of system
5164resources or data on the network itself. Incidentally, we won’t examine full-
5165blown system monitoring and trending packages (such as Nagios or MRTG)
5166in this chapter but instead will look at ways to interrogate your system to get
5167specific information about what’s going on right now. We’ll also see a couple
5168of ways of to detect potential problems before they happen and even how to
5169automatically deal with catastrophic failures when they do occur.
5170H A C K
5171#54
5172Steering syslog Hack #54
5173Make syslog work harder, and spend less time looking through huge log files
5174The default syslog installation on many distributions doesn’t do a very good
5175job of filtering classes of information into separate files. If you see a jumble
5176of messages from sendmail, sudo, bind, and other system services in /var/log/
5177messages, then you should probably review your /etc/syslog.conf.
5178There are a number of facilities and priorities on which syslog can filter. For
5179easy reference, here they are:
5180112 | Monitoring
5181#54 Steering syslog
5182HACK
5183Note that applications decide for themselves at what facility and priority to
5184log (and the best applications let you choose), so they may not always be
5185logged as you expect. Here’s a sample /etc/syslog.conf that attempts to shuf-
5186fle around what gets logged where:
5187auth.warning /var/log/auth
5188mail.err /var/log/maillog
5189kern.* /var/log/kernel
5190cron.crit /var/log/cron
5191*.err;mail.none /var/log/syslog
5192*.info;auth.none;mail.none /var/log/messages
5193#*.=debug /var/log/debug
5194local0.info /var/log/cluster
5195local1.err /var/log/spamerica
5196All of the above lines will log the specified priority (or higher) to the respec-
5197tive file. The special priority none tells syslog not to bother logging the speci-
5198fied facility at all. The local0 through local7 facilities are supplied for use
5199with your own programs, however you see fit. For example, the /var/log/
5200spamerica file fills with local1.err (or higher) messages that are generated by
5201our spam processing job. It’s nice to have those messages separate from the
5202standard mail delivery log (in /var/log/maillog.)
5203That commented *.=debug line is useful when debugging daemonized ser-
5204vices. It tells syslog to specifically log only debug priority messages of any
5205facility and generally shouldn’t be running (unless you don’t mind filling
5206your disks with debug logs). Another approach is to log debug information to
5207a fifo. This will make debug logs take up no space but will disappear unless a
5208process is watching it. To log to a fifo, first create it in the filesystem:
5209#mkfifo -m 0664 /var/log/debug
5210Facilities Priorities
5211auth debug
5212auth-priv info
5213cron notice
5214daemon warning
5215kern err
5216lpr crit
5217mail alert
5218news emerg
5219syslog
5220user
5221uucp
5222local0 - local7
5223Steering syslog #54
5224Monitoring | 113
5225HACK
5226Then amend the debug line in syslog.conf to include a | like this:
5227*.=debug |/var/log/debug
5228Now debug information is constantly logged to the fifo, and can be viewed
5229with a command like less -f /var/log/debug . This is also handy to set up if
5230you want a process to constantly watch all system messages and perhaps
5231notify you via email when a critical system message is seen. Try making a fifo
5232called /var/log/monitor, and add a rule like this to your syslog.conf:
5233*.* |/var/log/monitor
5234Now every message (at every priority) is passed to the /var/log/monitor fifo,
5235and any process watching it can react accordingly, all without taking up any
5236disk space.
5237Mark Who?
5238Do you notice a bunch of lines like this in /var/log/messages?
5239Dec 29 18:33:35 catlin -- MARK --
5240Dec 29 18:53:35 catlin -- MARK --
5241Dec 29 19:13:35 catlin -- MARK --
5242Dec 29 19:33:35 catlin -- MARK --
5243Dec 29 19:53:35 catlin -- MARK --
5244Dec 29 20:13:35 catlin -- MARK --
5245Dec 29 20:33:35 catlin -- MARK --
5246Dec 29 20:53:35 catlin -- MARK --
5247Dec 29 21:13:35 catlin -- MARK --
5248These are generated by the mark functionality of syslog, as a way of “touch-
5249ing base” with the system, so that you can theoretically tell if syslog has
5250unexpectedly died. Most times, this only serves to fill your logfiles, and
5251unless you are having problems with syslog, you probably don’t need it. To
5252turn this off, pass the -m 0 switch to syslogd (after first killing any running
5253syslogd), like this:
5254#killall syslogd; /usr/sbin/syslogd -m 0
5255Remote Logging
5256Modern versions of syslogd disable the ability to receive logs from remote
5257machines and for good reason. Without safeguards in place, it is entirely
5258possible for a random miscreant to fill up your disk with bogus syslog mes-
5259sages, as there is no host authentication available in syslog. Still, it is very
5260handy to have a centralized syslog, particularly when dealing with a cluster
5261of machines.
5262To turn on remote reception on the master, start syslogd with -r:
5263# /usr/sbin/syslogd -m 0 -r
5264114 | Monitoring
5265#55 Watching Jobs with watch
5266HACK
5267On each of the machines you’d like to log from, add this to the syslog.conf:
5268*.* @master.syslog.host.com
5269(Naturally, with your own hostname or IP address after the @.) It is a very
5270good idea to protect the syslog port on your syslog server. It listens on UDP
5271port 514 and is easily filtered with this iptables command:
5272# iptables -A INPUT -t filter -p udp --dport 514 -s ! $LOCAL_NET -j DROP
5273where $LOCAL_NET is the network (or host) that you would like to receive
5274sylog messages from (e.g., 192.168.1.0/24 or florian.nocat.net).
5275Once your system logs are better organized, it becomes much easier to find
5276the information you’re looking for. Even then, system logs can get a bit over-
5277whelming. For even more help dealing with log files, check out “Colorized
5278Log Analysis in Your Terminal” [Hack #75] .
5279H A C K
5280#55
5281Watching Jobs with watch Hack #55
5282Use watch to repeatedly run any command, and show you the results
5283If you have ever had a long-running background process, chances are you
5284are used to repeatedly checking to see if the command has finished using ps
5285and grep:
5286mc@escher:~$ps ax |grep tar
528710303 ? S 0:00 bash -c cd /; tar cf - home
528810304 ? S 0:42 tar cf - home
5289mc@escher:~$ps ax |grep tar
529010303 ? S 0:00 bash -c cd /; tar cf - home
529110304 ? S 0:43 tar cf - home
5292Or maybe you’re ftping a file and forget the hash command (and for some rea-
5293son, aren’t using ncftp, scp, or wget or curl), so you have no idea how far along
5294your transfer is. Naturally, you log in again and take a look at the file with ls:
5295mc@escher:~$ls -l xfree86.tgz
5296-rw-r--r-- 1 rob users 12741812 Jun 13 2001 xfree86.tgz
5297mc@escher:~$ls -l xfree86.tgz
5298-rw-r--r-- 1 rob users 12744523 Jun 13 2001 xfree86.tgz
5299Any time you find yourself running a command over and over again, try the
5300watch command. It will repeatedly cycle over any command you give it, and
5301whatever interval you’d like (defaulting to 2 seconds). It clears the screen on
5302each pass, making a nice, easy to read display.
5303mc@escher:~$watch 'ps ax |grep tar'
5304Every 2s: ps ax |grep tar|grep -v pts/0 Fri Sep 6 00:22:01 2002
530510303 ? S 0:00 bash -c cd /; tar cf - home
530610304 ? S 0:42 tar cf - home
5307What’s Holding That Port Open? #56
5308Monitoring | 115
5309HACK
5310You’ll only need to enclose the command in single quotes if you’re using
5311pipes (or other special characters that you don’t want interpolated before
5312watch runs). To specify a different time interval, try the -n switch:
5313mc@escher:~$watch -n1 ls -l xfree86.tgz
5314Every 1s: ls -l xfree86.tgz Fri Sep 6 00:31:41 2002
5315-rw-r--r-- 1 rob users 12756042 Jun 13 2001 xfree86.tgz
5316It will even highlight the differences on each pass, making changes leap out
5317in reverse type. Try the -d switch for that (it’s especially nice for watching
5318the output of a netstat -a | grep ESTAB , to watch network connections
5319come and go. Think of watch as your obsessive-compulsive little buddy. It
5320obsesses at your slightest whim, so you don’t have to. To exit, hit ^C.
5321See also:
5322• man watch
5323H A C K
5324#56
5325What’s Holding That Port Open? Hack #56
5326Associate a process with the port it is bound to easily with netstat
5327Generating a list of network ports that are in the Listen state on a Linux
5328server is simple with netstat:
5329root@catlin:~#netstat -ln
5330Active Internet connections (only servers)
5331Proto Recv-Q Send-Q Local Address Foreign Address State
5332tcp 0 0 0.0.0.0:5280 0.0.0.0:* LISTEN
5333tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN
5334tcp 0 0 10.42.3.2:53 0.0.0.0:* LISTEN
5335tcp 0 0 10.42.4.6:53 0.0.0.0:* LISTEN
5336tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN
5337tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
5338udp 0 0 10.42.3.2:53 0.0.0.0:*
5339udp 0 0 10.42.4.6:53 0.0.0.0:*
5340udp 0 0 127.0.0.1:53 0.0.0.0:*
5341udp 0 0 0.0.0.0:67 0.0.0.0:*
5342raw 0 0 0.0.0.0:1 0.0.0.0:* 7
5343So, we see the usual services (a web server on port 80, DNS on port 53, ssh on
5344port 22, dhcp on port 67), but what’s that process listening on 5280?
5345Finding out which programs are actually bound to those ports is simple with
5346recent versions of netstat. As long as you’re root, just add the -p switch (for
5347programs):
5348root@catlin:~#netstat -lnp
5349Active Internet connections (only servers)
5350Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
5351116 | Monitoring
5352#57 Checking On Open Files and Sockets with lsof
5353HACK
5354tcp 0 0 0.0.0.0:5280 0.0.0.0:* LISTEN 698/perl
5355tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 217/httpd
5356tcp 0 0 10.42.3.2:53 0.0.0.0:* LISTEN 220/named
5357tcp 0 0 10.42.4.6:53 0.0.0.0:* LISTEN 220/named
5358tcp 0 0 127.0.0.1:53 0.0.0.0:* LISTEN 220/named
5359tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 200/sshd
5360udp 0 0 0.0.0.0:32768 0.0.0.0:* 220/named
5361udp 0 0 10.42.3.2:53 0.0.0.0:* 220/named
5362udp 0 0 10.42.4.6:53 0.0.0.0:* 220/named
5363udp 0 0 127.0.0.1:53 0.0.0.0:* 220/named
5364udp 0 0 0.0.0.0:67 0.0.0.0:* 222/dhcpd
5365raw 0 0 0.0.0.0:1 0.0.0.0:* 7 222/dhcpd
5366Ah, that’s better. PID 698 is a Perl process that is bound to port 5280. We
5367now hunt it down with ps:
5368root@catlin:~#ps auwex |grep -w 698
5369nocat 698 0.0 2.0 5164 3840 ? S Aug25 0:00 /usr/bin/perl -w ./bin/gateway
5370PWD=/usr/local/nocat HOSTNAME=catlin.r
5371The ps aweux shows us all (a) non-interactive (x) processes with user infor-
5372mation (u) in wide format (w) with some environment bits appended (e).
5373We then grep on word boundaries (-w) for the PID.
5374That’s better: now we know that the nocat user is in the /usr/local/nocat/
5375running bin/gateway, a Perl process that is listening on port 5280. Without
5376the -p switch on netstat, associating an open port with a particular process is
5377much trickier.
5378Incidentally, if you’re not root, then the system won’t disclose which pro-
5379grams are running on which ports. If you see an error like this:
5380(No info could be read for "-p": geteuid()=1000 but you should be root.)
5381then come back when you’re root! (Or see “Make sudo Work Harder” [Hack
5382#12] for how to sudo yourself into an amazing simulation of root.)
5383H A C K
5384#57
5385Checking On Open Files and Sockets with
5386lsof Hack #57
5387Easily see which files, directories, and sockets your running processes are
5388holding open
5389Have you ever tried to umount a filesystem, only to find that some process
5390was still using it?
5391root@mouse:~#umount /mnt
5392umount: /mnt: device is busy
5393To quickly hunt down what processes are still using /mnt, try the lsof tool:
5394root@mouse:~#lsof /mnt
5395COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
5396bash 30951 rob cwd DIR 7,0 1024 2 /mnt
5397Checking On Open Files and Sockets with lsof #57
5398Monitoring | 117
5399HACK
5400Ah, apparently rob is cd’d to /mnt (since his bash process has it set as its
5401cwd). lsof will list all open files, directories, libraries, sockets, and devices
5402associated with a particular process. In the above example, we specified a
5403mount point and had lsof show us the associated processes. To do the
5404reverse (show files associated with a PID), use the -p switch:
5405root@mouse:~#lsof -p 30563
5406COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
5407inetd 30563 root cwd DIR 3,3 408 2 /
5408inetd 30563 root rtd DIR 3,3 408 2 /
5409inetd 30563 root txt REG 3,3 21432 39140 /usr/sbin/inetd
5410inetd 30563 root mem REG 3,3 432647 11715 /lib/ld-2.2.3.so
5411inetd 30563 root mem REG 3,3 4783716 11720 /lib/libc-2.2.3.so
5412inetd 30563 root mem REG 3,3 19148 11708 /lib/libnss_db-2.2.so
5413inetd 30563 root mem REG 3,3 238649 11728 /lib/libnss_files-2.2.3.so
5414inetd 30563 root mem REG 3,3 483324 11710 /lib/libdb-3.1.so
5415inetd 30563 root 0u CHR 1,3 647 /dev/null
5416inetd 30563 root 1u CHR 1,3 647 /dev/null
5417inetd 30563 root 2u CHR 1,3 647 /dev/null
5418inetd 30563 root 4u IPv4 847222 TCP *:telnet (LISTEN)
5419inetd 30563 root 5u IPv4 560439 TCP *:cvspserver (LISTEN)
5420If you’d rather specify the process by name, use -c:
5421root@mouse:~#lsof -c syslogd
5422COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
5423syslogd 25627 root cwd DIR 3,3 408 2 /
5424syslogd 25627 root rtd DIR 3,3 408 2 /
5425syslogd 25627 root txt REG 3,3 27060 5538 /usr/sbin/syslogd
5426syslogd 25627 root mem REG 3,3 432647 11715 /lib/ld-2.2.3.so
5427syslogd 25627 root mem REG 3,3 4783716 11720 /lib/libc-2.2.3.so
5428syslogd 25627 root mem REG 3,3 238649 11728 /lib/libnss_files-2.2.3.so
5429syslogd 25627 root mem REG 3,3 75894 11719 /lib/libnss_dns-2.2.3.so
5430syslogd 25627 root mem REG 3,3 225681 11724 /lib/libresolv-2.2.3.so
5431syslogd 25627 root mem REG 3,3 19148 11708 /lib/libnss_db-2.2.so
5432syslogd 25627 root mem REG 3,3 483324 11710 /lib/libdb-3.1.so
5433syslogd 25627 root 0u unix 0xdcc2d5b0 775254 /dev/log
5434syslogd 25627 root 1w REG 3,3 135744 5652 /var/log/debug
5435syslogd 25627 root 2w REG 3,3 107459 5651 /var/log/syslog
5436syslogd 25627 root 3w REG 3,3 107054 58317 /var/log/maillog
5437syslogd 25627 root 4w REG 3,3 4735 16 /var/log/authlog
5438You can also specify special devices on the command line. For example, let’s
5439see what the user on pts/0 is up to:
5440rob@mouse:~#lsof /dev/pts/0
5441COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
5442bash 29816 rob 0u CHR 136,0 2 /dev/pts/0
5443bash 29816 rob 1u CHR 136,0 2 /dev/pts/0
5444bash 29816 rob 2u CHR 136,0 2 /dev/pts/0
5445bash 29816 rob 255u CHR 136,0 2 /dev/pts/0
5446lsof 30882 root 0u CHR 136,0 2 /dev/pts/0
5447lsof 30882 root 1u CHR 136,0 2 /dev/pts/0
5448lsof 30882 root 2u CHR 136,0 2 /dev/pts/0
5449118 | Monitoring
5450#57 Checking On Open Files and Sockets with lsof
5451HACK
5452If you need to specify multiple switches, they are ORed with each other by
5453default. To require all switches (that is, to AND them) include the -a flag on
5454each switch you want to AND. For example, to see all of the open files asso-
5455ciated with vi processes that rob is running, try this:
5456root@mouse:~#lsof -u rob -ac vi
5457COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
5458vi 31059 rob cwd DIR 3,3 2824 39681 /home/rob
5459vi 31059 rob rtd DIR 3,3 408 2 /
5460vi 31059 rob txt REG 3,3 554504 9799 /usr/bin/vim
5461vi 31059 rob mem REG 3,3 432647 11715 /lib/ld-2.2.3.so
5462vi 31059 rob mem REG 3,3 282178 2825 /lib/libncurses.so.5.2
5463vi 31059 rob mem REG 3,3 76023 2831 /usr/lib/libgpm.so.1.18.0
5464vi 31059 rob mem REG 3,3 4783716 11720 /lib/libc-2.2.3.so
5465vi 31059 rob mem REG 3,3 249120 11721 /lib/libnss_compat-2.2.3.so
5466vi 31059 rob mem REG 3,3 357644 11725 /lib/libnsl-2.2.3.so
5467vi 31059 rob 0u CHR 136,1 3 /dev/pts/1
5468vi 31059 rob 1u CHR 136,1 3 /dev/pts/1
5469vi 31059 rob 2u CHR 136,1 3 /dev/pts/1
5470vi 31059 rob 3u REG 3,3 4096 15 /home/rob/.sushi.c.swp
5471If you’d like to examine open sockets and their associated processes (like a
5472netstat -p), try the -i switch:
5473rob@mouse:~#lsof -i
5474COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
5475sshd 69 root 3u IPv4 61 TCP *:ssh (LISTEN)
5476mysqld 126 mysql 3u IPv4 144 TCP *:3306 (LISTEN)
5477mysqld 128 mysql 3u IPv4 144 TCP *:3306 (LISTEN)
5478mysqld 129 mysql 3u IPv4 144 TCP *:3306 (LISTEN)
5479httpd 24905 root 22u IPv4 852520 TCP *:443 (LISTEN)
5480httpd 24905 root 23u IPv4 852521 TCP *:www (LISTEN)
5481httpd 28383 www 4u IPv4 917713 TCP nocat.net:www->65.192.187.158:8648
5482(ESTABLISHED)
5483httpd 28389 www 4u IPv4 917714 TCP nocat.net:www->65.192.187.158:9832
5484(ESTABLISHED)
5485httpd 28389 www 22u IPv4 852520 TCP *:443 (LISTEN)
5486httpd 28389 www 23u IPv4 852521 TCP *:www (LISTEN)
5487exim 29879 exim 0u IPv4 557513 TCP *:smtp (LISTEN)
5488inetd 30563 root 4u IPv4 847222 TCP *:telnet (LISTEN)
5489inetd 30563 root 5u IPv4 560439 TCP *:cvspserver (LISTEN)
5490sshd 30973 root 4u IPv4 901571 TCP nocat.net:ssh->some.where.net:52543
5491(ESTABLISHED)
5492sshd 30991 rob 4u IPv4 901577 TCP nocat.net:ssh->some.where.else.net:52544
5493(ESTABLISHED)
5494Note that you must be root to run lsof for many functions, including retriev-
5495ing open socket information. lsof is a complex and very flexible tool, giving
5496you as much (or as little) detail as you need about what files are in use by
5497every running process on your system.
5498Monitor System Resources with top #58
5499Monitoring | 119
5500HACK
5501See also:
5502• The latest version of lsof can be downloaded at ftp://vic.cc.purdue.edu/
5503pub/tools/unix/lsof/.
5504H A C K
5505#58
5506Monitor System Resources with top Hack #58
5507Use the top utility to get a better overview of what your system is up to
5508The top command can give you up-to-the-second reporting of system load,
5509memory usage, and CPU utilization. It is distributed as part of the procps
5510package. The simplest way to get started is to simply run top from the com-
5511mand line:
5512$top
5513You’ll be presented with a screenful of information updated every two
5514seconds.
55153:54pm up 1 day, 16 min, 2 users, load average: 0.00, 0.00, 0.00
551638 processes: 37 sleeping, 1 running, 0 zombie, 0 stopped
5517CPU states: 0.0% user, 0.7% system, 0.0% nice, 99.2% idle
5518Mem: 189984K av, 155868K used, 34116K free, 0K shrd, 42444K buff
5519Swap: 257032K av, 0K used, 257032K free 60028K cached
5520PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
55216195 rob 14 0 1004 1004 800 R 0.5 0.5 0:00 top
55221 root 8 0 212 212 180 S 0.0 0.1 0:13 init
55232 root 9 0 0 0 0 SW 0.0 0.0 0:00 keventd
55243 root 9 0 0 0 0 SW 0.0 0.0 0:00 kswapd
55254 root 9 0 0 0 0 SW 0.0 0.0 0:00 kreclaimd
55265 root 9 0 0 0 0 SW 0.0 0.0 0:00 bdflush
55276 root 9 0 0 0 0 SW 0.0 0.0 0:00 kupdated
55288 root -1 -20 0 0 0 SW< 0.0 0.0 0:00 mdrecoveryd
5529176 root 9 0 788 788 680 S 0.0 0.4 0:00 syslogd
5530179 root 9 0 1228 1228 444 S 0.0 0.6 0:00 klogd
5531182 root 8 0 1228 1228 1104 S 0.0 0.6 0:06 sshd
5532184 root 8 0 616 616 520 S 0.0 0.3 0:00 crond
5533186 daemon 9 0 652 652 560 S 0.0 0.3 0:00 atd
5534197 root 9 0 2544 2544 2396 S 0.0 1.3 0:00 httpd
5535200 root 9 0 3740 3740 1956 S 0.0 1.9 0:00 named
5536202 root 9 0 1004 1004 828 S 0.0 0.5 0:00 dhcpd
5537203 root 9 0 504 504 444 S 0.0 0.2 0:00 agetty
5538Hit ? while top is running to get a list of available commands. A couple of
5539very useful display keys are M (which sorts on resident memory size), P
5540(which sorts by CPU usage again), S (to toggle cumulative runtime, that is,
5541how long each process and all of its children have been running, in CPU sec-
5542onds), and i (to stop displaying idle processes.)
5543If you’re running top as root, there are a couple of other interactive and sort-
5544ing commands that you’ll likely find useful. The u key lets you filter out all
5545D ownload from Wow! eBook <www.wowebook.com>
5546120 | Monitoring
5547#59 Constant Load Average Display in the Titlebar
5548HACK
5549processes except those owned by a given user. Follow that up with k, which
5550lets you interactively kill a given PID (with any signal you like.) This can be
5551really handy for hunting down runaway processes, and killing them from
5552inside top (maybe even copying-and-pasting the offending PID to avoid the
5553dreaded kill typo).
5554It can be handy to leave top running in a terminal window on busy machines
5555that you’re logged into, but aren’t otherwise working on. If you’d like to see
5556a continual load average display in the title bar of a login window (for a sort
5557of mini-top while remaining in your shell), check out “Constant Load Aver-
5558age Display in the Titlebar” [Hack #59] .
5559See also:
5560• “Manipulating Processes Symbolically with procps” [Hack #17]
5561• “Constant Load Average Display in the Titlebar” [Hack #59]
5562• procps home: ftp://people.redhat.com/johnsonm/procps/
5563H A C K
5564#59
5565Constant Load Average Display in the
5566Titlebar Hack #59
5567Make your title bar work harder, so you don’t have to.
5568If you have managed a Linux server for any length of time, you’re probably
5569intimately familiar with the top utility. (If not, stop reading and immediately
5570go type top in the nearest terminal window, and hit ? when you get bored.)
5571If you’ve managed a number of Linux servers, then you probably know what
5572it’s like to have several tops running in multiple windows, all competing for
5573desktop real estate.
5574In computing, wasted resources are resources that could be better spent
5575helping you. Why not run a process that updates the titlebar of your termi-
5576nal with the current load average in real time, regardless of what else you’re
5577running?
5578Save this as a script called tl, and save it to your ~/bin directory:
5579Listing: tl
5580#!/usr/bin/perl -w
5581use strict;
5582$|++;
5583my $host=`/bin/hostname`;
5584chomp $host;
5585while(1) {
5586Network Monitoring with ngrep #60
5587Monitoring | 121
5588HACK
5589open(LOAD,"/proc/loadavg") || die "Couldn't open /proc/loadavg: $!\n";
5590my @load=split(/ /,<LOAD>);
5591close(LOAD);
5592print "\033]0;";
5593print "$host: $load[0] $load[1] $load[2] at ", scalar(localtime);
5594print "\007";
5595sleep 2;
5596}
5597When you’d like to have your titlebar replaced with the name, load average,
5598and current time of the machine you’re logged into, just run tl&. It will hap-
5599pily go on running in the background, even if you’re running an interactive
5600program like vim. If you have your titlebar already set to show the current
5601working directory, no problem. When you cd, the current working directory
5602is flashed momentarily, and then replaced with the time and load average
5603again. Need to see that directory one more time? Just hit Enter on a blank
5604line, and it will flash again.
5605Now instead of spreading terminals around and completely covering your
5606desktop, you can stack a pile of them together (leaving only your titlebars
5607showing), and enjoy an “at a glance” view of how hard each machine is work-
5608ing. When you need to work on a machine, just bring that window to the fore-
5609ground, and immediately begin working. Of course, this is all available to you
5610without installing a single piece of software on your local machine.
5611When you’re finished, don’t forget to killall tl before logging out. Or if
5612you’re terminally lazy, try this:
5613$ echo 'killall tl > /dev/null 2>&1' >> ~/.bash_logout
5614That will kill all of your running tl jobs on logout, without even having to lift
5615a finger. System administration just keeps getting easier, doesn’t it?
5616H A C K
5617#60
5618Network Monitoring with ngrep Hack #60
5619See who’s doing what, with a grep for your network interface
5620The ngrep utility is an interesting packet capture tool, similar to tcpdump or
5621snoop. It is unique in that it attempts to make it as easy as possible to match
5622which captured packets to print, by using a grep compatible format (com-
5623plete with regular expressions and a bunch of GNU grep’s switches). It also
5624converts the packets to ASCII (or hex) before printing.
5625For example, to see the contents of all http GET requests that pass through
5626your router, try this:
5627#ngrep -q GET
5628122 | Monitoring
5629#60 Network Monitoring with ngrep
5630HACK
5631If you’re only interested in a particular host, protocol, or port (or other
5632packet matching criteria), you can specify a bpf filter as well as a data pat-
5633tern. It uses a syntax similar to tcpdump:
5634#ngrep -qi rob@nocat.net port 25
5635T 10.42.4.7:65174 -> 209.204.146.26:25 [AP]
5636RCPT TO:<rob@nocat.net>..
5637T 209.204.146.26:25 -> 10.42.4.7:65174 [AP]
5638250 2.1.5 <rob@nocat.net>... Recipient ok..
5639T 10.42.4.7:65174 -> 209.204.146.26:25 [AP]
5640Date: Sun, 8 Sep 2002 23:55:18 -0700..Mime-Version: 1.0 (Apple Message fram
5641ework v543)..Content-Type: text/plain; charset=US-ASCII; format=flowed..Sub
5642ject: Greetings.....From: John Doe <johnd@somewhere.else.com>..To: rob@noca
5643t.net..Content-Transfer-Encoding: 7bit..Message-Id: <19DB8C16-C3C1-11D6-B23
56449-0003936D6AE0@somewhere.else.com>..X-Mailer: Apple Mail v2)....What does t
5645hat pgp command you mentioned do again?....Thanks,....--A Friend....
5646Since ngrep prints to STDOUT, you can do post-processing on the output to
5647make a nice printing filter. If you process the output yourself, add the -l
5648switch to make the output line buffered. For example, if you’re interested in
5649what people on your network are searching for online, try something like
5650this bit of Perl.
5651Listing: go-ogle
5652#!/usr/bin/perl
5653use Socket;
5654$|++;
5655open(NG,"ngrep -lqi '(GET|POST).*/(search|find)' |");
5656print "Go ogle online.\n";
5657my ($go,$i) = 0;
5658my %host = ( );
5659while(<NG>) {
5660if(/^T (\d+\.\d+.\d+\.\d+):\d+ -> (\d+\.\d+\.\d+\.\d+):80/) {
5661$i = inet_aton($1);
5662$host{$1} ||= gethostbyaddr($i, AF_INET) || $1;
5663$i = inet_aton($2);
5664$host{$2} ||= gethostbyaddr($i, AF_INET) || $2;
5665print "$host{$1} -> $host{$2} : ";
5666$go = 1;
5667next;
5668}
5669if(/(q|p|query|for)=(.*)?(&|HTTP)/) {
5670next unless $go;
5671my $q = $2;
5672$q =~ s/(\+|&.*)/ /g;
5673$q =~ s/%(\w+)/chr(hex($1))/ge;
5674Scanning Your Own Machines with nmap #61
5675Monitoring | 123
5676HACK
5677print "$q\n";
5678$go = 0;
5679}
5680}
5681I call the script go-ogle. This will run an ngrep looking for any GET or POST
5682request that includes search or find somewhere in the URL. The results look
5683something like this:
5684Go ogle online.
5685caligula.nocat.net -> www.google.com : o'reilly mac os x conference
5686caligula.nocat.net -> s1.search.vip.scd.yahoo.com : junk mail $$$
5687tiberius.nocat.net -> altavista.com : babel fish
5688caligula.nocat.net -> 166-140.amazon.com : Brazil
5689livia.nocat.net -> 66.161.12.119 : lart
5690It will unescape encoded strings in the query (note the ‘ in the google query
5691and the $$$ from yahoo). It will also convert IP addresses to hostnames for
5692you (since ngrep doesn’t seem to have that feature, probably so it can opti-
5693mize capturing for speed). The last two results are interesting: the Brazil query
5694was actually run on http://www.imdb.com/, and the last one was to http://www.
5695dictionary.com/. Evidently IMDB is now in a partnership with Amazon, and
5696Dictionary.com’s search machine doesn’t have a PTR record. It’s amazing
5697how much you can learn about the world by watching other people’s packets.
5698Note that you must be root to run ngrep, and for best results, it should be
5699run from the router at the edge of your network.
5700See also:
5701• man ngrep
5702• http://www.packetfactory.net/Projects/ngrep/
5703H A C K
5704#61
5705Scanning Your Own Machines with nmap Hack #61
5706Find out when servers and services come online anywhere on your network
5707If you haven’t used it before, nmap is a tremendously useful tool for identi-
5708fying machines and services on your network. It will perform a number of
5709different types of network scanning (from standard TCP and UDP to more
5710exotic scans like stealth TCP SYN scans, Xmas Tree and NULL probes, and
5711a bunch of other fun options).
5712Even more interesting is the OS fingerprinting code, which analyzes packets
5713returned by the target machine and compares the results against a database
5714of known operating systems. This is a fascinating bit of code, in that it can
5715typically identify the remote side’s operating system without connecting to
5716any actual services, and even return an estimated uptime for the machine
5717being scanned.
5718124 | Monitoring
5719#61 Scanning Your Own Machines with nmap
5720HACK
5721To perform a standard port sweep with OS fingerprinting, try the -O switch:
5722rob@catlin:~#nmap -O caligula
5723Starting nmap V. 3.00 ( www.insecure.org/nmap/ )
5724Interesting ports on caligula.rob.nocat (10.42.4.7):
5725(The 1600 ports scanned but not shown below are in state: closed)
5726Port State Service
572722/tcp open ssh
5728Remote operating system guess: Mac OS X 10.1 - 10.1.4
5729Uptime 5.760 days (since Tue Sep 3 19:14:36 2002)
5730Nmap run completed -- 1 IP address (1 host up) scanned in 31 seconds
5731If you’d like to nmap your entire network and have a bit of time to kill, you
5732can specify a network and subnet on the command line. This performs a TCP
5733SYN scan and fingerprinting for the first 64 addresses of 10.42.4.0:
5734root@catlin:~#nmap -OsS 10.42.4.0/26
5735Since nmap prints to STDOUT, you can save the output of a scan run and
5736compare it against previous runs for a differential report, quite easily. We’ll
5737run an Xmas tree scan and grep out a couple of lines (like the run time) to
5738eliminate false positives:
5739root@catlin:~#nmap -sX 10.42.4.0/26 | egrep -v '^(Nmap|Starting)' \
5740> nmap.output
5741Let’s run the same command again (say, the next day, at a random hour):
5742root@catlin:~#nmap -sX localhost | egrep -v '^(Nmap|Starting)' \
5743> nmap.output2
5744and let’s do a context diff to see what changed:
5745root@catlin:~#diff -c nmap.output*
5746*** nmap.output Mon Sep 9 14:45:06 2002
5747--- nmap.output2 Mon Sep 9 14:45:21 2002
5748***************
5749*** 1,7 ****
5750Interesting ports on catlin.rob.nocat (10.42.4.6):
5751! (The 1598 ports scanned but not shown below are in state: closed)
5752Port State Service
575322/tcp open ssh
575453/tcp open domain
575580/tcp open http
5756--- 1,8 ----
5757Interesting ports on catlin.rob.nocat (10.42.4.6):
5758! (The 1597 ports scanned but not shown below are in state: closed)
5759Port State Service
5760+ 21/tcp open ftp
576122/tcp open ssh
5762Disk Age Analysis #62
5763Monitoring | 125
5764HACK
576553/tcp open domain
576680/tcp open http
5767root@catlin:~#
5768Fascinating. It looks like catlin has picked up an ftp server at some point. This
5769technique will find new (and dead) hosts and services each time it is run. By
5770keeping an archive of nmap output (perhaps logged to time and date encoded
5771files, or even to a database) you can keep a log of the state of all machines on
5772your network. Turning it into a shell script and running it from cron is left as
5773an exercise (a hopefully fun, and definitely worthwhile exercise) for the reader.
5774See also:
5775• nmap’s home: http://www.insecure.org/nmap/
5776H A C K
5777#62
5778Disk Age Analysis Hack #62
5779Easily identify which parts of your disk change frequently
5780How can you quickly tell which parts of your filesystem are modified fre-
5781quently and which haven’t changed in months? It’s very straightforward,
5782with a proper application of Perl.
5783Here is a Perl script that will perform disk aging analysis on a filesystem. The
5784program breaks down disk space two ways, by last modified date and last
5785accessed date. A sample run looks like:
5786%diskage /usr/local
5787Disk aging analysis for /usr/local:
5788last num last num
5789Age (days) modified files accessed files
57900 - 30 260403 Kb 817 140303 Kb 6968
579131 - 60 11789 Kb 226 23140 Kb 199
579261 - 90 40168 Kb 1126 1087585 Kb 31625
579391 - 180 118927 Kb 995 0 Kb 0
5794181 - 365 85005 Kb 1889 0 Kb 0
5795366 - 9999 734735 Kb 33739 0 Kb 0
5796----------- ----- ----------- -----
5797Total 1251029 Kb 38792 1251029 Kb 38792
5798You can run the script with the -v option to list the last modified and last
5799accessed days for every file in the filesystem.
5800Listing: diskage
5801#!/usr/local/bin/perl
5802#
5803# Disk aging report generator
5804126 | Monitoring
5805#62 Disk Age Analysis
5806HACK
5807# Written by Seann Herdejurgen
5808#
5809# May 1998
5810use File::Find;
5811# Initialize variables
5812@levels=(30,60,90,180,365,9999);
5813# Check for verbose flag
5814if ($ARGV[0] eq "-v") {
5815$verbose++;
5816shift(@ARGV);
5817}
5818$ARGV[0]=$ENV{'PWD'} if ($ARGV[0] eq "");
5819foreach $dir (@ARGV) {
5820foreach $level (@levels) {
5821$modified{$level}=0;
5822$accessed{$level}=0;
5823$mfiles{$level}=0;
5824$afiles{$level}=0;
5825}
5826print("\nDisk aging analysis for $dir:\n\n");
5827print (" mod acc size file\n") if ($verbose);
5828# Traverse desired filesystems
5829find(\&wanted,$dir);
5830print(" last num last num\n");
5831print(" Age (days) modified files accessed files\n");
5832$msize=$asize=$mtotal=$atotal=$lastlevel=0;
5833foreach $level (@levels) {
5834printf("%4d - %4d %8d Kb %5d %8d Kb %5d\
5835n",$lastlevel,$level,$modified{$level}/
58361024,$mfiles{$level},$accessed{$level}/1024,$afiles{$level});
5837$msize+=$modified{$level}/1024;
5838$asize+=$accessed{$level}/1024;
5839$mtotal+=$mfiles{$level};
5840$atotal+=$afiles{$level};
5841$lastlevel=$level+1;
5842}
5843printf(" ----------- ----- ----------- -----\n");
5844printf(" Total %8d Kb %5d %8d Kb %5d\n",$msize,$mtotal,$asize,$atotal);
5845}
5846exit;
5847sub wanted {
5848(($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size) = lstat($_));
5849$mod=int(-M _);
5850Cheap IP Takeover #63
5851Monitoring | 127
5852HACK
5853$acc=int(-A _);
5854foreach $level (@levels) {
5855if ($mod<=$level) { $modified{$level}+=$size; $mfiles{$level}++; last; }
5856}
5857foreach $level (@levels) {
5858if ($acc<=$level) { $accessed{$level}+=$size; $afiles{$level}++; last; }
5859}
5860printf("%4d %4d %6d %s\n",$mod,$acc,$size,$_) if ($verbose);
5861}
5862H A C K
5863#63
5864Cheap IP Takeover Hack #63
5865Accomplish IP takeover with ping, bash, and a simple network utility
5866Directing traffic to one of several machines is fairly straightforward when
5867using round-robin DNS, as discussed in “Distributing Server Load with
5868Round-Robin DNS” [Hack #79] . But what happens when one of those servers
5869becomes unavailable? Here’s one scheme for monitoring the health of
5870another server and standing in for it if it fails.
5871First, we need to make a distinction between the server’s “real” IP address,
5872and the IP (or IPs) that it actually serves public content from. For this exam-
5873ple, we’ll be referring to two servers, Pinky and Brain. Pinky uses the IP
5874address 208.201.239.12 for its “real” IP on eth0, and also has an IP alias of
5875208.201.239.36 on eth0:0. Brain uses 208.201.239.13 on eth0, and 208.201.
5876239.37 on eth0:0. If you’ve never used IP aliases before, here’s the very
5877quick HOWTO:
5878#ifconfig eth0:0 1.2.3.4
5879Voila, you have another IP address (1.2.3.4) bound to eth0, called eth0:0.
5880You used to have to specifically compile IP aliasing into the kernel, but this
5881option seems to have gone away in recent kernels and is apparently on by
5882default. One important thing to remember about IP aliases is that if the
5883interface to which it is bound (in this case, eth0) is ever brought down, then
5884all of its associated aliases are also brought down. You can also make the
5885alias any alphanumeric string, although some versions of ifconfig only dis-
5886play the first four or five characters of the alias when displaying interfaces.
5887Once Pinky and Brain have their respective eth0:0 set, bind a service (like
5888Apache) to their aliased IPs, and set up round-robin DNS to point to both
5889with a single hostname (see “Distributing Server Load with Round-Robin
5890DNS” [Hack #79] ). We’ll assume that we’re setting up redundant web service for
5891www.oreillynet.com, resolving to either 208.201.239.36 or 208.201.239.37.
5892Now that roughly half of the traffic is going to each server, we’ll need Pinky
5893and Brain to monitor the health of each other. This can be done by pinging
5894each other’s real IP address, and watching the results. Save the following
5895into a script, and install it on Pinky.
5896128 | Monitoring
5897#63 Cheap IP Takeover
5898HACK
5899Listing: takeover
5900#!/bin/bash
5901OTHER="brain"
5902PUBLIC="208.201.239.37"
5903PAUSE=3
5904PATH=/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/sbin
5905MISSED=0
5906while true; do
5907if ! ping -c 1 -w 1 $OTHER > /dev/null; then
5908((MISSED++))
5909else
5910if [ $MISSED -gt 2 ]; then
5911ifconfig eth0:$OTHER down
5912fi
5913MISSED=0
5914fi;
5915if [ $MISSED -eq 2 ]; then
5916ifconfig eth0:$OTHER $PUBLIC
5917#
5918# ...but see discussion below...
5919#
5920fi
5921sleep $PAUSE;
5922done
5923Naturally, set OTHER to “pinky” and PUBLIC to “208.201.239.36” on the
5924copy that runs on Brain.
5925Let’s suppose that Brain suddenly stops responding on 208.201.239.13 (say a
5926network tech accidentally pulled the wrong plug when working on the rack).
5927After missing 3 pings in a row, Pinky will leap into action, bringing up eth0:
5928brain up as 208.201.239.37, the public IP that Brain is supposed to be serving.
5929It will then continue to watch Brain’s real IP address, and relinquish control
5930when it is back online. The ping -c 1 -w 1 means “send one ping packet, and
5931time out after one second, no matter what happens.” ping will return non-zero
5932if the packet didn’t come back in the one second time limit.
5933But this isn’t quite the entire solution. Although Pinky is now answering for
5934Brain, any machines on the same network as the two servers (notably, the
5935router just upstream at your ISP) will have the wrong MAC address cached for
5936208.201.239.37. With the wrong MAC address cached, no traffic will flow to
5937Pinky, since it will only respond to packets that bear its own MAC address.
5938How can we tell all of the machines on the 208.201.239.0 network that the
5939MAC address for 208.201.239.37 has been updated?
5940Running ntop for Real-Time Network Stats #64
5941Monitoring | 129
5942HACK
5943One way is to use the send_arp utility from the High Availability Linux
5944project. This very handy (and tiny) utility will craft an ARP packet to your
5945specifications and send it to a MAC address of your choice on the local net-
5946work. If we specify all ones (i.e., ff:ff:ff:ff:ff:ff ) for the destination,
5947then it effectively becomes a broadcast ARP packet. While most routers
5948won’t update their ARP tables when they see unrequested ARP broadcasts,
5949such a packet will signal them to resend an ARP request, to which Pinky will
5950obligingly reply. The advantage of using broadcast is that it will signal all
5951machines on the subnet simultaneously, instead of having to track all of the
5952MAC addresses of machines that need updating.
5953The syntax of send_arp is send_arp [Source IP] [Source MAC] [Target IP]
5954[Target MAC] . For example, our simple monitoring script above should run
5955the following when it detects that Brain is down:
5956send_arp 208.201.239.37 00:11:22:aa:bb:cc 208.201.239.37 fffffffffff
5957(Where 00:11:22:aa:bb:cc is the hardware MAC address of Pinky’s eth0.)
5958The script can continue to watch to watch when Brain’s real IP address
5959(208.201.239.13) becomes available. When it does, we can bring eth0:brain
5960back down and let Brain worry about updating the ARP cache again (which
5961it should be set to do on boot).
5962There are a number of improvements that could be made to this technique.
5963For one thing, just because 208.201.239.13 is up doesn’t guarantee that 208.
5964201.239.37 is also available. Also, ping isn’t the best test for service avail-
5965ability (a better test might be to actually request a web page from the other
5966machine and make sure that it has a closing </html> tag).
5967These improvements are left as an exercise to you, dear reader. Every site is dif-
5968ferent, so you’ll need to find the technique that works best with the tools that
5969you have at hand. After all, that’s exactly what a hack is, isn’t it?
5970See also:
5971• The Fake package from High Availability Linux, http://www.linux-ha.org/
5972failover/
5973• “Distributing Server Load with Round-Robin DNS” [Hack #79]
5974H A C K
5975#64
5976Running ntop for Real-Time Network Stats Hack #64
5977See who’s doing what on your network over time with ntop
5978If you’re looking for real-time network statistics, you should check out the
5979terrific ntop tool. It is a full-featured protocol analyzer with a web front-end,
5980complete with SSL and GD graphing support. ntop isn’t lightweight (requir-
5981ing more resources depending on the size of your network and the volume of
5982130 | Monitoring
5983#64 Running ntop for Real-Time Network Stats
5984HACK
5985net traffic) but can give you a very nice overview (and some critical details)
5986about who’s talking to whom on your network.
5987ntop needs to initially run as root (to throw your interfaces into promiscuous
5988mode and start capturing packets), but then releases its privileges to a user
5989that you specify. If you decide to run ntop for long periods of time, you’ll
5990probably be happiest running it on a dedicated monitoring box (with few
5991other services running on it for security and performance reasons.)
5992Here’s a quick reference on how to get ntop up and running quickly. First,
5993create an ntop user and group:
5994root@gemini:~#groupadd ntop
5995root@gemini:~#useradd -c "ntop user" -d /usr/local/etc/ntop \
5996-s /bin/true -g ntop ntop
5997Then unpack and build ntop as per the instructions in docs/BUILD-NTOP.
5998txt. We’ll assume that you have the source tree unpacked in /usr/local/src/
5999ntop-2.1.3/.
6000Create a directory for ntop in which to keep its capture database:
6001root@gemini:~#mkdir /usr/local/etc/ntop
6002(Note that it should be owned by root, and not by the ntop user.)
6003If you’d like to use SSL for https (instead of standard http), then copy the
6004default SSL key to /usr/local/etc/ntop:
6005root@gemini:#cp /usr/local/src/ntop-2.1.3/ntop/*pem /usr/local/etc/ntop
6006Note that the default SSL key will not be built with the correct hostname for
6007your server. If you’d like to make your own (and eliminate the SSL warnings in
6008your browser), check out “Generating an SSL cert and Certificate Signing
6009Request” [Hack #93] and “Creating Your Own CA” [Hack #94] for an example of
6010how to make your own key and sign it with your own CA.
6011Now we initialize the ntop databases and set an administrative password:
6012root@gemini:~#ntop -A -u ntop -P /usr/local/etc/ntop
601321/Sep/2002 20:30:23 Initializing GDBM...
601421/Sep/2002 20:30:23 Started thread (1026) for network packet analyser.
601521/Sep/2002 20:30:23 Started thread (2051) for idle hosts detection.
601621/Sep/2002 20:30:23 Started thread (3076) for DNS address resolution.
601721/Sep/2002 20:30:23 Started thread (4101) for address purge.
6018Please enter the password for the admin user:
6019Please enter the password again:
602021/Sep/2002 20:30:29 Admin user password has been set.
6021Finally, run ntop as a daemon, and start the SSL server on your favorite port
6022(4242, for example):
6023root@gemini:~#ntop -u ntop -P /usr/local/etc/ntop -W4242 -d
6024D ownload from Wow! eBook <www.wowebook.com>
6025Running ntop for Real-Time Network Stats #64
6026Monitoring | 131
6027HACK
6028By default, ntop will also run a standard http server on port 3000. You should
6029strongly consider locking down access to these ports at your firewall, or by
6030using command line iptables rules, as in “Creating a Firewall from the Com-
6031mand Line of any Server” [Hack #45] . Let ntop run for a while, then connect to
6032https://your.server.here:4242/. You can find out all sorts of details about what
6033traffic has been seen on your network, as in Figure 5-1.
6034And even look at various statistics graphically, as an easy-to-read (and
6035pretty-to-print) pie chart, as in Figure 5-2.
6036While tools like tcpdump and ethereal will give you detailed, interactive analysis
6037of network traffic, ntop delivers a wealth of statistical information in a slick and
6038easy-to-use web interface. When properly installed and locked down, it may
6039well become a favorite tool in your network analysis tool chest.
6040See also:
6041• http://www.ntop.org/
6042• http://www-serra.unipi.it/~ntop/ntop.html
6043Figure 5-1. ntop will generate real-time statistical graphs of your network traffic
6044132 | Monitoring
6045#65 Monitoring Web Traffic in Real Time with httptop
6046HACK
6047H A C K
6048#65
6049Monitoring Web Traffic in Real Time with
6050httptop Hack #65
6051See who’s hitting your web server the hardest up to the second with httptop
6052Advanced http log analysis packages like analog can give you very
6053detailed report on who has been looking at what on your site. Unfortu-
6054nately, log processing jobs can take quite a long time to run on a busy
6055site, giving you detailed retrospective reporting but nothing about what’s
6056happening right now.
6057Another approach is to log all relevant web activity to a database and per-
6058form queries on it for real-time statistics. This method can give you instanta-
6059neous, to-the-hit reporting, as long as you have a database and web server
6060powerhouse capable of supporting the additional load. On a very busy site,
6061this simply isn’t practical.
6062Somewhere in between lies httptop, a Perl script that attempts to give top-like
6063reporting in real time, without the overhead of a database server. It expects
6064to be given the path to an access_log file. Run the command like this:
6065$httptop -f combined /usr/local/apache/logs/access_log
6066Figure 5-2. When run overtime, ntop will reveal great detail about how individual clients
6067use your network
6068Monitoring Web Traffic in Real Time with httptop #65
6069Monitoring | 133
6070HACK
6071You’ll be presented with a screen that refreshes every couple of seconds,
6072sorted on hits per second. Just as with top, hit ? for a listing of available
6073keys. They include such niceties as sorting by most recent hits or total hits,
6074and optionally displaying the referring URL or domain, the requested URI,
6075or remote requesting host.
6076If you have a bunch of VirtualHosts and would like to be able to simulta-
6077neously run httptop across all of them, try this:
6078Add a line like this to each of your VirtualHost entries:
6079CustomLog /usr/local/apache/logs/combined-log vhost
6080and the following log definition (which should appear all on one line) some-
6081where in your global configuration:
6082LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\""
6083vhost
6084Now you can watch activity on all of your hosts simultaneously with a com-
6085mand, as:
6086$httptop -f vhost /usr/local/apache/logs/combined-log
6087Note that httptop requires the Time::HiRes and File::Tail Perl modules to be
6088installed first. If you don’t feel like typing all of this in, httptop (and most of
6089the examples in this book) are available from the examples.oreilly.com web
6090site. See the introduction for the exact URL.
6091Listing: httptop
6092#!/usr/bin/perl -w
6093#
6094=head1 NAME
6095httptop - display top(1)-like per-client HTTP access stats
6096=head1 SYNOPSIS
6097httptop [-f <format>] [-r <refresh_secs>] [-b <backtrack_lines>] <logdir |
6098path_to_log>
6099=cut
6100use Time::HiRes qw( time );
6101use File::Tail ();
6102use Term::ReadKey;
6103use Getopt::Std;
6104use strict;
6105134 | Monitoring
6106#65 Monitoring Web Traffic in Real Time with httptop
6107HACK
6108### Defaults you might be interested in adjusting.
6109my $Update = 2; # update every n secs
6110my $Backtrack = 250; # backtrack n lines on startup
6111my @Paths = qw(
6112%
6113/title/%/logs/access_log
6114/var/log/httpd/%/access_log
6115/usr/local/apache/logs/%/access_log
6116);
6117my $Log_Format = "combined";
6118my %Log_Fields = (
6119combined => [qw/ Host x x Time URI Response x Referer Client /],
6120vhost => [qw/ VHost Host x x Time URI Response x Referer Client /]
6121);
6122### Constants & other thingies. Nothing to see here. Move along.
6123my $Version = "0.4.1";
6124sub by_hits_per () { $b->{Rate} <=> $a->{Rate} }
6125sub by_total () { $b->{Total} <=> $a->{Total} }
6126sub by_age () { $a->{Last} <=> $b->{Last} }
6127my $last_field = "Client";
6128my $index = "Host";
6129my $show_help = 0;
6130my $order = \&by_hits_per;
6131my $Help = "htlwufd?q";
6132my %Keys = (
6133h => [ "Order by hits/second" => sub { $order = \&by_hits_per } ],
6134t => [ "Order by total recorded hits" => sub { $order = \&by_total } ],
6135l => [ "Order by most recent hits" => sub { $order = \&by_age } ],
6136w => [ "Show remote host" => sub { $index = "Host" } ],
6137u => [ "Show requested URI" => sub { $index = "URI" } ],
6138f => [ "Show referring URL" => sub { $index = "Referer" } ],
6139d => [ "Show referring domain" => sub { $index = "Domain" } ],
6140'?' => [ "Help (this thing here)" => sub { $show_help++ } ],
6141q => [ "Quit" => sub { exit } ]
6142);
6143my @Display_Fields = qw/ Host Date URI Response Client Referer Domain /;
6144my @Record_Fields = qw/ Host URI Referer Domain /;
6145my $Max_Index_Width = 50;
6146my $Initial_TTL = 50;
6147my @Months = qw/ Jan Feb Mar Apr May Jun Jul Aug Sep Nov Dec /;
6148my %Term = (
6149HOME => "\033[H",
6150CLS => "\033[2J",
6151START_TITLE => "\033]0;", # for xterms etc.
6152END_TITLE => "\007",
6153Monitoring Web Traffic in Real Time with httptop #65
6154Monitoring | 135
6155HACK
6156START_RV => "\033[7m",
6157END_RV => "\033[m"
6158);
6159my ( %hist, %opt, $spec );
6160$SIG{INT} = sub { exit };
6161END { ReadMode 0 };
6162### Subs.
6163sub refresh_output
6164{
6165my ( $cols, $rows ) = GetTerminalSize;
6166my $show = $rows - 3;
6167my $count = $show;
6168my $now = (shift || time);
6169for my $type ( values %hist ) {
6170for my $peer ( values %$type ) {
6171# if ( --$peer->{_Ttl} > 0 ) {
6172my $delta = $now - $peer->{Start};
6173if ( $delta >= 1 ) {
6174$peer->{ Rate } = $peer->{ Total } / $delta;
6175} else {
6176$peer->{ Rate } = 0
6177}
6178$peer->{ Last } = int( $now - $peer->{ Date } );
6179# } else {
6180# delete $type->{$peer}
6181# }
6182}
6183}
6184$count = scalar( values %{$hist{$index}} ) - 1 if $show >= scalar values
6185%{$hist{$index}};
6186my @list = ( sort $order values %{$hist{$index}} )[ 0 .. $count ];
6187my $first = 0;
6188$first = ( $first <= $_ ? $_ + 1 : $first ) for map { $_ ? length($_->
6189{$index}) : 0 } @list;
6190$first = $Max_Index_Width if $Max_Index_Width < $first;
6191print $Term{START_TITLE}, "Monitoring $spec at: ", scalar localtime,
6192$Term{END_TITLE} if $ENV{TERM} eq "xterm"; # UGLY!!!
6193my $help = "Help/?";
6194my $head = sprintf( "%-${first}s %6s %4s %4s %s (%d total)",
6195$index, qw{ Hits/s Tot Last }, $last_field,
6196scalar keys %{$hist{$index}}
6197);
6198136 | Monitoring
6199#65 Monitoring Web Traffic in Real Time with httptop
6200HACK
6201#
6202# Truncate status line if need be
6203#
6204$head = substr($head, 0, ($cols - length($help)));
6205print @Term{"HOME", "START_RV"}, $head, " " x ($cols - length($head) -
6206length($help)), $help, $Term{END_RV}, "\n";
6207for ( @list ) {
6208# $_->{_Ttl}++;
6209my $line = sprintf( "%-${first}s %6.3f %4d %3d %s",
6210substr( $_->{$index}, 0, $Max_Index_Width ), @$_{(qw{ Rate Total Last },
6211$last_field)} );
6212if ( length($line) > $cols ) {
6213substr( $line, $cols - 1 ) = "";
6214} else {
6215$line .= " " x ($cols - length($line));
6216}
6217print $line, "\n";
6218}
6219print " " x $cols, "\n" while $count++ < $show;
6220}
6221sub process_line
6222{
6223my $line = shift;
6224my $now = ( shift || time );
6225my %hit;
6226chomp $line;
6227@hit{@{$Log_Fields{$Log_Format}}} = grep( $_, split( /"([^"]+)"|\[([^]]+)\
6228]|\s/o, $line ) );
6229$hit{ URI } =~ s/HTTP\/1\S+//gos;
6230$hit{ Referer } = "<unknown>" if not $hit{Referer} or $hit{Referer} eq "-";
6231( $hit{Domain} = $hit{Referer} ) =~ s#^\w+://([^/]+).*$#$1#os;
6232$hit{ Client } ||= "<none>";
6233$hit{ Client } =~ s/Mozilla\/[\w.]+ \(compatible; /(/gos;
6234$hit{ Client } =~ s/[^\x20-\x7f]//gos;
6235# if $now is negative, try to guess how old the hit is based on the time
6236stamp.
6237if ( $now < 0 ) {
6238my @hit_t = ( split( m![:/\s]!o, $hit{ Time } ))[ 0 .. 5 ];
6239my @now_t = ( localtime )[ 3, 4, 5, 2, 1, 0 ];
6240my @mag = ( 3600, 60, 1 );
6241# If the hit didn't parse right, or didn't happen today, the hell with it.
6242return unless $hit_t[2] == ( $now_t[2] + 1900 )
6243and $hit_t[1] eq $Months[ $now_t[1] ]
6244and $hit_t[0] == $now_t[0];
6245Monitoring Web Traffic in Real Time with httptop #65
6246Monitoring | 137
6247HACK
6248splice( @hit_t, 0, 3 );
6249splice( @now_t, 0, 3 );
6250# Work backward to the UNIX time of the hit.
6251$now = time;
6252$now -= (shift( @now_t ) - shift( @hit_t )) * $_ for ( 3600, 60, 1 );
6253}
6254$hit{ Date } = $now;
6255for my $field ( @Record_Fields ) {
6256my $peer = ( $hist{$field}{$hit{$field}} ||= { Start => $now, _Ttl =>
6257$Initial_TTL } );
6258@$peer{ @Display_Fields } = @hit{ @Display_Fields };
6259$peer->{ Total }++;
6260}
6261}
6262sub display_help {
6263my $msg = "httptop v.$Version";
6264print @Term{qw/ HOME CLS START_RV /}, $msg, $Term{END_RV}, "\n\n";
6265print " " x 4, $_, " " x 8, $Keys{$_}[0], "\n" for ( split "", $Help );
6266print "\nPress any key to continue.\n";
6267}
6268### Init.
6269getopt( 'frb' => \%opt );
6270$Backtrack = $opt{b} if $opt{b};
6271$Update = $opt{r} if $opt{r};
6272$Log_Format = $opt{f} if $opt{f};
6273$spec = $ARGV[0];
6274die <<End unless $spec and $Log_Fields{$Log_Format};
6275Usage: $0 [-f <format>] [-r<refresh_secs>] [-b <backtrack_lines>] <logdir |
6276path_to_log>
6277Valid formats are: @{[ join ", ", keys %Log_Fields ]}.
6278End
6279for ( @Paths ) {
6280last if -r $spec;
6281( $spec = $_ ) =~ s/%/$ARGV[0]/gos;
6282}
6283die "No access_log $ARGV[0] found.\n" unless -r $spec;
6284my $file = File::Tail->new(
6285name => $spec,
6286interval => $Update / 2,
6287maxinterval => $Update,
6288tail => $Backtrack,
6289nowait => 1
6290) or die "$spec: $!";
6291138 | Monitoring
6292#65 Monitoring Web Traffic in Real Time with httptop
6293HACK
6294my $last_update = time;
6295my ( $line, $now );
6296# Backtracking.
6297while ( $Backtrack-- > 0 ) {
6298last unless $line = $file->read;
6299process_line( $line, -1 );
6300}
6301$file->nowait( 0 );
6302ReadMode 4; # Echo off.
6303print @Term{"HOME", "CLS"}; # Home & clear.
6304refresh_output;
6305### Main loop.
6306while (defined( $line = $file->read )) {
6307$now = time;
6308process_line( $line, $now );
6309while ( $line = lc ReadKey(-1) ) {
6310$show_help = 0 if $show_help;
6311$Keys{$line}[1]->() if $Keys{$line};
6312}
6313if ( $show_help == 1 ) {
6314display_help;
6315$show_help++; # Don't display help again.
6316} elsif ( $now - $last_update > $Update and not $show_help ) {
6317$last_update = $now;
6318refresh_output( $now );
6319}
6320}
6321__END__
6322=head1 DESCRIPTION
6323httptop is intended to be a top(1)-equivalent for httpd access activity.
6324httptop should be invoked with the path to an Apache access_log, or
6325alternately a string that uniquely identifies the directory in which to find
6326the access_log. The search paths can be configured in the source.
6327httptop has limited flexibility for dealing with logs of different formats.
6328Run 'httptop' without options to see which format names are available.
6329While httptop is running, you can obtain a list of terminal commands by
6330pressing '?'. As with top(1), pressing 'q' quits.
6331139
6332Chapter 6
6333C H A P T E R S I X
6334SSH
6335Hacks #66–71
6336Of all the types of hacks I encountered when preparing this book, it became
6337obvious early on that ssh would require its own chapter. The ssh tool pro-
6338vides a very flexible and cryptographically secure method for connecting
6339data streams together between machines over a network. Since the com-
6340mand line on a Linux system essentially consists of reading data from (and
6341writing data to) files and pipelines, ssh makes it possible to sling data around
6342a network more or less as if everything were taking place on a single
6343machine. It does this in a fast, safe, and intuitive way, and makes for some
6344very interesting (and powerful) hacks.
6345There are a couple of versions of ssh available for Linux. We’ll assume that
6346you’re using OpenSSH v3.4p1 or later in the examples for this section.
6347OpenSSH ships with all major Linux distributions, and is available at http://
6348www.openssh.com/.
6349H A C K
6350#66
6351Quick Logins with ssh Client Keys Hack #66
6352Using ssh keys instead of password authentication to speed up and automate
6353logins
6354When you’re an admin on more than a few machines, being able to navi-
6355gate quickly to a shell on any given server is critical. Having to type “ssh my.
6356server.com” (followed by a password) is not only tedious, but it breaks one’s
6357concentration. Suddenly having to shift from “where’s the problem?” to
6358“getting there” and back to “what’s all this, then?” has led more than one
6359admin to premature senility. It promotes the digital equivalent of “why did I
6360come into this room, anyway?” (In addition, the problem is only made
6361worse by /usr/games/fortune!)
6362At any rate, more effort spent logging into a machine means less effort spent
6363solving problems. Recent versions of ssh offer a secure alternative to end-
6364lessly entering a password: public key exchange.
6365140 | SSH
6366#66 Quick Logins with ssh Client Keys
6367HACK
6368To use public keys with an ssh server, you’ll first need to generate a public/
6369private key pair:
6370$ssh-keygen -t rsa
6371You can also use -t dsa for DSA keys, or -t rsa1 if you’re using Protocol v1.
6372(And shame on you if you are! Upgrade to v2 as soon as you can!)
6373After you enter the above command, you should see something like this:
6374Generating public/private rsa key pair.
6375Enter file in which to save the key (/home/rob/.ssh/id_rsa):
6376Just hit Enter there. It will then ask you for a pass phrase; just hit enter twice
6377(but read the Security note below). Here’s what the results should look like:
6378Enter passphrase (empty for no passphrase):
6379Enter same passphrase again:
6380Your identification has been saved in /home/rob/.ssh/id_rsa.
6381Your public key has been saved in /home/rob/.ssh/id_rsa.pub.
6382The key fingerprint is:
6383a6:5c:c3:eb:18:94:0b:06:a1:a6:29:58:fa:80:0a:bc rob@localhost
6384This created two files, ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub. To use this key-
6385pair on a server, try this:
6386$ssh server "mkdir .ssh; chmod 0700 .ssh"
6387$scp .ssh/id_rsa.pub server:.ssh/authorized_keys2
6388Of course, substitute your server name for server . It should ask for your
6389password both times. Now, simply ssh server and it should log you in
6390automagically without a password. And yes, it will use your shiny new pub-
6391lic key for scp, too.
6392If that didn’t work for you, check your file permissions on both ~/.ssh/* and
6393server:~/.ssh/*. Your private key ( id_rsa ) should be 0600 (and only be present
6394on your local machine), and everything else should be 0655 or better.
6395Terrific. So you can now ssh server quickly and with a minimum of fuss. Is it
6396possible to make it even quicker to connect to machines you frequently
6397touch? You bet, just check out “‘Turbo-mode’ ssh Logins” (#67) .
6398Security Concerns
6399Some consider the use of public keys a potential security risk. After all, one
6400only has to steal a copy of your private key to obtain access to your servers.
6401While this is true, the same is certainly true of passwords.
6402Ask yourself, how many times a day do you enter a password to gain shell
6403access to a machine (or scp a file)? How frequently is it the same password
6404on many (or all) of those machines? Have you ever used that password in a
6405way that might be questionable (on a web site, on a personal machine that
6406Turbo-mode ssh Logins #67
6407SSH | 141
6408HACK
6409isn’t quite up to date, or possibly with an ssh client on a machine that you
6410don’t directly control). If any of these possibilities sound familiar, then con-
6411sider that an ssh key in the same setting would make it virtually impossible
6412for an attacker to later gain unauthorized access (providing, of course, that
6413you keep your private key safe).
6414See Also:
6415• SSH: The Definitive Guide (O’Reilly)
6416• “‘Turbo-mode’ ssh Logins” (#67)
6417• “Running the ssh-Agent in a GUI” [Hack #69]
6418H A C K
6419#67
6420Turbo-mode ssh Logins Hack #67
6421Even faster logins from the command line
6422If you’ve just come from the previous hack, you’ve only seen half of the solu-
6423tion! Even with client keys, you still have to needlessly type ssh server every
6424time you want to ssh in. Back in the dark, insecure, unenlightened days of
6425rsh, there was an obscure feature that I happened to love that hasn’t (yet)
6426been ported to ssh. It used to be possible to symlink /usr/bin/rsh to a file of
6427the same name as your server, and rsh was smart enough to realize that if it
6428wasn’t called as rsh, that it should rsh to whatever name it was called as.
6429Of course, this is trivial to implement in shell. Create a file called ssh-to with
6430these two lines in it:
6431#!/bin/sh
6432ssh `basename $0` $*
6433(Those are backticks around basename $0 .) Now put that in your PATH (if
6434~/bin doesn’t exist or isn’t in your PATH already, it should be) and set up
6435symlinks to all of your favorite servers to it:
6436$cd bin
6437$ln -s ssh-to server1
6438$ln -s ssh-to server2
6439$ln -s ssh-to server3
6440Now, to ssh to server1 (assuming you’ve copied your public key over as
6441described previously) you can simply type “server1” and you’ll magically
6442end up with a shell on server1, without typing “ssh,” and without entering
6443your password. That $* at the end allows you to run arbitrary commands in
6444a single line (instead of spawning a shell), like this:
6445server1 uptime
6446D ownload from Wow! eBook <www.wowebook.com>
6447142 | SSH
6448#68 Using ssh-Agent Effectively
6449HACK
6450(This will simply show the uptime, number of users, and load average on
6451server1, then exit. Wrap it in a for loop and iterate over a list of servers to
6452get a pinpoint status of all of your machines.)
6453I believe that this is the quickest way of using ssh, short of setting up single
6454character aliases to do it for you (which is excessively hackish, unmaintain-
6455able, and unnecessary, although it does seem to impress some people):
6456$alias a='ssh alice'
6457$alias b='ssh bob'
6458$alias e='ssh eve'
6459...
6460At any rate, ssh has many options that make it a very flexible tool for securely
6461navigating between any number of servers. Now if we can just find a way to
6462make it log in and actually fix the machine for you, we’d have a real hack.
6463See also:
6464• “Quick Logins with ssh Client Keys” [Hack #66]
6465• “Using ssh-Agent Effectively” [Hack #68]
6466H A C K
6467#68
6468Using ssh-Agent Effectively Hack #68
6469Use ssh-agent to automatically manage your ssh client keys
6470The ssh-agent is a very handy component of ssh that manages your private
6471keys for you, passing your credentials along whenever they are required.
6472The ssh-agent manpage tells us the following:
6473ssh-agent is a program to hold private keys used for public key authentica-
6474tion (RSA, DSA). The idea is that ssh-agent is started in the beginning of an
6475X-session or a login session, and all other windows or programs are started
6476as clients to the ssh-agent program. Through use of environment variables
6477the agent can be located and automatically used for authentication when log-
6478ging in to other machines using ssh(1).
6479Practically, this means that with an agent running (and with properly config-
6480ured ssh clients), it is possible to ssh to multiple machines without requiring
6481a copy of your private key on each intervening machine (or typing in your
6482password on every connection).
6483Assume we already have an authorized ssh key (see “Quick Logins with ssh
6484Client Keys” [Hack #66] ) installed on each homer, bart, and lisa. If you ssh to
6485each machine from your local machine, there’s no problem:
6486rob@caligula:~$ssh homer
6487rob@homer:~$exit
6488logout
6489Using ssh-Agent Effectively #68
6490SSH | 143
6491HACK
6492Connection to homer.oreillynet.com closed.
6493rob@caligula:~$ssh bart
6494rob@bart:~$exit
6495logout
6496Connection to bart.oreillynet.com closed.
6497rob@caligula:~$ssh lisa
6498rob@lisa:~$exit
6499But what happens when we try to ssh from homer directly to bart?
6500rob@caligula:~$ssh homer
6501rob@homer:~$ssh bart
6502rob@bart's password:
6503This is where ssh-agent comes in handy. Rather than expose your private key
6504to unnecessary risk by placing a copy on all of your servers, simply start the
6505agent on your local machine like this:
6506rob@caligula:~$eval `ssh-agent`
6507Agent pid 8450
6508Then add your default ssh keys with the ssh-add command:
6509rob@caligula:~$ssh-add
6510Identity added: /home/rob/.ssh/id_rsa (/home/rob/.ssh/id_rsa)
6511Identity added: /home/rob/.ssh/id_dsa (/home/rob/.ssh/id_dsa)
6512Identity added: /home/rob/.ssh/identity (rob@caligula)
6513You’ll also need to check that homer, bart, and lisa are configured to for-
6514ward agent requests along. This is usually denied by default, but is con-
6515trolled with a line like this:
6516ForwardAgent yes
6517in your ~/.ssh/config or /usr/local/etc/ssh_config file. You can also specify it
6518from the command line with the -A switch.
6519Now, when you ssh from homer directly to bart, homer will first ask your
6520agent for any available credentials. Likewise, sshing from bart to lisa will first
6521cause bart to check with homer, who will forward the request back to your
6522agent. This makes it easy to skate from machine to machine very quickly:
6523rob@caligula:~$ssh homer
6524rob@homer:~$ssh bart
6525rob@bart:~$ssh lisa
6526rob@lisa:~$
6527Congratulations. You now have very simple network navigation, without
6528the worry of giving away your private ssh keys. Even copying files between
6529machines (with scp) is faster and easier than before.
6530But what happens if you have no opportunity to start an agent in the first
6531place? (This is frequently the case if your login is a graphical XDM prompt,
6532or if you’re running OS X.) Fear not. At least, not before reading “Running
6533the ssh-Agent in a GUI” [Hack #69] .
6534144 | SSH
6535#69 Running the ssh-Agent in a GUI
6536HACK
6537Surely, there is no way to make it even faster and easier to ssh to a machine,
6538without making it less secure. Or is there? For the answer, check out
6539“‘Turbo-mode’ ssh Logins” (#67) .
6540H A C K
6541#69
6542Running the ssh-Agent in a GUI Hack #69
6543Running ssh-agent in windowing environments
6544This ssh-agent method described in “Using ssh-Agent Effectively” [Hack #68]
6545works fine as long as that initial eval ssh-agent can be run before your win-
6546dow environment starts. One place to do this is at login, in your ~/.bash_
6547login (or ~/.login if running tcsh). But this can be a less than optimal solu-
6548tion if you’re using pass phrases on your keys (which you probably should
6549be). Your window system won’t start until you enter your key passwords,
6550after logging in. Another side effect to this approach is that if your ssh-agent
6551ever gets killed, you’ll have to quit X, start a new agent, and log in again.
6552And in some environments (such as OS X) there is never even an opportu-
6553nity to run commands before the graphical environment starts.
6554Rather than wastefully running an agent for every window you have open
6555(or copy-and-pasting the environment settings between them), try this code
6556in your ~/.profile:
6557if [ -f ~/.agent.env ]; then
6558. ~/.agent.env > /dev/null
6559if ! kill -0 $SSH_AGENT_PID > /dev/null 2>&1; then
6560echo "Stale agent file found. Spawning new agent..."
6561eval `ssh-agent | tee ~/.agent.env`
6562ssh-add
6563fi
6564else
6565echo "Starting ssh-agent..."
6566eval `ssh-agent | tee ~/.agent.env`
6567ssh-add
6568fi
6569This will maintain a ~/.agent.env file for you, with an environment pointing
6570at your currently running ssh-agent. If the agent dies, opening a new termi-
6571nal window will spawn one automatically (and add your keys for you),
6572which is shared between all subsequent terminal windows.
6573Ahh, that’s better. A single, respawning ssh-agent on demand.
6574See Also:
6575• “Quick Logins with ssh Client Keys” [Hack #66]
6576• “‘Turbo-mode’ ssh Logins” (#67)
6577X over ssh #70
6578SSH | 145
6579HACK
6580H A C K
6581#70
6582X over ssh Hack #70
6583Run remote X11 applications easily and securely with ssh
6584Surprisingly few people realize that ssh is perfectly capable of forwarding
6585X11 traffic. If X11 forwarding is permitted by the ssh server that you’re log-
6586ging into, starting X applications is as simple as:
6587rob@florian:~$ssh -X catlin
6588Last login: Thu Sep 5 22:59:25 2002 from florian.rob.nocat
6589Linux 2.4.18.
6590rob@catlin:~$xeyes &
6591[1] 12478
6592rob@catlin:~$
6593As long as you’re running X on your local machine, this will display xeyes on
6594your desktop. This xeyes is actually running on catlin, the machine we’re cur-
6595rently logged into. All X11 traffic is being encrypted and sent down the ssh con-
6596nection that we’re logged in under and is displayed locally.
6597The real work is done by ssh, which sets up a local X11 proxy server for you:
6598rob@catlin:~$echo $DISPLAY
6599catlin:10.0
6600X11 forwarding is normally disabled by default in openssh. To enable it, add
6601the following line to your sshd_config, and restart sshd:
6602X11Forwarding yes
6603While xeyes isn’t the most useful example of why you would want to do
6604this, here are a couple you might find more interesting:
6605ethereal
6606Does packet capturing and visual analysis on a server at your co/lo
6607vnc
6608Takes command of remote X desktops from your local terminal, as if
6609you were sitting at the console
6610gkrellm
6611Shows a nice graphical system status for your server (or even several at a
6612time)
6613To make X11 traffic forward over ssh automatically, try setting this in your
6614~/.ssh/config:
6615ForwardX11 yes
6616With this option in effect, you won’t need the -X switch any more. Combine
6617this with the ssh-to script (as described in “‘Turbo-mode’ ssh Logins” (#67) ),
6618and you have a very quick way of firing off secure, remote X11 jobs:
6619rob@florian:~$catlin ethereal &
6620146 | SSH
6621#71 Forwarding Ports over ssh
6622HACK
6623This will instantly give you an ethereal session running on catlin, encrypted
6624over ssh. Hanging up on the ssh connection (say, with ~.) will kill all run-
6625ning X applications instantly. But if you need to pause them temporarily,
6626suspending your ssh session (with ~^Z) and resuming later with fg works
6627just fine.
6628See also:
6629• Ethereal packet sniffer: http://www.ethereal.com/
6630• VNC (a free X remote desktop client): http://www.uk.research.att.com/vnc/
6631• gkrellm system monitor: http://www.gkrellm.net/
6632H A C K
6633#71
6634Forwarding Ports over ssh Hack #71
6635Keep network traffic to arbitrary ports secure with ssh port forwarding
6636In addition to providing remote shell access and command execution,
6637OpenSSH can forward arbitrary TCP ports to the other end of your connec-
6638tion. This can be very handy for protecting email, web, or any other traffic you
6639need to keep private (at least, all the way to the other end of the tunnel).
6640ssh accomplishes local forwarding by binding to a local port, performing
6641encryption, sending the encrypted data to the remote end of the ssh connec-
6642tion, then decrypting it and sending it to the remote host and port you spec-
6643ify. Start an ssh tunnel with the -L switch (short for Local):
6644root@laptop:~#ssh -f -N -L110:mailhost:110 -lusermailhost
6645Naturally, substitute user with your username, and mailhost with your
6646mail server’s name or IP address. Note that you will have to be root on lap-
6647top for this example, since you’ll be binding to a privileged port (110, the
6648POP port). You should also disable any locally running POP daemon (look
6649in /etc/inetd.conf) or it will get in the way.
6650Now to encrypt all of your POP traffic, configure your mail client to con-
6651nect to localhost port 110. It will happily talk to mailhost as if it were con-
6652nected directly, except that the entire conversation will be encrypted.
6653The -f forks ssh into the background, and -N tells it not to actually run a
6654command on the remote end (just do the forwarding). If your ssh server sup-
6655ports it, try the -C switch to turn on compression—this can significantly
6656improve the time it takes to download your email.
6657You can specify as many -L lines as you like when establishing the connec-
6658tion. To also forward outbound email traffic, try this:
6659root@laptop:~#ssh -f -N -L110:mailhost:110 -L25:mailhost:25 \
6660-lusermailhost
6661Forwarding Ports over ssh #71
6662SSH | 147
6663HACK
6664Set your outbound email host to localhost, and your email traffic will be
6665encrypted as far as mailhost. This generally is only useful if the email is
6666bound for an internal host, or if you can’t trust your local network connec-
6667tion (as is the case with most wireless networks). Obviously, once your
6668email leaves mailhost, it will be transmitted in the clear, unless you’ve
6669encrypted the message with a tool such as pgp or gpg.
6670If you’re already logged into a remote host and need to forward a port
6671quickly, try this:
6672• Hit Enter
6673• Type ~C
6674• You should be at an ssh> prompt; enter the -L line as you would from
6675the command line.
6676For example:
6677rob@catlin:~$
6678rob@catlin:~$~, thenC (it doesn't echo)
6679ssh>-L8080:localhost:80
6680Forwarding port.
6681Your current shell will then forward local port 8000 to catlin’s port 80, as if
6682you had entered it in the first place.
6683You can also allow other (remote) clients to connect to your forwarded port,
6684with the -g switch. If you’re logged in to a remote gateway that serves as a
6685NAT for a private network, then a command like this:
6686rob@gateway:~$ssh -f -g -N -L8000:localhost:80 10.42.4.6
6687will forward all connections from gateway’s port 8000 to internal host
668810.42.4.6’s port 80. If the gateway has a live Internet address, this will
6689allow anyone from the Net to connect to the web server on 10.42.4.6 as if
6690it were running on port 8000 of the gateway.
6691One last point worth mentioning: the forwarded host doesn’t have to be
6692localhost; it can be any host that the machine you’re connecting to can
6693access directly. For example, to forward local port 5150 to a web server
6694somewhere on an internal network, try this:
6695rob@remote:~$ssh -f -N -L5150:intranet.insider.nocat:80 gateway.nocat.net
6696Assuming that you’re running a TLD (“Running Your Own Top-Level
6697Domain” [Hack #80] ) of .nocat, and that gateway.nocat.net also has a connection
6698to the private .nocat network, all traffic to 5150 of remote will be obligingly
6699forwarded to intranet.insider.nocat:80. The address intranet.insider.nocat
6700148 | SSH
6701#71 Forwarding Ports over ssh
6702HACK
6703doesn’t have to resolve in DNS to remote; it isn’t looked up until the connec-
6704tion is made to gateway.nocat.net, then it’s gateway that does the lookup. To
6705securely browse that site from remote, try connecting to http://localhost:5150/.
6706Although ssh also has functionality for acting as a Socks 4 proxy (with the
6707-D switch), it just isn’t well suited for routing all network traffic to the
6708other end of a tunnel. Take a look at “Tunneling: IPIP Encapsulation” [Hack
6709#50] for a way to use vtun in conjunction with ssh to forward everything.
6710And see the documentation for the -D switch; it’s a pretty neat feature.
6711(What, did you think we’d do all of the work for you? ;)
6712ssh is an incredibly flexible tool, with much more functionality than I can
6713cover here. See the references below for more fun things you can do with ssh.
6714See also:
6715• man ssh
6716• SSH, The Secure Shell: The Definitive Guide (O’Reilly)
6717• “Tunneling: IPIP Encapsulation” [Hack #50]
6718• “Quick Logins with ssh Client Keys” [Hack #66]
6719• “Running Your Own Top-Level Domain” [Hack #80]
6720149
6721Chapter 7
6722C H A P T E R S E V E N
6723Scripting
6724Hacks #72–75
6725Sometimes, you want to do something more complicated than can be
6726expressed on a single command line. Once you’ve done something compli-
6727cated more than two or three times, you’ll probably begin to think that there
6728must be a better way to do it (which doesn’t involve a whole bunch of typ-
6729ing each time). This is usually when sysadmins begin to evolve into that
6730strange breed of problem solver known as the programmer.
6731This section is not a crash course in programming, but rather a demonstra-
6732tion of a couple of pieces of applied programming hackery. I’ve found that
6733the best way to learn how to program is to learn by doing, and the easiest
6734way to get started is to see how other people have solved similar problems.
6735The examples in this section are useful on their own but are even more use-
6736ful as a starting point in building your own custom tools. Even if you’re an
6737old hand at scripting, take a look at some of these examples for some ideas
6738on how to use some lesser known invocation switches and language fea-
6739tures to get more done with less effort.
6740H A C K
6741#72
6742Get Settled in Quickly with movein.sh Hack #72
6743Keep your local environment in sync on all of your servers
6744When you use a machine for some time, you will inevitably end up custom-
6745izing it to your liking. As we saw in “At Home in Your Shell Environment”
6746[Hack #10] , the shell environment is a tremendously flexible tool that can be
6747fine tuned to your precise specifications.
6748Usually these little tweaks take weeks (or even years) to perfect and span
6749several files: different environment settings (depending on if it’s a login shell
6750or a subshell), editor preferences, email settings, mysql preferences, aliases
6751to be set and options for every occasion, etc. When the environment is pre-
6752cisely programmed, tasks become easier and the system becomes more fun
6753to work with.
6754D ownload from Wow! eBook <www.wowebook.com>
6755150 | Scripting
6756#72 Get Settled in Quickly with movein.sh
6757HACK
6758Of course, all of this goes out the window when you log into a remote
6759machine. It can be terribly frustrating to fire off your favorite command, only
6760to see bash: xyz: command not found. Or to run ls and not see the colors. Or
6761any of a thousand little adjustments that make life on your home machine
6762such a joy, but working on remote machines such a barren, joyless existence.
6763You can always copy over your settings by hand, as most of them are kept as
6764dotfiles in your home directory (like .bashrc and .vimrc). But it’s difficult to
6765remember them all by hand, and it leads to the classic versioning problem:
6766once you make a change to your local copy, the remote copies all need to be
6767updated again.
6768Make your life simpler with this very simple shell script.
6769Listing: movein.sh
6770#!/bin/sh
6771if [ -z "$1" ]; then
6772echo "Usage: `basename $0` hostname"
6773exit
6774fi
6775cd ~/.skel
6776tar zhcf - . | ssh $1 "tar zpvxf -"
6777Call it movein.sh, and stick it in your ~/bin directory. Now create a ~/.skel
6778directory, and make symlinks to all of the files you’d like to copy to remote
6779servers:
6780rob@caligula:~/.skel$ls -al
6781total 12
6782drwxr-xr-x 6 rob staff 204 Sep 9 20:52 .
6783drwxr-xr-x 37 rob staff 1258 Sep 9 20:57 ..
6784lrwxr-xr-x 1 rob staff 11 Sep 9 20:52 .bash_login -> ../.bash_login
6785lrwxr-xr-x 1 rob staff 11 Sep 9 20:52 .bashrc -> ../.bashrc
6786lrwxr-xr-x 1 rob staff 11 Sep 9 20:52 .my.cnf -> ../.my.cnf
6787lrwxr-xr-x 1 rob staff 11 Sep 9 20:52 .pinerc -> ../.pinerc
6788drwxr-xr-x 3 rob staff 102 Sep 9 20:51 .ssh
6789lrwxr-xr-x 1 rob staff 9 Sep 9 20:52 .vimrc -> ../.vimrc
6790lrwxr-xr-x 1 rob staff 6 Sep 9 21:27 bin -> ../bin
6791Note that ~/.skel/.ssh is a special case: it’s a directory, not a symlink. DO
6792NOT SYMLINK YOUR ~/.ssh TO ~/.skel! The last thing you need is to
6793copy your private ssh key all over the place; that’s what the ssh-agent is
6794for (see “Using ssh-Agent Effectively” [Hack #68] ). Instead, make a directory
6795called ~/.skel/.ssh and make a symlink like this:
6796rob@caligula:~/.skel$cd .ssh
6797rob@caligula:~/.skel/.ssh$ls -al
6798total 4
6799Global Search and Replace with Perl #73
6800Scripting | 151
6801HACK
6802drwxr-xr-x 3 rob staff 102 Sep 9 20:51 .
6803drwxr-xr-x 6 rob staff 204 Sep 9 20:52 ..
6804lrwxr-xr-x 1 rob staff 26 Sep 9 20:51 authorized_keys2 ->
6805../../.ssh/id_dsa.pub
6806This is a link called authorized_keys2, and it points to your live public key.
6807You are using public key ssh connections, right? If not, consult “Quick Log-
6808ins with ssh Client Keys” [Hack #66] .
6809Now when this script runs, it will copy the contents of ~/.skel to the host
6810you specify on the command line, straight into your home directory. The h
6811flag to tar means “copy these symlinks as if they were files, not symlinks,” so
6812you end up with a copy of the contents of each link on the remote end. If
6813you make changes to your local copy, just run the script again, and it will
6814overwrite everything on the remote end.
6815Only include the .ssh directory as mentioned above if you’d like to be able to
6816log into the remote host automatically, without a password. As long as your
6817local machine’s keys are kept secure, there is no inherent security risk in
6818leaving extra authorized_keys2 files lying around. That’s what public key
6819cyptography is all about.
6820See also:
6821• “Quick Logins with ssh Client Keys” [Hack #66]
6822• “Using ssh-Agent Effectively” [Hack #68]
6823• “At Home in Your Shell Environment” [Hack #10]
6824• SSH: The Secure Shell, The Definitive Guide (O’Reilly)
6825H A C K
6826#73
6827Global Search and Replace with Perl Hack #73
6828Manipulate files and streams with arbitrary Perl substitutions, without a script
6829There are a couple of switches that make Perl a very useful command-line
6830editing tool. Learn these switches, and you too can learn how to mumble
6831magic Perl one-liners to confound your friends (and frighten your project
6832manager).
6833The first is -e. Give Perl a -e followed by a line of code, and it will run it as if
6834it were an ordinary Perl script:
6835rob@catlin:~$perl -e 'print "Hi, Ma!\n"'
6836Hi, Ma!
6837Note that a trailing ; isn’t needed on one-liners. It’s generally a good idea to
6838wrap your line in single quotes to prevent the shell from attempting to inter-
6839pret special characters (like the ! and \ above).
6840152 | Scripting
6841#73 Global Search and Replace with Perl
6842HACK
6843The next switch is a little more complicated, it but becomes second nature
6844once you start using it: -p. From perldoc perlrun:
6845-p causes Perl to assume the following loop around your
6846program, which makes it iterate over filename arguments
6847somewhat likesed:
6848LINE:
6849while (<>) {
6850... # your program goes here
6851} continue {
6852print or die "-p destination: $!\n";
6853}
6854The line in question ( $_ ) is automatically printed on every iteration of the
6855loop. If you combine -p and -e, you get a one-liner that iterates over every
6856file on the command line or on the text fed to it on STDIN. For example,
6857here’s a complicated cat command:
6858rob@catlin:~$perl -pe 1 /etc/hosts
6859#
6860# hosts This file describes a number of hostname-to-address
6861# mappings for the TCP/IP subsystem.
6862#
6863# For loopbacking.
6864127.0.0.1 localhost
6865The 1 is just a return code (as if you entered 1; in a Perl script, which is
6866equivalent to return 1; ). Since the lines are printed automatically, we don’t
6867really need the program we specify with -e to do anything.
6868Where it gets interesting is in providing a bit of code to manipulate the cur-
6869rent line before it gets printed. For example, suppose you wanted to append
6870the local machine name to the localhost line:
6871rob@catlin:~$perl -pe 's/localhost/localhost $ENV{HOSTNAME}/' /etc/hosts
6872#
6873# hosts This file describes a number of hostname-to-address
6874# mappings for the TCP/IP subsystem.
6875#
6876# For loopbacking.
6877127.0.0.1 localhost catlin.nocat.net
6878or maybe you’d like to manipulate your inetd settings:
6879rob@caligula:~$perl -pe 's/^(\s+)?(telnet|shell|login|exec)/# $2/' \
6880/etc/inetd.conf
6881That will print the contents of /etc/inetd.conf to STDOUT, commenting out
6882any uncommented telnet, shell, login, or exec lines along the way. Naturally,
6883we could redirect that back out to a file, but if we just want to edit a file in
6884place, there’s a better way: the -i switch.
6885Mincing Your Data into Arbitrary Chunks (in bash) #74
6886Scripting | 153
6887HACK
6888-i lets you edit files in place. So, to comment out all of the above lines in /
6889etc/inetd.conf, you might try:
6890root@catlin:~#perl -pi -e 's/^(\s+)?(telnet|shell|login|exec)/# $2/' /etc/
6891inetd.conf
6892or better yet:
6893root@catlin:~#perl -pi.orig -e 's/^(\s+)?(telnet|shell|login|exec)/# $2/' /
6894etc/inetd.conf
6895The second example will backup /etc/inetd.conf to /etc/inetd.conf.orig before
6896changing the original. Don’t forget to HUP inetd to make your changes take.
6897It’s just as easy to edit multiple files in place at the same time. You can spec-
6898ify any number of files (or wildcards) on the command line:
6899rob@catlin:~$perl -pi.bak -e 's/bgcolor=#ffffff/bgcolor=#000000/i' *.html
6900This will change the background color of all html pages in the current direc-
6901tory from white to black. Don’t forget that trailing i to make the match case
6902insensitive (to match bgcolor=#FFFFFF or even BGColor=#FfFffF).
6903What if you’re in the middle of working on a CVS project, and need to
6904change the CVS server that you’d like to commit to? It’s easy, if you pipe the
6905output of a find through an xargs running a perl -pi -e:
6906schuyler@ganesh:~/nocat$find -name Root | xargs perl -pi -e 's/cvs.
6907oldserver.com/cvs.newserver.org/g'
6908Then reset your $CVSROOT and do your CVS check in as normal, and your
6909project will automagically end up checked into cvs.newserver.org.
6910Using Perl from the command line can help you do some powerful transfor-
6911mations on the fly. Study your regular expressions, and use it wisely, and it
6912can save piles of hand edits.
6913See also:
6914• Programming Perl (O’Reilly)
6915• perldoc perlrun
6916• “At Home in Your Shell Environment” [Hack #10]
6917• “CVS: Making Changes to a Module” [Hack #30]
6918H A C K
6919#74
6920Mincing Your Data into Arbitrary Chunks (in
6921bash) Hack #74
6922Use bash arithmetic and dd to chop large binary files into reasonable chunks
6923Here’s an example of how to use environment variables and arithmetic eval-
6924uation in bash to chop any file into arbitrary sized chunks.
6925154 | Scripting
6926#74 Mincing Your Data into Arbitrary Chunks (in bash)
6927HACK
6928Listing: mince
6929#!/bin/bash
6930if [ -z "$2" -o ! -r "$1" ]; then
6931echo "Usage: mince [file] [chunk size]"
6932exit 255
6933fi
6934SIZE=`ls -l $1 | awk '{print $5}'`
6935if [ $2 -gt $SIZE ]; then
6936echo "Your chunk size must be smaller than the file size!"
6937exit 254
6938fi
6939CHUNK=$2
6940TOTAL=0
6941PASS=0
6942while [ $TOTAL -lt $SIZE ]; do
6943PASS=$((PASS + 1))
6944echo "Creating $1.$PASS..."
6945dd conv=noerror if=$1 of=$1.$PASS bs=$CHUNK skip=$((PASS - 1)) count=1 2> /
6946dev/null
6947TOTAL=$((TOTAL + CHUNK))
6948done
6949echo "Created $PASS chunks out of $1."
6950Note that we take advantage of conv=noerror, since the last chunk is almost
6951guaranteed to run beyond the end of your file. Using this option makes dd
6952blithely continue to write bits until we run out of source file, at which point
6953it exits (but doesn’t throw an error, and more importantly, doesn’t refuse to
6954write the last chunk).
6955This could be handy for slicing large files into floppy-sized (zip disk, cd-r,
6956dvd, Usenet) chunks prior to archiving. As it uses dd’s skip feature, it will
6957work on any sized file, regardless of the amount of available RAM (provided
6958that you supply a reasonable chunk size). Since the block size (bs) is set to
6959whatever you have selected as your chunk size, it runs quite quickly, espe-
6960cially with chunks larger than a couple of kilobytes.
6961It saves your chunks as multiple files (ending in a . followed by the chunk
6962number) in the current directory. Running ls FILENAME.* will show you your
6963chunks in numerical order. But what if you have more than nine of them?
6964ls FILENAME.* | sort -n -t . +2
6965How do you reassemble them?
6966cat `ls FILENAME.* | sort -n -t . +2` > FILENAME.complete
6967Colorized Log Analysis in Your Terminal #75
6968Scripting | 155
6969HACK
6970Don’t believe me?
6971diff FILENAME FILENAME.complete
6972(This of course assumes that your filename has only one . in it. If you have a
6973ridiculously.long.filename.with.multiple.dots, consult man sort(1).)
6974H A C K
6975#75
6976Colorized Log Analysis in Your Terminal Hack #75
6977View your log files in an xterm window in full, living color
6978If you find yourself slowly going cross-eyed while looking at line after line of
6979system logs, then you should consider using tools to help you organize your
6980logs. While a properly configured syslog (as shown in “Steering syslog” [Hack
6981#54] ) goes a long way toward logfile sanity, it can still be a bit overwhelming
6982to sift through a multi-megabyte /var/log/messages looking for patterns.
6983Just as a colorized ls can help identify types of files at a glance, a colorized
6984grep is a handy tool for making patterns leap out of a sea of grey lines. There
6985are a number of X applications that will do this for you, but why not make it
6986easy to view your colorized logs from the command line?
6987Save this as ~/bin/rcg (short for Regex Colored Glasses):
6988#!/usr/bin/perl -w
6989use strict;
6990use Term::ANSIColor qw(:constants);
6991my %target = ( );
6992while (my $arg = shift) {
6993my $clr = shift;
6994if(($arg =~ /^-/) | (!$clr)) {
6995print "Usage: rcg [regex] [color] [regex] [color] ...\n";
6996exit;
6997}
6998#
6999# Ugly, lazy, pathetic hack here
7000#
7001$target{$arg} = eval($clr);
7002}
7003my $rst = RESET;
7004while(<>) {
7005foreach my $x (keys(%target)) {
7006s/($x)/$target{$x}$1$rst/g;
7007}
7008print;
7009}
7010156 | Scripting
7011#75 Colorized Log Analysis in Your Terminal
7012HACK
7013rcg is a simple filter that uses Term::ANSIColor to colorize arbitrary regexs,
7014specified on the commandline. It is intended to help visually slog through
7015log files.
7016You must pass rcg an even number of command line parameters. The odd
7017terms specify the regex, the even terms specify the color.
7018Suppose you wanted anything with the word sendmail in your messages log
7019to show up magenta, instead of grey:
7020$rcg sendmail MAGENTA < /var/log/messages | less -r
7021The less -r is optional but handy (as it displays the intended colors in less,
7022instead of the ESC characters.)
7023You can use any arbitrary regex as an odd term:
7024$rcg '\d+\.\d+\.\d+\.\d+' GREEN < /var/log/maillog
7025Or chain colors together:
7026$tail -50 /var/log/messages | rcg WARNING 'BOLD . YELLOW . ON_RED'
7027You can specify any number of regex/color code pairs on a single command-
7028line. This is where teeny shell scripts or aliases would come in handy (one
7029for messages, one for firewall logs, one for Apache).
7030See the Term::ANSIColor docs for the full list of colors and combinations.
7031Some other useful strings:
7032\w+=\S+
7033Variables, such as TERM=xyz
7034\d+\.\d+\.\d+\.\d+
7035Probably an IP address
7036^(J|F|M|A|S|O|N|D)\w\w (\d|)\d
7037Might be a date
7038\b\d\d:\d\d:\d\d\b
7039Possibly the time
7040.*last message repeated.*
7041Makes this “BOLD . WHITE”
7042Use your imagination, but be warned: color params are just eval()'d. Theo-
7043retically, many valid Perl expressions can be substituted for regexes or col-
7044ors; exploiting this, uh, feature is left as an exercise to the reader. You
7045probably shouldn’t be running arbitrary rcg lines as root, unless you wrote
7046them yourself. Also note that colorization is applied in arbitrary order, so it’s
7047not possible to guarantee the behavior of overlapping regexes.
7048157
7049Chapter 8
7050C H A P T E R E I G H T
7051Information Servers
7052Hacks #76–100
7053Linux is a powerful platform for building information servers. But the infor-
7054mation systems themselves are rarely a part of Linux. Usually, Linux is sim-
7055ply a “life support system” for more complicated, dedicated information
7056services.
7057In this final section, we’ll look at three major applications. They all run quite
7058well on Linux and have become the backbone of Internet information ser-
7059vices. BIND (by the Internet Software Consortium) is by far the most com-
7060mon DNS information server on the planet, serving the Domain Name to IP
7061address information that keeps the Internet running. For more generic infor-
7062mation needs, MySQL is a very lightweight, fast, and scalable SQL database
7063that drives many enterprise network applications. Finally, when it comes to
7064serving information to users, Apache is by far the most popular web server
7065on the planet. Apache is run on more servers than all of the rest of the
7066known web servers combined and for good reason: it is mature, stable, fast,
7067and full of useful and interesting features.
7068If you’re looking for more information about running any of these pack-
7069ages, consult the online documentation for each. These applications are run-
7070ning the current incarnation of the Internet, and have been widely and
7071extensively documented. In addition, O’Reilly has a number of good books
7072on all three applications, such as DNS and Bind, MySQL Reference Manual,
7073and Apache: The Definitive Guide to name a few. MySQL by Paul DuBois
7074(New Riders) is also an excellent guide to running MySQL.
7075In the following hacks, we’ll see some non-obvious techniques for getting
7076these servers to deliver information the way you want them. We will take a
7077look at some methods for making them scale to large installations, while
7078keeping maintenance of even very complex sites to a minimum.
7079158 | Information Servers
7080#76 Running BIND in a chroot Jail
7081HACK
7082H A C K
7083#76
7084Running BIND in a chroot Jail Hack #76
7085Keep your named isolated from the rest of the system with the judicious use
7086of chroot
7087The vast majority of the Internet relies on BIND for its name resolution needs.
7088While tremendous effort has gone toward shoring up potential security holes
7089in BIND, you can never be absolutely certain that any code is completely free
7090of possible exploits. To minimize the possible damage done by remote root
7091exploits (due to buffer overflows, bugs, or misconfiguration), many sites
7092choose to run their named service in a chroot jail. This helps ensure that even
7093if the named process is compromised, the attacker’s job won’t be finished yet.
7094While a chroot jail isn’t necessarily impenetrable, it poses a very difficult chal-
7095lenge for a would-be system cracker.
7096These steps outline the minimum effort required to get BIND 9 running in
7097a chroot jail. DNS security is a large and complex issue, and should be
7098taken very seriously. Consult the resources at the end of this hack for more
7099information.
7100To begin with, we’ll want to run named as some user other than root. Cre-
7101ate a named user and group that will only be used for running the named
7102process:
7103root@gemini:~#groupadd -g 53 named
7104root@gemini:~#useradd -u 53 -g named -c "chroot BIND user" \
7105-d /var/named/jail -m named
7106We’ll instruct named to chroot to /var/named/jail at startup. We will need to
7107create enough of a skeleton file structure under this directory to allow named
7108to start normally. Create the /var structure, and copy your DNS data into it:
7109root@gemini:~#cd ~named
7110root@gemini:/var/named/jail#mkdir -p var/{run,named}
7111root@gemini:/var/named/jail#cp -Rav /var/named/data var/named/
7112If you act as a slave for any zones, then the named process will need write
7113access to a directory (to keep track of updated slave data). Create a direc-
7114tory specifically for slave data, independent of your regular data files:
7115root@gemini:/var/named/jail#mkdir var/named/slave
7116root@gemini:/var/named/jail#chown named.named var/named/slave
7117Next, create the dev/ and etc/ directories, and copy the critical system files and
7118devices to them:
7119root@gemini:/var/named/jail#mkdir {dev,etc}
7120root@gemini:/var/named/jail#cp -av /dev/{null,random} dev/
7121root@gemini:/var/named/jail#cp -av \
7122/etc/{localtime,named.conf,rndc.key} etc/
7123Running BIND in a chroot Jail #76
7124Information Servers | 159
7125HACK
7126Clean up the ownership and permissions of these directories:
7127root@gemini:/var/named/jail#chown root.root .
7128root@gemini:/var/named/jail#chmod 0755 .
7129root@gemini:/var/named/jail#chown named.named var/named/data/
7130root@gemini:/var/named/jail#chmod 0700 var/named/data/
7131root@gemini:/var/named/jail#chown named.named var/run/
7132If you’re using syslog for your DNS logs, you’ll need to add a switch like this to
7133your syslogd startup rc. It will tell syslog to listen on that socket as well (which
7134is needed since the system’s /dev/log won’t be reachable from inside the jail):
7135syslogd -m 0-a /var/named/jail/dev/log
7136If you’re using filesystem logs, you won’t need to change your syslogd con-
7137figuration. Just be sure that the log files are writable by the named user, and
7138that your log directory exists under /var/named/jail/.
7139Finally, fire up named, passing the username, chroot directory, and initial
7140configuration file on the command line:
7141root@gemini:~#/usr/sbin/named -u named -t /var/named/jail \
7142-c /etc/named.conf
7143If your named doesn’t start cleanly, take a look at the system’s /var/log/syslog
7144or /var/log/messages, and track down the source of the trouble. It can be
7145tricky to be sure that your permissions are all what they need to be. Here’s a
7146sample recursive ls output from a machine with a running chroot’d BIND:
7147root@gemini:/var/named/jail#ls -lR
7148.:
7149total 12
7150drwxr-xr-x 2 root root 4096 Sep 20 12:45 dev/
7151drwxr-xr-x 2 root root 4096 Sep 20 13:08 etc/
7152drwxr-xr-x 5 root root 4096 Sep 20 13:04 var/
7153./dev:
7154total 0
7155crw-rw-rw- 1 root root 1, 3 Jul 17 1994 null
7156crw-r--r-- 1 root root 1, 8 Dec 11 1995 random
7157./etc:
7158total 16
7159-rw-r--r-- 1 root root 1017 Sep 20 12:46 localtime
7160-r--r--r-- 1 root root 1381 Sep 20 13:08 named.conf
7161-rw------- 1 root root 77 Sep 11 04:22 rndc.key
7162./var:
7163total 12
7164drwxr-xr-x 4 root root 4096 Sep 20 13:01 named/
7165drwxr-xr-x 2 named named 4096 Sep 20 13:15 run/
7166./var/named:
7167total 8
7168160 | Information Servers
7169#77 Views in BIND 9
7170HACK
7171drwx------ 3 named named 4096 Sep 20 13:03 data/
7172drwxr-xr-x 2 named named 4096 Sep 20 12:43 slave/
7173./var/named/data:
7174total 42
7175-rw-r--r-- 1 root root 381 Apr 30 16:32 localhost.rev
7176-rw-r--r-- 1 root root 2769 Sep 20 13:03 named.ca
7177-r--r--r-- 1 root root 1412 Sep 17 16:44 nocat.net
7178./var/named/slave:
7179total 0
7180./var/run:
7181total 4
7182-rw-r--r-- 1 named named 6 Sep 20 13:15 named.pid
7183A chroot’d environment can help keep daemons sequestered away from the
7184rest of a running system, but they’re not a magic bullet for instant system
7185security. This hack should get you started, but be sure to read up on the
7186complexities of using and running BIND.
7187See also:
7188• http://www.losurs.org/docs/howto/Chroot-BIND.html
7189• DNS Bind, Fourth edition (O’Reilly)
7190H A C K
7191#77
7192Views in BIND 9 Hack #77
7193Change the results that your DNS server returns depending on where the
7194requests are coming from
7195Until recently, presenting one view of a zone to one community of hosts and
7196another view to others has entailed running multiple sets of name servers or
7197multiple name server processes on a single host in a tricky configuration.
7198Nobody ever said being two-faced is easy.
7199BIND 9 introduced a new feature called views that makes delivering differ-
7200ent versions of a zone and even different name server configurations easy.
7201Basic Syntax
7202The key to configuring views is the new BIND 9 configuration statement,
7203view. view takes a view name as an argument and has one required substate-
7204ment, match-clients. The view name is simply a convenient mnemonic for
7205identifying the view; I usually use names such as “internal” and “external”
7206or “Internet.” If you use a name that conflicts with a reserved word in
7207named.conf, be sure to quote it. I quote all of my view names, because I can’t
7208remember which words are reserved.
7209D ownload from Wow! eBook <www.wowebook.com>
7210Views in BIND 9 #77
7211Information Servers | 161
7212HACK
7213The match-clients substatement takes an address match list as an argument.
7214Only queriers whose IP addresses match this address match list will see the
7215configuration specified in the enclosing view. If a querier’s IP address
7216matches multiple view statement’s match-clients substatements, the first
7217view statement is the one that applies.
7218Here’s an example of a simple view statement:
7219view "global" {
7220match-clients { any; };
7221};
7222This view statement doesn’t do anything useful, because it applies to all que-
7223ries (in the absence of other view statements) and it doesn’t include any sub-
7224statements besides match-clients, which would change the configuration of
7225this view from the default configuration.
7226If you don’t specify a particular substatement within a view, the view inher-
7227its the global setting for that substatement. So, for example, if you don’t turn
7228recursion off within a view, that view would inherit the recursion setting
7229from your options statement (specified by the recursion substatement), or if
7230you don’t have one, would inherit the global default, which is recursion yes.
7231Here’s an example of turning recursion off within a view:
7232view "external" {
7233match-clients { any; };
7234recursion no;
7235};
7236Here’s a complete named.conf file that includes the previous view statement:
7237/*
7238* BIND 9 name server allowing recursive queries from
7239* localhost, disallowing from anywhere else
7240*/
7241options {
7242directory "/var/named";
7243};
7244view "localhost" {
7245match-clients { localhost; };
7246recursion yes; /* this is the default */
7247};
7248view "external" {
7249match-clients { any; };
7250recursion no;
7251};
7252This name server allows recursive queries only if they come from the local
7253host. (The localhost access control list [ACL] is predefined to be all of the IP
7254162 | Information Servers
7255#77 Views in BIND 9
7256HACK
7257addresses of the host that runs the name server, plus the loopback address.)
7258Queries from all other addresses are treated as non-recursive.
7259Here’s a similar configuration, this one of a name server that allows recur-
7260sive queries from our internal network but not from the Internet:
7261/*
7262* Same name server serving an internal network and the
7263* Internet
7264*/
7265options {
7266directory "/var/named";
7267};
7268view "internal" {
7269match-clients { localnets; };
7270recursion yes; /* this is the default */
7271};
7272view "external" {
7273match-clients { any; };
7274recursion no;
7275};
7276This configuration takes advantage of the built-in ACL localnets, which is pre-
7277defined as all of the networks to which our name server is directly connected.
7278Defining Zones in Views
7279In order to present a version of a zone to only some queriers, you need to
7280define the zone within a view. You simply use a zone statement as a view
7281substatement. Otherwise, its syntax is the same as if it were a top-level state-
7282ment. Note that if you define even one zone within a view, all of your zones
7283need to be defined inside views.
7284Here’s an example:
7285/*
7286* A name server showing different zone data to different
7287* networks
7288*/
7289options {
7290directory "/var/named";
7291};
7292view "internal" {
7293match-clients { localnets; };
7294recursion yes; /* this is the default */
7295zone "oreilly.com" {
7296type master;
7297Views in BIND 9 #77
7298Information Servers | 163
7299HACK
7300file "db.oreilly.com.internal";
7301allow-transfer { any; };
7302};
7303};
7304view "external" {
7305match-clients { any; };
7306recursion no;
7307zone "oreilly.com" {
7308type master;
7309file "db.oreilly.com.external";
7310allow-transfer { none; };
7311};
7312};
7313Note that the zone oreilly.com is defined in both the internal and the exter-
7314nal view, but the zone data file for oreilly.com is db.oreilly.com.internal in
7315the internal view and db.oreilly.com.external in the external view. Presum-
7316ably, the contents of the zone data files are different.
7317If you prefer to use different subdirectories for the internal and external zone
7318data, you can do that, too. For example, the file substatement for oreilly.
7319com in the internal view could be file "internal/db.oreilly.com"; and the
7320external view could use file "external/db.oreilly.com"; .
7321This way, you can keep all of your internal zone data files in the /var/named/
7322internal directory and your external zone data files in the /var/named/
7323external directory.
7324Views in Slave Name Servers
7325One minor wrinkle in views is the configuration of slave name servers. Many
7326people configure a primary master name server with multiple views, then
7327want to configure a slave with the same views. Unfortunately, when the
7328slave tries to transfer the zones (or the two versions of the same zone) from
7329the primary master name server, it only sees one version of the zone, the one
7330defined in the view the slave’s IP address can see.
7331The solution to this problem is to configure an IP address alias on the slave
7332name server, giving it two IP addresses from which to initiate zone transfers.
7333Then configure the primary master to present one view to one of the slave’s
7334IP addresses and the other view to the slave’s other IP address. Finally, force
7335the slave to initiate zone transfers from the appropriate IP address within
7336each view.
7337Here’s an example. Our slave has two IP addresses, 192.168.0.2 and 192.
7338168.0.254. Our primary master has just one, 192.168.0.1. First, here’s the
7339slave’s named.conf file:
7340164 | Information Servers
7341#77 Views in BIND 9
7342HACK
7343options {
7344directory "/var/named";
7345};
7346view "internal" {
7347match-clients { localnets; };
7348recursion yes;
7349zone "oreilly.com" {
7350type slave;
7351masters { 192.168.0.1; };
7352transfer-source 192.168.0.2;
7353file "internal/bak.oreilly.com ";
7354allow-transfer { any; };
7355};
7356};
7357view "external" {
7358match-clients { any; };
7359recursion no;
7360zone "oreilly.com" {
7361type slave;
7362masters { 192.168.0.1; };
7363transfer-source 192.168.0.254;
7364file "external/bak.oreilly.com ";
7365allow-transfer { none; };
7366};
7367};
7368Notice that the slave is configured to initiate transfers of oreilly.com within
7369the internal view from the IP address 192.168.0.2 and within the external
7370view from 192.168.0.254.
7371Now, here’s the primary master’s named.conf file:
7372options {
7373directory "/var/named";
7374};
7375view "internal" {
7376match-clients { !192.168.0.254; localnets; };
7377recursion yes;
7378zone "oreilly.com" {
7379type master;
7380file "internal/db.oreilly.com ";
7381allow-transfer { any; };
7382};
7383};
7384view "external" {
7385match-clients { any; };
7386recursion no;
7387Setting Up Caching DNS with Authority for Local Domains #78
7388Information Servers | 165
7389HACK
7390zone "oreilly.com" {
7391type master;
7392file "external/db.oreilly.com ";
7393allow-transfer { 192.168.0.254; };
7394};
7395};
7396Notice that the internal view’s match-clients substatement explicitly places
7397192.168.0.254 into the external view by negating it in the address match list.
7398Any time the slave initiates a zone transfer from that IP address, it’ll get the
7399version of oreilly.com described by the zone data file /var/named/external/db.
7400oreilly.com.
7401H A C K
7402#78
7403Setting Up Caching DNS with Authority for
7404Local Domains Hack #78
7405Get BIND running quickly with a forwarding, caching server
7406Running BIND can be very tricky business if you have a particularly com-
7407plex network topology. Multiple DMZs, public versus private IP addresses,
7408and delegated subdomains can make DNS administration a full time job for
7409a large site. If you’re looking for a way to alleviate some of the complexity,
7410see “Views in BIND 9” [Hack #77] . Or if you’re feeling particularly adventur-
7411ous, try wildcard domain matching and delegation, as described in “Ultra-
7412hosting: Mass Web Site Hosting with Wildcards, Proxy, and Rewrite” [Hack
7413#100] .
7414But in the majority of small to medium installations, BIND is really only
7415needed for two things: to act as the authoritative source for a domain or two
7416and provide forwarding to another DNS server for all other requests.
7417Here is a simple (but complete) named.con f that does exactly that:
7418options {
7419directory "/var/named";
7420pid-file "/var/run/named.pid";
7421statistics-file "/var/named/named.stats";
7422};
7423logging {
7424channel default_out {
7425file "/var/log/named.log";
7426};
7427category default { default_out; };
7428category config { default_out; };
7429category xfer-in { default_out; };
7430category xfer-out { default_out; };
7431category lame-servers { null; };
7432};
7433166 | Information Servers
7434#78 Setting Up Caching DNS with Authority for Local Domains
7435HACK
7436zone "0.0.127.in-addr.arpa" in {
7437type master;
7438file "data/localhost.rev";
7439};
7440zone "." {
7441type hint;
7442file "rootservers.cache";
7443};
7444// Authoritative domains go here
7445zone "nocat.net" {
7446type master;
7447file "data/nocat.net";
7448};
7449This makes us authoritative for the domain nocat.net, with its data stored in
7450the file /var/named/data/nocat.net. Requests made for domains other than
7451nocat.net (or for the loopback network 127.0.0.0) are automatically for-
7452warded along according to the root servers contained in rootservers.cache. A
7453suitable rootserver cache should have shipped with BIND, but if you can’t
7454find it, try something like this:
7455#dig @a.root-servers.net > rootservers.cache
7456If your network doesn’t have unrestricted access to the Internet (e.g., you
7457are behind a restrictive firewall), then forwarding to authoritative servers
7458may not work. If that’s the case, try an explicit forwarding rule in place of
7459the hint entry above:
7460zone "." {
7461type forward;
7462forward only;
7463forwarders { 192.168.1.1; };
7464}
7465Naturally, replace 192.168.1.1 with the IP address of a valid name server for
7466your network. This will direct all DNS traffic to your network’s DNS server,
7467which presumably has permission to do domain lookups through the firewall.
7468See also:
7469• “Views in BIND 9” [Hack #77]
7470• “Ultrahosting: Mass Web Site Hosting with Wildcards, Proxy, and
7471Rewrite” [Hack #100]
7472• DNS & BIND, 4th Edition (O’Reilly)
7473Distributing Server Load with Round-Robin DNS #79
7474Information Servers | 167
7475HACK
7476H A C K
7477#79
7478Distributing Server Load with Round-Robin
7479DNS Hack #79
7480Direct traffic to multiple servers simultaneously with round-robin DNS
7481If you serve a particularly popular site, you will eventually find the wall at
7482which your server simply can’t serve any more requests. In the web server
7483world, this is called the Slashdot effect, and it isn’t a pretty site (er, sight).
7484While adding RAM, upgrading processors, and using faster drives and buses
7485will help in the short term, you may eventually find that a single machine
7486can’t possibly perform the work that needs to be done.
7487One way to overcome the limits of the monolithic server is to distribute the
7488load across many machines. By adding a second (or third) server to the avail-
7489able pool of machines, you can not only increase performance but also add
7490to the stability of the network. If you have a hot spare (or three) running all
7491of the time, then if one develops trouble, the others can take over for it with-
7492out any downtime.
7493Perhaps the easiest way to distribute the load of public traffic is to have the
7494public do the work of distributing the load for you. Through the magic of
7495round-robin DNS, inbound requests to a single host name can be directed to
7496come from any number of IP addresses.
7497In BIND 9, this is as easy as adding multiple A records for a single host. For
7498example, suppose we use this in the zone file for oreillynet.com:
7499www 60 IN A 208.201.239.36
7500www 60 IN A 208.201.239.37
7501Now, when a hosts looks up www.oreillynet.com in DNS, about half of the
7502time they will see:
7503rob@caligula:~$host www.oreillynet.com
7504www.oreillynet.com has address 208.201.239.36
7505www.oreillynet.com has address 208.201.239.37
7506and the rest of the time, they get:
7507rob@caligula:~$host www.oreillynet.com
7508www.oreillynet.com has address 208.201.239.37
7509www.oreillynet.com has address 208.201.239.36
7510As most applications only use the first address returned by DNS, this works
7511rather nicely. Approximately half of the requests will go to each address, and
7512therefore the load of two servers should be roughly half of that of a single
7513server. We set the TTL low (to 60 seconds) to prevent any intervening cach-
7514ing DNS servers from hanging onto one sort order for too long, which will
7515hopefully help keep the number of requests to each host more or less equal.
7516168 | Information Servers
7517#80 Running Your Own Top-Level Domain
7518HACK
7519It is only useful to spread out the load if all of the servers are in agreement
7520about what they’re serving. If your data gets out of sync, then browsers
7521might get one version of a web page on the first hit and another when they
7522hit reload. If you’re looking for a way to keep that from happening, take a
7523look at “Keeping Parts of Filesystems in sync with rsync” [Hack #41] .
7524Also, keep in mind that doing actual IP takeover (in case one host is unable
7525to perform its duties) is tricky business. If one server dies, you can’t change
7526DNS and wait for it to propagate to the entire Internet. You’ll need some-
7527thing that takes over for the down server immediately. Take a look at one
7528method of how to do this in “Cheap IP Takeover” [Hack #63] .
7529See also:
7530• “Keeping Parts of Filesystems in sync with rsync” [Hack #41]
7531• “Cheap IP Takeover” [Hack #63]
7532H A C K
7533#80
7534Running Your Own Top-Level Domain Hack #80
7535Set up your own TLD in BIND for ease of navigation
7536If you admin a network that uses private addressing, you’ve almost certainly
7537encountered the disassociated schizophrenia of trying to maintain zone files
7538that properly reflect internal and external IP addresses. With the introduc-
7539tion of views in “Views in BIND 9” [Hack #77] , supporting multiple address
7540ranges in a single domain has been significantly streamlined.
7541While using views is one way to attack the problem, consider the ease of set-
7542ting up your own top-level domain. Normally, zone entries in named.conf
7543look something like this:
7544zone "oreillynet.com" {
7545type master;
7546file "data/oreillynet.com";
7547};
7548This is an entry appropriate for an authoritative DNS server for the oreillynet.
7549com subdomain. The actual top level domains (i.e., .com, .net, .org, .int, etc.)
7550are only delegated to the mysterious 13 known as the root DNS servers. Even
7551though your servers won’t be consulted by the rest of the Internet, it can be
7552handy to set up your very own TLD that works only on your local network.
7553For example, suppose you have a group of machines that use the private
7554192.168.1.0/24 network. These machines aren’t directly reachable from the
7555Internet, and you don’t really want to advertise their DNS information to
7556would-be network crackers. Try a non-standard TLD:
7557Monitoring MySQL Health with mtop #81
7558Information Servers | 169
7559HACK
7560zone "bp" {
7561type master;
7562file "data/bp";
7563allow-transfer { 192.168.1/24; };
7564allow-query { 192.168.1/24; };
7565};
7566(The bp is short for BackPlane, and more to the point, is just plain short.)
7567With the above added to your zone file, set up a master record for bp just as
7568you would any other domain:
7569$TTL 86400
7570@ IN SOA ns.bp. root.homer.bp. (
75712002090100 ; Serial
757210800 ; Refresh after 3 hours
75733600 ; Retry after 1 hour
7574604800 ; Expire (1 week)
757560 ; Negative expiry time
7576)
7577IN NS ns.bp.
7578ns IN A 192.168.1.1
7579homer IN A 192.168.1.10
7580bart IN A 192.168.1.11
7581lisa IN A 192.168.1.12
7582Reload named, and you should be able to simply ping homer.bp . If you’d
7583like other name servers to maintain slave copies of your TLD, just add them
7584as usual:
7585zone "bp" {
7586type slave;
7587file "db.bp";
7588masters { 192.168.1.1; };
7589};
7590In this way, you can extend your new TLD across your entire private net-
7591work architecture. If you’re running tunnels over the Internet (as in “Tun-
7592neling: IPIP Encapsulation” [Hack #50] ) to connect remote offices or friends,
7593support for your TLD could theoretically grow to be as large as you like.
7594H A C K
7595#81
7596Monitoring MySQL Health with mtop Hack #81
7597Display MySQL threads in real time in a format similar to top
7598Much like its top counterpart, the mtop utility gives real time, running statis-
7599tics of your mysql server all in a terminal window. On a busy database
7600server, this can give you very precise details about what queries are running
7601(and taking up all of your resources).
7602170 | Information Servers
7603#81 Monitoring MySQL Health with mtop
7604HACK
7605When running mtop, you’ll need to pass at least the following two switches
7606on the command line:
7607mysql --dbuser=monitor --password=n0telling
7608Naturally, substituting your own database username and password. If you’re
7609running mtop from some host other than your database server, also specify
7610the --host={mysql_host} switch. Once it’s running, you’ll be presented with
7611a top-like screen that refreshes every few seconds:
7612load average: 0.72, 0.47, 0.26 mysqld 3.23.51 up 33 day(s), 4:48 hrs
76132 threads: 2 running, 0 cached. Queries/slow: 71.5K/0 Cache Hit: 99.99%
7614Opened tables: 42 RRN: 4.0M TLW: 0 SFJ: 0 SMP: 0
7615ID USER HOST DB TIME COMMAND STATE INFO
761626049 root localhost test Query show full processlist
761726412 root localhost nocat 1 Query Writing to n select * from Member where
7618User like '%rob%'
7619---
7620Here we see the attached users, the hosts they are connecting from, the que-
7621ries they’re running (and on which databases), as well as how long each
7622thread has been executing (in seconds). The numbers on the left are the
7623thread ID of each running query, not the mysql PID. If a particular query is
7624in danger of becoming a slow query, the line will turn magenta. If the query
7625then becomes a slow query, the color changes to yellow. If the thread is still
7626running after twice MySQL’s long_query_time value has passed, the line
7627becomes red. This makes it easy to tell at a glance if some queries are taking
7628a particularly long time to execute.
7629Just as with top, hitting ? while mtop is running shows all available keystrokes:
7630mtop ver 0.6.2/2002905, Copyright (c) 2002, Marc Prewitt/Chelsea Networks
7631A top users display for mysql
7632These single-character commands are available:
7633q - quit
7634? - help; show this text
7635f - flush status
7636k - kill processes; send a kill to a list of ids
7637s - change the number of seconds to delay between updates
7638m - toggle manual refresh mode on/off
7639d - filter display with regular expression (user/host/db/command/state/info)
7640h - display process for only one host
7641u - display process for only one user
7642i - toggle all/non-Sleeping process display
7643o - reverse the sort order
7644e - explain a process; show query optimizer info
7645t - show mysqld stats (show status/mysqladmin ext)
7646Monitoring MySQL Health with mtop #81
7647Information Servers | 171
7648HACK
7649v - show mysqld variables (show variables/mysqladmin vars)
7650z - zoom in on a process, show sql statement detail
7651r - show replication status for master/slaves
7652Probably the two most commonly used features are explain (e) and kill (k).
7653Hitting e will prompt you for a thread ID. Type in the number of the query
7654you’re interested in, and it will show you details about what mysql is actu-
7655ally doing when running the query:
7656Id: 27134 User: root Host: localhost Db: nocat Time: 0
7657Command: Query State: cleaning up
7658select *
7659FROM Member
7660WHERE User like '%rob%'
7661table |type |possible_keys |key | ken_len|ref | rows|
7662Member |ALL | | | | | 9652|where used
7663Likewise, the k key will allow you to kill a thread by supplying its thread ID.
7664This is tremendously handy for killing long-running (or process intensive,
7665poorly optimized) queries, without having to run mysqladmin multiple times
7666from the command line.
7667]One suggested method for skipping the need for specifying the user and
7668password on the command line is to create a mysqltop user with restricted
7669privileges. From perldoc mtop:
7670The most convenient way to setup your system to use mtop is to create a
7671database user called mysqltop which has no password. For security pur-
7672poses, this user should have all privileges set to N except Process_priv, which
7673must be set to Y.
7674To grant these privileges, execute the following from the MySQL command
7675prompt:
7676mysql> grant select on test.* to mysqltop;
7677mysql> grant select on test.* to mysqltop@localhost;
7678mysql> update user set process_priv='y' where user='mysqltop';
7679mysql> flush privileges;
7680With mtop installed and in your PATH, you can save yourself a lot of typing
7681(and guesswork) compared to hunting down misbehaving threads with
7682mysqladmin alone.
7683See also:
7684• The mtop package is available at http://mtop.sourceforge.net/ (which
7685requires Perl 5)
7686• Curses.pm (available through CPAN)
7687172 | Information Servers
7688#82 Setting Up Replication in MySQL
7689HACK
7690H A C K
7691#82
7692Setting Up Replication in MySQL Hack #82
7693Keep live copies of your database running to increase performance and
7694provide redundancy
7695As of Version 3.23.33, database replication is implemented in MySQL. It is
7696accomplished by maintaining a binary log of each database action on one
7697machine (the master), and keeping it in sync with each of the replicated cop-
7698ies on other machines (the slaves). As of MySQL 3.23, replication is only
7699one-way (that is, you may make changes to the database on master that get
7700propagated to each slave, but changes to the slaves are not sent back to the
7701master). If you need bidirectional replication, take a look at the bleeding
7702edge code in MySQL 4.
7703The most common reason for setting up replication is to distribute the load
7704of handling database requests across multiple machines. This will not only
7705help with performance, but will add a bit of redundancy should one of your
7706database servers fail. Your application must be smart enough to know that
7707write requests must be handed off to the master or else your replicated cop-
7708ies will get out of sync with the master. This is nobody’s idea of a fun time,
7709so make sure that every place in your code that needs to write to the data-
7710base only does so to the master copy.
7711Here are nine easy steps to getting replication running in MySQL 3.23.33 (or
7712later). Before starting, make sure that all of your machines are running the
7713same version of MySQL (preferably, the latest stable version).
77141. Decide which database server will be the master, and which will be the
7715slave(s). Generally, the master is your most capable machine. If you
7716have a particularly popular site that needs to handle many database
7717requests, you should strongly consider a multiprocessor machine with
7718plenty of RAM and a hardware raid. If your database server is under-
7719powered, then your entire dynamic application will suffer as other serv-
7720ers block while waiting for the database to be updated, no matter how
7721many machines you distribute the load to.
77222. Create a replication user on the master. This is a MySQL user just like
7723any other, only with FILE permission for each database that you wish to
7724replicate:
7725mysql>grant FILE on webdb.* to replicant@'%.mynetwork.edu' identified by
7726'seCret';
77273. Enable the binlog on the master, and pick which databases will be rep-
7728licated. Include something like this in the [mysqld] section of the /etc/
7729my.cnf on the master:
7730log-bin
7731server-id=1
7732Setting Up Replication in MySQL #82
7733Information Servers | 173
7734HACK
7735Also include binlog-do-db lines for each database you’d like to make
7736available via replication.:
7737binlog-do-db=webdb
7738Now bring down MySQL and immediately bring it back up again to
7739make your changes live:
7740root@db:/usr/local/mysql#mysqladmin shutdown; ./bin/safe_mysqld&
77414. Bring down the database on the master, and make an archive of the
7742data/ directory. No, recent mysqldumps are not sufficient for this job;
7743you will need an actual copy of the data/ directory. This may take a few
7744seconds to several minutes, depending on how much data you have in
7745your databases:
7746root@db:/usr/local/mysql#mysqladmin shutdown; tar cvf ~/data.tar data/
7747Your database server will be unavailable during the entire time that the
7748copy is running, so be sure to run this at an off-peak time. With any
7749luck, you’ll only have to do this once.
77505.Bring the database back up on the master. If it looks like your tar ran
7751okay, turn MySQL back on:
7752root@db:/usr/local/mysql#./bin/safe_mysqld&
77536. Copy the archive to each of the slaves, and make sure MySQL isn’t run-
7754ning on them. Try something like this on each slave:
7755root@slave:~#mysqladmin shutdown; scp db:data.tar .
77567. Pick a unique ID for each of the slaves, and enable replication. Include
7757the following in the [mysqld] section of the /etc/my.cnf on each of the
7758slaves:
7759master-host=db.mynetwork.edu
7760master-user=replicant
7761master-password=seCret
7762server-id=10
7763Note that the server-id must be unique for each slave. Choose any inte-
7764ger that hasn’t already been used for a slave (or the master).
7765If this slave will only be serving as a slave to the master-host (and won’t
7766be serving any read/write databases of its own) then you can get a bit of
7767a performance boost by including these options as well:
7768low-priority-updates
7769skip-bdb
7770delay-key-write-for-all-tables
7771This will eliminate the overhead of some of the code that is only used
7772when writing to the database. If you’re acting as a slave, you’ll never
7773actually write to a database, and so won’t need MySQL to be ready to
7774handle it.
7775D ownload from Wow! eBook <www.wowebook.com>
7776174 | Information Servers
7777#82 Setting Up Replication in MySQL
7778HACK
77798. Extract the data archive on each slave. This should do it:
7780root@slave:/usr/local/mysql#mv data data.old; tar vxf ~/data.tar
7781For good measure, make sure that the new data/ directory and its con-
7782tents are owned by the mysql user and group:
7783root@slave:/usr/local/mysql#chown -R mysql.mysql data/
7784If you don’t need anything from the original data/ directory on the slaves,
7785feel free torm -rf them (rather than backing them up todata.old/).
77869. Start MySQL on each slave, and watch the error log for problems.
7787Finally, fire up MySQL on each slave:
7788root@slave:/usr/local/mysql#./bin/safe_mysqld&
7789and watch the database log for errors:
7790root@slave:/usr/local/mysql#tail -f data/slave.err
7791(Naturally, substituting your machine’s real name for slave.) If you see a
7792message to the effect of Starting replication at position XYZ, then con-
7793gratulations: you’re replicating! Try making an update to the master,
7794and then do a query on a slave to see if the change propagates. Updates
7795typically happen almost instantaneously. If there are any problems with
7796replication, they will be reported in the mysql log on the slave that
7797encountered the problem.
7798Usually, problems with replication tend to be permission problems (check
7799your GRANT syntax in step 2) or else your data set is out of sync on the
7800slaves (repeat steps 4, 5, 6, 8, and 9, very carefully.) Proceed slowly, and
7801always consult the mysql error logs on each machine.
7802If you intend to add more slaves at a later date, then keep a copy of your
7803data.tar handy. You can use it at any time to create a new slave (it will sync
7804up with the master over the network, replaying the binlog that has been gen-
7805erated since the data.tar archive was created).
7806Simply enabling replication and maintaining a replicated database installa-
7807tion are very different propositions. As long as your application is smart
7808enough to only make updates to the master, and if your hardware is reli-
7809able, then you generally won’t run into difficulty. But should your slave cop-
7810ies ever get out of sync for any reason, then you’ll need to be armed with
7811much more information than I can present in this hack. Consult the excel-
7812lent resources below before attempting to run a replicated installation of any
7813complexity.
7814See also:
7815• MySQL Reference Manual (O’Reilly)
7816• MySQL online documentation, http://www.mysql.com/doc/en/
7817Replication.html
7818MySQL Server Tuning #84
7819Information Servers | 175
7820HACK
7821H A C K
7822#83
7823Restoring a Single Table from a Large MySQL
7824Dump Hack #83
7825Here is a method for restoring a single mysql table from a huge mysqldump
7826Like a good admin, you faithfully dump your mysql tables every night, and
7827save them to the filesystem in compressed form (presumably to be picked up
7828by a backup script later). You probably have something like this running in
7829cron on your database server (or one of its replicated slaves):
7830for x in `mysql -Bse show databases`; do
7831mysqldump $x | gzip -9 > /var/spool/mysqldump/$x.`date +%Y%m%d`.gz
7832done
7833This will cover you if anything catastrophic happens to your live database.
7834But if your database grows to an appreciable size, doing partial restores can
7835be difficult. On a database with several million rows, your dumps suddenly
7836become massive piles of data that need to be sifted through. How can you
7837easily restore a single table out of a several hundred megabyte compressed
7838dump?
7839Here’s a simple method using Perl. Create a script called extract-table, with
7840this in it:
7841#!/usr/bin/perl -wn
7842BEGIN { $table = shift @ARGV }
7843print if /^create table $table\b/io .. /^create table (?!$table)\b/io;
7844To extract the User table from the dump of a database called randomdb, try
7845something like this:
7846# zcat /var/spool/mysqldump/randomdb.20020901.gz | extract-table Users > ~/
7847Users.dump
7848Now you can restore your Users table with a simple:
7849# mysql randomdb -e "drop table Users"
7850# mysql randomdb < ~/Users.dump
7851H A C K
7852#84
7853MySQL Server Tuning Hack #84
7854Try these practical steps to help make your MySQL server run as efficiently
7855as it can
7856Many Linux administrators find themselves suddenly the “DBA in resi-
7857dence” when there is nobody else willing (or able) to take on the job. Many
7858people specialize in tuning and maintaining databases for a living and don’t
7859touch sysadmin responsibilities at all, and yet more than one Linux adminis-
7860trator I’ve known has been required to take on DBA responsibilities with lit-
7861tle training (and certainly no increase in pay). While this hack won’t turn
7862176 | Information Servers
7863#84 MySQL Server Tuning
7864HACK
7865you into a DBA expert, it will hopefully show you some practical steps that
7866have helped improve performance in real-world installations.
7867Here are five steps you can take to optimize your MySQL installation,
7868roughly in increasing order of difficulty (and effectiveness).
78691. Run mysqlcheck -o database . This will optimize your tables, reclaiming
7870lost space by “defragging” your database. This is especially useful if you
7871have recently changed the structure of your database, or have deleted a
7872large amount of data from it.
78732. Renice mysqld. If you are running a dedicated MySQL server, you can
7874tell the scheduler to run mysql at a much higher priority than other
7875tasks. The mysql manual recommends adding a line like this to your
7876safe_mysqld script:
7877renice -20 $$
7878However, I have also found it necessary to find the following hunk of
7879code:
7880NOHUP_NICENESS="nohup"
7881if test -w /
7882then
7883NOHUP_NICENESS=`nohup nice 2>&1`
7884if test $? -eq 0 && test x"$NOHUP_NICENESS" != x0 && nice --1 echo foo > /
7885dev/null 2>&1
7886then
7887NOHUP_NICENESS="nice --$NOHUP_NICENESS nohup"
7888else
7889NOHUP_NICENESS="nohup"
7890fi
7891fi
7892and replace it with simply:
7893NOHUP_NICENESS="nohup nice --20"
7894Now safe_mysqld and all of the mysqld processes will run at the great-
7895est possible priority. It does this at the expense of all other processes, so
7896if you are trying to run other services along with mysql on the same
7897machine, you may want to pick a higher number (somewhere in the -10
7898to -5 range is probably a bit more conservative).
78993. Create indices. If you have long running queries, one very good optimi-
7900zation you can make is to add appropriate indices. If you see a long run-
7901ning query when running mtop (as discussed in “Monitoring MySQL
7902Health with mtop” [Hack #81] ), then consider creating a relevant index:
7903mysql>create index name on Member (Name(10));
7904Indexing is a trade-off of disk space for performance, and in this age of
7905inexpensive storage, there’s no excuse for neglecting your indices. Hav-
7906ing too many indices usually doesn’t hurt, but doing a linear search (or a
7907MySQL Server Tuning #84
7908Information Servers | 177
7909HACK
7910unique insert) on a large table that doesn’t have an associated index can
7911make your server crawl.
79124.Check your server variables. The default server variables are designed to
7913provide safe, sensible settings for modestly equipped machines. If you
7914have a large amount of RAM (512MB or more) then you can see tremen-
7915dous benefits by increasing the size of the default buffers and caches.
7916Here are some variables that we run on a production database server (a
7917dual Pentium 4/1.0GHz with 2GB RAM and lots of fast disk space). Put
7918them in the [mysqld] section of your /etc/my.cnf:
7919set-variable = key_buffer=384M
7920set-variable = max_allowed_packet=1M
7921set-variable = table_cache=512
7922set-variable = sort_buffer=2M
7923set-variable = record_buffer=2M
7924set-variable = myisam_sort_buffer_size=64M
7925set-variable = tmp_table_size=8M
7926set-variable = max_connections=768
7927Note that many of these are straight out of the my-huge.cnf sample
7928included with the mysql distribution (we haven’t had to change them,
7929since they work fine for our installation.) Since we run Apache::DBI on
7930our web servers, we also run with a wait_timeout of a few minutes:
7931set-variable = wait_timeout=120
7932This helps prevent idle DBI threads from hanging out and taking up all
7933available max_connections.
79345. Patch glibc and threads. When you have exhausted the abilities of the
7935default installation of glibc, consider patching it (see “Optimizing glibc,
7936linuxthreads, and the Kernel for a Super MySQL Server” [Hack #86] ) to
7937allow for smaller threads and more open files. This is normally not an
7938issue except for very large, very busy MySQL installations.
7939As with many topics in this book, database tuning and administration is a
7940much more complicated topic than can be covered in a few short pages. Con-
7941sult the resources below for more authoritative discussion on the subject.
7942See also:
7943• MySQL Reference Manual (O’Reilly)
7944• MySQL (New Riders)
7945• http://www.mysql.com/doc/en/Linux.html
7946• http://www.mysql.com/doc/en/SHOW_VARIABLES.html
7947178 | Information Servers
7948#85 Using proftpd with a mysql Authentication Source
7949HACK
7950H A C K
7951#85
7952Using proftpd with a mysql Authentication
7953Source Hack #85
7954Eliminate the need for user accounts for ftp users with proftpd and mysql
7955The proftpd ftp daemon is a powerful ftp daemon with a configuration syn-
7956tax much like Apache. It has a whole slew of options not available in most
7957ftp daemons, including ratios, virtual hosting, and a modularized design that
7958allows people to write their own modules.
7959One such module is mod_sql, which allows proftpd to use a SQL database as
7960its back end authentication source. Currently, mod_sql supports MySQL
7961and PostgreSQL. This can be a good way to help lock down access to your
7962server, as inbound users will authenticate against the database (and there-
7963fore not require an actual shell account on the server). In this hack, we’ll get
7964proftpd authenticating against a MySQL database.
7965First, download and build the source to proftpd and mod_sql:
7966~$bzcat proftpd-1.2.6.tar.bz2 | tar xf -
7967~/proftpd-1.2.6/contrib$tar zvxf ../../mod_sql-4.08.tar.gz
7968~/proftpd-1.2.6/contrib$cd ..
7969~/proftpd-1.2.6$ ./configure --with-modules=mod_sql:mod_sql_mysql \
7970--with-includes=/usr/local/mysql/include/ \
7971--with-libraries=/usr/local/mysql/lib/
7972(Naturally, substitute the path to your mysql install, if it isn’t in /usr/local/
7973mysql/.) Now, build the code and install it:
7974rob@catlin:~/proftpd-1.2.6$make && sudo make install
7975Next, create a database for proftpd to use (assuming that you already have
7976mysql up and running):
7977$mysqladmin create proftpd
7978then permit read-only access to it from proftpd:
7979$mysql -e "grant select on proftpd.* to proftpd@localhost \
7980identified by 'secret';"
7981Create two tables in the database, with this schema:
7982CREATE TABLE users (
7983userid varchar(30) NOT NULL default '',
7984password varchar(30) NOT NULL default '',
7985uid int(11) default NULL,
7986gid int(11) default NULL,
7987homedir varchar(255) default NULL,
7988shell varchar(255) default NULL,
7989UNIQUE KEY uid (uid),
7990UNIQUE KEY userid (userid)
7991) TYPE=MyISAM;
7992Using proftpd with a mysql Authentication Source #85
7993Information Servers | 179
7994HACK
7995CREATE TABLE groups (
7996groupname varchar(30) NOT NULL default '',
7997gid int(11) NOT NULL default '0',
7998members varchar(255) default NULL
7999) TYPE=MyISAM;
8000One quick way to create the tables is to save the above to a file called
8001proftpd.schema and run a command like mysql proftpd < proftpd.schema .
8002Now we need to tell proftpd to use this database for authentication. Add the
8003following lines to /usr/local/etc/proftpd.conf:
8004SQLConnectInfo proftpd proftpd secret
8005SQLAuthTypes crypt backend
8006SQLMinUserGID 111
8007SQLMinUserUID 111
8008The SQLConnectInfo line takes the form database user password. You could
8009also specify a database on another host (even on another port) with some-
8010thing like:
8011SQLConnectInfo proftpd@dbhost:5678 somebody somepassword
8012The SQLAuthTypes line lets you create users with passwords stored in the
8013standard Unix crypt format, or mysql’s PASSWORD() function. Be warned
8014that if you’re using mod_sql’s logging facilities, that the password may be
8015exposed in plain text, so keep those logs private.
8016The SQLAuthTypes line as specified won’t allow blank passwords; if you need
8017that functionality, also include the empty keyword. The SQLMinUserGID and
8018SQLMinUserUID specify the minimum group and user id that proftpd will per-
8019mit on login. It’s a good idea to make this greater than zero (to prohibit root
8020logins) but should be as low as you need to allow proper permissions in the
8021filesystem. On this system, we have a user and group called www, with both
8022its uid and gid set to 111. As we’ll want web developers to be able to log in
8023with these permissions, we’ll need to set the minimum values to 111.
8024Finally, we’re ready to create users in the database. This will create the user
8025jimbo, with effective user rights as www/www, and dump him in the /usr/
8026local/apache/htdocs/ directory at login:
8027mysql -e "insert into users values ('jimbo',PASSWORD('sHHH'),'111', \
8028'111', '/usr/local/apache/htdocs','/bin/bash');" proftpd
8029The password for jimbo is encrypted with mysql’s PASSWORD() function
8030before being stored. The /bin/bash line is passed to proftpd to pass proftpd’s
8031RequireValidShell directive. It has no bearing on granting actual shell access
8032to the user jimbo.
8033180 | Information Servers
8034#86 Optimizing glibc, linuxthreads, and the Kernel for a Super MySQL Server
8035HACK
8036At this point, you should be able to fire up proftpd and log in as user jimbo,
8037with a password of sHHH. If you are having trouble getting connected, try
8038running proftpd in the foreground with debugging on, like this:
8039#proftpd -n -d 5
8040Watch the messages as you attempt to connect, and you should be able to
8041track down the source of difficulty. In my experience, it’s almost always due
8042to a failure to set something properly in proftpd.conf, usually regarding per-
8043missions.
8044The mod_sql module can do far more than I’ve shown here; it can connect
8045to existing mysql databases with arbitrary table names, log all activity to the
8046database, modify its user lookups with an arbitrary WHERE clause, and
8047much more.
8048See also:
8049• The mod_sql home is at http://www.lastditcheffort.org/~aah/proftpd/
8050mod_sql/
8051• The proftpd home is at http://www.proftpd.org/
8052H A C K
8053#86
8054Optimizing glibc, linuxthreads, and the Kernel
8055for a Super MySQL Server Hack #86
8056Make sure that your database system’s OS is running as efficiently as
8057possible with these tweaks
8058If you have a very busy MySQL server (serving, say, 800+ queries/second
8059and hundreds of simultaneous clients), you may begin to run into limita-
8060tions of the operating system that prevent mysql from operating as effi-
8061ciently as it could. In extreme cases (on systems with a misbehaving threads
8062library), mysql has been observed to suddenly take up all available CPU
8063cycles, artificially driving the load to 100+ once a critical resource threshold
8064has been reached.
8065By building a dedicated mysql server, and adjusting some default values in
8066glibc, linuxthreads, and the Linux kernel, it is possible to make a single data-
8067base machine serve thousands of simultaneous requests. Naturally, you’ll
8068need hardware capable of supporting the load, but with these modifica-
8069tions, the OS and MYSQL build will probably not be the limiting perfor-
8070mance factor.
8071WARNING: the following hack makes changes to critical, sensitive areas of
8072your server. Don’t go monkeying with your libc and kernel source lightly, and
8073even then only if you have a complete, verified, offline (and preferably off-site)
8074backup of your entire system. This procedure should only be followed on
8075Optimizing glibc, linuxthreads, and the Kernel for a Super MySQL Server #86
8076Information Servers | 181
8077HACK
8078dedicated MySQL server machines, and then only when you are certain that
8079everything else is in order (especially your /etc/my.cnf variables). You have
8080been warned!
8081Step 1: Build glib. Download the glibc source code from http://www.gnu.org/
8082software/libc/libc.html. The latest version as of this writing is glibc 2.2.5.
8083Also pick up a copy of linuxthreads to go along with whichever glibc you
8084download.
8085Expand the glibc archive, and install the linuxthreads module (as per the
8086instructions in the glibc distribution). Next, follow the hacks suggested at
8087http://www.mysql.com/doc/en/Linux.html, including:
8088• sysdeps/unix/sysv/linux/bits/local_lim.h set PTHREAD_THREADS_MAX
8089to 4096
8090• linuxthreads/internals.h set STACK_SIZE to 256 KB
8091Now, when running configure , include this switch: --prefix=/usr/local/
8092glibc-2.2.5 . This will build glibc into its own directory, rather than replace
8093the system’s existing glibc. This is critical when running make install , as
8094attempting to overwrite a running system’s glibc will typically result in
8095library soup! A broken libc is difficult to fix without statically built system
8096tools and a boot disk, and I don’t recommend attempting it unless you have
8097time on your hands (and a full backup of the machine you just broke).
8098Building glibc to an alternate directory (like /usr/local/glibc-2.2.5) neatly
8099sidesteps the problems of upgrading the system’s libc.
8100Now build glibc and install it. If all went well, you should have a shiny new
8101glibc installed to /usr/local/glibc-2.2.5/.
8102Step 2: The Kernel. Make the following changes to your kernel source tree:
8103• include/linux/limits.h set NR_OPEN to 4096
8104• include/linux/fs.h set INR_OPEN to 4096
8105Now rebuild your kernel (and modules), install the new kernel, and reboot.
8106Step 3: Build a New MySQL. Download and unpack the MySQL source tree.
8107When you run configure , include the --use-other-libc=/usr/local/glibc-
81082.2.5 switch. This will link mysql against your new glibc instead of the sys-
8109tem’s glibc. Now build and install MySQL as normal.
8110Step 4: Expand the Maximum Filehandles at Boot. Add the following line to
8111your /etc/rc.d/rc.local (or other part of the boot sequence, before safe_mysqld
8112runs):
8113echo 65536 > /proc/sys/fs/file-max
8114182 | Information Servers
8115#87 Apache Toolbox
8116HACK
8117Now reboot (or run the above command by hand), and fire up safe_mysqld.
8118In benchmarks we have run in-house, the maximum number of available
8119connections was increased from several hundred to 4090, with no observed
8120CPU spike. We run dedicated database servers configured with these set-
8121tings in production, with absolutely no performance or stability issues.
8122If you have a significant amount of database traffic, you should consider dis-
8123tributing the load across multiple machines discussed in “Setting Up Repli-
8124cation in MySQL” [Hack #82] .
8125H A C K
8126#87
8127Apache Toolbox Hack #87
8128Use this great installation script to automatically download, configure,
8129compile, and install Apache (and friends)
8130Bryan Andrew’s Apache Toolbox is a Swiss army knife of a script, providing
8131a customisable, menu-driven interface to downloading and compiling
8132Apache, mod_perl, MySQL, PHP—and more! The Apache Toolbox has out-
8133of-the-box support for:
8134Apache
8135Our favorite web server
8136SSL
8137Secure Sockets Layer for secure web server interactions
8138PHP, mod_perl, and mod_fastcgi
8139For speedy scripting language support
8140MySQL
8141The ubiquitous lightning fast database server
8142OpenLDAP, mod_auth_ldap, mod_auth_radius, mod_auth_pop3, mod_auth_
8143sys and mod_accessref
8144Various means of authentication and authorization
8145WebDAV and mod_layout
8146For simple, yet powerful Web design functionality
8147mod_dynvhost, mod_throttle, mod_gzip, and mod_bandwidth
8148For efficient hosting and server control
8149And more! Apache Toolbox is customisable and has support (it’s just a shell
8150script after all) for anything else you’d care to plug in.
8151Apache Toolbox comes in two different flavours: just the script, which will
8152download the various component sources as needed, or the full package,
8153including the script and all required sources. The toolbox even catches RPM
8154conflicts should they arise.
8155Apache Toolbox #87
8156Information Servers | 183
8157HACK
8158The Apache Toolbox itself is simply a shell script that runs from the com-
8159mand line. It must be run as root (as it’ll remind you) to install all the vari-
8160ous bits into their appropriate locations. The first step is a menu-driven
8161GUIsh interface to selecting the various packages you wish to install:
8162-----------------------------------------------------------------------
8163Apache Toolbox 1.5.59
8164Support: http://www.apachetoolbox.com
8165-----------------------------------------------------------------------
8166[+] apache) Apache submenu...
8167[-] php) PHP submenu (v4.2.2)...
8168[-] rpm) Build an RPM with your choices?
8169[-] page2) Apache Modules PAGE 2 ...
8170-----------------------------------------------------------------------
8171[-] 1) GD 2.0.1 [-] 2) -SQL DB Menus-
8172[-] 4) Mod Python 2.7.8 [-] 5) Mod_SSL+OpenSSL
8173[-] 6) -Mod Throttle 312 [-] 7) -WebDAV 1.0.3-1.3.6
8174[-] 8) -Mod FastCGI [-] 9) -Mod AuthNDS 0.5
8175[-] 10) -Frontpage 2002 [-] 11) -Mod GZIP 1.3.19.1a
8176[-] 12) -Mod DynaVHost [-] 13) -Mod Roaming
8177[-] 14) -Mod AccessRef 1.0.1 [-] 15) -Mod AuthSYS
8178[-] 16) -Mod Bandwidth [-] 17) -Mod Perl 1.27
8179[-] 18) -Mod Auth LDAP [-] 19) -Apache Jakarta
8180[-] 20) -Mod Auth Radius [-] 21) -Mod Auth POP3
8181[-] 22) -Mod Layout 3.2 [-] 23) -Mod DTCL
8182q) Quit 99) Descriptions
8183go) Compile selections...
8184-----------------------------------------------------------------------
8185Not sure what all these goodies are? Type 99<enter> for complete descrip-
8186tions. After putting together your grocery list of preferred features, type go
8187and the toolbox springs into action. First stop: RPM conflict checks:
8188-------------------------------------------------------
8189-------------- Scanning for RPM's ---------------------
8190-------------------------------------------------------
8191Testing for PHP RPM... not found.
8192Testing for PHP IMAP RPM... not found.
8193Testing for GD RPM... not found.
8194Testing for GD Devel RPM... not found.
8195Testing for Apache RPM... not found.
8196...
8197Testing for OpenLDAP RPM... not found.
8198Testing for OpenLDAP Devel RPM... not found.
8199[+] Wget found!
8200-------------------------------------------------------
8201The toolbox confirms Apache install path (/usr/local/apache), allowing you
8202to change the final destination per your preference, then continues on its
8203way.
8204D ownload from Wow! eBook <www.wowebook.com>
8205184 | Information Servers
8206#87 Apache Toolbox
8207HACK
8208Any time a source (tar.gz archive) isn’t found, the toolbox script prompts
8209you for permission to download it:
8210[+] Setting up Apache source...
8211[-] apache_1.3.26.tar.gz detection failed
8212Do you wish to download it now? [y/n] y
8213--21:35:40--
8214--ftp://mirrors.partnersforever.net:21/pub/apache/dist/
8215apache_1.3.26.tar.gz => `apache_1.3.26.tar.gz'
8216Connecting to mirrors.partnersforever.net:21... connected!
8217Logging in as anonymous ... Logged in!
8218==> TYPE I ... done. ==> CWD pub/apache/dist ... done.
8219==> PORT ... done. ==> RETR apache_1.3.26.tar.gz ... done.
8220Length: 2,303,147 (unauthoritative)
82210K -> .......... .......... .......... .......... [ 2%]
822250K -> .......... .......... .......... .......... [ 5%]
8223...
8224The toolbox then silently hums away configuring, integrating, preheating,
8225and so on:
8226[+] Uncompressed Apache source...
8227[+] Getting apache pre-configured
8228[+] Apache pre-configured
8229[+] Apache httpd.conf-dist updated for SSI support
8230[+] Getting GD lib's with PNG and zlib support ready...
8231...
8232After a short while, the toolbox gives you the chance to edit the Apache con-
8233figuration script, if you’re just that bold. If all’s well you should end up with
8234a rather familiar site if you’ve ever configured Apache before:
8235Configuring for Apache, Version 1.3.26
8236+ using installation path layout: Apache (config.layout)
8237+ activated php4 module (modules/php4/libphp4.a)
8238+ Warning: You have enabled the suEXEC feature. Be aware
8239+ that you need root privileges to complete the final
8240+ installation step.
8241Creating Makefile
8242Creating Configuration.apaci in src
8243Creating Makefile in src
8244+ configured for Linux platform
8245...
8246Creating Makefile in src/modules/standard
8247Creating Makefile in src/modules/php4
8248[+] Done Configuring Apache source
8249----------------------------------------------------------
8250If there where _no_ errors run "cd apache_1.3.26;make" now.
8251Start debugging and have a blast...
8252Run "make install" in the apache source
8253directory to install apache 1.3.26
8254----------------------------------------------------------
8255Display the Full Filename in Indexes #88
8256Information Servers | 185
8257HACK
8258Mosey on over to the Apache directory, as specified, make, make test, and
8259make install!
8260See Also:
8261• See the original article at http://www.onlamp.com/pub/a/apache/2000/11/
826217/wrangler.html
8263• Apache Toolbox
8264• Linux Apache MySQL PHP (LAMP) Guide (Linux Help.Net)
8265• HTTP Wrangler columns (O’Reilly Network)
8266H A C K
8267#88
8268Display the Full Filename in Indexes Hack #88
8269Stop truncating the filenames in your auto-indexed directories
8270Have you ever noticed that Apache will truncate filenames in directory list-
8271ings? It’s terribly frustrating to go to a web page full of fun archives, only to
8272see something like:
8273Index of /~rob/stuff/kernel
8274Name Last modified Size Description
8275Parent Directory 03-Sep-2002 00:33 -
8276patched-linux-2.4.12..> 11-Oct-2001 00:59 22.0M
8277patched-linux-2.4.12..> 11-Oct-2001 00:59 1k
8278patched-linux-2.4.12..> 11-Oct-2001 00:59 27.1M
8279patched-linux-2.4.12..> 11-Oct-2001 00:59 1k
8280patched-linux-2.4.13..> 23-Oct-2001 22:28 22.0M
8281patched-linux-2.4.13..> 23-Oct-2001 22:28 1k
8282patched-linux-2.4.13..> 23-Oct-2001 22:28 27.2M
8283patched-linux-2.4.13..> 23-Oct-2001 22:28 1k
8284patched-linux-2.4.14..> 05-Nov-2001 15:30 22.1M
8285patched-linux-2.4.14..> 05-Nov-2001 15:30 1k
8286patched-linux-2.4.14..> 05-Nov-2001 15:30 27.4M
8287patched-linux-2.4.14..> 05-Nov-2001 15:30 1k
8288patched-linux-2.4.15..> 22-Nov-2001 22:18 22.6M
8289patched-linux-2.4.15..> 22-Nov-2001 22:18 1k
8290patched-linux-2.4.15..> 22-Nov-2001 22:18 28.0M
8291patched-linux-2.4.15..> 22-Nov-2001 22:18 1k
8292How can you tell which files are which without hovering over every link? Fix
8293it by setting your IndexOptions line in the httpd.conf to include a Name-
8294Width option:
8295IndexOptions FancyIndexingNameWidth=*
8296By default, Apache only ships with FancyIndexing on. Now issue an
8297apachectl restart, and reload that page to see the full filename:
8298186 | Information Servers
8299#89 Quick Configuration Changes with IfDefine
8300HACK
8301Index of /~rob/stuff/kernel
8302Parent Directory 03-Sep-2002 00:33 - Description
8303patched-linux-2.4.12.tar.bz2 11-Oct-2001 00:59 22.0M
8304patched-linux-2.4.12.tar.bz2.sign 11-Oct-2001 00:59 1k
8305patched-linux-2.4.12.tar.gz 11-Oct-2001 00:59 27.1M
8306patched-linux-2.4.12.tar.gz.sign 11-Oct-2001 00:59 1k
8307patched-linux-2.4.13.tar.bz2 23-Oct-2001 22:28 22.0M
8308patched-linux-2.4.13.tar.bz2.sign 23-Oct-2001 22:28 1k
8309patched-linux-2.4.13.tar.gz 23-Oct-2001 22:28 27.2M
8310patched-linux-2.4.13.tar.gz.sign 23-Oct-2001 22:28 1k
8311patched-linux-2.4.14.tar.bz2 05-Nov-2001 15:30 22.1M
8312patched-linux-2.4.14.tar.bz2.sign 05-Nov-2001 15:30 1k
8313patched-linux-2.4.14.tar.gz 05-Nov-2001 15:30 27.4M
8314patched-linux-2.4.14.tar.gz.sign 05-Nov-2001 15:30 1k
8315patched-linux-2.4.15.tar.bz2 22-Nov-2001 22:18 22.6M
8316patched-linux-2.4.15.tar.bz2.sign 22-Nov-2001 22:18 1k
8317patched-linux-2.4.15.tar.gz 22-Nov-2001 22:18 28.0M
8318patched-linux-2.4.15.tar.gz.sign 22-Nov-2001 22:18 1k
8319Much better. Install this by default on all of your Apache installations, and
8320your users will thank you.
8321H A C K
8322#89
8323Quick Configuration Changes with IfDefine Hack #89
8324Make changes to your running Apache configuration without editing httpd.
8325conf
8326Apache allows you to modify its configuration quickly and easily with the
8327IfDefine and IfModule directives. IfDefine allows you to designate blocks of
8328configuration to only be active when a command line flag has been passed,
8329like so:
8330#/usr/local/apache/bin/httpd -DSSL
8331The above is a common example, defining (-D) a flag called SSL. This will
8332enable parts of the configuration that look like:
8333<IfDefine SSL>
8334# anything in here will be enabled if -DSSL
8335# has been passed on the command line.
8336</IfDefine>
8337Likewise, the IfModule directive checks to see if a module is currently loaded
8338(i.e., uncommented) in the Apache configuration file. If mod_userdir.c is
8339loaded, for example, then the following configuration will be enabled. If it’s
8340not, it will be silently ignored.
8341<IfModule mod_userdir.c>
8342UserDir public_html
8343</IfModule>
8344Quick Configuration Changes with IfDefine #89
8345Information Servers | 187
8346HACK
8347As Apache has grown, the httpd.conf file has become steadily more modular-
8348ized as shown above, allowing you to tweak the various modules that load
8349without fear of breaking your installation due to missing dependencies.
8350With the above examples in mind, you can easily change how Apache oper-
8351ates by passing simple command lines. Let’s assume that every Friday you
8352release a newsletter to ten thousand people. You know from experience that
8353the weekends are when your web server gets hit the heaviest. Here’s a snip-
8354pet from a Apache configuration file using IfDefines:
8355<IfDefine !WEEKEND>
8356MinSpareServers 1
8357MaxSpareServers 5
8358StartServers 1
8359MaxClients 150
8360</IfDefine>
8361<IfDefine WEEKEND>
8362MinSpareServers 5
8363MaxSpareServers 15
8364StartServers 5
8365MaxClients 300
8366</IfDefine>
8367With the above configuration, if you start up your server normally:
8368#/usr/local/apache/bin/httpd
8369then you’ll use the default Apache settings for the number of servers to start
8370and maintain during normal running operation. However, before you leave
8371work on Friday, you can simply stop the daemon and restart it with:
8372#/usr/sbin/httpd -DWEEKEND
8373Upon restart, you’ll have a configuration that’s better tuned for the week-
8374end traffic.
8375Here’s an example of one way to use IfModule:
8376<IfModule libperl.so>
8377PerlModule Apache::Registry
8378<Location /usr/local/httpd/cgi-bin>
8379SetHandler perl-script
8380PerlHandler Apache::Registry
8381Options +ExecCGI
8382</Location>
8383</IfModule>
8384In this case, mod_perl is activated for all of our (hopefully well-written) CGI
8385scripts automatically, whenever the mod_perl module is loaded. This could
8386also be useful during the scenario described above—by turning on mod_perl,
8387our scripts will respond faster during the heavier traffic. If we don’t need the
8388188 | Information Servers
8389#90 Simplistic Ad Referral Tracking
8390HACK
8391overhead of mod_perl during the weekdays (or to make debugging cgi prob-
8392lems easier), we can simply comment mod_perl’s LoadModule and
8393AddModule lines, and we’ll be back to normal.
8394You can use the ideas presented above to create developer specific configura-
8395tions, usable with a -DEVELOPER command, or even a -DVIRTUAL_HOSTS
8396flag, which can be removed when your web hosting clients haven’t paid:
8397<IfDefine VIRTUAL_HOSTS>
8398# all your various <VirtualHosts> blocks would
8399# go in here. Without the command line flag of
8400# -DVIRTUAL_HOSTS (to signify that your clients
8401# have paid you, for instance), then all virtual
8402# hosts would be disabled.
8403</IfDefine>
8404<IfDefine EVELOPER>
8405# notice our trickery with "EVELOPER"
8406# to make the command line more readable.
8407# when -DEVELOPER has been passed, we load
8408# various modules that aren't used during production.
8409LoadModule proxy_module libexec/httpd/libproxy.so
8410LoadModule expires_module libexec/httpd/mod_expires.so
8411LoadModule usertrack_module libexec/httpd/mod_usertrack.so
8412AddModule mod_proxy.c
8413AddModule mod_expires.c
8414AddModule mod_usertrack.c
8415</IfDefine>
8416If your site uses apachectl to bring Apache up and down, you can edit
8417apachectl to include your favorite defines (it’s just a shell script.) Find this
8418line in apachectl:
8419HTTPD=/usr/local/apache/bin/httpd
8420and change it to something like this:
8421HTTPD="/usr/local/apache/bin/httpd -DSSL -DEVELOPER -DVIRTUAL_HOSTS"
8422Now when you run apachectl start , your -D defines will be properly
8423passed to httpd.
8424H A C K
8425#90
8426Simplistic Ad Referral Tracking Hack #90
8427Create simple user tracking in print ads using the QUERY_STRING
8428You’re planning on advertising in a number of magazines, web sites, and
8429newspapers, and you realize that you’d like to gauge how much traffic is
8430coming in from each advertisement—it’ll help you better spend your money
8431in the future. Unfortunately, you have no time to implement something
8432complicated, so you’re looking for a low-tech solution, as well as something
8433that will make it easy to analyze the results.
8434Simplistic Ad Referral Tracking #90
8435Information Servers | 189
8436HACK
8437Apache happily logs all environment variables passed to the pages it serves
8438to the outside world. Thus, the hack is simple: pass an advertisement-based
8439QUERY_STRING to the main page of your site. The QUERY_STRING will
8440be ignored if you don’t specifically act on it (with SSI, PHP, CGI, etc.), but
8441Apache will still log the information to it’s access_log.
8442For example, say you’re advertising in the New York Times. Instead of say-
8443ing “Come visit us at http://www.GatesMcFaddenCo.com,” change the adver-
8444tisement to specifically match the readership:
8445http://www.GatesMcFaddenCo.com/?nyt
8446In the above example, anytime someone types in that address, Apache will
8447serve the main page normally but silently log that a QUERY_STRING of
8448“nyt” has been passed. Using a log analyzer (like analog, or the simple one
8449below), you can then find out how many people visited your site from see-
8450ing your ad in the New York Times.
8451Since you’re using QUERY_STRINGs, you can also act upon them with
8452Server Side Includes, PHP, or any other server-side language. For instance,
8453the example HTML page below would show different greetings based on the
8454web site address that the user typed in.
8455<html>
8456<head>
8457<title>Apache Hack #12396 - Simplistic Ad Referrel Tracker</title>
8458</head>
8459<body>
8460<!--#if expr="\"$QUERY_STRING\" = \"nyt\"" -->
8461<h1>Welcome New York Times Reader!</h1>
8462<!--#elif expr="\"$QUERY_STRING\" = \"xml\"" -->
8463<h1>Welcome XML.com Reader!</h1>
8464<!--#else -->
8465<h1>Welcome To Our Site!</h1>
8466<!--#endif -->
8467</body>
8468</html>
8469The above will show special greetings if the user types in a QUERY_
8470STRING that corresponds to advertisements at XML.com or at the New
8471York Times. If the user didn’t type in either, then a generic welcome is
8472shown. Some sites have been known to change more than just the greeting,
8473customizing the color, logos, and internal ads served.
8474Be careful when you’re choosing your QUERY_STRING—if it’s too long,
8475hard to remember, or suitably threatening, then the user may mistype the
8476address (creating bad statistics) or else not type the QUERY_STRING at all.
8477In a worst case scenario, they may not even make the attempt to visit. These
8478are all bad choices:
8479190 | Information Servers
8480#90 Simplistic Ad Referral Tracking
8481HACK
8482# this one is simply too long.
8483http://www.gamegrene.com/?newyorktimes-04-21
8484# this one would be hard to remember or type.
8485http://www.gamegrene.com/?04xlmfo3d
8486# and this may worry people.
8487http://www.gamegrene.com/?track=nyt
8488If you are advertising online, encoding query strings in anchor tags works
8489just fine (and can be as long as you like, as it’s all just one click as far as the
8490user knows). To analyze your stats upon request, try the following Perl
8491script.
8492Listing: referral-report.pl
8493#!/usr/bin/perl -w
8494use strict; my %ads;
8495# define each of your QUERY_STRINGS,
8496# and the matching "real name" here.
8497# this script will only look for
8498# QUERY_STRINGS that are letters and
8499# numbers only (no dashes, etc.)
8500$ads{"xml"} = "XML.com";
8501$ads{"nyt"} = "New York Times";
8502$ads{"ora"} = "O'Reilly and Associates";
8503# your Apache access log location.
8504my $logfile = "/var/log/httpd/access_log";
8505# define some counters.
8506my %ads_count; my $total;
8507# open the logfile.
8508open(LOG, "<$logfile") or die $!;
8509# and loop through each line.
8510while (<LOG>) {
8511# skip over lines we're not interested in.
8512next unless /GET/; next unless /\?(\w+)\s/;
8513# save the query_string.
8514my $query_string = $1;
8515# move on if not defined, else increment.
8516next unless exists $ads{$query_string};
8517$total++; $ads_count{$query_string}++;
8518}
8519Mimicking FTP Servers with Apache #91
8520Information Servers | 191
8521HACK
8522# and print out the data.
8523print "There were a total of $total ad referrals.\n";
8524foreach ( sort keys %ads_count ) {
8525print "$ads{$_} had $ads_count{$_} ad referrals.\n";
8526}
8527# close logfile and
8528# exit the program.
8529close(LOG); exit;
8530The results of this script would look like:
8531There were a total of 17 ad referrals.
8532XML.com had 6 ad referrals.
8533New York Times had 7 ad referrals.
8534O'Reilly and Associates had 4 ad referrals.
8535Being simplistic, this code does not track multiple hits from the same IP
8536address, nor does it create percentages based on the total amount of refer-
8537rals received. Both should be easy for any Perl hacker to implement.
8538H A C K
8539#91
8540Mimicking FTP Servers with Apache Hack #91
8541Set up multiple levels of anonymous access in Apache
8542You want to set up a portion of your web site to act like an FTP server—
8543some users should have accounts for downloading, while others should have
8544anonymous downloading access. Still others should be allowed to down-
8545load one type of file but not another. You don’t trust the intelligence of your
8546clientele to handle installing and using an FTP program, so you’d like to
8547make things work within a normal browsing environment.
8548Using an oft-ignored Apache module included by default, you can allow
8549authenticated anonymous access to certain parts of your site, as well as falling
8550back on normal authentication methods with mod_auth. This becomes handy
8551if you’ve got a large amount of documents that you don’t want indexed by
8552search engines, or you want to create the effect of multiple levels of user
8553power, such as anonymous, privileged, and leech. You want to do it all
8554through a browser and all without additional programming.
8555The first step is to enable the anon_auth_module, which is normally com-
8556mented out within the Apache configuration file. To do so, look for the fol-
8557lowing lines (which may be slightly different in your installation) and
8558remove the #:
8559#LoadModule anon_auth_module libexec/httpd/mod_auth_anon.so
8560#AddModule mod_auth_anon.c
8561With the above uncommented and Apache restarted, you can now start add-
8562ing relevant directives to httpd.conf, or an .htaccess file located in the directory
8563D ownload from Wow! eBook <www.wowebook.com>
8564192 | Information Servers
8565#91 Mimicking FTP Servers with Apache
8566HACK
8567you want protected. Below, we’ll assume you’re using an .htaccess file. Plop
8568the following into your .htaccess and save to the directory you want to protect:
8569AuthName "anonymous/your email address"
8570AuthType Basic
8571Require valid-user
8572Anonymous orko bender
8573Anonymous_Authoritative on
8574You should recognize the first three directives, as they’re used when you
8575normally protect a directory with mod_auth. The Anonymous directive con-
8576trols what usernames should be considered an anonymous user—in this
8577case, we’ve got orko and bender, but we could just have easily chosen mun-
8578dania such as anonymous, anon, or guest.
8579The Anonymous_Authoritative controls whether we want to pass unautho-
8580rized usernames and passwords off to another authentication scheme for
8581processing. If we say on, then anonymity is king, either the visitor logs in
8582with orko or bender or they’re not allowed access.
8583On the other hand, if we say off, then we can include the full functionality of
8584mod_auth—authentication via passwords, groups, and so forth. Take a look
8585at the configuration below. If the user does not log in with heenie or retro-
8586girl, then the username is passed off to the AuthUserFile, where it’s also
8587checked against for those accounts. If they don’t exist in that file either, then
8588the user is denied:
8589AuthName "anonymous/your email address"
8590AuthUserFile /Library/WebServer/.htpasswd
8591AuthType Basic
8592Require valid-user
8593Anonymous heenie retrogirl
8594Anonymous_Authoritative off
8595As is typical, you can be as simplistic or as complicated as you need. The fol-
8596lowing configuration will allow any user to get a directory listing (allowing
8597them to see what you have available). Any user listed in the AuthUserFile
8598can get access to all .jpg files, as well as any anonymous user logging in with
8599the mrs_decepticon or spiderj usernames, assuming they enter a valid email
8600address (one with a “@” and “.” character—Apache doesn’t check that the
8601hostname exists). Finally, only the eustace user can download .mp3 files:
8602AuthType basic
8603AuthName "anonymous/your email address"
8604AuthUserFile /Library/WebServer/.htpasswd
8605Anonymous mrs_decepticon spiderj
8606Anonymous_Authoritative off
8607Anonymous_VerifyEmail on
8608Rotate and compress Apache Server Logs #92
8609Information Servers | 193
8610HACK
8611<Files *.jpg>
8612Require valid-user
8613</Files>
8614<Files *.mp3>
8615Require user eustace
8616<Files>
8617You can, of course, get even more convoluted, restricting by IP address or
8618hostname, environment variables, and so forth. Just make sure you com-
8619ment your craziness—it’s very easy to get confused about who has access to
8620what.
8621Unfortunately, you can’t duplicate the other side of FTP and that’s upload-
8622ing. Neither can any of the modules that are shipped with Apache by
8623default. To do so, you’d have to use one of the many CGI scripts that accept
8624file uploads, or else craft your own.
8625H A C K
8626#92
8627Rotate and compress Apache Server Logs Hack #92
8628Use this tiny Perl script to compress all of your Apache logs automatically,
8629even when you add more to your site
8630Here’s a handy script that will rotate and gzip your Apache logs for you
8631automatically. It reads your httpd.conf, descends all Included configuration
8632files, and makes a note of each unique Log file. It then renames each to a file
8633with a datestamp, restarts Apache, then gzip compresses the datestamped
8634log files. As a bonus, it chgrps the compressed logs to whatever group you
8635have set in $gid, and sets permissions up for that group to be able to read or
8636write to them. This makes it simple for other (non-root) processes to come
8637along later and post-process the compressed logs.
8638Run this nightly (or weekly, depending on your traffic) in cron to collect a
8639regular repository of compressed logs, and keep your live log files down to a
8640manageable size.
8641Listing: logflume.pl
8642#!/usr/bin/perl -w
8643#
8644# logflume.pl
8645#
8646# Roll over and compress Apache log files, following Includes within the
8647# httpd.conf (and all other configuration files).
8648#
8649#
8650use strict;
8651$|++;
8652194 | Information Servers
8653#93 Generating an SSL cert and Certificate Signing Request
8654HACK
8655my $server_root = "/usr/local/apache";
8656my $conf = "$server_root/conf/httpd.conf";
8657my $gid = "wwwadmin";
8658my (%logs, %included, @files, @gzip);
8659my $date = `date +%Y%m%d`; chomp $date;
8660push @files, $conf;
8661for $conf (@files) {
8662open(CONF, "<$conf") || die "Cannot open config file $conf: $!\n";
8663while (<CONF>) {
8664chomp;
8665next if /^(\s+)?#/;
8666if (/(Transfer|Custom|Error)Log\s+(\S+)(\s+)/i) {
8667$logs{$2}++;
8668} elsif (/^(ResourceConfig|Include)\s+(\S+)/i) {
8669if(!$included{$2}) {
8670push @files, $2;
8671$included{$2}++;
8672}
8673}
8674}
8675close CONF;
8676}
8677for my $logfile (sort keys %logs) {
8678$logfile = "$server_root/$logfile" unless ($logfile =~ m|^/|);
8679rename($logfile, "$logfile.$date");
8680push(@gzip, "$logfile.$date");
8681}
8682system("$server_root/bin/apachectl restart");
8683for my $logfile (@gzip) {
8684system("gzip $logfile");
8685system("chgrp $gid $logfile.gz");
8686system("chmod 664 $logfile.gz");
8687}
8688H A C K
8689#93
8690Generating an SSL cert and Certificate Signing
8691Request Hack #93
8692Make an SSL key, CSR, and cert for use with Apache
8693In order to use Apache with mod_ssl or Apache-ssl, you’ll need a certificate
8694signed by a trusted Certificate Authority. In this example, we’ll assume that
8695Generating an SSL cert and Certificate Signing Request #93
8696Information Servers | 195
8697HACK
8698you’re generating a cert to be used at https://propaganda.discordia.eris/. To
8699generate a key with OpenSSL:
8700hagbard@fnord:~/certs$openssl genrsa 512/1024 \
8701> propaganda.discordia.eris.key
8702warning, not much extra random data, consider using the -rand option
8703Generating RSA private key, 512 bit long modulus
8704..++++++++++++
8705...++++++++++++
8706e is 65537 (0x10001)
8707This just makes the private key, not the cert. If you’d like to protect this key
8708with a passphrase, use the -des3 option on the command line:
8709hagbard@fnord:~/certs$openssl genrsa -des3 512/1024 \
8710> propaganda.discordia.eris.key
8711warning, not much extra random data, consider using the -rand option
8712Generating RSA private key, 512 bit long modulus
8713.......++++++++++++
8714.....++++++++++++
8715e is 65537 (0x10001)
8716Enter PEM pass phrase:
8717Verifying password - Enter PEM pass phrase:
8718But be warned: you’ll need to enter this phrase every time you restart
8719Apache, which can be inconvenient when performing regular maintenance
8720(such as rotating http logs). Weigh the inconvenience against the potential
8721damage done if some miscreant should acquire this key. If you lose the pass-
8722phrase, it is essentially unrecoverable, so keep it safe!
8723Next you’ll need to generate the Certificate Signing Request, to submit to a
8724trusted CA (such as Thawte/VeriSign) for signing. Type in everything appear-
8725ing in boldface, substituting your own information where appropriate:
8726hagbard@fnord:~/certs$ openssl req -new -key propaganda.discordia.eris.key \
8727> propaganda.discordia.eris.csr
8728Using configuration from /usr/local/ssl/openssl.cnf
8729You are about to be asked to enter information that will be incorporated
8730into your certificate request.
8731What you are about to enter is what is called a Distinguished Name or a DN.
8732There are quite a few fields but you can leave some blank
8733For some fields there will be a default value,
8734If you enter '.', the field will be left blank.
8735Country Name (2 letter code) [AU]:US
8736State or Province Name (full name) [Some-State]:Texas
8737Locality Name (eg, city) []:Mad Dog
8738Organization Name (eg, company) [Internet Widgits Pty Ltd]:Discordia, Inc.
8739Organizational Unit Name (eg, section) []:Operations
8740Common Name (eg, YOUR name) []:propaganda.discordia.eris
8741Email Address []:norton@discordia.eris
8742196 | Information Servers
8743#94 Creating Your Own CA
8744HACK
8745Please enter the following 'extra' attributes
8746to be sent with your certificate request
8747A challenge password []:
8748An optional company name []:
8749Finally, you’re ready to create the cert itself. If you’re going to use a well-
8750known CA, then you’ll need to send the .csr file to them for signing. In the
8751meantime, you can self-sign your cert and use it while you’re waiting for the
8752CSR to be processed:
8753hagbard@fnord:~/certs$openssl req -x509 \
8754-key propaganda.discordia.eris.key \
8755-in propaganda.discordia.eris.csr \
8756> propaganda.discordia.eris.crt
8757Using configuration from /usr/local/ssl/openssl.cnf
8758hagbard@fnord:~/certs$
8759If you intend to build your own Certificate Authority, then use your own CA
8760key to sign your cert.
8761See also:
8762• “Creating Your Own CA” [Hack #94]
8763• The OpenSSL home at http://www.openssl.org/
8764H A C K
8765#94
8766Creating Your Own CA Hack #94
8767Become your own Certificate Authority, and sign your own (or others’) SSL
8768certs
8769Well-known Certificate Authorities (such as Thawte/VeriSign) exist to serve
8770as an authoritative, trusted third-party for authentication. They are in the
8771business of signing SSL certificates that are used on sites that deal with sen-
8772sitive information (such as account numbers or passwords). If a site’s SSL
8773certificate is signed by a trusted authority, then presumably it is possible to
8774verify the identity of a server supplying that cert’s credentials. In order to
8775receive a certificate “blessed” by a well known CA, you have to prove to
8776them beyond a shadow of doubt that not only are you who you claim to be,
8777but that you have the right to use the certificate in the way you intend. For
8778example, I may be able to prove to a CA that I am really Rob Flickenger, but
8779they probably won’t issue me a signed cert for Microsoft Corporation, as I
8780have no rights to use that name. Yes, they probably wouldn’t do that. Not
8781again.
8782OpenSSL is perfectly capable of generating everything you need to run your
8783own Certificate Authority. The CA.pl utility makes the process very simple.
8784Creating Your Own CA #94
8785Information Servers | 197
8786HACK
8787In these examples, you’ll need to type anything in boldface, and enter pass-
8788words wherever appropriate (that don’t echo to the screen.) To establish
8789your new Certificate Authority:
8790hagbard@fnord:~/certs$/usr/local/ssl/misc/CA.pl -newca
8791CA certificate filename (or enter to create)
8792Making CA certificate ...
8793Using configuration from /usr/local/ssl/openssl.cnf
8794Generating a 1024 bit RSA private key
8795...............++++++
8796......................................++++++
8797writing new private key to './demoCA/private/cakey.pem'
8798Enter PEM pass phrase:
8799Verifying password - Enter PEM pass phrase:
8800-----
8801You are about to be asked to enter information that will be incorporated
8802into your certificate request.
8803What you are about to enter is what is called a Distinguished Name or a DN.
8804There are quite a few fields but you can leave some blank
8805For some fields there will be a default value,
8806If you enter '.', the field will be left blank.
8807-----
8808Country Name (2 letter code) [AU]:US
8809State or Province Name (full name) [Some-State]:California
8810Locality Name (eg, city) []:Sebastopol
8811Organization Name (eg, company) [Internet Widgits Pty Ltd]:Illuminatus
8812Enterprises, Ltd
8813Organizational Unit Name (eg, section) []:Administration
8814Common Name (eg, YOUR name) []:Hagbard Celine
8815Email Address []:hagbardceline1723@yahoo.com
8816Congratulations. You’re the proud owner of your very own Certificate
8817Authority. Take a look around:
8818hagbard@fnord:~/certs$ls
8819demoCA/
8820hagbard@fnord:~/certs$cd demoCA/
8821hagbard@fnord:~/certs/demoCA$ls -l
8822total 24
8823-rw-r--r-- 1 rob users 1407 Sep 8 14:12 cacert.pem
8824drwxr-xr-x 2 rob users 4096 Sep 8 14:12 certs/
8825drwxr-xr-x 2 rob users 4096 Sep 8 14:12 crl/
8826-rw-r--r-- 1 rob users 0 Sep 8 14:12 index.txt
8827drwxr-xr-x 2 rob users 4096 Sep 8 14:12 newcerts/
8828drwxr-xr-x 2 rob users 4096 Sep 8 14:12 private/
8829-rw-r--r-- 1 rob users 3 Sep 8 14:12 serial
8830The public key for your new Certificate Authority is contained in cacert.
8831pem, and the private key is in private/cakey.pem. You can now use this pri-
8832vate key to sign other SSL certs.
8833198 | Information Servers
8834#94 Creating Your Own CA
8835HACK
8836To use your CA’s authority to sign SSL certs, you’ll need to make a new cert
8837that a web server (such as Apache) can use. First, generate a private key and
8838certificate request, as shown in “Generating an SSL cert and Certificate Signing
8839Request” [Hack #93] . Now you can sign the new request with your own CA’s key:
8840hagbard@fnord:~/certs$openssl ca -policy policy_anything \
8841-out propaganda.discordia.eris.crt \
8842-infiles propaganda.discordia.eris.csr
8843Using configuration from /usr/local/ssl/openssl.cnf
8844Enter PEM pass phrase:
8845Check that the request matches the signature
8846Signature ok
8847The Subjects Distinguished Name is as follows
8848countryName :PRINTABLE:'US'
8849stateOrProvinceName :PRINTABLE:'Texas'
8850localityName :PRINTABLE:'Mad Dog'
8851organizationName :PRINTABLE:'Discordia, Inc.'
8852organizationalUnitName:PRINTABLE:'Operations'
8853commonName :PRINTABLE:'propaganda.discordia.eris'
8854emailAddress :IA5STRING:'hail@discordia.eris'
8855Certificate is to be certified until Sep 8 22:49:26 2003 GMT (365 days)
8856Sign the certificate? [y/n]:y
88571 out of 1 certificate requests certified, commit? [y/n]y
8858Write out database with 1 new entries
8859Data Base Updated
8860Now to use the .crt and .key with Apache + mod_ssl (or Apache-ssl), install
8861them as you normally would (perhaps with lines like these):
8862SSLCertificateFile /usr/local/apache/conf/ssl.crt/propaganda.discordia.eris.
8863crt
8864SSLCertificateKeyFile /usr/local/apache/conf/ssl.key/propaganda.discordia.
8865eris.key
8866This is all lots of fun, but what happens when a client actually connects to
8867https://propaganda.discordia.eris/? Won’t the browser throw an error about
8868not recognizing the Certificate Authority that signed the SSL cert? Natu-
8869rally. Unless, of course, you’ve installed your CA’s public key to the client
8870browser ahead of time. See the next hack if you’d like to do that.
8871See also:
8872• man CA.pl
8873• The OpenSSL home at http://www.openssl.org/
8874• http://www.cert.org/advisories/CA-2001-04.html
8875Disclaimer: no, I honestly had nothing to do with the Microsoft Corpora-
8876tion cert snafu. But it does illustrate one of the fundamental facts of life
8877online: it’s difficult to know who to trust.
8878D ownload from Wow! eBook <www.wowebook.com>
8879Distributing Your CA to Client Browsers #95
8880Information Servers | 199
8881HACK
8882H A C K
8883#95
8884Distributing Your CA to Client Browsers Hack #95
8885Installing your shiny new CA cert to client browsers is just a click away
8886There are two possible formats that browsers will accept for new certificate
8887authority certs: pem and der. Early versions of Netscape expected pem for-
8888mat, but recent versions will accept either. Internet Explorer is just the
8889opposite (early IE would only accept der format, but recent versions will
8890take both). Other browsers will generally accept either format. You can gen-
8891erate a der from your existing pem with a single openssl command:
8892hagbard@fnord:~/certs$openssl x509 -in demoCA/cacert.pem \
8893-outform DER -out cacert.der
8894Also, add the following line to your conf/mime.types file in your Apache
8895installation:
8896application/x-x509-ca-cert der pem crt
8897Now restart Apache for the change to take effect. You should now be able to
8898place both the cacert.der and demoCA/cacert.pem files anywhere on your web
8899server, and have clients install the new cert by simply clicking on either link.
8900You will get a dialog box in your browser when downloading the new certif-
8901icate authority, asking if you’d like to continue. Accept the certificate, and
8902that’s all there is to it. Now SSL certs that are signed by your CA will be
8903accepted without warning the user, as in Figure 8-1.
8904Keep in mind that Certificate Authorities aren’t to be taken lightly. If you
8905accept a new CA in your browser, you had better trust it completely—a mis-
8906chievous CA manager could sign all sorts of certs that you should never
8907Figure 8-1. Click OK to accept the new Certificate Authority or View to read the fine
8908print
8909200 | Information Servers
8910#95 Distributing Your CA to Client Browsers
8911HACK
8912trust, but your browser would never complain (since you claimed to trust
8913the CA when you imported it). Be very careful about who you extend your
8914trust to when using SSL enabled browsers. It’s worth looking around in the
8915CA cache that ships with your browser to see exactly who you trust by
8916default.
8917For example, did you know that AOL/Time Warner has its own CA? How
8918about GTE? Or VISA? CA certs for all of these entities (and many others)
8919ship with Netscape 7.0 for Linux and are all trusted authorities for web sites,
8920email, and application add-ons by default. Keep this in mind when brows-
8921ing to SSL-enabled sites: if any one of the default authorities have signed
8922online content, then your browser will trust it without requiring operator
8923acknowledgment, as in Figure 8-2.
8924If you value your browser’s security (and, by extension, the security of your
8925client machine), then make it a point to review your trusted CA relationships.
8926See also:
8927• The OpenSSL FAQ at http://www.openssl.org/support/faq.cgi
8928• “Generating an SSL cert and Certificate Signing Request” [Hack #93]
8929• “Creating Your Own CA” [Hack #94]
8930Figure 8-2. It’s a good idea to review exactly who your browser considers trustworthy.
8931Serving multiple sites with the same DocumentRoot #96
8932Information Servers | 201
8933HACK
8934H A C K
8935#96
8936Serving multiple sites with the same
8937DocumentRoot Hack #96
8938Through creative use of mod_rewrite, several sites can share a
8939DocumentRoot and yet still appear to be independent sites
8940Occasionally, it is useful for a couple of sites to share the same DocumentRoot
8941but have unique front pages. Perhaps you want to share a graphics or javas-
8942cript directory, but for some reason can’t be bothered to set up an appropriate
8943Alias entry. Or maybe you need to be able to share most of a directory struc-
8944ture, but don’t want to FollowSymLinks. Whatever your reason for needing a
8945common DocumentRoot, mod_rewrite will provide a solution.
8946Let’s assume that you are running a site called www.fruit.yum, and would
8947like to serve information about apples and oranges. You’ve already had
8948much success with http://www.fruit.yum/apples and http://www.fruit.yum/
8949oranges, but you are suddenly able to obtain the much coveted apple.yum
8950and orange.yum domain names. You’d like to keep the existing www.fruit.
8951yum directory structure for users who browse to directories under your new
8952domain names (for example, to http://www.apple.yum/order or http://www.
8953orange.yum/faq), but would like to deliver a custom page to users who sim-
8954ply browse to the front page (e.g., http://www.apple.yum/).
8955This is no problem with mod_rewrite. Let’s assume that you already have a
8956VirtualHost entry for www.fruit.yum like this:
8957<VirtualHost *>
8958ServerName www.fruit.yum
8959ServerAdmin webmaster@fruit.yum
8960DocumentRoot /home/www/htdocs
8961CustomLog /home/www/logs/access_log combined
8962ErrorLog /home/www/logs/error_log
8963</VirtualHost>
8964Add a couple of new VirtualHost entries for your new domains but with
8965some additional RewriteRule lines:
8966<VirtualHost *>
8967ServerName www.apples.yum
8968ServerAdmin webmaster@fruit.yum
8969DocumentRoot /home/www/htdocs
8970CustomLog /home/www/logs/access_log combined
8971ErrorLog /home/www/logs/error_log
8972RewriteEngine On
8973202 | Information Servers
8974#96 Serving multiple sites with the same DocumentRoot
8975HACK
8976RewriteRule ^/$ /home/www/htdocs/apples/index.html
8977RewriteRule ^/index.html$ /home/www/htdocs/apples/index.html
8978</VirtualHost>
8979<VirtualHost *>
8980ServerName www.oranges.yum
8981ServerAdmin webmaster@fruit.yum
8982DocumentRoot /home/www/htdocs
8983CustomLog /home/www/logs/access_log combined
8984ErrorLog /home/www/logs/error_log
8985RewriteEngine On
8986RewriteRule ^/$ /home/www/htdocs/oranges/index.html
8987RewriteRule ^/index.html$ /home/www/htdocs/oranges/index.html
8988</VirtualHost>
8989As these are handled as internal rewrites, the URL line will not change on
8990the client’s browser. As far as they are concerned, browsing to http://www.
8991oranges.yum/ appears to display an independent site. Browsing directly to
8992http://www.fruit.yum/oranges/ is functionally equivalent, as long as relative
8993links (or appropriate Alias lines) are used in all of your html. If you’d like a
8994more general solution that will work for as many www.*.yum addresses as
8995you care to register, you might try something like this:
8996<VirtualHost *>
8997ServerName www.fruit.yum
8998ServerAlias www.*.yum
8999ServerAdmin webmaster@fruit.yum
9000DocumentRoot /home/www/htdocs
9001CustomLog /home/www/logs/access_log combined
9002ErrorLog /home/www/logs/error_log
9003RewriteCond %{HTTP_HOST} www.(.*).yum
9004RewriteCond %{HTTP_HOST} !www.fruit.yum
9005RewriteRule (.*) $1 [E=SITE:%1]
9006RewriteCond %{REQUEST_URI} ^/index.html$
9007RewriteCond /home/www/htdocs/%{ENV:SITE}/index.html -f
9008RewriteRule .* /home/www/htdocs/%{ENV:SITE}/index.html
9009RewriteCond %{REQUEST_URI} ^/$
9010RewriteCond /home/www/htdocs/%{ENV:SITE}/index.html -f
9011RewriteRule .* /home/www/htdocs/%{ENV:SITE}/index.html
9012</VirtualHost>
9013Delivering Content Based on the Query String Using mod_rewrite #97
9014Information Servers | 203
9015HACK
9016This will strip off the middle part of the requested domain name (the bit
9017between www. and .yum) and check to see if an index.html exists in the
9018directory of that name under the DocumentRoot for the site. If so, and if the
9019request was for / or /index.html, then it performs an internal rewrite to it.
9020Working with mod_rewrite can be confusing. One tool that helps greatly
9021with debugging rewrite problems is the RewriteLog functionality. If you’re
9022stuck on figuring out a RewriteRule, try a couple of lines like this in your
9023VirtualHost entry:
9024RewriteLog /tmp/rewrite.log
9025RewriteLogLevel 9
9026This will turn on full debugging to the log you specify. It is handy to watch it
9027with a tail -f /tmp/rewrite.log or less -f /tmp/rewrite.log as you
9028request various URLs in a browser. Don’t run with this enabled for very long
9029on a production server, as all rewrite actions will be logged (and conse-
9030quently make your Apache run slower).
9031H A C K
9032#97
9033Delivering Content Based on the Query String
9034Using mod_rewrite Hack #97
9035Control content delivery based on a URL’s query string without a CGI script
9036It can be useful to use the query string of a URL line (that is, anything after
9037the first ?) to direct what content Apache will serve. While the query string
9038is normally used by CGI applications to read and write program variables,
9039mod_rewrite can use it without the need for an external script.
9040For example, suppose you have a publication system that splits large arti-
9041cles into multiple pages. The content is normally delivered by a script the
9042takes the page= query string and delivers the requested page number. If you
9043can save a local copy of each page to the filesystem, then you can use a
9044RewriteRule to deliver the cached copy, and bypass the overhead of running
9045the script on every hit.
9046Let’s suppose we’re trying to serve the second page of an article at http://
9047mysite.com/news/article.html?page=2. Install the following rules:
9048RewriteCond %{QUERY_STRING} page=([0123456789]+)
9049RewriteCond /home/www/htdocs/%{REQUEST_URI}.%1 -f
9050RewriteRule .* /home/www/htdocs/%{REQUEST_URI}.%1 [L]
9051If /home/www/htdocs/news/article.html.2 exists, then it will be served as an
9052internal redirect and exit before the publication script is run. If the file
9053doesn’t exist, then processing will fall through the last rule, and will be
9054picked up by the normal publication delivery method (presumably by run-
9055ning a script specified by a ScriptAlias called /news). This will work for any
9056number of pages that end in an integer.
9057204 | Information Servers
9058#98 Using mod_proxy on Apache for Speed
9059HACK
9060Likewise, the absence of a query string might be meaningful. Suppose your
9061content delivery script will manipulate content depending on a variety of
9062variables, but if no variables are specified, it returns predictable contend
9063(say, the default view of the first page of an article.) This is another case
9064where a judicious RewriteRule can save you the unnecessary computing
9065overhead of dynamic rendering:
9066RewriteCond %{QUERY_STRING} =""
9067RewriteCond /home/www/static/%{REQUEST_URI} -f
9068RewriteRule .* /home/www/static/%{REQUEST_URI} [L]
9069Here we assume that if a query string exists, then it must necessarily involve
9070dynamic content. Otherwise, check first to see if a local copy of the
9071requested content exists (under /home/www/static/) and if so, serve it. If the
9072local copy doesn’t exist, then processing falls through to the next set of
9073directives (which most likely runs the dynamic content generator). As long
9074as an external process manages the expiration of old content from the cache
9075(via cron or a dedicated daemon that watches for content changes), then this
9076caching will be completely seamless from the user’s point of view. To expire
9077old content, simply delete the cached version, and future hits will be gener-
9078ated dynamically.
9079You can direct Apache to take any action you like based on the query string.
9080The power of mod_rewrite is really limited only by your imagination (and
9081the strength of your regex-fu).
9082See also:
9083• “Distributing Load with Apache RewriteMap” [Hack #99]
9084• “Ultrahosting: Mass Web Site Hosting with Wildcards, Proxy, and
9085Rewrite” [Hack #100]
9086• Apache’s mod_rewrite guide can be found at http://httpd.apache.org/
9087docs/mod/mod_rewrite.html
9088H A C K
9089#98
9090Using mod_proxy on Apache for Speed Hack #98
9091Offload complex dynamic requests to another apache (or another machine
9092entirely)
9093A tremendous amount of effort has gone into optimizing Apache to make it
9094rip files from the filesystem and throw them at incoming http requests as
9095quickly as possible. Unfortunately, sites that solely serve content that sits in
9096files within the filesystem are not typically very popular sites. The huge
9097Using mod_proxy on Apache for Speed #98
9098Information Servers | 205
9099HACK
9100demand for interactive content has given birth to many projects that are spe-
9101cifically designed to give the end user a highly customizable, dynamic
9102browsing experience.
9103Unfortunately, as the interactivity of a web site increases, performance typi-
9104cally goes out the window. Interactive content demands a programming lan-
9105guage, and interpreting arbitrary code is very expensive (compared to
9106serving static files). One classic “worst case” example is a Perl CGI that
9107makes database requests and sends email by spawning an external sendmail.
9108Compared to the time it takes to serve a static file, a single CGI request of
9109that sort takes an eternity (and many requests therefore take many eterni-
9110ties, bringing your entire server to a crawl).
9111Apache modules such as mod_perl and mod_php (along with a large helping
9112of programming sanity) can significantly alleviate the pain of spawning
9113external processes on every hit. But even with interpreters built into Apache
9114itself, the rate of serving dynamic requests rarely approaches that of a static
9115web server. One reason for this is that most hits suffer the performance pen-
9116alty of supporting embedded languages, regardless of whether the request is
9117dynamic or not.
9118Consider the case of a large mod_perl installation. After having run for a few
9119minutes, an individual httpd process may consume 40 or 50MB of RAM as
9120Perl modules are cached into memory, for speedy execution. Now imagine
9121that a request for a single-pixel transparent gif is made (as this is one trick
9122commonly used by web designers to make arbitrary-sized blank spaces on a
9123web page, exact to the pixel). In the absolute worst case, suppose that all
9124available httpd processes are busy, so Apache decides to spawn another. The
9125server must now invoke the behemoth of mod_perl, creating a child that
9126takes up 50MB of RAM and a huge amount of processing time, just to serve
9127a single static file of only a few bytes. Clearly, if we could only invoke the
9128shaggy shoggoth of mod_perl when absolutely necessary, we could reap
9129huge benefits every time someone requested a graphic (or other file that is
9130guaranteed to require no dynamic processing).
9131You might consider running two installations of Apache: one with every
9132module, bell, whistle, and vibra-slap that your dynamic content manage-
9133ment system requires, and another that is as stripped-down and simple as
9134you can possibly make it. If all requests come into the lightweight Apache,
9135then we can serve static content as normal, and proxy dynamic requests to
9136the dynamic content servers.
9137Suppose you’re running the slimmed-down Apache on standard port 80,
9138and the application server on port 8088. Using the following rewrite rule,
9139206 | Information Servers
9140#99 Distributing Load with Apache RewriteMap
9141HACK
9142you can proxy requests for anything but graphics and archives to the appli-
9143cation server. Put this at the bottom of the VirtualHost entry for your site
9144(on the server running on port 80):
9145RewriteEngine On
9146RewriteCond %{REQUEST_URI} !.*\.(jpg|gif|pdf|png|zip|tgz|gz)$
9147RewriteRule ^/(.*) http://%{HTTP_HOST}:8088/$1 [P]
9148That [P] at the end of the RewriteRule line means proxy. This will cause the
9149lightweight Apache to make a connection to the dynamic server running on
91508088 and make the request as if the client itself had made it. When it
9151receives the reply, it passes it along to the client as if it had served the
9152request itself. As far as the client is concerned, there is only one web server:
9153the really fast one running on port 80. And as long as the site definitions for
9154this site in both apache installations point to the same DocumentRoot, this
9155method will work just fine.
9156If your application uses cookies, you should also add a ProxyPassReverse
9157line just above the rewrite rules with something like this:
9158ProxyPassReverse / http://%{HTTP_HOST}:8088/
9159This tells Apache to translate http headers when proxying, so that cookies
9160get propagated from the application server to the client as they should.
9161If you have the hardware for it, you could help distribute the load by using
9162one machine for the proxy server and another for the application server. In
9163that case, try rules like this:
9164ProxyPassReverse / http://your.application.server.here/
9165RewriteEngine On
9166RewriteCond %{REQUEST_URI} !.*\.(jpg|gif|pdf|png|zip|tgz|gz)$
9167RewriteRule ^/(.*) http://your.application.server.here/$1 [P]
9168Of course, with multiple physical servers, you’ll have to keep your filesys-
9169tems in sync (see “Keeping Parts of Filesystems in sync with rsync” [Hack #41] ),
9170or else people will see conflicting pages every time your site gets updated. If
9171even two servers aren’t capable of serving all of your dynamic content, take a
9172look at “Distributing Load with Apache RewriteMap” [Hack #99] for one
9173method of spreading load across as many application servers as you like.
9174H A C K
9175#99
9176Distributing Load with Apache RewriteMap Hack #99
9177Scale to any number of web application servers with RewriteMap
9178As we saw in the previous hack, it is possible to transparently serve content
9179from any arbitrary web server using mod_proxy and mod_rewrite. Rather
9180than using an external redirect [R] , the proxy target [P] makes the content
9181D ownload from Wow! eBook <www.wowebook.com>
9182Distributing Load with Apache RewriteMap #99
9183Information Servers | 207
9184HACK
9185appear to come from the server from which the client originally requested
9186content, without changing the URL line in the client browser. As we saw
9187earlier, one application of this technique is to spread the load of serving
9188pages across two machines, making it possible to serve many more hits than
9189a single machine can handle.
9190But what if you need to serve even more traffic, to the point that a single
9191application server is no longer sufficient? We need a mechanism that will
9192allow the proxy servers to choose from an available pool of application serv-
9193ers, preferably in a way that we can direct more hits at more capable serv-
9194ers. The RewriteMap directive gives us this functionality.
9195Here’s an example of how to specify a RewriteMap:
9196RewriteMap server rnd:/usr/local/apache/conf/servers.map
9197The RewriteMap line just defines a map called server, physically bound to
9198the servers.map file in the conf/ directory. The rnd: makes this a randomized
9199plain text map.
9200The format of the servers.map file is very straightforward:
9201web www1|www2|www3|www4|www5
9202The left-hand side is an arbitrary variable name that we can use just about
9203anywhere in our configuration file, as in this RewriteRule:
9204RewriteRule ^/(.*) http://${server:web}.oreillynet.com/$1 [P]
9205The right-hand side of the lines of the servers.map file consists of strings to
9206be returned in random order, separated by pipes. In the case of the Rewrit-
9207eRule above, a random value (one of www1, www2, www3, www4, or
9208www5) will be chosen from the servers.map and will replace the ${server:
9209web} variable. This results in a proxy rule to http://www4.oreillynet.com/ (or
9210any of the other www* servers) with the original URI appended to it.
9211The only trouble with a truly random map is that all of the servers specified
9212in it will receive approximately the same amount of traffic over time. If
9213www1 is a quad 2.4GHz Xeon with 8GB RAM, and www2 is a 486SX/33
9214with 4MB RAM, you’ll probably want to direct more traffic at www1 than at
9215www2. To do this, simply specify the more capable servers multiple times in
9216the servers.map:
9217web www1|www1|www1|www1|www2|www3|www3|www4|www5|www5
9218In this example, www1 will receive 4 out of 10 hits, www2 will receive 1 out
9219of 10, www3 will receive 2, www4 will get 1, and www5 will pick up the
9220remaining 2. You can fine tune this in real time by watching top (or even tl, as
9221discussed in “Constant Load Average Display in the Titlebar” [Hack #59] ) on
9222every application server while manipulating the servers.map. The mtime of the
9223file is checked on every hit, and Apache will automatically reload the map
9224208 | Information Servers
9225#100 Ultrahosting: Mass Web Site Hosting with Wildcards, Proxy, and Rewrite
9226HACK
9227whenever it changes, without needing a restart. This means that an intelligent
9228systems monitoring tool can watch all of www* and rotate them out of the
9229map in real time whenever it finds a web server that’s having trouble. If care-
9230fully planned, this can mean web services with zero downtime, routing around
9231problem servers, all without ever having to restart Apache.
9232Working with multiple application servers can be tricky business. Obvi-
9233ously, you’ll need to keep your filesystems in sync, using NFS or possibly
9234rsync, as discussed in “Constant Load Average Display in the Titlebar” [Hack
9235#59] . But even the best planned server schemes can sometimes go awry. What
9236happens if you notice an Internal Server Error that goes away if you click
9237Reload? One of the application servers has developed a problem, and there’s
9238no way to tell which one is having trouble (without consulting the error log
9239on every machine). Here’s a handy rewrite rule that will let you select which
9240application server to steer a particular request:
9241RewriteCond %{QUERY_STRING} appserver=(.*)
9242RewriteRule ^/(.*) http://%1.oreillynet.com/$1 [P,L]
9243Now you can select a particular application server by adding a query string
9244to the URL in your browser, like this:
9245http://www.oreillynet.com/some/path/?appserver=www3
9246This will help you quickly track down the server with the problem. Com-
9247bined with front-end proxy servers, round robin DNS, and replicated data-
9248bases, this makes a very straightforward configuration that will scale to very
9249large installations.
9250See Also:
9251• Apache mod_rewrite documentation at http://httpd.apache.org/docs/mod/
9252mod_rewrite.html
9253• Proxy Server Hack
9254• “Using rsync over ssh” [Hack #38]
9255• “Distributing Server Load with Round-Robin DNS” [Hack #79]
9256• “Setting Up Replication in MySQL” [Hack #82]
9257H A C K
9258100
9259Ultrahosting: Mass Web Site Hosting with
9260Wildcards, Proxy, and Rewrite Hack #100
9261Support thousands of internal web servers without lifting a finger
9262Suppose you have a large private network hiding from the Internet behind a
9263NAT router. Your network layout looks something like Figure 8-3.
9264Ultrahosting: Mass Web Site Hosting with Wildcards, Proxy, and Rewrite #100
9265Information Servers | 209
9266HACK
9267You want to be able to allow anyone on your private network to set up their
9268own web server. But like all good network administrators, you are smart and
9269lazy and don’t want to fiddle with updating forwarding rules on your fire-
9270wall every time someone needs to make a change. Through the careful use of
9271named virtual hosts, mod_proxy, and mod_rewrite, you can reduce the
9272administrative overhead of your entire network to simple DNS updates.
9273Then there is little keeping you from delegating that responsibility to the
9274departments that wanted the web servers in the first place.
9275To start, you’ll need Apache running on your gateway machine, with mod_
9276rewrite and mod_proxy installed. You’ll also need a DNS server running
9277your own top-level internal domain (as discussed in “Running Your Own
9278Top-Level Domain” [Hack #80] ). We’ll assume that you own the Internet
9279domain shelbyville.com, and have the internal TLD of .springfield already set
9280up, serving your internal machines.
9281Add the following to the Apache configuration on your gateway machine:
9282Port 80
9283BindAddress *
9284NameVirtualHost *
9285<VirtualHost *>
9286ServerName mux.shelbyville.com
9287ServerAlias *.shelbyville.com
9288RewriteEngine On
9289RewriteCond %{HTTP_HOST} (.*).shelbyville.com
9290RewriteRule ^/(.*) http://%1.springfield/$1 [P]
9291</VirtualHost>
9292Figure 8-3. Typical corporate lans use private addressing internally and at least one
9293Internet gateway providing Network Address Translation
9294IS Department
9295(wiggum.springfield)
929610.0.5.0/24
9297Corporate
9298(krabappel.springfield)
929910.0.6.0/24
9300Sales
9301(syzlak.springfield)
930210.0.7.0/24
9303Research
9304(frink.springfield)
930510.0.8.0/24
9306Facilities
9307(willy.springfield)
930810.0.9.0/24
9309Linux
9310gateway
9311gateway.springfield
931210.0.0.1
9313Internet
9314208.201.239.36
9315gateway.shelbyville.com
9316210 | Information Servers
9317#100 Ultrahosting: Mass Web Site Hosting with Wildcards, Proxy, and Rewrite
9318HACK
9319Briefly, this configuration translates to:
9320• Listen on any available IP address, on port 80
9321• Accept http requests for any host ending in .shelbyville.com
9322• Strip shelbyville.com from the http host that was requested
9323• Make a connection (via Proxy) to what’s left of the hostname with .spring-
9324field appended to it
9325• Post the original URI and return the results to the client
9326This works because hosts are specified by name in the http header under the
9327http 1.1 specification and aren’t tied to a particular IP address. Apache will
9328look up the contents of %1 (everything before .shelbyville.com) via the sys-
9329tem resolver and attempt to connect to it. Since the gateway is using a DNS
9330server that serves your .springfield TLD, it will proxy to the proper internal
9331host.
9332For example, suppose that the original request was for http://jimbo.shelbyville.
9333com/index.html. After the RewriteCond line, the %1 variable simply contains
9334jimbo. It then attempts to proxy to %1 (aka jimbo) with .springfield appended
9335to it. The result? A proxy request to the internal web server jimbo.springfield,
9336with the original URI passed along as if the gateway weren’t even there.
9337This is the simplest configuration, but it will break internal servers that
9338require cookies. To support cookies on servers residing on the internal net-
9339work, try something like this:
9340<VirtualHost *>
9341ServerName mux.shelbyville.com
9342ServerAlias *.shelbyville.com
9343RewriteEngine On
9344RewriteCond %{HTTP_HOST} (.*).shelbyville.com
9345RewriteRule (.*) $1 [E=WHERETO:%1.springfield]
9346ProxyPassReverse / http://%{ENV:WHERETO}/
9347RewriteRule ^/(.*) http://%{ENV:WHERETO}/$1 [P]
9348</VirtualHost>
9349This uses a “fake-out” RewriteRule that is only called to invoke the side effect
9350of setting the %{WHERETO} environment variable. This gets set to the original
9351requested http host with .shelbyville.com stripped off, but with .springfield
9352appended. We need to do that to be able to feed the amended hostname to
9353ProxyPassReverse.
9354Ultrahosting: Mass Web Site Hosting with Wildcards, Proxy, and Rewrite #100
9355Information Servers | 211
9356HACK
9357By manipulating the DNS configuration for .shelbyville.com and .springfield,
9358you can bring internal web servers up and down at will, without ever touch-
9359ing the Apache configuration on the gateway. Of course, to make the job
9360even easier, you could use wildcard DNS for .shelbyville.com:
9361*.shelbyville.com IN A 12.34.56.78
9362(Naturally, substituting the external IP address of the gateway for 12.34.56.
936378). Now, requests for anything.shelbyville.com will be fed to the gateway
9364and proxied, without ever changing the zone file again. That just leaves
9365internal DNS maintenance (for .springfield). The simplest way to dispense
9366with that responsibility is to divide it into multiple subdomains and defer to
9367other internal DNS servers. For example, you could put something like this
9368in your named.conf:
9369zone "wiggum.springfield" {
9370type slave;
9371file "wiggum.db";
9372masters { 10.42.5.6; };
9373};
9374zone "krabappel.springfield" {
9375type slave;
9376file "krabappel.db";
9377masters { 10.42.6.43; };
9378};
9379zone "syzlak.springfield" {
9380type slave;
9381file "syzlak.db";
9382masters { 10.42.7.2; };
9383};
9384and so on. Now each department can have their own master DNS server and
9385can add new hosts (and therefore new, Internet-ready web servers) without
9386even dropping you an email. They can get to their new servers from the
9387Internet by browsing to an address like http://ralph.wiggum.shelbyville.com/,
9388which translates to http://ralph.wiggum.springfield/ internally and is ulti-
9389mately looked up from the DNS server for the division that is responsible for
9390the wiggum subdomain. As far as the Internet users are concerned, the infor-
9391mation came directly from the gateway machine, and aren’t even aware that
9392.springfield even exists.
9393Now, just what will you do with all of the time that this saves you?
9394213
9395We’d like to hear your suggestions for improving our indexes. Send email to index@oreilly.com.
9396Index
9397` (backtick), 14, 20
9398^D, 21
9399. (dot) and CVS tags, 56
9400/ (forward slash), 66
9401| (pipes), efficiency of, 15
9402? (question mark) and top, 119
9403; (semicolon) and Perl one-liners, 151
9404~ expansion facility, 21
9405Numbers
94061024th cylinder limit, kernel install, 73
9407A
9408access
9409ACL (access control list), local
9410host, 161
9411hosts, assigning trust-levels to, 89
9412lost root password, 6
9413ad referral tracking, 188–191
9414advanced routing tools, 98
9415anon_auth_module, 191
9416Apache, 157, 182–211
9417ad referral tracking, 188–191
9418anon_auth module, enabling, 191
9419Apache Toolbox script, 182–185
9420installation, 183–185
9421supported applications, 182
9422apachectl, 188
9423certificate generation and signing
9424requests, 194–196
9425certificates, distribution to client
9426browsers, 199–200
9427configuration files, automatic
9428updates across a network, 28
9429ftp servers, mimicking, 191–193
9430controlling access, 191
9431httpd.conf, 187
9432configuring display of full
9433filenames, 185
9434IfDefine, configuration changes
9435using, 186–188
9436IfModule directive, 186, 187
9437mass web site hosting, 208–211
9438mod_proxy, speeding content
9439delivery using, 204–206
9440mod_rewrite, 201–204
9441multiple installations, increasing
9442efficiency using, 205
9443RewriteMap, load distribution
9444using, 206–208
9445server logs, rotation and
9446compression, 193
9447serving cookies, 206
9448sites with a shared
9449DocumentRoot, 201–203
9450ssl keys and certificates, using, 198
9451URL query strings, content delivery
9452using, 203
9453archiving with pax, 67–72
9454restoring, 68
9455and renaming, 69
9456at, disabling, 23
9457automount, 4
9458214 | Index
9459B
9460backtick (`), 14, 20
9461backups, 64–87
9462of the boot sector, 72
9463CDR/CDRWs, 84–87
9464over a network, 65–67
9465rsync, 66
9466scp command, 65
9467tar, using over ssh, 65
9468with pax, 67–72
9469incremental backups, 71
9470restoring compressed archives, 68
9471snapshot-style, 79–84
9472balance-push.sh script, 77
9473bash
9474~ expansion facility, 21
9475clearing history, 22
9476files, breaking into chunks, 153
9477login shells, csutomizing, 22
9478manpage, 20
9479shell environment,
9480customizing, 20–22
9481tab completion, 15
9482.bash_logout, 22
9483.bashrc, 22
9484network configuration, 66
9485binary files
9486chunking, 153–155
9487reassembling, 154
9488BIND, 157, 158–169
9489caching DNS with local domain
9490authority, setup, 165
9491running in a chroot jail (in V.
94929), 158–160
9493server load distribution (in V. 9), 167
9494views (in V. 9), 160–165
9495match-clients substatement, 161
9496named.conf, 161
9497named.conf, primary master, 164
9498named.conf, slave name
9499servers, 163
9500syntax, 160
9501zones, defining, 162
9502BIOS, 1024th cylinder boot sector
9503limit, 73
9504blacklists, 89–91
9505boot messages, viewing, 40
9506boot parameters, 7–9
9507boot sector, backing up, 72
9508boot time configuration, 2, 3–10
95091024th cylinder limit, 73
9510removing superfluous services, 3
9511unnecessary drivers,
9512eliminating, 40–42
9513C
9514cacert.pem, 197
9515cakey.pem, 197
9516CA.pl utility, 197
9517CDPATH variable, 21
9518CDR/CDRWs, 84
9519CD-ROMs, 85–87
9520Certificate Authorities, 196
9521certificates, acquiring from, 194
9522creating your own, 196–198
9523keys, file locations, 197
9524public keys, distributing to
9525clients, 199–200
9526chargen, 5
9527chattr, 18
9528chroot jail, 158–160
9529ci command (RCS), 48
9530cmdline files, viewing, 33
9531co command (RCS), 48
9532CodeRed virus, filtering, 93
9533command line, 2, 10–46
9534complex commands, creating, 12–15
9535watch, automating a repeated
9536command using, 114
9537compiles, parallel builds (multiprocessor
9538systems), 19
9539comsat, 5
9540Concurrent Versioning System (see CVS)
9541.config file, 41
9542configuration files, updating with
9543make, 27
9544console= (boot parameter), 8
9545Control D, 21
9546cp command
9547-a switch, 79
9548-l switch, 79
9549cpio
9550pax and, 67
9551crontab
9552automating snapshot-type
9553backups, 80
9554setuid and, 23
9555Index | 215
9556CVS (Concurrent Versioning
9557System), 52–63
9558anonymous repositories, creating, 62
9559branching development, 59
9560changing CVS servers, 153
9561commands, 55–59
9562config trees, 52
9563cvs-makerepos (Debian), 53
9564default editor, setting, 57
9565directories
9566removing, 59
9567structure, 55
9568documenting changes, 57
9569environment variables for, 54
9570files
9571merging, 58
9572permissions, setting, 61
9573watching and locking, 60
9574modules
9575checking out, 54
9576importing new, 53
9577pserver access method, 62
9578release tags, 53
9579repository, creating, 53
9580security, 60
9581connecting to remote
9582machines, 54
9583developer machines, 61
9584SSH setup, 61
9585uses, 52
9586vendor tags, 53
9587$CVSEDITOR environment variable, 57
9588$CVSROOT environment variable, 54,
958955
9590CVSROOT/readers file, 63
9591$CVS_RSH environment variable,
9592setting to ssh, 54
9593$CVSUMASK environment variable, 61
9594D
9595daemons
9596init, creating with, 9–10
9597databases (see MySQL)
9598data.tar, 173, 174
9599dd, 154
9600denial-of-service attacks, preventing
9601with iptables, 92
9602der certificate format, 199
9603$DEST_IP, 92
9604directories
9605backup of home, 67
9606copying recursively, 70
9607CVS, 55
9608removal from, 59
9609identifying largest users of disk
9610space, 31
9611replication, 65
9612disk age analysis, 125–127
9613disk space, determining largest users, 30
9614diskage script, 125
9615dmesg command, 40
9616DNS (domain name service)
9617caching, with local domain
9618authority, 165
9619logging on syslog, 159
9620round-robin load distribution, 167
9621DocumentRoot, sharing among several
9622sites, 201–203
9623domain name queries, automated
9624pattern matching, 29
9625downloads from Apache servers,
9626enabling, 191–193
9627drivers
9628built-in vs loadable modules, 40
9629unnecessary, eliminating, 40–42
9630drives, (E)IDE, optimizing
9631performance, 43–46
9632E
9633echo, 5
9634email, securing with ssh, 146
9635environ files, viewing, 33
9636environment variables
9637bash, customizing in, 20–22
9638for CVS, 54
9639viewing with /proc, 31–34
9640errors
9641diagnosing (example), 12–15
9642ethereal and network debugging, 100
9643ext2 filesystem
9644immutable files, creating in, 17
9645ext3 filesystem
9646immutable files, creating in, 17
9647$EXT_IFACE, 92
9648extract-table script, 175
9649D ownload from Wow! eBook <www.wowebook.com>
9650216 | Index
9651F
9652fifo, logging of debug info to, 112
9653file command, 68
9654file descriptor, 11
9655file descriptors
96563-9, 11
9657closing, 12
9658filenames
9659special characters in, handling, 15
9660files
9661associated processes,
9662finding, 116–119
9663breaking into arbitrary
9664chunks, 153–155
9665reassembling, 154
9666immutability flag, setting, 17
9667immutable, creating in ext2/ext3
9668filesystems, 17
9669rsync and synchronization across a
9670network, 74–79
9671security, enhancing, 18
9672filesystem
9673monitoring changes to, 125–127
9674filesystems
9675ISOs (CD-ROM data format), 85–87
9676find used with xargs, 17
9677finger, 5
9678finger, security, 24
9679firewalls, 88–91
9680access, setting trust-levels, 89
9681forged packets, blocking, 91
9682ftp, 5
9683mimicking with Apache, 191–193
9684G
9685Generic Routing Encapsulation
9686(GRE), 99
9687go-ogle script, 123
9688gpm, security, 24
9689GRE (Generic Routing
9690Encapsulation), 99
9691grep, colorizing output, 155
9692grep -v ^# /etc/inetd.conf, 4
9693H
9694hacks, xv
9695hdparm, 43–46
9696online resources, 46
9697hdX= (boot parameter), 8
9698High Availability Linux project, send_
9699arp utility, 129
9700history command
9701clearing your history (bash), 22
9702home directory, backing up, 67
9703hosts, access by, setting, 89
9704httpd.conf
9705display of full filename in indexes,
9706configuration, 185
9707httpd.conf, diagnosing problems with
9708RCS, 49
9709httptop, 132–138
9710required Perl modules, 133
9711I
9712-i, immutable flag, chattr and lsattr, 18
9713IDE hard drives, performance
9714optimization, 43
9715identd, 5
9716IfDefine, 186–188
9717IfModule, 186, 187
9718inbound port scans, blocking, 93
9719incremental backups using pax, 71
9720inetd, 4
9721inetd.conf, 4
9722CVS and, 63
9723information servers, 157–211
9724(see also Apache)
9725(see also BIND)
9726(see also MySQL)
9727Init
9728creating persistent daemons
9729using, 9–10
9730init, 9
9731inittab, 9
9732Intel Physical Address Extension (PAE)
9733mode, 42
9734$INT_IFACE, 92
9735I/O support parameter, hdparm, 44
9736IP aliasing, 127–129
9737IP masquerading, 91
9738IP tunneling, 97
9739IPIP, 99
9740IPIP tunneling
9741GRE, compared to, 99
9742iplimit patch, 93
9743iptables
9744custom chains, using in, 96
9745ipchains, compared to, 92
9746Index | 217
9747kernel patches, features enabled
9748with, 93
9749port forwarding and, 94
9750syslog port, protecting, 114
9751TCP flags and, 93
9752TCP ports, forwarding, 94
9753transparent Squid proxy, 92
9754iptables command, 88–94, 96–101
9755firewalls, creating, 88–94
9756IP masquerading, 91
9757ISOs (CD-ROM data format), 85–87
9758K
9759kernel
9760.config file, 41
9761drivers, 40–42
9762modules, build considerations, 41
9763modules, built-in vs. loadable, 40
9764minimum boot functionality, 41
9765RAM support, configuring, 42
9766viewing running processes, 31–34
9767L
9768LILO, 1024th cylinder boot sector
9769limit, 73
9770Linux
9771operating systems, 1
9772optimization, 2
9773Linux servers, 1
9774(see also servers)
9775load average, script for displaying in
9776titlebar, 120
9777log files, 111
9778logflume.pl script, 193
9779logging
9780DNS on syslog, 159
9781login shells, 22
9782logins
9783ssh, using to facilitate, 139–142
9784turbo-mode, 141
9785lp, security, 24
9786lpd, 5
9787lsattr, 18
9788lsof, 116–119
9789-a switch, 118
9790-c switch, 117
9791devices, specifying, 117
9792-i switch, 118
9793-p switch, 117
9794M
9795mail spool, determining largest user, 30
9796make command, for file updates, 27
9797master, 172
9798mem= (boot parameter), 8
9799mince script, 154
9800mod_proxy, 204–206
9801mod_rewrite, 201–204
9802query strings, using for content
9803delivery, 203
9804RewriteLog, 203
9805mod_sql module, 178
9806monitoring, 111–138
9807httptop, 132–138
9808IP monitoring and
9809takeover, 127–129
9810MAC address updating, 128
9811load averages, displaying, 120
9812MySQL threads, 169
9813ntop, 129–131
9814scanning the networks
9815machines, 123–125
9816syslog and, 111–114
9817top, 119
9818watch, 114
9819web traffic, 132–138
9820mouse drivers, security of, 24
9821movein.sh, 149
9822mtop, 169
9823available commands, 170
9824e (explain), 171
9825k (kill), 171
9826user, creating for, 171
9827multcount support parameter,
9828hdparm, 44
9829multicast packet tunneling, 99
9830multiprocessor systems, improving
9831compile efficiency, 19
9832MySQL, 157, 169–182
9833dedicated servers,
9834optimizing, 180–182
9835glibc, 181
9836kernel changes, 181
9837maximum filehandles,
9838expanding, 181
9839MySQL build, 181
9840risks, 180
9841mod-sql module, 178
9842monitoring, 169
9843218 | Index
9844MySQL (continued)
9845proftpd and user
9846authentication, 178–180
9847replication, 172
9848typical problems, 174
9849write requests, 172
9850replication user, 172
9851server tuning, 175
9852indices, 176
9853my.cnf, 177
9854mysqlcheck -o database, 176
9855mysqld, renicing, 176
9856safe_mysql script, 176
9857server variables, checking, 177
9858servers, master and slaves, 172
9859table restoration from a dump, 175
9860users, creating, 179
9861N
9862named, 4, 158–160
9863changing ownership from root, 158
9864named.conf for views, 161
9865named.conf
9866for caching DNS with local domain
9867authority, 165
9868NAT (network address translation),
9869circumvention with
9870vtun, 101–106
9871netfilter patches, 93
9872netstat, 115
9873netstat -lp, 4
9874networking, 88–110
9875firewalls, 88–91
9876networks
9877admin tasks, automating, 208–211
9878and Linux servers, 1
9879scanning of machines, 123
9880NFS services, 4
9881NFSfilesystem, file-syncing with, 74–79
9882ngrep, 121–123
9883syntax, 122
9884Nimda virus, filtering, 93
9885n>&m operator, 11
9886n>&m operator (Bourne shell), 11
9887nmap, 123–125
9888-O switch, 124
9889nmbd, 4
9890nosmp (boot parameter), 8
9891ntop, 129–131
9892admin password, setting, 130
9893databases, initializing, 130
9894SSL, running on, 130
9895O
9896OpenSSH, 139
9897X11 forwarding, enabling, 145
9898OpenSSL, key generation, 195
9899OS fingerprinting code, 123
9900P
9901passwords
9902lost root, getting around, 6
9903ssh keys, compared to, 140
9904patch-o-matic patches, 93
9905pax (portable archive exchange), 67–72
9906archives, restoring, 68
9907and renaming, 69
9908-c (exception) switch, 72
9909cpio and, 67
9910directories, recursive copying, 70
9911home directory, backup using, 67
9912incremental backups, 71
9913-n switch, 72
9914-r switch, 68
9915risks of misuse, 70
9916skipping files on restore, 72
9917tar and, 67
9918-z switch, 68
9919pem certificate format, 199
9920Perl
9921-e switch, 151
9922global search and replace, 151
9923-i switch, 152
9924-p switch, 152
9925pgrep, 35
9926ping
9927setuid and, 25
9928pkill, 35
9929portmap, 4
9930ports
9931checking for Listen state, 115
9932restricting access, 89–91
9933ssh, forwarding over, 146–148
9934syslog port, protecting, 114
9935preferences, automating with
9936scripts, 149
9937Index | 219
9938privileges, minimizing security risks
9939of, 25
9940/proc, 31–34
9941evaluating system security, 34
9942processes for open sockets and files,
9943identifying, 116–119
9944procps, 34–36
9945.profile
9946ssh-agent, inclusion in, 144
9947proftpd and mysql
9948authentication, 178–180
9949prompts
9950colorcoding in bash shell, 20
9951setting default, 20
9952ps
9953security and, 34
9954ps ax, 4
9955PS1 variable, 20
9956psd patch, 93
9957pserver, 62
9958installing, 62
9959remote use, 63
9960Q
9961QUERY_STRING, using for ad
9962tracking, 188–191
9963R
9964r* commands, setuid and, 24
9965RAM, 42
996664 GB, support to, 42
9967960MB, support above, 42
9968RCS (revision control system), 48–52
9969checking out files, 48
9970-r switch, 50
9971diagnosing problems with, 49
9972initializing a file, 48
9973locking files, 48
9974logging changes, 49, 50–52
9975log entries, correspondence to
9976versions, 51
9977rcs2log, 50–52
9978-v switch, 51
9979rcsdiff, 49, 51
9980repository, creating, 48
9981unlocking files, 48
9982real-time netork stats,
9983monitoring, 129–131
9984referral-report.pl script, 190
9985release tags (CVS), 53
9986replication in MySQL, purpose, 172
9987restricted port policies, 89–91
9988revision control, 47–63
9989(see also CVS)
9990(see also RCS)
9991revision control system (see RCS)
9992RewriteMap, 206–208
9993servers.map file, 207
9994rexec, 5
9995rgc (Regex Colored Glasses) script, 155
9996rinetd, 94
9997rlogin, 5
9998ro (boot parameter), 8
9999root= (boot parameter), 7
10000root password
10001loss of, repairing, 6
10002rootservers.cache, 166
10003rpc.mountd, 4
10004rpc.nfsd, 4
10005rp_filter, 91
10006rsync, 66
10007--delete flag, 67
10008“downward spiral” condition, 75
10009exclude-from feature, 75
10010excludes lists, 82
10011file syncing with, 74–79
10012Perl script in cron, 76–79
10013snapshot-type backups and, 79
10014ssh, using over, 66
10015rw (boot parameter), 8
10016S
10017safe_mysqld script, 176
10018Samba daemons, 4
10019scp command, 65
10020scripting, 149–156
10021automatic configuration of
10022preferences on remote
10023systems, 149
10024files, breaking into chunks, 153–155
10025reassembling, 154
10026global search and replace using
10027Perl, 151
10028logfiles, colorizing data in, 155
10029security
10030certificates, trustworthiness, 199
10031closing access to ex-users, 38–40
10032CVS and remote connections, 54
10033220 | Index
10034security (continued)
10035enhancing with non-standard
10036TLDs, 168
10037evaluating using /proc, 34
10038files, enhancement of, 18
10039history, clearing (bash), 22
10040name resolution and, 158
10041setuid/setgid binaries,
10042eliminating, 23–25
10043semicolon (;) and Perl one-liners, 151
10044send_arp, 129
10045sendmail
10046configuration files, automatic
10047updating, 27
10048servers
10049distributing load with round-robin
10050DNS, 167
10051monitoring (see monitoring)
10052updating asynchronously, 74
10053servers.map file, 207
10054services, removing unnecessary, 3
10055setuid and setgid binaries,
10056finding, 23–25
10057setuid and sudo, 27
10058single (boot parameter), 7
10059skill, 35
10060slaves, 172
10061smbd, 4
10062snapshot backups, 79–84
10063snapshot-type backups
10064rotating snapshots, 80
10065Snark rule, 21
10066snice, 35
10067sockets
10068identifying listeners, 4
10069sockets and associated processes,
10070identifying, 116–119
10071SQLAuthTypes, 179
10072SQLConnectInfo, 179
10073Squid proxy, 92
10074src directory, CVS, 55
10075SSH
10076CVS, using for, 61
10077ssh, 65–67, 139–148
10078backups with tar, 65
10079CVS and remote logins, 54
10080key generation, 140
10081keys, compared to passwords, 140
10082keys, usefulness of, 66
10083logins, using for, 139–142
10084turbo mode, 141
10085port 22, 90
10086ports, forwarding, 146–148
10087private keys, protecting, 150
10088rsync over, 66
10089security of, 24
10090ssh-agent, 142–144
10091GUI, running in, 144
10092ssh-to script, 141
10093traffic, securing with, 146
10094vtun, using with, 148
10095X11 traffic, forwarding over, 145
10096SSL and ntop, 130
10097standard error
10098redirecting, 10
10099standard output
10100redirecting, 10
10101sudo, 25–27
10102setuid and, 27
10103syntax, 26–27
10104groups, 27
10105users, 26
10106syslog, 111–114
10107DNS logs and, 159
10108filtering facilities and priorities, 111
10109MARK messages, 113
10110protecting the syslog port, 114
10111remote logging, risks of, 113
10112syslog.conf, 112
10113system administration
10114automated logout of idle users, 22
10115delegation of responsibilities, 25
10116environment, replicating across a
10117network, 149
10118make, system updates using, 27
10119network monitoring (see monitoring)
10120processes, manipulating with
10121procps, 34
10122runaway processes,
10123preventing, 36–38
10124ssh, using for logins across the
10125network, 139
10126user accounts, closing, 38–40
10127system administrators, xvi, 1
10128system resource limits, setting, 36–38
10129Index | 221
10130T
10131tab completion, 15
10132takeover script, 128
10133tar, 66
10134backups over ssh, 65
10135directories, recursive copying, 70
10136pax and, 67
10137tcpdump and network debugging, 100
10138telnet, 5
10139terminal windows
10140titlebar, customizing, 20
10141tl script, 120
10142TLDs (top level domains), 168
10143TLDs (top-level domains)
10144running your own, 168
10145TLDs (top-level domains), pattern
10146matching to, 29
10147TMOUT variable, 22
10148top, 119
10149? (question mark), 119
10150filtering of results, 119
10151sort criteria, specifying, 119
10152top command, 34
10153traceroute
10154setuid and, 25
10155transparent Squid proxy, 92
10156tun, 101
10157tunneling
10158IP tunneling, 97
10159NAT, circumvention with vtun over
10160ssh, 101–106
10161U
10162unmaskirq parameter, hdparm, 45
10163URL query strings and content
10164delivery, 203
10165users
10166closing user accounts, 38–40
10167using_dma parameter, hdparm, 45
10168UUCP and setuid/setgid binaries, 25
10169V
10170vendor tags (CVS), 53
10171views (see under BIND)
10172VirtualHosts, monitoring, 133
10173virus filters, 93
10174visudo, 26
10175vmstat, 35
10176vtun, 101–110
10177NAT, circumvention ssh, 101–106
10178ssh, using over, 105
10179vtund server, launching, 103
10180vtund.conf, 102–110
10181client-side, 103
10182automatic generation via Perl
10183script, 106–110
10184server-side, 102
10185W
10186wall
10187setgid and, 24
10188watch, 114
10189web servers
10190diagnosing error conditions
10191(example), 12–15
10192web sites, performance costs of
10193interactivity, 205
10194web traffic, monitoring, 132–138
10195whitelists, 89–91
10196whitespace in filenames, handling, 16
10197write
10198setgid and, 24
10199X
10200X11 traveling, forwarding over ssh, 145
10201xargs, 16
10202find, used with, 17
10203D ownload from Wow! eBook <www.wowebook.com>
10204Colophon
10205Our look is the result of reader comments, our own experimentation, and
10206feedback from distribution channels. Distinctive covers complement our
10207distinctive approach to technical topics, breathing personality and life into
10208potentially dry subjects.
10209The tool on the cover of Linux Server Hacks is an ax. An ax is a chopping
10210tool based on one of the six simple machines of physics: the wedge. Though
10211the ax is one of the earliest man-made tools, dating back anywhere from
10212100,000 to 500,000, its simplicity and efficiency make it indispensable to
10213this day.
10214Sarah Sherman was the production editor and copyeditor for Linux Server
10215Hacks. Colleen Gorman, Mary Brady, and Claire Cloutier provided quality
10216control. John Bickelhaupt wrote the index. Linley Dolby provided produc-
10217tion assistance.
10218Edie Freedman designed the cover of this book. The cover image is an orig-
10219inal photograph from the CMCD collection. Emma Colby produced the
10220cover layout with QuarkXPress 4.1 using Adobe’s ITC Garamond and
10221Helvetica Neue fonts.
10222David Futato designed the interior layout. This book was converted to
10223FrameMaker 5.5.6 with a format conversion tool created by Erik Ray, Jason
10224McIntosh, Neil Walls, and Mike Sierra that uses Perl and XML technolo-
10225gies. The text font is Linotype Birka; the heading font is Adobe Helvetica
10226Neue Condensed; and the code font is LucasFont’s TheSans Mono
10227Condensed. The illustrations that appear in the book were produced by
10228Robert Romano and Jessamyn Read using Macromedia FreeHand 9 and
10229Adobe Photoshop 6. This colophon was written by Linley Dolby.