WxnHZxjq

1www.it-ebooks.info
2Linux Shell Scripting
3Cookbook
4Solve real-world shell scripting problems with over 110
5simple but incredibly effective recipes
6Sarath Lakshman
7BIRMINGHAM - MUMBAI
8www.it-ebooks.info
9Linux Shell Scripting Cookbook
10Copyright © 2011 Packt Publishing
11All rights reserved. No part of this book may be reproduced, stored in a retrieval system,
12or transmitted in any form or by any means, without the prior written permission of the
13publisher, except in the case of brief quotations embedded in critical articles or reviews.
14Every effort has been made in the preparation of this book to ensure the accuracy of the
15information presented. However, the information contained in this book is sold without
16warranty, either express or implied. Neither the author, nor Packt Publishing, and its
17dealers and distributors will be held liable for any damages caused or alleged to be
18caused directly or indirectly by this book.
19Packt Publishing has endeavored to provide trademark information about all of the
20companies and products mentioned in this book by the appropriate use of capitals.
21However, Packt Publishing cannot guarantee the accuracy of this information.
22First published: January 2011
23Production Reference: 1200111
24Published by Packt Publishing Ltd.
2532 Lincoln Road
26Olton
27Birmingham, B27 6PA, UK.
28ISBN 978-1-849513-76-0
29www.packtpub.com
30Cover Image by Charwak A ( charwak86@gmail.com )
31www.it-ebooks.info
32Credits
33Author
34Sarath Lakshman
35Reviewers
36Aric Pedersen
37John L Weatherwax
38Acquisition Editor
39Steven Wilding
40Development Editor
41Hyacintha D'Souza
42Technical Editor
43Hithesh Uchil
44Indexer
45Tejal Daruwale
46Editorial Team Leader
47Aanchal Kumar
48Project Team Leader
49Priya Mukherji
50Project Coordinator
51Jovita Pinto
52Proofreader
53Lesley Harrison
54Production Coordinator
55Kruthika Bangera
56Cover Work
57Kruthika Bangera
58www.it-ebooks.info
59About the Author
60Sarath Lakshman is a 21 year old who was bitten by the Linux bug during his teenage
61years. He is an undergraduate student of Computer Science and Engineering at Model
62Engineering College, Cochin, Kerala.
63Sarath is a life hacker who loves to explore innovations. He is a GNU/Linux enthusiast
64and hactivist of free and open source software. He spends most of his time hacking
65with computers and having fun with his great friends. Sarath is well known as the
66developer of SLYNUX (2005)—a user friendly GNU/Linux distribution for Linux newbies.
67The free and open source software projects he has contributed to are PiTiVi Video editor,
68SLYNUX GNU/Linux distro, Swathantra Malayalam Computing, School-Admin, Istanbul,
69and the Pardus Project. He has authored many articles for the Linux For You magazine
70on various domains of FOSS technologies. He had made a contribution to several
71different open source projects during his multiple Google Summer of Code projects.
72He currently lives with his parents in Kerala. Sarath can be reached via his website
73http://www.sarathlakshman.com .
74I would like to thank my friends and family for the great support and
75encouragement they have given me for all my endeavors. I would like
76to thank my friends Anu Mahadevan and Neenu Jacob for the tireless
77enthusiasm and patience to read through the chapter developments
78and providing comments during development. I would also like to thank
79Mr. Atanu Datta for helping me come up with the chapter titles. I extend
80my gratitude to the team at Packt Publishing who helped me in making this
81book happen.
82www.it-ebooks.info
83About the Reviewers
84Aric Pedersen is the author of cPanel User Guide and Tutorial (ISBN 978-1-904811-
8592-3) and Web Host Manager Administration Guide (ISBN 978-1-904811-50-3), both
86written for Packt Publishing. He also served as a reviewer for CUPS Administrative Guide
87(ISBN 978-1-84719-258-5) and Linux E-mail Second Edition (ISBN 978-1-847198-64-8)
88published by Packt Publishing.
89Aric has over nine years of experience working as a System Administrator. He currently
90works for  Hostdime.com , the world-class web host; and also for  Netenberg.com ,
91makers of Fantastico, the world's most popular web script installer for cPanel servers.
92I would like to thank PMUG ( http://www.pmug-nj.org/ ) for years of
93interesting content and camaraderie.
94John L Weatherwax has been in love with computers ever since he had access to
95one. An applied mathematician by trade, he began his study of computer programming
96with the Pascal programming language on the computer that his father bought way
97before most people had even thought of using a computer for anything truly practical.
98As an undergraduate, he obtained dual degrees in physics and mathematics from the
99University of Missouri Columbia and as a graduate student he obtained a Ph.D. from M.I.T.
100in Applied Mathematics. After graduating, he worked for many years applying pattern
101recognition techniques to various problems in electrical engineering before transitioning to
102mathematical finance. In school, he was strongly influenced by the UNIX/Linux philosophy
103of using simpler component programs together to solve larger problems. Many years later,
104while exploring various trading strategies as a quantitative analysis at a Wall Street firm,
105he still uses UNIX commands and the UNIX philosophy on a daily basis. Most of the work
106he does now is related to using mathematical models to predict prices and trade various
107financial instruments. He spends his free time working on applied mathematics and pattern
108recognition (much of which can be found on his website at  http://waxworksmath.com/ )
109and spending time with his family.
110www.it-ebooks.info
111www.PacktPub.com
112Support files, eBooks, discount offers, and more
113You might want to visit  www.PacktPub.com for support files and downloads related to
114your book.
115Did you know that Packt offers eBook versions of every book published, with PDF and
116ePub files available? You can upgrade to the eBook version at  www.PacktPub.com and
117as a print book customer, you are entitled to a discount on the eBook copy. Get in touch
118with us at  service@packtpub.com for more details.
119At  www.PacktPub.com , you can also read a collection of free technical articles, sign up
120for a range of free newsletters and receive exclusive discounts and offers on Packt books
121and eBooks.
122http://PacktLib.PacktPub.com
123Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book
124library. Here, you can access, read, and search across Packt's entire library of books.
125Why Subscribe?
126• Fully searchable across every book published by Packt
127• Copy and paste, print, and bookmark content
128• On demand and accessible via web browser
129Free Access for Packt account holders
130If you have an account with Packt at  www.PacktPub.com , you can use this to access
131PacktLib today and view nine entirely free books. Simply use your login credentials for
132immediate access.
133www.it-ebooks.info
134www.it-ebooks.info
135www.it-ebooks.info
136This book is dedicated to my parents who showed me the bright choices in life.
137www.it-ebooks.info
138www.it-ebooks.info
139Table of Contents
140Preface  1
141Chapter 1: Shell Something Out  7
142Introduction  7
143Printing in the terminal  9
144Playing with variables and environment variables  12
145Doing math calculations with the shell  17
146Playing with file descriptors and redirection  19
147Arrays and associative arrays  25
148Visiting aliases  27
149Grabbing information about terminal  29
150Getting, setting dates, and delays  30
151Debugging the script  33
152Functions and arguments  35
153Reading the output of a sequence of commands  38
154Reading "n" characters without pressing Return  40
155Field separators and iterators  41
156Comparisons and tests  44
157Chapter 2: Have a Good Command  49
158Introduction  50
159Concatenating with cat  50
160Recording and playback of terminal sessions  53
161Finding files and file listing  55
162Playing with xargs  63
163Translating with tr  69
164Checksum and verification  72
165Sorting, unique and duplicates  75
166Temporary file naming and random numbers  80
167Splitting files and data  81
168www.it-ebooks.info
169ii
170Table of Contents
171Slicing file names based on extension  84
172Renaming and moving files in bulk  86
173Spell checking and dictionary manipulation  89
174Automating interactive input  90
175Chapter 3: File In, File Out  95
176Introduction  96
177Generating files of any size  96
178Intersection and set difference (A-B) on text files  97
179Finding and deleting duplicate files  100
180Making directories for a long path  103
181File permissions, ownership, and sticky bit  104
182Making files immutable  109
183Generating blank files in bulk  110
184Finding a symbolic link and its target  111
185Enumerating file type statistics  113
186Loopback files and mounting  115
187Creating ISO files, Hybrid ISO  117
188Finding difference between files, patching  120
189head and tail – printing the last or first 10 lines  122
190Listing only directories – alternative methods  125
191Fast command-line navigation using pushd and popd  126
192Counting number of lines, words, and characters in a file  128
193Printing directory tree  129
194Chapter 4: Texting and Driving  131
195Introduction  132
196Basic regular expression primer  132
197Searching and mining "text" inside a file with grep  136
198Column-wise cutting of a file with cut  142
199Frequency of words used in a given file  146
200Basic sed primer  147
201Basic awk primer  150
202Replacing strings from a text or file  156
203Compressing or decompressing JavaScript  158
204Iterating through lines, words, and characters in a file  161
205Merging multiple files as columns  162
206Printing the nth word or column in a file or line  163
207Printing text between line numbers or patterns  164
208Checking palindrome strings with a script  165
209Printing lines in the reverse order  169
210Parsing e-mail addresses and URLs from text  171
211www.it-ebooks.info
212iii
213Table of Contents
214Printing n lines before or after a pattern in a file  172
215Removing a sentence in a file containing a word  174
216Implementing head, tail, and tac with awk  175
217Text slicing and parameter operations  177
218Chapter 5: Tangled Web? Not At All!  179
219Introduction  180
220Downloading from a web page  180
221Downloading a web page as formatted plain text  183
222A primer on cURL  183
223Accessing Gmail from the command line  188
224Parsing data from a website  189
225Image crawler and downloader  191
226Web photo album generator  193
227Twitter command-line client  195
228define utility with Web backend  197
229Finding broken links in a website  199
230Tracking changes to a website  200
231Posting to a web page and reading response  203
232Chapter 6: The Backup Plan  205
233Introduction  205
234Archiving with tar  206
235Archiving with cpio  211
236Compressing with gunzip (gzip)  212
237Compressing with bunzip (bzip)  215
238Compressing with lzma  217
239Archiving and compressing with zip  219
240squashfs – the heavy compression filesystem  220
241Cryptographic tools and hashes  222
242Backup snapshots with rsync  224
243Version control based backup with Git  227
244Cloning hard drive and disks with dd  230
245Chapter 7: The Old-boy Network  233
246Introduction  233
247Basic networking primer  234
248Let's ping!  241
249Listing all the machines alive on a network  243
250Transferring files  247
251Setting up an Ethernet and wireless LAN with script  250
252Password-less auto-login with SSH  253
253Running commands on remote host with SSH  255
254www.it-ebooks.info
255iv
256Table of Contents
257Mounting a remote drive at a local mount point  259
258Multi-casting window messages on a network  260
259Network traffic and port analysis  262
260Chapter 8: Put on the Monitor's Cap  265
261Introduction  266
262Disk usage hacks  266
263Calculating execution time for a command  272
264Information about logged users, boot logs, and failure boot  274
265Printing the 10 most frequently-used commands  276
266Listing the top 10 CPU consuming process in a hour  278
267Monitoring command outputs with watch  281
268Logging access to files and directories  282
269Logfile management with logrotate  283
270Logging with syslog  285
271Monitoring user logins to find intruders  286
272Remote disk usage health monitor  289
273Finding out active user hours on a system  292
274Chapter 9: Administration Calls  295
275Introduction  295
276Gathering information about processes  296
277Killing processes and send or respond to signals  304
278which, whereis, file, whatis, and loadavg explained  307
279Sending messages to user terminals  309
280Gathering system information  311
281Using /proc – gathering information  312
282Scheduling with cron  313
283Writing and reading MySQL database from Bash  316
284User administration script  321
285Bulk image resizing and format conversion  325
286Index  329
287www.it-ebooks.info
288Preface
289GNU/Linux is a remarkable operating system that comes with a complete development
290environment that is stable, reliable, and extremely powerful. The shell, being the native
291interface to communicate with the operating system, is capable of controlling the entire
292operating system. An understanding of shell scripting helps you to have better awareness
293of the operating system and helps you to automate most of the manual tasks with a few
294lines of script, saving you an enormous amount of time. Shell scripts can work with many
295external command-line utilities for tasks such as querying information, easy text manipulation,
296scheduling task running times, preparing reports, sending mails, and so on. There are
297numerous commands on the GNU/Linux shell, which are documented but hard to understand.
298This book is a collection of essential command-line script recipes along with detailed
299descriptions tuned with practical applications. It covers most of the important commands
300in Linux with a variety of use cases, accompanied by plenty of examples. This book helps
301you to perform complex data manipulations involving tasks such as text processing, file
302management, backups, and more with the combination of few commands.
303Do you want to become the command-line wizard who performs any complex text manipulation
304task in a single line of code? Have you wanted to write shell scripts and reporting tools for fun or
305serious system administration? This cookbook is for you. Start reading!.
306What this book covers
307Chapter 1, Shell Something Out, has a collection of recipes that covers the basic tasks such
308as printing in the terminal, performing mathematical operations, arrays, operators, functions,
309aliases, file redirection, and so on by using Bash scripting. This chapter is an introductory
310chapter for understanding the basic concepts and features in Bash.
311Chapter 2, Have a Good Command, shows various commands that are available with GNU/
312Linux that come under practical usages in different circumstances. It introduces various
313essential commands such as cat, md5sum, find, tr, sort, uniq, split, rename, look, and so on.
314This chapter travels through different practical usage examples that users may come across
315and that they could make use of.
316www.it-ebooks.info
317Preface
3182
319Chapter 3, File In, File Out, contains a collection of task recipes related to files and file
320systems. This chapter explains how to generate large size files, installing a file system on files
321and mounting files, finding and removing duplicate files, counting lines in a file, creating ISO
322images, collecting details about files, symbolic link manipulation, file permissions and file
323attributes, and so on.
324Chapter 4, Texting and Driving, has a collection of recipes that explains most of the command-
325line text processing tools well under GNU/Linux with a number of task examples. It also has
326supplementary recipes for giving a detailed overview of regular expressions and commands
327such as sed and awk. This chapter goes through solutions to most of the frequently used text
328processing tasks in a variety of recipes.
329Chapter 5, Tangled Web? Not At All!, has a collection of shell-scripting recipes that are
330adherent to the Internet and Web. This chapter is intended to help readers understand how to
331interact with the web using shell scripts to automate tasks such as collecting and parsing data
332from web pages, POST and GET to web pages, writing clients to web services, downloading
333web pages, and so on.
334Chapter 6, The Backup Plan, shows several commands used for performing data backup,
335archiving, compression, and so on, and their usages with practical script examples. It
336introduces commands such as tar, gzip, bunzip, cpio, lzma, dd, rsync, git, squashfs, and much
337more. This chapter also walks through essential encryption techniques.
338Chapter 7, The Old-boy Network, has a collection of recipes that talks about networking on
339Linux and several commands useful to write network-based scripts. The chapter starts with
340an introductory basic networking primer. Important tasks explained in the chapter include
341password-less login with SSH, transferring files through network, listing alive machines on a
342network, multi-cast messaging, and so on.
343Chapter 8, Put on the Monitor's Cap, walks through several recipes related to monitoring
344activities on the Linux system and tasks used for logging and reporting. The chapter explains
345tasks such as calculating disk usage, monitoring user access, CPU usage, syslog, frequently
346used commands, and much more.
347Chapter 9, Administration Calls, has a collection of recipes for system administration. This
348chapter explains different commands to collect details about the system, user management
349using scripting, sending messages to users, bulk image resizing, accessing MySQL databases
350from shell, and so on.
351www.it-ebooks.info
352Preface
3533
354What you need for this book
355Basic user experience with any GNU/Linux platform will help you easily follow the book.
356We have tried to keep all the recipes in the book precise and as simple to follow as possible.
357Your curiosity for learning with the Linux platform is the only prerequisite for the book.
358Step-by-step explanations are provided for solving the scripting problems explained in the
359book. In order to run and test the examples in the book, an Ubuntu Linux installation is
360recommended, however, any other Linux distribution is enough for most of the tasks. You will
361find the book to be a straightforward reference to essential shell scripting tasks as well as a
362learning aid to code real-world efficient scripts.
363Who this book is for
364If you are a beginner or an intermediate user who wants to master the skill of quickly writing
365scripts to perform various tasks without reading entire manpages, this book is for you. You can
366start writing scripts and one-liners by simply looking at a similar recipe and its descriptions
367without any working knowledge of shell scripting or Linux. Intermediate or advanced users
368as well as system administrators or developers and programmers can use this book as a
369reference when they face problems while coding.
370Conventions
371In this book, you will find a number of styles of text that distinguish between different kinds of
372information. Here are some examples of these styles, and an explanation of their meaning.
373Code words in text are shown as follows: "We can use formatted strings with  printf ."
374A block of code is set as follows:
375#!/bin/bash
376#Filename: printf.sh
377printf "%-5s %-10s %-4s\n" No Name Mark
378printf "%-5s %-10s %-4.2f\n" 1 Sarath 80.3456
379printf "%-5s %-10s %-4.2f\n" 2 James 90.9989
380printf "%-5s %-10s %-4.2f\n" 3 Jeff 77.564
381Any command-line input or output is written as follows:
382$ chmod +s executable_file
383# chown root.root executable_file
384# chmod +s executable_file
385$ ./executable_file
386www.it-ebooks.info
387Preface
3884
389Warnings or important notes appear in a box like this.
390Tips and tricks appear like this.
391Reader feedback
392Feedback from our readers is always welcome. Let us know what you think about this
393book—what you liked or may have disliked. Reader feedback is important for us to develop
394titles that you really get the most out of.
395To send us general feedback, simply send an e-mail to  feedback@packtpub.com , and
396mention the book title via the subject of your message.
397If there is a book that you need and would like to see us publish, please send us a note in the
398SUGGEST A TITLE form on  www.packtpub.com or e-mail  suggest@packtpub.com .
399If there is a topic that you have expertise in and you are interested in either writing or
400contributing to a book, see our author guide on  www.packtpub.com/authors .
401Customer support
402Now that you are the proud owner of a Packt book, we have a number of things to help you to
403get the most from your purchase.
404Downloading the example code for this book
405You can download the example code files for all Packt books you have
406purchased from your account at http://www.PacktPub.com. If you
407purchased this book elsewhere, you can visit http://www.PacktPub.com/
408support and register to have the files e-mailed directly to you.
409www.it-ebooks.info
410Preface
4115
412Errata
413Although we have taken every care to ensure the accuracy of our content, mistakes do happen.
414If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be
415grateful if you would report this to us. By doing so, you can save other readers from frustration
416and help us improve subsequent versions of this book. If you find any errata, please report them
417by visiting  http://www.packtpub.com/support , selecting your book, clicking on the errata
418submission form link, and entering the details of your errata. Once your errata are verified, your
419submission will be accepted and the errata will be uploaded on our website, or added to any
420list of existing errata, under the Errata section of that title. Any existing errata can be viewed by
421selecting your title from  http://www.packtpub.com/support .
422Piracy
423Piracy of copyright material on the Internet is an ongoing problem across all media. At Packt,
424we take the protection of our copyright and licenses very seriously. If you come across any
425illegal copies of our works, in any form, on the Internet, please provide us with the location
426address or website name immediately so that we can pursue a remedy.
427Please contact us at  copyright@packtpub.com with a link to the suspected pirated material.
428We appreciate your help in protecting our authors, and our ability to bring you valuable content.
429Questions
430You can contact us at  questions@packtpub.com if you are having a problem with any
431aspect of the book, and we will do our best to address it.
432www.it-ebooks.info
433www.it-ebooks.info
4341
435Shell Something Out
436In this chapter, we will cover:
437f Printing in the terminal
438f Playing with variables and environment variables
439f Doing Math calculations with the shell
440f Playing with file descriptors and redirection
441f Arrays and associative arrays
442f Visiting aliases
443f Grabbing information about the terminal
444f Getting, setting dates, and delays
445f Debugging the script
446f Functions and arguments
447f Reading output of a sequence of commands in a variable
448f Reading "n" characters without pressing Return
449f Field separators and iterators
450f Comparisons and tests
451Introduction
452UNIX-like systems are amazing operating system designs. Even after many decades, the
453UNIX-style architecture for operating systems serves as one of the best designs. One of the
454most important features of this architecture is the command-line interface or the shell. The
455shell environment helps users to interact with and access core functions of the operating
456system. The term scripting is more relevant in this context. Scripting is usually supported by
457interpreter-based programming languages. Shell scripts are files in which we write a sequence
458of commands that we need to perform. And the script file is executed using the shell utility.
459www.it-ebooks.info
460Shell Something Out
4618
462In this book we are dealing with Bash (Bourne Again Shell), which is the default shell
463environment for most GNU/Linux systems. Since GNU/Linux is the most prominent operating
464system based on a UNIX-style architecture, most of the examples and discussions are written
465by keeping Linux systems in mind.
466The primary purpose of this chapter is to give readers an insight about the shell environment
467and become familiar with the basic features that come around the shell. Commands are
468typed and executed in a shell terminal. When opened, in a terminal, a prompt is available. It is
469usually in the following format:
470username@hostname$
471Or:
472root@hostname#
473Or simply as  $ or  #.
474$ represents regular users and  # represents the administrative user root. Root is the most
475privileged user in a Linux system.
476A shell script is a text file that typically begins with a shebang, as follows:
477#!/bin/bash
478For any scripting language in a Linux environment, a script starts with a special line called
479shebang. Shebang is a line for which  #!  is prefixed to the interpreter path.  /bin/bash  is
480the interpreter command path for Bash.
481Execution of a script can be done in two ways. Either we can run the script as a command-line
482argument for  sh  or run a self executable with execution permission.
483The script can be run with the filename as a command-line argument as follows:
484$ sh script.sh # Assuming script is in the current directory.
485Or:
486$ sh /home/path/script.sh # Using full path of script.sh.
487If a script is run as a command-line argument for  sh , the shebang in the script is of no use.
488In order to self execute a shell script, it requires executable permission. While running as a
489self executable, it makes use of the shebang. It runs the script using the interpreter path that
490is appended to  #! in shebang. The execution permission for the script can be set as follows:
491$ chmod a+x script.sh
492www.it-ebooks.info
493Chapter 1
4949
495This command gives the  script.sh file the executable permission for all users. The script
496can be executed as:
497$ ./script.sh #./ represents the current directory
498Or:
499$ /home/path/script.sh # Full path of the script is used
500The shell program will read the first line and see that the shebang is  #!/bin/bash . It will
501identify the  /bin/bash and execute the script internally as:
502$ /bin/bash script.sh
503When a terminal is opened it initially executes a set of commands to define various settings
504like prompt text, colors, and many more. This set of commands (run commands) are read
505from a shell script called  .bashrc , which is located in the home directory of the user ( ~/.
506bashrc ). The bash shell also maintains a history of commands run by the user. It is available
507in the file  ~/.bash_history .  ~  is the shorthand for the user home directory path.
508In Bash, each command or command sequence is delimited by using a semicolon or a new
509line. For example:
510$ cmd1 ; cmd2
511This is equivalent to:
512$ cmd1
513$ cmd2
514Finally, the  # character is used to denote the beginning of unprocessed comments. A
515comment section starts with  # and proceeds up to the end of that line. The comment lines are
516most often used to provide comments about the code in the file or to stop a line of code from
517being executed.
518Now let's move on to the basic recipes in this chapter.
519Printing in the terminal
520The terminal is an interactive utility by which a user interacts with the shell environment.
521Printing text in the terminal is a basic task that most shell scripts and utilities need to perform
522regularly. Printing can be performed via various methods and in different formats.
523How to do it...
524echo  is the basic command for printing in the terminal.
525www.it-ebooks.info
526Shell Something Out
52710
528echo  puts a newline at the end of every invocation by default:
529$ echo "Welcome to Bash"
530Welcome to Bash
531Simply using double-quoted text with the  echo command prints the text in the terminal.
532Similarly, text without double-quotes also gives the same output:
533$ echo Welcome to Bash
534Welcome to Bash
535Another way to do the same task is by using single quotes:
536$ echo 'text in quote'
537These methods may look similar, but some of them have got a specific purpose and side
538effects too. Consider the following command:
539$ echo "cannot include exclamation - ! within double quotes"
540This will return the following:
541bash: !: event not found error
542Hence, if you want to print  ! , do not use within double-quotes or you may escape the  ! with a
543special escape character ( \ ) prefixed with it.
544$ echo Hello world !
545Or:
546$ echo 'Hello world !'
547Or:
548$ echo "Hello world \!" #Escape character \ prefixed.
549When using  echo with double-quotes, you should add  set +H before issuing  echo so that you
550can use  ! .
551The side effects of each of the methods are as follows:
552f When using  echo without quotes, we cannot use a semicolon as it acts as a delimiter
553between commands in the bash shell.
554f echo hello;hello  takes  echo hello as one command and the second  hello
555as the second command.
556f When using  echo with single quotes, the variables (for example,  $var will not be
557expanded) inside the quotes will not be interpreted by Bash, but will be displayed as is.
558www.it-ebooks.info
559Chapter 1
56011
561This means:
562$ echo '$var'  will return  $var
563whereas
564$ echo $var  will return the value of the variable  $var if defined or nothing at all if
565it is not defined.
566Another command for printing in the terminal is the  printf command.  printf uses the
567same arguments as the  printf command in the C programming language. For example:
568$ printf "Hello world"
569printf  takes quoted text or arguments delimited by spaces. We can use formatted strings
570with  printf . We can specify string width, left or right alignment, and so on. By default,
571printf  does not have newline as in the  echo command. We have to specify a newline when
572required, as shown in the following script:
573#!/bin/bash
574#Filename: printf.sh
575printf "%-5s %-10s %-4s\n" No Name Mark
576printf "%-5s %-10s %-4.2f\n" 1 Sarath 80.3456
577printf "%-5s %-10s %-4.2f\n" 2 James 90.9989
578printf "%-5s %-10s %-4.2f\n" 3 Jeff 77.564
579We will receive the formatted output:
580No Name Mark
5811 Sarath 80.35
5822 James 91.00
5833 Jeff 77.56
584%s ,  %c ,  %d , and  %f  are format substitution characters for which an argument can be placed
585after the quoted format string.
586%-5s can be described as a string substitution with left alignment ( - represents left
587alignment) with width equal to  5 . If  - was not specified, the string would have been aligned to
588the right. The width specifies the number of characters reserved for that variable. For  Name ,
589the width reserved is  10 . Hence, any name will reside within the 10-character width reserved
590for it and the rest of the characters will be filled with space up to 10 characters in total.
591For floating point numbers, we can pass additional parameters to round off the decimal places.
592For marks, we have formatted the string as  %-4.2f , where  .2 specifies rounding off to two
593decimal places. Note that for every line of the format string a  \n  newline is issued.
594www.it-ebooks.info
595Shell Something Out
59612
597There's more...
598It should be always noted that flags (such as -e, -n, and so on) for  echo and  printf  should
599appear before any strings in the command, else Bash will consider the flags as another string.
600Escaping newline in echo
601By default,  echo has a newline appended at the end of its output text. This can be avoided
602by using the  -n flag.  echo can also accept escape sequences in double-quoted strings as
603argument. For using escape sequences, use  echo as  echo -e "string containing
604escape sequences" . For example:
605echo -e "1\t2\t3"
606123
607Printing colored output
608Producing colored output on the terminal is very interesting stuff. We produce colored output
609using escape sequences.
610Color codes are used to represent each color. For example, reset=0, black=30, red=31,
611green=32, yellow=33, blue=34, magenta=35, cyan=36, and white=37.
612In order to print colored text, enter the following:
613echo -e "\e[1;31m This is red text \e[0m"
614Here  \e[1;31 is the escape string that sets the color to red and  \e[0m resets the color back.
615Replace  31 with the required color code.
616For a colored background, reset = 0, black = 40, red = 41, green = 42, yellow = 43, blue = 44,
617magenta = 45, cyan = 46, and white=47, are the color code that are commonly used.
618In order to print a colored background, enter the following:
619echo -e "\e[1;42m Green Background \e[0m"
620Playing with variables and environment
621variables
622Variables are essential components of every programming language and are used to hold
623varying data. Scripting languages usually do not require variable type declaration before its
624use. It can be assigned directly. In Bash, the value for every variable is string. If we assign
625variables with quotes or without quotes, they are stored as string. There are special variables
626used by the shell environment and the operating system environment to store special values,
627which are called environment variables.
628Let's look at the recipes.
629www.it-ebooks.info
630Chapter 1
63113
632Getting ready
633Variables are named with usual naming constructs. When an application is executing, it will be
634passed with a set of variables called environment variables. From the terminal, to view all the
635environment variables related to that terminal process, issue the  env  command. For every
636process, environment variables in its runtime can be viewed by:
637cat /proc/$PID/environ
638Set the  PID with the process ID of the relevant process (PID is always an integer).
639For example, assume that an application called gedit is running. We can obtain the process ID
640of gedit with the  pgrep command as follows:
641$ pgrep gedit
64212501
643You can obtain the environment variables associated with the process by executing the
644following command:
645$ cat /proc/12501/environ
646GDM_KEYBOARD_LAYOUT=usGNOME_KEYRING_PID=1560USER=slynuxHOME=/home/slynux
647Note that many environment variables are stripped off for convenience. The actual output may
648contain numerous variables.
649The above mentioned command returns a list of environment variables and their values.
650Each variable is represented as a  name=value pair and are separated by a null character
651( \0 ). If you can substitute the  \0 character with  \n , you can reformat the output to show
652each  variable=value pair in each line. Substitution can be made using the  tr command
653as follows:
654$ cat /proc/12501/environ | tr '\0' '\n'
655Now, let's see how to assign and manipulate variables and environment variables.
656How to do it...
657A variable can be assigned as follows:
658var=value
659var  is the name of a variable and value is the value to be assigned. If  value does not
660contain any white space characters (like a space), it need not be enclosed in quotes, else it
661must be enclosed in single or double quotes.
662www.it-ebooks.info
663Shell Something Out
66414
665Note that  var = value and  var=value  are different. It is a common mistake to write
666var =value  instead of  var=value . The later is the assignment operation, whereas
667the former is an equality operation.
668Printing the contents of a variable is done using by prefixing  $ with the variable name
669as follows:
670var="value" #Assignment of value to variable var.
671echo $var
672Or:
673echo ${var}
674The output is as follows:
675value
676We can use variable values inside  printf or  echo in double quotes.
677#!/bin/bash
678#Filename :variables.sh
679fruit=apple
680count=5
681echo "We have $count ${fruit}(s)"
682The output is as follows:
683We have 5 apple(s)
684Environment variables are variables that are not defined in the current process, but are
685received from the parent processes. For example,  HTTP_PROXY is an environment variable.
686This variable defines which proxy server should be used for an Internet connection.
687Usually, it is set as:
688HTTP_PROXY=http://192.168.0.2:3128
689export HTTP_PROXY
690The export command is used to set the  env variable. Now any application, executed from
691the current shell script will receive this variable. We can export custom variables for our
692own purposes in an application or shell script that is executed. There are many standard
693environment variables that are available for the shell by default.
694For example,  PATH . A typical  PATH variable will contain:
695$ echo $PATH
696/home/slynux/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/
697sbin:/bin:/usr/games
698www.it-ebooks.info
699Chapter 1
70015
701When given a command for execution, shell automatically searches for the executable in
702the list of directories in the  PATH environment variable (directory paths are delimited by
703the ":" character). Usually,  $PATH is defined in  /etc/environment or  /etc/profile or
704~/.bashrc . When we need to add a new path to the  PATH environment, we use:
705export PATH="$PATH:/home/user/bin"
706Or, alternately, we can use:
707$ PATH="$PATH:/home/user/bin"
708$ export PATH
709$ echo $PATH
710/home/slynux/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/
711sbin:/bin:/usr/games:/home/user/bin
712Here we have added  /home/user/bin to  PATH .
713Some of the well-known environment variables are:  HOME ,  PWD ,  USER ,  UID ,  SHELL , and so on.
714There's more...
715Let's see some more tips associated with regular and environment variables.
716Finding length of string
717Get the length of a variable value as follows:
718length=${#var}
719For example:
720$ var=12345678901234567890
721$ echo ${#var}
72220
723length is the number of characters in the string.
724Identifying the current shell
725Display the currently used shell as follows:
726echo $SHELL
727Or, you can also use:
728echo $0
729www.it-ebooks.info
730Shell Something Out
73116
732For example:
733$ echo $SHELL
734/bin/bash
735$ echo $0
736bash
737Check for super user
738UID is an important environment variable that can be used to check whether the current script
739has been run as root user or regular user. For example:
740if [ $UID -ne 0 ]; then
741echo Non root user. Please run as root.
742else
743echo "Root user"
744fi
745The UID for the root user is 0.
746Modifying the Bash prompt string (username@hostname:~$)
747When we open a terminal or run a shell, we see a prompt string like
748user@hostname: /home/$ . Different GNU/Linux distributions have slightly
749different prompts and different colors. We can customize the prompt text using the
750PS1 environment variable. The default prompt text for the shell is set using a line in the
751~/.bashrc  file.
752f We can list the line used to set the  PS1 variable as follows:
753$ cat ~/.bashrc | grep PS1
754PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
755f In order to set a custom prompt string, enter:
756slynux@localhost: ~$ PS1="PROMPT>"
757PROMPT> Type commands here # Prompt string changed.
758f We can use colored text by using the special escape sequences like  \e[1;31 (refer
759to the Printing in the terminal recipe of this chapter).
760There are also certain special characters that expand to system parameters. For example,
761\u expands to username,  \h expands to hostname, and  \w expands to the current
762working directory.
763www.it-ebooks.info
764Chapter 1
76517
766Doing math calculations with the shell
767Arithmetic operations are an essential requirement for every programming language. The
768Bash shell comes with a variety of methods for arithmetic operations.
769Getting ready
770The Bash shell environment can perform basic arithmetic operations using the commands
771let ,  (( )) , and  [] . The two utilities  expr and  bc are also very helpful in performing
772advanced operations.
773How to do it...
774A numeric value can be assigned as a regular variable assignment, which is stored as string.
775However, we use methods to manipulate as numbers.
776#!/bin/bash
777no1=4;
778no2=5;
779The  let command can be used to perform basic operations directly.
780While using  let , we use variable names without the  $ prefix, for example:
781let result=no1+no2
782echo $result
783f Increment operation:
784$ let no1++
785f Decrement operation:
786$ let no1--
787f Shorthands:
788let no+=6
789let no-=6
790These are equal to  let no=no+6 and  let no=no-6 respectively.
791f Alternate methods:
792The  [] operator can be used similar to the  let command as follows:
793result=$[ no1 + no2 ]
794www.it-ebooks.info
795Shell Something Out
79618
797Using  $ prefix inside  [] operators are legal, for example:
798result=$[ $no1 + 5 ]
799(( )) can also be used.  $  prefixed with a variable name is used when the  (( ))
800operator is used, as follows:
801result=$(( no1 + 50 ))
802expr  can also be used for basic operations:
803result=`expr 3 + 4`
804result=$(expr $no1 + 5)
805All of the above methods do not support floating point numbers, and operate on
806integers only.
807bc the precision calculator is an advanced utility for mathematical operations. It has
808a wide range of options. We can perform floating point operations and use advanced
809functions as follows:
810echo "4 * 0.56" | bc
8112.24
812no=54;
813result=`echo "$no * 1.5" | bc`
814echo $result
81581.0
816Additional parameters can be passed to  bc with prefixes to the operation with
817semicolon as delimiters through  stdin .
818‰ Specifying decimal precision (scale): In the following example the  scale=2
819parameter sets the number of decimal places to  2 . Hence the output of  bc
820will contain a number with two decimal places:
821echo "scale=2;3/8" | bc
8220.37
823‰ Base conversion with bc: We can convert from one base number system to
824another one. Let's convert from decimal to binary, and binary to octal:
825#!/bin/bash
826Description: Number conversion
827no=100
828echo "obase=2;$no" | bc
8291100100
830no=1100100
831echo "obase=10;ibase=2;$no" | bc
832100
833www.it-ebooks.info
834Chapter 1
83519
836‰ Calculating squares and square roots can be done as follows:
837echo "sqrt(100)" | bc #Square root
838echo "10^10" | bc #Square
839Playing with file descriptors and redirection
840File descriptors are integers that are associated with file input and output. They keep track
841of opened files. The best-known file descriptors are  stdin ,  stdout , and  stderr . We can
842redirect the contents of one file descriptor to another. The following recipe will give examples
843on how to manipulate and redirect with file descriptors.
844Getting ready
845While writing scripts we use standard input ( stdin ), standard output ( stdout ), and standard
846error ( stderr ) frequently. Redirection of output to a file by filtering the contents is one of the
847essential things we need to perform. While a command outputs some text, it can be either
848an error or an output (non-error) message. We cannot distinguish whether it is output text or
849an error text by just looking at it. However, we can handle them with file descriptors. We can
850extract text that is attached to a specific descriptor.
851File descriptors are integers associated with an opened file or data stream. File descriptors 0,
8521, and 2 are reserved as follows:
853f 0 –  stdin (standard input)
854f 1 –  stdout (standard output)
855f 2 –  stderr (standard error)
856How to do it...
857Redirecting or saving output text to a file can be done as follows:
858$ echo "This is a sample text 1" > temp.txt
859This would store the echoed text in  temp.txt by truncating the file, the contents will be
860emptied before writing.
861Next, consider the following example:
862$ echo "This is sample text 2" >> temp.txt
863This would append the text into the file.
864>  and  >>  operators are different. Both of them redirect text to a file, but the first one
865empties the file and then writes to it, whereas the later one adds the output to the end
866of the existing file.
867www.it-ebooks.info
868Shell Something Out
86920
870View the contents of the file as follows:
871$ cat temp.txt
872This is sample text 1
873This is sample text 2
874When we use a redirection operator, it won't print in the terminal but it is directed to a file.
875When redirection operators are used, by default, it takes standard output. In order to explicitly
876take a specific file descriptor, you must prefix the descriptor number to the operator.
877> is equivalent to  1> and similarly it applies for  >> (equivalent to 1>> ).
878Let's see what a standard error is and how you can redirect it.  stderr messages are printed
879when commands output an error message. Consider the following example:
880$ ls +
881ls: cannot access +: No such file or directory
882Here  + is an invalid argument and hence an error is returned.
883Successful and unsuccessful command
884When a command returns after error, it returns a non-zero exit status. The
885command returns zero when it terminates after successful completion. Return
886status can be read from special variable $? (run echo $? immediately after
887the command execution statement to print the exit status).
888The following command prints the  stderr text to the screen rather than to a file:
889$ ls + > out.txt
890ls: cannot access +: No such file or directory
891However, in the following command the  stdout output is empty, so an empty file  out.txt
892is generated:
893$ ls + 2> out.txt # works
894You can redirect  stderr  exclusively to a file and  stdout to another file as follows:
895$ cmd 2>stderr.txt 1>stdout.txt
896It is also possible to redirect  stderr and  stdout to a single file by converting  stderr
897to  stdout  using this preferred method:
898$ cmd 2>&1 output.txt
899or an alternate approach:
900$ cmd &> output.txt
901www.it-ebooks.info
902Chapter 1
90321
904Sometimes the output may contain unnecessary information (such as debug messages).
905If you don't want the output terminal burdened with the  stderr details, then you should
906redirect  stderr output to  /dev/null , which removes it completely. For example, consider
907that we have three files a1, a2, and a3. However, a1 does not have read-write-execute
908permission for the user. When you need to print the contents of files starting with  a , you can
909use the  cat command.
910Set up the test files as follows:
911$ echo a1 > a1
912$ cp a1 a2 ; cp a2 a3;
913$ chmod 000 a1 #Deny all permissions
914While displaying contents of the files using wildcards ( a* ), it will show an error message for file
915a1 as it does not have the proper read permission:
916$ cat a*
917cat: a1: Permission denied
918a1
919a1
920Here  cat: a1: Permission denied belongs to  stderr data. We can redirect  stderr
921data into a file, whereas  stdout remains printed in the terminal. Consider the following code:
922$ cat a* 2> err.txt #stderr is redirected to err.txt
923a1
924a1
925$ cat err.txt
926cat: a1: Permission denied
927Take a look at the following code:
928$ some_command 2> /dev/null
929In this case, the  stderr output is dumped to the  /dev/null file.  /dev/null is a special
930device file where any data received by the file is discarded. The null device is often called the
931bit bucket or black hole.
932When redirection is performed for  stderr or  stdout , the redirected text flows into a file.
933As the text has already been redirected and has gone into the file, no text remains to flow to
934the next command through pipe ( | ), and it appears to the next set of command sequence
935through  stdin .
936www.it-ebooks.info
937Shell Something Out
93822
939However, there is a tricky way to redirect data to a file as well as provide a copy of redirected
940data as  stdin for the next set of commands. This can be done using the  tee command. For
941example, to print the  stdout in the terminal as well as redirect  stdout into a file, the syntax
942for  tee is as follows:
943command | tee FILE1 FILE2
944In the following code,  stdin data is received by the  tee command. It writes a copy of  stdout
945to the file  out.txt and sends another copy as  stdin for the next command. The  cat –n
946command puts a line number for each line received from  stdin and writes it into  stdout :
947$ cat a* | tee out.txt | cat -n
948cat: a1: Permission denied
9491a1
9502a1
951Examine the contents of  out.txt as follows:
952$ cat out.txt
953a1
954a1
955Note that  cat: a1: Permission denied does not appear because it belongs to  stdin .
956tee can read from  stdin only.
957By default, the  tee command overwrites the file, but it can be used with appended options by
958providing the  -a  option, for example:
959$ cat a* | tee –a out.txt | cat –n.
960Commands appear with arguments in the format:  command FILE1 FILE2… or simply
961command FILE .
962We can use  stdin as a command argument. It can be done by using  – as the filename
963argument for the command as follows:
964$ cmd1 | cmd2 | cmd -
965For example:
966$ echo who is this | tee -
967who is this
968who is this
969Alternately, we can use  /dev/stdin as the output filename to use  stdin .
970Similarly, use  /dev/stderr for standard error and  /dev/stdout for standard output. These
971are special device files that correspond to  stdin ,  stderr , and  stdout .
972www.it-ebooks.info
973Chapter 1
97423
975There's more...
976A command that reads  stdin for input can receive data in multiple ways. Also, it is possible
977to specify file descriptors of our own using  cat and pipes, for example:
978$ cat file | cmd
979$ cmd1 | cmd2
980Redirection from file to command
981By using redirection, we can read data from a file as  stdin as follows:
982$ cmd < file
983Redirecting from a text block enclosed within a script
984Sometimes we need to redirect a block of text (multiple lines of text) as standard input.
985Consider a particular case where the source text is placed within the shell script. A practical
986usage example is writing a log file header data. It can be performed as follows:
987#!/bin/bash
988cat <<EOF>log.txt
989LOG FILE HEADER
990This is a test log file
991Function: System statistics
992EOF
993The lines that appear between  cat <<EOF >log.txt  and the next  EOF line will appear as
994stdin data. Print the contents of  log.txt as follows:
995$ cat log.txt
996LOG FILE HEADER
997This is a test log file
998Function: System statistics
999Custom file descriptors
1000A file descriptor is an abstract indicator for accessing a file. Each file access is associated
1001with a special number called a file descriptor. 0, 1, and 2 are reserved descriptor numbers for
1002stdin ,  stdout , and  stderr .
1003www.it-ebooks.info
1004Shell Something Out
100524
1006We can create our own custom file descriptors using the  exec command. If you are already
1007familiar with file programming with any other programming languages, you might have noticed
1008modes for opening files. Usually, three modes are used:
1009f Read mode
1010f Write with truncate mode
1011f Write with append mode
1012< is an operator used to read from the file to  stdin .  > is the operator used to write to a file with
1013truncation (data is written to the target file after truncating the contents).  >> is an operator used
1014to write to a file with append (data is appended to the existing file contents and the contents of
1015the target file will not be lost). File descriptors can be created with one of the three modes.
1016Create a file descriptor for reading a file, as follows:
1017$ exec 3<input.txt # open for reading with descriptor number 3
1018We could use it as follows:
1019$ echo this is a test line > input.txt
1020$ exec 3<input.txt
1021Now you can use file descriptor  3 with commands. For example,  cat <&3 as follows:
1022$ cat <&3
1023this is a test line
1024If a second read is required, we cannot reuse file descriptor  3 . It is needed to reassign file
1025descriptor  3 for read using  exec for making a second read.
1026Create a file descriptor for writing (truncate mode) as follows:
1027$ exec 4>output.txt # open for writing
1028For example:
1029$ exec 4>output.txt
1030$ echo newline >&4
1031$ cat output.txt
1032newline
1033Create a file descriptor for writing (append mode) as follows:
1034$ exec 5>>input.txt
1035www.it-ebooks.info
1036Chapter 1
103725
1038For example:
1039$ exec 5>>input.txt
1040$ echo appended line >&5
1041$ cat input.txt
1042newline
1043appended line
1044Arrays and associative arrays
1045Arrays are a very important component for storing a collection of data as separate entities
1046using indexes.
1047Getting ready
1048Bash supports regular arrays as well as associative arrays. Regular arrays are arrays which
1049can use only integers as its array index. But associative arrays are arrays which can take a
1050string as its array index.
1051Associative arrays are very useful in many types of manipulations. Associative array
1052support came with version 4.0 of Bash. Therefore, older versions of Bash will not
1053support associative arrays.
1054How to do it...
1055An array can be defined in many ways. Define an array using a list of values in a line,
1056as follows:
1057array_var=(1 2 3 4 5 6)
1058#Values will be stored in consecutive locations starting from index 0.
1059Alternately, define an array as a set of index-value pairs as follows:
1060array_var[0]="test1"
1061array_var[1]="test2"
1062array_var[2]="test3"
1063array_var[3]="test4"
1064array_var[4]="test5"
1065array_var[5]="test6"
1066www.it-ebooks.info
1067Shell Something Out
106826
1069Print the contents of an array at a given index using:
1070$ echo ${array_var[0]}
1071test1
1072index=5
1073$ echo ${array_var[$index]}
1074test6
1075Print all of the values in an array as a list using:
1076$ echo ${array_var[*]}
1077test1 test2 test3 test4 test5 test6
1078Alternately, you can use:
1079$ echo ${array_var[@]}
1080test1 test2 test3 test4 test5 test6
1081Print the length of an array (the number of elements in an array), as follows:
1082$ echo ${#array_var[*]}
10836
1084There's more...
1085Associative arrays have been introduced to Bash from version 4.0. They are useful entities to
1086solve many problems using the hashing technique. Let's go into more details.
1087Defining associative arrays
1088In an associative array, we can use any text data as an array index. However, ordinary arrays
1089can only use integers for array indexing.
1090Initially, a declaration statement is required to declare a variable name as an associative
1091array. A declaration can be made as follows:
1092$ declare -A ass_array
1093After the declaration, elements can be added to the associative array using two methods,
1094as follows:
10951. By using inline index-value list method, we can provide a list of index-value pairs:
1096$ ass_array=([index1]=val1 [index2]=val2)
1097www.it-ebooks.info
1098Chapter 1
109927
11002. Alternately, you could use separate index-value assignments:
1101$ ass_array[index1]=val1
1102$ ass_array[index2]=val2
1103For example, consider the assignment of prices for fruits using an associative array:
1104$ declare -A fruits_value
1105$ fruits_value=([apple]='100dollars' [orange]='150 dollars')
1106Display the content of an array as follows:
1107$ echo "Apple costs ${fruits_value[apple]}"
1108Apple costs 100 dollars
1109Listing of array indexes
1110Arrays have indexes for indexing each of the elements. Ordinary and associative arrays differ
1111in terms of index type. We can obtain the list of indexes in an array as follows:
1112$ echo ${!array_var[*]}
1113Or, we can also use:
1114$ echo ${!array_var[@]}
1115In the previous  fruits_value array example, consider the following:
1116$ echo ${!fruits_value[*]}
1117orange apple
1118This will work for ordinary arrays too.
1119Visiting aliases
1120An alias is basically a shortcut that takes the place of typing a long command sequence.
1121Getting ready
1122Aliases can be implemented in multiple ways, either by using functions or by using
1123the  alias command.
1124www.it-ebooks.info
1125Shell Something Out
112628
1127How to do it...
1128An alias can be implemented as follows:
1129$ alias new_command='command sequence'
1130Giving a shortcut to the install command,  apt-get install , can be done as follows:
1131$ alias install='sudo apt-get install'
1132Therefore, we can use  install pidgin instead of  sudo apt-get install pidgin .
1133The  alias command is temporary; aliasing exists until we close the current terminal only.
1134In order to keep these shortcuts permanent, add this statement to the  ~/.bashrc file.
1135Commands in  ~/.bashrc  are always executed when a new shell process is spawned.
1136$ echo 'alias cmd="command seq"' >> ~/.bashrc
1137To remove an alias, remove its entry from  ~/.bashrc  or use the  unalias command.
1138Another method is to define a function with a new command name and write it in  ~/.bashrc .
1139We can alias  rm  so that it will delete the original and keep a copy in a backup directory:
1140alias rm='cp $@ ~/backup; rm $@'
1141When you create an alias, if the item being aliased already exists, it will be replaced by this
1142newly aliased command for that user.
1143There's more...
1144There are situations when aliasing can also be a security breach. See how to identify them:
1145Escaping aliases
1146The  alias command can be used to alias any important command, and you may not always
1147want to run the command using the alias. We can ignore any aliases currently defined by
1148escaping the command we want to run. For example:
1149$ \command
1150The  \ character escapes the command, running it without any aliased changes. While running
1151privileged commands on an untrusted environment, it is always a good security practise to
1152ignore aliases by prefixing the command with  \ . The attacker might have aliased the privileged
1153command with his own custom command to steal the critical information that is provided to
1154the command by the user.
1155www.it-ebooks.info
1156Chapter 1
115729
1158Grabbing information about terminal
1159While writing command-line shell scripts, we will often need to heavily manipulate information
1160about the current terminal, such as number of columns, rows, cursor positions, masked
1161password fields, and so on. This recipe helps to learn about collecting and manipulating
1162terminal settings.
1163Getting ready
1164tput and  stty  are utilities that can be used for terminal manipulations. Let's see how to
1165use them to perform different tasks.
1166How to do it...
1167Get number of columns and rows in a terminal as follows:
1168tput cols
1169tput lines
1170In order to print the current terminal name, use:
1171tput longname
1172For moving the cursor to a position 100,100 you can enter:
1173tput cup 100 100
1174Set the background color for terminal as follows:
1175tput setb no
1176no can be a value in the range of 0 to 7.
1177Set the foreground color for text as follows:
1178tput setf no
1179no can be a value in the range of 0 to 7.
1180In order to make the text bold use:
1181tput bold
1182Start and end underlining by using:
1183tput smul
1184tput rmul
1185www.it-ebooks.info
1186Shell Something Out
118730
1188In order to delete from cursor to end of the line use:
1189tput ed
1190While typing a password, we should not display the characters typed. In the following example,
1191we will see how to do it using  stty :
1192#!/bin/sh
1193#Filename: password.sh
1194echo -e "Enter password: "
1195stty -echo
1196read password
1197stty echo
1198echo
1199echo Password read.
1200The  -echo option above disables output to the terminal, whereas  echo enables output.
1201Getting, setting dates, and delays
1202Many applications require printing dates in different formats, setting the date and time, and
1203performing manipulations based on the date and time. Delays are commonly used to provide
1204a wait time (for example, 1 second) during the program's execution. Scripting contexts, such
1205as performing a monitoring task every five seconds, demand the understanding of writing
1206delays in a program. This recipe will show you how to work with dates and time delays.
1207Getting ready
1208Dates can be printed in a variety of formats. We can also set dates from the command line. In
1209UNIX-like systems, dates are stored as an integer in seconds since 1970-01-01 00:00:00 UTC.
1210This is called epoch or UNIX time. Let's see how to read dates and set them.
1211How to do it...
1212You can read the date as follows:
1213$ date
1214Thu May 20 23:09:04 IST 2010
1215The epoch time can be printed as follows:
1216$ date +%s
12171290047248
1218www.it-ebooks.info
1219Chapter 1
122031
1221Epoch is defined as the number of seconds that have elapsed since midnight proleptic
1222Coordinated Universal Time (UTC) of January 1, 1970, not counting leap seconds. Epoch time
1223is useful when you need to calculate the difference between two dates or time. You may find
1224out the epoch times for two given timestamps and take the difference between the epoch
1225values. Therefore, you can find out the total number of seconds between two dates.
1226We can find out epoch from a given formatted date string. You can use dates in multiple date
1227formats as input. Usually, you don't need to bother about the date string format that you use
1228if you are collecting the date from a system log or any standard application generated output.
1229You can convert a date string into epoch as follows:
1230$ date --date "Thu Nov 18 08:07:21 IST 2010" +%s
12311290047841
1232The  --date option is used to provide a date string as input. However, we can use any date
1233formatting options to print output. Feeding input date from a string can be used to find out the
1234weekday, given the date.
1235For example:
1236$ date --date "Jan 20 2001" +%A
1237Saturday
1238The date format strings are listed in the following table:
1239Date component Format
1240Weekday %a (for example:. Sat)
1241%A (for example: Saturday)
1242Month %b (for example: Nov)
1243%B (for example: November)
1244Day %d (for example: 31)
1245Date in format (mm/dd/yy) %D (for example: 10/18/10)
1246Year %y (for example: 10)
1247%Y (for example: 2010)
1248Hour %I or %H (for example: 08)
1249Minute %M (for example: 33)
1250Second %S (for example: 10)
1251Nano second %N (for example:695208515)
1252epoch UNIX time in seconds %s (for example: 1290049486)
1253www.it-ebooks.info
1254Shell Something Out
125532
1256Use a combination of format strings prefixed with  + as an argument for the  date command to
1257print the date in the format of your choice. For example:
1258$ date "+%d %B %Y"
125920 May 2010
1260We can set the date and time as follows:
1261# date -s "Formatted date string"
1262For example:
1263# date -s "21 June 2009 11:01:22"
1264Sometimes we need to check the time taken by a set of commands. We can display it as follows:
1265#!/bin/bash
1266#Filename: time_take.sh
1267start=$(date +%s)
1268commands;
1269statements;
1270end=$(date +%s)
1271difference=$(( end - start))
1272echo Time taken to execute commands is $difference seconds.
1273An alternate method would be to use  timescriptpath to get the time that it took to execute
1274the script.
1275There's more...
1276Producing time intervals are essential when writing monitoring scripts that execute in a loop.
1277Let's see how to generate time delays.
1278Producing delays in a script
1279In order to delay execution in a script for some period of time, use  sleep:
1280$ sleep no_of_seconds .
1281For example, the following script counts from 0 to 40 by using  tput and  sleep :
1282#!/bin/bash
1283#Filename: sleep.sh
1284echo -n Count:
1285tput sc
1286count=0;
1287while true;
1288do
1289if [ $x -lt 40 ];
1290www.it-ebooks.info
1291Chapter 1
129233
1293then let count++;
1294sleep 1;
1295tput rc
1296tput ed
1297echo -n $count;
1298else exit 0;
1299fi
1300done
1301In the above example, a variable count is initialized to 0 and is incremented on every loop
1302execution. The  echo statement prints the text. We use  tput sc  to store the cursor position.
1303On every loop execution we write the new count in the terminal by restoring the cursor position
1304for the number. The cursor position is restored using  tput rc .  tput ed clears text from the
1305current cursor position to the end of the line, so that the older number can be cleared and the
1306count can be written. A delay of 1 second is provided in the loop by using the sleep command.
1307Debugging the script
1308Debugging is one of the critical features every programming language should implement to
1309produce race-back information when something unexpected happens. Debugging information
1310can be used to read and understand what caused the program to crash or to act in an
1311unexpected fashion. Bash provides certain debugging options that every sysadmin should
1312know. There are also some other tricky ways to debug.
1313Getting ready
1314No special utilities are required to debug shell scripts. Bash comes with certain flags that can
1315print arguments and inputs taken by the scripts. Let's see how to do it.
1316How to do it...
1317Add the  -x option to enable debug tracing of a shell script as follows:
1318$ bash -x script.sh
1319Running the script with the  -x flag will print each source line with current status. Note that
1320you can also use  sh –x script .
1321The  -x  flag outputs every line of script as it is executed to  stdout . However, we may require
1322only some portions of the source lines to be observed such that commands and arguments
1323are to be printed at certain portions. In such conditions we can use  set built-in to enable
1324and disable debug printing within the script.
1325f set -x : Displays arguments and commands upon their execution
1326f set +x : Disables debugging
1327www.it-ebooks.info
1328Shell Something Out
132934
1330f set –v : Displays input when they are read
1331f set +v : Disables printing input
1332For example:
1333#!/bin/bash
1334#Filename: debug.sh
1335for i in {1..6}
1336do
1337set -x
1338echo $i
1339set +x
1340done
1341echo "Script executed"
1342In the above script, debug information for  echo $i will only be printed as debugging is
1343restricted to that section using  -x and  +x .
1344The above debugging methods are provided by bash built-ins. But they always produce
1345debugging information in a fixed format. In many cases, we need debugging information in our
1346own format. We can set up such a debugging style by passing the  _DEBUG environment variable.
1347Look at the following example code:
1348#!/bin/bash
1349function DEBUG()
1350{
1351[ "$_DEBUG" == "on" ] && $@ || :
1352}
1353for i in {1..10}
1354do
1355DEBUG echo $i
1356done
1357We can run the above script with debugging set to "on" as follows:
1358$ _DEBUG=on ./script.sh
1359We prefix  DEBUG  before every statement where debug information is to be printed. If
1360_DEBUG=on  is not passed to script, debug information will not be printed. In Bash the
1361command ':' tells the shell to do nothing.
1362There's more...
1363We can also use other convenient ways to debug scripts. We can make use of shebang in a
1364trickier way to debug scripts.
1365www.it-ebooks.info
1366Chapter 1
136735
1368Shebang hack
1369The shebang can be changed from  #!/bin/bash to  #!/bin/bash –xv  to enable
1370debugging without any additional flags ( -xv flags themselves).
1371Functions and arguments
1372Like any other scripting languages, Bash also supports functions. Let's see how to define and
1373use functions.
1374How to do it...
1375A function can be defined as follows:
1376function fname()
1377{
1378statements;
1379}
1380Or alternately,
1381fname()
1382{
1383statements;
1384}
1385A function can be invoked just by using its name:
1386$ fname ; # executes function
1387Arguments can be passed to functions and can be accessed by our script:
1388fname arg1 arg2 ; # passing args
1389Following is the definition of the function  fname . In the  fname function, we have included
1390various ways of accessing the function arguments.
1391fname()
1392{
1393echo $1, $2; #Accessing arg1 and arg2
1394echo "$@"; # Printing all arguments as list at once
1395echo "$*"; # Similar to $@, but arguments taken as single entity
1396return 0; # Return value
1397}
1398www.it-ebooks.info
1399Shell Something Out
140036
1401Similarly, arguments can be passed to scripts and can be accessed by  script:$0 (the name
1402of the script):
1403f $1 is the first argument
1404f $2 is the second argument
1405f $n is the nth argument
1406f "$@"  expands as  "$1" "$2" "$3" and so on
1407f "$*"  expands as  "$1c$2c$3" , where  c is the first character of IFS
1408f "$@"  is the most used one.  "$*"  is used rarely since it gives all arguments as a
1409single string.
1410There's more...
1411Let's explore more tips on Bash functions.
1412Recursive function
1413Functions in Bash also support recursion (the function that can call itself). For example,
1414F() { echo $1; F hello; sleep 1; } .
1415Fork bomb
1416:(){ :|:& };:
1417This recursive function is a function that calls itself. It infinitely spawns
1418processes and ends up in a denial of service attack. & is postfixed with the
1419function call to bring the subprocess into the background. This is a dangerous
1420code as it forks processes and, therefore, it is called a fork bomb.
1421You may find it difficult to interpret the above code. See Wikipedia page
1422http://en.wikipedia.org/wiki/Fork_bomb for more details and
1423interpretation of the fork bomb.
1424It can be prevented by restricting the maximum number of processes that can
1425be spawned from the config file /etc/security/limits.conf.
1426Exporting functions
1427A function can be exported like environment variables using  export such that the scope of
1428the function can be extended to subprocesses, as follows:
1429export -f fname
1430www.it-ebooks.info
1431Chapter 1
143237
1433Reading command return value (status)
1434We can get the return value of a command or function as follows:
1435cmd;
1436echo $?;
1437$?  will give the return value of the command  cmd .
1438The return value is called exit status. It can be used to analyze whether a command
1439completed its execution successfully or unsuccessfully. If the command exits successfully,
1440the exit status will be zero, else it will be non-zero.
1441We can check whether a command terminated successfully or not as follows:
1442#!/bin/bash
1443#Filename: success_test.sh
1444CMD="command" #Substitute with command for which you need to test exit
1445status
1446$CMD
1447if [ $? –eq 0 ];
1448then
1449echo "$CMD executed successfully"
1450else
1451echo "$CMD terminated unsuccessfully"
1452fi
1453Passing arguments to commands
1454Arguments to commands can be passed in different formats. Suppose  –p and  -v are the
1455options available and  -k NO is another option that takes a number. Also the command takes
1456a filename as argument. It can be executed in multiple ways as follows:
1457$ command -p -v -k 1 file
1458Or:
1459$ command -pv -k 1 file
1460Or:
1461$ command -vpk 1 file
1462Or:
1463$ command file -pvk 1
1464www.it-ebooks.info
1465Shell Something Out
146638
1467Reading the output of a sequence
1468of commands
1469One of the best-designed features of shell scripting is the ease of combining many commands
1470or utilities to produce output. The output of one command can appear as the input of another,
1471which passes its output to another command, and so on. The output of this combination can
1472be read in a variable. This recipe illustrates how to combine multiple commands and how its
1473output can be read.
1474Getting ready
1475Input is usually fed into a command through  stdin  or arguments. Output appears as
1476stderr or  stdout . While we combine multiple commands, we usually use  stdin  to give
1477input and  stdout  for output.
1478Commands are called as filters. We connect each filter using pipes. The piping operator is " | ".
1479An example is as follows:
1480$ cmd1 | cmd2 | cmd3
1481Here we combine three commands. The output of  cmd1 goes to  cmd2 and output of  cmd2
1482goes to  cmd3 and the final output (which comes out of  cmd3 ) will be printed or it can be
1483directed to a file.
1484How to do it...
1485Have a look at the following code:
1486$ ls | cat -n > out.txt
1487Here the output of  ls  (the listing of the current directory) is passed to  cat -n .  cat –n  puts
1488line numbers to the input received through  stdin . Therefore, its output is redirected to the
1489out.txt file.
1490We can read the output of a sequence of commands combined by pipes as follows:
1491cmd_output=$(COMMANDS)
1492This is called the subshell method. For example:
1493cmd_output=$(ls | cat -n)
1494echo $cmd_output
1495www.it-ebooks.info
1496Chapter 1
149739
1498Another method, called back-quotes can also be used to store the command output as follows:
1499cmd_output=`COMMANDS`
1500For example:
1501cmd_output=`ls | cat -n`
1502echo $cmd_output
1503Back quote is different from the single quote character. It is the character on the ~ button in
1504the keyboard.
1505There's more...
1506There are multiple ways of grouping commands. Let's go through few of them.
1507Spawning a separate process with subshell
1508Subshells are separate processes. A subshell can be defined using the  ( ) operators as follows:
1509pwd;
1510(cd /bin; ls);
1511pwd;
1512When some commands are executed in a subshell none of the changes occur in the current
1513shell; changes are restricted to the subshell. For example, when the current directory in a
1514subshell is changed using the  cd command, the directory change is not reflected in the main
1515shell environment.
1516The  pwd  command prints the path of the working directory.
1517The  cd  command changes the current directory to the given directory path.
1518Subshell quoting to preserve spacing and newline character
1519Suppose we are reading the output of a command to a variable using a subshell or the back-
1520quotes method, we always quote them in double-quotes to preserve the spacing and newline
1521character (\n). For example:
1522$ cat text.txt
15231
15242
15253
1526$ out=$(cat text.txt)
1527$ echo $out
15281 2 3 # Lost \n spacing in 1,2,3
1529www.it-ebooks.info
1530Shell Something Out
153140
1532$ out="$(cat tex.txt)"
1533$ echo $out
15341
15352
15363
1537Reading "n" characters without pressing
1538Return
1539read is an important Bash command that can be used to read text from keyboard or standard
1540input. We can use  read to interactively read an input from the user, but  read is capable
1541of much more. Let's look at a new recipe to illustrate some of the most important options
1542available with the  read command.
1543Getting ready
1544Most of the input libraries in any programming language read the input from the keyboard; but
1545string input termination is done when  Return is pressed. There are certain critical situations
1546when  Return cannot be pressed, but the termination is done based on number of characters
1547or a single character. For example, in a game a ball is moved up when up + is pressed.
1548Pressing  + and then pressing  Return everytime to acknowledge the + press is not efficient.
1549The read command provides a way to accomplish this task without having to press  Return .
1550How to do it...
1551The following statement will read "n" characters from input into the variable  variable_name :
1552read -n number_of_chars variable_name
1553For example:
1554$ read -n 2 var
1555$ echo $var
1556Many other options are possible with  read . Let's see take a look at these.
1557Read a password in non-echoed mode as follows:
1558read -s var
1559Display a message with  read using:
1560read -p "Enter input:" var
1561www.it-ebooks.info
1562Chapter 1
156341
1564Read the input after a timeout as follows:
1565read -t timeout var
1566For example:
1567$ read -t 2 var
1568#Read the string that is typed within 2 seconds into variable var.
1569Use a delimiter character to end the input line as follows:
1570read -d delim_charvar
1571For example:
1572$ read -d ":" var
1573hello:#var is set to hello
1574Field separators and iterators
1575The Internal Field Separator is an important concept in shell scripting. It is very useful while
1576manipulating text data. We will now discuss delimiters that separate different data elements
1577from single data stream. An Internal Field Separator is a delimiter for a special purpose. An
1578Internal Field Separator (IFS) is an environment variable that stores delimiting characters. It
1579is the default delimiter string used by a running shell environment.
1580Consider the case where we need to iterate through words in a string or comma separated
1581values (CSV). In the first case we will use  IFS=" " and in the second, IFS="," . Let's see
1582how to do it.
1583Getting ready
1584Consider the case of CSV data:
1585data="name,sex,rollno,location"
1586#To read each of the item in a variable, we can use IFS.
1587oldIFS=$IFS
1588IFS=, now,
1589for item in $data;
1590do
1591echo Item: $item
1592done
1593IFS=$oldIFS
1594www.it-ebooks.info
1595Shell Something Out
159642
1597The output is as follows:
1598Item: name
1599Item: sex
1600Item: rollno
1601Item: location
1602The default value of IFS is a space component (newline, tab, or a space character).
1603When IFS is set as "," the shell interprets the comma as a delimiter character, therefore, the
1604$item variable takes substrings separated by a comma as its value during the iteration.
1605If IFS were not set as " , " then it would print the entire data as a single string.
1606How to do it...
1607Let's go through another example usage of IFS by taking  /etc/passwd  file into
1608consideration. In the  /etc/passwd  file, every line contains items delimited by  ":" . Each line
1609in the file corresponds to an attribute related to a user.
1610Consider the input: root:x:0:0:root:/root:/bin/bash . The last entry on each line
1611specifies the default shell for the user. In order to print users and their default shells, we
1612can use the IFS hack as follows:
1613#!/bin/bash
1614#Description: Illustration of IFS
1615line="root:x:0:0:root:/root:/bin/bash"
1616oldIFS=$IFS;
1617IFS=":"
1618count=0
1619for item in $line;
1620do
1621[ $count -eq 0 ] && user=$item;
1622[ $count -eq 6 ] && shell=$item;
1623let count++
1624done;
1625IFS=$oldIFS
1626echo $user\'s shell is $shell;
1627The output will be:
1628root's shell is /bin/bash
1629Loops are very useful in iterating through a sequence of values. Bash provides many types of
1630loops. Let's see how to use them.
1631www.it-ebooks.info
1632Chapter 1
163343
1634For loop:
1635for var in list;
1636do
1637commands; # use $var
1638done
1639list can be a string, or a sequence.
1640We can generate different sequences easily.
1641echo {1..50} can generate a list of numbers from 1 to 50
1642echo {a..z} or {A..Z} or we can generate partial list using  {a..h} . Similarly, by combining
1643these we can concatenate data.
1644In the following code, in each iteration, the variable  i will hold a character in the range  a to  z :
1645for i in {a..z}; do actions; done;
1646The  for loop can also take the format of the  for loop in C. For example:
1647for((i=0;i<10;i++))
1648{
1649commands; # Use $i
1650}
1651While loop:
1652while condition
1653do
1654commands;
1655done
1656For an infinite loop, use  true as the condition.
1657Until loop:
1658A special loop called  until is available with Bash. This executes the loop until the given
1659condition becomes true. For example:
1660x=0;
1661until [ $x -eq 9 ]; # [ $x -eq 9 ] is the condition
1662do let x++; echo $x;
1663done
1664www.it-ebooks.info
1665Shell Something Out
166644
1667Comparisons and tests
1668Flow control in a program is handled by comparison and test statements. Bash also comes
1669with several options to perform tests that are compatible with the UNIX system-level features.
1670Getting ready
1671We can use  if ,  if else , and logical operators to perform tests and certain comparison
1672operators to compare data items. There is also a command called  test available to perform
1673tests. Let's see how to use those commands.
1674How to do it...
1675If condition:
1676if condition;
1677then
1678commands;
1679fi
1680else if and else:
1681if condition;
1682then
1683commands;
1684elif condition;
1685then
1686commands
1687else
1688commands
1689fi
1690Nesting is also possible with if and else.  if conditions can be lengthy. We can use logical
1691operators to make them shorter as follows:
1692[ condition ] && action; # action executes if condition is true.
1693[ condition ] || action; # action executes if condition is false.
1694&& is the logical AND operation and  || is the logical OR operation. This is a very helpful trick
1695while writing Bash scripts. Now let's go into conditions and comparisons operations.
1696Mathematical comparisons:
1697Usually, conditions are enclosed in square brackets  [] . Note that there is a space between  [
1698or ] and operands. It will show an error if no space is provided. An example is as follows:
1699[ $var -eq 0 ] or [ $var -eq 0 ]
1700www.it-ebooks.info
1701Chapter 1
170245
1703Performing mathematical conditions over variables or values can be done as follows:
1704[ $var -eq 0 ] # It returns true when $var equal to 0.
1705[ $var -ne 0 ] # It returns true when $var not equals 0
1706Other important operators are:
1707f -gt : Greater than
1708f -lt : Less than
1709f -ge : Greater than or equal to
1710f -le : Less than or equal to
1711Multiple test conditions can be combined as follows:
1712[ $var1 -ne 0 -a $var2 -gt 2 ] # using AND -a
1713[ $var -ne 0 -o var2 -gt 2 ] # OR -o
1714Filesystem related tests:
1715We can test different filesystem related attributes using different condition flags as follows:
1716f [ -f $file_var ] : Returns true if the given variable holds a regular filepath or
1717filename.
1718f [ -x $var ] : Returns true if the given variable holds a file path or filename which
1719is executable.
1720f [ -d $var ] : Returns true if the given variable holds a directory path or directory
1721name.
1722f [ -e $var ] : Returns true if the given variable holds an existing file.
1723f [ -c $var ] : Returns true if the given variable holds path of a character device file.
1724f [ -b $var ] : Returns true if the given variable holds path of a block device file.
1725f [ -w $var ] : Returns true if the given variable holds path of a file which is writable.
1726f [ -r $var ] : Returns true if the given variable holds path of a file which is
1727readable.
1728f [ -L $var ] : Returns true if the given variable holds path of a symlink.
1729An example of the usage is as follows:
1730fpath="/etc/passwd"
1731if [ -e $fpath ]; then
1732echo File exists;
1733else
1734echo Does not exist;
1735fi
1736www.it-ebooks.info
1737Shell Something Out
173846
1739String comparisons:
1740While using string comparison, it is best to use double square brackets since use of single
1741brackets can sometimes lead to errors. Usage of single brackets sometimes lead to error. So it
1742is better to avoid them.
1743Two strings can be compared to check whether they are the same as follows;
1744f [[ $str1 = $str2 ]] : Returns true when str1 equals str2, that is, the text
1745contents of str1 and str2 are the same
1746f [[ $str1 == $str2 ]] : It is alternative method for string equality check
1747We can check whether two strings are not the same as follows:
1748f [[ $str1 != $str2 ]] : Returns true when str1 and str2 mismatches
1749We can find out the alphabetically smaller or larger string as follows:
1750f [[ $str1 > $str2 ]] : Returns true when str1 is alphabetically greater than str2
1751f [[ $str1 < $str2 ]] : Returns true when str1 is alphabetically lesser than str2
1752Note that a space is provided after and before  = . If space is not provided,
1753it is not a comparison, but it becomes an assignment statement.
1754f [[ -z $str1 ]] : Returns true if str1 holds an empty string
1755f [[ -n $str1 ]] : Returns true if str1 holds a non-empty string
1756It is easier to combine multiple conditions using the logical operators  && and  || as follows:
1757if [[ -n $str1 ]] && [[ -z $str2 ]] ;
1758then
1759commands;
1760fi
1761For example:
1762str1="Not empty "
1763str2=""
1764if [[ -n $str1 ]] && [[ -z $str2 ]];
1765then
1766echo str1 is non-empty and str2 is empty string.
1767fi
1768www.it-ebooks.info
1769Chapter 1
177047
1771The output is as follows:
1772str1 is non-empty and str2 is empty string.
1773The  test command can be used for performing condition checks. It helps to avoid usage
1774of many braces. The same set of test conditions enclosed within  [] can be used for the test
1775command.
1776For example:
1777if [ $var -eq 0 ]; then echo "True"; fi
1778can be written as
1779if test $var -eq 0 ; then echo "True"; fi
1780www.it-ebooks.info
1781www.it-ebooks.info
17822
1783Have a Good
1784Command
1785In this chapter, we will cover:
1786f Concatenating with cat
1787f Recording and playback of terminal sessions
1788f Finding files and file listing
1789f Command output as argument to a command (xargs)
1790f Translating with tr
1791f Checksum and verification
1792f Sorting, unique and duplicates
1793f Temporary file naming and random numbers
1794f Splitting files and data
1795f Slicing filenames based on extension
1796f Renaming files in bulk with rename and mv
1797f Spell check and dictionary manipulation
1798f Automating interactive input
1799www.it-ebooks.info
1800Have a Good Command
180150
1802Introduction
1803Commands are beautiful components of UNIX-like systems. They help us achieve many tasks
1804making our work easier. When you practise the use of commands everywhere, you will love
1805it. Many circumstances make you say "wow!". Once you've had a chance to try some of the
1806commands that Linux offers you to make your life easier and more productive, you'll wonder
1807how you did without using them before. Some of my personal favorite commands are  grep ,
1808awk ,  sed , and  find .
1809Using the UNIX/Linux command line is an art. You will get better at using it as you practice
1810and gain experience. This chapter will introduce you to some of the most interesting and
1811useful commands.
1812Concatenating with cat
1813cat is one of the first commands which a command line warrior must learn.  cat is a beautiful
1814and simple command. It is usually used to read, display, or concatenate the contents of a file,
1815but  cat is capable of more than just that.
1816Getting ready
1817We scratch our heads when we need to combine standard input data as well as data from a
1818file using a single-line command. The regular way of combining  stdin data as well as file data
1819is to redirect  stdin to a file and then append two files. But we can use the  cat command to
1820do it easily in a single invocation.
1821How to do it...
1822The  cat command is a very simple command that is used very frequently in daily life.  cat
1823stands for concatenate.
1824The general syntax of  cat  for reading a file's contents is:
1825$ cat file1 file2 file3 ...
1826This command outputs concatenated data from the files with file names provided as
1827command-line arguments. For example:
1828$ cat file.txt
1829This is a line inside file.txt
1830This is the second line inside file.txt
1831www.it-ebooks.info
1832Chapter 2
183351
1834How it works…
1835There are a lot of features that come along with  cat . Let's walk through several usage
1836techniques that are possible with  cat .
1837The  cat command not only can read from files and concatenate the data, but also can read
1838the input from the standard input.
1839In order to read from the standard input, use a pipe operator as follows:
1840OUTPUT_FROM_SOME COMMANDS | cat
1841Similarly, we can concatenate content from input files along with standard input using  cat .
1842Combine  stdin and data from another file, as follows:
1843$ echo 'Text through stdin' | cat – file.txt
1844In this code  - acts as filename for  stdin text.
1845There's more...
1846The  cat command has few other options for viewing files. Let's go through them.
1847Squeezing blank lines
1848Sometimes many empty lines in text need to be squeezed into one to make it readable or for
1849some other purpose. Squeeze adjacent blank lines in a text file by using the following syntax:
1850$ cat -s file
1851For example:
1852$ cat multi_blanks.txt
1853line 1
1854line2
1855line3
1856line4
1857$ cat -s multi_blanks.txt # Squeeze adjacent blank lines
1858line 1
1859line2
1860line3
1861www.it-ebooks.info
1862Have a Good Command
186352
1864line4
1865Alternately, we can remove all blank lines by using  tr as follows:
1866$ cat multi_blanks.txt | tr -s '\n'
1867line 1
1868line2
1869line3
1870line4
1871In the above usage of  tr , it squeezes adjacent ' \n' characters into a single ' \n'
1872(newline character).
1873Displaying tabs as ^I
1874It is hard to distinguish tabs and repeated space characters. While writing programs in
1875languages like Python, it keeps special meaning for tabs and spaces for indentation purposes.
1876They are treated differently. Therefore, the use of tab instead of spaces causes problems
1877in indentation. It may become difficult to track where the misplacement of the tab or space
1878occurred by looking through a text editor.  cat has a feature that can highlight tabs. This is
1879very helpful in debugging indentation errors. Use the  –T option with  cat to highlight tab
1880characters as ^I. An example is as follows:
1881$ cat file.py
1882def function():
1883var = 5
1884next = 6
1885third = 7
1886$ cat -T file.py
1887def function():
1888^Ivar = 5
1889next = 6
1890^Ithird = 7^I
1891Line numbers
1892Using the  –n flag for the  cat command will output each line with a line number prefixed. It
1893is to be noted that the  cat command never changes a file; instead it produces an output on
1894stdout with modifications to input according to the options provided. For example:
1895$ cat lines.txt
1896line
1897www.it-ebooks.info
1898Chapter 2
189953
1900line
1901line
1902$ cat -n lines.txt
19031 line
19042 line
19053 line
1906Recording and playback of terminal
1907sessions
1908When you need to show somebody how to do something in the terminal or you need to
1909prepare a tutorial on how to do something through command line, you would normally type the
1910commands manually and show them. Or you could record a screencast video and playback
1911the video to them. What if we can record the order and timing of the commands that we typed
1912before and replay these commands again so that others can watch as if they were typing?
1913The output of the commands gets displayed on the terminal until the playback is complete.
1914Sounds interesting? It can be done using the commands  script  and  scriptreplay .
1915Getting ready
1916script and  scriptreplay commands are available in most of the GNU/Linux distributions.
1917Recording the terminal sessions to a file will be interesting. You can create tutorials of
1918command-line hacks and tricks to achieve some task by recording the terminal sessions. You
1919can also share the recorded files for others to playback and see how to perform a particular
1920task using the command line.
1921How to do it...
1922We can start recording the terminal session as follows:
1923$ script -t 2> timing.log -a output.session
1924type commands;
1925…
1926..
1927exit
1928Two configuration files are passed to the  script command as arguments. One file is for
1929storing timing information ( timing.log ) at which each of the commands are run, whereas
1930the other file ( output.session ) is used for storing command output. The  -t flag is used to
1931dump timing data to  stderr .  2> is used to redirect  stderr to  timing.log .
1932www.it-ebooks.info
1933Have a Good Command
193454
1935By using the two files,  timing.log (stores timing information) and  output.session (stores
1936command output information), we can replay the sequence of command execution as follows:
1937$ scriptreplay timing.log output.session
1938# Plays the sequence of commands and output
1939How it works...
1940Usually, we record the desktop video to prepare tutorials. However, videos require good
1941amount of storage. But a terminal script file is just a text file. Therefore, it always has a file size
1942only in the order of Kilobytes.
1943You can share the files  timing.log and  output.session with anyone who wants to replay
1944a terminal session in their terminal.
1945The  script command can also be used to set up a terminal session that can be broadcasted
1946to multiple users. It is a very interesting experience. Let's see how to do it.
1947Open two terminals, Terminal1 and Terminal2.
19481. In Terminal1 enter the following command:
1949$ mkfifo scriptfifo
19502. In Terminal2 enter the following command:
1951$ cat scriptfifo
19523. Go back to Terminal1 and enter the following command:
1953$ script -f scriptfifo
1954$ commands;
1955When you need to end the session, type  exit  and press Return. It will show the message
1956"Script done, file is scriptfifo".
1957Now Terminal1 is the broadcaster and Terminal2 is the receiver.
1958When you type anything in real-time on Terminal1, it will be played on Terminal2 or any
1959terminal that supplies the following command:
1960cat scriptfifo
1961This method can be used when handling a tutorial session for many users in a computer lab
1962or over the Internet. It will save bandwidth as well as provide a real-time experience.
1963www.it-ebooks.info
1964Chapter 2
196555
1966Finding files and file listing
1967find is one of the great utilities in the UNIX/Linux command-line toolbox. It is a very useful
1968command for shell scripts, but most people do not use it effectively due to the lack of
1969understanding. This recipe deals with most of the use cases of  find and how it can be used
1970to solve problems of different criterions.
1971Getting ready
1972The  find command uses the following strategy:  find descends through a hierarchy of files,
1973matches the files that meet specified criteria, and performs some actions. Let's go through
1974different use cases of find and the basic usages.
1975How to do it...
1976In order to list all the files and folders from the current directory to the descending child
1977directories, use the following syntax:
1978$ find base_path
1979base_path can be any location from which the  find should start descending (for example,
1980/home/slynux/ ).
1981An example of this command is as follows:
1982$ find . -print
1983# Print lists of files and folders
1984. specifies current directory and  .. specifies the parent directory. This convention is followed
1985throughout the UNIX file system.
1986The  -print argument specifies to print the names (path) of the matching files. When  -print
1987is used  '\n' will be the delimiting character for separating each file.
1988The  -print0 argument specifies each matching file name printed with the delimiting
1989character  '\0' . This is useful when a filename contains a space character.
1990There's more...
1991In this recipe we have learned the usage of the most commonly-used  find command with an
1992example. The  find command is a powerful command-line tool and it is armed with a variety of
1993interesting options. Let's walk through some of these different options of the  find command.
1994www.it-ebooks.info
1995Have a Good Command
199656
1997Search based on file name or regular expression match
1998The  -name argument specifies a matching string for the filename. We can pass wildcards as
1999its argument text.  *.txt matches all the filenames ending with  .txt and prints them. The
2000–print option prints the filenames or file paths in the terminal that matches the conditions
2001(for example,  –name ) given as options to the  find command.
2002$ find /home/slynux -name "*.txt" –print
2003The  find command has an option  –iname (ignore case), which is similar to  -name .  –iname
2004matches the name ignoring the case.
2005For example:
2006$ ls
2007example.txt EXAMPLE.txt file.txt
2008$ find . -iname "example*" -print
2009./example.txt
2010./EXAMPLE.txt
2011If we want to match either of the multiple criterions, we can use OR conditions as shown below:
2012$ ls
2013new.txt some.jpg text.pdf
2014$ find . \( -name "*.txt" -o -name "*.pdf" \) -print
2015./text.pdf
2016./new.txt
2017The previous code will print all of the  .txt and  .pdf files, since the  find command
2018matches both  .txt and  .pdf files.  \( and \) is used to treat  -name "*.txt" -o -name
2019"*.pdf" as a single unit.
2020The  -path argument can be used to match the file path for files that match the wildcards.
2021-name always matches using the given filename. However,  -path matches the file path as a
2022whole. For example:
2023$ find /home/users -path "*slynux*" -print
2024This will match files as following paths.
2025/home/users/list/slynux.txt
2026/home/users/slynux/eg.css
2027The  -regex argument is similar to  -path , but  -regex matches the file paths based on
2028regular expressions.
2029www.it-ebooks.info
2030Chapter 2
203157
2032Regular expressions are an advanced form of wildcard matching. It enables to specify a text
2033with patterns. By using the patterns, we can make matches to the text and print them. A
2034typical example of text matching using regular expressions is: parsing all e-mail addresses
2035from a given pool of text. An e-mail address takes the form  name@host.root . So, it can be
2036generalized as  [a-z0-9]+@[a-z0-9]+.[a-z0-9]+ . The  + signifies that the previous class
2037of characters can occur one or more times, repeatedly, in the characters that follow.
2038The following command matches  .py or  .sh files:
2039$ ls
2040new.PY next.jpg test.py
2041$ find . -regex ".*\(\.py\|\.sh\)$"
2042./test.py
2043Similarly, using  -iregex ignores the case for the regular expressions that are available.
2044For example:
2045$ find . -iregex ".*\(\.py\|\.sh\)$"
2046./test.py
2047./new.PY
2048Negating arguments
2049find can also take negation of arguments using "!". For example:
2050$ find . ! -name "*.txt" -print
2051The above  find construct matches all the file names, as long as the name does not end with
2052.txt . The following example shows the result of the command:
2053$ ls
2054list.txt new.PY new.txt next.jpg test.py
2055$ find . ! -name "*.txt" -print
2056.
2057./next.jpg
2058./test.py
2059./new.PY
2060Search based on the directory depth
2061When the  find command is used it recursively walks through all the subdirectories as much
2062as possible until it reaches the leaf of the subdirectory tree. We can restrict the depth to which
2063the  find command traverses using some depth parameters given to the find.  -maxdepth
2064and  -mindepth  are the parameters.
2065www.it-ebooks.info
2066Have a Good Command
206758
2068In most of the cases, we need to search only in the current directory. It should not further
2069descend into the subdirectories from the current directory. In such cases, we can restrict the
2070depth to which the  find command should descend using depth parameters. In order to restrict
2071find from descending into the subdirectories from the current directory, the depth can be set
2072as 1. When we need to descend to two levels, the depth is set as 2, and so on for the rest of
2073the levels.
2074For specifying the maximum depth we use the  –maxdepth level parameter. Similarly,
2075we can also specify the minimum level at which the descending should start. If we want to
2076start searching from the second level onwards, we can set the minimum depth using the
2077–mindepth level parameter. Restrict the  find command to descend to a maximum depth
2078of 1, by using the following command:
2079$ find . -maxdepth 1 -type f -print
2080This command lists all the regular files only from the current directory. If there are
2081subdirectories, they are not printed or traversed. Similarly,  -maxdepth 2 traverses up to at
2082most two descending levels of subdirectories.
2083-mindepth is similar to  –maxdepth , but it sets the least depth level for the  find traversal. It
2084can be used to find and print the files that are located with a minimum level of depth from the
2085base path. For example, to print all the files that are at least two subdirectories distant from
2086the current directory use the following command:
2087$ find . -mindepth 2 -type f -print
2088./dir1/dir2/file1
2089./dir3/dir4/f2
2090Even if there are files in the current directory or  dir1 and  dir3 , it will not be printed.
2091-maxdepth and –mindepth should be specified as the third argument to
2092the find. If they are specified as the fourth or further arguments, it may affect
2093the efficiency of the find as it has to do unnecessary checks (for example,
2094if –maxdepth is specified as the fourth argument and –type as the third
2095argument, the find command first finds out all the files having the specified
2096–type and then finds all of the matched files having the specified depth.
2097However, if the depth were specified as the third argument and –type as the
2098fourth, find could collect all the files having at most the specified depth and
2099then check for the file type, which is the most efficient way of searching.
2100Search based on file type
2101UNIX-like operating systems consider every object as a file. There are different kinds of files
2102such as regular file, directory, character devices, block devices, symlinks, hardlinks, sockets,
2103FIFO, and so on.
2104The file search can be filtered out using the  -type  option. By using  –type , we can specify to
2105the  find command that it should only match files having a specified type.
2106www.it-ebooks.info
2107Chapter 2
210859
2109List only directories including descendants as follows:
2110$ find . -type d -print
2111It is hard to list directories and files separately. But  find helps to do it. List only regular files
2112as follows:
2113$ find . -type f -print
2114List only symbolic links as follows:
2115$ find . -type l -print
2116You can use the  type arguments from the following table to properly match the required
2117file type:
2118File type Type argument
2119Regular file f
2120Symbolic link l
2121Directory d
2122Character special device c
2123Block device b
2124Socket s
2125Fifo p
2126Search on up file times
2127UNIX/Linux file systems have three types of timestamp on each file. They are as follows:
2128f Access time (- atime ): This is the last timestamp of when the file was accessed
2129by some user
2130f Modification time (- mtime ): This is the last timestamp of when the file content
2131was modified
2132f Change time (- ctime ): This is the last timestamp of when the metadata for a file
2133(such as permissions or ownership) was modified
2134There is nothing called creation time in UNIX.
2135-atime ,  -mtime ,  -ctime are the time parameter options available with  find . They can be
2136specified with integer values in "number of days". These integer values are often attached with
2137- or + signs. The  - sign implies less than whereas the  + implies greater than. For example:
2138f Print all the files that were accessed within the last 7 days as follows:
2139$ find . -type f -atime -7 -print
2140www.it-ebooks.info
2141Have a Good Command
214260
2143f Print all the files that are having access time exactly 7 days old as follows:
2144$ find . -type f -atime 7 -print
2145f Print all the files that are having access time older than 7 days as follows:
2146$ find . -type f -atime +7 -print
2147Similarly, we can use the  –mtime parameter for search files based on modification time and
2148-ctime for search based on change time.
2149-atime ,  -mtime , and  –ctime are time-based parameters that use the time metric in days.
2150There are some other time-based parameters that use the time metric in minutes. These are
2151as follows:
2152f -amin (access time)
2153f -mmin (modification time)
2154f -cmin (change time)
2155For example:
2156In order to print all the files that are having access time older than seven minutes, use the
2157following command:
2158$ find . -type f -amin +7 -print
2159Another nice feature available with  find is the  –newer parameter. By using  -newer , we can
2160specify a reference file to compare with the timestamp. We can find all the files that are newer
2161(older modification time) than the specified file with the  –newer parameter.
2162For example, find all the files that are having a modification time greater than that of the
2163modification time of a given  file.txt file as follows:
2164$ find . -type f -newer file.txt -print
2165Timestamp manipulation flags for the  find command are very useful for writing system
2166backup and maintenance scripts.
2167Search based on file size
2168Based on the file sizes of the files, a search can be performed as follows:
2169$ find . -type f -size +2k
2170# Files having size greater than 2 kilobytes
2171$ find . -type f -size -2k
2172# Files having size less than 2 kilobytes
2173$ find . -type f -size 2k
2174# Files having size 2 kilobytes
2175www.it-ebooks.info
2176Chapter 2
217761
2178Instead of  k we can use different size units as the following:
2179f b – 512 byte blocks
2180f c – bytes
2181f w – two byte words
2182f k – Kilobyte
2183f M – Megabyte
2184f G – Gigabyte
2185Deleting based on the file matches
2186The  -delete flag can be used to remove files that are matched by  find .
2187Remove all the .swp files from the current directory as follows:
2188$ find . -type f -name "*.swp" -delete
2189Match based on the file permissions and ownership
2190It is possible to match files based on the file permissions. We can list out the files having
2191specified file permission as follows:
2192$ find . -type f -perm 644 -print
2193# Print files having permission 644
2194As an example usage case, we can consider the case of Apache web server. The PHP files in
2195the web server require proper permissions to execute. We can find out the PHP files that are
2196not having proper execute permissions as follows:
2197$ find . –type f –name "*.php" ! -perm 644 –print
2198We can also search files based on ownership of the files. The files owned by a specific user
2199can be found out using the  -user USER option.
2200The  USER argument can be a username or UID.
2201For example, to print the list of all files owned by the user slynux, you can use the
2202following command:
2203$ find . -type f -user slynux -print
2204Executing commands or actions with find
2205The  find command can be coupled with many of the other commands using the  -exec
2206option.  -exec is one of the most powerful features that comes with  find .
2207Let's see how to use the – exec option.
2208www.it-ebooks.info
2209Have a Good Command
221062
2211Consider the example in the previous section. We used  –perm to find out the files that do not
2212have proper permissions. Similarly, in the case where we need to change the ownership of all
2213files owned by a certain user (for example,  root ) to another user (for example,  www-data the
2214default Apache user in the web server), we can find all the files owned by root by using the  –
2215user option and using  –exec to perform ownership change operation.
2216You must run the find command as root for performing ownership change.
2217Let's have a look at the following example:
2218# find . -type f –user root –exec chown slynux {} \;
2219In this command,  {} is a special string used with the  –exec option. For each file match,  {}
2220will be replaced with the file name in place for  –exec . For example, if the  find command
2221finds two files  test1.txt and  test2.txt with owner slynux, the find command will perform:
2222chown slynux {}
2223This gets resolved to  chown slynux test1.txt and  chown slynux test2.txt .
2224Another usage example is to concatenate all the C program files in a given directory and write
2225it to a single file  all_c_files.txt . We can use  find to match all the C files recursively and
2226use the  cat command with the  -exec flag as follows:
2227$ find . -type f -name "*.c" -exec cat {} \;>all_c_files.txt
2228-exec is followed with any command.  {}  is a match. For every matched filename,  {} is
2229replaced with filename.
2230To redirect the data from  find to the  all_c_files.txt file, we used the  > operator instead
2231of  >> (append) because the entire output from the  find command is a single data stream
2232( stdin ).  >> is necessary only when multiple data streams are to be appended to a single file.
2233For example, to copy all the  .txt files that are older than 10 days to a directory  OLD , use the
2234following command:
2235$ find . -type f -mtime +10 -name "*.txt" -exec cp {} OLD \;
2236Similarly, the  find command can be coupled with many other commands.
2237-exec with multiple commands
2238We cannot use multiple commands along with the –exec parameter. It
2239accepts only a single command, but we can use a trick. Write multiple
2240commands in a shell script (for example, commands.sh) and use it
2241with –exec as follows:
2242–exec ./commands.sh {} \;
2243www.it-ebooks.info
2244Chapter 2
224563
2246-exec can be coupled with  printf to produce a very useful output. For example:
2247$ find . -type f -name "*.txt" -exec printf "Text file: %s\n" {} \;
2248Skip specified directories from the find
2249Skipping certain subdirectories for a performance improvement is sometimes required while
2250doing a directory search and performing some action. For example, when programmers look
2251for particular files on a development source tree, which is under a version control system
2252such as Git, the source hierarchy will always contain the  .git  directory in each of the
2253subdirectories ( .git stores version control related information for every directory). Since
2254version control related directories do not produce useful output, they should be excluded
2255from the search. The technique of excluding files and directories from the search is known as
2256pruning. It can be performed as follows:
2257$ find devel/source_path \( -name ".git" -prune \) -o \( -type f -print \)
2258# Instead of \( -type -print \), use required filter.
2259The above command prints the name (path) of all the files that are not from the
2260.git directories.
2261Here,  \( -name ".git" -prune \) is the exclude portion, which specifies that the  .git
2262directory should be excluded and  \( -type f -print \)  specifies the action to be
2263performed. The actions to be performed are placed in the second block  -type f –print
2264(the action specified here is to print the names and path of all the files).
2265Playing with xargs
2266We use pipes to redirect  stdout  (standard output) of a command to  stdin (standard input)
2267of another command. For example:
2268cat foo.txt | grep "test"
2269But, some of the commands accept data as command-line arguments rather than a data
2270stream through  stdin (standard input). In that case, we cannot use pipes to supply data
2271through command-line arguments.
2272We should go for alternate methods.  xargs is a command that is very helpful in handling
2273standard input data to the command-line argument conversions.  xargs can manipulate
2274stdin and convert to command-line arguments for the specified command. Also  xargs
2275can convert any one line or multiple line text input into other formats, such as multiple lines
2276(specified number of columns) or a single line and vice versa.
2277www.it-ebooks.info
2278Have a Good Command
227964
2280All the Bash hackers love one-line commands. One-liners are command sequences that are
2281joined by using the pipe operator, but do not use the semi colon terminator (;) between the
2282commands used. Crafting one-line commands makes tasks efficient and simpler to solve. It
2283requires proper understanding and practise to formulate one-liners for solving text processing
2284problems.  xargs is one of the important components for building one-liner commands.
2285Getting ready
2286The  xargs command should always appear immediately after a pipe operator.  xargs uses
2287standard input as the primary data stream source. It uses  stdin and executes another
2288command by providing command-line arguments for that executing command using the stdin
2289data source. For example:
2290command | xargs
2291How to do it...
2292The  xargs command can supply arguments to a command by reformatting the data received
2293through  stdin .
2294xargs can act as a substitute that can perform similar actions as the  -exec argument in
2295the case of the  find command. Let's see a variety of hacks that can be performed using the
2296xargs command.
2297f Converting multiple lines of input to a single line output:
2298Multiple line input can be converted simply by removing the new line character and
2299replacing with the " " (space) character. ' \n ' is interpreted as a newline, which is the
2300delimiter for the lines. By using  xargs , we can ignore all the newlines with spaces so
2301that multiple lines can be converted into a single line text as follows:
2302$ cat example.txt # Example file
23031 2 3 4 5 6
23047 8 9 10
230511 12
2306$ cat example.txt | xargs
23071 2 3 4 5 6 7 8 9 10 11 12
2308f Converting single line into multiple line output:
2309Given maximum no of arguments in a line  = n , we can split any  stdin (standard
2310input) text into lines of n arguments each. An argument is a piece of string delimited
2311by " " (space). Space is the default delimiter. A single line can be split into multiple
2312lines as follows:
2313www.it-ebooks.info
2314Chapter 2
231565
2316$ cat example.txt | xargs -n 3
23171 2 3
23184 5 6
23197 8 9
232010 11 12
2321How it works…
2322The  xargs command is appropriate to be applied to many problem scenarios with its rich and
2323simple options. Let's see how these options can be used wisely to solve problems.
2324We can also use our own delimiter towards separating arguments. In order to specify a custom
2325delimiter for input, use the  –d option as follows:
2326$ echo "splitXsplitXsplitXsplit" | xargs -d X
2327split split split split
2328In the above code,  stdin contains a string consisting of multiple 'X' characters. We can use
2329'X' as the input delimiter by using it with  –d . Here we have explicitly specified X as the input
2330delimiter, whereas in the default case  xargs takes Internal Field Separator (space) as the
2331input delimiter.
2332By using  –n along with the above command, we can split the input into multiple lines having
2333two words each as follows:
2334$ echo "splitXsplitXsplitXsplit" | xargs -d X -n 2
2335split split
2336split split
2337There's more...
2338We have learned how to format  stdin to different output as arguments from the above
2339examples. Now let's learn how to supply these formatted output as arguments to commands.
2340Passing formatted arguments to a command by reading stdin
2341Write a small custom echo for better understanding of example usages with xargs to provide
2342command arguments.
2343#!/bin/bash
2344#Filename: cecho.sh
2345echo $*'#'
2346www.it-ebooks.info
2347Have a Good Command
234866
2349When arguments are passed to the  cecho.sh , it will print the arguments terminated by the  #
2350character. For example:
2351$ ./cecho.sh arg1 arg2
2352arg1 arg2 #
2353Let's have a look at a problem:
2354f I have a list of arguments in a file (one argument in each line) to be provided to a
2355command (say  cecho.sh ). I need to provide arguments in two methods. In the first
2356method, I need to provide one argument each for the command as follows:
2357./cecho.sh arg1
2358./cecho.sh arg2
2359./cecho.sh arg3
2360Or, alternately, I need to provide two or three arguments each for each execution of
2361command. For two arguments each, it would be similar to the following:
2362./cecho.sh arg1 arg2
2363./cecho.sh arg3
2364f In the second method, I need to provide all arguments at once to the command
2365as follows:
2366./cecho.sh arg1 arg2 arg3
2367Run the above commands and note down the output before going through the following section.
2368The above problems can be solved using  xargs . We have the list of arguments in a file called
2369args.txt . The contents are as follows:
2370$ cat args.txt
2371arg1
2372arg2
2373arg3
2374For the first problem, we can execute the command multiple times with one argument per
2375execution, by using:
2376$ cat args.txt | xargs -n 1 ./cecho.sh
2377arg1 #
2378arg2 #
2379arg3 #
2380For executing a command with X arguments per each execution, use:
2381INPUT | xargs –n X
2382www.it-ebooks.info
2383Chapter 2
238467
2385For example:
2386$ cat args.txt | xargs -n 2 ./cecho.sh
2387arg1 arg2 #
2388arg3 #
2389For the second problem, we can execute the command at once with all the arguments, by using:
2390$ cat args.txt | xargs ./ccat.sh
2391arg1 arg2 arg3 #
2392In the above examples, we have supplied command-line arguments directly to a specific
2393command (for example,  cecho.sh ). We could only supply the arguments from the  args.
2394txt file. However, in realtime, we may also need to add some constant parameter with
2395the command (for example,  cecho.sh ) along with the arguments taken from  args.txt .
2396Consider the following example with the format:
2397./cecho.sh –p arg1 –l
2398In the above command execution  arg1 is the only variable text. All others should remain
2399constant. We should read arguments from a file ( args.txt ) and supply it as:
2400./cecho.sh –p arg1 –l
2401./cecho.sh –p arg2 –l
2402./cecho.sh –p arg3 –l
2403To provide a command execution sequence as shown,  xargs has an option  –I . By using  –I
2404we can specify a replacement string that will be replaced while  xargs expands. When  –I is
2405used with  xargs , it will execute as one command execution per argument.
2406Let's do it as follows:
2407$ cat args.txt | xargs -I {} ./cecho.sh -p {} -l
2408-p arg1 -l #
2409-p arg2 -l #
2410-p arg3 -l #
2411-I {} specifies the replacement string. For each of the arguments supplied for the
2412command, the  {} string will be replaced with arguments read through  stdin . When used
2413with  -I , the command is executed like in a loop. When there are three arguments the
2414command is executed three times along with the command  {} . Each time  {} is replaced with
2415arguments one by one.
2416www.it-ebooks.info
2417Have a Good Command
241868
2419Using xargs with find
2420xargs and  find are best friends. They can be combined to perform tasks easily. Usually,
2421people combine them in a wrong way. For example:
2422$ find . -type f -name "*.txt" -print | xargs rm -f
2423This is dangerous. It may sometimes cause removal of unnecessary files. Here, we cannot
2424predict the delimiting character (whether it is  '\n' or  ' ' ) for the output of the  find
2425command. Many of the filenames may contain a space character (' ') and hence  xargs may
2426misinterpret it as a delimiter (for example, "hell text.txt" is misinterpreted by  xargs as "hell"
2427and "text.txt").
2428Hence we must use  -print0 along with  find to produce an output with delimited character
2429null ( '\0' ) whenever we use the  find output as the  xargs input.
2430Let's use  find to match and list of all the  .txt files and remove them using  xargs :
2431$ find . -type f -name "*.txt" -print0 | xargs -0 rm -f
2432This removes all  .txt files.  xargs -0 interprets that the delimiting character is  \0 .
2433Counting number of lines of C code in a source code directory
2434over many C files.
2435This is a task most programmers do, that is, counting all C program files for LOC (Lines of
2436Code). The code for this task is as follows:
2437$ find source_code_dir_path -type f -name "*.c" -print0 | xargs -0 wc -l
2438While and subshell trick with stdin
2439xargs  is restricted to provide arguments in limited ways to supply arguments. Also, xargs
2440cannot supply arguments to multiple set of commands. For executing commands with
2441collected arguments from standard input, we have a very flexible method. I call it a subshell
2442hack. A subshell with a  while loop can be used to read arguments and execute commands in
2443a trickier way as follows:
2444$ cat files.txt | ( while read arg; do cat $arg; done )
2445# Equivalent to cat files.txt | xargs -I {} cat {}
2446Here, by replacing  cat $arg with any number of commands using a  while loop, we can
2447perform many command actions with same arguments. We can also pass the output to
2448other commands without using pipes. Subshell  ( ) tricks can be used in a variety of problem
2449environments. When enclosed within subshell operators, it acts as a single unit with multiple
2450commands inside.
2451$ cmd0 | ( cmd1;cmd2;cmd3) | cmd4
2452If  cmd1 is  cd / , within the subshell, the path of the working directory changes. However, this
2453change resides inside the subshell only.  cmd4 will not see the directory change.
2454www.it-ebooks.info
2455Chapter 2
245669
2457Translating with tr
2458tr is a small and beautiful command in the UNIX command-warrior toolkit. It is one of the
2459important commands frequently used to craft beautiful one-liner commands.
2460tr can be used to perform substitution of characters, deletion of the characters, and
2461squeezing of repeated characters from the standard input. It is often called translate, since it
2462can translate a set of characters to another set.
2463Getting ready
2464tr accepts input only through  stdin (standard input). It cannot accept input through
2465command-line arguments. It has the following invocation format:
2466tr [options] set1 set2
2467Input characters from  stdin are mapped from  set1 to  set2 and the output is written to
2468stdout (standard output).  set1 and  set2 are character classes or a set of characters.
2469If the length of sets is unequal,  set2 is extended to the length of  set1 by repeating the
2470last character, or else, if the length of  set2 is greater than that of  set1 , all the characters
2471exceeding the length of  set1 are ignored from  set2 .
2472How to do it...
2473In order to perform translation of characters in the input from uppercase to lowercase, use the
2474following command:
2475$ echo "HELLO WHO IS THIS" | tr 'A-Z' 'a-z'
2476'A-Z' and  'a-z' are the sets. We can specify custom sets as needed by appending
2477characters or character classes.
2478'ABD-}' ,  'aA.,' ,  'a-ce-x' ,  'a-c0-9' , and so on are valid sets. We can define sets
2479easily. Instead of writing continuous character sequences, we can use the  'startchar-
2480endchar' format. It can also be combined with any other characters or character classes. If
2481startchar-endchar are not a valid continuous character sequence, then they are taken
2482as a set of three characters (for example,  startchar ,  - , and  endchar ). You can also use
2483special characters such as  '\t' , '\n' , or any ASCII characters.
2484www.it-ebooks.info
2485Have a Good Command
248670
2487How it works…
2488By using  tr with the concept of sets, we can map characters from one set to another set easily.
2489Let's go through an example on how to use  tr for encrypting and decrypting numeric characters:
2490$ echo 12345 | tr '0-9' '9876543210'
249187654 #Encrypted
2492$ echo 87654 | tr '9876543210' '0-9'
249312345 #Decrypted
2494Let's try another interesting example.
2495ROT13 is a well known encryption algorithm. In the ROT13 scheme, the same function is used
2496to encrypt and decrypt text. The ROT13 scheme performs alphabetic rotation of characters for
249713 characters. Let's perform ROT13 using  tr as follows:
2498$ echo "tr came, tr saw, tr conquered." | tr
2499'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'
2500'NOPQRSTUVWXYZABCDEFGHIJKLMnopqrstuvwxyzabcdefghijklm'
2501The output will be:
2502ge pnzr, ge fnj, ge pbadhrerq.
2503By sending the encrypted text again to the same ROT13 function, we get:
2504$ echo ge pnzr, ge fnj, ge pbadhrerq. | tr
2505'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz'
2506'NOPQRSTUVWXYZABCDEFGHIJKLMnopqrstuvwxyzabcdefghijklm'
2507The output will be:
2508tr came, tr saw, tr conquered.
2509tr can be used to convert tab characters into space as follows:
2510$ cat text | tr '\t' ' '
2511There's more...
2512Deleting characters using tr
2513tr has an option  -d to delete a set of characters that appear on  stdin by using the specified
2514set of characters to be deleted as follows:
2515$ cat file.txt | tr -d '[set1]'
2516#Only set1 is used, not set2
2517www.it-ebooks.info
2518Chapter 2
251971
2520For example:
2521$ echo "Hello 123 world 456" | tr -d '0-9'
2522Hello world
2523# Removes the numbers from stdin and print
2524Complementing character set
2525We can use a set complement to  set1 by using the  -c flag.  -c [set] is equivalent to
2526specifying a set (complement-set) which contains complement characters of  [set] :
2527tr -c [set1] [set2]
2528The complement of  set1 means that it is the set having all the characters except for
2529characters in  set1 .
2530The best usage example is to delete all the characters from the input text except the ones
2531specified in the complement set. For example:
2532$ echo hello 1 char 2 next 4 | tr -d -c '0-9 \n'
25331 2 4
2534Here, the complement set is the set containing all numerals, the space character, and
2535newline. All other characters are removed since  –d is used with  tr .
2536Squeezing characters with tr
2537The  tr command is very helpful in many text processing contexts. Repeated continuous
2538characters should be squeezed to a single character in many circumstances. Squeezing of
2539whitespace is a frequently occurring task.
2540tr  provides the  –s option to squeeze repeating characters from the input. It can be
2541performed as follows:
2542$ echo "GNU is not UNIX. Recursive right ?" | tr -s ' '
2543GNU is not UNIX. Recursive right ?
2544# tr -s '[set]'
2545Let's use  tr in a tricky way to add a given list of numbers from a file as follows:
2546$ cat sum.txt
25471
25482
25493
25504
25515
2552$ cat sum.txt | echo $[ $(tr '\n' '+' ) 0 ]
255315
2554www.it-ebooks.info
2555Have a Good Command
255672
2557How does this hack work?
2558Here, the  tr command is used to replace  '\n' with the  '+' character, hence we form the
2559string  "1+2+3+..5+", but at the end of the string we have an extra  + operator. In order to
2560nullify the effect of the  + operator,  0 is appended.
2561$[ operation ] performs a numeric operation. Hence it forms the string as follows:
2562echo $[ 1+2+3+4+5+0 ]
2563If we use a loop to perform addition by reading numbers from a file, it would take few lines of
2564code. Here a one-liner does the trick. The skill of crafting one-liners is attained by practice.
2565Character classes
2566tr can use different character classes as sets. The different classes are as follows:
2567f alnum : Alphanumeric characters
2568f alpha : Alphabetic characters
2569f cntrl : Control (non-printing) characters
2570f digit : Numeric characters
2571f graph : Graphic characters
2572f lower : Lower-case alphabetic characters
2573f print : Printable characters
2574f punct : Punctuation characters
2575f space : Whitespace characters
2576f upper : Upper-case characters
2577f xdigit : Hexadecimal characters
2578We can select the required classes and use them with as follows:
2579tr [:class:] [:class:]
2580For example:
2581tr '[:lower:]' '[:upper:]'
2582Checksum and verification
2583Checksum programs are used to generate checksum key strings from the files and verify
2584the integrity of the files later by using that checksum string. A file might be distributed over
2585the network or any storage media to different destinations. Due to many reasons, there are
2586chances for the file being corrupted due to a few bits missing during the data transfer. These
2587errors happen most often while downloading the files from the Internet, transferring through
2588the network, CD ROM damage, and so on.
2589www.it-ebooks.info
2590Chapter 2
259173
2592Hence, we need to know whether the received file is the correct one or not by applying some
2593kind of test. The special key string that is used for this file integrity test is known as checksum.
2594We calculate the checksum for the original file as well as the received file. By comparing both
2595of the checksums, we can verify whether the received file is the correct one or not. If the
2596checksums (calculated from original file at the source location and the one calculated from
2597destination) are equal, it means that we have received the correct file without causing any
2598erroneous data loss during the data transfer, or else, the user has to repeat the data transfer
2599and try the checksum comparison again.
2600Checksums are crucial while writing backup scripts or maintenance scripts that consist of
2601transfer of files through the network. By using checksum verification, files corrupted during the
2602data transfer over the network can be identified and those files can be resend again from the
2603source to the destination. Thus the integrity of the data received can always be ensured.
2604Getting ready
2605The most famous and widely-used checksum techniques are  md5sum and  sha1sum . They
2606generate checksum strings by applying the corresponding algorithm to the file content. Let's
2607see how can we generate a checksum and from a file and verify the integrity of a file.
2608How to do it...
2609In order to compute the  md5sum , use the following command:
2610$ md5sum filename
261168b329da9893e34099c7d8ad5cb9c940 filename
2612An  md5sum is a 32 character hexadecimal string, as given above.
2613We redirect the checksum output into a file and use that MD5 file for verification as follows:
2614$ md5sum filename > file_sum.md5
2615How it works...
2616The syntax for  md5sum checksum calculation is as follows:
2617$ md5sum file1 file2 file3 ..
2618When multiple files are used, the output will contain a checksum for each of the file having
2619one checksum string per line, as follows:
2620[checksum1] file1
2621[checksum1] file2
2622[checksum1] file3
2623www.it-ebooks.info
2624Have a Good Command
262574
2626The integrity of a file can be verified by using the generated file as follows:
2627$ md5sum -c file_sum.md5
2628# It will output message whether checksum matches or not
2629Or, alternately, if we need to check all the files using all  .md5 info available, use:
2630$ md5sum *.md5
2631SHA1 is another commonly used checksum algorithm like md5sum. It generates a
263240-character hex code from a given input file. The command used for calculating a SHA1
2633string is  sha1sum . Its usage is very similar to that of  md5sum . Replace  md5sum with  sha1sum
2634in all the commands mentioned previously in this recipe. Instead of  file_sum.md5 , change
2635the output filename to  file_sum.sha1 .
2636Checksum verification is much useful to verify the integrity of files that we download from the
2637Internet. The ISO images that we download from the Internet are usually much more prone
2638to erroneous bits. Therefore, to check whether we received the file correctly, checksums are
2639widely used. For the same file data the checksum program will always produce the same
2640checksum string.
2641There's more...
2642Checksum are also useful when used with a number of files. Let's see how to apply checksum
2643to many files and verify correctness.
2644Checksum for directories
2645Checksums are calculated for files. Calculating the checksum for a directory would mean that
2646we will need to calculate the checksums for all the files in the directory, recursively.
2647It can be achieved by the command  md5deep or  sha1deep . Install the package  md5deep to
2648make these commands available. An example of this command is as follows:
2649$ md5deep -rl directory_path > directory.md5
2650# -r for enable recursive.
2651# -l for using relative path. By default it writes absolute file path in
2652output
2653Alternately, combine it with  find to calculate checksums recursively:
2654$ find directory_path -type f -print0 | xargs -0 md5sum >> directory.md5
2655To verify, use the following command:
2656$ md5sum -c directory.md5
2657www.it-ebooks.info
2658Chapter 2
265975
2660Sorting, unique and duplicates
2661Sorting is a common task that we always encounter with text files. Hence, in text processing
2662tasks, sort is very useful.  sort commands help us to perform sort operations over text files
2663and  stdin . Most often, it can also be coupled with many other commands to produce the
2664required output.  uniq is another command that is often used along with a  sort command.
2665It helps to extract unique lines from a text or  stdin .  sort and  uniq can be coupled to find
2666duplicates. This recipe illustrates most of the use cases with  sort and  uniq commands.
2667Getting ready
2668The  sort command accepts input as filenames as well as from  stdin (Standard input)
2669and outputs the result by writing into  stdout . The  uniq command follows the same
2670sequence of operation.
2671How to do it...
2672We can easily sort a given set of files (for example,  file1.txt and  file2.txt ) as follows:
2673$ sort file1.txt file2.txt .. > sorted.txt
2674Or:
2675$ sort file1.txt file2.txt .. -o sorted.txt
2676In order to find the unique lines from a sorted file, use:
2677$ cat sorted_file.txt | uniq> uniq_lines.txt
2678How it works…
2679There are numerous scenarios where  sort and  uniq commands can be used. Let's go
2680through various options and usage techniques.
2681For numerical sort use:
2682$ sort -n file.txt
2683To sort in reverse order use:
2684$ sort -r file.txt
2685For sorting by months (in the order Jan, Feb, March) use:
2686$ sort -M months.txt
2687www.it-ebooks.info
2688Have a Good Command
268976
2690A file can be tested whether sorted or not as follows:
2691#!/bin/bash
2692#Desc: Sort
2693sort -C file ;
2694if [ $? -eq 0 ]; then
2695echo Sorted;
2696else
2697echo Unsorted;
2698fi
2699# If we are checking numerical sort, it should be sort -nC
2700In order to merge two sorted files without sorting again, use:
2701$ sort -m sorted1 sorted2
2702There's more...
2703Sort according to the keys or columns
2704We use sort by column if we need to sort a text as follows:
2705$ cat data.txt
27061 mac 2000
27072 winxp 4000
27083 bsd 1000
27094 linux 1000
2710We can sort this in many ways; currently it is numeric sorted by serial number (the first
2711column). We can also sort by second column and third column.
2712-k specifies the key by which the sort is to be performed. Key is the column number by which
2713sort is to be done.  -r specifies the sort command to sort in the reverse order. For example:
2714# Sort reverse by column1
2715$ sort -nrk 1 data.txt
27164  linux 1000
27173  bsd  1000
27182  winxp 4000
27191  mac  2000
2720# -nr means numeric and reverse
2721# Sort by column 2
2722$ sort -k 2 data.txt
2723www.it-ebooks.info
2724Chapter 2
272577
27263  bsd  1000
27274  linux 1000
27281  mac  2000
27292  winxp 4000
2730Always be careful about the -n option for numeric sort. The sort command
2731treats alphabetical sort and numeric sort differently. Hence, in order to
2732specify numeric sort the –n option should be provided.
2733Usually, by default, keys are columns in the text file. Columns are separated by space
2734characters. But in certain circumstances, we will need to specify keys as a group of characters
2735in the given character number range (for example, key1= character4-character8). In such
2736cases where keys are to be specified explicitly as a range of characters, we can specify the
2737keys as ranges with the character position at key starts and key ends as follows:
2738$ cat data.txt
27391010hellothis
27402189ababbba
27417464dfddfdfd
2742$ sort -nk 2,3 data.txt
2743The highlighted characters are to be used as numeric keys. In order to extract them, use their
2744start-pos and end-pos as the key format.
2745In order to use the first character as the key, use:
2746$ sort -nk 1,1 data.txt
2747Make the sort's output xargs compatible with  \0 terminator, by using the following command:
2748$ sort -z data.txt | xargs -0
2749#Zero terminator is used to make safe use with xargs
2750Sometimes the text may contain unnecessary extraneous characters like spaces. To sort by
2751ignoring them in dictionary order by ignoring punctuations and folds, use:
2752$ sort -bd unsorted.txt
2753The option  –b is used to ignore leading blanks from the file and the  –d option is used to
2754specify sort in the dictionary order.
2755uniq
2756uniq is a command used to find out the unique lines from the given input ( stdin or from
2757filename as command argument) by eliminating the duplicates. It can also be used to find out
2758the duplicate lines from the input . uniq can be applied only for sorted data input. Hence,  uniq
2759is to be used always along with the  sort command using pipe or using a sorted file as input.
2760www.it-ebooks.info
2761Have a Good Command
276278
2763You can produce the unique lines (unique lines means that all lines in the input are printed,
2764but the duplicate lines are printed only once) from the given input data as follows:
2765$ cat sorted.txt
2766bash
2767foss
2768hack
2769hack
2770$ uniq sorted.txt
2771bash
2772foss
2773hack
2774Or:
2775$ sort unsorted.txt | uniq
2776Or:
2777$ sort -u unsorted.txt
2778Display only unique lines (the lines which are not repeated or duplicate in input file) as follows:
2779$ uniq -u sorted.txt
2780bash
2781foss
2782Or:
2783$ sort unsorted.txt | uniq -u
2784In order to count how many times each of the line appears in the file, use the following
2785command:
2786$ sort unsorted.txt | uniq -c
27871 bash
27881 foss
27892 hack
2790Find duplicate lines in the file as follows:
2791$ sort unsorted.txt | uniq -d
2792hack
2793www.it-ebooks.info
2794Chapter 2
279579
2796To specify keys, we can use the combination of  -s  and  -w arguments.
2797f -s specifies the number for the first  N characters to be skipped
2798f -w specifies the maximum number of characters to be compared
2799This comparison key is used as the index for the  uniq operation as follows:
2800$ cat data.txt
2801u:01:gnu
2802d:04:linux
2803u:01:bash
2804u:01:hack
2805We need to use the highlighted characters as the uniqueness key. This is used to ignore the
2806first 2 characters ( -s 2 ) and the max number of comparison characters is specified using the
2807–w option  (-w 2) :
2808$ sort data.txt | uniq -s 2 -w 2
2809d:04:linux
2810u:01:bash
2811While we use output from one command as input to the xargs command, it is always preferable
2812to use a zero byte terminator for each of the lines of the output, which acts as source for
2813xargs . While using the  uniq commands output as the source for  xargs , we should use a zero
2814terminated output. If a zero byte terminator is not used, space characters are by default taken
2815as delimiter to split the arguments in the  xargs command. For example, a line with text "this
2816is a line" from  stdin will be taken as four separate arguments by the  xargs . But, actually, it
2817is a single line. When a zero byte terminator is used,  \0 is used as the delimiter character and
2818hence, a single line including space is interpreted as a single argument.
2819Zero byte terminated output can be generated from the  uniq command as follows:
2820$ uniq -z file.txt
2821The following command removes all the files, with filenames read from  files.txt :
2822$ uniq –z file.txt | xargs -0 rm
2823If multiple line entries of filenames exist in the file, the  uniq command writes the filename
2824only once to  stdout .
2825www.it-ebooks.info
2826Have a Good Command
282780
2828String pattern generation with uniq
2829Here is an interesting question for you: We have a string containing repeated characters. How
2830can we find the number of times each of the character appears in the string and output a
2831string in the following format?
2832Input: ahebhaaa
2833Output: 4a1b1e2h
2834Each of the characters is repeated once, and each of them is prefixed with the number of
2835times they appear in the string. We can solve this using  uniq and  sort as follows:
2836INPUT= "ahebhaaa"
2837OUTPUT=` echo $INPUT | sed 's/[^\n]/&\n/g' | sed '/^$/d' | sort | uniq
2838-c | tr -d ' \n'`
2839echo $OUTPUT
2840In the above code, we can split each of the piped commands as follows:
2841echo $ INPUT # Print the input to stdout
2842sed 's/./&\n/g'
2843Append a newline character to each of the characters so that only one character appears in
2844one line. This is done to make the characters sortable by using the  sort command. The  sort
2845command can take only items delimited by newline.
2846f sed '/^$/d' : Here the last character is replaced as character  +\n . Hence an extra
2847newline is formed and it will form a blank line at the end. This command removes the
2848blank line from the end.
2849f sort : Since each character appears in each line, it can be sorted so that it can serve
2850as input to uniq.
2851f uniq –c : This command prints each of the line with how many times they got
2852repeated(count).
2853f tr –d ' \n' : This removes the space characters and newline characters from the
2854input so that output can be produced in the given format.
2855Temporary file naming and random numbers
2856While writing shell scripts, we use often require to store temporary data. The most suitable
2857location to store temporary data is  /tmp (which will be cleaned out by the system on reboot).
2858We can use two methods to generate standard filenames for temporary data.
2859www.it-ebooks.info
2860Chapter 2
286181
2862How to do it...
2863tempfile is not seen in non-Debian Linux distributions. The  tempfile command comes
2864shipped with Debian-based distributions, such as Ubuntu, Debian, and many more.
2865The following code will assign a temporary filename to the variable  temp_file :
2866temp_file=$(tempfile)
2867Use  echo $temp_file to print the temporary file name in the terminal.
2868The output will look similar to  /tmp/fileaZWm8Y .
2869Sometimes we may use a filename with a random number attached to it as a temporary
2870filename. This can be done as follows:
2871temp_file="/tmp/file-$RANDOM"
2872The  $RANDOM environment variable always returns a random number.
2873How it works…
2874Instead of using the  tempfile command, we can also use our own temporary. Most
2875experienced UNIX programmers use the following conventions:
2876temp_file="/tmp/var.$$"
2877The  .$$ suffix is attached.  $$ is expanded as the process ID of the current script
2878upon execution.
2879Splitting files and data
2880Splitting of files into many smaller pieces becomes essential in certain situations. Earlier,
2881when memory was limited with devices like floppy disks, it was crucial to split files into smaller
2882file sizes to transfer data in many disks. However, nowadays we split files for other purposes,
2883such as readability, for generating logs, and so on.
2884How to do it...
2885Generate a test file ( data.file ) of 100kb as follows:
2886$ dd if=/dev/zero bs=100k count=1 of=data.file
2887The above command creates a file filled with zeros with the size of 100kb.
2888www.it-ebooks.info
2889Have a Good Command
289082
2891You can split files into smaller files by specifying the split size as follows:
2892$ split -b 10k data.file
2893$ ls
2894data.file xaa xab xac xad xae xaf xag xah xai xaj
2895It will split  data.file into many files, each of a 10k chunk. The chunks will be named in
2896the manner  xab ,  xac ,  xad , and so on. This means it will have alphabetic suffixes. To use the
2897numeric suffixes, use an additional  -d argument. It is also possible to specify a suffix length
2898using  -a length as follows:
2899$ split -b 10k data.file -d -a 4
2900$ ls
2901data.file x0009 x0019 x0029 x0039 x0049 x0059 x0069 x0079
2902Instead of the  k (kilobyte) suffix we can use  M for MB,  G for GB,  c for byte,  w for word, and so on.
2903There's more…
2904The  split command has more options. Let's go through them.
2905Specifying filename prefix for the split files
2906The above split files have a filename prefix "x". We can also use our own filename prefix by
2907providing a prefix filename. The last command argument for the split command is  PREFIX . It
2908is in the format:
2909$ split [COMMAND_ARGS] PREFIX
2910Let's run the previous command with the prefix filename for split files:
2911$ split -b 10k data.file -d -a 4 split_file
2912$ ls
2913data.file split_file0002 split_file0005 split_file0008 strtok.c
2914split_file0000 split_file0003 split_file0006 split_file0009
2915split_file0001 split_file0004 split_file0007
2916In order to split files based on number of lines in each split rather than chunk size,
2917use  -l no_of_lines as follows:
2918$ split -l 10 data.file
2919# Splits into files of 10 lines each.
2920There is another interesting utility called  csplit . It can be used to split log file-based
2921specified conditions and string match options. Let's see how to work with it.
2922www.it-ebooks.info
2923Chapter 2
292483
2925csplit is a variant of the  split utility. The  split utility can only split files based on chunk
2926size or based on the number of lines.  csplit makes the split based on context based split. It
2927can be used to split files based on existence of a certain word or text content.
2928Look at the example log:
2929$ cat server.log
2930SERVER-1
2931[connection] 192.168.0.1 success
2932[connection] 192.168.0.2 failed
2933[disconnect] 192.168.0.3 pending
2934[connection] 192.168.0.4 success
2935SERVER-2
2936[connection] 192.168.0.1 failed
2937[connection] 192.168.0.2 failed
2938[disconnect] 192.168.0.3 success
2939[connection] 192.168.0.4 failed
2940SERVER-3
2941[connection] 192.168.0.1 pending
2942[connection] 192.168.0.2 pending
2943[disconnect] 192.168.0.3 pending
2944[connection] 192.168.0.4 failed
2945We may need to split the files into  server1.log ,  server2.log , and  server3.log from
2946the contents for each  SERVER in each file. This can be done as follows:
2947$ csplit server.log /SERVER/ -n 2 -s {*} -f server -b "%02d.log" ; rm
2948server00.log
2949$ ls
2950server01.log server02.log server03.log server.log
2951The details of the command are as follows:
2952f /SERVER/ is the line used to match a line by which the split is to be carried out.
2953f /[REGEX]/ is the format. It copies from current line (first line) upto the matching line
2954that contains  "SERVER" excluding match line.
2955f {*} is used to specify to repeat splitting based on match upto the end of the file. By
2956using  {integer} , we can specify no of times it is to be continued.
2957f -s is the flag to make the command silent rather than printing other messages.
2958f -n is used to specify the number of digits to be used as suffix. 01, 02, 03, and so on.
2959www.it-ebooks.info
2960Have a Good Command
296184
2962f -f  is used for specifying the filename prefix for split files ("server" is the prefix in the
2963previous example).
2964f -b is used to specify the suffix format.  "%02d.log" is similar to the  printf
2965argument format in C. Here the filename = prefix + suffix = "server" + "%02d.
2966log" .
2967We remove  server00.log since the first split file is an empty file (the match word is the first
2968line of the file).
2969Slicing filenames based on extension
2970Several custom shell scripts perform manipulations based on file names. We may need to
2971perform actions like renaming the files by preserving extension, converting files from one
2972format to another (change the extension by preserving the name), extracting a portion of the
2973file name, and so on. The shell comes with inbuilt functionalities for slicing filenames based
2974on different conditions. Let's see how to do it.
2975How to do it…
2976The name from  name.extension can be easily extracted by using the  % operator. You can
2977extract the name from  "sample.jpg" as follows:
2978file_jpg="sample.jpg"
2979name=${file_jpg%.*}
2980echo File name is: $name
2981The output is:
2982File name is: sample
2983The next task is to extract the extension of a file from its filename. The extension can be
2984extracted using the  # operator.
2985Extract  .jpg from the a filename stored in variable  file_jpg as follows:
2986extension=${file_jpg#*.}
2987echo Extension is: jpg
2988The output is:
2989Extension is: jpg
2990How it works..
2991In the first task, in order to extract the name from the file name in the format  name.extension
2992we have used the  % operator.
2993www.it-ebooks.info
2994Chapter 2
299585
2996${VAR%.*} can be interpreted as:
2997f Remove the string match from the  $VARIABLE for the wildcard pattern that appears
2998to the right-hand side of % (.* in the previous example). Evaluating from the right to
2999the left direction should make the wildcard match.
3000f Let  VAR=sample.jpg . Therefore, the wildcard match for  .* from right to left is