TDPhYAWQ

1.jpg . Thus it is removed from the  $VAR string and the output will be "sample".
2% is a non-greedy operation. It finds the minimal match for the wildcard from the right to
3left. There is an operator  %% , which is similar to  % . But it is greedy in nature. That means it
4matches the maximal string for the wildcard.
5For example, we have:
6VAR=hack.fun.book.txt
7By using the % operator, we have:
8$ echo ${VAR%.*}
9The output will be:  hack.fun.book .
10The operator  % performs a non-greedy match for  .* from right to left ( .txt ).
11By using the %% operator, we have:
12$ echo ${VAR%%.*}
13The output will be:  hack
14The  %% operator matches greedy match for  .* from right to left (. fun.book.txt ).
15In the second task, we have used the  # operator to extract the extension from the filename. It
16is similar to  % . But it evaluates from left to right.
17${VAR#*.} can be interpreted as:
18Remove the string match from the  $VARIABLE for the wildcard pattern match appears right
19side to the  # ( *. in the above example). Evaluating from the left to right direction should make
20the wildcard match.
21Similarly, as in the case of  %% , we have another greedy operator for  # , which is  ## .
22It makes greedy matches by evaluating from left to right and removes the match string from
23the specified variable.
24Let's use this example:
25VAR=hack.fun.book.txt
26www.it-ebooks.info
27Have a Good Command
2886
29By using the  # operator, we have:
30$ echo ${VAR#*.}
31The output will be:  fun.book.txt .
32The operator  # performs a non-greedy match for  *. from left to right ( hack. ).
33By using the ## operator, we have:
34$ echo ${VAR##*.}
35The output will be:  txt .
36The operator ## matches greedy match for *. from left to right (txt).
37The ## operator is more preferred over the # operator to extract an extension
38from a filename since the filename may contain multiple '.' characters. Since
39## makes greedy match, it always extract extensions only.
40Here is practical example that can be used to extract different portions of a domain name,
41given URL=" www.google.com ":
42$ echo ${URL%.*} # Remove rightmost .*
43www.google
44$ echo ${URL%%.*} # Remove right to leftmost .* (Greedy operator)
45www
46$ echo ${URL#*.} # Remove leftmost part before *.
47google.com
48$ echo ${URL##*.} # Remove left to rightmost part before *. (Greedy
49operator)
50com
51Renaming and moving files in bulk
52Renaming a number of files is one of the tasks we frequently come across. A simple example
53is, when you download photos from your digital camera to the computer you may delete
54unnecessary files and it causes discontinuous numbering of image files. Sometimes you
55many need to rename them with custom prefix and continuous numbering for filenames.
56We sometimes use third-party tools for performing rename operations. We can use Bash
57commands to perform a rename operation in a couple of seconds.
58Moving all the files having a particular substring in the filename (for example, same prefix for
59filenames) or with a specific file type to a given directory is another use case we frequently
60perform. Let's see how to write scripts to perform these kinds of operations.
61www.it-ebooks.info
62Chapter 2
6387
64Getting ready
65The  rename command helps to change file names using Perl regular expressions. By
66combining the commands  find ,  rename , and  mv , we can perform a lot of things.
67How to do it...
68The easiest way of renaming image files in the current directory to our own filename with a
69specific format is by using the following script:
70#!/bin/bash
71#Filename: rename.sh
72#Description: Rename jpg and png files
73count=1;
74for img in *.jpg *.png
75do
76new=image-$count.${img##*.}
77mv "$img" "$new" 2> /dev/null
78if [ $? -eq 0 ];
79then
80echo "Renaming $img to $new"
81let count++
82fi
83done
84The output is as follows:
85$ ./rename.sh
86Renaming hack.jpg to image-1.jpg
87Renaming new.jpg to image-2.jpg
88Renaming next.jpg to image-3.jpg
89The script renames all the  .jpg and  .png files in the current directory to new filenames in
90the format  image-1.jpg ,  image-2.jpg ,  image-3.jpg ,  image-4.png , and so on.
91How it works…
92In the above rename script, we have used a  for loop to iterate through the names of all files
93ending with a  .jpg extension. The wildcard  *.jpg and * .png are used to match all the
94JPEG and PNG files. We can do a small improvisation over the extension match. The  .jpg
95wildcard matches only the extension in lowercase. However, we can make it case insensitive
96by replacing  .jpg with  .[jJ][pP][gG] . Hence it can match files like  file.jpg as well as
97file.JPG or  file.Jpg . In Bash, when characters are enclosed in  [] , it means to match
98one character from the set of characters enclosed in  [] .
99www.it-ebooks.info
100Have a Good Command
10188
102for img in *.jpg *.png in the above code will be expanded as follows:
103for img in hack.jpg new.jpg next.jpg
104We have initialized a variable  count=1 in order to keep track of the image number. The
105next step is to rename the file using the  mv command. The new name of the file should be
106formulated for renaming.  ${img##*.} in the script parses the extension of the filename
107currently in the loop (see the Slicing file names based on extension recipe for interpretation of
108${img##*.} ).
109let count++ is used to increment the file number for each execution of loop.
110You can see that error redirection ( stderr ) to  /dev/null is done for the  mv command using
111the  2> operator. This is to stop the error messages being printed into the terminal.
112Since we use  *.png and  *.jpg , if atleast one image for a wildcard match is not present,
113the shell will interpret the wildcard itself as a string. In the above output, you can see that
114.png files are not present. Hence it will take  *.png as yet another filename and execute
115mv *.png image-X.png , which will cause an error. An  if statement with  [ $? –eq 0 ]
116is used to check the exit status ( $? ). The value of  $? will be 0 if the last executed command
117is successful, else it returns non-zero. When the  mv command fails, it returns non-zero and,
118therefore, the message "Renaming file" will not be shown to the user, as well as the count will
119not be incremented.
120There are a variety of other ways to perform rename operations. Let's walk through a few
121of them.
122Renaming  *.JPG  to  *.jpg:
123$ rename *.JPG *.jpg
124Replace space in the filenames with the  "_" character as follows:
125$ rename 's/ /_/g' *
126# 's/ /_/g' is the replacement part in the filename and  * is the wildcard for the target
127files. It can be  *.txt or any other wildcard pattern.
128You can convert any filename of files from uppercase to lowercase and vice versa as follows:
129$ rename 'y/A-Z/a-z/' *
130$ rename 'y/a-z/A-Z/' *
131In order to recursively move all the .mp3 files to a given directory, use:
132$ find path -type f -name "*.mp3" -exec mv {} target_dir \;
133Recursively rename all the files by replacing space with " _" character as follows:
134$ find path -type f -exec rename 's/ /_/g' {} \;
135www.it-ebooks.info
136Chapter 2
13789
138Spell checking and dictionary manipulation
139Most Linux distributions come with a dictionary file. However, I find few people are aware of
140the dictionary file and hence many people fail to make use of them. There is a command-line
141utility called  aspell that functions as a spell checker. Let's go through few scripts that make
142use of the dictionary file and the spell checker.
143How to do it...
144The  /usr/share/dict/ directory contains some of the dictionary files. Dictionary files are
145text files that contain a list of dictionary words. We can use this list to check whether a word is
146a dictionary word or not.
147$ ls /usr/share/dict/
148american-english british-english
149In order to check whether the given word is a dictionary word, use the following script:
150#!/bin/bash
151#Filename: checkword.sh
152word=$1
153grep "^$1$" /usr/share/dict/british-english -q
154if [ $? -eq 0 ]; then
155echo $word is a dictionary word;
156else
157echo $word is not a dictionary word;
158fi
159The usage is as follows:
160$ ./checkword.sh ful
161ful is not a dictionary word
162$ ./checkword.sh fool
163fool is a dictionary word
164How it works...
165In  grep ,  ^ is the word start marker character and the character  $ is the word end marker.
166-q is used to suppress any output and to be silent.
167www.it-ebooks.info
168Have a Good Command
16990
170Or, alternately, we can use the spell check,  aspell , to check whether a word is in a dictionary
171or not as follows:
172#!/bin/bash
173#Filename: aspellcheck.sh
174word=$1
175output=`echo \"$word\" | aspell list`
176if [ -z $output ]; then
177echo $word is a dictionary word;
178else
179echo $word is not a dictionary word;
180fi
181The  aspell list command returns output text when the given input is not a dictionary
182word, and does not output anything when a dictionary word is the input. A  -z check ensures
183whether  $output is an empty string or not.
184List all words in a file starting with a given word as follows:
185$ look word filepath
186Or alternately, use:
187$ grep "^word" filepath
188By default, if the filename argument is not given to the  look command, it uses the default
189dictionary ( /usr/share/dict/words ) and returns an output.
190$ look word
191# When used like this it takes default dictionary as file
192For example:
193$ look android
194android
195android's
196androids
197Automating interactive input
198Automating interactive input for command-line utilities are extremely useful for writing
199automation tools or testing tools. There will be many situations when we deal with commands
200that read inputs interactively. Interactive input is the input typed by the user only when
201the command asks for some input. An example for execution of a command and supply of
202interactive input is as follows:
203www.it-ebooks.info
204Chapter 2
20591
206$ command
207Enter a number: 1
208Enter name : hello
209You have entered 1,hello
210Getting ready
211Automating utilities which can automate the acceptance of input as in the above mentioned
212manner are useful to supply input to local commands as well as for remote applications. Let's
213see how to automate them.
214How to do it...
215Think about the sequence of an interactive input. From the previous code we can formulate
216the steps of the sequence as follows:
2171[Return]hello[Return]
218Converting the above steps  1,Return,hello,Return by observing the characters that are
219actually typed in the keyboard, we can formulate the following string.
220"1\nhello\n"
221The  \n character is sent when we press  Return . By appending return ( \n ) characters, we get
222the actual string that is passed to the  stdin (standard input).
223Hence by sending the equivalent string for the characters typed by the user, we can automate
224the passing of input in the interactive processes.
225How it works…
226Let's write a script that reads input interactively and uses this script for automation examples:
227#!/bin/bash
228#Filename: interactive.sh
229read -p "Enter number:" no ;
230read -p "Enter name:" name
231echo You have entered $no, $name;
232Let's automate the sending of input to the command as follows:
233$ echo -e "1\nhello\n" | ./interactive.sh
234You have entered 1, hello
235Thus crafting inputs with  \n works.
236www.it-ebooks.info
237Have a Good Command
23892
239We have used  echo -e to produce the input sequence. If the input is large we can use an
240input file and redirection operator to supply input.
241$ echo -e "1\nhello\n" > input.data
242$ cat input.data
2431
244hello
245You can also manually craft the input file without  echo commands by hand typing. For
246example:
247$ ./interactive.sh < input.data
248This redirects interactive input data from a file.
249If you are a reverse engineer, you may have played with buffer overflow exploits. To exploit
250them we need to redirect shellcode like  "\xeb\x1a\x5e\x31\xc0\x88\x46" , which is
251written in hex. These characters cannot be typed directly through keyboard since, keys for
252these characters are not present in the keyboard. Therefore we should use:
253echo -e "\xeb\x1a\x5e\x31\xc0\x88\x46"
254This will redirect shellcode to a vulnerable executable.
255We have described a method to automate interactive input programs by redirecting expected
256input text through  stdin (standard input). We are sending the input without checking the
257input the program asks for. We are sending the input by expecting the program to ask input
258in a specific (static) order. If the program asks input randomly or in a changing order, or
259sometimes certain inputs are never asked, the above method fails. It will send wrong inputs to
260different input prompts by the program. In order to handle dynamic input supply and provide
261input by checking the input requirements by the program on runtime, we have a great utility
262called  expect . The  expect command supplies correct input for the correct input prompt by
263the program. Let's see how to use  expect .
264There's more...
265Automation of interactive input can also be done using other methods. Expect scripting is
266another method for automation. Let's go through it.
267Automating with expect
268The  expect utility does not come by default with most of the common Linux distributions. You
269have to install the expect package manually using package manager.
270expect expects for a particular input prompt and sends data by checking message
271in the input prompt.
272www.it-ebooks.info
273Chapter 2
27493
275#!/usr/bin/expect
276#Filename: automate_expect.sh
277spawn ./interactive .sh
278expect "Enter number:"
279send "1\n"
280expect "Enter name:"
281send "hello\n"
282expect eof
283Run as:
284$ ./automate_expect.sh
285In this script:
286f spawn parameter specifies which command is to be automated
287f expect parameter provides the expected message
288f send is the message to be sent.
289f expect eof defines the end of command interaction
290www.it-ebooks.info
291www.it-ebooks.info
2923
293File In, File Out
294In this chapter, we will cover:
295f Generating files of any size
296f Intersection and set difference (A-B) on text files
297f Finding and deleting duplicate files
298f Making directories for a long path
299f File permissions, ownership and sticky bit
300f Making files immutable
301f Generating blank files in bulk
302f Finding symbolic links and its target
303f Enumerating file type statistics
304f Loopback files and mounting
305f Creating ISO files, Hybrid ISO
306f Finding difference between files, patching
307f head and tail - printing the last or first 10 lines
308f Listing only directories - alternative methods
309f Fast command line directory navigation using pushd and popd
310f Counting the number of lines, words, and characters in a file
311f Printing directory tree
312www.it-ebooks.info
313File In, File Out
31496
315Introduction
316UNIX treats every object in the operating system as a file. We can find the files associated with
317every action performed and can make use of them for different system- or process-related
318manipulations. For example, the command terminal that we use is associated with a device
319file. We can write to the terminal by writing to the corresponding device file for that specific
320terminal. Files take different forms such as directories, regular files, block devices, character
321special devices, symbolic links, sockets, named pipes, and so on. Filename, size, file type,
322modification time, access time, change time, inode, links associated, and the filesystem the
323file is on are all attributes and properties that files can have. This chapter deals with recipes
324that handle any of the operations or properties related to files.
325Generating files of any size
326For various reasons, you may need to generate a file filled with random data. It may be for
327creating a test file to perform tests, such as an application efficiency test that uses a large file
328as input, or to test the splitting of files into many, or to create loopback filesystems (loopback
329files are files that can contain a filesystem itself and these files can be mounted similar to a
330physical device using the  mount command). It is hard to create such files by writing specific
331programs. So we use general utilities.
332How to do it...
333The easiest way to create a large sized file with a given size is to use the  dd command. The  dd
334command clones the given input and writes an exact copy to the output. Input can be  stdin ,
335a device file, a regular file, or so on. Output can be  stdout , a device file, a regular file, and so
336on. An example of the  dd command is as follows:
337$ dd if=/dev/zero of=junk.data bs=1M count=1
3381+0 records in
3391+0 records out
3401048576 bytes (1.0 MB) copied, 0.00767266 s, 137 MB/s
341The above command will create a file called  junk.data that is exactly 1MB in size. Let's go
342through the parameters:  if stands for  – input file,  of stands for  – output file,  bs stands for
343BYTES for a block, and  count stands for the number of blocks of  bs specified to be copied.
344Here we are only creating a file 1MB in size by specifying  bs as 1MB with a count of 1. If  bs
345was set to  2M and a count to 2, the total file size would be 4MB.
346www.it-ebooks.info
347Chapter 3
34897
349We can use various units for Block Size (BS) as follows. Append any of the following
350characters to the number to specify the size in bytes:
351Unit size Code
352Byte (1B) c
353Word (2B) w
354Block (512B) b
355Kilo Byte (1024B) k
356Mega Byte (1024 KB) M
357Giga Byte (1024 MB) G
358We can generate a file of any size using this. Instead of MB we can use any other unit
359notations such as the ones mentioned in the previous table.
360/dev/zero is a character special device, which infinitely returns the zero byte ( \0 ).
361If the input parameter ( if ) is not specified, it will read the input from  stdin by default. Similarly,
362if the output parameter ( of ) is not specified, it will use  stdout as the default output sink.
363The  dd command can also be used to measure the speed of memory operations by transferring
364a large quantity of data and checking the command output (for example,  1048576 bytes
365(1.0 MB) copied, 0.00767266 s, 137 MB/s as seen the previous example).
366Intersection and set difference (A-B)
367on text files
368Intersection and set difference operations are commonly used in mathematical classes on set
369theory. However, similar operations on text are also very helpful in some scenarios.
370Getting ready
371The  comm command is a utility to perform comparison between the two files. It has many nice
372options to arrange the output in such a way that we can perform intersection, difference, and
373set difference operations.
374f Intersection: The intersection operation will print the lines that the specified files
375have in common with one another.
376f Difference: The difference operation will print the lines that the specified files contain
377and that are not the same in all of those files.
378f Set difference: The set difference operation will print the lines in file "A" that do not
379match those in all of the set of files specified ("B" plus "C" for example).
380www.it-ebooks.info
381File In, File Out
38298
383How to do it...
384Note that  comm takes sorted files as input. Take a look at the following example:
385$ cat A.txt
386apple
387orange
388gold
389silver
390steel
391iron
392$ cat B.txt
393orange
394gold
395cookies
396carrot
397$ sort A.txt -o A.txt ; sort B.txt -o B.txt
398$ comm A.txt B.txt
399apple
400carrot
401cookies
402gold
403iron
404orange
405silver
406steel
407The first column of the output contains lines that are in  A.txt excluding common lines in
408two files. The second column contains lines that are in  B.txt excluding common lines. The
409third column contains the common lines from  A.txt and  B.txt . Each of the columns are
410delimited by using the tab ( \t ) character.
411Some options are available to format the output as per our requirement. For example:
412f -1 removes first column from output
413f -2 removes the second column
414f -3 removes the third column
415www.it-ebooks.info
416Chapter 3
41799
418In order to print the intersection of two files, we need to remove the first and second columns
419and print the third column only as follows:
420$ comm A.txt B.txt -1 -2
421gold
422orange
423Print lines that are uncommon in two files as follows:
424$ comm A.txt B.txt -3
425apple
426carrot
427cookies
428iron
429silver
430steel
431Using the  -3 argument in the  comm command removes the third column from the output.
432But, it writes column-1 and column-2 to the output. The column-1 contains the lines in  A.txt
433excluding the lines in  B.txt . Similarly, column-2 has the lines from  B.txt excluding the lines
434in  A.txt . As the output is a two-column output, it is not that useful. Columns have their fields
435blank for each of the unique lines. Hence both columns will not have the content on the same
436line. Either one of the two columns will have the content. In order to make it in a usable output
437text format, we need to remove the blank fields and make two columns into a single column
438output as follows:
439apple
440carrot
441cookies
442iron
443silver
444steel
445In order to produce such an output, we need to remove the  \t character at the beginning of
446the lines. We can remove the  \t character from the start of each line and unify the columns
447into one as follows:
448$ comm A.txt B.txt -3 | sed 's/^\t//'
449apple
450carrot
451cookies
452iron
453silver
454steel
455www.it-ebooks.info
456File In, File Out
457100
458The  sed command is piped to the  comm output. The  sed removes the  \t character at the
459beginning of the lines. The  s in the  sed script stands for substitute.  /^\t/ matches the
460\t at the beginning of the lines ( ^ is the start of the line marker).  // (no character) is the
461replacement string for every  \t at the beginning of the line. Hence every  \t at the start of the
462line gets removed.
463A set difference operation on two files can be performed as explained in the following
464paragraphs.
465The set difference operation enables you to compare two files and print all the lines that are
466in the file  A.txt or  B.txt excluding the common lines in  A.txt and  B.txt . When  A.txt
467and  B.txt are given as arguments to the  comm command, the output will contain column-1
468with the set difference for  A.txt with respect to  B.txt and column-2 will contain the set
469difference for  B.txt with respect to  A.txt .
470By removing the unnecessary columns, we can produce the set difference for  A.txt and
471B.txt as follows:
472f Set difference for A.txt:
473$ comm A.txt B.txt -2 -3
474-2 -3  removes the second and third columns.
475f Set difference for B.txt:
476$ comm A.txt B.txt -1 -3
477-2 -3  removes the second and third columns.
478Finding and deleting duplicate files
479Duplicate files are copies of the same files. In some circumstances, we may need to remove
480duplicate files and keep a single copy of them. Identification of duplicate files by looking at the
481file content is an interesting task. It can be done using a combination of shell utilities. This
482recipe deals with finding out duplicate files and performing operations based on the result.
483Getting ready
484Duplicate files are files with different names but same data. We can identify the duplicate files
485by comparing the file content. Checksums are calculated by looking at the file contents. Since
486files with exactly the same content will produce duplicate checksum values, we can use this to
487remove duplicate lines.
488www.it-ebooks.info
489Chapter 3
490101
491How to do it...
492Generate some test files as follows:
493$ echo "hello" > test ; cp test test_copy1 ; cp test test_copy2;
494$ echo "next" > other;
495# test_copy1 and test_copy2 are copy of test
496The code for the script to remove the duplicate files is as follows:
497#!/bin/bash
498#Filename: remove_duplicates.sh
499#Description: Find and remove duplicate files and keep one sample of
500each file.
501ls -lS | awk 'BEGIN {
502getline;getline;
503name1=$8; size=$5
504}
505{ name2=$8;
506if (size==$5)
507{
508"md5sum "name1 | getline; csum1=$1;
509"md5sum "name2 | getline; csum2=$1;
510if ( csum1==csum2 )
511{print name1; print name2 }
512};
513size=$5; name1=name2;
514}' | sort -u > duplicate_files
515cat duplicate_files | xargs -I {} md5sum {} | sort | uniq -w 32 | awk
516'{ print "^"$2"$" }' | sort -u > duplicate_sample
517echo Removing..
518comm duplicate_files duplicate_sample -2 -3 | tee /dev/stderr | xargs
519rm
520echo Removed duplicates files successfully.
521Run it as:
522$ ./remove_duplicates.sh
523www.it-ebooks.info
524File In, File Out
525102
526How it works...
527The commands above will find the copies of same file in a directory and remove all except one
528copy of the file. Let's go through the code and see how it works.  ls -lS will list the details of
529the files sorted by file size in the current directory. awk will read the output of  ls -lS and
530perform comparisons on columns and rows of the input text to find out the duplicate files.
531The logic behind the previous code is as follows:
532f We list the files sorted by file size so that the similarly sized files will be grouped
533together. The files having the same file size are identified as a first step to finding files
534that are the same. Next, we calculate the checksum of the files. If the checksums
535match, then the files are duplicates and one set of the duplicates are removed.
536f The  BEGIN{} block of  awk is executed first before lines are read from the file.
537Reading of lines takes place in the {} block and after the end of reading and
538processing all lines, the END{} block statements are executed. The output of  ls
539-lS is:
540total 16
5414 -rw-r--r-- 1 slynux slynux 5 2010-06-29 11:50 other
5424 -rw-r--r-- 1 slynux slynux 6 2010-06-29 11:50 test
5434 -rw-r--r-- 1 slynux slynux 6 2010-06-29 11:50 test_copy1
5444 -rw-r--r-- 1 slynux slynux 6 2010-06-29 11:50 test_copy2
545f The output of the first line tells us the total number of files, which in this case is not
546useful. We use  getline to read the first line and then dump it. We need to compare
547each of the lines and the next line for sizes. For that we read the first line explicitly
548using  getline and store name and size (which are the eighth and fifth columns).
549Hence a line is read ahead using  getline . Now, when  awk enters the  {} block (in
550which the rest of the lines are read) that block is executed for every read offline. It
551compares size obtained from the current line and the previously stored size kept in
552the  size variable. If they are equal, it means two files are duplicates by size. Hence
553they are to be further checked by  md5sum .
554We have played some tricky ways to reach the solution.
555The external command output can be read inside  awk as:
556"cmd"| getline
557Then we receive the output in line  $0 and each column output can be received in
558$1,$2,..$n , and so on. Here we read the md5sum of files in the  csum1 and  csum2
559variables. Variables  name1 and  name2 are used to store consecutive file names. If the
560checksums of two files are the same, they are confirmed to be duplicates and are printed.
561www.it-ebooks.info
562Chapter 3
563103
564We need to find a file each from the group of duplicates so that we can remove all other
565duplicates except one. We calculate the  md5sum of the duplicates and print one file from
566each group of duplicates by finding unique lines by comparing  md5sum only from each line
567using  -w 32 (the first 32 characters in the  md5sum output; usually,  md5sum output consists
568of a 32 character hash followed by the filename). Therefore, one sample from each group of
569duplicates is written in  duplicate_sample .
570Now, we need to remove all the files listed in  duplicate_files , excluding the files listed
571in  duplicate_sample . The  comm  command prints files in  duplicate_files but not in
572duplicate_sample .
573For that, we use a set difference operation (refer to the intersection, difference, and set
574difference recipes).
575comm always accepts files that are sorted. Therefore,  sort -u is used as a filter before
576redirecting to  duplicate_files and  duplicate_sample .
577Here the  tee command is used to perform a trick so that it can pass filenames to the  rm
578command as well as  print .  tee writes lines that appear as  stdin to a file and sends them
579to  stdout . We can also print text to the terminal by redirecting to  stderr .  /dev/stderr is
580the device corresponding to  stderr (standard error). By redirecting to a  stderr device file,
581text that appears through  stdin will be printed in the terminal as standard error.
582See also
583f Basic awk primer of Chapter 4 explains the awk command.
584f Checksum and verification of Chapter 2 explains the md5sum command.
585Making directories for a long path
586There are circumstances when we are required to make a tree of empty directories. If some
587intermediate directories exist in the given path, it will also have to incorporate checks to see
588whether the directory exists or not. It will make the code larger and inefficient. Let's see the
589use case and the recipe to solve the issue.
590Getting ready
591mkdir is the command for creating directories. For example:
592$ mkdir dirpath
593If the directory already exists, it will return a "File exists" error message, as follows:
594mkdir: cannot create directory `dir_name': File exists
595www.it-ebooks.info
596File In, File Out
597104
598You are given a directory path ( /home/slynux/test/hello/child ). The directory
599/home/slynux  already exist. We need to create rest of the directories  (/home/slynux/
600test ,  /home/slynux/test/hello , and  /home/slynux/test/hello ) in the path.
601The following code is used to figure out whether each directory in a path exists:
602if [ -e /home/slynux ]; then
603# Create next level directory
604fi
605-e is a parameter used in the condition construct  [ ] , to determine whether a file exists. In
606UNIX-like systems, directory is also a type of file.  [ -e FILE_PATH ] returns true if the
607file exists.
608How to do it...
609The following sequence of code needs to be executed to create directories in a tree in several
610levels:
611$ mkdir /home 2> /dev/null
612$ mkdir /home/slynux 2> /dev/null
613$ mkdir /home/slynux/test 2> /dev/null
614$ mkdir /home/slynux/test/hello 2> /dev/null
615$ mkdir /home/slynux/test/hello/child 2> /dev/null
616If an error, such as "Directory exists", is encountered, it is ignored and the error message
617is dumped to the  /dev/null device using the  2> redirection. But this is lengthy and non-
618standard. The standard one-liner to perform this action is:
619$ mkdir -p /home/slynux/test/hello/child
620This single command takes the place of the five different commands listed above. It ignores
621if any level of directory exists and creates the missing directories.
622File permissions, ownership, and sticky bit
623File permissions and ownership are one of the distinguishing features of UNIX/Linux file
624systems such as extended (ext FS). In many circumstances while working on UNIX/Linux
625platforms, we come across issues related to permissions and ownership. This recipe is a walk
626through different use cases of permissions and ownership.
627Getting ready
628In Linux systems, each file is associated with many types of permissions. Out of these
629permissions, three set of permissions (user, group, and others) are commonly manipulated.
630www.it-ebooks.info
631Chapter 3
632105
633The user is the owner of the file. The group is the collection of users (as defined by the
634system) that are permitted some access to the file. Others are any entity other than the user
635or group owner of the file.
636Permissions of a file can be listed by using the  ls -l command:
637-rw-r--r-- 1 slynux slynux 2497 2010-02-28 11:22 bot.py
638-rw-r--r-- 1 slynux slynux 16237 2010-02-06 21:42 c9.php
639drwxr-xr-x 2 slynux slynux 4096 2010-05-27 14:31a.py
640-rw-r--r-- 1 slynux slynux 539 2010-02-10 09:11 cl.pl
641The first column of output specifies the following. The first letter corresponds to:
642f "-"—if it is a regular file.
643f "d"—if it is a directory
644f "c"—for a character device
645f "b"—for a block device
646f "l"—if it is a symbolic link
647f "s"—for a socket
648f "p"—for a pipe
649The rest of the portions can be divided into three groups of three letters each ( ------ ). The
650first  --- three characters correspond the permissions of the user (owner), the second set
651of three characters correspond to the permissions of the group, and the third set of three
652characters correspond to the permissions of others. Each character in the nine character
653sequence (nine permissions) specifies whether a permission is set or unset. If the permission
654is set, a character appears in the corresponding position, else a '-' character appears in that
655position, which means that the corresponding permission is unset (unavailable).
656Let's take a look at what each of these three character set means for the user, group,
657and others.
658User:
659Permission string:  rwx------
660The first letter in the three letters specifies whether the user has read permission for the file.
661If the read permission is set for the user, the character  r will appear as the first character.
662Similarly, the second character specifies write (modify) permission ( w ) and the third character
663specifies whether the user has execute ( x ) permission (the permission to run the file). The
664execute permission is usually set for executable files. User has one more special permission
665called setuid ( S ), which appears in the position of execute ( x ). The setuid permission enables
666an executable file to be executed effectively as its owner, even when the executable is run by
667another user.
668www.it-ebooks.info
669File In, File Out
670106
671An example for a file with setuid permission set is as follows:
672-rwS------
673The read, write, and execute permissions are also applied to the directories. However, the
674interpretation of read, write, and execute permissions are slightly different in the context of
675directories as follows:
676f Read permission ( r ) for the directories enables to read the list of files and sub-
677directories in the directory
678f Write permission ( w ) for a directory enables to create or remove files and directories
679from a directory
680f Execute permission ( x ) specifies whether the access to the files and directories in a
681directory is possible or not
682Group:
683Permission string:  ---rwx---
684The second set of three characters specifies the group permissions. The interpretation of
685permissions  rwx is the same as the permissions for user. Instead of setuid, the group has
686a setgid ( S ) bit. It enables to run an executable file with an effective group as the owner
687group. But the group, which initiates the command, may be different. An example of group
688permission is as follows:
689----rwS---
690Others:
691Permission string:  ------rwx
692Other permissions appear as the last three character set in the permission string. Others have
693the same read, write, and execute permissions as the user and group. But it does not have
694permission  S (like setuid and setgid).
695Directories have a special permission called sticky bit. When a sticky bit is set for a directory,
696the user who created the directory can only delete the files in the directory even if group and
697others have write permissions. The sticky bit appears in the position of execute character ( x )
698in the others permission set. It is represented as character  t or  T .  t appears in the position of
699x if the execute permission is unset and the sticky bit is set. If the sticky bit and the execute
700permission is set, character  T appears in the position of  x .
701For example:
702------ rwt ,  ------ rwT
703A typical example of a directory with sticky bit turned on by default is  /tmp . The sticky bit is a
704type of write-protection.
705www.it-ebooks.info
706Chapter 3
707107
708In each of the  ls -l output line, the string  slynux slynux corresponds to the owned user
709and owned group. Here the first 'slynux' is the user and the second 'slynux' is the group owner.
710How to do it...
711In order to set permissions for files, we use the  chmod command.
712Assume that we need to set permission:  rwx rw- r--
713This could be set using chmod as follows:
714$ chmod u=rwx g=rw o=r filename
715Here:
716f u = specifies user permissions
717f g = specifies group permissions
718f o = specifies others permissions
719In order to add additional permissions on the current file, use + to add permission to user,
720group or others and use – to remove the permissions. Add the executable permission to a file,
721which is already having the permission  rwx rw- r-- as follows:
722$ chmod o+x filename
723This command adds the  x permission for others.
724Add the executable permission to all permission categories that is, for user, group, and others
725as follows:
726$ chmod a+x filename
727Here  a means all.
728In order to remove any permission, use -. For example:
729$ chmod a-x filename
730Permissions can also be set using octal numbers. Permissions are denoted by three-digit octal
731numbers in which each of the digit corresponds to user, group, and other in the order.
732Read, write, and execute permissions have unique octal numbers as follows:
733f r-- = 4
734f -w- = 2
735f --x = 1
736www.it-ebooks.info
737File In, File Out
738108
739We can get the required combination of permissions by adding the octal values for the
740required permission sets. For example:
741f rw- = 4 + 2 = 6
742f r-x = 4 + 1 = 5
743The permission  rwx rw- r-- in numeric method is as follows:
744f rwx = 4 + 2 + 1 = 7
745f rw- = 4 + 2 = 6
746f r-- = 4
747Therefore,  rwx rw- r--  is equal to 764, and the command for setting the permissions
748using octal values is:
749$ chmod 764 filename
750There's more...
751Let's go through some additional tasks that can be performed for files and directories.
752Changing ownership
753In order to change ownership of files, use the  chown command as follows:
754$ chown user.group filename
755For example:
756$ chown slynux.slynux test.sh
757Here,  slynux is the user as well as the group.
758Setting the sticky bit
759The sticky bit is an interesting type of permission applied to directories. By setting the sticky
760bit, it restricts only the user owning it to delete the files even though group and others have
761sufficient permissions.
762In order to set the sticky bit,  +t is applied on a directory with  chmod as follows:
763$ chmod a+t directory_name
764Applying permissions recursively to files
765Sometimes it may be required to recursively change the permissions of all the files and
766directories inside the current directory. This can be done as follows:
767$ chmod 777 . –R
768The  -R option specifies to apply change permission recursively.
769www.it-ebooks.info
770Chapter 3
771109
772We have used "." to specify the path as the current working directory. It is equivalent to:
773$ chmod 777 "$(pwd)" –R.
774Sarath Lakshman 7 January 2011 8:41 PM
775Applying ownership recursively
776We can apply the ownership recursively by using the  -R flag with the  chown command as
777follows:
778$ chown user.group . -R
779Running an executable as a different user (setuid)
780Some executables need to be executed as a different user (other than the current user that
781initiates the execution of the file), effectively, whenever they are executed, by using the file
782path, such as  ./executable_name . A special permission attribute for files called  setuid
783permission enables to effectively execute as the file owner when any other user runs the
784program.
785First change the ownership to the user to which it needs to be executed every time and login
786as the owner user. Then, run the following command:
787$ chmod +s executable_file
788# chown root.root executable_file
789# chmod +s executable_file
790$ ./executable_file
791Now it executes effectively as the root user every time.
792setuid is restricted such that  setuid won't work for scripts, but only for Linux ELF binaries.
793This is a fix for ensuring security.
794Making files immutable
795Files on extended type file systems, which are common in Linux (for example, ext2, ext3, ext4,
796and so on) can be made immutable. Certain type of file attributes help to set the immutable
797attribute to the file. When a file is made immutable, any user or super user cannot remove
798the file until the immutable attribute is removed from the file. We can easily find out the file
799system type of any mounted partition by looking at the  /etc/mtab file. The first column of
800the file specifies the partition device path (for example,  /dev/sda5 ) and the third column
801specifies the file system type (for example, ext3). Let's see how to make files immutable.
802www.it-ebooks.info
803File In, File Out
804110
805Getting ready
806chattr can be used for to make files immutable. However, it is not the only extended
807attribute that can be changed by chattr.
808Making a file immutable is one of the methods for securing files from modification. The best
809known example is in the case of the  /etc/shadow file. The shadow file consists of encrypted
810passwords of every user in the current system. By injecting encrypted passwords, we can login
811into the system. Users can, usually, change their password by using the  passwd command.
812When you execute the  passwd command, it actually modifies the  /etc/shadow file. We can
813make the shadow file immutable so that no user is able to change the password. Let's see
814how to do it.
815How to do it...
816A file can be made immutable as follows:
817chattr +i file
818Or:
819$ sudo chattr +i file
820The file is therefore made immutable. Now try the following command:
821rm file
822rm: cannot remove `file': Operation not permitted
823In order to make it writable, remove the immutable attribute as follows:
824chattr -i file
825Generating blank files in bulk
826Sometimes we many need to generate test cases. We may use programs that operate on
8271000s of files. But how are test files generated?
828Getting ready
829touch is a command that can create blank files or modify the timestamp of files if they
830already exist. Let's take a look at how to use them.
831How to do it...
832A blank file with the name  filename will be created using the following command:
833$ touch filename
834www.it-ebooks.info
835Chapter 3
836111
837Generate bulk files with a different name pattern as follows:
838for name in {1..100}.txt
839do
840touch $name
841done
842In the above code  {1..100} will be expanded as a string "1, 2, 3, 4, 5, 6, 7...100". Instead
843of  {1..100}.txt , we can use various shorthand patterns such as  test{1..200}.c ,
844test{a..z}.txt , and so on.
845If a file already exists, then the  touch command changes all timestamps associated with
846the file to the current time. However, if we want to specify that only certain stamps are to be
847modified, we use the following options:
848f touch -a  modifies only the access time
849f touch -m  modifies only the modification time
850Instead of using the current time for the timestamp, we can specify the time and date with
851which to stamp the file as follows:
852$ touch -d "Fri Jun 25 20:50:14 IST 1999" filename
853The date string that is used with  –d need not always be in the same format. It will accept any
854standard date formats. We can omit time from the string and provide handy date formats like
855"Jan 20 2010".
856Finding a symbolic link and its target
857Symbolic links are common with UNIX-like systems. We may come across various
858manipulations based on symbolic links. This recipe may not be having any practical purpose,
859but it gives practice of handling symbolic links that may be helpful in writing shell scripts for
860other purposes.
861Getting ready
862Symbolic links are just pointers to other files. They are similar in function to aliases in Mac OS
863X or shortcuts in Windows. When symbolic links are removed, they will not cause any harm to
864the original file.
865How to do it...
866We can create a symbolic link as follows:
867$ ln -s target symbolic_link_name
868www.it-ebooks.info
869File In, File Out
870112
871For example:
872$ ln –l -s /var/www/ ~/web
873This creates a symbolic link (called "web") in the logged in user's home directory. The link
874points to  /var/www/ . This is seen in the output of the following command:
875$ ls web
876lrwxrwxrwx 1 slynux slynux 8 2010-06-25 21:34 web -> /var/www
877web -> /var/www specifies that  web points to  /var/www .
878For every symbolic link, the permission notation block ( lrwxrwxrwx ) starts with letter "l",
879which represents a symlink.
880So, in order to print symbolic links in the current directory, use the following command:
881$ ls -l | grep "^l" | awk '{ print $8 }'
882grep will filter the lines from the  ls -l output such that it displays only lines starting with l.
883^ is the start marker for the string.  awk is used to print the eighth column. Hence it prints the
884eighth column, which is the filename.
885Another way to print symbolic links is to use  find as follows:
886$ find . -type l -print
887In the above command, in the  find argument  type we have specified "l", which will instruct
888the  find command to search only for symbolic link files. The  –print option is used to print
889the list of symbolic links to the standard output ( stdout ). The path from which the file search
890should begin is given as '.', which means it is the current directory.
891In order to print the target of a symbolic link use the following command:
892$ ls -l web | awk '{ print $10 }'
893/var/www
894The  ls –l command lists many details with each of the line corresponding to the details of
895a file.  ls –l web lists the details for the file called  web , which is a symbolic link. The tenth
896column in the output of  ls –l contains the link to which the file points to (if the file is a
897symbolic link). Hence in order to find the target associated with a symbolic link, we can use
898awk to print the tenth column from the file details listing (the output from  ls –l ).
899Or, alternately, we can use the standard way of reading the target path for a given symbolic link
900using the command  readlink . It is the most preferred method and can be used as follows:
901$ readlink web
902/var/www
903www.it-ebooks.info
904Chapter 3
905113
906Enumerating file type statistics
907There are many file types. It will be an interesting exercise to write a script that can enumerate
908through all the files inside a directory, its descendants, and print a report that provides details
909on types of files (files with different file types) and the count of each file type present. This
910recipe is an exercise on how to write scripts that can enumerate through a bulk of files and
911collecting details.
912Getting ready
913The file command can be used to find out the type of the file by looking at the contents of the
914file. In UNIX/Linux systems, file types are not determined based on the extension of the file
915(like the Microsoft Windows platform does). This recipe aims at collecting file type statistics
916of a number of files. For storing the count of files of the same type, we can use an associative
917array and the  file command can be used to fetch the file type details from each of the files.
918How to do it...
919In order to print the file type of a file use the following command:
920$ file filename
921$ file /etc/passwd
922/etc/passwd: ASCII text
923Print the file type only by excluding the filename as follows:
924$ file -b filename
925ASCII text
926The script for files statistics is as follows:
927#!/bin/bash
928# Filename: filestat.sh
929if [ $# -ne 1 ];
930then
931echo $0 basepath;
932echo
933fi
934path=$1
935declare -A statarray;
936while read line;
937do
938www.it-ebooks.info
939File In, File Out
940114
941ftype=`file -b "$line"`
942let statarray["$ftype"]++;
943done< <(find $path -type f -print)
944echo ============ File types and counts =============
945for ftype in "${!statarray[@]}";
946do
947echo $ftype : ${statarray["$ftype"]}
948done
949The usage is as follows:
950$ ./filestat.sh /home/slynux/temp
951A sample output is shown below:
952$ ./filetype.sh /home/slynux/programs
953============ File types and counts =============
954Vim swap file : 1
955ELF 32-bit LSB executable : 6
956ASCII text : 2
957ASCII C program text : 10
958How it works...
959Here an associative array named  statarray is declared so that it can take file type as file
960indices and store the count of each file type in the array.  let is used to increment the count
961each time when a file type is encountered. The  find command is used to get the list of file
962paths recursively. A  while loop is used to iterate line by line through the  find command's
963output. The input line  ftype=`file -b "$line"` in the previous script is used to find out
964the file type using the  file command. The  –b option specifies file command to print only file
965type (without filename in the output). The file type output consists of more details, such as
966image encoding used and resolution (in the case of an image file). But we are not interested
967in more details, we need only the basic information. Details are comma separated as in the
968following example:
969$ file a.out -b
970ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically
971linked (uses shared libs), for GNU/Linux 2.6.15, not stripped
972We need to extract only the "ELF 32-bit LSB executable" from the above details. Hence we use
973cut –d, -f1 , which specifies to use " , " as the delimiter and print only the first field.
974www.it-ebooks.info
975Chapter 3
976115
977done< <(find $path –type f –print); is an important bit of code. The logic is as
978follows:
979while  read line;
980do something
981done< filename
982Instead of the filename we used the output of  find .
983<(find $path -type f -print) is equivalent to a filename. But it substitutes filename
984with subprocess output. Note that there is an additional  < .
985${!statarray[@]} is used to return the list of array indexes.
986Loopback files and mounting
987Loopback filesystems are very interesting components of Linux like systems. We usually
988create filesystems on devices (for example, disk drive partitions). These storage devices
989are available as device files like  /dev/device_name . In order to use the storage device
990filesystem, we need to mount it at some directory called a mount point. Loopback filesystems
991are those that we create in files rather than a physical device. We can mount those files as
992devices at a mount point. Let's see how to do it.
993Getting ready
994Loopback filesystems reside on a file. We mount these files by attaching it to a device file. An
995example of a loopback filesystem is the initial ramdisk file, which you would see at  boot/
996initrd.img . It stores an initial filesystem for the kernel in a file.
997Let's see how to create an ext4 filesystem on a file of size 1GB.
998How to do it...
999The following command will create a file that is 1 GB in size.
1000$ dd if=/dev/zero of=loopbackfile.img bs=1G count=1
10011024+0 records in
10021024+0 records out
10031073741824 bytes (1.1 GB) copied, 37.3155 s, 28.8 MB/s
1004You can see that the size of the created file exceeds 1GB. This is because the hard disk is a
1005block device and hence storage is allocated by integral multiples of blocks size.
1006www.it-ebooks.info
1007File In, File Out
1008116
1009Now format the 1GB file using the  mkfs command as follows:
1010# mkfs.ext4 loopbackfile.img
1011This command formats it to ext4. Check the file type using the following command:
1012$ sudo file loopbackfile.img
1013loopbackfile.img: Linux rev 1.0 ext4 filesystem data, UUID=c9d56c42-f8e6-
10144cbd-aeab-369d5056660a (extents) (large files) (huge files)
1015Now you can mount the loopback file as follows:
1016$ sudo mkdir /mnt/loopback
1017# mount -o loop loopback.img /mnt/loopback
1018The  -o loop additional option is used to mount any loopback file systems.
1019This is the shortcut method. We do not attach it to any devices. But internally it attaches to a
1020device called  /dev/loop1 or  loop2 .
1021We can do it manually as follows:
1022# losetup /dev/loop1 loopback.img
1023# mount /dev/loop1 /mnt/loopback
1024The first method cannot be used in all circumstances. Suppose we want to create a hard disk
1025file, and then want to partition it and mount a sub partition, we cannot use  mount -o loop .
1026We have to use the second method. Partition a zeros dumped file as follows:
1027# losetup /dev/loop1 loopback.img
1028# fdisk /dev/loop1
1029Create partitions in  loopback.img in order to mount the first partition as follows:
1030# losetup -o 32256 /dev/loop2 loopback.img
1031Now  /dev/loop2 represents first partition.
1032-o is the offset flag.  32256 bytes are for a DOS partition scheme. The first partition starts
1033after an offset of 32256 bytes from the start of the hard disk.
1034We can set up the second partition by specifying the required offset. After mounting we can
1035perform all regular operations as we can on physical devices.
1036In order to  umount , use the following syntax:
1037# umount mount_point
1038www.it-ebooks.info
1039Chapter 3
1040117
1041For example:
1042# umount /mnt/sda1
1043Or, alternately, we can use device file path as an argument to the  umount command as:
1044# umount /dev/sda1
1045Note that  umount command should be executed as a root user since it is a privileged command.
1046There's more...
1047Let's explore more about additional mount options.
1048Mounting ISO files as loopback
1049An ISO file is an archive of any optical media. We can mount ISO files in the same way that we
1050mount physical discs by using loopback mounting.
1051A mount point is just a directory, which is used as access path to contents of a device through
1052a filesystem. We can even use a non-empty directory as the mount path. Then the mount path
1053will contain data from the devices rather than original contents until the device is unmounted.
1054For example:
1055# mkdir /mnt/iso
1056# mount -o loop linux.iso /mnt/iso
1057Now perform operations using files from  /mnt/iso . ISO is a read-only filesystem.
1058Flush changes immediately with sync
1059While making changes on a mounted device, they are not immediately written to the physical
1060devices. They are only written when the buffer is full. But we can force writing of changes
1061immediately by using the  sync command as follows:
1062# sync
1063You should execute the  sync command as root.
1064Creating ISO files, Hybrid ISO
1065An ISO image is an archive format that stores the exact storage images of optical disks like
1066CD ROMs, DVD ROMs, and so on. It is a common use case that we burn ISO images to optical
1067disks. But what if you want to create an image of an optical disk? For that we need to create
1068an ISO image from an optical disk. Many people rely on third-party utilities to create an ISO
1069image from an optical disk. However, using the command line, it's just a single line job.
1070www.it-ebooks.info
1071File In, File Out
1072118
1073Also, many people don't distinguish between bootable and non-bootable optical disks.
1074Bootable disks are capable of booting from themselves and also running an operating system
1075or another product. Non-bootable ISOs cannot do that. The practice that people usually follow
1076is to copy files from a bootable CD-ROM and paste it to another location for keeping the copy.
1077After that, they use the copied directory to burn a CD ROM. But then, it will lose its bootable
1078nature. To preserve the bootable nature, it should be copied as a disk image or an ISO file.
1079Nowadays, most people use devices such as flash drives or hard disks as a replacement for
1080optical disks. When we write a bootable ISO to a flash drive it will no longer be bootable unless
1081we use a special hybrid ISO image designed specifically for the purpose.
1082This recipe will give you an insight on ISO images and manipulations.
1083Getting ready
1084As we described many times in this book, UNIX handles everything as files. Every device is a
1085file. Hence what if we want to copy an exact image of a device? We need to read all data from
1086it and write to another file, right?
1087As we know, the  cat command can be used to read any data and redirection can be used
1088to write to a file.
1089How to do it...
1090In order to create an ISO image from  /dev/cdrom use the following command:
1091# cat /dev/cdrom > image.iso
1092This will work, it will read all the bytes from the device and write an ISO image.
1093Using the  cat command for creating an ISO image is a tricky way to do it. But the most
1094preferred way to create an ISO image is to use the  dd utility.
1095# dd if=/dev/cdrom of=image.iso
1096mkisofs is a command used to create ISO system. The output file of  mkisofs can be written
1097to CD ROM or DVD ROM using utilities like  cdrecord . We can use  mkisofs to create an ISO
1098file using a directory containing all the required files that should appear as contents of an ISO
1099file as follows:
1100$ mkisofs -V "Label" -o image.iso source_dir/
1101The  –o option in the  mkisofs command specifies the ISO file path. The  source_dir is the
1102path of the directory that should be used as source content for the ISO and the  –V option
1103specifies the label that should be used for the ISO file.
1104www.it-ebooks.info
1105Chapter 3
1106119
1107There's more...
1108Let's learn more commands and techniques related to ISO files.
1109Hybrid ISO that boots off flash drive or hard disk
1110Usually, bootable ISO files cannot be transferred or written to a USB storage device and boot
1111the OS from the USB key. But special type of ISO files called hybrid ISOs can be flashed and
1112they are capable of booting from such devices.
1113We can convert standard ISO files into hybrid ISOs with the  isohybrid command. The
1114isohybrid command is a new utility and most Linux distros don't include this by default. You
1115can download the syslinux package from:  http://syslinux.zytor.com .
1116Have a look at the following command:
1117# isohybrid image.iso
1118Using this command, we will have a hybrid ISO with the file name  image.iso and it can be
1119written to USB storage devices.
1120Write the ISO to a USB storage by using the following command:
1121# dd if=image.iso of=/dev/sdb1
1122Use the appropriate device instead of  sdb1 .
1123Or, you can use  cat as follows:
1124# cat image.iso > /dev/sdb1
1125Burning an ISO from command line
1126The  cdrecord command is used to burn an ISO file into a CD ROM or DVD ROM. It can be
1127used to burn the image to the CD ROM as follows:
1128# cdrecord -v dev=/dev/cdrom image.iso
1129Some extra options are as follows:
1130f We can specify the burning speed with the  –speed option as follows:
1131-speed SPEED
1132For example:
1133# cdrecord –v dev=/dev/cdrom image.iso –speed 8
1134The speed is 8x, which is specified as 8.
1135www.it-ebooks.info
1136File In, File Out
1137120
1138f A CD ROM can be burned in multisessions such that we can burn data multiple
1139times on a disk. Multisession burning can be performed using the  –multi option as
1140follows:
1141# cdrecord –v dev=/dev/cdrom image.iso -multi
1142Playing with CD Rom tray
1143Try the following commands and have fun:
1144f $ eject
1145This command is used to eject the tray.
1146f $ eject -t
1147This command is used to close the tray.
1148Try to write a loop that opens the tray and closes the tray for "N" number of times.
1149Finding difference between files, patching
1150When multiple versions of a file are available, it is very useful when we can find the
1151differences between files being highlighted rather than comparing two files manually by
1152looking through them. If the files are of 1000s of lines, they are practically very difficult and
1153time consuming to compare. This recipe illustrates how to generate differences between
1154files highlighted with line numbers. When working on large files by multiple developers, when
1155one of them has made changes and these changes need to be shown to the other, sending
1156the entire source code to other developers is costly in consumption of space and time to
1157manually check the changes. Sending a different file is helpful. It consists of only lines that
1158are changed, added, or removed and line numbers are attached with it. This difference file is
1159called a patch file. We can add the changes specified in the patch file to the original source
1160code by using the patch command. We can also revert the changes by patching again. Let's
1161see how to do this.
1162How to do it...
1163The  diff  command utility is used to generate difference files.
1164In order to generate difference information, create the following files:
1165f File 1:  version1.txt
1166this is the original text
1167line2
1168line3
1169line4
1170happy hacking !
1171www.it-ebooks.info
1172Chapter 3
1173121
1174f File 2:  version2.txt
1175this is the original text
1176line2
1177line4
1178happy hacking !
1179GNU is not UNIX
1180Non-unified  diff output (without the  –u flag) will be as follows:
1181$ diff version1.txt version2.txt
11823d2
1183<line3
11846c5
1185> GNU is not UNIX
1186The unified  diff output will be as follows::
1187$ diff -u version1.txt version2.txt
1188--- version1.txt  2010-06-27 10:26:54.384884455 +0530
1189+++ version2.txt  2010-06-27 10:27:28.782140889 +0530
1190@@ -1,5 +1,5 @@
1191this is the original text
1192line2
1193-line3
1194line4
1195happy hacking !
1196-
1197+GNU is not UNIX
1198The  -u option is used to produce unified output. Everyone prefers unified output, as the
1199unified output is more readable and because it is easier to interpret the difference that is
1200being made between two files.
1201In unified  diff , the lines starting with  + are the newly added lines and the lines starting with
1202– are the removed lines.
1203A patch file can be generated by redirecting the  diff output to a file, as follows:
1204$ diff -u version1.txt version2.txt > version.patch
1205Now using the patch command we can apply changes to any of the two files. When applied to
1206version1.txt , we get  version2.txt file. When applied to  version2.txt , we receive
1207version1.txt .
1208www.it-ebooks.info
1209File In, File Out
1210122
1211Apply the patch by using the following command:
1212$ patch -p1 version1.txt < version.patch
1213patching file version1.txt
1214We now have  version1.txt with the same contents as that of  version2.txt .
1215In order to revert the changes back, use the following command:
1216$ patch -p1 version1.txt < version.patch
1217patching file version1.txt
1218Reversed (or previously applied) patch detected! Assume -R? [n] y
1219#Changes are reverted.
1220Revert the changes without prompting the user with  y/n by using the  –R option along with the
1221patch command.
1222There's more...
1223Let's go through additional features available with  diff .
1224Generating diff against directories
1225The  diff  command can also act recursively against directories. It will generate a difference
1226output for all the descendant files in the directories.
1227Use the following command:
1228$ diff -Naur directory1 directory2
1229The interpretation of each of the above options is as follows:
1230f -N is for treating absent files as empty
1231f -a is to consider all files as text files
1232f -u is to produce unified output
1233f -r is to recursively traverse through the files in the directories
1234head and tail – printing the last or first
123510 lines
1236When looking into a large file, which consists of thousands of lines, we will not use a
1237command like  cat to print the entire file contents. Instead we look for a sample (for example,
1238the first 10 lines of the file or the last 10 lines of the file). We may also need to print the first n
1239lines or last n lines. Also we may need to print all the lines except the last "n" lines or all lines
1240except first "n" lines.
1241www.it-ebooks.info
1242Chapter 3
1243123
1244Another use case is to print lines from n-th to m-th lines.
1245The commands  head and  tail can help us do this.
1246How to do it...
1247The  head command always reads the header portion of the input file.
1248Print first 10 lines as follows:
1249$ head file
1250Read the data from stdin as follows:
1251$ cat text | head
1252Specify the number of first lines to be printed as follows:
1253$ head -n 4 file
1254This command prints four lines.
1255Print all lines excluding the last  N lines as follows:
1256$ head -n -N file
1257Note that it is negative N.
1258For example, to print all the lines except the last 5 lines use the following code:
1259$ seq 11 | head -n -5
12601
12612
12623
12634
12645
12656
1266The following command will, however, print from 1 to 5:
1267$ seq 100 | head -n 5
1268Printing by excluding the last lines is a very important usage of  head . But people always look
1269at some other complex methods to do the same.
1270Print the last 10 lines of a file as follows:
1271$ tail file
1272www.it-ebooks.info
1273File In, File Out
1274124
1275In order to read from  stdin , you can use the following code:
1276$ cat text | tail
1277Print the last 5 lines as follows:
1278$ tail -n 5 file
1279In order to print all lines excluding first N lines, use the following code:
1280$ tail -n +(N+1)
1281For example, to print all lines except the first 5 lines, N + 1 = 6, therefore the command will be
1282as follows:
1283$ seq 100 | tail -n +6
1284This will print from 6 to 100.
1285One of the important usages of  tail is to read a constantly growing file. Since new lines are
1286constantly appended to the end of the file,  tail can be used to display all new lines as they
1287are written to the file. When we run  tail simply, it will read the last 10 lines and exit. However,
1288by that time, new lines would have been appended to the file by some process. In order to
1289constantly monitor the growth of file,  tail has a special option  -f or  --follow , which enables
1290tail to follow the appended lines and keep being updated with the data growth:
1291$ tail -f growing_file
1292An example of such growing files are logfiles. The command to monitor the growth of the files
1293would be:
1294# tail -f /var/log/messages
1295or
1296$ dmesg | tail -f
1297We frequently run  dmesg to look at kernel ring buffer messages either to debug the USB
1298devices or to look at the  sdX ( X is the minor number for the  sd device). The  tail -f can
1299also add a sleep interval  -s , so that we can set the interval during which the file updates are
1300monitored.
1301tail has the interesting property that allows it to terminate after a given process ID dies.
1302Suppose we are reading a growing file, and a process  Foo is appending data to the file,
1303tail -f should be executed until process  Foo dies.
1304$ PID=$(pidof Foo)
1305$ tail -f file --pid $PID
1306When the process  Foo terminates,  tail also terminates.
1307www.it-ebooks.info
1308Chapter 3
1309125
1310Let's work on an example.
1311Create a new file  file.txt and open the file in gedit (You can use any text editor).
1312Add new lines to the file and make frequent file saves in gedit.
1313Now run:
1314$ PID=$(pidof gedit)
1315$ tail -f file.txt --pid $PID
1316When you make frequent changes to the file, it will be written to the terminal by the tail
1317command. When you close the  gedit , the  tail command will get terminated.
1318Listing only directories – alternative
1319methods
1320Though listing only directories seems to be a simple task, many would not be able to do it.
1321I have seen this often, even when asked to people who are good at shell scripting. This
1322recipe is worth knowing since it introduces multiple ways of listing only directories with
1323various tricky techniques.
1324Getting ready
1325There are multiple ways of listing directories only. When you ask people about these techniques,
1326the first answer that they would probably give is  dir . But, it is wrong. The  dir command is just
1327another command like  ls with fewer options than  ls . Let's see how to list directories.
1328How to do it...
1329There are four ways in which directories in the current path can be displayed. They are:
1330f $ ls -d */
1331Only the above combination with  -d will print directories.
1332f $ ls -F | grep "/$"
1333When the  -F parameter is used, all entries are appended with some type of file
1334character such as  @ ,  * ,  | , and so on. For directories, entries are appended with the  /
1335character. We use  grep to filter only entries ending with the  /$ end of line indicator.
1336f $ ls -l | grep "^d"
1337The first character of  ls -d output lines of each file entries is the type of file
1338character. For directory, the type of file character is  "d" . Hence we use  grep to filter
1339lines starting with  "d" .  ^  is the start of line indicator.
1340www.it-ebooks.info
1341File In, File Out
1342126
1343f $ find . -type d -maxdepth 1 -print
1344The  find command can take the parameter  type as directory and  maxdepth is set
1345to  1 since it should not search the directories of descendants.
1346Fast command-line navigation using pushd
1347and popd
1348When dealing with multiple locations on a terminal or shell prompt, our common practice is
1349to copy and paste the paths. Copy-paste is only effective when mouse is used. When there is
1350only command-line access without a GUI, it is hard to deal with navigation through multiple
1351paths. For example, if we are dealing with locations  /var/www ,  /home/slynux , and  /
1352usr/src , when we need to navigate these locations one by one, it is really difficult to type
1353the path every time when we need to switch between the paths. Hence the command-line
1354interface (CLI) based navigation techniques such as pushd and popd are used. Let's see how
1355to practice them.
1356Getting ready
1357pushd and  popd are used to switch between multiple directories without the copy-paste of
1358directory paths.  pushd and  popd operate on a stack. We know that stack is a Last In First
1359Out (LIFO) data structure. It will store the directory paths in a stack and switch between them
1360using push and pop operations.
1361How to do it...
1362We omit the use of the  cd command while using  pushd and  popd .
1363In order to push and change directory to a path use:
1364~ $ pushd /var/www
1365Now the stack contains  /var/www ~ and the current directory is changed to  /var/www .
1366Now again push the next directory path as follows:
1367/var/www $ pushd /usr/src
1368Now the stack contains  /usr/src /var/www ~ and the current directory is  /usr/src .
1369You can similarly push as many directory paths as needed.
1370www.it-ebooks.info
1371Chapter 3
1372127
1373View the stack contents by using the following command:
1374$ dirs
1375/usr/src /var/www ~ /usr/share /etc
13760 1 2 3 4
1377When you want to switch to any path in the list, number each path from  0 to  n, then use the
1378path number for which we need to switch, for example:
1379$ pushd +3
1380It will rotate the stack and switch to the directory  /usr/share .
1381pushd will always add paths to the stack, to remove paths from the stack use  popd .
1382Remove a last pushed path and change directory to the next directory by using:
1383$ popd
1384Suppose the stack is  /usr/src /var/www ~ /usr/share /etc such that the current
1385directory is  /usr/src ,  popd will change the stack to  /var/www ~ /usr/share /etc and
1386change the directory to  /var/www .
1387In order to remove a specific path from the list, use  popd +no .
1388The  no is counted as  0 to  n from left to right.
1389There's more...
1390Let's go through essential directory navigation practices.
1391Most frequently used directory switching
1392pushd and  popd can be used when there are more than three directory paths are used. But
1393when you use only two locations, there is an alternative and easier way. That is  cd - .
1394If the current path is  /var/www , perform the following:
1395/var/www $ cd /usr/src
1396/usr/src $ # do something
1397Now to switch back to  /var/www , you don't have to type it out again, but just execute:
1398/usr/src $ cd -
1399Now you can switch to  /usr/src as follows:
1400/var/www $ cd -
1401www.it-ebooks.info
1402File In, File Out
1403128
1404Counting number of lines, words, and
1405characters in a file
1406Counting the number of lines, words, and characters from a text or file are very useful for
1407text manipulations. In several cases, count of words or characters are used in indirect
1408ways to perform some hacks to produce required output patterns and results. This book
1409includes some of such tricky examples in other chapters. Counting LOC (Lines of Code) is an
1410important application for developers. We may need to count special types of files excluding
1411unnecessary files. A combination of  wc with other commands help to perform that.
1412Getting ready
1413wc is the utility used for counting. It stands for Word Count (wc). Let's see how to use  wc
1414to count lines, words, and characters.
1415How to do it...
1416Count number of lines as follows:
1417$ wc -l file
1418In order to use  stdin as input, use the following command:
1419$ cat file | wc -l
1420Count the number of words as follows:
1421$ wc -w file
1422$ cat file | wc -w
1423In order to count number of characters, use:
1424$ wc -c file
1425$ cat file | wc -c
1426For example, we can count the characters in a text as follows:
1427echo -n 1234 | wc -c
14284
1429-n is used to avoid an extra newline character.
1430When  wc is executed without any options as:
1431$ wc file
1432it will print number of lines, words, and characters delimited by tabs.
1433www.it-ebooks.info
1434Chapter 3
1435129
1436There's more...
1437Let's go through additional options available with  wc command.
1438Print length of longest length line
1439wc can be also used to print the length of longest line using the  –L option:
1440$ wc file -L
1441Printing directory tree
1442Graphically representing directories and filesystem as tree hierarchy is quite useful when
1443preparing tutorials and documents. Also they are sometimes useful in writing certain
1444monitoring scripts that helps to look at the filesystem using easy-to-read tree representations.
1445Let's see how to do it.
1446Getting ready
1447The  tree command is the hero that helps to print graphical trees of files and directories.
1448Usually,  tree does not come with Linux distributions. You need to install it using the
1449package manager.
1450How to do it...
1451The following is a sample UNIX file system tree to show an example:
1452$ tree ~/unixfs
1453unixfs/
1454|-- bin
1455| |-- cat
1456| `-- ls
1457|-- etc
1458| `-- passwd
1459|-- home
1460| |-- pactpub
1461| | |-- automate.sh
1462| | `-- schedule
1463| `-- slynux
1464|-- opt
1465|-- tmp
1466`-- usr
14678 directories, 5 files
1468www.it-ebooks.info
1469File In, File Out
1470130
1471The tree command comes with many interesting options, let us look at few of them.
1472Highlight only files matched by pattern as follows:
1473$ tree path -P PATTERN # Pattern should be wildcard
1474For example:
1475$ tree PATH -P "*.sh" # Replace PATH with a directory path
1476|-- home
1477| |-- pactpub
1478| | `-- automate.sh
1479Highlight only files excluding the match pattern by using:
1480$ tree path -I PATTERN
1481In order to print size along with files and directories use the  -h option as follows:
1482$ tree -h
1483There's more...
1484Let's see an interesting option that is available with the  tree command.
1485HTML output for tree
1486It is possible to generate HTML output from the  tree command. For example, use the
1487following command to create an HTML file with tree output.
1488$ tree PATH -H http://localhost -o out.html
1489Replace  http://localhost with the URL where you would like to host the file. Replace
1490PATH with a real path for the base directory. For the current directory use '.' as the PATH.
1491The web page generated from the directory listing will look as follows:
1492www.it-ebooks.info
14934
1494Texting and Driving
1495In this chapter, we will cover:
1496f A basic regular expression primer
1497f Searching and mining "text" inside a file with grep
1498f Column-wise cutting of a file with cut
1499f Determining the frequency of words used in a given file
1500f A basic sed primer
1501f A basic awk primer
1502f Replacing strings from a text or file
1503f Compressing or decompressing JavaScript
1504f Iterating through lines, words, and characters in a file
1505f Merging multiple files as columns
1506f Printing the nth word or column in a file or line
1507f Printing text between line numbers or patterns
1508f Checking palindrome strings with a script
1509f Printing lines in the reverse order
1510f Parsing e-mail address and URLs from text
1511f Printing a set number of lines before or after a pattern in a file
1512f Removing a sentence in a file containing a word
1513f Implementing head, tail, and tac with awk
1514f Text slicing and parameter operations
1515www.it-ebooks.info
1516Texting and Driving
1517132
1518Introduction
1519The Shell Scripting language is packed with essential problem-solving components for UNIX/
1520Linux systems. Bash can always provide some quick solutions to the problems in a UNIX
1521environment. Text processing is one of the key areas where shell scripting is used. It comes
1522with beautiful utilities such as sed, awk, grep, cut, and so on, which can be combined to solve
1523text processing related problems. Most of the programming languages are designed to be
1524generic, and hence it takes a lot of effort to write programs that can process text and produce
1525the desired output. Since Bash is a language that is designed by also keeping text processing
1526in mind, it has a lot of functionalities.
1527Various utilities help to process a file in fine detail as a character, line, word, column, row,
1528and so on. Hence we can manipulate a text file in many ways. Regular expressions are the
1529core of pattern matching techniques. Most of the text processing utilities come with regular
1530expression support. By using suitable regular expression strings, we can produce the desired
1531output such as filtering, stripping, replacing, searching, and much more.
1532This chapter includes a collection of recipes, which walks through many contexts of problems
1533based on text processing that will be helpful in writing real scripts.
1534Basic regular expression primer
1535Regular expressions are the heart of the pattern-matching based text-processing techniques.
1536For fluency in writing text-processing tools, one must have basic understanding of regular
1537expressions. Regular expressions are a form of tiny, highly-specialized programming language
1538used to match text. Using wild card techniques, the scope of matching text with patterns is
1539very limited. This recipe is a walk through of basic regular expressions.
1540Getting ready
1541Regular expressions are the language used in most text processing utilities. Hence you will
1542use the techniques learned in this recipe in many other recipes.  [a-z0-9_]+@[a-z0-9]+\.
1543[a-z]+ is an example of regular expression for matching an e-mail address.
1544Does this seem weird? Don't worry, it is really simple once you understand the concepts.
1545How to do it...
1546In this section, we will go through regex, the POSIX character class, and meta characters.
1547Let's first go through the basic components of regular expressions (regex).
1548www.it-ebooks.info
1549Chapter 4
1550133
1551regex Description Example
1552^ The start of the line marker. ^tux matches a string that
1553starts the line with tux.
1554$ The end of the line marker. tux$ matches strings of a
1555line that ends with tux.
1556. Matches any one character. Hack. matches Hack1,
1557Hacki but not Hack12,
1558Hackil, only one additional
1559character matches.
1560[] Matches any one of the characters enclosed in
1561[chars].
1562coo[kl] matches cook or
1563cool.
1564[^] Matches any one of the characters EXCEPT those
1565that are enclosed in [^chars].
15669[^01] matches 92, 93
1567but not 91 or 90.
1568[-] Matches any character within the range specified
1569in [].
1570[1-5] matches any digits
1571from 1 to 5.
1572? The preceding item must match one or zero times. colou?r matches
1573color or colour but not
1574colouur.
1575+ The preceding item must match one or more
1576times.
1577Rollno-9+ matches
1578Rollno-99, Rollno-9
1579but not Rollno-.
1580* The preceding item must match zero or more
1581times.
1582co*l matches cl, col,
1583coool.
1584() Creates a substring from the regex match. ma(tri)?x matches max
1585or matrix.
1586{n} The preceding item must match n times. [0-9]{3} matches any
1587three-digit number. [0-9]
1588{3} can be expanded as:
1589[0-9][0-9][0-9].
1590{n,} Minimum number of times that the preceding item
1591should match.
1592[0-9]{2,} matches any
1593number, that is, two digits or
1594more.
1595{n, m} Specifies the minimum and maximum number of
1596times the preceding item should match.
1597[0-9]{2,5} matches any
1598number that is having two
1599digits to five digits.
1600| Alternation—one of the items on either of sides of
1601| should match.
1602Oct (1st | 2nd)
1603matches Oct 1st or Oct
16042nd.
1605\ The escape character for escaping any of the
1606special characters mentioned above.
1607a\.b matches a.b but
1608not ajb. It ignores special
1609meaning of .by prefexing \.
1610www.it-ebooks.info
1611Texting and Driving
1612134
1613A POSIX character class is a special meta sequence of the form  [:...:] that can be used to
1614match a range of specified characters. The POSIX classes are as follows:
1615Regex Description Example
1616[:alnum:] Alphanumeric character [[:alnum:]]+
1617[:alpha:] Alphabet character (lowercase and uppercase) [[:alpha:]]{4}
1618[:blank:] Space and tab [[:blank:]]*
1619[:digit:] Digit [[:digit:]]?
1620[:lower:] Lowercase alphabet [[:lower:]]{5,}
1621[:upper:] Uppercase alphabet ([[:upper:]]+)?
1622[:punct:] Punctuation [[:punct:]]
1623[:space:] All whitespace characters including newline,
1624carriage return, and so on.
1625[[:space:]]+
1626Meta characters are a type of Perl-style regular expression that is supported by a subset of
1627text processing utilities. Not all of the utilities will support the following notations. But the
1628above character classes and regular expression are universally accepted.
1629Regex Description Example
1630\b Word boundary \bcool\b matches only cool not
1631coolant.
1632\B Non-word boundary cool\B matches coolant and not cool.
1633\d Single digit character b\db matches b2b not bcb.
1634\D Single non-digit b\Db matches bcb not b2b.
1635\w Single word character(alnum and _) \w matches 1 or a not &.
1636\W Single non-word character \w matches & not 1 or a.
1637\n Newline \n Matches a new line.
1638\s Single whitespace  x\sx matches xx not xx.
1639\S Single non-space x\Sx matches xkx not xx.
1640\r Carriage return \r matches carriage return.
1641How it works...
1642The tables seen in the previous section are the key element tables for regular expressions.
1643By using the suitable keys from the tables, we can construct any suitable regular expression
1644string to match text according to the context. regex is a generic language to match text.
1645Therefore, we are not introducing any tools in this recipe. However, it follows in the other
1646recipes in this chapter.
1647www.it-ebooks.info
1648Chapter 4
1649135
1650Let's see a few examples of text matching:
1651f In order to match all words in a given text, we can write the regex as:
1652( ?[a-zA-Z]+ ?)
1653"?" is the notation for optional space that precedes and follows a word. The
1654[a-zA-Z]+ notation represents one or more alphabet characters (a-z and A-Z).
1655f To match an IP address, we can write the regex as:
1656[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}
1657or
1658[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]{1,3}\.[[:digit:]]
1659{1,3}
1660We know that an IP address is in the form 192.168.0.2. It is in the form of four
1661integers (each from 0-255) separated by dots (for example, 192.168.0.2).
1662[0-9] or [:digit:] represents a match for digits 0-9.  {1,3} matches one to three
1663digits and  \. matches ".".
1664There's more...
1665Let's see how the special meanings of certain characters are specified in the regular
1666expressions.
1667Treatment of special characters
1668Regular expressions use some characters such as  $ ,  ^ ,  . ,  * ,  + ,  { , and  } as special characters.
1669But what if we want to use these characters as non-special characters (a normal text
1670character)? Let's see an example.
1671regex:  [a-z]*.[0-9]
1672How is this interpreted?
1673It can be zero or more [a-z] ([a-z]*) , then any one character ( . ), and then one character in
1674the set [0-9] such that it matches  abcdeO9 .
1675It can also be interpreted as one of  [a-z] , then a character  * , then a character  . (period),
1676and a digit such that it matches  x*.8 .
1677In order to overcome this problem, we precede the character with a forward slash "\" (doing
1678this is called "escaping the character"). Characters such as * that have multiple meanings are
1679prefixed with "\" to make them into a special meaning or to make them non special. Whether
1680special characters or non-special characters are to be escaped varies depending on the tool
1681that you are using.
1682www.it-ebooks.info
1683Texting and Driving
1684136
1685Searching and mining "text" inside a file
1686with grep
1687Searching inside a file is an important use case in text processing. We may need to
1688search through thousands of lines in a file to find out some required data by using certain
1689specifications. This recipe will help you learn how to locate data items of a given specification
1690from a pool of data.
1691Getting ready
1692The  grep  command is the master UNIX utility for searching in the text. It accepts regular
1693expressions and wild cards. We can produce output in various formats using the numerous
1694interesting options that come with  grep . Let's see how to do it.
1695How to do it...
1696Search in a file for a word as follows:
1697$ grep match_pattern filename
1698this is the line containing match_pattern
1699Or:
1700$ grep "match_pattern" filename
1701this is the line containing match_pattern
1702It will return lines of text that contain the given  match_pattern .
1703We can also read from  stdin as follows:
1704$ echo -e "this is a word\nnext line" | grep word
1705this is a word
1706Perform a search in multiple files using a single  grep invocation as follows:
1707$ grep "match_text" file1 file2 file3 ...
1708We can highlight the word in the line by using the  --color option as follows:
1709$ grep word filename –-color=auto
1710this is the line containing word
1711Usually, the  grep command considers  match_text as a wildcard. To use regular expressions
1712as input arguments, the  -E option should be added—which means extended regular expression.
1713Or we can a use regular expression enabled  grep command,  egrep . For example:
1714www.it-ebooks.info
1715Chapter 4
1716137
1717$ grep -E "[a-z]+"
1718Or:
1719$ egrep "[a-z]+"
1720In order to output only the matching portion of text in a file, use the  –o  option as follows:
1721$ echo this is a line. | grep -o -E "[a-z]+\."
1722line
1723Or:
1724$ echo this is a line. | egrep -o "[a-z]+\."
1725line.
1726In order to print all of the lines, except the line containing  match_pattern , use:
1727$ grep -v match_pattern file
1728The  –v option added to  grep inverts the match results.
1729Count the number of lines in which a matching string or regex match appears in a file or text
1730as follows:
1731$ grep -c "text" filename
173210
1733It should be noted that  -c counts only the number of matching lines, not the number of times
1734a match is made. For example:
1735$ echo -e "1 2 3 4\nhello\n5 6" | egrep -c "[0-9]"
17362
1737Even though there are 6 matching items, it prints  2 since there are only  2 matching lines.
1738Multiple matches in a single line are counted only once.
1739In order to count the number of matching items in a file, use the following hack:
1740$ echo -e "1 2 3 4\nhello\n5 6" | egrep -o "[0-9]" | wc -l
17416
1742Print the line number of the match string as follows:
1743$ cat sample1.txt
1744gnu is not unix
1745linux is fun
1746bash is art
1747$ cat sample2.txt
1748www.it-ebooks.info
1749Texting and Driving
1750138
1751planetlinux
1752$ grep linux -n sample1.txt
17532:linux is fun
1754Or:
1755$ cat sample1.txt | grep linux -n
1756If multiple files are used, it will also print the filename with the result as follows:
1757$ grep linux -n sample1.txt sample2.txt
1758sample1.txt:2:linux is fun
1759sample2.txt:2:planetlinux
1760Print the character or byte offset at which a pattern matches as follows:
1761$ echo gnu is not unix | grep -b -o "not"
17627:not
1763The character offset for a string in a line is a counter from 0 starting with the first character. In
1764the above example, "not" is at the seventh offset position (that is,  not starts from the seventh
1765character in the line ( gnu is not unix ).
1766The  –b option is always used with  –o .
1767To search over many files and find out in which of the files a certain text matches use:
1768$ grep -l linux sample1.txt sample2.txt
1769sample1.txt
1770sample2.txt
1771The inverse of the  –l argument is  –L . The  -L argument returns a list of non-matching files.
1772There's more...
1773We have used the basic usage examples for the  grep command. But the  grep command
1774comes with rich features. Let's go through the different options available along with  grep .
1775Recursively search many files
1776To recursively search for a text over many directories of descendants use:
1777$ grep "text" . -R -n
1778In this command  "." specifies the current directory.
1779www.it-ebooks.info
1780Chapter 4
1781139
1782For example:
1783$ cd src_dir
1784$ grep "test_function()" . -R -n
1785./miscutils/test.c:16:test_function();
1786test_function() exists in line number 16 of  miscutils/test.c .
1787This is one of the most frequently used commands by developers. It is used to
1788find the file of source code in which a certain text exists.
1789Ignoring case of pattern
1790The  –i argument helps match patterns to be evaluated without considering if the characters
1791are uppercase or lowercase. For example:
1792$ echo hello world | grep -i "HELLO"
1793hello
1794grep by matching multiple patterns
1795Usually, we can specify single pattern for matching. However, we can use an argument  -e to
1796specify multiple patterns for matching as follows:
1797$ grep -e "pattern1" -e "pattern"
1798For example:
1799$ echo this is a line of text | grep -e "this" -e "line" -o
1800this
1801line
1802There is also another way to specify multiple patterns. We can use a pattern file for reading
1803patterns. Write patterns to match line by line and execute  grep with a  -f argument as follows:
1804$ grep -f pattern_file source_filename
1805For example:
1806$ cat pat_file
1807hello
1808cool
1809$ echo hello this is cool | grep -f pat_file
1810hello this is cool
1811www.it-ebooks.info
1812Texting and Driving
1813140
1814Include and exclude files (wild card pattern) in grep search
1815grep can include or exclude files in which to search. We can specify include files or exclude
1816files using wild card patterns.
1817To search only  .c and  .cpp files recursively in a directory by excluding all other file types, use:
1818$ grep "main()" . -r --include *.{c,cpp}
1819Note that  some{string1,string2,string3} expands as  somestring1 somestring2
1820somestring3 .
1821Exclude all README files in the search as follows:
1822$ grep "main()" . -r –-exclude "README"
1823To exclude directories use the  --exclude-dir option.
1824To read a list of files to exclude from a file use  --exclude-from FILE .
1825Using grep with xargs with zero-byte suffix
1826The  xargs command is often used to provide a list of file names as a command-line
1827argument to another command. When filenames are used as command-line arguments, it is
1828recommended to use a zero-byte terminator for the file names instead of a space terminator.
1829Some of the file names can contain a space character and it will be misinterpreted as a
1830terminator and a single file name may be broken into two file names (for example,  New file.
1831txt can be interpreted as two filenames  New and  file.txt ). This problem can be avoided
1832by using a zero-byte suffix. We use  xargs so as to accept  stdin text from commands like
1833grep ,  find , and so on. Such commands can output text to the  stdout with a zero-byte
1834suffix. In order to specify that the input terminator for filenames is zero byte ( \0 ), we should
1835use  –0 with  xargs.
1836Create some test files as follows:
1837$ echo "test" > file1
1838$ echo "cool" > file2
1839$ echo "test" > file3
1840In the following command sequence,  grep outputs filenames with a zero byte terminator ( \0 ).
1841It is specified by using the  –Z option with  grep .  xargs -0 reads the input and separates file
1842names with a zero byte terminator:
1843$ grep "test" file* -lZ | xargs -0 rm
1844Usually,  -Z is used along with  -l .
1845www.it-ebooks.info
1846Chapter 4
1847141
1848Silent output for grep
1849The previously mentioned usages of  grep return output in different formats. There are
1850some cases when we need to know whether a file contains the specified text or not. We have
1851to perform a test condition that returns true or false. It can be performed using the quiet
1852condition ( -q ). In quiet mode, the  grep command does not write any output to the standard
1853output. Instead it runs the command and returns exit status based on success or failure.
1854We know that a command returns 0 if success and non-zero if failure.
1855Let's go through a script that makes uses of  grep in quiet mode for testing whether a match
1856text appears in a file or not.
1857#!/bin/bash
1858#Filename: silent_grep.sh
1859#Description: Testing whether a file contain a text or not
1860if [ $# -ne 2 ];
1861then
1862echo "$0 match_text filename"
1863fi
1864match_text=$1
1865filename=$2
1866grep -q $match_text $filename
1867if [ $? -eq 0 ];
1868then
1869echo "The text exists in the file"
1870else
1871echo "Text does not exist in the file"
1872fi
1873The  silent_grep.sh script can be run as follows by providing a match word ( Student ) and
1874a filename ( student_data.txt ) as the command argument:
1875$ ./silent_grep.sh Student student_data.txt
1876The text exists in the file
1877Print lines before and after text matches
1878Context-based printing is a one of the nice features of  grep . Suppose a matching line for a
1879given match text is found,  grep usually prints only the matching lines. But we may need "n"
1880lines after the matching lines or "n" lines before the matching line or both. It can be performed
1881using context line control in  grep . Let's see how to do it.
1882www.it-ebooks.info
1883Texting and Driving
1884142
1885In order to print three lines after a match, use the  -A option:
1886$ seq 10 | grep 5 -A 3
18875
18886
18897
18908
1891In order to print three lines before the match, use the  -B option:
1892$ seq 10 | grep 5 -B 3
18932
18943
18954
18965
1897Print three lines after and before the match, use the  -C option as follows:
1898$ seq 10 | grep 5 -C 3
18992
19003
19014
19025
19036
19047
19058
1906If there are multiple matches, each section is delimited by a line "--":
1907$ echo -e "a\nb\nc\na\nb\nc" | grep a -A 1
1908a
1909b
1910--
1911a
1912b
1913Column-wise cutting of a file with cut
1914We may need to cut text by column rather than row. Let's assume that we have a text file
1915containing student reports with columns, such as  No ,  Name ,  Mark , and  Percentage . We
1916need to extract only the name of students to another file or any n-th column in the file or
1917extract two or more columns. This recipe will illustrate how to perform this task.
1918www.it-ebooks.info
1919Chapter 4
1920143
1921Getting ready
1922cut is a small utility that often comes to our help for cutting in column fashion. It can also
1923specify the delimiter that separates each column. In  cut terminology, each column is known
1924as a field.
1925How to do it...
1926In order to extract the first field or column, use the following syntax:
1927cut -f FIELD_LIST filename
1928FIELD_LIST is a list of columns that are to be displayed. The list consists of column numbers
1929delimited by commas. For example:
1930$ cut -f 2,3 filename
1931Here, the second and the third columns are displayed.
1932cut can also read input text from  stdin .
1933Tab is the default delimiter for fields or columns. If lines without delimiters are found, they
1934are also printed. To avoid printing lines that do not have delimiter characters, attach the  -s
1935option along with  cut . An example of using the  cut command for columns is as follows:
1936$ cat student_data.txt
1937No Name Mark Percent
19381 Sarath 45 90
19392 Alex 49 98
19403 Anu 45 90
1941$ cut -f1 student_data.txt
1942No
19431
19442
19453
1946Extract multiple fields as follows:
1947$ cut -f2,4 student_data.txt
1948Name Percent
1949Sarath 90
1950Alex 98
1951Anu 90
1952www.it-ebooks.info
1953Texting and Driving
1954144
1955To print multiple columns, provide a list of column numbers separated by commas as
1956argument to  -f .
1957We can also complement the extracted fields using the  --complement option. Suppose you
1958have many fields and you want to print all the columns except the third column, use:
1959$ cut -f3 –-complement student_data.txt
1960No Name Percent
19611 Sarath 90
19622 Alex 98
19633 Anu 90
1964To specify the delimiter character for the fields, use the  -d option as follows:
1965$ cat delimited_data.txt
1966No;Name;Mark;Percent
19671;Sarath;45;90
19682;Alex;49;98
19693;Anu;45;90
1970$ cut -f2 -d";" delimited_data.txt
1971Name
1972Sarath
1973Alex
1974Anu
1975There's more...
1976The  cut command has more options to specify the character sequences to be displayed as
1977columns. Let's go through the additional options available with  cut .
1978Specifying range of characters or bytes as fields
1979Suppose that we don't rely on delimiters, but we need to extract fields such that we need to
1980define a range of characters (counting from 0 as start of line) as a field, such extractions are
1981possible with  cut .
1982Let's see what notations are possible:
1983N- from N-th byte, character or field, to end of line
1984N-M from N-th to M-th (included) byte, character or field
1985-M from first to M-th (included) byte, character or field
1986www.it-ebooks.info
1987Chapter 4
1988145
1989We use the above notations to specify fields as range of bytes or characters with the following
1990options:
1991f -b for bytes
1992f -c for characters
1993f -f for defining fields
1994For example:
1995$ cat range_fields.txt
1996abcdefghijklmnopqrstuvwxyz
1997abcdefghijklmnopqrstuvwxyz
1998abcdefghijklmnopqrstuvwxyz
1999abcdefghijklmnopqrstuvwxy
2000You can print the first to fifth characters as follows:
2001$ cut -c1-5 range_fields.txt
2002abcde
2003abcde
2004abcde
2005abcde
2006The first two characters can be printed as follows:
2007$ cut range_fields.txt -c-2
2008ab
2009ab
2010ab
2011ab
2012Replace  -c with  -b to count in bytes.
2013We can specify output delimiter while using with  -c ,  -f and  -b as:
2014--output-delimiter "delimiter string"
2015When multiple fields are extracted with  -b or  -c ,  --output-delimiter is a must. Else, you
2016cannot distinguish between fields if it is not provided. For example:
2017$ cut range_fields.txt -c1-3,6-9 --output-delimiter ","
2018abc,fghi
2019abc,fghi
2020abc,fghi
2021abc,fghi
2022www.it-ebooks.info
2023Texting and Driving
2024146
2025Frequency of words used in a given file
2026Finding the frequency of words used in a file is an interesting exercise to apply the text
2027processing skills. It can be done in many different ways. Let's see how to do it.
2028Getting ready
2029We can use associative arrays, awk, sed, grep, and so on to solve this problem in different ways.
2030How to do it...
2031Words are alphabetic characters delimited by space and dot. First we should parse all the
2032words in the given file. Hence the count of each word needs to be found out. Words can be
2033parsed by using regex with any of the tools such as sed, awk, or grep.
2034To find out the count of each word, we can have a different approach. One way of doing it is
2035to loop through each word, and then use another loop to go through the words and check
2036if they are equal. If they are equal, increment a count and print it at the end of file. This is
2037an inefficient method. In an associative array, we use the word as the array index and count
2038as the array value. We will only need one loop to achieve this by looping through each word.
2039array[word] = array[word] + 1 while initially its value is set  0 . Hence we can get an
2040array containing the counts for each word.
2041Now let's do it. Create the shell script as follows:
2042#!/bin/bash
2043#Name: word_freq.sh
2044#Description: Find out frequency of words in a file
2045if [ $# -ne 1 ];
2046then
2047echo "Usage: $0 filename";
2048exit -1
2049fi
2050filename=$1
2051egrep -o "\b[[:alpha:]]+\b" $filename | \
2052awk '{ count[$0]++ }
2053END{ printf("%-14s%s\n","Word","Count") ;
2054for(ind in count)
2055{ printf("%-14s%d\n",ind,count[ind]); }
2056}'
2057www.it-ebooks.info
2058Chapter 4
2059147
2060A sample output is as follows:
2061$ ./word_freq.sh words.txt
2062Word Count
2063used 1
2064this 2
2065counting 1
2066How it works...
2067Here  egrep -o "\b[[:alpha:]]+\b" $filename is used to output only words. The  -o
2068option will print the matching character sequence delimited by a newline character. Hence we
2069receive words in each line.
2070\b is the word boundary character.  [:alpha:] is a character class for alphabets.
2071The  awk  command is used to avoid the iteration through each word. Since  awk , by default,
2072executes the statements in the  { } block for each row, we don't need a specific loop for doing
2073that. Hence the count is incremented as  count[$0]++ using the associative array. Finally, in
2074the  END{} block, we print the words and their count by iterating through the words.
2075See also
2076f Arrays and associative arrays of Chapter 1, explains the arrays in Bash
2077f Basic awk primer, explains the awk command
2078Basic sed primer
2079sed stands for stream editor. It is a very essential tool for text processing. It is a marvelous
2080utility that can play around regular expressions. A well-known usage of the  sed command is
2081for text replacement. This recipe will cover most of the frequently used  sed techniques.
2082How to do it…
2083sed can be used to replace occurrences of a string with another string in a given text. It can
2084be matched using regular expressions.
2085$ sed 's/pattern/replace_string/' file
2086or
2087$ cat file | sed 's/pattern/replace_string/' file
2088This command reads from  stdin .
2089www.it-ebooks.info
2090Texting and Driving
2091148
2092To save the changes along with the substitutions to the same file, use the -i option. Most of
2093the users follow multiple redirections to save the file after making a replacement as follows:
2094$ sed 's/text/replace/' file > newfile
2095$ mv newfile file
2096However, it can be done in just one line, for example:
2097$ sed -i 's/text/replace/' file
2098The previously seen  sed  commands will replace the first occurrence of the pattern in each line.
2099But in order to replace every occurrence, we need to add the  g parameter at the end as follows:
2100$ sed 's/pattern/replace_string/g' file
2101The  /g suffix means that it will substitute every occurrence. However, sometimes we need not
2102replace the first "N" occurrences, but only the rest of them. There is a built-in option to ignore
2103the first "N" occurrences and replace from the"N+1th"occurrence onwards.
2104Have a look at the following commands:
2105$ echo this thisthisthis | sed 's/this/THIS/2g'
2106thisTHISTHISTHIS
2107$ echo this thisthisthis | sed 's/this/THIS/3g'
2108thisthisTHISTHIS
2109$ echo this thisthisthis | sed 's/this/THIS/4g'
2110thisthisthisTHIS
2111Place  /Ng when it needs to start the replacement from the N-th occurrence.
2112/ in  sed is a delimiter character. We can use any delimiter characters as follows:
2113sed 's:text:replace:g'
2114sed 's|text|replace|g'
2115When the delimiter character appears inside the pattern, we have to escape it using  \ prefix as:
2116sed 's|te\|xt|replace|g'
2117\| is a delimiter appearing in the pattern replaced with escape.
2118There's more...
2119The  sed command comes with numerous options for text manipulation. By combining the
2120options available with  sed in logical sequences, many complex problems can be solved in one
2121line. Let's see some different options available with  sed .
2122www.it-ebooks.info
2123Chapter 4
2124149
2125Removing blank lines
2126Removing blank lines is a simple technique using  sed to remove blank lines. Blanks can be
2127matched with regular expression  ^$ :
2128$ sed '/^$/d' file
2129/pattern/d will remove lines matching the pattern.
2130For blank lines, the line end marker appears next to the line start marker.
2131Matched string notation (&)
2132In  sed we can use  & as the matched string for the substitution pattern such that we can use
2133the matched string in replacement string.
2134For example:
2135$ echo this is an example | sed 's/\w\+/[&]/g'
2136[this] [is] [an] [example]
2137Here the regex  \w\+ matches every word. Then we replace it with  [&] .  &  corresponds to the
2138word that is matched.
2139Substring match notation (\1)
2140& is a string which corresponds to match string for the given pattern. But we can also match
2141the substrings of the given pattern. Let's see how to do it.
2142$ echo this is digit 7 in a number | sed 's/digit \([0-9]\)/\1/'
2143this is 7 in a number
2144It replaces  digit 7 with  7 . The substring matched is  7 .  \(pattern\) is used to match the
2145substring. The pattern is enclosed in  () and is escaped with slashes. For the first substring
2146match, the corresponding notation is  \1 , for the second it is  \2 , and so on. Go through the
2147following example with multiple matches:
2148$ echo seven EIGHT | sed 's/\([a-z]\+\) \([A-Z]\+\)/\2 \1/'
2149EIGHT seven
2150([a-z]\+\) matches the first word and  \([A-Z]\+\) matches the second word.  \1 and
2151\2 are used for referencing them. This type of referencing is called back referencing. In the
2152replacement part, their order is changed as  \2 \1 and hence it appears in reverse order.
2153Combination of multiple expressions
2154The combination of multiple  sed using a pipe can be replaced as follows:
2155sed 'expression' | sed 'expression'
2156www.it-ebooks.info
2157Texting and Driving
2158150
2159Which is equivalent to:
2160$ sed 'expression; expression'
2161Quoting
2162Usually, it is seen that the  sed expression is quoted using single quotes. But double-quotes
2163can also be used. Double-quotes expand the expression by evaluating it. Using double-quotes
2164is useful when we want to use some variable string in a  sed expression.
2165For example:
2166$ text=hello
2167$ echo hello world | sed "s/$text/HELLO/"
2168HELLO world
2169$text is evaluated as "hello".
2170Basic awk primer
2171awk is a tool designed to work with data streams. It is very interesting as it can operate on
2172columns and rows. It supports many inbuilt functionalities such as arrays, functions, and so
2173on, as in the C programming language. Flexibility is the greatest advantage of it.
2174How to do it…
2175The structure of an  awk script looks like this:
2176awk ' BEGIN{ print "start" } pattern { commands } END{ print "end" }
2177file
2178The  awk command can read from  stdin also.
2179An  awk script usually consists of three parts:  BEGIN ,  END , and a common statements block
2180with the pattern match option. The three of them are optional and any of them can be absent
2181in the script. The script is usually enclosed in single-quotes or double-quotes as follows:
2182awk 'BEGIN { statements } { statements } END { end statements }'
2183Or, alternately, use:
2184awk "BEGIN { statements } { statements } END { end statements }"
2185For example:
2186$ awk 'BEGIN { i=0 } { i++ } END{ print i}' filename
2187Or:
2188$ awk "BEGIN { i=0 } { i++ } END{ print i }" filename
2189www.it-ebooks.info
2190Chapter 4
2191151
2192How it works…
2193The  awk command works in the following manner:
21941. Execute the statements in the  BEGIN { commands } block.
21952. Read one line from the file or  stdin , and execute  pattern { commands } .
2196Repeat this step until the end of the file is reached.
21973. When the end of the input stream is reached, execute the  END { commands } block.
2198The  BEGIN block is executed before  awk starts reading lines from the input stream. It is an
2199optional block. The statements such as variable initialization, printing the output header for
2200an output table, and so on are common statements that are written in the  BEGIN block.
2201The  END block is similar to the  BEGIN block. The  END block gets executed when  awk has
2202completed reading all the lines from the input stream. The statements like printing results
2203after analyzing all the values calculated for all the lines or printing the conclusion are the
2204commonly-used statements in the  END block (for example, after comparing all the lines, print
2205the maximum number from a file). This is an optional block.
2206The most important block is the common commands with the pattern block. This block is also
2207optional. If this block is not provided, by default  { print } gets executed so as to print each
2208of the lines read. This block gets executed for each line read by  awk .
2209It is like a while loop for line read with provided statements inside the body of the loop.
2210When a line is read, it checks whether the provided pattern matches the line. The pattern can
2211be a regular expression match, conditions, range of lines match, and so on. If the current read
2212line matches with the pattern, it executes the statements enclosed in  { } .
2213The pattern is optional. If pattern is not used, all the lines are matched and statements inside
2214{ } are executed.
2215Let's go through the following example:
2216$ echo -e "line1\nline2" | awk 'BEGIN{ print "Start" } { print } END{
2217print "End" } '
2218Start
2219line1
2220line2
2221End
2222When  print is used without an argument, it will print the current line. There are two
2223important things to be kept in mind about  print . When the arguments of the print are
2224separated by commas, they are printed with a space delimiter. Double-quotes are used as the
2225concatenation operator in the context of  print in  awk .
2226www.it-ebooks.info
2227Texting and Driving
2228152
2229For example:
2230$ echo | awk '{ var1="v1"; var2="v2"; var3="v3"; \
2231print var1,var2,var3 ; }'
2232The above statement will print the values of the variables as follows:
2233v1 v2 v3
2234The  echo command writes a single line into the standard output. Hence the statements in
2235the  { } block of  awk are executed once. If standard input to  awk contains multiple lines, the
2236commands in  awk will be executed multiple times.
2237Concatenation can be used as follows:
2238$ echo | awk '{ var1="v1"; var2="v2"; var3="v3"; \
2239print var1"-"var2"-"var3 ; }'
2240The output will be:
2241v1-v2-v3
2242{ } is like a block in a loop iterating through each line of a file.
2243Usually, we place initial variable assignments, such as var=0; and
2244statements to print the file header in the BEGIN block. In the END{} block,
2245we place statements such as printing results and so on.
2246There's more…
2247The  awk command comes with lot of rich features. In order to master the art of awk
2248programming you should be familiar with the important  awk options and functionalities. Let's
2249go through the essential functionalities of  awk .
2250Special variables
2251Some special variables that can be used with  awk are as follows:
2252f NR : It stands for number of records and corresponds to current line number under
2253execution.
2254f NF : It stands for number of fields and corresponds to number of fields in the current
2255line under execution (Fields are delimited by space).
2256f $0 : It is a variable that contain the text content of current line under execution.
2257f $1 : It is a variable that holds the text of the first field.
2258f $2 : It is the variable that holds the test of the second field text.
2259www.it-ebooks.info
2260Chapter 4
2261153
2262For example:
2263$ echo -e "line1 f2 f3\nline2 f4 f5\nline3 f6 f7" | \
2264awk '{
2265print "Line no:"NR",No of fields:"NF, "$0="$0, "$1="$1,"$2="$2,"$3="$3
2266}'
2267Line no:1,No of fields:3 $0=line1 f2 f3 $1=line1 $2=f2 $3=f3
2268Line no:2,No of fields:3 $0=line2 f4 f5 $1=line2 $2=f4 $3=f5
2269Line no:3,No of fields:3 $0=line3 f6 f7 $1=line3 $2=f6 $3=f7
2270We can print last field of a line as  print $NF , last but second as  $(NF-1) and so on.
2271awk provides the  printf() function with same syntax as in C. We can also use that instead
2272of print.
2273Let's see some basic  awk  usage examples.
2274Print the second and third field of every line as follows:
2275$awk '{ print $3,$2 }' file
2276In order to count the number of lines in a file, use the following command:
2277$ awk 'END{ print NR }' file
2278Here we only use the  END block.  NR will be updated on entering each line by  awk with its line
2279number. When it reaches the end line it will have the value of last line number. Hence, in the
2280END block  NR will have the value of last line number.
2281You can sum up all the numbers from each line of field 1 as follows:
2282$ seq 5 | awk 'BEGIN{ sum=0; print "Summation:" }
2283{ print $1"+"; sum+=$1 } END { print "=="; print sum }'
2284Summation:
22851+
22862+
22873+
22884+
22895+
2290==
229115
2292www.it-ebooks.info
2293Texting and Driving
2294154
2295Passing a variable value from outside to awk
2296By using the  -v  argument, we can pass external values (other than from  stdin ) to  awk
2297as follows:
2298$ VAR=10000
2299$ echo | awk -v VARIABLE=$VAR'{ print VARIABLE }'
23001
2301There is a flexible alternate method to pass many variable values from outside  awk .
2302For example:
2303$ var1="Variable1" ; var2="Variable2"
2304$ echo | awk '{ print v1,v2 }' v1=$var1 v2=$var2
2305Variable1 Variable2
2306When input is given through a file rather than standard input, use:
2307$ awk '{ print v1,v2 }' v1=$var1 v2=$var2 filename
2308In the above method, variables are specified as key-value pairs separated by space
2309( v1=$var1 v2=$var2 ) as command arguments to awk soon after the BEGIN, { } and END
2310blocks.
2311Reading a line explicitly using getline
2312Usually,  grep  reads all lines in a file by default. If you want to read one specific line, you can
2313use the  getline function. Sometimes we may need to read the first line from the  BEGIN block.
2314The syntax is:  getline var
2315The variable  var will contain the content for the line.
2316If the  getline is called without an argument, we can access the content of the line by using
2317$0 ,  $1 , and  $2 .
2318For example:
2319$ seq 5 | awk 'BEGIN { getline; print "Read ahead first line", $0 } {
2320print $0 }'
2321Read ahead first line 1
23222
23233
23244
23255
2326www.it-ebooks.info
2327Chapter 4
2328155
2329Filtering lines processed by awk with filter patterns
2330We can specify some conditions for lines to be processed. For example:
2331$ awk 'NR < 5' # Line number less than 5
2332$ awk 'NR==1,NR==4' #Line numbers from 1-5
2333$ awk '/linux/' # Lines containing the pattern linux (we can specify
2334regex)
2335$ awk '!/linux/' # Lines not containing the pattern linux
2336Setting delimiter for fields
2337By default, the delimiter for fields is space. We can explicitly specify a delimiter using
2338-F "delimiter" :
2339$ awk -F: '{ print $NF }' /etc/passwd
2340Or:
2341awk 'BEGIN { FS=":" } { print $NF }' /etc/passwd
2342We can set the output fields separator by setting  OFS="delimiter" in the  BEGIN block.
2343Reading command output from awk
2344In the following code,  echo will produces a single blank line. The  cmdout variable will contain
2345output of command  grep root /etc/passwd and it will print the line containing  root :
2346The syntax for reading out of the 'command' in a variable 'output' is as follows:
2347"command" | getline output ;
2348For example:
2349$ echo | awk '{ "grep root /etc/passwd" | getline cmdout ; print cmdout
2350}'
2351root:x:0:0:root:/root:/bin/bash
2352By using  getline  we can read the output of external shell commands in a variable
2353called  cmdout .
2354awk supports associative arrays, which can use text as the index.
2355Using loop inside awk
2356A  for loop is available in  awk . It has the format:
2357for(i=0;i<10;i++) { print $i ; }
2358Or:
2359for(i in array) { print array[i]; }
2360www.it-ebooks.info
2361Texting and Driving
2362156
2363awk comes with many built-in string manipulation functions. Let's have a look at a few of them:
2364f length(string) : It returns the string length.
2365f index(string, search_string) : It returns the position at which the
2366search_string is found in the string.
2367f split(string, array, delimiter) : It stores the list of strings generated by
2368using the delimiter in the array.
2369f substr(string, start-position, end-position) : It returns the substring
2370created from the string by using start and end character offets.
2371f sub(regex, replacement_str, string) : It replaces the first occurring regular
2372expression match from the string with  replacment_str .
2373f gsub(regex, replacment_str, string : It is similar to  sub() . But it replaces
2374every regular expression match.
2375f match(regex, string) : It returns the result of whether a regular expression
2376(regex) match is found in the string or not. It returns non-zero if match is found, else
2377it returns zero. Two special variables are associated with  match() . They are  RSTART
2378and  RLENGTH . The  RSTART variable contains the position at which the regular
2379expression match starts. The  RLENGTH variable contains the length of the string
2380matched by the regular expression.
2381Replacing strings from a text or file
2382String replacement is a frequently-used text-processing task. It can be done easily with regular
2383expressions by matching the required text.
2384Getting ready
2385When we hear the term 'replace', every system admin will recall sed.  sed is the universal tool
2386under UNIX-like systems to make replacements in text or in a file. Let's see how to do it.
2387How to do it...
2388The  sed primer recipe contains most of the usages of  sed . You can replace a string or pattern
2389as follows:
2390$ sed 's/PATTERN/replace_text/g' filename
2391Or:
2392$ stdin | sed 's/PATTERN/replace_text/g'
2393We can also use double quote (") instead of single quote ('). When double quote (") is used, we
2394can specify variables inside the  sed pattern and replacement strings. For example:
2395www.it-ebooks.info
2396Chapter 4
2397157
2398$ p=pattern
2399$ r=replaced
2400$ echo "line containing apattern" | sed "s/$p/$r/g"
2401line containing a replaced
2402We can also use it without  g in  sed .
2403$ sed 's/PATTEN/replace_text/' filename
2404Then it will replace the occurrence of  PATTERN first time it appears only.  /g stands for global.
2405That means, it will replace every occurrence of  PATTERN in the file.
2406There's more...
2407We have seen basic text replacement with  sed . Let's see how to save the replaced text in the
2408source file itself.
2409Making replacement saved in the file
2410When a filename is passed to  sed , it's output will be available to  stdout . Instead of sending
2411the output stream into  stdout , to make changes saved in the file, use the  –i  option as
2412follows:
2413$ sed 's/PATTERN/replacement/' -i filename
2414For example, replace all three-digit numbers with another specified number in a file as follows:
2415$ cat sed_data.txt
241611 abc 111 this 9 file contains 111 11 88 numbers 0000
2417$ cat sed_data.txt | sed 's/\b[0-9]\{3\}\b/NUMBER/g'
241811 abc NUMBER this 9 file contains NUMBER 11 88 numbers 0000
2419The above one-liner replaces three-digit numbers only. \b[0-9]\{3\}\b is the regular
2420expression used to match three-digit numbers. [0-9] is the range of digits, that is, from 0 to 9.
2421{3} is used for matching the preceding character thrice.  \ in  \{3\} is used to give a special
2422meaning for  { and  } .  \b is the word boundary marker.
2423See also
2424f Basic sed primer, explains the sed command
2425www.it-ebooks.info
2426Texting and Driving
2427158
2428Compressing or decompressing JavaScript
2429JavaScript is widely used in designing websites. While writing JavaScript code, we use several
2430white spaces, comments, and tabs for readability and maintenance of code. But the use
2431of a lot of white spaces and tabs in JavaScript causes the file size to increase. As the file
2432size increases, it increases page load times. Hence most of the professional websites use
2433compressed JavaScripts for fast loading. Compression is mostly squeezing white spaces
2434and newline characters. Once JavaScript is compressed, it can be decompressed by adding
2435enough white space and newline characters, which makes it readable. Usually, obfuscated
2436code also can be made readable by inserting white space and newlines. This recipe is an
2437attempt to hack similar capabilities in the shell.
2438Getting ready
2439We are going to write a JavaScript compressor or obfuscation tool. Also a decompressing tool
2440can be designed. We are going to get our hands dirty using text and character replacement
2441tools  tr and  sed . Let's see how to do it.
2442How to do it...
2443Let's go through the logical sequences and the code required for compressing and
2444decompressing the JavaScript.
2445$ cat sample.js
2446functionsign_out()
2447{
2448$("#loading").show();
2449$.get("log_in",{logout:"True"},
2450function(){
2451window.location="";
2452});
2453}
2454The following are the tasks we need to perform for compressing the JavaScript:
24551. Remove newline and tab characters.
24562. Squeeze spaces.
24573. Replace comments /* content */.
2458www.it-ebooks.info
2459Chapter 4
2460159
24614. Replace the following with substitutions:
2462‰ "{ " with "{"
2463‰ " }" with "}"
2464‰ " (" with "("
2465‰ ") " with ")"
2466‰ ", " with ","
2467‰ " ; " with ";" (we need to remove all extra spaces)
2468To decompress or to make the JavaScript more readable, we can use the following tasks:
24691. Replace ";" with ";\n".
24702. Replace "{" with "{\n" and "}" with "\n}".
2471How it works...
2472Let's compress the JavaScript by performing these tasks:
24731. Remove the'\n' and '\t' characters:
2474tr -d '\n\t'
24752. Remove extra spaces:
2476tr -s ' ' or sed 's/[ ]\+/ /g'
24773. Remove comments:
2478sed 's:/\*.*\*/::g'
2479‰ : is used as a sed delimiter to avoid the need of escaping  / since we need to
2480use  /* and  */
2481‰ * in the sed is escaped as  \*
2482‰ .* is used to match all text in between  /* and  */
24834. Remove all spaces preceding and suffixing the  { , } , ( , ) , ; , : , and comma.
2484sed 's/ \?\([{}();,:]\) \?/\1/g'
2485The above  sed statement can be parsed as follows:
2486f / \?\([{}();,:]\) \?/ in the  sed code is the match part and  /\1 /g is the
2487replacement part.
2488www.it-ebooks.info
2489Texting and Driving
2490160
2491f \([{}();,:]\) is used to match any one character in the set  [ { }( ) ; , : ]
2492(inserted spaces for readability).  \( and  \) are group operators used to memorize the
2493match and back reference in the replacement part.  ( and  ) are escaped to give them a
2494special meaning as a group operator.  \? precedes and follows the group operators.
2495It is to match the space character that may precede or follow any of the characters in
2496the set.
2497f In the replacement part, the match string (that is, the combination of  : a space
2498(optional), a character from the set, and again optional space) is replaced with
2499the character matched. It uses a back reference to the character matched and
2500memorized using the group operator  () . Back-referenced characters refer to a group
2501match by using the  \1 symbol.
2502Combine the above tasks using a pipe as follows:
2503$ catsample.js | \
2504tr -d '\n\t' | tr -s ' ' \
2505| sed 's:/\*.*\*/::g' \
2506| sed 's/ \?\([{}();,:]\) \?/\1/g'
2507The output is as follows:
2508functionsign_out(){$("#loading").show();$.get("log_
2509in",{logout:"True"},function(){window.location="";});}
2510Let's write a decompression script for making obfuscated code readable as follows:
2511$ cat obfuscated.txt | sed 's/;/;\n/g; s/{/{\n\n/g; s/}/\n\n}/g'
2512Or:
2513$ cat obfuscated.txt | sed 's/;/;\n/g' | sed 's/{/{\n\n/g' | sed 's/}/\n\
2514n}/g'
2515In the previous command:
2516f s/;/;\n/g  replaces  ; with  \n;
2517f s/{/{\n\n/g replaces  { with  {\n\n
2518f s/}/\n\n}/g replaces  } with  \n\n}
2519See also
2520f Translating with tr of Chapter 2, explains the tr command
2521f Basic sed primer, explains the sed command
2522www.it-ebooks.info
2523Chapter 4
2524161
2525Iterating through lines, words, and
2526characters in a file
2527Iterating through character, word, and lines in a file is a frequently required script element
2528while writing different text processing and file operation scripts. Even though it is simple to
2529perform, we make simple mistakes and it gets erroneous without getting the expected output.
2530This recipe will help you out to learn how to do it.
2531Getting ready
2532Iteration with a simple loop and redirection from  stdin or file are basic components of
2533performing the mentioned tasks.
2534How to do it...
2535In this recipe we discuss about performing three tasks of iterating through line, word, and
2536characters. Let's see how each of these tasks can be performed.
25371. Iterate through each line in a file:
2538We can use a  while loop to read from standard input. Hence it will read a line in
2539each iteration.
2540Use file redirection to  stdin as follows:
2541while read line;
2542do
2543echo $line;
2544done < file.txt
2545Use subshell as follows:
2546cat file.txt | ( while read line; do echo $line; done )
2547Here  cat file.txt can be replaced with the output of any command sequence.
25482. Iterate through each word in a line
2549We can use a  while loop to iterate through words in a line as follows:
2550for word in $line;
2551do
2552echo $word;
2553done
2554www.it-ebooks.info
2555Texting and Driving
2556162
25573. Iterate through each character in a word
2558We can use a  for loop to iterate a variable  i from  0 to the length of string. A
2559character can be extracted from the string in each iteration using the special notation
2560${string:start_position:No_of_characters} .
2561for((i=0;i<${#word};i++))
2562do
2563echo ${word:i:1} ;
2564done
2565How it works...
2566Reading lines of a file and reading words in a line are direct ways. But reading a character of a
2567word is a little hack. We use the substring extraction technique.
2568${word:start_position:no_of_characters} returns a substring of a string held in
2569variable  word .
2570${#word} returns the length of the variable  word .
2571See also
2572f Field separators and iterators of Chapter 1, explains different loops in Bash.
2573f Text slicing and parameter operations, explains extracting characters from a string.
2574Merging multiple files as columns
2575There are different cases when we require to concatenate files in columns. We may need each
2576file's content to appear in separate columns. Usually, the  cat command concatenates in a
2577line- or row-wise fashion.
2578How to do it...
2579paste is the command that can be used for column-wise concatenation. The  paste
2580command can be used with the following syntax:
2581$ paste file1 file2 file3 …
2582Let's try an example as follows:
2583$ cat paste1.txt
25841
25852
2586www.it-ebooks.info
2587Chapter 4
2588163
25893
25904
25915
2592$ cat paste2.txt
2593slynux
2594gnu
2595bash
2596hack
2597$ paste paste1.txt paste2.txt
25981slynux
25992gnu
26003bash
26014hack
26025
2603The default delimiter is Tab. We can also explicitly specify the delimiter using  –d . For example:
2604$ paste paste1.txt paste2.txt -d ","
26051,slynux
26062,gnu
26073,bash
26084,hack
26095,
2610See also
2611f Column-wise cutting of a file with cut, explains extracting data from text files
2612Printing the nth word or column in a file
2613or line
2614We may get a file having a number of columns and only a few will actually be useful. In order
2615to print only relevant columns or fields, we filter it.
2616Getting ready
2617The most widely-used method is to use  awk  for doing this task. It can be also done using  cut .
2618www.it-ebooks.info
2619Texting and Driving
2620164
2621How to do it...
2622To print the fifth column use the following command:
2623$ awk '{ print $5 }' filename
2624We can also print multiple columns and we can insert our custom string in between columns.
2625For example, to print the permission and filename of each file in the current directory, use:
2626$ ls -l | awk '{ print $1" : " $8 }'
2627-rw-r--r-- : delimited_data.txt
2628-rw-r--r-- : obfuscated.txt
2629-rw-r--r-- : paste1.txt
2630-rw-r--r-- : paste2.txt
2631See also
2632f Basic awk primer, explains the awk command
2633f Column-wise cutting of a file with cut, explains extracting data from text files
2634Printing text between line numbers
2635or patterns
2636We may require to print certain section of text lines based on conditions such as a range of
2637line numbers, range matched by start and end pattern and so on. Let's see how to do it.
2638Getting ready
2639We can use utilities such as awk, grep, and sed to perform the printing of a section based on
2640conditions. Still I found  awk to be the simplest one to understand. Let's do it using  awk .
2641How to do it...
2642In order to print lines of text in a range of line numbers, M to N, use the following syntax:
2643$ awk 'NR==M, NR==N' filename
2644Or, it can take  stdin input as follows:
2645$ cat filename | awk 'NR==M, NR==N'
2646www.it-ebooks.info
2647Chapter 4
2648165
2649Replace  M and  N with numbers as follows:
2650$ seq 100 | awk 'NR==4,NR==6'
26514
26525
26536
2654To print lines of text in a section with  start_pattern and  end_pattern , use the following
2655syntax:
2656$ awk '/start_pattern/, /end _pattern/' filename
2657For example:
2658$ cat section.txt
2659line with pattern1
2660line with pattern2
2661line with pattern3
2662line end with pattern4
2663line with pattern5
2664$ awk '/pa.*3/, /end/' section.txt
2665line with pattern3
2666line end with pattern4
2667The patterns used in  awk are regular expressions.
2668See also
2669f Basic awk primer, explains the awk command
2670Checking palindrome strings with a script
2671Checking whether a string is palindrome is one of the first lab exercises in a C programming
2672course. However, here we have included this recipe to give you an idea of how to solve similar
2673problems in which pattern matching can be extended in a way that previously occurring
2674patterns repeat in the text.
2675Getting ready
2676The  sed command has the capability to remember a previously-matched sub pattern. It is
2677called back referencing. We can solve palindrome problems by using back referencing. We
2678can solve this using multiple ways in Bash.
2679www.it-ebooks.info
2680Texting and Driving
2681166
2682How to do it...
2683sed can remember previously matched regular expression patterns, thereby we can identify
2684whether duplicates of a character exists in a string. This capability to remember and reference
2685previously matched patterns is called back-reference.
2686Let's see how we can apply back-referencing in a simpler manner to solve the problem. For
2687example:
2688$ sed -n '/\(.\)\1/p' filename
2689\(.\) corresponds to memorize the one sub string inside ( ). Here it is . (period) which is also
2690sed's single character wildcard character.
2691\1 corresponds to the memory of the first match inside ().  \2  corresponds to the second
2692match. Hence we can memorize many blocks enclosed in  () .  () appears as  \( \) to give  (
2693and  ) special meaning rather than just a character.
2694The previous  sed  statement will print any pattern matching two exactly the same.
2695The structure of all palindrome words is as follows:
2696f Even number of characters and a sequence of characters concatenated with same
2697characters in reverse order
2698f Odd number of characters with a sequence of characters concatenated with reverse
2699of same characters, but a common character in between the first sequence and its
2700reverse
2701Therefore, for matching both, we can keep an optional character in between while writing the
2702regular expression.
2703A  sed  regex matching a three-letter palindrome word will look like the following:
2704'/\(.\).\1/p'
2705We can place an extra character ( . ) in between the character sequence and its reverse
2706sequence.
2707Let's write a script that can match a palindrome string of any length as follows:
2708#!/bin/bash
2709#Filename: match_palindrome.sh
2710#Description: Find out palindrome strings from a given file
2711if [ $# -ne 2 ];
2712then
2713echo "Usage: $0 filename string_length"
2714exit -1
2715fi
2716www.it-ebooks.info
2717Chapter 4
2718167
2719filename=$1 ;
2720basepattern='/^\(.\)'
2721count=$(( $2 / 2 ))
2722for((i=1;i<$count;i++))
2723do
2724basepattern=$basepattern'\(.\)' ;
2725done
2726if [ $(( $2 % 2 )) -ne 0 ];
2727then
2728basepattern=$basepattern'.' ;
2729fi
2730for((count;count>0;count--))
2731do
2732basepattern=$basepattern'\'"$count" ;
2733done
2734basepattern=$basepattern'$/p'
2735sed -n "$basepattern" $filename
2736Use the dictionary file as the input file to get a list of palindrome words of a given string length.
2737For example:
2738$ ./match_palindrome.sh /usr/share/dict/british-english 4
2739noon
2740peep
2741poop
2742sees
2743How it works...
2744The working of the above script is simple. Most of the work is done to generate the  sed  script
2745for a regular expression and a back-reference string generation.
2746Let's go through its working with the help of some worked out examples.
2747f If you want to match the character and back-reference it, we use  \(.\) to match one
2748character and  \1 to reference it. Hence, in order match a two letter palindrome and
2749print it, we use:
2750sed '/\(.\)\1/p'
2751Now, to specify that match string from the beginning of the line, we add line-begin
2752market ^ so that it will become  sed'/^\(.\)\1/p' .  /p is used to print the match.
2753www.it-ebooks.info
2754Texting and Driving
2755168
2756f If we want to match four character palindrome, we use:
2757sed '/^\(.\)\(.\)\2\1/p'
2758We have used two  \(.\) to match two characters and remember them. Anything
2759enclosed within  \( and  \) will be remembered by  sed and can be back-referenced.
2760\2\1 is used to back-reference in the reverse order of the matched characters.
2761In the above script, we have a variable called  basepattern , which contains the  sed script.
2762The pattern is generated using a  for loop based on the number of characters in the
2763palindrome string.
2764Initially, basepattern is initialized as  basepattern='/^\(.\)' , which corresponds to a one-
2765character match. A  for loop is used to concatenate  \(.\) with  basepattern for half the
2766number of times of the length of palindrome string. Again a  for loop is used to concatenate
2767back-references in the reverse order (like  '\4\3\2\1' ) half the number of times the length
2768of palindrome string. Finally, in order to support palindrome strings with odd length an
2769optional character ( . ) is enclosed between match regex and back-references.
2770Thus the  sed palindrome match pattern is crafted. This crafted string is used to find out the
2771palindrome strings from the dictionary file.
2772In the above script, we have used  sed pattern generation using  for loops. Actually there is no
2773need to generate pattern separately. The  sed command has its own loop implementation using
2774labels and goto.  sed is a vast language. Palindrome check can be done in a single line using a
2775complex  sed script. It is hard to explain it from scratch. Just try out the following script:
2776$ word="malayalam"
2777$ echo $word | sed ':loop ; s/^\(.\)\(.*\)\1/\2/; t loop; /^.\?$/{ s/.*/
2778PALINDROME/ ; q; }; s/.*/NOT PALINDROME/ '
2779PALINDROME
2780If you are interested in deep scripting with  sed , refer to the complete  sed and  awk reference
2781book: sed & awk, Second Edition by Dale Dougherty and Arnold Robbins.
2782Try to parse the above one-line  sed script to test the palindrome using the book.
2783There's more...
2784Now let's see some other options, or possibly some pieces of general information that are
2785relevant to this task.
2786Simplest and direct method
2787The simplest method to check whether a string is a palindrome is by using the rev command.
2788The  rev command takes a file or  stdin as input and prints the reversed string of every line.
2789www.it-ebooks.info
2790Chapter 4
2791169
2792Let's do it:
2793string="malayalam"
2794if [[ "$string" == "$(echo $string | rev )" ]];
2795then
2796echo "Palindrome"
2797else
2798echo "Not palindrome"
2799fi
2800The  rev command can be used along with other commands to solve different problems. Let's
2801look at an interesting example to reverse the words in a sentence:
2802sentence='this is line from sentence'
2803echo $sentence | rev | tr ' ' '\n' | tac | tr '\n' ' ' | rev
2804The output is as follows:
2805sentence from line is this
2806In the above one-liner, the characters are reversed first using the  rev command. Then the
2807words are separated into a word per line by replacing space with the  \n character by using
2808the  tr command. Now the lines are reversed in order using the  tac command. Again, lines
2809are merged into a line using  tr . Now  rev is again applied so that a line with words is in the
2810reverse order.
2811See also
2812f Basic sed primer, explains the sed command
2813f Comparisons and tests of Chapter 1, explains the string comparison operators
2814Printing lines in the reverse order
2815This is a simple recipe. It may not seem very useful but it can be used to emulate the stack
2816data structure in Bash. This is something interesting. Let's print the lines of text in a file in
2817reverse order.
2818Getting ready
2819A little hack with  awk  can do the task. However, there is a direct command  tac to do the
2820same as well.  tac is the reverse of  cat .
2821www.it-ebooks.info
2822Texting and Driving
2823170
2824How to do it...
2825Let's do it with  tac first. The syntax is as follows:
2826tac file1 file2 …
2827It can also read from  stdin as follows:
2828$ seq 5 | tac
28295
28304
28313
28322
28331
2834In  tac ,  \n is the line separator. But we can also specify our own separator by using the  -s
2835"separator" option.
2836Let's do it in  awk as follows:
2837$ seq 9 | \
2838awk '{ lifo[NR]=$0; lno=NR }
2839END{ for(;lno>-1;lno--){ print lifo[lno]; }
2840}'
2841\ in the shell script is used to conveniently break a single line command sequence into
2842multiple lines.
2843How it works...
2844The  awk script is very simple. We store each of the lines into an associative array with the
2845line number as array index (NR gives line number). In the end,  awk executes the  END block.
2846In order to get last line number  lno=NR is used in the { } block. Hence it iterates from the last
2847line number to  0 and prints the lines stored in the array in reverse order.
2848See also
2849f Implementing head, tail, and tac with awk, explains writing tac using awk
2850www.it-ebooks.info
2851Chapter 4
2852171
2853Parsing e-mail addresses and URLs from text
2854Parsing required text from a given file is a common task that we encounter in text processing.
2855Items such as e-mail, URL, and so on can be found out with the help of correct regex
2856sequences. Mostly, we need to parse e-mail addresses from a contact list of a e-mail client
2857which is composed of many unwanted characters and words or from a HTML web page.
2858Getting ready
2859This problem can be solved with utilities egrep.
2860How to do it...
2861The regular expression pattern to match an e-mail address is:
2862egrep regex: [A-Za-z0-9.]+@[A-Za-z0-9.]+\.[a-zA-Z]{2,4}
2863For example:
2864$ cat url_email.txt
2865this is a line of text contains,<email> #slynux@slynux.com. </email>
2866and email address, blog "http://www.google.com", test@yahoo.com
2867dfdfdfdddfdf;cool.hacks@gmail.com<br />
2868<ahref="http://code.google.com"><h1>Heading</h1>
2869$ egrep -o '[A-Za-z0-9.]+@[A-Za-z0-9.]+\.[a-zA-Z]{2,4}' url_email.txt
2870slynux@slynux.com
2871test@yahoo.com
2872cool.hacks@gmail.com
2873The  egrep regex pattern for an HTTP URL is:
2874http://[a-zA-Z0-9\-\.]+\.[a-zA-Z]{2,4}
2875For example:
2876$ egrep -o "http://[a-zA-Z0-9.]+\.[a-zA-Z]{2,3}" url_email.txt
2877http://www.google.com
2878http://code.google.com
2879www.it-ebooks.info
2880Texting and Driving
2881172
2882How it works...
2883The regular expressions are really easy to design part by part. In the e-mail regex, we all know
2884that an e-mail address takes the form  name@domain.some_2-4_letter . Here the same is
2885written in regex language as follows:
2886[A-Za-z0-9.]+@[A-Za-z0-9.]+\.[a-zA-Z]{2,4}
2887[A-Za-z0-9.]+ means that some combination of characters in the  [] block should appear
2888one or more times (that is the meaning of  + ) before a literal  @ character appears. Then
2889[A-Za-z0-9.] also should appear one or more times ( + ). The pattern  \. means that a literal
2890period should appear and finally the last part should be of length 2 to 4 alphabetic characters.
2891The case of an HTTP URL is similar to that of an e-mail address but without the  name@ match
2892part of e-mail regex.
2893http://[a-zA-Z0-9.]+\.[a-zA-Z]{2,3}
2894See also
2895f Basic sed primer, explains the sed command
2896f Basic regular expression primer, explains how to use regular expressions
2897Printing n lines before or after a pattern
2898in a file
2899Printing a section of text by pattern matching is frequently used in text processing. Sometimes
2900we may need the lines of text before a pattern or after a pattern appears in a text. For
2901example, consider that there is a file containing the rating of film actors where each line
2902corresponds to a film actor's details, and we need to find out the rating of an actor along with
2903the details of actors who are nearest to them in rating. Let's see how to do it.
2904Getting ready
2905grep is the best tool for searching and finding text in a file. Usually,  grep prints a matching
2906line or matching text for a given pattern. But the context line control options in  grep enables it
2907to print before, after, and before-after lines around the line of pattern match.
2908How to do it...
2909This technique can be better explained with a film actor list. For example:
2910www.it-ebooks.info
2911Chapter 4
2912173
2913$ cat actress_rankings.txt | head -n 20
29141 Keira Knightley
29152 Natalie Portman
29163 Monica Bellucci
29174 Bonnie Hunt
29185 Cameron Diaz
29196 Annie Potts
29207 Liv Tyler
29218 Julie Andrews
29229 Lindsay Lohan
292310 Catherine Zeta-Jones
292411 CateBlanchett
292512 Sarah Michelle Gellar
292613 Carrie Fisher
292714 Shannon Elizabeth
292815 Julia Roberts
292916 Sally Field
293017 TéaLeoni
293118 Kirsten Dunst
293219 Rene Russo
293320 JadaPinkett
2934In order to print three lines after the match "Cameron Diaz" along with the matching line, use
2935the following command:
2936$ grep -A 3 "Cameron Diaz" actress_rankings.txt
29375 Cameron Diaz
29386 Annie Potts
29397 Liv Tyler
29408 Julie Andrews
2941In order to print the matched line and the preceding three lines, use the following command:
2942$ grep -B 3 "Cameron Diaz" actress_rankings.txt
29432 Natalie Portman
29443 Monica Bellucci
29454 Bonnie Hunt
29465 Cameron Diaz
2947Print the matched line and the two lines before and after the matched line as follows:
2948$ grep -C 2 "Cameron Diaz" actress_rankings.txt
29493 Monica Bellucci
29504 Bonnie Hunt
2951www.it-ebooks.info
2952Texting and Driving
2953174
29545 Cameron Diaz
29556 Annie Potts
29567 Liv Tyler
2957Are you wondering where I got this ranking from?
2958I parsed a website having full of images and HTML content just using basic sed, awk, and grep
2959commands. See the chapter: Tangled Web? Not at all.
2960See also
2961f Searching and mining "text" inside a file with grep, explains the grep command.
2962Removing a sentence in a file containing
2963a word
2964Removing a sentence containing a word is a simple task when a correct regular expression is
2965identified. This is just an exercise on solving similar problems.
2966Getting ready
2967sed  is the best utility for making substitutions. Hence let's use  sed to replace the matched
2968sentence with a blank.
2969How to do it...
2970Let's create a file with some text to carry out the substitutions. For example:
2971$ cat sentence.txt
2972Linux refers to the family of Unix-like computer operating systems
2973that use the Linux kernel. Linux can be installed on a wide variety of
2974computer hardware, ranging from mobile phones, tablet computers and video
2975game consoles, to mainframes and supercomputers. Linux is predominantly
2976known for its use in servers. It has a server market share ranging
2977between 20–40%. Most desktop computers run either Microsoft Windows or
2978Mac OS X, with Linux having anywhere from a low of an estimated 1–2% of
2979the desktop market to a high of an estimated 4.8%. However, desktop use
2980of Linux has become increasingly popular in recent years, partly owing
2981to the popular Ubuntu, Fedora, Mint, and openSUSE distributions and the
2982emergence of netbooks and smart phones running an embedded Linux.
2983We will remove the sentence containing the words "mobile phones". Use the following  sed
2984expression for this task:
2985www.it-ebooks.info
2986Chapter 4
2987175
2988$ sed 's/ [^.]*mobile phones[^.]*\.//g' sentence.txt
2989Linux refers to the family of Unix-like computer operating systems
2990that use the Linux kernel. Linux is predominantly known for its use
2991in servers. It has a server market share ranging between 20–40%. Most
2992desktop computers run either Microsoft Windows or Mac OS X, with Linux
2993having anywhere from a low of an estimated 1–2% of the desktop market to
2994a high of an estimated 4.8%. However, desktop use of Linux has become
2995increasingly popular in recent years, partly owing to the popular Ubuntu,
2996Fedora, Mint, and openSUSE distributions and the emergence of netbooks
2997and smart phones running an embedded Linux.
2998How it works...
2999Let's evaluate the  sed  regex 's/ [^.]*mobile phones[^.]*\.//g' .
3000It has the format  's/substitution_pattern/replacement_string/g .