· 9 years ago · Jun 01, 2017, 04:18 AM
1It has the format 's/substitution_pattern/replacement_string/g .
2It replaces every occurrence of substitution_pattern with the replacement string.
3Here the substitution pattern is the regex for a sentence. Every sentence is delimited by "."
4and the first character is a space. Therefore, we need to match the text that is in the format
5"space" some text MATCH_STRING some text "dot". A sentence may contain any characters
6except a "dot", which is the delimiter. Hence we have used [^.]. [^.]* matches a combination of
7any characters except dot. In between the text match string "mobile phones" is placed. Every
8match sentence is replaced by // (nothing).
9See also
10f Basic sed primer, explains the sed command
11f Basic regular expression primer, explains how to use regular expressions
12Implementing head, tail, and tac with awk
13Mastering text-processing operations comes with practice. This recipe will help us practice
14incorporating some of the commands that we have just learned with some that we already
15know.
16Getting ready
17The commands head , tail , uniq , and tac operate line by line. Whenever we need line by
18line processing, we can always use awk . Let's emulate these commands with awk .
19www.it-ebooks.info
20Texting and Driving
21176
22How to do it...
23Let's see how different commands can be emulated with different basic text processing
24commands, such as head, tail, and tac.
25The head command reads the first ten lines of a file and prints them out:
26$ awk 'NR <=10' filename
27The tail command prints the last ten lines of a file:
28$ awk '{ buffer[NR % 10] = $0; } END { for(i=1;i<11;i++) { print
29buffer[i%10] } }' filename
30The tac command prints the lines of input file in reverse order:
31$ awk '{ buffer[NR] = $0; } END { for(i=NR; i>0; i--) { print buffer[i] }
32}' filename
33How it works...
34In the implementation of head using awk , we print the lines in the input stream having a line
35number less than or equal to 10 . The line number is available using the special variable NR .
36In the implementation of the tail command a hashing technique is used. The buffer array
37index is determined by a hashing function NR % 10 , where NR is the variable that contains the
38Linux number of current execution. $0 is the line in the text variable. Hence % maps all the lines
39having the same remainder in the hash function to a particular index of an array. In the END{}
40block, it can iterate through ten index values of an array and print the lines stored in a buffer.
41In the tac command emulation, it simply stores all the lines in an array. When it appears in
42the END{} block, NR will be holding the line number of the last line. Then it is decremented in
43a for loop until it reaches 1 and it prints the lines stored in each iteration statement.
44See also
45f Basic awk primer, explains the awk command
46f head and tail - printing the last or first 10 lines of Chapter 3, explains the commands
47head and tail
48f Sorting, unique and duplicates of Chapter 2, explains the uniq command
49f Printing lines in reverse order, explains the tac command
50www.it-ebooks.info
51Chapter 4
52177
53Text slicing and parameter operations
54This recipe walks through some of the simple text replacement techniques and parameter
55expansion short hands available in Bash. A few simple techniques can often help us avoid
56having to write multiple lines of code.
57How to do it...
58Let's get into the tasks.
59Replacing some text from a variable can be done as follows:
60$ var="This is a line of text"
61$ echo ${var/line/REPLACED}
62This is a REPLACED of text"
63line is replaced with REPLACED .
64We can produce a sub-string by specifying the start position and string length, by using the
65following syntax:
66${variable_name:start_position:length}
67To print from the fifth character onward use the following command:
68$ string=abcdefghijklmnopqrstuvwxyz
69$ echo ${string:4}
70efghijklmnopqrstuvwxyz
71To print eight characters starting from the fifth character, use:
72$ echo ${string:4:8}
73efghijkl
74The index is specified by counting the start letter as 0 . We can also specify counting from last
75letter as -1 . It is but used inside a parenthesis. (-1) is the index for the last letter.
76echo ${string:(-1)}
77z
78$ echo ${string:(-2):2}
79yz
80See also
81f Iterating through lines, words, and characters in a file, explains slicing of a character
82from a word
83www.it-ebooks.info
84www.it-ebooks.info
855
86Tangled Web?
87Not At All!
88In this chapter, we will cover:
89f Downloading from a web page
90f Downloading a web page as formatted plain text
91f A primer on cURL
92f Accessing unread Gmail mails from the command line
93f Parsing data from a website
94f Creating an image crawler and downloader
95f Creating a web photo album generator
96f Building a Twitter command-line client
97f Define utility with Web backend
98f Finding broken links in a website
99f Tracking changes to a website
100f Posting to a web page and reading response
101www.it-ebooks.info
102Tangled Web? Not At All!
103180
104Introduction
105The Web is becoming the face of technology. It is the central access point for data processing.
106Though shell scripting cannot do everything that languages like PHP can do on the Web, there
107are still many tasks to which shell scripts are ideally suited. In this chapter we will explore
108some recipes that can be used to parse website content, download and obtain data, send
109data to forms, and automate website usage tasks and similar activities. We can automate
110many activities that we perform interactively through a browser with a few lines of scripting.
111Access to the functionalities provided by the HTTP protocol with command-line utilities
112enables us to write scripts that are suitable to solve most of the web-automation utilities.
113Have fun while going through the recipes of this chapter.
114Downloading from a web page
115Downloading a file or a web page from a given URL is simple. A few command-line download
116utilities are available to perform this task.
117Getting ready
118wget is a file download command-line utility. It is very flexible and can be configured with
119many options.
120How to do it...
121A web page or a remote file can be downloaded using wget as follows:
122$ wget URL
123For example:
124$ wget http://slynux.org
125--2010-08-01 07:51:20-- http://slynux.org/
126Resolving slynux.org... 174.37.207.60
127Connecting to slynux.org|174.37.207.60|:80... connected.
128HTTP request sent, awaiting response... 200 OK
129Length: 15280 (15K) [text/html]
130Saving to: "index.html"
131100%[======================================>] 15,280 75.3K/s in
1320.2s
1332010-08-01 07:51:21 (75.3 KB/s) - "index.html" saved [15280/15280]
134www.it-ebooks.info
135Chapter 5
136181
137It is also possible to specify multiple download URLs as follows:
138$ wget URL1 URL2 URL3 ..
139A file can be downloaded using wget using the URL as:
140$ wget ftp://example_domain.com/somefile.img
141Usually, files are downloaded with the same filename as in the URL and the download log
142information or progress is written to stdout .
143You can specify the output file name with the -O option. If the file with the specified filename
144already exists, it will be truncated first and the downloaded file will be written to the specified
145file.
146You can also specify a different logfile path rather than printing logs to stdout by using
147the -o option as follows:
148$ wget ftp://example_domain.com/somefile.img -O dloaded_file.img -o log
149By using the above command, nothing will be printed on screen. The log or progress will be
150written to log and the output file will be dloaded_file.img .
151There is a chance that downloads might break due to unstable Internet connections. Then we
152can use the number of tries as an argument so that once interrupted, the utility will retry the
153download that many times before giving up.
154In order to specify the number of tries, use the -t flag as follows:
155$ wget -t 5 URL
156There's more...
157The wget utility has several additional options that can be used under different problem
158domains. Let's go through a few of them.
159Restricted with speed downloads
160When we have a limited Internet downlink bandwidth and many applications sharing the
161internet connection, if a large file is given for download, it will suck all the bandwidth and
162may cause other process to starve for bandwidth. The wget command comes with a built-in
163option to specify the maximum bandwidth limit the download job can possess. Hence all the
164applications can simultaneously run smoothly.
165We can restrict the speed of wget by using the --limit-rate argument as follows:
166$ wget --limit-rate 20k http://example.com/file.iso
167In this command k (kilobyte) and m (megabyte) specify the speed limit.
168www.it-ebooks.info
169Tangled Web? Not At All!
170182
171We can also specify the maximum quota for the download. It will stop when the quota is
172exceeded. It is useful when downloading multiple files limited by the total download size. This
173is useful to prevent the download from accidently using too much disk space.
174Use --quota or –Q as follows:
175$ wget -Q 100m http://example.com/file1 http://example.com/file2
176Resume downloading and continue
177If a download using wget gets interrupted before it is completed, we can resume the
178download where we left off by using the -c option as follows:
179$ wget -c URL
180Using cURL for download
181cURL is another advanced command-line utility. It is much more powerful than wget .
182cURL can be used to download as follows:
183$ curl http://slynux.org > index.html
184Unlike wget , curl writes the downloaded data into standard output ( stdout ) rather than to a
185file. Therefore, we have to redirect the data from stdout to the file using a redirection operator.
186Copying a complete website (mirroring)
187wget has an option to download the complete website by recursively collecting all the URL
188links in the web pages and downloading all of them like a crawler. Hence we can completely
189download all the pages of a website.
190In order to download the pages, use the --mirror option as follows:
191$ wget --mirror exampledomain.com
192Or use:
193$ wget -r -N -l DEPTH URL
194-l specifies the DEPTH of web pages as levels. That means it will traverse only that much
195number of levels. It is used along with –r (recursive). The -N argument is used to enable time
196stamping for the file. URL is the base URL for a website for which the download needs to be
197initiated.
198Accessing pages with HTTP or FTP authentication
199Some web pages require authentication for HTTP or FTP URLs. This can be provided by using
200the --user and --password arguments:
201$ wget –-user username –-password pass URL
202www.it-ebooks.info
203Chapter 5
204183
205It is also possible to ask for a password without specifying the password inline. In order to do
206that use --ask-password instead of the --password argument.
207Downloading a web page as formatted
208plain text
209Web pages are HTML pages containing a collection of HTML tags along with other elements,
210such as JavaScript, CSS, and so on. But the HTML tags define the base of a web page. We
211may need to parse the data in a web page while looking for specific content, and this is
212something Bash scripting can help us with. When we download a web page, we receive an
213HTML file. In order to view formatted data, it should be viewed in a web browser. However, in
214most of the circumstances, parsing a formatted text document will be easier than parsing
215HTML data. Therefore, if we can get a text file with formatted text similar to the web page seen
216on the web browser, it is more useful and it saves a lot of effort required to strip off HTML
217tags. Lynx is an interesting command-line web browser. We can actually get the web page as
218plain text formatted output from Lynx. Let's see how to do it.
219How to do it...
220Let's download the webpage view, in ASCII character representation, in a text file using the
221–dump flag with the lynx command:
222$ lynx -dump URL > webpage_as_text.txt
223This command will also list all the hyper-links ( <a href="link"> ) separately under a
224heading References as the footer of the text output. This would help us avoid parsing of links
225separately using regular expressions.
226For example:
227$ lynx -dump http://google.com > plain_text_page.txt
228You can see the plain text version of text by using the cat command as follows:
229$ cat plain_text_page.txt
230A primer on cURL
231cURL is a powerful utility that supports many protocols including HTTP, HTTPS, FTP, and much
232more. It supports many features including POST, cookie, authentication, downloading partial
233files from a specified offset, referers, user agent strings, extra headers, limit speed, maximum
234file size, progress bars, and so on. cURL is useful for when we want to play around with
235automating a web page usage sequence and to retrieve data. This recipe is a list of the most
236important features of cURL.
237www.it-ebooks.info
238Tangled Web? Not At All!
239184
240Getting ready
241cURL doesn't come with any of the main Linux distros by default, so you may have to install it
242using the package manager. By default, most distributions ship with wget .
243cURL usually dumps downloaded files to stdout and progress information to stderr . To
244avoid progress information from being shown, we always use the --silent option.
245How to do it…
246The curl command can be used to perform different activities such as downloading, sending
247different HTTP requests, specifying HTTP headers, and so on. Let's see how to perform
248different tasks with cURL.
249$ curl URL --silent
250The above command dumps the downloaded file into the terminal (the downloaded data is
251written to stdout ).
252The --silent option is used to prevent the curl command from displaying progress
253information. If progress information is required, remove --silent .
254$ curl URL –-silent -O
255The -O option is used to write the downloaded data into a file with the filename parsed from
256the URL rather than writing into the standard output.
257For example:
258$ curl http://slynux.org/index.html --silent -O
259index.html will be created.
260It writes a web page or file to the filename as in the URL instead of writing to stdout . If
261filenames are not there in the URL, it will produce an error. Hence, make sure that the URL
262is a URL to a remote file. curl http://slynux.org -O --silent will display an error
263since the filename cannot be parsed from the URL.
264$ curl URL –-silent -o new_filename
265The -o option is used to download a file and write to a file with a specified file name.
266In order to show the # progress bar while downloading, use –-progress instead of
267–-silent .
268$ curl http://slynux.org -o index.html --progress
269################################## 100.0%
270www.it-ebooks.info
271Chapter 5
272185
273There's more...
274In the previous sections we have learned how to download files and dump HTML pages to the
275terminal. There several advanced options that come along with cURL. Let's explore more
276on cURL.
277Continue/Resume downloading
278cURL has advanced resume download features to continue at a given offset unlike wget . It
279helps to download portions of files by specifying an offset.
280$ curl URL/file -C offset
281The offset is an integer value in bytes.
282cURL doesn't require us to know the exact byte offset if we want to resume downloading a file.
283If you want cURL to figure out the correct resume point, use the -C - option, like this:
284$ curl -C - URL
285cURL will automatically figure out where to restart the download of the specified file.
286Set referer string with cURL
287Referer is a string in the HTTP header used to identify the page from which the user reaches
288the current web page. When a user clicks on a link from web page A and it reaches web page
289B, the referer header string in the page B will contain a URL of page A.
290Some dynamic pages check the referer string before returning HTML data. For example, a web
291page shows a Google logo attached page when a user navigates to a website by searching on
292Google, and shows a different page when they navigate to the web page by manually typing
293the URL.
294The web page can write a condition to return a Google page if the referer is www.google.com
295or else return a different page.
296You can use --referer with the curl command to specify the referer string as follows:
297$ curl –-referer Referer_URL target_URL
298For example:
299$ curl –-referer http://google.com http://slynux.org
300Cookies with cURL
301Using curl we can specify as well as store cookies encountered during HTTP operations.
302In order to specify cookies, use the --cookie "COOKIES" option.
303www.it-ebooks.info
304Tangled Web? Not At All!
305186
306Cookies should be provided as name=value . Multiple cookies should be delimited by a
307semicolon ";". For example:
308$ curl http://example.com –-cookie "user=slynux;pass=hack"
309In order to specify a file to which cookies encountered are to be stored, use the --cookie-
310jar option. For example:
311$ curl URL –-cookie-jar cookie_file
312Setting a user agent string with cURL
313Some web pages that check the user-agent won't work if there is no user-agent specified. You
314may have noticed that certain websites work well only in Internet Explorer (IE). If a different
315browser is used, the website will show a message that it will work only on IE. This is because
316the website checks for a user agent. You can set the user agent as IE with curl and see that
317it returns a different web page in this case.
318Using cURL it can be set using --user-agent or –A as follows:
319$ curl URL –-user-agent "Mozilla/5.0"
320Additional headers can be passed with cURL. Use –H "Header" to pass multiple additional
321headers. For example:
322$ curl -H "Host: www.slynux.org" -H "Accept-language: en" URL
323Specifying bandwidth limit on cURL
324When the available bandwidth is limited and multiple users are sharing the Internet, in order
325to perform the sharing of bandwidth smoothly, we can limit the download rate to a specified
326limit from curl by using the --limit-rate option as follows:
327$ curl URL --limit-rate 20k
328In this command k (kilobyte) and m (megabyte) specify the download rate limit.
329Specifying the maximum download size
330The maximum download file size for cURL can be specified using the --max-filesize
331option as follows:
332$ curl URL --max-filesize bytes
333It will return a non-zero exit code if the file size exceeds. It will return zero if it succeeds.
334Authenticating with cURL
335HTTP authentication or FTP authentication can be done using cURL with the -u argument.
336www.it-ebooks.info
337Chapter 5
338187
339The username and password can be specified using -u username:password . It is possible
340to not provide a password such that it will prompt for password while executing.
341If you prefer to be prompted for the password, you can do that by using only -u username .
342For example:
343$ curl -u user:pass http://test_auth.com
344In order to be prompted for the password use:
345$ curl -u user http://test_auth.com
346Printing response headers excluding data
347It is useful to print only response headers to apply many checks or statistics. For example, to
348check whether a page is reachable or not, we don't need to download the entire page contents.
349Just reading the HTTP response header can be used to identify if a page is available or not.
350An example usage case for checking the HTTP header is to check the file size before
351downloading. We can check the Content-Length parameter in the HTTP header to find out
352the length of a file before downloading. Also, several useful parameters can be retrieved from
353the header. The Last-Modified parameter enables to know the last modification time for
354the remote file.
355Use the –I or –head option with curl to dump only HTTP headers without downloading the
356remote file. For example:
357$ curl -I http://slynux.org
358HTTP/1.1 200 OK
359Date: Sun, 01 Aug 2010 05:08:09 GMT
360Server: Apache/1.3.42 (Unix) mod_gzip/1.3.26.1a mod_log_bytes/1.2
361mod_bwlimited/1.4 mod_auth_passthrough/1.8 FrontPage/5.0.2.2635 mod_
362ssl/2.8.31 OpenSSL/0.9.7a
363Last-Modified: Thu, 19 Jul 2007 09:00:58 GMT
364ETag: "17787f3-3bb0-469f284a"
365Accept-Ranges: bytes
366Content-Length: 15280
367Connection: close
368Content-Type: text/html
369See also
370f Posting to a web page and reading response
371www.it-ebooks.info
372Tangled Web? Not At All!
373188
374Accessing Gmail from the command line
375Gmail is a widely-used free e-mail service from Google : http://mail.google.com/ .
376Gmail allows you to read your mail via authenticated RSS feeds. We can parse the RSS feeds
377with the sender's name and an e-mail with subject. It will help to have a look at unread mails
378in the inbox without opening the web browser.
379How to do it...
380Let's go through the shell script to parse the RSS feeds for Gmail to display the unread mails:
381#!/bin/bash
382Filename: fetch_gmail.sh
383#Description: Fetch gmail tool
384username="PUT_USERNAME_HERE"
385password="PUT_PASSWORD_HERE"
386SHOW_COUNT=5 # No of recent unread mails to be shown
387echo
388curl -u $username:$password --silent "https://mail.google.com/mail/
389feed/atom" | \
390tr -d '\n' | sed 's:</entry>:\n:g' |\
391sed 's/.*<title>\(.*\)<\/title.*<author><name>\([^<]*\)<\/
392name><email>
393\([^<]*\).*/Author: \2 [\3] \nSubject: \1\n/' | \
394head -n $(( $SHOW_COUNT * 3 ))
395The output will be as follows:
396$ ./fetch_gmail.sh
397Author: SLYNUX [ slynux@slynux.com ]
398Subject: Book release - 2
399Author: SLYNUX [ slynux@slynux.com ]
400Subject: Book release - 1
401.
402… 5 entries
403How it works...
404The script uses cURL to download the RSS feed by using user authentication. User authentication
405is provided by the -u username:password argument. You can use -u user without providing
406the password. Then while executing cURL it will interactively ask for the password.
407www.it-ebooks.info
408Chapter 5
409189
410Here we can split the piped commands into different blocks to illustrate how they work.
411tr -d '\n' removes the newline character so that we restructure each mail entry with \n
412as the delimiter. sed 's:</entry>:\n:g' replaces every </entry> with a newline so that
413each mail entry is delimited by a newline and hence mails can be parsed one by one. Have a
414look at the source of https://mail.google.com/mail/feed/atom for XML tags used in
415the RSS feeds. <entry> TAGS </entry> corresponds to a single mail entry.
416The next block of script is as follows:
417sed 's/.*<title>\(.*\)<\/title.*<author><name>\([^<]*\)<\/
418name><email>
419\([^<]*\).*/Author: \2 [\3] \nSubject: \1\n/'
420This script matches the substring title using <title>\(.*\)<\/title , the sender name
421using <author><name>\([^<]*\)<\/name> , and e-mail using <email>\([^<]*\) . Then
422back referencing is used as follows:
423f Author: \2 [\3] \nSubject: \1\n is used to replace an entry for a mail with
424the matched items in an easy-to-read format. \1 corresponds to the first substring
425match, \2 for the second substring match, and so on.
426f The SHOW_COUNT=5 variable is used to take the number of unread mail entries to be
427printed on terminal.
428f head is used to display only SHOW_COUNT*3 lines from the first line. SHOW_COUNT is
429used three times in order to show three lines of the output.
430See also
431f A primer on cURL, explains the curl command
432f Basic sed primer of Chapter 4, explains the sed command
433Parsing data from a website
434It is often useful to parse data from web pages by eliminating unnecessary details. sed and awk
435are the main tools that we will use for this task. You might have come across a list of access
436rankings in a grep recipe in the previous chapter Texting and driving; it was generated by parsing
437the website page http://www.johntorres.net/BoxOfficefemaleList.html .
438Let's see how to parse the same data using text-processing tools.
439www.it-ebooks.info
440Tangled Web? Not At All!
441190
442How to do it...
443Let's go through the command sequence used to parse details of actresses from the website:
444$ lynx -dump http://www.johntorres.net/BoxOfficefemaleList.html | \ grep
445-o "Rank-.*" | \
446sed 's/Rank-//; s/\[[0-9]\+\]//' | \
447sort -nk 1 |\
448awk '
449{
450for(i=3;i<=NF;i++){ $2=$2" "$i }
451printf "%-4s %s\n", $1,$2 ;
452}' > actresslist.txt
453The output will be as follows:
454# Only 3 entries shown. All others omitted due to space limits
4551 Keira Knightley
4562 Natalie Portman
4573 Monica Bellucci
458How it works...
459Lynx is a command-line web browser; it can dump the text version of the website as we
460would see in a web browser rather than showing us the raw code. Hence it avoids the job of
461removing the HTML tags. We parse the lines starting with Rank, using sed as follows:
462sed 's/Rank-//; s/\[[0-9]\+\]//'
463These lines could be then sorted according to the ranks. awk is used here to keep the spacing
464between rank and the name uniform by specifying the width. %-4s specifies a four-character
465width. All the fields except the first field are concatenated to form a single string as $2 .
466See also
467f Basic sed primer of Chapter 4, explains the sed command
468f Basic awk primer of Chapter 4, explains the awk command
469f Downloading a web page as formatted plain text, explains the lynx command
470www.it-ebooks.info
471Chapter 5
472191
473Image crawler and downloader
474Image crawlers are very useful when we need to download all the images that appear in a web
475page. Instead of going through the HTML sources and picking all the images, we can use a
476script to parse the image files and download them automatically. Let's see how to do it.
477How to do it...
478Let's write a Bash script to crawl and download the images from a web page as follows:
479#!/bin/bash
480#Description: Images downloader
481#Filename: img_downloader.sh
482if [ $# -ne 3 ];
483then
484echo "Usage: $0 URL -d DIRECTORY"
485exit -1
486fi
487for i in {1..4}
488do
489case $1 in
490-d) shift; directory=$1; shift ;;
491*) url=${url:-$1}; shift;;
492esac
493done
494mkdir -p $directory;
495baseurl=$(echo $url | egrep -o "https?://[a-z.]+")
496curl –s $url | egrep -o "<img src=[^>]*>" |
497sed 's/<img src=\"\([^"]*\).*/\1/g' > /tmp/$$.list
498sed -i "s|^/|$baseurl/|" /tmp/$$.list
499cd $directory;
500while read filename;
501do
502curl –s -O "$filename" --silent
503done < /tmp/$$.list
504An example usage is as follows:
505$ ./img_downloader.sh http://www.flickr.com/search/?q=linux -d images
506www.it-ebooks.info
507Tangled Web? Not At All!
508192
509How it works...
510The above image downloader script parses an HTML page, strips out all tags except <img> ,
511then parses src="URL" from the <img> tag and downloads them to the specified directory.
512This script accepts a web page URL and the destination directory path as command-line
513arguments. The first part of the script is a tricky way to parse command-line arguments.
514The [ $# -ne 3 ] statement checks whether the total number of arguments to the script
515is three, else it exits and returns a usage example.
516If it is 3 arguments, then parse the URL and the destination directory. In order to do that a
517tricky hack is used:
518for i in {1..4}
519do
520case $1 in
521-d) shift; directory=$1; shift ;;
522*) url=${url:-$1}; shift;;
523esac
524done
525A for loop is iterated four times (there is no significance to the number four, it is just to iterate
526a couple of times to run the case statement).
527The case statement will evaluate the first argument ( $1 ), and matches -d or any other
528string arguments that are checked. We can place the -d argument anywhere in the format as
529follows:
530$ ./img_downloader.sh -d DIR URL
531Or:
532$ ./img_downloader.sh URL -d DIR
533shift is used to shift arguments such that when shift is called $1 will be assigned with
534$2 , when again called $1=$3 and so on as it shifts $1 to the next arguments. Hence we can
535evaluate all arguments through $1 itself.
536When -d is matched ( -d) ), it is obvious that the next argument is the value for the
537destination directory. *) corresponds to default match. It will match anything other than
538-d . Hence while iteration $1="" or $1=URL in the default match, we need to take $1=URL
539avoiding "" to overwrite. Hence we use the url=${url:-$1} trick. It will return a URL value
540if already not "" else it will assign $1 .
541egrep -o "<img src=[^>]*>" will print only the matching strings, which are the <img>
542tags including their attributes. [^>]* used to match all characters except the closing > , that
543is, <img src="image.jpg" …. > .
544www.it-ebooks.info
545Chapter 5
546193
547sed 's/<img src=\"\([^"]*\).*/\1/g' parses src="url" so that all image URLs
548can be parsed from the <img> tags already parsed.
549There are two types of image source paths: relative and absolute. Absolute paths contain full
550URLs that start with http:// or https:// . Relative URLs starts with / or image_name itself.
551An example of an absolute URL is: http://example.com/image.jpg
552An example of a relative URL is: /image.jpg
553For relative URLs the starting / should be replaced with the base URL to transform it to
554http://example.com/image.jpg .
555For that transformation, we initially find out baseurl sed by parsing.
556Then replace every occurrence of the starting / with baseurl sed as sed -i
557"s|^/|$baseurl/|" /tmp/$$.list .
558Then a while loop is used to iterate the list line by line and download the URL using curl .
559The --silent argument is used with curl to avoid other progress messages from being
560printed on the screen.
561See also
562f A primer on cURL, explains the curl command
563f Basic sed primer of Chapter 4, explains the sed command
564f Searching and mining "text" inside a file with grep of Chapter 4, explains the grep
565command
566Web photo album generator
567Web developers commonly design photo album pages for websites that consist of a number
568of image thumbnails on the page. When thumbnails are clicked, a large version of the
569picture will be displayed. But when many images are required, copying the <img> tag every
570time, resizing the image to create a thumbnail, placing them in the thumbs directory, testing
571the links, and so on are real hurdles. It takes a lot of time and repeats the same task. It
572can be automated easily by writing a simple Bash script. By writing a script, we can create
573thumbnails, place them in exact directories, and generate the code fragment for <img> tags
574automatically in few seconds. This recipe will teach you how to do it.
575Getting ready
576We can perform this task with a for loop that iterates every image in the current directory.
577The usual Bash utilities such as cat and convert (image magick) are used. These will
578generate an HTML album, using all the images, to index.html . In order to use convert ,
579make sure you have Imagemagick installed.
580www.it-ebooks.info
581Tangled Web? Not At All!
582194
583How to do it...
584Let's write a Bash script to generate a HTML album page:
585#!/bin/bash
586#Filename: generate_album.sh
587#Description: Create a photo album using images in current directory
588echo "Creating album.."
589mkdir -p thumbs
590cat <<EOF > index.html
591<html>
592<head>
593<style>
594body
595{
596width:470px;
597margin:auto;
598border: 1px dashed grey;
599padding:10px;
600}
601img
602{
603margin:5px;
604border: 1px solid black;
605}
606</style>
607</head>
608<body>
609<center><h1> #Album title </h1></center>
610<p>
611EOF
612for img in *.jpg;
613do
614convert "$img" -resize "100x" "thumbs/$img"
615echo "<a href=\"$img\" ><img src=\"thumbs/$img\" title=\"$img\" />
616</a>" >> index.html
617done
618cat <<EOF >> index.html
619</p>
620</body>
621</html>
622EOF
623echo Album generated to index.html
624www.it-ebooks.info
625Chapter 5
626195
627Run the script as follows:
628$ ./generate_album.sh
629Creating album..
630Album generated to index.html
631How it works...
632The initial part of the script is to write the header part of the HTML page.
633The following script redirects all the contents up to EOF (excluding) to the index.html :
634cat <<EOF > index.html
635contents...
636EOF
637The header includes the HTML and stylesheets.
638for img in *.jpg; will iterate through names of each file and will perform actions.
639convert "$img" -resize "100x" "thumbs/$img" will create images of 100px width
640as thumbnails.
641The following statement will generate the required <img> tag and appends it to the index.html :
642echo "<a href=\"$img\" ><img src=\"thumbs/$img\" title=\"$img\" /></
643a>" >> index.html
644Finally, the footer HTML tags are appended with cat again.
645See also
646f Playing with file descriptors and redirection of Chapter 1, explains EOF and stdin
647redirection.
648Twitter command-line client
649Twitter is the hottest micro blogging platform as well as the latest buzz of online social media.
650Tweeting and reading tweets is fun. What if we can do both from command line? It is pretty
651simple to write a command-line Twitter client. Twitter has RSS feeds and hence we can make
652use of them. Let's see how to do it.
653Getting ready
654We can use cURL to authenticate and send twitter updates as well as download the RSS feed
655pages to parse the tweets. Just four lines of code can do it. Let's do it.
656www.it-ebooks.info
657Tangled Web? Not At All!
658196
659How to do it...
660Let's write a Bash script using the curl command to manipulate twitter APIs:
661#!/bin/bash
662#Filename: tweets.sh
663#Description: Basic twitter client
664USERNAME="PUT_USERNAME_HERE"
665PASSWORD="PUT_PASSWORD_HERE"
666COUNT="PUT_NO_OF_TWEETS"
667if [[ "$1" != "read" ]] && [[ "$1" != "tweet" ]];
668then
669echo -e "Usage: $0 send status_message\n OR\n $0 read\n"
670exit -1;
671fi
672if [[ "$1" = "read" ]];
673then
674curl --silent -u $USERNAME:$PASSWORD http://twitter.com/statuses/
675friends_timeline.rss | \
676grep title | \
677tail -n +2 | \
678head -n $COUNT | \
679sed 's:.*<title>\([^<]*\).*:\n\1:'
680elif [[ "$1" = "tweet" ]];
681then
682status=$( echo $@ | tr -d '"' | sed 's/.*tweet //')
683curl --silent -u $USERNAME:$PASSWORD -d status="$status" http://
684twitter.com/statuses/update.xml > /dev/null
685echo 'Tweeted :)'
686fi
687Run the script as follows:
688$ ./tweets.sh tweet Thinking of writing a X version of wall command
689"#bash"
690Tweeted :)
691$ ./tweets.sh read
692bot: A tweet line
693t3rm1n4l: Thinking of writing a X version of wall command #bash
694www.it-ebooks.info
695Chapter 5
696197
697How it works...
698Let's see the working of above script by splitting it into two parts. The first part is
699about reading tweets. To read tweets the script downloads the RSS information from
700http://twitter.com/statuses/friends_timeline.rss and parses the lines
701containing the <title> tag. Then it strips off the <title> and </title> tags using sed
702to form the required tweet text. Then a COUNT variable is used to remove all other text except
703the number of recent tweets by using the head command. tail –n +2 is used to remove an
704unnecessary header text "Twitter: Timeline of friends".
705In the sending tweet part, the -d status argument of curl is used to post data to Twitter
706using their API: http://twitter.com/statuses/update.xml .
707$1 of the script will be the tweet in the case of sending a tweet. Then to obtain the status we
708take $@ (list of all arguments of the script) and remove the word "tweet" from it.
709See also
710f A primer on cURL, explains the curl command
711f head and tail - printing the last or first 10 lines of Chapter 3, explains the commands
712head and tail
713define utility with Web backend
714Google provides Web definitions for any word by using the search query define:WORD . We
715need a GUI web browser to fetch the definitions. However, we can automate it and parse the
716required definitions by using a script. Let's see how to do it.
717Getting ready
718We can use lynx , sed , awk , and grep to write the define utility.
719How to do it...
720Let's go through the code for the define utility script to fetch definitions from Google search:
721#!/bin/bash
722#Filename: define.sh
723#Description: A Google define: frontend
724limit=0
725if [ ! $# -ge 1 ];
726then
727echo -e "Usage: $0 WORD [-n No_of_definitions]\n"
728exit -1;
729www.it-ebooks.info
730Tangled Web? Not At All!
731198
732fi
733if [ "$2" = "-n" ];
734then
735limit=$3;
736let limit++
737fi
738word=$1
739lynx -dump http://www.google.co.in/search?q=define:$word | \
740awk '/Defini/,/Find defini/' | head -n -1 | sed 's:*:\n*:; s:^[ ]*::'
741| \
742grep -v "[[0-9]]" | \
743awk '{
744if ( substr($0,1,1) == "*" )
745{ sub("*",++count".") } ;
746print
747} ' > /tmp/$$.txt
748echo
749if [ $limit -ge 1 ];
750then
751cat /tmp/$$.txt | sed -n "/^1\./, /${limit}/p" | head -n -1
752else
753cat /tmp/$$.txt;
754fi
755Run the script as follows:
756$ ./define.sh hack -n 2
7571. chop: cut with a hacking tool
7582. one who works hard at boring tasks
759How it works...
760We will look into the core part of the definition parser. Lynx is used to obtain the plain text
761version of the web page. http://www.google.co.in/search?q=define:$word is
762the URL for the web definition web page. Then we reduce the text between "Definitions on
763web" and "Find definitions". All the definitions are occurring in between these lines of text
764( awk '/Defini/,/Find defini/' ).
765www.it-ebooks.info
766Chapter 5
767199
768's:*:\n*:' is used to replace * with * and newline in order to insert a newline in between
769each definition, and s:^[ ]*:: is used to remove extra spaces in the start of lines. Hyperlinks
770are marked as [number] in lynx output. Those lines are removed by grep -v , the invert match
771lines option. Then awk is used to replace the * occurring at start of the line with a number so
772that each definition can assign a serial number. If we have read a -n count in the script, it has to
773output only a few definitions as per count. So awk is used to print the definitions with number 1
774to count (this makes it easier since we replaced * with the serial number).
775See also
776f Basic sed primer of Chapter 4, explains the sed command
777f Basic awk primer of Chapter 4, explains the awk command
778f Searching and mining "text" inside a file with grep of Chapter 4, explains the grep
779command
780f Downloading a web page as formatted plain text, explains the lynx command
781Finding broken links in a website
782I have seen people manually checking each and every page on a site to search for broken links.
783It is possible only for websites having very few pages. When the number of pages become large,
784it will become impossible. It becomes really easy if we can automate finding broken links. We
785can find the broken links by using HTTP manipulation tools. Let's see how to do it.
786Getting ready
787In order to identify the links and find the broken ones from the links, we can use lynx and
788curl . It has an option -traversal , which will recursively visit pages in the website and build
789the list of all hyperlinks in the website. We can use cURL to verify whether each of the links
790are broken or not.
791How to do it...
792Let's write a Bash script with the help of the curl command to find out the broken links on a
793web page:
794#!/bin/bash
795#Filename: find_broken.sh
796#Description: Find broken links in a website
797if [ $# -eq 2 ];
798then
799echo -e "$Usage $0 URL\n"
800exit -1;
801fi
802www.it-ebooks.info
803Tangled Web? Not At All!
804200
805echo Broken links:
806mkdir /tmp/$$.lynx
807cd /tmp/$$.lynx
808lynx -traversal $1 > /dev/null
809count=0;
810sort -u reject.dat > links.txt
811while read link;
812do
813output=`curl -I $link -s | grep "HTTP/.*OK"`;
814if [[ -z $output ]];
815then
816echo $link;
817let count++
818fi
819done < links.txt
820[ $count -eq 0 ] && echo No broken links found.
821How it works...
822lynx -traversal URL will produce a number of files in the working directory. It includes
823a file reject.dat which will contain all the links in the website. sort -u is used to build a
824list by avoiding duplicates. Then we iterate through each link and check the header response
825by using curl -I . If the header contains first line HTTP/1.0 200 OK as the response, it
826means that the target is not broken. All other responses correspond to broken links and are
827printed out to stdout .
828See also
829f Downloading a web page as formatted plain text, explains the lynx command
830f A primer on cURL, explains the curl command
831Tracking changes to a website
832Tracking changes to a website is helpful to web developers and users. Checking a website
833manually in intervals is really hard and impractical. Hence we can write a change tracker
834running at repeated intervals. When a change occurs, it can play a sound or send a
835notification. Let's see how to write a basic tracker for the website changes.
836www.it-ebooks.info
837Chapter 5
838201
839Getting ready
840Tracking changes in terms of Bash scripting means fetching websites at different times and
841taking the difference using the diff command. We can use curl and diff to do this.
842How to do it...
843Let's write a Bash script by combining different commands to track changes in a web page:
844#!/bin/bash
845#Filename: change_track.sh
846#Desc: Script to track changes to webpage
847if [ $# -eq 2 ];
848then
849echo -e "$Usage $0 URL\n"
850exit -1;
851fi
852first_time=0
853# Not first time
854if [ ! -e "last.html" ];
855then
856first_time=1
857# Set it is first time run
858fi
859curl --silent $1 -o recent.html
860if [ $first_time -ne 1 ];
861then
862changes=$(diff -u last.html recent.html)
863if [ -n "$changes" ];
864then
865echo -e "Changes:\n"
866echo "$changes"
867else
868echo -e "\nWebsite has no changes"
869fi
870else
871echo "[First run] Archiving.."
872fi
873cp recent.html last.html
874www.it-ebooks.info
875Tangled Web? Not At All!
876202
877Let's look at the output of the track_changes.sh script when changes are made to the web
878page and when the changes are not made to the page:
879f First run:
880$ ./track_changes.sh http://web.sarathlakshman.info/test.html
881[First run] Archiving..
882f Second Run:
883$ ./track_changes.sh http://web.sarathlakshman.info/test.html
884Website has no changes
885f Third run after making changes to the web page:
886$ ./test.sh http://web.sarathlakshman.info/test_change/test.html
887Changes:
888--- last.html 2010-08-01 07:29:15.000000000 +0200
889+++ recent.html 2010-08-01 07:29:43.000000000 +0200
890@@ -1,3 +1,4 @@
891<html>
892+added line :)
893<p>data</p>
894</html>
895How it works...
896The script checks whether the script is running for the first time using [ ! -e "last.html"
897]; . If last.html doesn't exist, that means it is the first time and hence the webpage must
898be downloaded and copied as last.html .
899If it is not the first time, it should download the new copy ( recent.html ) and check the
900difference using the diff utility. If changes are there, it should print the changes and finally it
901should copy recent.html to last.html .
902See also
903f A primer on cURL, explains the curl command
904www.it-ebooks.info
905Chapter 5
906203
907Posting to a web page and reading response
908POST and GET are two types of requests in HTTP to send information to or retrieve information
909from a website. In a GET request, we send parameters (name-value pairs) through the web
910page URL itself. In the case of POST, it won't be attached with the URL. POST is used when a
911form needs to be submitted. For example, a username, the password to be submitted, and the
912login page to be retrieved.
913POSTing to pages comes as frequent use while writing scripts based on web page retrievals.
914Let's see how to work with POST. Automating the HTTP GET and POST request by sending
915POST data and retrieving output is a very important task that we practice while writing shell
916scripts that parse data from websites.
917Getting ready
918Both cURL and wget can handle POST requests by arguments. They are to be passed as
919name-value pairs.
920How to do it...
921Let's see how to POST and read HTML response from a real website using curl :
922$ curl URL -d "postvar=postdata2&postvar2=postdata2"
923We have a website ( http://book.sarathlakshman.com/lsc/mlogs/ ) and it is used
924to submit the current user information such as hostname and username. Assume that, in
925the home page of the website there are two fields HOSTNAME and USER, and a SUBMIT
926button. When the user enters a hostname, a user name, and clicks on the SUBMIT button,
927the details will be stored in the website. This process can be automated using a single line of
928curl command by automating the POST request. If you look at the website source (use the
929view source option from the web browser), you can see an HTML form defined similar to the
930following code:
931<form action="http://book.sarathlakshman.com/lsc/mlogs/submit.php"
932method="post" >
933<input type="text" name="host" value="HOSTNAME" >
934<input type="text" name="user" value="USER" >
935<input type="submit" >
936</form>
937Here, http://book.sarathlakshman.com/lsc/mlogs/submit.php is the target
938URL. When the user enters the details and clicks on the Submit button. The host and user
939inputs are sent to submit.php as a POST request and the response page is returned on the
940browser.
941www.it-ebooks.info
942Tangled Web? Not At All!
943204
944We can automate the POST request as follows:
945$ curl http://book.sarathlakshman.com/lsc/mlogs/submit.php -d "host=test-
946host&user=slynux"
947<html>
948You have entered :
949<p>HOST : test-host</p>
950<p>USER : slynux</p>
951<html>
952Now curl returns the response page.
953-d is the argument used for posting. The string argument for -d is similar to the GET request
954semantics. var=value pairs are to be delimited by & .
955The -d argument should always be given in quotes. If quotes are not used, &
956is interpreted by the shell to indicate this should be a background process.
957There's more
958Let's see how to perform POST using cURL and wget .
959POST in curl
960You can POST data in curl by using -d or –data as follows:
961$ curl –-data "name=value" URL -o output.html
962If multiple variables are to be sent, delimit them with & . Note that when & is used the
963name-value pairs should be enclosed in quotes, else the shell will consider & as a special
964character for background process. For example:
965$ curl -d "name1=val1&name2=val2" URL -o output.html
966POST data using wget
967You can POST data using wget by using -–post-data "string" . For example:
968$ wget URL –post-data "name=value" -O output.html
969Use the same format as cURL for name-value pairs.
970See also
971f A primer on cURL, explains the curl command
972f Downloading from a web page explains the wget command
973www.it-ebooks.info
9746
975The Backup Plan
976In this chapter, we will cover:
977f Archiving with tar
978f Archiving with cpio
979f Compressing with gunzip (gzip)
980f Compressing with bunzip (bzip)
981f Compressing with lzma
982f Archiving and compressing with zip
983f Heavy compression squashfs fileystem
984f Encrypting files and folders (with standard algorithms)
985f Backup snapshots with rsync
986f Version controlled backups with git
987f Cloning disks with dd
988Introduction
989Taking snapshots and backups of data are regular tasks we come across. When it comes
990to a server or large data storage systems, regular backups are important. It is possible
991to automate backups via shell scripting. Archiving and compression seems to find usage
992in the everyday life of a system admin or a regular user. There are various compression
993formats that can be used in various ways so that best results can be obtained. Encryption is
994another task that comes under frequent usage for protection of data. In order to reduce the
995size of encrypted data, usually files are archived and compressed before encrypting. Many
996standard encryption algorithms are available and it can be handled with shell utilities. This
997chapter walks through different recipes for creating and maintaining files or folder archives,
998compression formats, and encrypting techniques with shell. Let's go through the recipes.
999www.it-ebooks.info
1000The Backup Plan
1001206
1002Archiving with tar
1003The tar command can be used to archive files. It was originally designed for storing data on
1004tape archives (tar). It allows you to store multiple files and directories as a single file. It can
1005retain all the file attributes, such as owner, permissions, and so on. The file created by the tar
1006command is often referred to as a tarball.
1007Getting ready
1008The tar command comes by default with all UNIX like operating systems. It has a simple
1009syntax and is a portable file format. Let's see how to do it.
1010tar has got a list of arguments: A , c , d , r , t , u , x , f , and v . Each of these letters can be used
1011independently for different purposes corresponding to it.
1012How to do it...
1013To archive files with tar, use the following syntax:
1014$ tar -cf output.tar [SOURCES]
1015For example:
1016$ tar -cf output.tar file1 file2 file3 folder1 ..
1017In this command, -c stands for "create file" and –f stands for "specify filename".
1018We can specify folders and filenames as SOURCES . We can use a list of file names or
1019wildcards such as *.txt to specify the sources.
1020It will archive the source files into a file called output.tar .
1021The filename must appear immediately after the –f and should be the last option in the
1022argument group (for example, -cvvf filename.tar and -tvvf filename.tar ).
1023We cannot pass hundreds of files or folders as command-line arguments because there is a
1024limit. So it is safer to use the append option if many files are to be archived.
1025There's more...
1026Let's go through additional features that are available with the tar command.
1027Appending files to an archive
1028Sometimes we may need to add files to an archive that already exists (an example usage is
1029when thousands of files are to be archived and when they cannot be specified in one line as
1030command-line arguments).
1031www.it-ebooks.info
1032Chapter 6
1033207
1034Append option: -r
1035In order to append a file into an already existing archive use:
1036$ tar -rvf original.tar new_file
1037List the files in an archive as follows:
1038$ tar -tf archive.tar
1039yy/lib64/
1040yy/lib64/libfakeroot/
1041yy/sbin/
1042In order to print more details while archiving or listing, use the -v or the –vv flag. These flags
1043are called verbose ( v ), which will enable to print more details on the terminal. For example,
1044by using verbose you could print more details, such as the file permissions, owner group,
1045modification date, and so on.
1046For example:
1047$ tar -tvvf archive.tar
1048drwxr-xr-x slynux/slynux 0 2010-08-06 09:31 yy/
1049drwxr-xr-x slynux/slynux 0 2010-08-06 09:39 yy/usr/
1050drwxr-xr-x slynux/slynux 0 2010-08-06 09:31 yy/usr/lib64/
1051Extracting files and folders from an archive
1052The following command extracts the contents of the archive to the current directory:
1053$ tar -xf archive.tar
1054The -x option stands for extract.
1055When –x is used, the tar command extracts the contents of the archive to the current
1056directory. We can also specify the directory where the files need to be extracted by using the
1057–C flag, as follows:
1058$ tar -xf archive.tar -C /path/to/extraction_directory
1059The command extracts the contents of an archive to insert image a specified directory. It
1060extracts the entire contents of the archive. We can also extract only a few files by specifying
1061them as command arguments:
1062$ tar -xvf file.tar file1 file4
1063The command above extracts only file1 and file4 , and ignores other files in the archive.
1064www.it-ebooks.info
1065The Backup Plan
1066208
1067stdin and stdout with tar
1068While archiving, we can specify stdout as the output file so that another command appearing
1069through a pipe can read it as stdin and then do some process or extract the archive.
1070This is helpful in order to transfer data through a Secure Shell (SSH) connection (while on a
1071network). For example:
1072$ mkdir ~/destination
1073$ tar -cf - file1 file2 file3 | tar -xvf - -C ~/destination
1074In the example above, file1 , file2 , and file3 are combined into a tarball and then
1075extracted to ~/destination . In this command:
1076f -f specifies stdout as the file for archiving (when the -c option used)
1077f -f specifies stdin as the file for extracting (when the -x option used)
1078Concatenating two archives
1079We can easily merge multiple tar files with the -A option.
1080Let's pretend we have two tarballs: file1.tar and file2.tar . We can merge the contents
1081of file2.tar to file1.tar as follows:
1082$ tar -Af file1.tar file2.tar
1083Verify it by listing the contents:
1084$ tar -tvf file1.tar
1085Updating files in an archive with timestamp check
1086The append option appends any given file to the archive. If the same file is inside the archive
1087is given to append, it will append that file and the archive will contain duplicates. We can
1088use the update option -u to specify only append files that are newer than the file inside the
1089archive with the same name.
1090$ tar -tf archive.tar
1091filea
1092fileb
1093filec
1094This command lists the files in the archive.
1095In order to append filea only if filea has newer modification time than filea inside
1096archive.tar , use:
1097$ tar -uvvf archive.tar filea
1098www.it-ebooks.info
1099Chapter 6
1100209
1101Nothing happens if the version of filea outside the archive and the filea inside
1102archive.tar have the same timestamp.
1103Use the touch command to modify the file timestamp and then try the tar command again:
1104$ tar -uvvf archive.tar filea
1105-rw-r--r-- slynux/slynux 0 2010-08-14 17:53 filea
1106The file is appended since its timestamp is newer than the one inside the archive.
1107Comparing files in archive and file system
1108Sometimes it is useful to know whether a file in the archive and a file with the same filename
1109in the filesystem are the same or contain any differences. The –d flag can be used to print the
1110differences:
1111$ tar -df archive.tar filename1 filename2 ...
1112For example:
1113$ tar -df archive.tar afile bfile
1114afile: Mod time differs
1115afile: Size differs
1116Deleting files from archive
1117We can remove files from a given archive using the –delete option. For example:
1118$ tar -f archive.tar --delete file1 file2 ..
1119Let's see another example:
1120$ tar -tf archive.tar
1121filea
1122fileb
1123filec
1124Or, we can also use the following syntax:
1125$ tar --delete --file archive.tar [FILE LIST]
1126For example:
1127$ tar --delete --file archive.tar filea
1128$ tar -tf archive.tar
1129fileb
1130filec
1131www.it-ebooks.info
1132The Backup Plan
1133210
1134Compression with tar archive
1135The tar command only archives files, it does not compress them. For this reason, most people
1136usually add some form of compression when working with tarballs. This significantly decreases
1137the size of the files. Tarballs are often compressed into one of the following formats:
1138f file.tar.gz
1139f file.tar.bz2
1140f file.tar.lzma
1141f file.tar.lzo
1142Different tar flags are used to specify different compression formats.
1143f -j for bunzip2
1144f -z for gzip
1145f --lzma for lzma
1146They are explained in the following compression-specific recipes.
1147It is possible to use compression formats without explicitly specifying special options as
1148above. tar can compress by looking at the given extension of the output or input file names.
1149In order for tar to support compression automatically by looking at the extensions, use -a or
1150--auto-compress with tar .
1151Excluding a set of files from archiving
1152It is possible to exclude a set of files from archiving by specifying patterns. Use
1153--exclude [PATTERN] for excluding files matched by wildcard patterns.
1154For example, to exclude all .txt files from archiving use:
1155$ tar -cf arch.tar * --exclude "*.txt"
1156Note that the pattern should be enclosed in double quotes.
1157It is also possible to exclude a list of files provided in a list file with the -X flag as follows:
1158$ cat list
1159filea
1160fileb
1161$ tar -cf arch.tar * -X list
1162Now it excludes filea and fileb from archiving.
1163www.it-ebooks.info
1164Chapter 6
1165211
1166Excluding version control directories
1167We usually use tarballs for distributing source code. Most of the source code is maintained
1168using version control systems such as subversion, Git, mercurial, cvs, and so on. Code
1169directories under version control will contain special directories used to manage versions like
1170.svn or .git . However, these directories aren't needed by the code itself and so should be
1171eliminated from the tarball of the source code.
1172In order to exclude version control related files and directories while archiving use the
1173--exclude-vcs option along with tar . For example:
1174$ tar --exclude-vcs -czvvf source_code.tar.gz eye_of_gnome_svn
1175Printing total bytes
1176It is sometimes useful if we can print total bytes copied to the archive. Print the total bytes
1177copied after archiving by using the -- totals option as follows:
1178$ tar -cf arc.tar * --exclude "*.txt" --totals
1179Total bytes written: 20480 (20KiB, 12MiB/s)
1180See also
1181f Compressing with gunzip (gzip), explains the gzip command
1182f Compressing with bunzip (bzip2), explains the bzip2 command
1183f Compressing with lzma, explains the lzma command
1184Archiving with cpio
1185cpio is another archiving format similar to tar . It is used to store files and directories in a file
1186with attributes such as permissions, ownership, and so on. But it is not commonly used as
1187much as tar . However, cpio seems to be used in RPM package archives, initramfs files for
1188the Linux kernel, and so on. This recipe will give minimal usage examples of cpio .
1189How to do it...
1190cpio takes input filenames through stdin and it writes the archive into stdout . We have to
1191redirect stdout to a file to receive the output cpio file as follows:
1192Create test files:
1193$ touch file1 file2 file3
1194We can archive the test files as follows:
1195$ echo file1 file2 file3 | cpio -ov > archive.cpio
1196www.it-ebooks.info
1197The Backup Plan
1198212
1199In this command:
1200f -o specifies the output
1201f -v is used for printing a list of files archived
1202By using cpio, we can also archive using files as absolute paths. /usr/
1203somedir is an absolute path as it contains the full path starting from root (/).
1204A relative path will not start with / but it starts the path from the current
1205directory. For example, test/file means that there is a directory test and
1206the file is inside the test directory.
1207While extracting, cpio extracts to the absolute path itself. But incase of tar it
1208removes the / in the absolute path and converts it as relative path.
1209In order to list files in a cpio archive use the following command:
1210$ cpio -it < archive.cpio
1211This command will list all the files in the given cpio archive. It reads the files from stdin .
1212In this command:
1213f -i is for specifying the input
1214f -t is for listing
1215In order to extract files from the cpio archive use:
1216$ cpio -id < archive.cpio
1217Here, -d is used for extracting.
1218It overwrites files without prompting. If the absolute path files are present in the archive, it will
1219replace the files at that path. It will not extract files in the current directory like tar .
1220Compressing with gunzip (gzip)
1221gzip is a commonly used compression format in GNU/Linux platforms. Utilities such as gzip ,
1222gunzip , and zcat are available to handle gzip compression file types. gzip can be applied
1223on a file only. It cannot archive directories and multiple files. Hence we use a tar archive
1224and compress it with gzip . When multiple files are given as input it will produce several
1225individually compressed ( .gz ) files. Let's see how to operate with gzip .
1226How to do it...
1227In order to compress a file with gzip use the following command:
1228$ gzip filename
1229www.it-ebooks.info
1230Chapter 6
1231213
1232$ ls
1233filename.gz
1234Then it will remove the file and produce a compressed file called filename.gz .
1235Extract a gzip compressed file as follows:
1236$ gunzip filename.gz
1237It will remove filename.gz and produce an uncompressed version of filename.gz .
1238In order to list out the properties of a compressed file use:
1239$ gzip -l test.txt.gz
1240compressed uncompressed ratio uncompressed_name
124135 6 -33.3% test.txt
1242The gzip command can read a file from stdin and also write a compressed file into
1243stdout .
1244Read from stdin and out as stdout as follows:
1245$ cat file | gzip -c > file.gz
1246The -c option is used to specify output to stdout .
1247We can specify the compression level for gzip . Use --fast or the --best option to provide
1248low and high compression ratios, respectively.
1249There's more...
1250The gzip command is often used with other commands. It also has advanced options to
1251specify the compression ratio. Let's see how to work with these features.
1252Gzip with tarball
1253We usually use gzip with tarballs. A tarball can be compressed by using the –z option passed
1254to the tar command while archiving and extracting.
1255You can create gzipped tarballs using the following methods:
1256f Method - 1
1257$ tar -czvvf archive.tar.gz [FILES]
1258Or:
1259$ tar -cavvf archive.tar.gz [FILES]
1260The -a option specifies that the compression format should automatically be
1261detected from the extension.
1262www.it-ebooks.info
1263The Backup Plan
1264214
1265f Method - 2
1266First, create a tarball:
1267$ tar -cvvf archive.tar [FILES]
1268Compress it after tarballing as follows:
1269$ gzip archive.tar
1270If many files (a few hundreds) are to be archived in a tarball and need to be compressed, we
1271use Method - 2 with few changes. The issue with giving many files as command arguments
1272to tar is that it can accept only a limited number of files from the command line. In order
1273to solve this issue, we can create a tar file by adding files one by one using a loop with an
1274append option ( -r ) as follows:
1275FILE_LIST="file1 file2 file3 file4 file5"
1276for f in $FILE_LIST;
1277do
1278tar -rvf archive.tar $f
1279done
1280gzip archive.tar
1281In order to extract a gzipped tarball, use the following:
1282f -x for extraction
1283f -z for gzip specification
1284Or:
1285$ tar -xavvf archive.tar.gz -C extract_directory
1286In the above command, the -a option is used to detect the compression format automatically.
1287zcat – reading gzipped files without extracting
1288zcat is a command that can be used to dump an extracted file from a .gz file to stdout
1289without manually extracting it. The .gz file remains as before but it will dump the extracted
1290file into stdout as follows:
1291$ ls
1292test.gz
1293$ zcat test.gz
1294A test file
1295# file test contains a line "A test file"
1296$ ls
1297test.gz
1298www.it-ebooks.info
1299Chapter 6
1300215
1301Compression ratio
1302We can specify compression ratio, which is available in range 1 to 9, where:
1303f 1 is the lowest, but fastest
1304f 9 is the best, but slowest
1305You can also specify the ratios in between as follows:
1306$ gzip -9 test.img
1307This will compress the file to the maximum.
1308See also
1309f Archiving with tar, explains the tar command
1310Compressing with bunzip (bzip)
1311bunzip2 is another compression technique which is very similar to gzip . bzip2 typically
1312produces smaller (more compressed) files than gzip . It comes with all Linux distributions.
1313Let's see how to use bzip2 .
1314How to do it...
1315In order to compress with bzip2 use:
1316$ bzip2 filename
1317$ ls
1318filename.bz2
1319Then it will remove the file and produce a compressed file called filename.bzip2 .
1320Extract a bzipped file as follows:
1321$ bunzip2 filename.bz2
1322It will remove filename.bz2 and produce an uncompressed version of filename .
1323bzip2 can read a file from stdin and also write a compressed file into stdout .
1324In order to read from stdin and read out as stdout use:
1325$ cat file | bzip2 -c > file.tar.bz2
1326-c is used to specify output to stdout .
1327www.it-ebooks.info
1328The Backup Plan
1329216
1330We usually use bzip2 with tarballs. A tarball can be compressed by using the -j option
1331passed to the tar command while archiving and extracting.
1332Creating a bzipped tarball can be done by using the following methods:
1333f Method - 1
1334$ tar -cjvvf archive.tar.bz2 [FILES]
1335Or:
1336$ tar -cavvf archive.tar.bz2 [FILES]
1337The -a option specifies to automatically detect compression format from the extension.
1338f Method - 2
1339First create the tarball:
1340$ tar -cvvf archive.tar [FILES]
1341Compress it after tarballing:
1342$ bzip2 archive.tar
1343If we need to add hundreds of files to the archive, the above commands may fail. To fix that
1344issue, use a loop to append files to the archive one by one using the –r option. See the similar
1345section from the recipe, Compressing with gunzip (gzip).
1346Extract a bzipped tarball as follows:
1347$ tar -xjvvf archive.tar.bz2 -C extract_directory
1348In this command:
1349f -x is used for extraction
1350f -j is for bzip2 specification
1351f -C is for specifying the directory to which the files are to be extracted
1352Or, you can use the following command:
1353$ tar -xavvf archive.tar.bz2 -C extract_directory
1354-a will automatically detect the compression format.
1355There's more...
1356bunzip has several additional options to carry out different functions. Let's go through few
1357of them.
1358Keeping input files without removing them
1359While using bzip2 or bunzip2 , it will remove the input file and produce a compressed output
1360file. But we can prevent it from removing input files by using the –k option.
1361www.it-ebooks.info
1362Chapter 6
1363217
1364For example:
1365$ bunzip2 test.bz2 -k
1366$ ls
1367test test.bz2
1368Compression ratio
1369We can specify the compression ratio, which is available in the range of 1 to 9 (where 1 is the
1370least compression, but fast, and 9 is the highest possible compression but much slower).
1371For example:
1372$ bzip2 -9 test.img
1373This command provides maximum compression.
1374See also
1375f Archiving with tar, explains the tar command
1376Compressing with lzma
1377lzma is comparatively new when compared to gzip or bzip2 . lzma offers better
1378compression rates than gzip or bzip2 . As lzma is not preinstalled on most Linux distros,
1379you may need to install it using the package manager.
1380How to do it...
1381In order to compress with lzma use the following command:
1382$ lzma filename
1383$ ls
1384filename.lzma
1385This will remove the file and produce a compressed file called filename.lzma .
1386To extract an lzma file use:
1387$ unlzma filename.lzma
1388This will remove filename.lzma and produce an uncompressed version of the file.
1389The lzma command can also read a file from stdin and write the compressed file to stdout .
1390www.it-ebooks.info
1391The Backup Plan
1392218
1393In order to read from stdin and read out as stdout use:
1394$ cat file | lzma -c > file.lzma
1395-c is used to specify output to stdout .
1396We usually use lzma with tarballs. A tarball can be compressed by using the --lzma option
1397passed to the tar command while archiving and extracting.
1398There are two methods to create a lzma tarball:
1399f Method - 1
1400$ tar -cvvf --lzma archive.tar.lzma [FILES]
1401Or:
1402$ tar -cavvf archive.tar.lzma [FILES]
1403The -a option specifies to automatically detect the compression format from the
1404extension.
1405f Method - 2
1406First, create the tarball:
1407$ tar -cvvf archive.tar [FILES]
1408Compress it after tarballing:
1409$ lzma archive.tar
1410If we need to add hundreds of files to the archive, the above commands may fail. To fix that
1411issue, use a loop to append files to the archive one by one using the –r option. See the
1412similar section from the recipe, Compressing with gunzip (gzip).
1413There's more...
1414Let's go through additional options associated with lzma utilities
1415Extracting an lzma tarball
1416In order to extract a tarball compressed with lzma compression to a specified directory, use:
1417$ tar -xvvf --lzma archive.tar.lzma -C extract_directory
1418In this command, -x is used for extraction. --lzma specifies the use of lzma to
1419decompress the resulting file.
1420Or, we could also use:
1421$ tar -xavvf archive.tar.lzma -C extract_directory
1422The -a option specifies to automatically detect the compression format from the extension.
1423www.it-ebooks.info
1424Chapter 6
1425219
1426Keeping input files without removing them
1427While using lzma or unlzma , it will remove the input file and produce an output file. But we
1428can prevent from removing input files and keep them by using the -k option. For example:
1429$ lzma test.bz2 -k
1430$ ls
1431test.bz2.lzma
1432Compression ratio
1433We can specify the compression ratio, which is available in the range of 1 to 9 (where 1 is the
1434least compression, but fast, and 9 is the highest possible compression but much slower).
1435You can also specify ratios in between as follows:
1436$ lzma -9 test.img
1437This command compresses the file to the maximum.
1438See also
1439f Archiving with tar, explains the tar command
1440Archiving and compressing with zip
1441ZIP is a popular compression format used on many platforms. It isn't as commonly used as
1442gzip or bzip2 on Linux platforms, but files from the Internet are often saved in this format.
1443How to do it...
1444In order to archive with ZIP, the following syntax is used:
1445$ zip archive_name.zip [SOURCE FILES/DIRS]
1446For example:
1447$ zip file.zip file
1448Here, the file.zip file will be produced.
1449Archive directories and files recursively as follows:
1450$ zip -r archive.zip folder1 file2
1451In this command, -r is used for specifying recursive.
1452www.it-ebooks.info
1453The Backup Plan
1454220
1455Unlike lzma , gzip , or bzip2 , zip won't remove the source file after archiving. zip is similar
1456to tar in that respect, but zip can compress files where tar does not. However, zip adds
1457compression too.
1458In order to extract files and folders in a ZIP file, use:
1459$ unzip file.zip
1460It will extract the files without removing filename.zip (unlike unlzma or gunzip ).
1461In order to update files in the archive with newer files in the filesystem, use the -u flag:
1462$ zip file.zip -u newfile
1463Delete a file from a zipped archive, by using –d as follows:
1464$ zip -d arc.zip file.txt
1465In order to list the files in an archive use:
1466$ unzip -l archive.zip
1467squashfs – the heavy compression filesystem
1468squashfs is a heavy-compression based read-only filesystem that is capable of compressing
14692 to 3GB of data onto a 700 MB file. Have you ever thought of how Linux Live CDs work?
1470When a Live CD is booted it loads a complete Linux environment. Linux Live CDs make use
1471of a read-only compressed filesystem called squashfs. It keeps the root filesystem on a
1472compressed filesystem file. It can be loopback mounted and files can be accessed. Thus when
1473some files are required by processes, they are decompressed and loaded onto the RAM and
1474used. Knowledge of squashfs can be useful when building a custom live OS or when required
1475to keep files heavily compressed and to access them without entirely extracting the files.
1476For extracting a large compressed file, it will take a long time. However, if a file is loopback
1477mounted, it will be very fast since the required portion of the compressed files are only
1478decompressed when the request for files appear. In regular decompression, all the data is
1479decompressed first. Let's see how we can use squashfs.
1480Getting ready
1481If you have an Ubuntu CD just locate a .squashfs file at CDRom ROOT/casper/
1482filesystem.squashfs . squashfs internally uses compression algorithms such as gzip
1483and lzma . squashfs support is available in all of the latest Linux distros. However, in order
1484to create squashfs files, an additional package squashfs-tools needs to be installed from
1485package manager.
1486www.it-ebooks.info
1487Chapter 6
1488221
1489How to do it...
1490In order to create a squashfs file by adding source directories and files, use:
1491$ mksquashfs SOURCES compressedfs.squashfs
1492Sources can be wildcards, or file, or folder paths.
1493For example:
1494$ sudo mksquashfs /etc test.squashfs
1495Parallel mksquashfs: Using 2 processors
1496Creating 4.0 filesystem on test.squashfs, block size 131072.
1497[=======================================] 1867/1867 100%
1498More details will be printed on terminal. They are limited to save space
1499In order to mount the squashfs file to a mount point, use loopback mounting as follows:
1500# mkdir /mnt/squash
1501# mount -o loop compressedfs.squashfs /mnt/squash
1502You can copy contents by accessing /mnt/squashfs .
1503There's more...
1504The squashfs file system can be created by specifying additional parameters. Let's go
1505through the additional options.
1506Excluding files while creating a squashfs file
1507While creating a squashfs file, we can exclude a list of files or a file pattern specified using
1508wildcards.
1509Exclude a list of files specified as command-line arguments by using the -e option. For
1510example:
1511$ sudo mksquashfs /etc test.squashfs -e /etc/passwd /etc/shadow
1512The –e option is used to exclude passwd and shadow files.
1513It is also possible to specify a list of exclude files given in a file with –ef as follows:
1514$ cat excludelist
1515/etc/passwd
1516/etc/shadow
1517$ sudo mksquashfs /etc test.squashfs -ef excludelist
1518If we want to support wildcards in excludes lists, use -wildcard as an argument.
1519www.it-ebooks.info
1520The Backup Plan
1521222
1522Cryptographic tools and hashes
1523Encryption techniques are used mainly to protect data from unauthorized access. There are
1524many algorithms available and we use a common set of standard algorithms. There are a few
1525tools available in a Linux environment for performing encryption and decryption. Sometimes
1526we use encryption algorithm hashes for verifying data integrity. This section will introduce a few
1527commonly-used cryptographic tools and a general set of algorithms that these tools can handle.
1528How to do it...
1529Let's see how to use the tools such as crypt, gpg, base64, md5sum, sha1sum, and openssl:
1530f crypt
1531The crypt command is a simple cryptographic utility, which takes a file from stdin
1532and a passphrase as input and outputs encrypted data into stdout .
1533$ crypt <input_file> output_file
1534Enter passphrase:
1535It will interactively ask for a passphrase. We can also provide a passphrase through
1536command-line arguments.
1537$ crypt PASSPHRASE < input_file > encrypted_file
1538In order to decrypt the file use:
1539$ crypt PASSPHRASE -d < encrypted_file > output_file
1540f gpg (GNU privacy guard)
1541gpg (GNU privacy guard) is a widely-used encryption scheme used for protecting files
1542with key signing techniques that enables to access data by authentic destination only.
1543gpg signatures are very famous. The details of gpg are outside the scope of this book.
1544Here we can learn how to encrypt and decrypt a file.
1545In order to encrypt a file with gpg use:
1546$ gpg -c filename
1547This command reads the passphrase interactively and generates filename.gpg .
1548In order to decrypt a gpg file use:
1549$ gpg filename.gpg
1550This command reads a passphrase and decrypts the file.
1551f Base64
1552Base64 is a group of similar encoding schemes that represents binary data in an
1553ASCII string format by translating it into a radix-64 representation. The base64
1554command can be used to encode and decode the Base64 string.
1555www.it-ebooks.info
1556Chapter 6
1557223
1558In order to encode a binary file into Base64 format, use:
1559$ base64 filename > outputfile
1560Or:
1561$ cat file | base64 > outputfile
1562It can read from stdin .
1563Decode Base64 data as follows:
1564$ base64 -d file > outputfile
1565Or:
1566$ cat base64_file | base64 -d > outputfile
1567f md5sum and sha1sum
1568md5sum and sha1sum are unidirectional hash algorithms, which cannot be reversed
1569to form the original data. These are usually used to verify the integrity of data or for
1570generating a unique key from a given data. For every file it generates a unique key by
1571analyzing its content.
1572$ md5sum file
15738503063d5488c3080d4800ff50850dc9 file
1574$ sha1sum file
15751ba02b66e2e557fede8f61b7df282cd0a27b816b file
1576These types of hashes are ideal for storing passwords. Passwords are stored as its
1577hashes. When a user wants to authenticate, the password is read and converted to
1578the hash. Then hash is compared to the one that is stored already. If they are same,
1579the password is authenticated and access is provided, else it is denied. Storing
1580original password strings is risky and poses a security risk of exposing the password.
1581f Shadowlike hash (salted hash)
1582Let's see how to generate shadow like salted hash for passwords.
1583The user passwords in Linux are stored as its hashes in the /etc/shadow file. A
1584typical line in /etc/shadow will look like this:
1585test:$6$fG4eWdUi$ohTKOlEUzNk77.4S8MrYe07NTRV4M3LrJnZP9p.qc1bR5c.
1586EcOruzPXfEu1uloBFUa18ENRH7F70zhodas3cR.:14790:0:99999:7:::
1587In this line $6$fG4eWdUi$ohTKOlEUzNk77.4S8MrYe07NTRV4M3LrJnZP9p.
1588qc1bR5c.EcOruzPXfEu1uloBFUa18ENRH7F70zhodas3cR is the shadow hash
1589corresponding to its password.
1590In some situations, we may need to write critical administration scripts that may need
1591to edit passwords or add users manually using a shell script. In that case we have to
1592generate a shadow password string and write a similar line as above to the shadow
1593file. Let's see how to generate a shadow password using openssl .
1594www.it-ebooks.info
1595The Backup Plan
1596224
1597Shadow passwords are usually salted passwords. SALT is an extra string used to
1598obfuscate and make the encryption stronger. The salt consists of random bits that are
1599used as one of the inputs to a key derivation function that generates the salted hash
1600for the password.
1601For more details on salt, see the Wikipedia page http://en.wikipedia.org/
1602wiki/Salt_(cryptography) .
1603$ openssl passwd -1 -salt SALT_STRING PASSWORD
1604$1$SALT_STRING$323VkWkSLHuhbt1zkSsUG.
1605Replace SALT_STRING with a random string and PASSWORD with the password you
1606want to use.
1607Backup snapshots with rsync
1608Backing up data is something that most sysadmins need to do regularly. We may need to
1609backup data in a web server or from remote locations. rsync is a command that can be
1610used to synchronize files and directories from one location to another while minimizing data
1611transfer using file difference calculations and compression. The advantage of rsync over the
1612cp command is that rsync uses strong difference algorithms. Also, it supports data transfer
1613across networks. While making copies, it compares the files in the original and destination
1614locations and will only copy the files that are newer. It also supports compression, encryption,
1615and a lot more. Let's see how we can work with rsync .
1616How to do it...
1617In order to copy a source directory to a destination (to create a mirror) use:
1618$ rsync -av source_path destination_path
1619In this command:
1620f -a stands for archiving
1621f -v (verbose) prints the details or progress on stdout
1622The above command will recursively copy all the files from the source path to the destination
1623path. We can specify paths as remote or localhost paths.
1624It can be in the format /home/slynux/data , slynux@192.168.0.6:/home/backups/
1625data , and so on.
1626/home/slynux/data specifies the absolute path in the machine in which the rsync
1627command is executed. slynux@192.168.0.6:/home/backups/data specifies that the
1628path is /home/backups/data in the machine with IP address 192.168.0.6 and is logged
1629in as user slynux .
1630www.it-ebooks.info
1631Chapter 6
1632225
1633In order to back up data to a remote server or host, use:
1634$ rsync -av source_dir username@host:PATH
1635To keep a mirror at the destination, run the same rsync command scheduled at regular
1636intervals. It will copy only changed files to the destination.
1637Restore the data from remote host to localhost as follows:
1638$ rsync -av username@host:PATH destination
1639The rsync command uses SSH to connect to another remote machine. Provide the remote
1640machine address in the format user@host , where user is the username and host is the IP
1641address or domain name attached to the remote machine. PATH is the absolute path address
1642where the data needs to be copied. rsync will ask for the user password as usual for SSH
1643logic. This can be automated (avoid user password probing) by using SSH keys.
1644Make sure that the OpenSSH is installed and running on the remote machine.
1645Compressing data while transferring through the network can significantly optimize the
1646speed of the transfer. We can use the rsync option –z to specify to compress data while
1647transferring through a network. For example:
1648$ rsync -avz source destination
1649For the PATH format, if we use / at the end of the source, rsync will copy
1650contents of that end directory specified in the source_path to the destination.
1651If / not at the end of the source, rsync will copy that end directory itself to the
1652destination.
1653For example, the following command copies the content of the test directory:
1654$ rsync -av /home/test/ /home/backups
1655The following command copies the test directory to the destination:
1656$ rsync -av /home/test /home/backups
1657If / is at the end of destination_path, rsync will copy the source to the
1658destination directory.
1659If / is not used at the end of the destination path, rsync will create a folder,
1660named similar to the source directory, at the end of the destination path and
1661copy the source into that directory.
1662For example:
1663$ rsync -av /home/test /home/backups/
1664www.it-ebooks.info
1665The Backup Plan
1666226
1667This command copies the source ( /home/test ) to an existing folder called backups .
1668$ rsync -av /home/test /home/backups
1669This command copies the source ( /home/test ) to a directory named backups by creating
1670that directory.
1671There's more...
1672The rsync command has several additional functionalities that can be specified using its
1673command-line options. Let's go through them.
1674Excluding files while archiving with rsync
1675Some files need not be updated while archiving to a remote location. It is possible to tell rsync
1676to exclude certain files from the current operation. Files can be excluded by two options:
1677--exclude PATTERN
1678We can specify a wildcard pattern of files to be excluded. For example:
1679$ rsync -avz /home/code/some_code /mnt/disk/backup/code --exclude "*.txt"
1680This command excludes .txt files from backing up.
1681Or, we can specify a list of files to be excluded by providing a list file.
1682Use --exclude-from FILEPATH .
1683Deleting non-existent files while updating rsync backup
1684We archive files as tarball and transfer the tarball to the remote backup location. When we
1685need to update the backup data, we create a TAR file again and transfer the file to the backup
1686location. By default, rsync does not remove files from the destination if they no longer exist
1687at the source. In order to remove the files from the destination that do not exist at the source,
1688use the rsync --delete option:
1689$ rsync -avz SOURCE DESTINATION --delete
1690Scheduling backups at intervals
1691You can create a cron job to schedule backups at regular intervals.
1692A sample is as follows:
1693$ crontab -e
1694Add the following line:
16950 */10 * * * rsync -avz /home/code user@IP_ADDRESS:/home/backups
1696The above crontab entry schedules the rsync to be executed every 10 hours.
1697www.it-ebooks.info
1698Chapter 6
1699227
1700*/10 is the hour position of the crontab syntax. /10 specifies to execute the backup every
170110 hours. If */10 is written in the minutes position, it will execute every 10 minutes.
1702Have a look at the Scheduling with cron recipe in Chapter 9 to understand how to configure
1703crontab .
1704Version control based backup with Git
1705People use different strategies in backing up data. Differential backups are more efficient
1706than making copies of the entire source directory to a target the backup directory with the
1707version number using date or time of a day. It causes wastage of space. We only need to
1708copy the changes that occurred to files from the second time that the backups occur. This is
1709called incremental backups. We can manually create incremental backups using tools like
1710rsync . But restoring this sort of backup can be difficult. The best way to maintain and restore
1711changes is to use version control systems. They are very much used in software development
1712and maintenance of code, since coding frequently undergoes changes. Git (GNU it) is a very
1713famous and is the most efficient version control systems available. Let's use Git for backup
1714of regular files in non-programming context. Git can be installed by your distro's package
1715manager. It was written by Linus Torvalds.
1716Getting ready
1717Here is the problem statement:
1718We have a directory that contains several files and subdirectories. We need to keep track of
1719changes occurring to the directory contents and back them up. If data becomes corrupted or
1720goes missing, we must be able to restore a previous copy of that data. We need to backup the
1721data at regular intervals to a remote machine. We also need to take the backup at different
1722locations in the same machine (localhost). Let's see how to implement it using Git.
1723How to do it...
1724In the directory which is to be backed up use:
1725$ cd /home/data/source
1726Let it be the directory source to be tracked.
1727Set up and initiate the remote backup directory. In the remote machine, create the backup
1728destination directory:
1729$ mkdir -p /home/backups/backup.git
1730$ cd /home/backups/backup.git
1731$ git init --bare
1732www.it-ebooks.info
1733The Backup Plan
1734228
1735The following steps are to be performed in the source host machine:
17361. Add user details to Git in the source host machine:
1737$ git config --global user.name "Sarath Lakshman"
1738#Set user name to "Sarath Lakshman"
1739$ git config --global user.email slynux@slynux.com
1740# Set email to slynux@slynux.com
1741Initiate the source directory to backup from the host machine. In the source directory in
1742the host machine whose files are to be backed up, execute the following commands:
1743$ git init
1744Initialized empty Git repository in /home/backups/backup.git/
1745# Initialize git repository
1746$ git commit --allow-empty -am "Init"
1747[master (root-commit) b595488] Init
17482. In the source directory, execute the following command to add the remote git
1749directory and synchronize backup:
1750$ git remote add origin user@remotehost:/home/backups/backup.git
1751$ git push origin master
1752Counting objects: 2, done.
1753Writing objects: 100% (2/2), 153 bytes, done.
1754Total 2 (delta 0), reused 0 (delta 0)
1755To user@remotehost:/home/backups/backup.git
1756* [new branch] master -> master
17573. Add or remove files for Git tracking.
1758The following command adds all files and folders in the current directory to the
1759backup list:
1760$ git add *
1761We can conditionally add certain files only to the backup list as follows:
1762$ git add *.txt
1763$ git add *.py
1764We can remove the files and folders not required to be tracked by using:
1765$ git rm file
1766It can be a folder or even a wildcard as follows:
1767$ git rm *.txt
1768www.it-ebooks.info
1769Chapter 6
1770229
17714. Check-pointing or marking backup points.
1772We can mark checkpoints for the backup with a message using the following
1773command:
1774$ git commit -m "Commit Message"
1775We need to update the backup at the remote location at regular intervals. Hence, set
1776up a cron job (for example, backing up every five hours).
1777Create a file crontab entry with lines:
17780 */5 * * * /home/data/backup.sh
1779Create a script /home/data/backup.sh as follows:
1780#!/bin/ bash
1781cd /home/data/source
1782git add .
1783git commit -am "Commit - @ $(date)"
1784git push
1785Now we have set up the backup system.
17865. Restoring data with Git.
1787In order to view all backup versions use:
1788$ git log
1789Update the current directory to the last backup by ignoring any recent changes.
1790‰ To revert back to any previous state or version, look into the commit ID,
1791which is a 32-character hex string. Use the commit ID with git checkout .
1792‰ For commit ID 3131f9661ec1739f72c213ec5769bc0abefa85a9 it will be:
1793$ git checkout 3131f9661ec1739f72c213ec5769bc0abefa85a9
1794$ git commit -am "Restore @ $(date) commit ID:
17953131f9661ec1739f72c213ec5769bc0abefa85a9"
1796$ git push
1797‰ In order to view the details about versions again, use:
1798$ git log
1799If the working directory is broken due to some issues, we need to fix the directory with
1800the backup at the remote location.
1801Then we can recreate the contents from the backup at the remote location as follows:
1802$ git clone user@remotehost:/home/backups/backup.git
1803This will create a directory backup with all contents.
1804www.it-ebooks.info
1805The Backup Plan
1806230
1807Cloning hard drive and disks with dd
1808While working with hard drives and partitions, we may need to create copies or make backups
1809of full partitions rather than copying all contents (not only hard disk partitions but also copy an
1810entire hard disk without missing any information, such as boot record, partition table, and so
1811on). In this situation we can use the dd command. It can be used to clone any type of disks,
1812such as hard disks, flash drives, CDs, DVDs, floppy disks, and so on.
1813Getting ready
1814The dd command expands to Data Definition. Since its improper usage leads to loss of data,
1815it is nicknamed as "Data Destroyer". Be careful while using the order of arguments. Wrong
1816arguments can lead to loss of entire data or can become useless. dd is basically a bitstream
1817duplicator that writes the entire bit stream from a disk to a file or a file to a disk. Let's see how
1818to use dd .
1819How to do it...
1820The syntax for dd is as follows:
1821$ dd if=SOURCE of=TARGET bs=BLOCK_SIZE count=COUNT
1822In this command:
1823f if stands for input file or input device path
1824f of stands for target file or target device path
1825f bs stands for block size (usually, it is given in the power of 2, for example, 512, 1024,
18262048, and so on). COUNT is the number of blocks to be copied (an integer).
1827Total bytes copied = BLOCK_SIZE * COUNT
1828bs and count are optional.
1829By specifying COUNT we can limit the number of bytes to be copied from input file to target. If
1830COUNT is not specified, dd will copy from input file until it reaches the end of file (EOF) marker.
1831In order to copy a partition into a file use:
1832# dd if=/dev/sda1 of=sda1_partition.img
1833Here /dev/sda1 is the device path for the partition.
1834Restore the partition using the backup as follows:
1835# dd if=sda1_partition.img of=/dev/sda1
1836You should be careful about the argument if and of . Improper usage may lead to data loss.
1837www.it-ebooks.info
1838Chapter 6
1839231
1840By changing the device path /dev/sda1 to the appropriate device path, any disk can be
1841copied or restored.
1842In order to permanently delete all of the data in a partition, we can make dd to write zeros into
1843the partition by using the following command:
1844# dd if=/dev/zero of=/dev/sda1
1845/dev/zero is a character device. It always returns infinite zero '\0' characters.
1846Clone one hard disk to another hard disk of the same size as follows:
1847# dd if=/dev/sda of=/dev/sdb
1848Here /dev/sdb is the second hard disk.
1849In order to take the image of a CD ROM (ISO file) use:
1850# dd if=/dev/cdrom of=cdrom.iso
1851There's more...
1852When a file system is created in a file which is generated using dd , we can mount it to a
1853mount point. Let's see how to work with it.
1854Mounting image files
1855Any file image created using dd can be mounted using the loopback method. Use the -o
1856loop with the mount command.
1857# mkdir /mnt/mount_point
1858# mount -o loop file.img /mnt/mount_point
1859Now we can access the contents of the image files through the location /mnt/mount_point .
1860See also
1861f Creating ISO files, Hybrid ISO of Chapter 3, explains how to use dd to create an ISO
1862file from a CD
1863www.it-ebooks.info
1864www.it-ebooks.info
18657
1866The Old-boy Network
1867In this chapter, we will cover:
1868f Basic networking primer
1869f Let's ping!
1870f Listing all the machines alive on a network
1871f Transferring files through network
1872f Setting up an Ethernet and wireless LAN with script
1873f Password-less auto-login with SSH
1874f Running commands on remote host with SSH
1875f Mounting remote drive at local mount point
1876f Multi-casting window messages on a network
1877f Network traffic and port analysis
1878Introduction
1879Networking is the act of interconnecting machines through a network and configuring the
1880nodes in the network with different specifications. We use TCP/IP as our networking stack
1881and all operations are based on it. Networks are an important part of every computer system.
1882Each node connected in the network is assigned a unique IP address for identification. There
1883are many parameters in networking, such as subnet mask, route, ports, DNS, and so on,
1884which require a basic understanding to follow.
1885www.it-ebooks.info
1886The Old-boy Network
1887234
1888Several applications that make use of a network operate by opening and connecting to
1889firewall ports. Every application may offer services such as data transfer, remote shell login,
1890and so on. Several interesting management tasks can be performed on a network consisting
1891of many machines. Shell scripts can be used to configure the nodes in a network, test the
1892availability of machines, automate execution of commands at remote hosts, and so on. This
1893chapter focuses on different recipes that introduce interesting tools or commands related to
1894networking and also how they can be used for solving different problems.
1895Basic networking primer
1896Before digging through recipes based on networking, it is essential for you to have a basic
1897understanding of setting up a network, the terminology and commands for assigning an IP
1898address, adding routes, and so on. This recipe will give an overview of different commands
1899used in GNU/Linux for networking and their usages from the basics.
1900Getting ready
1901Every node in a network requires many parameters to be assigned to work successfully and
1902interconnect with other machines. Some of the different parameters are the IP address,
1903subnet mask, gateway, route, DNS, and so on.
1904This recipe will introduce commands ifconfig , route , nslookup , and host .
1905How to do it...
1906Network interfaces are used to connect to a network. Usually, in the context of UNIX-like
1907Operating Systems, network interfaces follow the eth0, eth1 naming convention. Also, other
1908interfaces, such as usb0, wlan0, and so on, are available for USB network interfaces, wireless
1909LAN, and other such networks.
1910ifconfig is the command that is used to display details about network interfaces, subnet
1911mask, and so on.
1912ifconfig is available at /sbin/ifconfig . Some GNU/Linux distributions will display an
1913error "command not found" when ifconfig is typed. This is because /sbin in not included
1914in the user's PATH environment variable. When a command is typed, the Bash looks in the
1915directories specified in PATH variable.
1916By default, in Debian, ifconfig is not available since /sbin is not in PATH.
1917/sbin/ifconfig is the absolute path, so try run ifconfig with the absolute path (that is,
1918/sbin/ifconfig ). For every system, there will be a by default interface 'lo' called loopback
1919that points to the current machine. For example:
1920$ ifconfig
1921lo Link encap:Local Loopback
1922www.it-ebooks.info
1923Chapter 7
1924235
1925inet addr:127.0.0.1 Mask:255.0.0.0
1926inet6addr: ::1/128 Scope:Host
1927UP LOOPBACK RUNNING MTU:16436 Metric:1
1928RX packets:6078 errors:0 dropped:0 overruns:0 frame:0
1929TX packets:6078 errors:0 dropped:0 overruns:0 carrier:0
1930collisions:0 txqueuelen:0
1931RX bytes:634520 (634.5 KB) TX bytes:634520 (634.5 KB)
1932wlan0 Link encap:EthernetHWaddr 00:1c:bf:87:25:d2
1933inet addr:192.168.0.82 Bcast:192.168.3.255 Mask:255.255.252.0
1934inet6addr: fe80::21c:bfff:fe87:25d2/64 Scope:Link
1935UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
1936RX packets:420917 errors:0 dropped:0 overruns:0 frame:0
1937TX packets:86820 errors:0 dropped:0 overruns:0 carrier:0
1938collisions:0 txqueuelen:1000
1939RX bytes:98027420 (98.0 MB) TX bytes:22602672 (22.6 MB)
1940The left-most column in the ifconfig output lists the name of network interfaces and the
1941right-hand columns show the details related to the corresponding network interface.
1942There's more...
1943There are several additional commands that frequently come under usage for querying and
1944configuring the network. Let's go through the essential commands and usage.
1945Printing the list of network interfaces
1946Here is a one-liner command sequence to print the list of network interface available
1947on a system.
1948$ ifconfig | cut -c-10 | tr -d ' ' | tr -s '\n'
1949lo
1950wlan0
1951The first 10 characters of each line in the ifconfig output is reserved for writing the
1952name of the network interface. Hence we use cut to extract the first 10 characters of each
1953line. tr -d ' ' deletes every space character in each line. Now the \n newline character is
1954squeezed using tr -s '\n' to produce a list of interface names.
1955www.it-ebooks.info
1956The Old-boy Network
1957236
1958Assigning and displaying IP addresses
1959The ifconfig command displays details of every network interface available on the system.
1960However, we can restrict it to a specific interface by using:
1961$ ifconfig iface_name
1962For example:
1963$ ifconfig wlan0
1964wlan0 Link encap:Ethernet HWaddr 00:1c:bf:87:25:d2
1965inet addr:192.168.0.82 Bcast:192.168.3.255
1966Mask:255.255.252.0
1967From the outputs of the previously mentioned command, our interests lie in the IP address,
1968broadcast address, hardware address, and subnet mask. They are as follows:
1969f HWaddr 00:1c:bf:87:25:d2 is the hardware address (MAC address)
1970f inet addr:192.168.0.82 is the IP address
1971f Bcast:192.168.3.255 is the broadcast address
1972f Mask:255.255.252.0 is the subnet mask
1973In several scripting contexts, we may need to extract any of these addresses from the script
1974for further manipulations.
1975Extracting the IP address is a common task. In order to extract the IP address from the
1976ifconfig output use:
1977$ ifconfig wlan0 | egrep -o "inet addr:[^ ]*" | grep -o "[0-9.]*"
1978192.168.0.82
1979Here the first command egrep -o "inet addr:[^ ]*" will print inet
1980addr:192.168.0.82 .
1981The pattern starts with inet addr: and ends with some non-space character sequence
1982(specified by [^ ]* ). Now in the next pipe, it prints the character combination of digits and '.'.
1983In order to set the IP address for a network interface, use:
1984# ifconfig wlan0 192.168.0.80
1985You will need to run the above command as root. 192.168.0.80 is the address to be set.
1986Set the subnet mask along with IP address as follows:
1987# ifconfig wlan0 192.168.0.80 netmask 255.255.252.0
1988www.it-ebooks.info
1989Chapter 7
1990237
1991Spoofing Hardware Address (MAC Address)
1992In certain circumstances where authentication or filtering of computers on a network is
1993provided by using the hardware address, we can use hardware address spoofing. The
1994hardware address appears in ifconfig output as HWaddr 00:1c:bf:87:25:d2 .
1995We can spoof the hardware address at the software level as follows:
1996# ifconfig eth0 hw ether 00:1c:bf:87:25:d5
1997In the above command, 00:1c:bf:87:25:d5 is the new MAC address to be assigned.
1998This can be useful when we need to access the Internet through MAC authenticated service
1999providers that provide access to the Internet for a single machine.
2000Name server and DNS (Domain Name Service)
2001The elementary addressing scheme for the Internet is IP addresses (dotted decimal form, for
2002example, 202.11.32.75 ). However, the resources on the Internet (for example, websites)
2003are accessed through a combination of ASCII characters called URLs or domain names. For
2004example, google.com is a domain name. It actually corresponds to an IP address. Typing the
2005IP address in the browser can also access the URL www.google.com .
2006This technique of abstracting IP addresses with symbolic names is called Domain Name Service
2007(DNS). When we enter google.com , the DNS servers configured with our network resolve the
2008domain name into the corresponding IP address. While on a local network, we setup the local
2009DNS for naming local machines on the network symbolically using their hostnames.
2010Name servers assigned to the current system can be viewed by reading /etc/resolv.conf .
2011For example:
2012$ cat /etc/resolv.conf
2013nameserver 8.8.8.8
2014We can add name servers manually as follows:
2015# echo nameserver IP_ADDRESS >> /etc/resolv.conf
2016How can we obtain the IP address for a corresponding domain name?
2017The easiest method to obtain an IP address is by trying to ping the given domain name and
2018looking at the echo reply. For example:
2019$ ping google.com
2020PING google.com (64.233.181.106) 56(84) bytes of data.
2021Here 64.233.181.106 is the corresponding IP address.
2022A domain name can have multiple IP addresses assigned. In that case, the DNS server will
2023return one address among the list of IP addresses. To obtain all the addresses assigned to
2024the domain name, we should use a DNS lookup utility.
2025www.it-ebooks.info
2026The Old-boy Network
2027238
2028DNS lookup
2029There are different DNS lookup utilities available from the command line. These will request a
2030DNS server for an IP address resolution. host and nslookup are two DNS lookup utilities.
2031When host is executed it will list out all of the IP addressed attached to the domain name.
2032nslookup is another command that is similar to host , which can be used to query details
2033related to DNS and resolving of names. For example:
2034$ host google.com
2035google.com has address 64.233.181.105
2036google.com has address 64.233.181.99
2037google.com has address 64.233.181.147
2038google.com has address 64.233.181.106
2039google.com has address 64.233.181.103
2040google.com has address 64.233.181.104
2041It may also list out DNS resource records like MX (Mail Exchanger) as follows:
2042$ nslookup google.com
2043Server: 8.8.8.8
2044Address: 8.8.8.8#53
2045Non-authoritative answer:
2046Name: google.com
2047Address: 64.233.181.105
2048Name: google.com
2049Address: 64.233.181.99
2050Name: google.com
2051Address: 64.233.181.147
2052Name: google.com
2053Address: 64.233.181.106
2054Name: google.com
2055Address: 64.233.181.103
2056Name: google.com
2057Address: 64.233.181.104
2058Server: 8.8.8.8
2059The last line above corresponds to the default nameserver used for DNS resolution.
2060www.it-ebooks.info
2061Chapter 7
2062239
2063Without using the DNS server, it is possible to add a symbolic name to IP address resolution
2064just by adding entries into file /etc/hosts .
2065In order to add an entry, use the following syntax:
2066# echo IP_ADDRESS symbolic_name >> /etc/hosts
2067For example:
2068# echo 192.168.0.9 backupserver.com >> /etc/hosts
2069After adding this entry, whenever a resolution to backupserver.com occurs, it will resolve
2070to 192.168.0.9 .
2071Setting default gateway, showing routing table information
2072When a local network is connected to another network, it needs to assign some machine
2073or network node through which an interconnection takes place. Hence the IP packets with
2074a destination exterior to the local network should be forwarded to the node machine, which
2075is interconnected to the external network. This special node machine, which is capable of
2076forwarding packets to the external network, is called a gateway. We set the gateway for every
2077node to make it possible to connect to an external network.
2078The operating system maintains a table called the routing table, which contains information
2079on how packets are to be forwarded and through which machine node in the network. The
2080routing table can be displayed as follows:
2081$ route
2082Kernel IP routing table
2083Destination Gateway Genmask Flags Metric Ref UseIface
2084192.168.0.0 * 255.255.252.0 U 2 0 0wlan0
2085link-local * 255.255.0.0 U 1000 0 0wlan0
2086default p4.local 0.0.0.0 UG 0 0 0wlan0
2087Or, you can also use:
2088$ route -n
2089Kernel IP routing table
2090Destination Gateway Genmask Flags Metric Ref Use Iface
2091192.168.0.0 0.0.0.0 255.255.252.0 U 2 0 0 wlan0
2092169.254.0.0 0.0.0.0 255.255.0.0 U 1000 0 0 wlan0
20930.0.0.0 192.168.0.4 0.0.0.0 UG 0 0 0 wlan0
2094Using -n specifies to display the numerical addresses. When -n is used it will display every
2095entry with a numerical IP addresses, else it will show symbolic host names instead of IP
2096addresses under the DNS entries for IP addresses that are available.
2097www.it-ebooks.info
2098The Old-boy Network
2099240
2100A default gateway is set as follows:
2101# route add default gw IP_ADDRESS INTERFACE_NAME
2102For example:
2103# route add default gw 192.168.0.1 wlan0
2104Traceroute
2105When an application requests a service through the Internet, the server may be at a distant
2106location and connected through any number of gateways or device nodes. The packets
2107travel through several gateways and reach the destination. There is an interesting command
2108traceroute that displays the address of all intermediate gateways through which the
2109packet travelled to reach the destination. traceroute information helps us to understand
2110how many hops each packet should take in order reach the destination. The number of
2111intermediate gateways or routers gives a metric to measure the distance between two nodes
2112connected in a large network. An example of the output from traceroute is as follows:
2113$ traceroute google.com
2114traceroute to google.com (74.125.77.104), 30 hops max, 60 byte packets
21151 gw-c6509.lxb.as5577.net (195.26.4.1) 0.313 ms 0.371 ms 0.457 ms
21162 40g.lxb-fra.as5577.net (83.243.12.2) 4.684 ms 4.754 ms 4.823 ms
21173 de-cix10.net.google.com (80.81.192.108) 5.312 ms 5.348 ms 5.327 ms
21184 209.85.255.170 (209.85.255.170) 5.816 ms 5.791 ms 209.85.255.172
2119(209.85.255.172) 5.678 ms
21205 209.85.250.140 (209.85.250.140) 10.126 ms 9.867 ms 10.754 ms
21216 64.233.175.246 (64.233.175.246) 12.940 ms 72.14.233.114
2122(72.14.233.114) 13.736 ms 13.803 ms
21237 72.14.239.199 (72.14.239.199) 14.618 ms 209.85.255.166
2124(209.85.255.166) 12.755 ms 209.85.255.143 (209.85.255.143) 13.803 ms
21258 209.85.255.98 (209.85.255.98) 22.625 ms 209.85.255.110
2126(209.85.255.110) 14.122 ms
2127*
21289 ew-in-f104.1e100.net (74.125.77.104) 13.061 ms 13.256 ms 13.484 ms
2129See also
2130f Playing with variables and environment variables of Chapter 1, explains the PATH
2131variable
2132f Searching and mining "text" inside a file with grep of Chapter 4, explains the grep
2133command
2134www.it-ebooks.info
2135Chapter 7
2136241
2137Let's ping!
2138ping is the most basic network command, and one that every user should first know. It is a
2139universal command that is available on major Operating Systems. It is also a diagnostic tool
2140used for verifying the connectivity between two hosts on a network. It can be used to find out
2141which machines are alive on a network. Let us see how to use ping.
2142How to do it...
2143In order to check the connectivity of two hosts on a network, the ping command uses
2144Internet Control Message Protocol (ICMP) echo packets. When these echo packets are sent
2145towards a host, the host responds back with a reply if it is reachable or alive.
2146Check whether a host is reachable as follows:
2147$ ping ADDRESS
2148The ADDRESS can be a hostname, domain name, or an IP address itself.
2149ping will continuously send packets and the reply information is printed on the terminal. Stop
2150the pinging by pressing Ctrl + C .
2151For example:
2152f When the host is reachable the output will be similar to the following:
2153$ ping 192.168.0.1
2154PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
215564 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=1.44 ms
2156^C
2157--- 192.168.0.1 ping statistics ---
21581 packets transmitted, 1 received, 0% packet loss, time 0ms
2159rtt min/avg/max/mdev = 1.440/1.440/1.440/0.000 ms
2160$ ping google.com
2161PING google.com (209.85.153.104) 56(84) bytes of data.
216264 bytes from bom01s01-in-f104.1e100.net (209.85.153.104): icmp_
2163seq=1 ttl=53 time=123 ms
2164^C
2165--- google.com ping statistics ---
21661 packets transmitted, 1 received, 0% packet loss, time 0ms
2167rtt min/avg/max/mdev = 123.388/123.388/123.388/0.000 ms
2168www.it-ebooks.info
2169The Old-boy Network
2170242
2171f When a host is unreachable the output will be similar to:
2172$ ping 192.168.0.99
2173PING 192.168.0.99 (192.168.0.99) 56(84) bytes of data.
2174From 192.168.0.82 icmp_seq=1 Destination Host Unreachable
2175From 192.168.0.82 icmp_seq=2 Destination Host Unreachable
2176Once the host is not reachable, the ping returns a Destination Host Unreachable
2177error message.
2178There's more
2179In addition to checking the connectivity between two points in a network, the ping command
2180can be used with additional options to get useful information. Let's go through the additional
2181options of ping .
2182Round trip time
2183The ping command can be used to find out the Round Trip Time (RTT) between two hosts on a
2184network. RTT is the time required for the packet to reach the destination host and come back to
2185the source host. The RTT in milliseconds can be obtained from ping. An example is as follows:
2186--- google.com ping statistics ---
21875 packets transmitted, 5 received, 0% packet loss, time 4000ms
2188rtt min/avg/max/mdev = 118.012/206.630/347.186/77.713 ms
2189Here the minimum RTT is 118.012ms, the average RTT is 206.630ms, and the maximum RTT is
2190347.186ms. The mdev (77.713ms) parameter in the ping output stands for mean deviation.
2191Limiting number of packets to be sent
2192The ping command sends echo packets and waits for the reply of echo indefinitely until it is
2193stopped by pressing Ctrl + C . However, we can limit the count of echo packets to be sent by
2194using the -c flag.
2195The usage is as follows:
2196-c COUNT
2197For example:
2198$ ping 192.168.0.1 -c 2
2199PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
220064 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=4.02 ms
220164 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=1.03 ms
2202www.it-ebooks.info
2203Chapter 7
2204243
2205--- 192.168.0.1 ping statistics ---
22062 packets transmitted, 2 received, 0% packet loss, time 1001ms
2207rtt min/avg/max/mdev = 1.039/2.533/4.028/1.495 ms
2208In the previous example, the ping command sends two echo packets and stops.
2209This is useful when we need to ping multiple machines from a list of IP addresses through a
2210script and checks its statuses.
2211Return status of ping command
2212The ping command returns exit status 0 when it succeeds and returns non-zero when it
2213fails. Successful means, destination host is reachable, where failure is when destination host
2214is unreachable.
2215The return status can be easily obtained as follows:
2216$ ping ADDRESS -c2
2217if [ $? -eq 0 ];
2218then
2219echo Successful ;
2220else
2221echo Failure
2222fi
2223Listing all the machines alive on a network
2224When we deal with a large local area network, we may need to check the availability of other
2225machines in the network, whether alive or not. A machine may not be alive in two conditions:
2226either it is not powered on or due to a problem in the network. By using shell scripting, we can
2227easily find out and report which machines are alive on the network. Let's see how to do it.
2228Getting ready
2229In this recipe, we use two methods. The first method uses ping and the second method uses
2230fping . fping doesn't come with a Linux distribution by default. You may have to manually
2231install fping using a package manager.
2232How to do it...
2233Let's go through the script to find out all the live machines on the network and alternate
2234methods to find out the same.
2235www.it-ebooks.info
2236The Old-boy Network
2237244
2238f Method 1:
2239We can write our own script using the ping command to query list of IP addresses
2240and check whether they are alive or not as follows:
2241#!/bin/bash
2242#Filename: ping.sh
2243# Change base address 192.168.0 according to your network.
2244for ip in 192.168.0.{1..255} ;
2245do
2246ping $ip -c 2 &> /dev/null ;
2247if [ $? -eq 0 ];
2248then
2249echo $ip is alive
2250fi
2251done
2252The output is as follows:
2253$ ./ping.sh
2254192.168.0.1 is alive
2255192.168.0.90 is alive
2256f Method 2:
2257We can use an existing command-line utility to query the status of machines on a
2258network as follows:
2259$ fping -a 192.160.1/24 -g 2> /dev/null
2260192.168.0.1
2261192.168.0.90
2262Or, use:
2263$ fping -a 192.168.0.1 192.168.0.255 -g
2264How it works...
2265In Method 1, we used the ping command to find out the alive machines on the network.
2266We used a for loop for iterating through the list of IP addresses. The list is generated as
2267192.168.0.{1..255} . The {start..end} notation will expand and will generate a list of
2268IP addresses, such as 192.168.0.1 , 192.168.0.2 , 192.168.0.3 till 192.168.0.255 .
2269www.it-ebooks.info
2270Chapter 7
2271245
2272ping $ip -c 2 &> /dev/null will run a ping to the corresponding IP address in each
2273execution of loop. -c 2 is used to restrict the number of echo packets to be sent to two
2274packets. &> /dev/null is used to redirect both stderr and stdout to /dev/null so that
2275it won't be printed on the terminal. Using $? we evaluate the exit status. If it is successful, the
2276exit status is 0 else non-zero. Hence the successful IP addresses are printed. We can also
2277print the list of unsuccessful IP addresses to give the list of unreachable IP addresses.
2278Here is an exercise for you. Instead of using a range of IP
2279addresses hard-coded in the script, modify the script to
2280read a list of IP addresses from a file or stdin.
2281In this script, each ping is executed one after the other. Even though all the IP addresses
2282are independent each other, the ping command is executed due to a sequential program, it
2283takes a delay of sending two echo packets and receiving them or the timeout for a reply for
2284executing the next ping command.
2285When it comes to 255 addresses, the delay is large. Let's run all the ping commands in
2286parallel to make it much faster. The core part of the script is the loop body. To make the ping
2287commands in parallel, enclose the loop body in ( )& . ( ) encloses a block of commands
2288to run as the sub-shell and & sends it to the background by leaving the current thread. For
2289example:
2290(
2291ping $ip -c2 &> /dev/null ;
2292if [ $? -eq 0 ];
2293then
2294echo $ip is alive
2295fi
2296)&
2297wait
2298The for loop body executes many background process and it comes out of the loop and it
2299terminates the script. In order to present the script to terminate until all its child process end,
2300we have a command called wait . Place a wait at the end of the script so that it waits for the
2301time until all the child ( ) subshell processes complete.
2302The wait command enables a script to be terminated only after all its child
2303process or background processes terminate or complete.
2304Have a look at fast_ping.sh from the code provided with the book.
2305www.it-ebooks.info
2306The Old-boy Network
2307246
2308Method 2 uses a different command called fping . It can ping a list of IP addresses
2309simultaneously and respond very quickly. The options available with fping are as follows:
2310f The -a option with fping specifies to print all alive machine's IP addresses
2311f The -u option with fping specifies to print all unreachable machines
2312f The -g option specifies to generate a range of IP addresses from slash-subnet mask
2313notation specified as IP/mask or start and end IP addresses as:
2314$ fping -a 192.160.1/24 -g
2315Or
2316$ fping -a 192.160.1 192.168.0.255 -g
2317f 2>/dev/null is used to dump error messages printed due to unreachable host to a
2318null device
2319It is also possible to manually specify a list of IP addresses as command-line arguments or as
2320a list through stdin . For example:
2321$ fping -a 192.168.0.1 192.168.0.5 192.168.0.6
2322# Passes IP address as arguments
2323$ fping -a <ip.list
2324# Passes a list of IP addresses from a file
2325There's more...
2326The fping command can be used for querying DNS data from a network. Let's see how to do it.
2327DNS lookup with fping
2328fping has an option -d that returns host names by using DNS lookup for each echo reply. It
2329will print out host names rather than IP addresses on ping replies.
2330$ cat ip.list
2331192.168.0.86
2332192.168.0.9
2333192.168.0.6
2334$ fping -a -d 2>/dev/null <ip.list
2335www.local
2336dnss.local
2337www.it-ebooks.info
2338Chapter 7
2339247
2340See also
2341f Playing with file descriptors and redirection of Chapter 1, explains the data
2342redirection
2343f Comparisons and tests of Chapter 1, explains numeric comparisons
2344Transferring files
2345The major purpose of the networking of computers is for resource sharing. Among resource
2346sharing, the most prominent use is in file sharing. There are different methods by which we
2347can transfer files between different nodes on a network. This recipe discusses how to transfer
2348files using commonly used protocols FTP, SFTP, RSYNC, and SCP.
2349Getting ready
2350The commands for performing file transfer over the network are mostly available by default
2351with Linux installations. Files via FTP can be transferred by using the lftp command. Files via
2352a SSH connection can be transferred by using sftp , RSYNC using SSH with rsync command
2353and transfer through SSH using scp .
2354How to do it...
2355File Transfer Protocol (FTP) is an old file transfer protocol for transferring files between
2356machines on a network. We can use the command lftp for accessing FTP enabled servers
2357for file transfer. It uses Port 21. FTP can only be used if an FTP server is installed on the
2358remote machine. FTP is used by many public websites to share files.
2359To connect to an FTP server and transfer files in between, use:
2360$ lftp username@ftphost
2361Now it will prompt for a password and then display a logged in prompt as follows:
2362lftp username@ftphost:~>
2363You can type commands in this prompt. For example:
2364f To change to a directory, use cd directory
2365f To change directory of local machine, use lcd
2366f To create a directory use mkdir
2367f To download a file, use get filename as follows:
2368lftp username@ftphost:~> get filename
2369www.it-ebooks.info
2370The Old-boy Network
2371248
2372f To upload a file from the current directory, use put filename as follows:
2373lftp username@ftphost:~> put filename
2374f An lftp session can be exited by using the quit command
2375Auto completion is supported in the lftp prompt.
2376There's more...
2377Let's go through some additional techniques and commands used for file transfer through a
2378network.
2379Automated FTP transfer
2380ftp is another command used for FTP-based file transfer. lftp is more flexible for usage.
2381lftp and the ftp command open an interactive session with user (it prompts for user input
2382by displaying messages). What if we want to automate a file transfer instead of using the
2383interactive mode? We can automate FTP file transfers by writing a shell script as follows:
2384#!/bin/bash
2385#Filename: ftp.sh
2386#Automated FTP transfer
2387HOST='domain.com'
2388USER='foo'
2389PASSWD='password'
2390ftp -i -n $HOST <<EOF
2391user ${USER} ${PASSWD}
2392binary
2393cd /home/slynux
2394puttestfile.jpg
2395getserverfile.jpg
2396quit
2397EOF
2398The above script has the following structure:
2399<<EOF
2400DATA
2401EOF
2402This is used to send data through stdin to the FTP command. The recipe, Playing with file
2403descriptors and redirection in Chapter 1, explains various methods for redirection into stdin .
2404The -i option of ftp turns off the interactive session with user. user ${USER} ${PASSWD}
2405sets the username and password. binary sets the file mode to binary.
2406www.it-ebooks.info
2407Chapter 7
2408249
2409SFTP (Secure FTP)
2410SFTP is an FTP-like file transfer system that runs on top of an SSH connection. It makes use of
2411an SSH connection to emulate an FTP interface. It doesn't require an FTP server at the remote
2412end to perform file transfer but it requires an OpenSSH server to be installed and running. It is
2413an interactive command, which offers an sftp prompt.
2414The following commands are used to perform the file transfer. All other commands remain
2415same for every automated FTP session with specific HOST, USER, and PASSWD:
2416cd /home/slynux
2417put testfile.jpg
2418get serverfile.jpg
2419In order to run sftp , use:
2420$ sftp user@domainname
2421Similar to lftp , an sftp session can be exited by typing the quit command.
2422The SSH server sometimes will not be running at the default Port 22. If it is running at a
2423different port, we can specify the port along with sftp as -oPort=PORTNO .
2424For example:
2425$ sftp -oPort=422 user@slynux.org
2426-oPort should be the first argument of the sftp command.
2427RSYNC
2428rsync is an important command-line utility that is widely used for copying files over networks
2429and for taking backup snapshots. This is better explained in separate recipe,
2430Backup snapshots with rsync, that explains the usage of rsync .
2431SCP (Secure Copy)
2432SCP is a file copy technique which is more secure than the traditional remote copy tool called
2433rcp . The files are transferred through an encrypted channel. SSH is used as an encryption
2434channel. We can easily transfer files to a remote machine as follows:
2435$ scp filename user@remotehost:/home/path
2436This will prompt for a password. It can be made password less by using autologin SSH
2437technique. The recipe, Password-less auto-login with SSH, explains SSH autologin.
2438Therefore, file transfer using scp doesn't require specific scripting. Once SSH login is automated,
2439the scp command can be executed without an interactive prompt for the password.
2440www.it-ebooks.info
2441The Old-boy Network
2442250
2443Here remotehost can be IP address or domain name. The format of the scp command is:
2444$ scp SOURCE DESTINATION
2445SOURCE or DESTINATION can be in the format username@localhost:/path for example:
2446$ scp user@remotehost:/home/path/filename filename
2447The above command copies a file from the remote host to the current directory with the given
2448filename.
2449If SSH is running at a different port than 22, use -oPort with the same syntax as sftp .
2450Recursive copying with SCP
2451By using scp we can recursively copy a directory between two machines on a network as
2452follows with the -r parameter:
2453$ scp -r /home/slynux user@remotehost:/home/backups
2454# Copies the directory /home/slynux recursively to remote location
2455scp can also copy files by preserving permissions and mode by using the -p parameter.
2456See also
2457f Playing with file descriptors and redirection of Chapter 1, explains the standard input
2458using EOF
2459Setting up an Ethernet and wireless LAN
2460with script
2461An Ethernet is simple to configure. Since it uses physical cables, there are no special
2462requirements such as authentication. However, a wireless LAN requires authentication—for
2463example, a WEP key as well as the ESSID of the wireless network to connect. Let's see how to
2464connect to a wireless as well as a wired network by writing a shell script.
2465Getting ready
2466To connect to a wired network, we need to assign an IP address and subnet mask by using the
2467ifconfig utility. But for a wireless network connection, it will require additional utilities, such
2468as iwconfig and iwlist , to configure more parameters.
2469www.it-ebooks.info
2470Chapter 7
2471251
2472How to do it...
2473In order to connect to a network from a wired interface, execute the following script:
2474#!/bin/bash
2475#Filename: etherconnect.sh
2476#Description: Connect Ethernet
2477#Modify the parameters below according to your settings
2478######### PARAMETERS ###########
2479IFACE=eth0
2480IP_ADDR=192.168.0.5
2481SUBNET_MASK=255.255.255.0
2482GW=192.168.0.1
2483HW_ADDR='00:1c:bf:87:25:d2'
2484# HW_ADDR is optional
2485#################################
2486if [ $UID -ne 0 ];
2487then
2488echo "Run as root"
2489exit 1
2490fi
2491# Turn the interface down before setting new config
2492/sbin/ifconfig $IFACE down
2493if [[ -n $HW_ADDR ]];
2494then
2495/sbin/ifconfig hw ether $HW_ADDR
2496echo Spoofed MAC ADDRESS to $HW_ADDR
2497fi
2498/sbin/ifconfig $IFACE $IP_ADDR netmask $SUBNET_MASK
2499route add default gw $GW $IFACE
2500echo Successfully configured $IFACE
2501The script for connecting to a wireless LAN with WEP is as follows:
2502#!/bin/bash
2503#Filename: wlan_connect.sh
2504#Description: Connect to Wireless LAN
2505#Modify the parameters below according to your settings
2506######### PARAMETERS ###########
2507IFACE=wlan0
2508IP_ADDR=192.168.1.5
2509SUBNET_MASK=255.255.255.0
2510www.it-ebooks.info
2511The Old-boy Network
2512252
2513GW=192.168.1.1
2514HW_ADDR='00:1c:bf:87:25:d2'
2515#Comment above line if you don't want to spoof mac address
2516ESSID="homenet"
2517WEP_KEY=8b140b20e7
2518FREQ=2.462G
2519#################################
2520KEY_PART=""
2521if [[ -n $WEP_KEY ]];
2522then
2523KEY_PART="key $WEP_KEY"
2524fi
2525# Turn the interface down before setting new config
2526/sbin/ifconfig $IFACE down
2527if [ $UID -ne 0 ];
2528then
2529echo "Run as root"
2530exit 1;
2531fi
2532if [[ -n $HW_ADDR ]];
2533then
2534/sbin/ifconfig $IFACE hw ether $HW_ADDR
2535echo Spoofed MAC ADDRESS to $HW_ADDR
2536fi
2537/sbin/iwconfig $IFACE essid $ESSID $KEY_PART freq $FREQ
2538/sbin/ifconfig $IFACE $IP_ADDR netmask $SUBNET_MASK
2539route add default gw $GW $IFACE
2540echo Successfully configured $IFACE
2541How it works...
2542The commands ifconfig , iwconfig , and route are to be run as root. Hence a check for
2543the root user is performed at the beginning of the scripts.
2544The Ethernet connection script is pretty straightforward and it uses the concepts explained in
2545the recipe, Basic networking primer. Let's go through the commands used for connecting to
2546the wireless LAN.
2547www.it-ebooks.info
2548Chapter 7
2549253
2550A wireless LAN requires some parameters such as the essid , key , and frequency to connect
2551to the network. The essid is the name of the wireless network to which we need to connect.
2552Some Wired Equivalent Protocol (WEP) networks use a WEP key for authentication, whereas
2553some networks don't. The WEP key is usually a 10-letter hex passphrase. Next comes the
2554frequency assigned to the network. iwconfig is the command used to attach the wireless
2555card with the proper wireless network, WEP key, and frequency.
2556We can scan and list the available wireless network by using the utility iwlist . To scan, use
2557the following command:
2558# iwlist scan
2559wlan0 Scan completed :
2560Cell 01 - Address: 00:12:17:7B:1C:65
2561Channel:11
2562Frequency:2.462 GHz (Channel 11)
2563Quality=33/70 Signal level=-77 dBm
2564Encryption key:on
2565ESSID:"model-2"
2566The Frequency parameter can be extracted from the scan result, from the line
2567Frequency:2.462 GHz (Channel 11) .
2568See also
2569f Comparisons and tests of Chapter 1, explains string comparisons.
2570Password-less auto-login with SSH
2571SSH is widely used with automation scripting. By using SSH, it is possible to remotely execute
2572commands at remote hosts and read their output. SSH is authenticated by using username
2573and password. Passwords are prompted during the execution of SSH commands. But in
2574automation scripts, SSH commands may be executed hundreds of times in a loop and hence
2575providing passwords each time is impractical. Hence we need to automate logins. SSH has
2576a built-in feature by which SSH can auto-login using SSH keys. This recipe describes how to
2577create SSH keys and facilitate auto-login.
2578www.it-ebooks.info
2579The Old-boy Network
2580254
2581How to do it...
2582The SSH uses public key-based and private key-based encryption techniques for automatic
2583authentication. An authentication key has two elements: a public key and a private key pair.
2584We can create an authentication key using the ssh-keygen command. For automating the
2585authentication, the public key must be placed at the server (by appending the public key to the
2586~/.ssh/authorized_keys file) and its private key file of the pair should be present at the
2587~/.ssh directory of the user at client machine, which is the computer you are logging in from.
2588Several configurations (for example, path and name of the authorized_keys file) regarding
2589the SSH can be configured by altering the configuration file /etc/ssh/sshd_config .
2590There are two steps towards the setup of automatic authentication with SSH. They are:
25911. Creating the SSH key from the machine, which requires a login to a remote machine.
25922. Transferring the public key generated to the remote host and appending it to
2593~/.ssh/authorized_keys file.
2594In order to create an SSH key, enter the ssh-keygen command with the encryption algorithm
2595type specified as RSA as follows:
2596$ ssh-keygen -t rsa
2597Generating public/private rsa key pair.
2598Enter file in which to save the key (/home/slynux/.ssh/id_rsa):
2599Created directory '/home/slynux/.ssh'.
2600Enter passphrase (empty for no passphrase):
2601Enter same passphrase again:
2602Your identification has been saved in /home/slynux/.ssh/id_rsa.
2603Your public key has been saved in /home/slynux/.ssh/id_rsa.pub.
2604The key fingerprint is:
2605f7:17:c6:4d:c9:ee:17:00:af:0f:b3:27:a6:9c:0a:05slynux@slynux-laptop
2606The key's randomart image is:
2607+--[ RSA 2048]----+
2608| . |
2609| o . .|
2610| E o o.|
2611| ...oo |
2612| .S .+ +o.|
2613| . . .=....|
2614| .+.o...|
2615| . . + o. .|
2616| ..+ |
2617+-----------------+
2618www.it-ebooks.info
2619Chapter 7
2620255
2621You need to enter a passphrase for generating the public-private key pair. It is also possible
2622to generate the key pair without entering a passphrase, but it is insecure. We can write
2623monitoring scripts that use automated login from the script to several machines. In such
2624cases, you should leave the passphrase empty while running the ssh-keygen command to
2625prevent the script from asking for a passphrase while running.
2626Now ~/.ssh/id_rsa.pub and ~/.ssh/id_rsa has been generated. id_dsa.pub is the
2627generated public key and id_dsa is the private key. The public key has to be appended to the
2628~/.ssh/authorized_keys file on remote servers where we need to auto-login from the
2629current host.
2630In order to append a key file, use:
2631$ ssh USER@REMOTE_HOST "cat >> ~/.ssh/authorized_keys" < ~/.ssh/id_rsa.
2632pub
2633Password:
2634Provide the login password in the previous command.
2635The auto-login has been set up. From now on, SSH will not prompt for passwords during
2636execution. You can test this with the following command:
2637$ ssh USER@REMOTE_HOST uname
2638Linux
2639You will not be prompted for a password.
2640Running commands on remote host
2641with SSH
2642SSH is an interesting system administration tool that enables to control remote hosts by login
2643with a shell. SSH stands for Secure Shell. Commands can be executed on the shell received
2644by login to remote host as if we run commands on localhost. It runs the network data transfer
2645over an encrypted tunnel. This recipe will introduce different ways in which commands can be
2646executed on the remote host.
2647Getting ready
2648SSH doesn't come by default with all GNU/Linux distributions. Therefore, you may have to
2649install the openssh-server and openssh-client packages using a package manager.
2650SSH service runs by default on port number 22.
2651www.it-ebooks.info
2652The Old-boy Network
2653256
2654How to do it...
2655To connect to a remote host with the SSH server running, use:
2656$ ssh username@remote_host
2657In this command:
2658f username is the user that exist at the remote host.
2659f remote_host can be domain name or IP address.
2660For example:
2661$ ssh mec@192.168.0.1
2662The authenticity of host '192.168.0.1 (192.168.0.1)' can't be
2663established.
2664RSA key fingerprint is 2b:b4:90:79:49:0a:f1:b3:8a:db:9f:73:2d:75:d6:f9.
2665Are you sure you want to continue connecting (yes/no)? yes
2666Warning: Permanently added '192.168.0.1' (RSA) to the list of known
2667hosts.
2668Password:
2669Last login: Fri Sep 3 05:15:21 2010 from 192.168.0.82
2670mec@proxy-1:~$
2671It will interactively ask for a user password and upon successful authentication it will return
2672the shell for the user.
2673By default, the SSH server runs at Port 22. But certain servers run the SSH service at different
2674ports. In that case use -p port_no with the ssh command to specify the port.
2675In order to connect to an SSH server running at port 422, use:
2676$ ssh user@locahost -p 422
2677You can execute commands in the shell that corresponds to the remote host. Shell is an
2678interactive tool in which a user types and runs commands. However, in shell scripting contexts,
2679we do not need an interactive shell. We need to automate several tasks. We require to execute
2680several commands at the remote shell and display or store its output at localhost. Issuing a
2681password every time is not practical for an automated script, hence autologin for SSH should
2682be configured.
2683The recipe, Password-less auto-login with SSH, explains the SSH commands.
2684Make sure that auto-login is configured before running automated scripts that use SSH.
2685www.it-ebooks.info
2686Chapter 7
2687257
2688To run a command on the remote host and display its output on the localhost shell, use the
2689following syntax:
2690$ ssh user@host 'COMMANDS'
2691For example:
2692$ ssh mec@192.168.0.1 'whoami'
2693Password:
2694mec
2695Multiple commands can be given by using semicolon delimiter in between the commands as:
2696$ ssh user@host 'command1 ; command2 ; command3'
2697Commands can be sent through stdin and the output of the commands will be available to
2698stdout .
2699The syntax will be as follows:
2700$ ssh user@remote_host "COMMANDS" > stdout.txt 2> errors.txt
2701The COMMANDS string should be quoted in order to prevent a semicolon character to act as
2702delimiter in the localhost shell. We can also pass any command sequence that involves piped
2703statements to the SSH command through stdin as follows:
2704$ echo "COMMANDS" | sshuser@remote_host> stdout.txt 2> errors.txt
2705For example:
2706$ ssh mec@192.168.0.1 "echo user: $(whoami);echo OS: $(uname)"
2707Password:
2708user: slynux
2709OS: Linux
2710In this example, the commands executed on the remote host are:
2711echo user: $(whoami);
2712echo OS: $(uname)
2713It can be generalized as:
2714COMMANDS="command1; command2; command3"
2715$ ssh user@hostname "$COMMANDS"
2716We can also pass a more complex subshell in the command sequence by using the ( )
2717subshell operator.
2718www.it-ebooks.info
2719The Old-boy Network
2720258
2721Let's write an SSH based shell script that collects the uptime of a list of remote hosts. Uptime
2722is the time for which the system is powered on. The uptime command is used to display how
2723long the system has been powered on.
2724It is assumed that all systems in the IP_LIST have a common user test .
2725#!/bin/bash
2726#Filename: uptime.sh
2727#Description: Uptime monitor
2728IP_LIST="192.168.0.1 192.168.0.5 192.168.0.9"
2729USER="test"
2730for IP in $IP_LIST;
2731do
2732utime=$(ssh $USER@$IP uptime | awk '{ print $3 }' )
2733echo $IP uptime: $utime
2734done
2735The expected output is:
2736$ ./uptime.sh
2737192.168.0.1 uptime: 1:50,
2738192.168.0.5 uptime: 2:15,
2739192.168.0.9 uptime: 10:15,
2740There's more...
2741The ssh command can be executed with several additional options. Let's go through them.
2742SSH with compression
2743The SSH protocol also supports data transfer with compression, which comes in handy when
2744bandwidth is an issue. Use the -C option with the ssh command to enable compression as
2745follows:
2746$ ssh -C user@hostname COMMANDS
2747Redirecting data into stdin of remote host shell commands
2748Sometimes we need to redirect some data into stdin of remote shell commands. Let's see
2749how to do it. An example is as follows:
2750$ echo "text" | ssh user@remote_host 'cat >> list'
2751www.it-ebooks.info
2752Chapter 7
2753259
2754Or:
2755# Redirect data from file as:
2756$ ssh user@remote_host 'cat >> list' < file
2757cat >> list appends the data received through stdin to the file list. Here this command
2758is executed at the remote host. But the data is passed to stdin from localhost.
2759See also
2760f Password-less auto-login with SSH, explains how to configure auto-login to execute
2761commands without prompting for password.
2762Mounting a remote drive at a local mount
2763point
2764Having a local mount point to access remote host file-system would be really helpful while
2765carrying out both read and write data transfer operations. SSH is the most common transfer
2766protocol available in a network and hence we can make use of it with sshfs . sshfs enables
2767you to mount a remote filesystem to a local mount point. Let's see how to do it.
2768Getting ready
2769sshfs doesn't come by default with GNU/Linux distributions. Install sshfs by using a
2770package manager. sshfs is an extension to the fuse file system package that allows
2771supported OSes to mount a wide variety of data as if it were a local file system.
2772How to do it...
2773In order to mount a filesytem location at a remote host to a local mount point, use:
2774# sshfs user@remotehost:/home/path /mnt/mountpoint
2775Password:
2776Issue the user password when prompted.
2777Now data at /home/path on the remote host can be accessed via a local mount point /mnt/
2778mountpoint .
2779In order to unmount after completing the work, use:
2780# umount /mnt/mountpoint
2781www.it-ebooks.info
2782The Old-boy Network
2783260
2784See also
2785f Running commands on remote host with SSH, explains the ssh command.
2786Multi-casting window messages on
2787a network
2788The administrator of a network may often require to send messages to the nodes on the
2789network. Displaying pop-up windows on the user's desktop would be helpful to alert the user
2790with a piece of information. Using a GUI toolkit with shell scripting can achieve this task. This
2791recipe discusses how to send a popup window with custom messages to remote hosts.
2792Getting ready
2793For implementing a GUI pop window, zenity can be used. Zenity is a scriptable GUI toolkit for
2794creating windows consisting of textbox, input box, and so on. SSH can be used for connecting
2795to the remote shell on a remote host. Zenity doesn't come installed by default with GNU/Linux
2796distributions. Use a package manager to install zenity.
2797How to do it...
2798Zenity is one of the scriptable dialog creation toolkit. There are other toolkits, such as gdialog,
2799kdialog, xdialog, and so on. Zenity seems to be one flexible toolkit that is adherent to the
2800GNOME Desktop Environment.
2801In order to create an info box with zenity, use:
2802$ zenity --info --text "This is a message"
2803# It will display a window with "This is a message" as text.
2804Zenity can be used to create windows with input box, combo input, radio button, pushbutton,
2805and more. They are not in the scope of this recipe. Check the man page of zenity for more.
2806Now, we can use SSH to run these zenity statements on a remote machine. In order to run this
2807statement on the remote host through SSH, run:
2808$ ssh user@remotehost 'zenity --info --text "This is a message"'
2809But this will return an error like:
2810(zenity:3641): Gtk-WARNING **: cannot open display:
2811This is because zenity depends on Xserver. Xsever is a daemon which is responsible for
2812plotting graphical elements on the screen which consists of the GUI. A bare GNU/Linux system
2813consists of only a text terminal or shell prompts.
2814www.it-ebooks.info
2815Chapter 7
2816261
2817Xserver uses a special environment variable, DISPLAY , to track the Xserver instance that is
2818running on the system.
2819We can manually set DISPLAY=:0 to instruct Xserver about the Xserver instance.
2820The previous SSH command can be rewritten as:
2821$ ssh username@remotehost 'export DISPLAY=:0 ; zenity --info --text "This
2822is a message"'
2823This statement will display a pop up at remotehost if the user with username has been
2824logged in any of the window managers.
2825In order to multicast the popup window to multiple remote hosts, write a shell script as follows:
2826#!/bin/bash
2827#Filename: multi_cast_window.sh
2828# Description: Multi-cast window popups
2829IP_LIST="192.168.0.5 192.168.0.3 192.168.0.23"
2830USER="username"
2831COMMAND='export DISPLAY=:0 ;zenity --info --text "This is a message" '
2832for host in $IP_LIST;
2833do
2834ssh $USER@$host "$COMMAND" &
2835done
2836How it works...
2837In the above script, we have a list of IP addresses to which the window should be popped up.
2838A loop is used to iterate through IP addresses and execute the SSH command.
2839In the SSH statement, at the end we have post fixed & . & will send an SSH statement to the
2840background. It is done to facilitate parallelization in the execution of several SSH statements.
2841If & was not used, it will start the SSH session, execute the zenity dialog, and wait for the user
2842to close that pop up window. Unless the user at the remote host closes the window, the next
2843SSH statement in the loop will not be executed. In order to move away from this blocking of
2844the loop from further execution by waiting for the SSH session to terminate, the & trick is used.
2845See also
2846f Running commands on remote host with SSH, explains the ssh command.
2847www.it-ebooks.info
2848The Old-boy Network
2849262
2850Network traffic and port analysis
2851Network ports are essential parameters of network-based applications. Applications open
2852ports on the host and communicate to a remote host through opened ports at the remote
2853host. Having awareness of opened and closed ports is essential for security context. Malwares
2854and root kits may be running on the system with custom ports and custom services that allow
2855attackers to capture unauthorized access to data and resources. By getting the list of opened
2856ports and services running on the ports, we can analyze and defend the system from being
2857controlled by root kits and the list helps to remove them efficiently. The list of opened ports
2858is not only helpful for malware detection, but also for collecting information about opened
2859ports on the system enables to debug network based applications. It helps to analyse whether
2860certain port connections and port listening functionalities are working fine. This recipe
2861discusses various utilities for port analysis.
2862Getting ready
2863Various commands are available for listening to ports and services running on each port (for
2864example, lsof and netstat ). These commands are, by default, available on all GNU/Linux
2865distributions.
2866How to do it...
2867In order to list all opened ports on the system along with the details on each service attached
2868to it, use:
2869$ lsof -i
2870COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
2871firefox-b 2261 slynux 78u IPv4 63729 0t0 TCP localhost:47797-
2872>localhost:42486 (ESTABLISHED)
2873firefox-b 2261 slynux 80u IPv4 68270 0t0 TCP slynux-laptop.
2874local:41204->192.168.0.2:3128 (CLOSE_WAIT)
2875firefox-b 2261 slynux 82u IPv4 68195 0t0 TCP slynux-laptop.
2876local:41197->192.168.0.2:3128 (ESTABLISHED)
2877ssh 3570 slynux 3u IPv6 30025 0t0 TCP localhost:39263-
2878>localhost:ssh (ESTABLISHED)
2879ssh 3836 slynux 3u IPv4 43431 0t0 TCP slynux-laptop.
2880local:40414->boneym.mtveurope.org:422 (ESTABLISHED)
2881GoogleTal 4022 slynux 12u IPv4 55370 0t0 TCP localhost:42486
2882(LISTEN)
2883GoogleTal 4022 slynux 13u IPv4 55379 0t0 TCP localhost:42486-
2884>localhost:32955 (ESTABLISHED)
2885Each entry in the output of lsof corresponds to each service that opens a port for
2886communication. The last column of the output consists of lines similar to:
2887www.it-ebooks.info
2888Chapter 7
2889263
2890slynux-laptop.local:34395->192.168.0.2:3128 (ESTABLISHED)
2891In this output slynux-laptop.local:34395 corresponds to localhost part and
2892192.168.0.2:3128 corresponds to remote host.
289334395 is the port opened from current machine, and 3128 is the port to which the service
2894connects at remote host.
2895In order to list out the opened ports from current machine, use:
2896$ lsof -i | grep ":[0-9]\+->" -o | grep "[0-9]\+" -o | sort | uniq
2897The :[0-9]\+-> regex for grep is used to extract the host port portion ( :34395-> ) from the
2898lsof output. The next grep is used to extract the port number (which is numeric). Multiple
2899connections may occur through the same port and hence multiple entries of the same port may
2900occur. In order to display each port once, they are sorted and the unique ones are printed.
2901There's more...
2902Let's go through additional utilities that can be used for viewing the opened port and network
2903traffic related information.
2904Opened port and services using netstat
2905netstat is another command for network service analysis. Explaining all the features of
2906netstat is not in the scope of this recipe. We will now look at how to list services and port
2907numbers.
2908Use netstat -tnp to list opened ports and services as follows:
2909$ netstat -tnp
2910(Not all processes could be identified, non-owned process info
2911will not be shown, you would have to be root to see it all.)
2912Active Internet connections (w/o servers)
2913Proto Recv-Q Send-Q Local Address Foreign Address State
2914PID/Program name
2915tcp 0 0 192.168.0.82:38163 192.168.0.2:3128
2916ESTABLISHED 2261/firefox-bin
2917tcp 0 0 192.168.0.82:38164 192.168.0.2:3128 TIME_
2918WAIT -
2919tcp 0 0 192.168.0.82:40414 193.107.206.24:422
2920ESTABLISHED 3836/ssh
2921tcp 0 0 127.0.0.1:42486 127.0.0.1:32955
2922ESTABLISHED 4022/GoogleTalkPlug
2923tcp 0 0 192.168.0.82:38152 192.168.0.2:3128
2924ESTABLISHED 2261/firefox-bin
2925tcp6 0 0 ::1:22 ::1:39263
2926ESTABLISHED -
2927tcp6 0 0 ::1:39263 ::1:22
2928ESTABLISHED 3570/ssh
2929www.it-ebooks.info
2930www.it-ebooks.info
29318
2932Put on the Monitor's
2933Cap
2934In this chapter, we will cover:
2935f Disk usage hacks
2936f Calculating the execution time for a command
2937f Information about logged users, boot logs, failure boots
2938f Printing the 10 most frequently-used commands
2939f Listing the top 10 CPU consuming process in 1 hour
2940f Monitoring command outputs with watch
2941f Logging access to files and directories
2942f Logfile management with logrotate
2943f Logging with syslog
2944f Monitoring user logins to find intruders
2945f Remote disk usage health monitoring
2946f Finding out active user hours on a system
2947www.it-ebooks.info
2948Put on the Monitor’s Cap
2949266
2950Introduction
2951An operating system consists of a collection of system software, designed for different
2952purposes, serving different task sets. Each of these programs requires to be monitored by the
2953operating system or the system administrator in order to know whether it is working properly
2954or not. We will also use a technique called logging by which important information is written to
2955a file while the application is running. By reading this file, we can understand the timeline of
2956the operations that are taking place with a particular software or a daemon. If an application
2957or a service crashes, this information helps to debug the issue and enables us to fix any
2958issues. Logging and monitoring also helps to gather information from a pool of data. Logging
2959and monitoring are important tasks for ensuring security in the operating system and for
2960debugging purposes.
2961This chapter deals with different commands that can be used to monitor different activities. It
2962also goes through logging techniques and their usages.
2963Disk usage hacks
2964Disk space is a limited resource. We frequently perform disk usage calculation on hard
2965disks or any storage media to find out the free space available on the disk. When free space
2966becomes scarce, we will need to find out large-sized files that are to be deleted or moved in
2967order to create free space. Disk usage manipulations are commonly used in shell scripting
2968contexts. This recipe will illustrate various commands used for disk manipulations and
2969problems where disk usages can be calculated with a variety of options.
2970Getting ready
2971df and du are the two significant commands that are used for calculating disk usage in Linux.
2972The command df stands for disk free and du stands for disk usage. Let's see how we can use
2973them to perform various tasks that involve disk usage calculation.
2974How to do it...
2975To find the disk space used by a file (or files), use:
2976$ du FILENAME1 FILENAME2 ..
2977For example:
2978$ du file.txt
29794
2980www.it-ebooks.info
2981Chapter 8
2982267
2983The result is, by default, shown as size in bytes.
2984In order to obtain the disk usage for all files inside a directory along with the individual disk
2985usage for each file showed in each line, use:
2986$ du -a DIRECTORY
2987-a outputs results for all files in the specified directory or directories recursively.
2988Running du DIRECTORY will output a similar result, but it will show only the
2989size consumed by subdirectories. However, they do not show the disk usage
2990for each of the files. For printing the disk usage by files, -a is mandatory.
2991For example:
2992$ du -a test
29934 test/output.txt
29944 test/process_log.sh
29954 test/pcpu.sh
299616 test
2997An example of using du DIRECTORY is as follows:
2998$ du test
299916 test
3000There's more...
3001Let's go through additional usage practices for the du command.