uDW0GP1h

1Displaying disk usage in KB, MB, or Blocks
2By default, the disk usage command displays the total bytes used by a file. A more human-
3readable format is when disk usage is expressed in standard units KB, MB, or GB. In order to
4print the disk usage in a display-friendly format, use  –h as follows:
5du -h FILENAME
6For example:
7$ du -sh test/pcpu.sh
84.0K test/pcpu.sh
9# Multiple file arguments are accepted
10www.it-ebooks.info
11Put on the Monitor’s Cap
12268
13Or:
14# du -h DIRECTORY
15$ du -h hack/
1616K hack/
17Displaying the grand total sum of disk usage
18Suppose we need to calculate the total size taken by all the files or directories, displaying
19individual file sizes won't help.  du has an option  -c such that it will output the total disk usage
20of all files and directories given as an argument. It appends a line SIZE total with the result.
21The syntax is as follows:
22$ du -c FILENAME1 FILENAME2..
23For example:
24du -c process_log.sh pcpu.sh
254 process_log.sh
264 pcpu.sh
278 total
28Or:
29$ du -c DIRECTORY
30For example:
31$ du -c test/
3216 test/
3316 total
34Or:
35$ du -c *.txt
36# Wildcards
37-c can be used along with other options like  -a and  -h . It gives the same output as without
38using  -c . The only difference is that it appends an extra line containing the total size.
39There is another option  –s (summarize), which will print only the grand total as the output.
40It will print the total sum, and flag  -h can be used along with it to print in human readable
41format. This command has frequent use in practice. The syntax is as follows:
42$ du -s FILES(s)
43$ du -sh DIRECTORY
44www.it-ebooks.info
45Chapter 8
46269
47For example:
48$ du -sh slynux
49680K slynux
50Printing files in specified units
51We can force  du to print the disk usage in specified units. For example:
52f Print size in bytes (by default) by using:
53$ du -b FILE(s)
54f Print the size in kilobytes by using:
55$ du -k FILE(s)
56f Print the size in megabytes by using:
57$ du -m FILE(s)
58f Print size in given BLOCK size specified by using:
59$ du -B BLOCK_SIZE FILE(s)
60Here,  BLOCK_SIZE is specified in bytes.
61An example consisting of all the commands is as follows:
62$ du pcpu.sh
634 pcpu.sh
64$ du -b pcpu.sh
65439  pcpu.sh
66$ du -k pcpu.sh
674 pcpu.sh
68$ du -m pcpu.sh
691 pcpu.sh
70$ du -B 4 pcpu.sh
711024 pcpu.sh
72Excluding files from disk usage calculation
73There are circumstances when we need to exclude certain files from disk usage calculation.
74Such excluded files can be specified in two ways:
751. Wildcards
76We can specify a wildcard as follows:
77$ du --exclude "WILDCARD" DIRECTORY
78www.it-ebooks.info
79Put on the Monitor’s Cap
80270
81For example:
82$ du --exclude "*.txt" FILES(s)
83# Excludes all .txt files from calculation
842. Exclude list
85We can specify a list of files to be excluded from a file as follows:
86$ du --exclude-from EXCLUDE.txt DIRECTORY
87# EXCLUDE.txt is the file containing list
88There are also some other handy options available with  du to restrict the disk usage
89calculation. We can specify the maximum depth of the hierarchy that the  du should traverse
90as a whole by calculating disk usage with the  --max-depth parameter. Specifying a depth of
911 calculates the sizes of files in the current directory. Depth  2 will calculate files in the current
92directory and the next subdirectory and stop traversal at that second subdirectory.
93For example:
94$ du --max-depth 2 DIRECTORY
95du can be restricted to traverse only a single file system by using the -x
96argument. Suppose du DIRECTORY is run, it will traverse through every
97possible subdirectory of DIRECTORY recursively. A subdirectory in the
98directory hierarchy may be a mount point (for example, /mnt/sda1 is a
99subdirectory of /mnt and it is a mount point for the device /dev/sda1). du
100will traverse that mount point and calculate the sum of disk usage for that
101device filesystem also. In order to prevent du from traversing and to calculate
102from other mount points or filesystems, use the -x flag along with other du
103options. du –x / will exclude all mount points in /mnt/ for disk usage
104calculation.
105While using  du make sure that the directories or files it traverses have the proper read
106permissions.
107Finding the 10 largest size files from a given directory
108Finding large-size files is a regular task we come across. We regularly require to delete those
109huge size files or move them. We can easily find out large-size files using  du and  sort
110commands. The following one-line script can achieve this task:
111$ du -ak SOURCE_DIR | sort -nrk 1 | head
112Here  -a specifies all directories and files. Hence  du traverses the  SOURCE_DIR and
113calculates the size of all files. The first column of the output contains the size in Kilobytes
114since  -k is specified and the second column contains the file or folder name.
115www.it-ebooks.info
116Chapter 8
117271
118sort is used to perform numerical sort with column 1 and reverse it.  head is used to parse
119the first 10 lines from the output.
120For example:
121$ du -ak /home/slynux | sort -nrk 1 | head -n 4
12250220 /home/slynux
12343296 /home/slynux/.mozilla
12443284 /home/slynux/.mozilla/firefox
12543276 /home/slynux/.mozilla/firefox/8c22khxc.default
126One of the drawbacks of the above one-liner is that it includes directories in the result.
127However, when we need to find only the largest files and not directories we can improve the
128one-liner to output only the large-size files as follows:
129$ find . -type f -exec du -k {} \; | sort -nrk 1 | head
130We used  find to filter only files to  du rather than allow  du to traverse recursively by itself.
131Disk free information
132The  du command provides information about the usage, whereas  df provides information
133about free disk space. It can be used with and without  -h . When  -h is issued with  df it prints
134the disk space in human readable format.
135For example:
136$ df
137Filesystem 1K-blocks Used Available Use% Mounted on
138/dev/sda1 9611492 2276840 6846412 25% /
139none 508828 240 508588 1% /dev
140none 513048 168 512880 1% /dev/shm
141none 513048 88 512960 1% /var/run
142none 513048 0 513048 0% /var/lock
143none 513048 0 513048 0% /lib/init/rw
144none 9611492 2276840 6846412 25% /var/lib/
145ureadahead/debugfs
146$ df -h
147FilesystemSize Used Avail Use% Mounted on
148/dev/sda1 9.2G 2.2G 6.6G 25% /
149none 497M 240K 497M 1% /dev
150none 502M 168K 501M 1% /dev/shm
151www.it-ebooks.info
152Put on the Monitor’s Cap
153272
154none 502M 88K 501M 1% /var/run
155none 502M 0 502M 0% /var/lock
156none 502M 0 502M 0% /lib/init/rw
157none 9.2G 2.2G 6.6G 25% /var/lib/ureadahead/debugfs
158Calculating execution time for a command
159While testing an application or comparing different algorithms for a given problem, execution
160time taken by a program is very critical. A good algorithm should execute in minimum amount
161of time. There are several situations in which we need to monitor the time taken for execution
162by a program. For example, while learning about sorting algorithms, how do you practically
163state which algorithm is faster? The answer to this is to calculate the execution time for the
164same data set. Let's see how to do it.
165How to do it...
166time is a command that is available with any UNIX-like operating systems. You can prefix
167time with the command you want to calculate execution time, for example:
168$ time COMMAND
169The command will execute and its output will be shown. Along with output, the  time
170command appends the time taken in  stderr . An example is as follows:
171$ time ls
172test.txt
173next.txt
174real 0m0.008s
175user 0m0.001s
176sys 0m0.003s
177It will show real, user, and system times for execution. The three different times can be
178defined as follows:
179f Real is wall clock time—the time from start to finish of the call. This is all elapsed time
180including time slices used by other processes and the time that the process spends
181when blocked (for example, if it is waiting for I/O to complete).
182f User is the amount of CPU time spent in user-mode code (outside the kernel) within
183the process. This is only the actual CPU time used in executing the process. Other
184processes and the time that the process spends when blocked do not count towards
185this figure.
186www.it-ebooks.info
187Chapter 8
188273
189f Sys is the amount of CPU time spent in the kernel within the process. This means
190executing the CPU time spent in system calls within the kernel, as opposed to library
191code, which is still running in the user space. Like 'user time', this is only the CPU
192time used by the process.
193An executable binary of the time command is available at /usr/bin/time
194as well as a shell built-in named time exists. When we run time, it calls the
195shell built-in by default. The shell built-in time has limited options. Hence,
196we should use an absolute path for the executable (/usr/bin/time) for
197performing additional functionalities.
198We can write this time statistics to a file using the  -o filename option as follows:
199$ /usr/bin/time -o output.txt COMMAND
200The filename should always appear after the  –o flag.
201In order to append the time statistics to a file without overwriting, use the  -a flag along with
202the  -o option as follows:
203$ /usr/bin/time –a -o output.txt COMMAND
204We can also format the time outputs using format strings with the  -f option. A format string
205consists of parameters corresponding to specific options prefixed with  % . The format strings
206for real time, user time, and sys time are as follows:
207f Real time -  %e
208f User -  %U
209f sys -  %S
210By combining parameter strings, we can create formatted output as follows:
211$ /usr/bin/time -f "FORMAT STRING" COMMAND
212For example:
213$ /usr/bin/time -f "Time: %U" -a -o timing.log uname
214Linux
215Here  %U is the parameter for user time.
216When formatted output is produced, the formatted output of the command is written to the
217standard output and the output of the  COMMAND , which is timed, is written to standard error.
218We can redirect the formatted output using a redirection operator ( > ) and redirect the time
219information output using the ( 2> ) error redirection operator. For example:
220$ /usr/bin/time -f "Time: %U" uname> command_output.txt 2>time.log
221$ cat time.log
222Time: 0.00
223$ cat command_output.txt
224Linux
225www.it-ebooks.info
226Put on the Monitor’s Cap
227274
228Many details regarding a process can be collected using the  time command. The important
229details include, exit status, number of signals received, number of context switches made,
230and so on. Each parameter can be displayed by using a suitable format string.
231The following table shows some of the interesting parameters that can be used:
232Parameter Description
233%C Name and command-line arguments of the command being timed.
234%D Average size of the process's unshared data area, in kilobytes.
235%E Elapsed real (wall clock) time used by the process in [hours:]minutes:seconds.
236%x Exit status of the command.
237%k Number of signals delivered to the process.
238%W Number of times the process was swapped out of the main memory.
239%Z System's page size in bytes. This is a per-system constant, but varies between
240systems.
241%P Percentage of the CPU that this job got. This is just user + system times divided
242by the total running time. It also prints a percentage sign.
243%K Average total (data + stack + text) memory usage of the process, in kilobytes.
244%w Number of times that the program was context-switched voluntarily, for instance
245while waiting for an I/O operation to complete.
246%c Number of times the process was context-switched involuntarily (because the
247time slice expired).
248For example, the page size can be displayed using the  %Z parameters as follows:
249$ /usr/bin/time -f "Page size: %Z bytes" ls> /dev/null
250Page size: 4096 bytes
251Here the output of the timed command is not required and hence the standard output is
252directed to the  /dev/null device in order to prevent it from writing to the terminal.
253More format strings parameters are available. Read  man time for more details.
254Information about logged users, boot logs,
255and failure boot
256Collecting information about the operating environment, logged in users, the time for which
257the computer has been powered on, and any boot failures are very helpful. This recipe will go
258through a few commands used to gather information about a live machine.
259Getting ready
260This recipe will introduce the commands  who ,  w ,  users ,  uptime ,  last , and  lastb .
261www.it-ebooks.info
262Chapter 8
263275
264How to do it...
265To obtain information about users currently logged in to the machine use:
266$ who
267slynux pts/0 2010-09-29 05:24 (slynuxs-macbook-pro.local)
268slynux tty7 2010-09-29 07:08 (:0)
269Or:
270$ w
27107:09:05 up 1:45, 2 users, load average: 0.12, 0.06, 0.02
272USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
273slynux pts/0 slynuxs 05:24 0.00s 0.65s 0.11s sshd: slynux
274slynux tty7 :0 07:08 1:45m 3.28s 0.26s gnome-session
275It will provide information about logged in users, the pseudo TTY used by the users, the
276command that is currently executing from the pseudo terminal, and the IP address from
277which the users have logged in. If it is localhost, it will show the hostname.  who and  w format
278outputs with slight difference. The  w command provides more detail than  who .
279TTY is the device file associated with a text terminal. When a terminal is newly spawned by
280the user, a corresponding device is created in  /dev/ (for example,  /dev/pts/3 ). The device
281path for the current terminal can be found out by typing and executing the command  tty .
282In order to list the users currently logged in to the machine, use:
283$ users
284Slynux slynux slynux hacker
285If a user has opened multiple pseudo terminals, it will show that many entries for the same
286user. In the above output, the user  slynux has opened three pseudo terminals. The easiest
287way to print unique users is to use  sort and  uniq to filter as follows:
288$ users | tr ' ' '\n' | sort | uniq
289slynux
290hacker
291We have used  tr to replace  ' ' with  '\n' . Then combination of  sort and  uniq will produce
292unique entries for each user.
293In order to see how long the system has been powered on, use:
294$ uptime
29521:44:33 up 3:17, 8 users, load average: 0.09, 0.14, 0.09
296www.it-ebooks.info
297Put on the Monitor’s Cap
298276
299The time that follows the word  up indicates the time for which the system has been powered
300on. We can write a simple one-liner to extract the uptime only.
301Load average in uptime's output is a parameter that indicates system load. This is explained
302in more detail in the chapter, Administration Calls!. In order to get information about previous
303boot and user logged sessions, use:
304$ last
305slynux tty7 :0 Tue Sep 28 18:27 still logged in
306reboot system boot 2.6.32-21-generi Tue Sep 28 18:10 - 21:46 (03:35)
307slynux pts/0 :0.0 Tue Sep 28 05:31 - crash (12:39)
308The  last command will provide information about logged in sessions. It is actually a log of
309system logins that consists of information such as  tty from which it has logged in, login time,
310status, and so on.
311The  last command uses the log file  /var/log/wtmp for input log data. It is also possible to
312explicitly specify the log file for the last command using the  –f option. For example:
313$ last –f /var/log/wtmp
314In order to obtain info about login sessions for a single user, use:
315$ last USER
316Get information about reboot sessions as follows:
317$ last reboot
318reboot system boot 2.6.32-21-generi Tue Sep 28 18:10 - 21:48 (03:37)
319reboot system boot 2.6.32-21-generi Tue Sep 28 05:14 - 21:48 (16:33)
320In order to get information about failed user login sessions use:
321# lastb
322test tty8 :0 Wed Dec 15 03:56 - 03:56 (00:00)
323slynux tty8 :0 Wed Dec 15 03:55 - 03:55 (00:00)
324You should run  lastb as the root user.
325Printing the 10 most frequently-used
326commands
327Terminal is the tool used to access the shell prompt where we type and execute commands.
328Users run many commands in the shell. Many of them are frequently used. A user's nature
329can be identified easily by looking at the commands he frequently uses. This recipe is a small
330exercise to find out 10 most frequently-used commands.
331www.it-ebooks.info
332Chapter 8
333277
334Getting ready
335Bash keeps track of previously typed commands by the user and stores in the file  ~/.bash_
336history . But it only keeps a specific number (say 500) of the recently executed commands.
337The history of commands can be viewed by using the command history or  cat ~/.bash_
338history . We will use this for finding out frequently-used commands.
339How to do it...
340We can get the list of commands from  ~/.bash_history , take only the command excluding
341the arguments, count the occurrence of each command, and find out the 10 commands with
342the highest count.
343The following script can be used to find out frequently-used commands:
344#!/bin/bash
345#Filename: top10_commands.sh
346#Description: Script to list top 10 used commands
347printf "COMMAND\tCOUNT\n" ;
348cat ~/.bash_history | awk '{ list[$1]++; } \
349END{
350for(i in list)
351{
352printf("%s\t%d\n",i,list[i]); }
353}'| sort -nrk 2 | head
354A sample output is as follows:
355$ ./top10_commands.sh
356COMMAND COUNT
357ping 80
358ls 56
359cat 35
360ps 34
361sudo 26
362du 26
363cd 26
364ssh 22
365sftp 22
366clear 21
367www.it-ebooks.info
368Put on the Monitor’s Cap
369278
370How it works...
371In the above script, the history file  ~/.bash_history is the source file used. The source
372input is passed to  awk through a pipe. Inside  awk , we have an associative array list. This
373array can use command names as index and it stores the count of the commands in array
374locations. Hence for each arrival or occurrence of a command it will increment by one
375( list[$1]++ ).  $1 is used as the index.  $1 is the first word of text in a line input. If  $0
376were used it would contain all the arguments for the command also. For example, if  ssh
377192.168.0.4 is a line from  .bash_history ,  $0 equals to  ssh 192.168.0.4 and  $1
378equals to  ssh .
379Once all the lines of the history files are traversed, we will have the array with command names
380as indexes and their count as the value. Hence command names with maximum count values
381will be the commands most frequently used. Hence in the  END{} block of  awk , we traverse
382through the indexes of commands and print all command names and their counts.  sort -nrk
3832 will perform a numeric sort based on the second column ( COUNT ) and reverse it. Hence we
384use the  head command to extract only the first 10 commands from the list. You can customize
385the top 10 to top 5 or any other number by using the argument  head -n NUMBER .
386Listing the top 10 CPU consuming process
387in a hour
388CPU time is a major resource and sometimes we require to keep track of the processes that
389consume the most CPU cycles in a period of time. In regular desktops or laptops, it might not
390be an issue that the CPU is heavily consumed. However, for a server that handles numerous
391requests, CPU is a critical resource. By monitoring the CPU usage for a certain period we can
392identify the processes that keep the CPU busy all the time and optimize them to efficiently
393use the CPU or to debug them due to any other issues. This recipe is a practice with process
394monitoring and logging.
395Getting ready
396ps is a command used for collecting details about the processes running on the system. It can
397be used to gather details such as CPU usage, commands under execution, memory usage,
398status of process, and so on. Processes that consume the CPU for one hour can be logged,
399and the top 10 can be determined by proper usage of  ps and text processing. For more details
400on the  ps command, see the chapter: Administration Calls!.
401www.it-ebooks.info
402Chapter 8
403279
404How to do it...
405Let's go through the following shell script for monitoring and calculating CPU usages in one hour:
406#!/bin/bash
407#Name: pcpu_usage.sh
408#Description: Script to calculate cpu usage by processes for 1 hour
409SECS=3600
410UNIT_TIME=60
411#Change the SECS to total seconds for which monitoring is to be
412performed.
413#UNIT_TIME is the interval in seconds between each sampling
414STEPS=$(( $SECS / $UNIT_TIME ))
415echo Watching CPU usage... ;
416for((i=0;i<STEPS;i++))
417do
418ps -eo comm,pcpu | tail -n +2 >> /tmp/cpu_usage.$$
419sleep $UNIT_TIME
420done
421echo
422echo CPU eaters :
423cat /tmp/cpu_usage.$$ | \
424awk '
425{ process[$1]+=$2; }
426END{
427for(i in process)
428{
429printf("%-20s %s",i, process[i] ;
430}
431}' | sort -nrk 2 | head
432rm /tmp/cpu_usage.$$
433#Remove the temporary log file
434A sample output is as follows:
435$ ./pcpu_usage.sh
436Watching CPU usage...
437CPU eaters :
438Xorg 20
439www.it-ebooks.info
440Put on the Monitor’s Cap
441280
442firefox-bin 15
443bash 3
444evince 2
445pulseaudio 1.0
446pcpu.sh 0.3
447wpa_supplicant 0
448wnck-applet 0
449watchdog/0 0
450usb-storage 0
451How it works...
452In the above script, the major input source is  ps -eocomm, pcpu .  comm stands for
453command name and  pcpu stands for the CPU usage in percent. It will output all the process
454names and the CPU usage in percent. For each process there exists a line in the output. Since
455we need to monitor the CPU usage for one hour, we repeatedly take usage statistics using
456ps -eo comm,pcpu | tail -n +2 and append to a file  /tmp/cpu_usage.$$ running
457inside a  for loop with 60 seconds wait in each iteration. This wait is provided by  sleep 60 . It
458will execute  ps once in each minute.
459tail -n +2 is used to strip off the header and  COMMAND %CPU in the  ps output.
460$$ in  cpu_usage.$$ signifies that it is the process ID of the current script. Suppose PID is
4611345, during execution it will be replaced as  /tmp/cpu_usage.1345 . We place this file in  /
462tmp since it is a temporary file.
463The statistics file will be ready after one hour and will contain 60 entries corresponding to
464the process status for each minute. Then  awk is used to sum the total CPU usage for each
465process. An associative array process is used for the summation of CPU usages. It uses
466the process name as an array index. Finally, it sorts the result with a numeric reverse sort
467according to the total CPU usage and pass through head to obtain top 10 usage entries.
468See also
469f Basic awk primer of Chapter 4, explains the awk command
470f head and tail - printing the last or first ten lines of Chapter 3, explains the tail
471command
472www.it-ebooks.info
473Chapter 8
474281
475Monitoring command outputs with watch
476We might need to continuously watch the output of a command for a period of time in equal
477intervals. For example, for a large file copy, we need to watch the growing file size. In order to do
478that, newbies repeatedly type commands and press return a number of times. Instead we can
479use the watch command to view output repeatedly. This recipe explains how to do that.
480How to do it...
481The  watch command can be used to monitor the output of a command on the terminal at
482regular intervals. The syntax of the  watch command is as follows:
483$ watch COMMAND
484For example:
485$ watch ls
486Or:
487$ watch 'COMMANDS'
488For example:
489$ watch 'ls -l | grep "^d"'
490# list only directories
491This command will update the output at a default interval of two seconds.
492We can also specify the time interval at which the output needs to be updated, by using  -n
493SECONDS . For example:
494$ watch -n 5 'ls -l'
495#Monitor the output of ls -l at regular intervals of 5 seconds
496There's more
497Let's explore an additional feature of the  watch command.
498Highlighting the differences in watch output
499In  watch , there is an option for updating the differences that occur during the execution of
500the command at an update interval to be highlighted using colors. Difference highlighting can
501be enabled by using the  -d option as follows:
502$ watch -d 'COMMANDS'
503www.it-ebooks.info
504Put on the Monitor’s Cap
505282
506Logging access to files and directories
507Logging of file and directory access is very helpful to keep track of changes that are
508happening to files and folders. This recipe will describe how to log user accesses.
509Getting ready
510The  inotifywait command can be used to gather information about file accesses. It doesn't
511come by default with every Linux distro. You have to install the  inotify-tools package by
512using a package manager. It also requires the Linux kernel to be compiled with inotify support.
513Most of the new GNU/Linux distributions come with inotify enabled in the kernel.
514How to do it...
515Let's walk through the shell script to monitor the directory access:
516#/bin/bash
517#Filename: watchdir.sh
518#Description: Watch directory access
519path=$1
520#Provide path of directory or file as argument to script
521inotifywait -m -r -e create,move,delete $path -q
522A sample output is as follows:
523$ ./watchdir.sh .
524./ CREATE new
525./ MOVED_FROM new
526./ MOVED_TO news
527./ DELETE news
528How it works...
529The previous script will log events create, move, and delete files and folders from the given
530path. The  -m option is given for monitoring the changes continuously rather than going to exit
531after an event happens.  -r is given for enabling a recursive watch the directories.  -e specifies
532the list of events to be watched.  -q is to reduce the verbose messages and print only required
533ones. This output can be redirected to a log file.
534We can add or remove the event list. Important events available are as follows:
535www.it-ebooks.info
536Chapter 8
537283
538Event Description
539access When some read happens to a file.
540modify When file contents are modified.
541attrib When metadata is changed.
542move When a file undergoes move operation.
543create When a new file is created.
544open When a file undergoes open operation.
545close When a file undergoes close operation.
546delete When a file is removed.
547Logfile management with logrotate
548Logfiles are essential components of a Linux system's maintenance. Logfiles help to keep
549track of events happening on different services on the system. This helps the sysadmin
550to debug issues and also provides statistics on events happening on the live machine.
551Management of logfiles is required because as time passes the size of a logfile gets bigger
552and bigger. Therefore, we use a technique called rotation to limit the size of the logfile and if
553the logfile reaches a size beyond the limit, it will strip the logfile and store the older entries
554from the logfile in an archive. Hence older logs can be stored and kept for future reference.
555Let's see how to rotate logs and store them.
556Getting ready
557logrotate is a command every Linux system admin should know. It helps to restrict the size
558of logfile to the given SIZE. In a logfile, the logger appends information to the log file. Hence
559the recent information appears at the bottom of the log file.  logrotate will scan specific
560logfiles according to the configuration file. It will keep the last 100 kilobytes (for example,
561specified SIZE = 100k) from the logfile and move rest of the data (older log data) to a new
562file  logfile_name.1 with older entries. When more entries occur in the logfile ( logfile_
563name.1 ) and it exceeds the SIZE, it updates the logfile with recent entries and creates
564logfile_name.2 with older logs. This process can easily be configured with  logrotate .
565logrotate can also compress the older logs as  logfile_name.1.gz ,  logfile_name2.
566gz , and so on. The option for whether older log files are to be compressed or not is available
567with the  logrotate configuration.
568How to do it...
569logrotate has the configuration directory at  /etc/logrotate.d . If you look at this
570directory by listing contents, many other logfile configurations can be found.
571www.it-ebooks.info
572Put on the Monitor’s Cap
573284
574We can write our custom configuration for our logfile (say  /var/log/program.log ) as follows:
575$ cat /etc/logrotate.d/program
576/var/log/program.log {
577missingok
578notifempty
579size 30k
580compress
581weekly
582rotate 5
583create 0600 root root
584}
585Now the configuration is complete.  /var/log/program.log in the configuration specifies
586the logfile path. It will archive old logs in the same directory path. Let's see what each of these
587parameters are:
588Parameter Description
589missingok Ignore if the logfile is missing and return without rotating the log.
590notifempty Only rotate the log if the source logfile is not empty.
591size 30k Limit the size of the logfile for which the rotation is to be made. It
592can be 1M for 1MB.
593compress Enable compression with gzip for older logs.
594weekly Specify the interval at which the rotation is to be performed. It
595can be weekly, yearly, or daily.
596rotate 5 It is the number of older copies of logfile archives to be kept.
597Since 5 is specified, there will be program.log.1.gz,
598program.log.2.gz, and so on till program.log.5.gz.
599create 0600 root root Specify the mode, user, and the group of the logfile archive to be
600created.
601The options specified in the table are optional; we can specify the required options only in
602the  logrotate configuration file. There are numerous options available with  logrotate .
603Please refer to the man pages ( http://linux.die.net/man/8/logrotate ) for more
604information on  logrotate .
605www.it-ebooks.info
606Chapter 8
607285
608Logging with syslog
609Logfiles are an important component of applications that provide services to the users. An
610applications writes status information to its logfile while it is running. If any crash occurs or we
611need to enquire some information about the service, we look into the logfile. You can find lots
612of logfiles related to different daemons and applications in the  /var/log directory. It is the
613common directory for storing log files. If you read through a few lines of the logfiles, you can
614see that lines in the log are in a common format. In Linux, creating and writing log information
615to logfiles at  /var/log are handled by a protocol called syslog. It is handled by the  syslogd
616daemon. Every standard application makes use of syslog for logging information. In this recipe,
617we will discuss how to make use of  syslogd for logging information from a shell script.
618Getting ready
619Logfiles are useful for helping you deduce what is going wrong with a system. Hence while
620writing critical applications, it is always a good practice to log the progress of application with
621messages into a logfile. We will learn the command logger to log into log files with  syslogd .
622Before getting to know how to write into logfiles, let's go through a list of important logfiles
623used in Linux:
624Log file Description
625/var/log/boot.log Boot log information.
626/var/log/httpd Apache web server log.
627/var/log/messages Post boot kernel information.
628/var/log/auth.log User authentication log.
629/var/log/dmesg System boot up messages.
630/var/log/mail.log Mail server log.
631/var/log/Xorg.0.log X Server log.
632How to do it...
633In order to log to the syslog file  /var/log/messages use:
634$ logger LOG_MESSAGE
635For example:
636$ logger This is a test log line
637$ tail -n 1 /var/log/messages
638Sep 29 07:47:44 slynux-laptop slynux: This is a test log line
639www.it-ebooks.info
640Put on the Monitor’s Cap
641286
642The logfile  /var/log/messages is a general purpose logfile. When the  logger command is
643used, it logs to  /var/log/messages by default. In order to log to the syslog with a specified
644tag, use:
645$ logger -t TAG This is a message
646$ tail -n 1 /var/log/messages
647Sep 29 07:48:42 slynux-laptop TAG: This is a message
648syslog handles a number of logfiles in  /var/log . However, while logger sends a message,
649it uses the tag string to determine in which logfile it needs to be logged.  syslogd decides
650to which file the log should be made by using the  TAG associated with the log. You can see
651the tag strings and associated logfiles from the configuration files located in the  /etc/
652rsyslog.d/ directory.
653In order to log to the system log with the last line from another logfile use:
654$ logger -f /var/log/source.log
655See also
656f head and tail - printing the last or first 10 lines of Chapter 3, explains the head and
657tail commands
658Monitoring user logins to find intruders
659Logfiles can be used to gather details about the state of the system. Here is an interesting
660scripting problem statement:
661We have a system connected to the Internet with SSH enabled. Many attackers are trying to
662log in to the system. We need to design an intrusion detection system by writing a shell script.
663Intruders are defined as users who are trying to log in with multiple attempts for more than
664two minutes and whose attempts are all failing. Such users are to be detected and a report
665should be generated with the following details:
666f User account to which a login is attempted
667f Number of attempts
668f IP address of the attacker
669f Host mapping for IP address
670f Time range for which login attempts are performed.
671www.it-ebooks.info
672Chapter 8
673287
674Getting started
675We can write a shell script that can scan through the logfiles and gather the required
676information from them. Here, we are dealing with SSH login failures. The user authentication
677session log is written to the log file  /var/log/auth.log . The script should scan the log file
678to detect the failure login attempts and perform different checks on the log to infer the data.
679We can use the  host command to find out the host mapping from the IP address.
680How to do it…
681Let's write an intruder detection script that can generate a report of intruders by using the
682authentication logfile as follows:
683#!/bin/bash
684#Filename: intruder_detect.sh
685#Description: Intruder reporting tool with auth.log input
686AUTHLOG=/var/log.auth.log
687if [[ -n $1 ]];
688then
689AUTHLOG=$1
690echo Using Log file : $AUTHLOG
691fi
692LOG=/tmp/valid.$$.log
693grep -v "invalid" $AUTHLOG > $LOG
694users=$(grep "Failed password" $LOG | awk '{ print $(NF-5) }' | sort |
695uniq)
696printf "%-5s|%-10s|%-10s|%-13s|%-33s|%s\n" "Sr#" "User" "Attempts" "IP
697address" "Host_Mapping" "Time range"
698ucount=0;
699ip_list="$(egrep -o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" $LOG | sort |
700uniq)"
701for ip in $ip_list;
702do
703grep $ip $LOG > /tmp/temp.$$.log
704for user in $users;
705do
706grep $user /tmp/temp.$$.log> /tmp/$$.log
707cut -c-16 /tmp/$$.log > $$.time
708tstart=$(head -1 $$.time);
709start=$(date -d "$tstart" "+%s");
710tend=$(tail -1 $$.time);
711end=$(date -d "$tend" "+%s")
712limit=$(( $end - $start ))
713www.it-ebooks.info
714Put on the Monitor’s Cap
715288
716if [ $limit -gt 120 ];
717then
718let ucount++;
719IP=$(egrep -o "[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+" /tmp/$$.log | head
720-1 );
721TIME_RANGE="$tstart-->$tend"
722ATTEMPTS=$(cat /tmp/$$.log|wc -l);
723HOST=$(host $IP | awk '{ print $NF }' )
724printf "%-5s|%-10s|%-10s|%-10s|%-33s|%-s\n" "$ucount" "$user"
725"$ATTEMPTS" "$IP" "$HOST" "$TIME_RANGE";
726fi
727done
728done
729rm /tmp/valid.$$.log /tmp/$$.log $$.time /tmp/temp.$$.log 2> /dev/null
730A sample output is as follows:
731How it works…
732In the  intruder_detect.sh script, we use the  auth.log file as input. We can either
733provide a log file as input to the script by using a command-line argument to the script or, by
734default, it reads the  /var/log/auth.log file. We need to log details about login attempts
735for valid user names only. When a login attempt for an invalid user occurs, a log similar to
736Failed password for invalid user bob from 203.83.248.32 port 7016
737ssh2 is logged to  auth.log . Hence, we need to exclude all lines in the log file having the
738word "invalid". The  grep command with the invert option ( -v ) is used to remove all logs
739corresponding to invalid users. The next step is to find out the list of users for which login
740attempts occurred and failed. The SSH will log lines similar to  sshd[21197]: Failed
741password for bob1 from 203.83.248.32 port 50035 ssh2 for a failed password.
742www.it-ebooks.info
743Chapter 8
744289
745Hence we should find all the lines with words "failed password". Now all the unique IP
746addresses are to be found out for extracting all the log lines corresponding to each IP address.
747The list of IP address is extracted by using a regular expression for IP address and the  egrep
748command. A  for loop is used to iterate through IP address and the corresponding log lines
749are found using  grep and are written to a temporary file. The sixth word from the last word
750in the log line is the user name (for example, bob1 ). The  awk command is used to extract
751the sixth word from the last word.  NF returns the column number of the last word. Therefore,
752NF-5 gives the column number of the sixth word from the last word. We use  sort  and  uniq
753commands to produce a list of users without duplication.
754Now we should collect the failed login log lines containing the name of each users. A  for loop
755is used for reading the lines corresponding to each user and the lines are written to a temporary
756file. The first 16 characters in each of the log lines is the timestamp. The  cut  command is used
757to extract the timestamp. Once we have all the timestamps for failed login attempts for a user,
758we should check the difference in time between the first attempt and the last attempt. The first
759log line corresponds to the first attempt and last log line corresponds to last attempt. We have
760used  head -1 to extract the first line and  tail -1 to extract the last line. Now we have a time
761stamp for first ( tstart ) and last attempt ( tends ) in string format. Using the  date command,
762we can convert the date in string representation to total seconds in UNIX Epoch time (the recipe,
763Getting, setting dates, and delays of Chapter 1, explains Epoch time).
764The variables start and end have a time in seconds corresponding to the start and end
765timestamps in the date string. Now, take the difference between them and check whether it
766exceeds two minutes (120 seconds). Thus, the particular user is termed as an intruder and the
767corresponding entry with details are to be produced as a log. IP addresses can be extracted
768from the log by using a regular expression for IP address and the  egrep command. The number
769of attempts is the number of log lines for the user. The number of lines can be found out by
770using the  wc command. The host name mapping can be extracted from the output of the host
771command by running with IP address as argument. The time range can be printed using the
772timestamp we extracted. Finally, the temporary files used in the script are removed.
773The above script is aimed only at illustrating a model for scanning the log and producing a
774report from it. It has tried to make the script smaller and simpler to leave out the complexity.
775Hence it has few bugs. You can improve the script by using better logic.
776Remote disk usage health monitor
777A network consists of several machines with different users. The network requires centralized
778monitoring of disk usage of remote machines. The system administrator of the network
779needs to log the disk usage of all the machines in the network every day. Each log line should
780contain details such as the date, IP address of the machine, device, capacity of the device,
781used space, free space, percentage usage, and health status. If the disk usage of any of the
782partitions in any remote machine exceeds 80 percent, the health status should be set to
783ALERT, else it should be set to SAFE. This recipe will illustrate how to write a monitoring script
784that can collect details from remote machines in a network.
785www.it-ebooks.info
786Put on the Monitor’s Cap
787290
788Getting ready
789We need to collect the disk usage statistics from each machine on the network, individually,
790and write a log file in the central machine. A script that collects the details and writes the
791log can be scheduled to run everyday at a particular time. The SSH can be used to log in to
792remote systems to collect disk usage data.
793How to do it…
794First we have to set up a common user account on all the remote machines in the network.
795It is for the disklog program to log in to the system. We should configure auto-login with SSH
796for that particular user (the recipe, Password-less auto-login with SSH in Chapter 7, explains
797configuration of auto-login). We assume that there is a user called test in all remote machines
798configured with auto-login. Let's go through the shell script:
799#!/bin/bash
800#Filename: disklog.sh
801#Description: Monitor disk usage health for remote systems
802logfile="diskusage.log"
803if [[ -n $1 ]]
804then
805logfile=$1
806fi
807if [ ! -e $logfile ]
808then
809printf "%-8s %-14s %-9s %-8s %-6s %-6s %-6s %s\n" "Date" "IP
810address" "Device" "Capacity" "Used" "Free" "Percent" "Status" >
811$logfile
812fi
813IP_LIST="127.0.0.1 0.0.0.0"
814#provide the list of remote machine IP addresses
815(
816for ip in $IP_LIST;
817do
818ssh slynux@$ip 'df -H' | grep ^/dev/ > /tmp/$$.df
819while read line;
820do
821cur_date=$(date +%D)
822printf "%-8s %-14s " $cur_date $ip
823echo $line | awk '{ printf("%-9s %-8s %-6s %-6s
824%-8s",$1,$2,$3,$4,$5); }'
825pusg=$(echo $line | egrep -o "[0-9]+%")
826www.it-ebooks.info
827Chapter 8
828291
829pusg=${pusg/\%/};
830if [ $pusg -lt 80 ];
831then
832echo SAFE
833else
834echo ALERT
835fi
836done< /tmp/$$.df
837done
838) >> $logfile
839We can schedule using the cron utility to run the script at regular intervals. For example, to run
840the script everyday at 10 am, write the following entry in the  crontab :
84100 10 * * * /home/path/disklog.sh /home/user/diskusg.log
842Run the command  crontab –e . Add the above line and save the text editor.
843You can run the script manually as follows:
844$ ./disklog.sh
845A sample output log for the above script is as follows:
846How it works…
847In the  disklog.sh script, we can provide the logfile path as a command-line argument or
848else it will use the default logfile. If the logfile does not exists, it will write the logfile header
849text into the new file.  –e $logfile is used to check whether the file exists or not. The list of
850IP addresses of remote machines are stored in the variable  IP_LIST delimited with spaces.
851It should be made sure that all the remote systems listed in the  IP_LIST have a common
852user  test with auto-login with SSH configured. A  for loop is used to iterate through each of
853the IP addresses. A remote command  df –H is executed to get the disk free usage data using
854the  ssh command. It is stored in a temporary file. A  while loop is used to read the file line
855by line. Data is extracted using  awk and is printed. The date is also printed. The percentage
856usage is extracted using the  egrep command and  % is replaced with  none to get the numeric
857value of percent. It is checked whether the percentage value exceeds 80. If it is less than 80,
858the status is set as SAFE and if greater than or equal to 80, the status is set as ALERT. The
859entire printed data should be redirected to the logfile. Hence the portion of code is enclosed
860in a subshell  () and the standard output is redirected to the logfile.
861www.it-ebooks.info
862Put on the Monitor’s Cap
863292
864See also
865f Scheduling with cron of Chapter 9, explains the crontab command
866Finding out active user hours on a system
867Consider a web server with shared hosting. Many users log in to and log out of the server
868every day. The user activity gets logged in the server's system log. This recipe is a practice task
869to make use of the system logs and to find out how many hours each of the users have spent
870on the server and rank them according to the total usage hours. A report should be generated
871with the details, such as the rank, user, first logged in date, last logged in date, number of
872times logged in, and total usage hours. Let's see how we can approach this problem.
873Getting ready
874The  last command is used to list the details about the login sessions of the users in a
875system. The log data is stored in the  /var/log/wtmp file. By individually adding the session
876hours for each user we can find out the total usage hours.
877How to do it…
878Let's go through the script to find out active users and generate the report:
879#!/bin/bash
880#Filename: active_users.sh
881#Description: Reporting tool to find out active users
882log=/var/log/wtmp
883if [[ -n $1 ]];
884then
885log=$1
886fi
887printf "%-4s %-10s %-10s %-6s %-8s\n" "Rank" "User" "Start" "Logins"
888"Usage hours"
889last -f $log | head -n -2 > /tmp/ulog.$$
890cat /tmp/ulog.$$ | cut -d' ' -f1 | sort | uniq> /tmp/users.$$
891(
892while read user;
893do
894grep ^$user /tmp/ulog.$$ > /tmp/user.$$
895seconds=0
896while read t
897do
898www.it-ebooks.info
899Chapter 8
900293
901s=$(date -d $t +%s 2> /dev/null)
902let seconds=seconds+s
903done< <(cat /tmp/user.$$ | awk '{ print $NF }' | tr -d ')(')
904firstlog=$(tail -n 1 /tmp/user.$$ | awk '{ print $5,$6 }')
905nlogins=$(cat /tmp/user.$$ | wc -l)
906hours=$(echo "$seconds / 60.0" | bc)
907printf "%-10s %-10s %-6s %-8s\n" $user "$firstlog" $nlogins $hours
908done< /tmp/users.$$
909) | sort -nrk 4 | awk '{ printf("%-4s %s\n", NR, $0) }'
910rm /tmp/users.$$ /tmp/user.$$ /tmp/ulog.$$
911A sample output is as follows:
912$ ./active_users.sh
913Rank User Start Logins Usage hours
9141 easyibaa Dec 11 531 11437311943
9152 demoproj Dec 10 350 7538718253
9163 kjayaram Dec 9 213 4587849555
9174 cinenews Dec 11 85 1830831769
9185 thebenga Dec 10 54 1163118745
9196 gateway2 Dec 11 52 1120038550
9207 soft132 Dec 12 49 1055420578
9218 sarathla Nov 1 45 969268728
9229 gtsminis Dec 11 41 883107030
92310 agentcde Dec 13 39 840029414
924How it works…
925In the  active_users.sh script, we can either provide the  wtmp log file as a command-line
926argument or it will use the  defaulwtmp log file. The  last –f command is used to print the
927logfile contents. The first column in the logfile is the user name. By using  cut  we extract the
928first column from the logfile. Then the unique users are found out by using the  sort  and
929uniq  commands. Now for each user, the log lines corresponding to their login sessions are
930found out using  grep and are written to a temporary file. The last column in the last log is the
931duration for which the user logged a session. Hence in order to find out the total usage hours
932for a user, the session durations are to be added. The usage duration is in  (HOUR:SEC)
933format and it is to be converted into seconds using the date command.
934www.it-ebooks.info
935Put on the Monitor’s Cap
936294
937In order to extract the session hours for the users, we have used the  awk command. For
938removing the parenthesis,  tr –d is used. The list of usage hour string is passed to the
939standard input for the  while  loop using the  <( COMMANDS ) operator. It acts as a file input.
940Each hour string, by using the  date command, is converted into seconds and added to the
941variable  seconds . The first login time for a user is in the last line and it is extracted. The
942number of login attempts is the number of log lines. In order to calculate the rank of each
943user according to the total usage hours, the data record is to be sorted in the descending
944order with usage hours as the key. For specifying the number reverse sort  -nr  option is used
945along with the  sort command.  –k4 is used to specify the key column (usage hour). Finally,
946the output of the sort is passed to  awk . The  awk  command prefixes a line number to each of
947the lines, which becomes the rank for each user.
948www.it-ebooks.info
9499
950Administration Calls
951In this chapter, we will cover:
952f Gathering information about processes
953f Killing processes and send or respond to signals
954f Which, whereis, file, whatis, and loadavg explained
955f Sending messages to user terminals
956f Gathering system information
957f Using /proc – gathering information
958f Scheduling with cron
959f Writing and reading MySQL database from Bash
960f User administration script
961f Bulk image resizing and format conversion
962Introduction
963A GNU/Linux ecosystem consists of running programs, services, connected devices,
964filesystems, users, and a lot more. Having an overview of the entire system and managing the
965OS as a whole, according to the way we want, is the primary purpose of system administration.
966One should be armed with the knowledge of commonly-used commands and proper usage
967practices to gather system information and manage resources to write script and automation
968tools that perform management tasks. This chapter will introduce several commands and
969methods for gathering information about your system and make use of these commands to
970write administration scripts.
971www.it-ebooks.info
972Administration Calls
973296
974Gathering information about processes
975Processes are the running instance of a program. Several processes run on a computer and
976each process is assigned a unique identification number called a process ID. It is an integer.
977Multiple instances of the same program with the same name can be executed at a time. But
978they all will have different process IDs. A process consists of several attributes, such as which
979user owns the process, the amount of memory used by the program, the amount of CPU used by
980the program, and so on. This recipe will go through how to gather information about processes.
981Getting ready
982Important commands related to process management are  top ,  ps , and  pgrep . Let's see how
983we can gather information about processes.
984How to do it...
985ps is an important tool for gathering information about the processes.  ps provides information
986on a user who owns the process, the time when a process started, command path used for
987executing the process, process ID (PID), the terminal it is attached with (TTY), the memory
988used by the process, CPU used by the process, and so on. For example:
989$ ps
990PID TTY TIME CMD
9911220 pts/0 00:00:00 bash
9921242 pts/0 00:00:00 ps
993The  ps command is usually used with a set of parameters. When it is run without any
994parameter,  ps will display processes that are running on the current (TTY) terminal. The first
995column shows the process ID (PID), the second column is the TTY (terminal), the third column
996is how much time has elapsed since the process started, and finally CMD (the command).
997In order to show more columns consisting of more information, use  -f (this stands for full)
998as follows:
999$ ps -f
1000UID PID PPID C STIME TTY TIME CMD
1001slynux 1220 1219 0 18:18 pts/0 00:00:00 -bash
1002slynux 1587 1220 0 18:59 pts/0 00:00:00 ps -f
1003The above  ps commands are not useful since it does not provide any information about
1004processes other than the ones attached to the current terminal. In order to get information
1005about every process running on the system, add the  -e (every) option. The  -ax (all) option
1006will also produce an identical output.
1007www.it-ebooks.info
1008Chapter 9
1009297
1010The -x argument along with -a specifies to remove the TTY restriction
1011imparted, by default, by ps. Usually, using ps without arguments prints
1012processes that are attached to terminal only.
1013Run  ps -e or  ps –ef else  ps -ax or  ps –axf :
1014$ ps -e | head
1015PID TTY TIME CMD
10161 ? 00:00:00 init
10172 ? 00:00:00 kthreadd
10183 ? 00:00:00 migration/0
10194 ? 00:00:00 ksoftirqd/0
10205 ? 00:00:00 watchdog/0
10216 ? 00:00:00 events/0
10227 ? 00:00:00 cpuset
10238 ? 00:00:00 khelper
10249 ? 00:00:00 netns
1025It will be a long list. The example filters the output using  head so we only get the first 10 entries.
1026The  ps command supports several information to be displayed along with the process name
1027and process ID. By default,  ps shows the information as different columns. Most of them are
1028not useful for us. We can actually specify the columns to be displayed using the  -o flag. Hence
1029we can print only the required columns. Different parameters associated with a process
1030are specified with options for that parameter. The list of parameters and usage of  -o are
1031discussed next.
1032In order to display the required columns of output using  ps , use:
1033$ ps [OTHER OPTIONS] -o parameter1,parameter2,parameter3 ..
1034Parameters for -o are delimited by using the comma (,) operator. It should
1035be noted that there is no space in between the comma operator and next
1036parameter. Mostly, the -o option is combined with the -e (every) option
1037(-oe) since it should list every process running in the system. However, when
1038certain filters are used along with –o, such as those used for listing the
1039processes owned by specified users, -e is not used along with –o. Usage of
1040-e with a filter will nullify the filter and it will show all process entries.
1041www.it-ebooks.info
1042Administration Calls
1043298
1044An example is as follows. Here,  comm stands for COMMAND and  pcpu is percent of CPU usage:
1045$ ps -eo comm,pcpu | head
1046COMMAND %CPU
1047init 0.0
1048kthreadd 0.0
1049migration/0 0.0
1050ksoftirqd/0 0.0
1051watchdog/0 0.0
1052events/0 0.0
1053cpuset 0.0
1054khelper 0.0
1055netns 0.0
1056The different parameters that can be used with the  -o option and their descriptions are
1057as follows:
1058Parameter Description
1059pcpu Percentage of CPU
1060pid Process ID
1061ppid Parent Process ID
1062pmem Percentage of Memory
1063comm Executable file name
1064cmd Simple command
1065user The user who started process
1066nice The priority (niceness)
1067time Cumulative CPU time
1068etime Elapsed time since the process started
1069tty The associated TTY device
1070euid The effective user
1071stat Process state
1072There's more...
1073Let's go through additional usage examples of process manipulation commands.
1074top
1075top is a very important command for system administrators. The  top command will, by
1076default, output a list of top CPU consuming processes. The command is used as follows:
1077$ top
1078It will display several parameters along with the top CPU consuming processes.
1079www.it-ebooks.info
1080Chapter 9
1081299
1082Sorting ps output with respect to a parameter
1083Output of the  ps command can be sorted according to specified columns with the  --sort
1084parameter.
1085The ascending or descending order can be specified by using the  + (ascending) or  -
1086(descending) prefix to the parameter as follows:
1087$ ps [OPTIONS] --sort -paramter1,+parameter2,parameter3..
1088For example, to list the top 10 CPU consuming processes use:
1089$ ps -eo comm,pcpu --sort -pcpu | head
1090COMMAND %CPU
1091Xorg 0.1
1092hald-addon-stor 0.0
1093ata/0 0.0
1094scsi_eh_0 0.0
1095gnome-settings- 0.0
1096init 0.0
1097hald 0.0
1098pulseaudio 0.0
1099gdm-simple-gree 0.0
1100Here processes are sorted in descending order by percentage of CPU usage and  head is
1101applied to extract the top 10 processes.
1102We can use  grep to extract entries in the  ps output related to a given process name or
1103another parameter. In order to find out entries about running bash processes use:
1104$ ps -eo comm,pid,pcpu,pmem | grep bash
1105bash 1255 0.0 0.3
1106bash 1680 5.5 0.3
1107Finding process ID when given command names
1108Suppose several instances of a command are being executed, we may need to identify the
1109process ID of the processes. This information can be found by using the  ps or the  pgrep
1110command. We can use  ps as follows:
1111$ ps -C COMMAND_NAME
1112Or:
1113$ ps -C COMMAND_NAME -o pid=
1114www.it-ebooks.info
1115Administration Calls
1116300
1117The  -o user defined format specifier was described in the earlier part of the recipe. But here
1118you can see  = appended with  pid . This is to remove the header PID in the output of  ps . In
1119order to remove headers for each column, append  = to the parameter. For example:
1120$ ps -C bash -o pid=
11211255
11221680
1123This command lists the process IDs of bash processes.
1124Alternately, there is a handy command called  pgrep . You should use  pgrep to get a quick list
1125of process IDs for a particular command. For example:
1126$ pgrep COMMAND
1127$ pgrep bash
11281255
11291680
1130pgrep requires only a portion of the command name as its input argument to
1131extract a Bash command, for example, pgrep ash or pgrep bas will also
1132work. But ps requires you to type the exact command.
1133pgrep accepts many more output-filtering options. In order to specify a delimiter character for
1134output rather than using a newline as the delimiter use:
1135$ pgrep COMMAND -d DELIMITER_STRING
1136$ pgrep bash -d ":"
11371255:1680
1138Specify a list of owners of the user for the matching processes as follows:
1139$ pgrep -u root,slynux COMMAND
1140In this command,  root and  slynux are users.
1141Return the count of matching processes as follows:
1142$ pgrep -c COMMAND
1143Filters with ps for real user or ID, effective user or ID
1144With  ps , it is possible to group processes based on the real and effective user name or ID
1145specified. Specified arguments can be used to filter the  ps output by checking whether each
1146entry belongs to a specific, effective user or real user from the list of arguments and shows
1147only the entries matching them. This can be done as follows:
1148www.it-ebooks.info
1149Chapter 9
1150301
1151f Specify an effective users list by using  -u EUSER1, EUSER2 and so on
1152f Specify a real users list by using  -U RUSER1, RUSER2 and so on
1153For example:
1154$ ps -u root -U root -o user,pcpu
1155This command will show all processes running with  root as the effective user ID and real
1156user ID, and will also show the user and percentage CPU usage columns.
1157Mostly, we find -o along with -e as -eo. But when filters are applied -o
1158should act alone as mentioned above.
1159TTY filter for ps
1160The  ps output can be selected by specifying the TTY to which the process is attached. Use the
1161-t option to specify the TTY list as follows:
1162$ ps -t TTY1, TTY2 ..
1163For example:
1164$ ps -t pts/0,pts/1
1165PID TTY TIME CMD
11661238 pts/0 00:00:00 bash
11671835 pts/1 00:00:00 bash
11681864 pts/0 00:00:00 ps
1169Information about process threads
1170Usually, information about process threads are hidden in the  ps output. We can show
1171information about threads in the  ps output by adding the  -L option. Then it will show two
1172columns NLWP and NLP. NLWP is the thread count for a process and NLP is the thread ID for
1173each entry in PS. For example:
1174$ ps -eLf
1175Or:
1176$ ps -eLf --sort -nlwp | head
1177UID PID PPID LWP C NLWP STIME TTY TIME CMD
1178root 647 1 647 0 64 14:39 ? 00:00:00 /usr/sbin/
1179console-kit-daemon --no-daemon
1180root 647 1 654 0 64 14:39 ? 00:00:00 /usr/sbin/
1181console-kit-daemon --no-daemon
1182www.it-ebooks.info
1183Administration Calls
1184302
1185root 647 1 656 0 64 14:39 ? 00:00:00 /usr/sbin/
1186console-kit-daemon --no-daemon
1187root 647 1 657 0 64 14:39 ? 00:00:00 /usr/sbin/
1188console-kit-daemon --no-daemon
1189root 647 1 658 0 64 14:39 ? 00:00:00 /usr/sbin/
1190console-kit-daemon --no-daemon
1191root 647 1 659 0 64 14:39 ? 00:00:00 /usr/sbin/
1192console-kit-daemon --no-daemon
1193root 647 1 660 0 64 14:39 ? 00:00:00 /usr/sbin/
1194console-kit-daemon --no-daemon
1195root 647 1 662 0 64 14:39 ? 00:00:00 /usr/sbin/
1196console-kit-daemon --no-daemon
1197root 647 1 663 0 64 14:39 ? 00:00:00 /usr/sbin/
1198console-kit-daemon --no-daemon
1199This command lists 10 processes with maximum number of threads.
1200Specifying output width and columns to be displayed
1201We can specify the columns to be displayed in the  ps output using the user-defined output
1202format specifier  -o . Another way to specify the output format is with "standard" options.
1203Practice them according to your usage style. Try these options:
1204f -f ps –ef
1205f u ps -e u
1206f ps ps -e w (w stands for wide output)
1207Showing environment variables for a process
1208Understanding which environment variables a process is depended on is a very useful bit of
1209information we might need. Whether or not a process works might be heavily dependent on
1210the environmental variables set. We can debug and make use of environment data for fixing
1211several problems related to running of processes.
1212In order to list environment variables along with  ps entries use:
1213$ ps -eo cmd e
1214For example:
1215$ ps -eo pid,cmd e | tail -n 3
12161162 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
12171172 sshd: slynux [priv]
12181237 sshd: slynux@pts/0
1219www.it-ebooks.info
1220Chapter 9
1221303
12221238 -bash USER=slynux LOGNAME=slynux HOME=/home/slynux PATH=/usr/
1223local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
1224MAIL=/var/mail/slynux SHELL=/bin/bash SSH_CLIENT=10.211.55.2 49277 22
1225SSH_CONNECTION=10.211.55.2 49277 10.211.55.4 22 SSH_TTY=/dev/pts/0
1226TERM=xterm-color LANG=en_IN XDG_SESSION_COOKIE=d1e96f5cc8a7a3bc3a0a73e44c
122795121a-1286499339.592429-1573657095
1228An example of where this type of environment tracing can come in handy is in tracing
1229problems with the apt-get package manager. If you use an HTTP proxy to connect to the
1230internet, you may need to set environment variables  http_proxy=host:port . But
1231sometimes even when it is set, the  apt-get command will not select the proxy and hence it
1232returns an error. Then you can actually look at an environment variable and track the issue.
1233We may need some applications to be run automatically with scheduling tools such as
1234crontab . But it might be dependent on some environment variables. Suppose we want
1235to open a GUI-windowed application at a given time. We schedule it using  crontab at a
1236specified time. However, you will notice that the application will not start at a given time if an
1237entry like the following is given:
123800 10 * * * /usr/bin/windowapp
1239This is because a windowed application always depends on the DISPLAY environment variable.
1240Environment variables need to be passed to the application.
1241First run  windowapp manually, and then run  ps -C windowapp -eo cmd e .
1242Find out the environment variables. Prefix them before a command name appears in
1243crontab . The issue will get resolved.
1244Modify the entry as follows:
124500 10 * * * DISPLAY=:0 /usr/bin/windowapp
1246DISPLAY=:0 can be obtained from the  ps output.
1247See also
1248f Scheduling with cron, explains how to schedule tasks
1249www.it-ebooks.info
1250Administration Calls
1251304
1252Killing processes and send or respond
1253to signals
1254Termination of processes is an important task we always come across. Sometimes we
1255may need to terminate all the instances of a program. The command line provides several
1256options for terminating programs. An important concept regarding processes in UNIX-like
1257environments is that of signals. Signals are an inter-process communication mechanism
1258used to interrupt running process to perform some action. Termination of a program is also
1259performed by using the signals technique. This recipe is an introduction to signals and the
1260usage of signals.
1261Getting ready
1262Signals are an inter-process mechanism available in Linux. We can interrupt a process by using
1263a specific signal. Each signal is associated with an integer value. When a process receives a
1264signal, it responds by executing a signal handler. In Shell scripting also, it is possible to send
1265and receive signals and respond according to the signals.  KILL is a signal used to terminate a
1266process. Events such as  Ctrl +  C ,  Ctrl +  Z are also types of signals. The  kill command is used
1267to send signals to processes and the  trap command is used to handle the received signals.
1268How to do it...
1269In order to list all the signals available, use:
1270$ kill -l
1271It will print the signal number and signal names.
1272Terminate a process as follows:
1273$ kill PROCESS_ID_LIST
1274The  kill command issues a TERM signal by default. The process ID list is to be specified with
1275space as a delimiter between process IDs.
1276In order to specify a signal to be sent to a process via the  kill command use:
1277$ kill -s SIGNAL PID
1278The  SIGNAL argument is either a signal name or a signal number. Though there are many
1279signals specified for different purposes, we frequently use only a few signals. They are as follows:
1280f SIGHUP 1 —hangup detection on death of controlling process or terminal
1281f SIGINT 2 —signal which is emitted when  Ctrl +  C is pressed
1282www.it-ebooks.info
1283Chapter 9
1284305
1285f SIGKILL 9 —signal used to force kill the process
1286f SIGTERM -15 —signal used to terminate a process by default
1287f SIGTSTP 20 —signal emitted when  Ctrl +  Z is pressed
1288We frequently use force kill for processes. In order to force kill a process, use:
1289$ kill -s SIGKILL PROCESS_ID
1290Or:
1291$ kill -9 PROCESS_ID
1292There's more...
1293Let's walk through additional commands used for terminating and signalling processes.
1294kill family of commands
1295The  kill command takes the process ID as argument. There are also a few other commands in
1296the  kill family that accept the command name as argument and send a signal to the process.
1297The  killall command terminates the process by name as follows:
1298$ killall process_name
1299In order to send a signal to a process by name use:
1300$ killall -s SIGNAL process_name
1301In order to force kill process by name use:
1302$ killall -9 process_name
1303For example:
1304$ killall -9 gedit
1305Specify the process by name, which is specified by users who own it, by using:
1306$ killall -u USERNAME process_name
1307In order to ask interactively before killing processes, use the  -i argument along with
1308killall .
1309The  pkill command is similar to the  kill command but it, by default, accepts a process
1310name instead of a process ID. For example:
1311$ pkill process_name
1312$ pkill -s SIGNAL process_name
1313www.it-ebooks.info
1314Administration Calls
1315306
1316SIGNAL is the signal number.  SIGNAL name is not supported with  pkill .
1317It provides many of the same options that the  kill command does. Check the  pkill
1318manpages for more details.
1319Capturing and responding to signals
1320trap is a command used to assign signal handler to signals in a script. Once a function is
1321assigned to a signal using the  trap command, while the script runs and it receives a signal,
1322this function is executed upon reception of a corresponding signal.
1323The syntax is as follows:
1324trap 'signal_handler_function_name' SIGNAL LIST
1325SIGNAL LIST is delimited by space. It can be a signal number or a signal name.
1326Let's write a shell script that responds to the  SIGINT signal:
1327#/bin/bash
1328#Filename: sighandle.sh
1329#Description: Signal handler
1330function handler()
1331{
1332echo Hey, received signal : SIGINT
1333}
1334echo My process ID is $$
1335# $$ is a special variable that returns process ID of current process/
1336script
1337trap 'handler' SIGINT
1338#handler is the name of the signal handler function for SIGINT signal
1339while true;
1340do
1341sleep 1
1342done
1343Run this script in a terminal. When the script is running, if you press  Ctrl +  C it will show the
1344message by executing the signal handler associated with it.  Ctrl +  C is a  SIGINT signal.
1345The  while loop is used to keep the process running without going to termination by using an
1346infinite loop. Thus the process is kept running infinitely so that it can respond to the signals
1347that are sent to the process asynchronously by another process. The loop that is used to keep
1348the process alive infinitely is often called as the event loop. If an infinite loop is not available,
1349the script will terminate after executing the statements. But for signal handler scripts, it has to
1350wait and respond to the signals.
1351We can send a signal to the script by using the  kill command and the process ID of the script:
1352$ kill -s SIGINT PROCESS_ID
1353www.it-ebooks.info
1354Chapter 9
1355307
1356The  PROCESS_ID of the above script will be printed when it is executed. Or you can find it out
1357by using the  ps command
1358If no signal handlers are specified for signals, it will call the default signal handlers assigned
1359by the operating system. Generally, pressing  Ctrl +  C will terminate a program since the
1360default handler provided by the operating system will terminate the process. But the custom
1361handler defined here specifies a custom action upon receipt of the signal.
1362We can define signal handlers for any signals available ( kill -l ), by using the  trap
1363command. It is also possible to set a single signal handler for multiple signals.
1364which, whereis, file, whatis, and loadavg
1365explained
1366This recipe aims to explain a few commands we come across. Understanding these
1367commands is helpful for users.
1368How to do it...
1369Let's go through each of the commands and their usage examples.
1370f which
1371The  which command is used to find the location of a command. We type commands
1372in the terminal without knowing the location where the executable file is stored.
1373When we type a command, the terminal looks for the command in a set of locations
1374and executes the executable file if found at the location. This set of locations is
1375specified using an environment variable  PATH . For example:
1376$ echo $PATH
1377/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/
1378games
1379We can export  PATH and can add our own locations to be searched when command
1380names are typed. For example, to add  /home/slynux/bin to  PATH use the
1381following command:
1382$ export PATH=$PATH:/home/slynux/bin
1383# /home/slynux/bin is added to PATH
1384The  which command outputs the location of the command given as argument. For
1385example:
1386$ which ls
1387/bin/ls
1388www.it-ebooks.info
1389Administration Calls
1390308
1391f whereis
1392whereis is similar to the  which command. But it not only returns the path of the
1393command, it will also print the location of the manpage, if available, and also the
1394path of the source code for the command if available. For example:
1395$ whereis ls
1396ls: /bin/ls /usr/share/man/man1/ls.1.gz
1397f file
1398The  file command is an interesting and frequently-used command. It is used for
1399determining the file type:
1400$ file FILENAME
1401This will print the details of the file regarding its file type.
1402An example is as follows:
1403$ file /bin/ls
1404/bin/ls: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV),
1405dynamically linked (uses shared libs), for GNU/Linux 2.6.15,
1406stripped
1407f whatis
1408The  whatis command outputs a one-line description of the command given as an
1409argument. It parses information from the manpage. For example:
1410$ whatis ls
1411ls (1) - list directory contents
1412apropos
1413Sometimes we need to search if some command related
1414to a word exists. Then we can search the manpages for
1415strings in the command. For this we can use:
1416apropos COMMAND
1417f Load average
1418Load average is an important parameter for total load on the running system. It
1419specifies the average of the total number of runnable processes on the system. It
1420is specified by three values. The first value indicates the average in one minute,
1421the second indicates average in five minutes, and third indicates the average in 15
1422minutes.
1423It can be obtained by running  uptime . For example:
1424$ uptime
142512:40:53 up 6:16, 2 users, load average: 0.00, 0.00, 0.00
1426www.it-ebooks.info
1427Chapter 9
1428309
1429Sending messages to user terminals
1430A system administrator may need to send messages to the terminal screen of every user or a
1431specified user on all the machines over a network. This recipe is a guide to perform this task.
1432Getting ready
1433wall is a command that is used to write messages on the terminals of all logged in users. It
1434can be used to convey messages to all logged in users in a server or multiple access machines.
1435Sending messages to all users may, sometimes, not be useful. We may need to send messages
1436to specific users or a specific terminal. Terminals are treated as devices in a Linux system and
1437hence these opened terminals will have a corresponding device node file at  /dev/pts/ . Writing
1438data to a specific device will display messages on the corresponding terminal.
1439How to do it...
1440In order to broadcast a message to all users and all logged in terminals, use:
1441$ cat message | wall
1442Or:
1443$ wall < message
1444Broadcast Message from slynux@slynux-laptop
1445(/dev/pts/1) at 12:54 ...
1446This is a message
1447The message outline will show who sent the message (which user and which host). The
1448message gets OR is displayed to the current terminal if some other users send a message, only
1449if the "write message" option is enabled. By default, in most distros "write message" is enabled
1450by default. If the sender of the message is root, then the message gets displayed on the screen
1451irrespective of whether the "write message" option is enabled or disabled by the user.
1452In order to enable write messages use:
1453$ mesg y
1454In order to disable write messages use:
1455$ mesg n
1456www.it-ebooks.info
1457Administration Calls
1458310
1459Let's write a script for sending messages specifically to a given user's terminal:
1460#/bin/bash
1461#Filename: message_user.sh
1462#Description: Script to send message to specified user logged
1463terminals.
1464USER=$1
1465devices=`ls /dev/pts/* -l | awk '{ print $3,$9 }' | grep $USER | awk
1466'{ print $2 }'`
1467for dev in $devices;
1468do
1469cat /dev/stdin > $dev
1470done
1471Run the script as:
1472./message_user.sh USERNAME < message.txt
1473# Pass message through stdin and username as argument
1474The output will be as follows:
1475$ cat message.txt
1476A message to slynux. Happy Hacking!
1477# ./message_user.sh slynux < message.txt
1478# Run message_user.sh as root since the message is to be send to a
1479specifc user.
1480Now, slynux's terminal will receive the message text.
1481How it works...
1482The  /dev/pts directory will contain character devices corresponding to each of the logged
1483in terminals on the system. We can find out who logged into which terminal by looking at
1484the owner of the device files. The  ls -l output will contain the owner name and the device
1485path. This information is extracted by using  awk . Then it uses  grep to extract the lines
1486corresponding to specified user only. The username is accepted as the first argument for the
1487script as stored as variable USER. Then a list of terminals for a given user is made. A  for
1488loop is used to iterate through each device path.  /dev/stdin will contain standard input
1489data passed to the current process. Therefore, by reading  /dev/stdin , data is read and
1490redirected to the corresponding terminal (TTY) devices. Hence the message gets displayed.
1491www.it-ebooks.info
1492Chapter 9
1493311
1494Gathering system information
1495Collecting information about the current system from the command line is very important
1496in logging system data. The different system information data includes hostname, kernel
1497version, Linux distro name, CPU information, memory information, disk partition information,
1498and so on. This recipe will show you different sources in a Linux system to gather information
1499about the system.
1500How to do it...
1501In order to print the hostname of the current system, use:
1502$ hostname
1503Or:
1504$ uname -n
1505Print long details about the Linux kernel version, hardware architecture, and more by using:
1506$ uname -a
1507In order to print the kernel release, use:
1508$ uname -r
1509Print the machine type as follows:
1510$ uname –m
1511In order to print details about CPU details, use:
1512$ cat /proc/cpuinfo
1513In order to extract the processor name, use:
1514$ cat /proc/cpuinfo | head -n 5 | tail -1
1515The fifth line contains the processor name. Therefore, the first five lines are extracted first.
1516Then the last one line is extracted to print processor name.
1517Print details about memory or RAM as follows:
1518$ cat /proc/meminfo
1519Print the total memory (RAM) available on the system as follows:
1520$ cat /proc/meminfo | head -1
1521MemTotal: 1026096 kB
1522www.it-ebooks.info
1523Administration Calls
1524312
1525In order to list out the partitions information available on the system, use:
1526$ cat /proc/partitions
1527Or:
1528$ fdisk -l
1529Get the entire details about the system as follows:
1530$ lshw
1531Using /proc – gathering information
1532/proc is an in-memory pseudo filesystem available on GNU/Linux operating systems. It was
1533introduced to provide an interface to read several system parameters from a user space. It
1534is very interesting and we can gather lots of information from it. Let's see few of the features
1535available with the  proc filesystem.
1536How to do it...
1537If you look at  /proc , you can see several files and directories. Some of them are already
1538explained in another recipe in this chapter. You can simply  cat files in  /proc and the
1539subdirectories to get information. All of them are well-formatted text.
1540There will be a directory in  /proc for every process that is running on the system. The
1541directory name for a process in  /proc is same as that of process ID of that process.
1542Suppose for Bash, the process ID is 4295 ( pgrep bash ),  /proc/4295 will exist. Each of
1543the directories corresponding to the process will contain a lot of information regarding that
1544process. Few of the important files in  /proc/PID are as follows.
1545f environ —contains environment variables associated with that process.
1546By  cat /proc/4295/environ we can display all the environment variables passed
1547to that process.
1548f cwd —is a symlink to a working directory of the process.
1549f exe —is a symlink to the running executable for the current process.
1550$ readlink /proc/4295/exe
1551/bin/bash
1552f fd —is the directory consisting of entries on file descriptors used by the process.
1553www.it-ebooks.info
1554Chapter 9
1555313
1556Scheduling with cron
1557It is a common requirement to schedule the execution of scripts at a given time or at given
1558time intervals. The GNU/Linux system comes with different utilities for scheduling tasks.
1559cron is such a utility that allows tasks to automatically run in the background of the system
1560at regular intervals by use of the  cron daemon. The  cron utility makes use of a file called
1561"cron table" that stores a list of schedule of scripts or commands to be executed and the
1562time at which they are to be executed. It is a very useful utility. A common example usage is
1563to schedule downloads of files from the Internet during the free hours (certain ISPs provide
1564free usage - usually during the night when most people are sleeping). Users are not required
1565to wake up in the night to start the download. Users can write a cron entry and schedule the
1566download. You can also schedule to drop the Internet connection automatically and shut down
1567the system when the free usage hours end.
1568Getting ready
1569The cron scheduling utility comes with all the GNU/Linux distributions by default. Once we
1570write the cron table entry, the commands will be executed at the time specified for execution.
1571The command  crontab is used to add schedule entries to the cron schedule domain. A cron
1572schedule is a simple text file. Each user has his or her own cron schedule. A cron schedule is
1573often called a cron job.
1574How to do it…
1575In order to schedule tasks, we should know the format for writing the cron table. A cron job
1576specifies the path of a script or command to be executed and the time at which it is to be
1577executed. Each cron table consists of six sections in the following order:
1578f Minute (0 - 59)
1579f Hour (0 - 23)
1580f Day (1 - 31)
1581f Month (1 - 12)
1582f Weekday (0 - 6)
1583f COMMAND (the script or command to be executed at the specified time)
1584The first five sections specify the time at which an instance of the command is to be executed.
1585There are a few additional options to specify the time schedule.
1586www.it-ebooks.info
1587Administration Calls
1588314
1589An asterisk ( * ) is used to specify that the command should be executed at every instance of
1590time. That is, if  * is written in the hours field in the cron job, the command will be executed
1591for every hour. Similarly, if you would like to execute the command at multiple instances of
1592a particular time period, specify the time period separated by comma in the corresponding
1593time field (for example, for running the command at the fifth minute and tenth minute, enter
15945,10 in the minutes field). We also have another nice option to run the command at particular
1595divisions of time. Use  */5 in the minutes field for running the command at every five minutes.
1596We can apply this to any time field. A cron table entry can consist of one or more lines of cron
1597jobs. Each line in the cron table entry is a single job. For example:
1598f Let's write a sample  crontab entry for illustration:
159902 * * * * /home/slynux/test.sh
1600This cron job will execute the  test.sh script at the second minute of all hours on all
1601days.
1602f In order to run the script at fifth, sixth, and seventh hours on all days, use:
160300 5,6,7 * * /home/slynux/test.sh
1604f Execute  script.sh at every hour on Sundays as follows:
160500 */12 * * 0 /home/slynux/script.sh
1606f Shut down the computer at 2am everyday as follows:
160700 02 * * * /sbin/shutdown -h
1608Now, let us see how to schedule a cron job. You can execute the  crontab command in
1609multiple ways to schedule the scripts.
1610When you run the  crontab manually, use the  –e option to enter the cron job:
1611$ crontab –e
161202 02 * * * /home/slynux/script.sh
1613When  crontab –e is entered, the default text editor (usually vi) is opened up and the user
1614can type the cron job and save it. This cron job will be scheduled and executed at specified
1615time intervals.
1616There are two other methods we usually use when we invoke the  crontab command inside a
1617script for scheduling tasks:
16181. Create a text file (for example,  task.cron ) and write the cron job.
1619Then run the  crontab with the filename as the command argument:
1620$ crontab task.cron
16212. By using the next method we can specify the cron job inline without creating a
1622separate file. For example:
1623www.it-ebooks.info
1624Chapter 9
1625315
1626crontab<<EOF
162702 * * * * /home/slynux/script.sh
1628EOF
1629The cron job needs to be written in between  crontab<<EOF and EOF .
1630Cron jobs are executed with privileges with which the  crontab command is executed. If you
1631need to execute commands that require higher privileges, such as a command for shutting
1632down the computer, run the  crontab command as root.
1633The commands specified in the cronjob are written with the full path to the command. This
1634is because the environment in which a cron job is executed is different from the one that we
1635execute on a terminal. Hence the  PATH environment variable may not be set. If your command
1636requires certain environment variables to be set for running, you should explicitly set the
1637environment variables.
1638There's more…
1639The  crontab command has more options. Let's see a few of them.
1640Specifying environment variables
1641Many of the commands require environment variables to be set properly for execution. We can
1642set environment variables by inserting a line with variable assignment statement in the cron
1643table of the user.
1644For example, if you are using a proxy server for connecting to the Internet, to schedule
1645a command that uses Internet you have to set the HTTP proxy environment variable
1646http_proxy . It can be done as follows:
1647crontab<<EOF
1648http_proxy=http://192.168.03:3128
164900 * * * * /home/slynux/download.sh
1650EOF
1651Viewing the cron table
1652We can list the existing cronjobs using the  –l  option:
1653$ crontab –l
165402 05 * * * /home/user/disklog.sh
1655The  crontab –l lists the existing entries in the cron table for the current user.
1656We can also view the cron table for other users by specifying username with the  –u option as
1657follows:
1658$ crontab –l –u slynux
165909 10 * * * /home/slynux/test.sh
1660www.it-ebooks.info
1661Administration Calls
1662316
1663You should run as root when you use the  –u option to gain higher privilege.
1664Removing the cron table
1665We can remove the crontable for the current user using the  –r option:
1666$ crontab –r
1667In order to remove crontab for another user, use:
1668# crontab –u slynux –r
1669Run as root to get higher privilege.
1670Writing and reading MySQL database
1671from Bash
1672MySQL is a widely used Database System. Usually, MySQL databases are used as the storage
1673systems for applications that are written in languages such as PHP, Python, C++, and so on.
1674Accessing and manipulating MySQL databases from shell script will be interesting. We can
1675write scripts to write contents from a text file or CSV (Comma Separated Values) into tables
1676and interact with the MySQL database to read and manipulate data. For example, we can read
1677all the e-mail addresses stored in a guestbook program's database by running a query from
1678the shell script. In this recipe, we will see how to read and write to a MySQL database from
1679Bash. For illustration, here is an example problem:
1680I have a CSV file containing details of students. I need to insert the contents of the file to a
1681database table. From this data, I need to generate a separate rank list for each department.
1682Getting ready
1683In order to handle MySQL databases, you should have mysql-server and mysql-client packages
1684installed on your system. These tools do not come with a Linux distribution by default.
1685Since MySQL comes with a username and password for authentication, you should have a
1686username and password to run the scripts.
1687How to do it…
1688The above problem can be solved using Bash utilities such as  sort ,  awk , and so on.
1689Alternately, we can solve it by using an SQL database table. We will write three scripts for the
1690purpose of creating a database and table, inserting student data into the table, and reading
1691and displaying processed data from the table.
1692www.it-ebooks.info
1693Chapter 9
1694317
1695Create the database and table script as follows:
1696#!/bin/bash
1697#Filename: create_db.sh
1698#Description: Create MySQL database and table
1699USER="user"
1700PASS="user"
1701mysql -u $USER -p$PASS <<EOF 2> /dev/null
1702CREATE DATABASE students;
1703EOF
1704[ $? -eq 0 ] && echo Created DB || echo DB already exist
1705mysql -u $USER -p$PASS students <<EOF 2> /dev/null
1706CREATE TABLE students(
1707id int,
1708name varchar(100),
1709mark int,
1710dept varchar(4)
1711);
1712EOF
1713[ $? -eq 0 ] && echo Created table students || echo Table students
1714already exist
1715mysql -u $USER -p$PASS students <<EOF
1716DELETE FROM students;
1717EOF
1718The script for inserting data into the table is as follows:
1719#!/bin/bash
1720#Filename: write_to_db.sh
1721#Description: Read from CSV and write to MySQLdb
1722USER="user"
1723PASS="user"
1724if [ $# -ne 1 ];
1725then
1726echo $0 DATAFILE
1727echo
1728exit 2
1729fi
1730data=$1
1731while read line;
1732do
1733oldIFS=$IFS
1734www.it-ebooks.info
1735Administration Calls
1736318
1737IFS=,
1738values=($line)
1739values[1]="\"`echo ${values[1]} | tr ' ' '#' `\""
1740values[3]="\"`echo ${values[3]}`\""
1741query=`echo ${values[@]} | tr ' #' ', ' `
1742IFS=$oldIFS
1743mysql -u $USER -p$PASS students <<EOF
1744INSERT INTO students VALUES($query);
1745EOF
1746done< $data
1747echo Wrote data into DB
1748The script for the query from the database is as follows:
1749#!/bin/bash
1750#Filename: read_db.sh
1751#Description: Read from the database
1752USER="user"
1753PASS="user"
1754depts=`mysql -u $USER -p$PASS students <<EOF | tail -n +2
1755SELECT DISTINCT dept FROM students;
1756EOF`
1757for d in $depts;
1758do
1759echo Department : $d
1760result="`mysql -u $USER -p$PASS students <<EOF
1761SET @i:=0;
1762SELECT @i:=@i+1 as rank,name,mark FROM students WHERE dept="$d" ORDER
1763BY mark DESC;
1764EOF`"
1765echo "$result"
1766echo
1767done
1768The data for the input CSV file ( studentdata.csv ) is as follows:
17691,Navin M,98,CS
17702,Kavya N,70,CS
17713,Nawaz O,80,CS
17724,Hari S,80,EC
17735,Alex M,50,EC
1774www.it-ebooks.info
1775Chapter 9
1776319
17776,Neenu J,70,EC
17787,Bob A,30,EC
17798,Anu M,90,AE
17809,Sruthi,89,AE
178110,Andrew,89,AE
1782Execute the scripts in the following sequence:
1783$ ./create_db.sh
1784Created DB
1785Created table students
1786$ ./write_to_db.sh studentdat.csv
1787Wrote data into DB
1788$ ./read_db.sh
1789Department : CS
1790rank name mark
17911 Navin M 98
17922 Nawaz O 80
17933 Kavya N 70
1794Department : EC
1795rank name mark
17961 Hari S 80
17972 Neenu J 70
17983 Alex M 50
17994 Bob A 30
1800Department : AE
1801rank name mark
18021 Anu M 90
18032 Sruthi 89
18043 Andrew 89
1805www.it-ebooks.info
1806Administration Calls
1807320
1808How it works…
1809We will now see the explanation of the above scripts one by one. The first script  create_db.sh
1810is used to create database called  students and a table named  students inside it. We need
1811the MySQL username and password to access or modify data in the DBMS. The variables  USER
1812and  PASS are used to store the username and password. The  mysql command is used for
1813MySQL manipulations. The  mysql  command can specify the username by using  –u and the
1814password by using  –pPASSWORD . The other command argument for the  mysql command is the
1815database name. If a database name is specified as an argument to the  mysql command, it will
1816use that for database operations, else we have to explicitly specify in the SQL query about which
1817database is to be used with the  use database_name query. The  mysql command accepts
1818the queries to be executed through standard input ( stdin ). The convenient way of supplying
1819multiple lines through  stdin is by using the  <<EOF method. The text that appears in between
1820<<EOF and  EOF is passed to  mysql as standard input. In the  CREATE DATABASE query, we
1821have redirected  stderr to  /dev/null in order to prevent displaying an error message. Also,
1822in the table creation query, we have redirected  stderr to  /dev/null to ignore any errors that
1823occur. Then we check the exit status for the  mysql command by using the exit status variable
1824$? to know if a table or database already exists. If the database or table already exists, a
1825message is displayed to notify that. Else we will create them.
1826The next script  write_to_db.sh accepts a filename of the student data CSV file. We read
1827each line of the CSV file by using the  while loop. So in each iteration a line with comma
1828separated values will be received. We then need to formulate the values in the line to an SQL
1829query. For that, the easiest way to store data items in the comma separated line is by using
1830an array. We know that an array assignment is in the form  array=(val1 val2 val3) .
1831Here the space character is the Internal Field Separator (IFS). We have a line with comma
1832separated values, hence by changing the IFS to a comma, we can easily assign values to
1833the array ( IFS=, ). The data items in the comma separated line are  id ,  name ,  mark , and
1834department .  id and  mark are integer values whereas  name and  dept are strings (strings
1835must be quoted). Also the name can contain space characters. Space can conflict with the
1836Internal Field Separator. Hence we should replace the space in the name with some character
1837( # ) and replace it later after formulating the query. In order to quote the strings, the values in
1838the array are prefixed and suffixed with  \" . The  tr is used to substitute space in the name to
1839# . Finally, the query is formed by replacing the space character with comma and replacing  #
1840with space and this query is executed.
1841The third script  read_db.sh is used to find out the department and print the rank list of
1842students for each department. The first query is used to find distinct names of departments.
1843We use a  while loop to iterate through each department and run the query to display student
1844details in the order of highest marks.  SET @i=0 is an SQL construct used to set the variable
1845i=0 . On each row it is incremented and is displayed as the rank of the student.
1846www.it-ebooks.info
1847Chapter 9
1848321
1849User administration script
1850GNU/Linux is a multi user operating system. Many users can log in and perform several
1851activities at a time. There are several administration tasks that are handled with user
1852management. The tasks includes setting the default shell for the user, disabling a user
1853account, disabling a shell account, adding new users, removing users, setting a password,
1854setting an expiry date for a user account, and so on. This recipe aims at writing a user
1855management tool that can handle all of these tasks.
1856How to do it…
1857Let's go through the user administration script:
1858#!/bin/bash
1859#Filename: user_adm.sh
1860#Description: A user administration tool
1861function usage()
1862{
1863echo Usage:
1864echo Add a new user
1865echo $0 -adduser username password
1866echo
1867echo Remove an existing user
1868echo $0 -deluser username
1869echo
1870echo Set the default shell for the user
1871echo $0 -shell username SHELL_PATH
1872echo
1873echo Suspend a user account
1874echo $0 -disable username
1875echo
1876echo Enable a suspended user account
1877echo $0 -enable username
1878echo
1879echo Set expiry date for user account
1880echo $0 -expiry DATE
1881echo
1882echo Change password for user account
1883echo $0 -passwd username
1884echo
1885echo Create a new user group
1886echo $0 -newgroup groupname
1887echo
1888www.it-ebooks.info
1889Administration Calls
1890322
1891echo Remove an existing user group
1892echo $0 -delgroup groupname
1893echo
1894echo Add a user to a group
1895echo $0 -addgroup username groupname
1896echo
1897echo Show details about a user
1898echo $0 -details username
1899echo
1900echo Show usage
1901echo $0 -usage
1902echo
1903exit
1904}
1905if [ $UID -ne 0 ];
1906then
1907echo Run $0 as root.
1908exit 2
1909fi
1910case $1 in
1911-adduser) [ $# -ne 3 ] && usage ; useradd $2 -p $3 -m ;;
1912-deluser) [ $# -ne 2 ] && usage ; deluser $2 --remove-all-files;;
1913-shell) [ $# -ne 3 ] && usage ; chsh $2 -s $3 ;;
1914-disable) [ $# -ne 2 ] && usage ; usermod -L $2 ;;
1915-enable) [ $# -ne 2 ] && usage ; usermod -U $2 ;;
1916-expiry) [ $# -ne 3 ] && usage ; chage $2 -E $3 ;;
1917-passwd) [ $# -ne 2 ] && usage ; passwd $2 ;;
1918-newgroup) [ $# -ne 2 ] && usage ; addgroup $2 ;;
1919-delgroup) [ $# -ne 2 ] && usage ; delgroup $2 ;;
1920-addgroup) [ $# -ne 3 ] && usage ; addgroup $2 $3 ;;
1921-details) [ $# -ne 2 ] && usage ; finger $2 ; chage -l $2 ;;
1922-usage) usage ;;
1923*) usage ;;
1924esac
1925A sample output is as follows:
1926# ./user_adm.sh -details test
1927Login: test Name:
1928Directory: /home/test Shell: /bin/sh
1929Last login Tue Dec 21 00:07 (IST) on pts/1 from localhost
1930No mail.
1931www.it-ebooks.info
1932Chapter 9
1933323
1934No Plan.
1935Last password change : Dec 20, 2010
1936Password expires : never
1937Password inactive : never
1938Account expires : Oct 10, 2010
1939Minimum number of days between password change : 0
1940Maximum number of days between password change : 99999
1941Number of days of warning before password expires : 7
1942How it works…
1943The  user_adm.sh script can be used to perform many user management tasks. You can
1944follow the  usage() text for the proper usage of the script. A function  usage() is defined
1945to display how to execute the script with different options for the user when any of the
1946parameters given by user gets wrong or has run the  –usage parameter. A case statement is
1947used to match the command arguments and execute the corresponding commands according
1948to that. The valid command options for the  user_adm.sh script are:  -adduser ,  -deluser ,
1949-shell ,  -disable ,  -enable ,  -expiry ,  -passwd ,  -newgroup ,  -delgroup ,  -addgroup ,
1950-details , and  -usage . When the  *) case is matched, it means its a wrong option and
1951hence  usage() is invoked. For each match case, we have used  [ $# -ne 3 ] && usage .
1952It is used for checking number of arguments. If the number of command arguments are not
1953equal to required number, the  usage() function is invoked and the script will exit without
1954executing further. In order to run the user management commands, the script needs to be
1955run as root. Hence a check for user ID 0 (the root has user ID 0) is performed. If the user has
1956a non-zero user ID, this means it is executing as non-root. Hence a message to run as root is
1957displayed and the script exits.
1958Let's explain each case one by one:
1959f -useradd :
1960The  useradd command can be used to create a new user. It has the syntax:
1961useradd USER –p PASSWORD
1962The  -m option is used to create the home directory
1963It is also possible to provide the full name of the user by using the  –c FULLNAME
1964option.
1965f -deluser :
1966The  deluser command can be used to remove the user. The syntax is:
1967deluser USER
1968--remove-all-files is used to remove all files associated with the user including
1969the home directory.
1970www.it-ebooks.info
1971Administration Calls
1972324
1973f -shell :
1974The  chsh command is used to change the default shell for the user. The syntax is:
1975chsh USER –s SHELL
1976f -disable and  –enable :
1977The  usermod command is used to manipulate several attributes related to user
1978accounts.
1979usermod –L USER locks the user account and  usermod –U USER unlocks the
1980user account.
1981f -expiry :
1982The  chage command is used manipulate user account expiry information.
1983The syntax is:
1984chage –E DATE
1985There are additional options as follows:
1986‰ -m MIN_DAYS (set the minimum number of days between password
1987changes to  MIN_DAYS )
1988‰ -M MAX_DAYS (set the maximum number of days during which a password
1989is valid)
1990‰ -W WARN_DAYS (set the number of days of warning before a password
1991change is required)
1992f -passwd :
1993The  passwd command is used to change passwords for the users. The syntax is:
1994passwd USER
1995The command will prompt to enter new password.
1996f -newgroup and  addgroup :
1997The  addgroup command will add a new usergroup to the system. The syntax is:
1998addgroup GROUP
1999In order to add an existing user to a group use:
2000addgroup USER GROUP
2001-delgroup
2002The delgroup command will remove a user group. The syntax is:
2003delgroup GROUP
2004f -details :
2005The  finger USER command will display the user information for the user, which
2006includes details such as user home directory path, last login time, default shell, and
2007so on. The  chage –l  command will display the user account expiry information.
2008www.it-ebooks.info
2009Chapter 9
2010325
2011Bulk image resizing and format conversion
2012All of us use digital cameras and download photos from the cameras as well as the Internet.
2013When we need to deal with large number of image files, we can use scripts to easily perform
2014actions on the files in bulk. A regular task we come across with photos is resizing the file. Also,
2015format conversion from one image format to another comes to use (for example, JPEG to PNG
2016conversion). When we download pictures from a camera, the large resolution pictures take
2017a large size. But we may need pictures of lower sizes that are convenient to store and e-mail
2018over the internet. Hence we resize it to lower resolutions. This recipe will discuss how to use
2019scripts for image management.
2020Getting ready
2021Imagemagick is an excellent tool for manipulating images that can work across several image
2022formats and different constructs with rich options. Most of the GNU/Linux distributions don't
2023come with Imagemagick installed. You need to manually install the package.  convert is the
2024command that we will use frequently.
2025How to do it..
2026In order to convert from one image format to another image format use:
2027$ convert INPUT_FILE OUTPUT_FILE
2028For example:
2029$ convert file1.png file2.png
2030We can resize an image size to a specified image size either by specifying the scale
2031percentage or by specifying width and height of the output image.
2032Resize the image by specifying the  WIDTH or  HEIGHT as follows:
2033$ convert image.png -resize WIDTHxHEIGHT image.png
2034For example:
2035$ convert image.png -resize 1024x768 image.png
2036It is required to provide either  WIDTH or  HEIGHT so that the other will be automatically
2037calculated and resized so as to preserve the image size ratio:
2038$ convert image.png -resize WIDTHx image.png
2039For example:
2040$ convert image.png -resize 1024x image.png
2041www.it-ebooks.info
2042Administration Calls
2043326
2044Resize the image by specifying the percentage scale factor as follows:
2045$ convert image.png -resize "50%" image.png
2046Let's see a script for image management:
2047#!/bin/bash
2048#Filename: image_help.sh
2049#Description: A script for image management
2050if [ $# -ne 4 -a $# -ne 6 -a $# -ne 8 ];
2051then
2052echo Incorrect number of arguments
2053exit 2
2054fi
2055while [ $# -ne 0 ];
2056do
2057case $1 in
2058-source) shift; source_dir=$1 ; shift ;;
2059-scale) shift; scale=$1 ; shift ;;
2060-percent) shift; percent=$1 ; shift ;;
2061-dest) shift ; dest_dir=$1 ; shift ;;
2062-ext) shift ; ext=$1 ; shift ;;
2063*) echo Wrong parameters; exit 2 ;;
2064esac;
2065done
2066for img in `echo $source_dir/*` ;
2067do
2068source_file=$img
2069if [[ -n $ext ]];
2070then
2071dest_file=${img%.*}.$ext
2072else
2073dest_file=$img
2074fi
2075if [[ -n $dest_dir ]];
2076then
2077dest_file=${dest_file##*/}
2078dest_file="$dest_dir/$dest_file"
2079fi
2080if [[ -n $scale ]];
2081then
2082PARAM="-resize $scale"
2083www.it-ebooks.info
2084Chapter 9
2085327
2086elif [[ -n $percent ]];
2087then
2088PARAM="-resize $percent%"
2089fi
2090echo Processing file : $source_file
2091convert $source_file $PARAM $dest_file
2092done
2093The following is a sample output, to scale the images in the directory  sample_dir to  20% size:
2094$ ./image_help.sh -source sample_dir -percent 20%
2095Processing file :sample/IMG_4455.JPG
2096Processing file :sample/IMG_4456.JPG
2097Processing file :sample/IMG_4457.JPG
2098Processing file :sample/IMG_4458.JPG
2099In order to scale the images to width 1024 use:
2100$ ./image_help.sh -source sample_dir –scale 1024x
2101Change the files to PNG format by adding  –ext png along with the above commands.
2102Scale or convert files with specified destination directory as follows:
2103$ ./image_help.sh -source sample -scale 50% -ext png -dest newdir
2104# newdir is the new destination directory
2105How it works…
2106The above  image_help.sh script can accept several command-line arguments, such as
2107- source ,  -percent ,  -scale ,  –ext , and  -dest . A brief explanation of each is as follows:
2108f The  -source parameter is used to specify the source directory for the images.
2109f The  –percent parameter is used to specify the scale percent and  –scale is used to
2110specify scale width and height.
2111f Either  –percent or  –scale is used. Both of them do not appear simultaneously.
2112f The  –ext parameter is used to specify the target file format.  –ext is optional; if it is
2113not specified, format conversion is not performed.
2114f The  –dest parameter is used to specify the destination directory for scale or
2115conversion of image files.  –dest is optional. If  –dest is not specified, the destination
2116directory will be same as the source directory. As the first step in the script, it checks
2117whether the number of command arguments given to the script are correct. Either 4
2118or 6 or 8 parameters can appear.
2119www.it-ebooks.info
2120Administration Calls
2121328
2122Now, by using a  while loop and case statement, we will parse the command-line arguments
2123corresponding to variables. $# is a special variable that returns the number of arguments.
2124The  shift command shifts the command arguments one position to left, so that on each
2125execution of shift, we can access command arguments one by one, by using the same  $1
2126variable rather than using  $1 ,  $2 ,  $3, and so on. The case statement matches the value of
2127$1 . It is like a switch statement in the C programming language. When a case is matched, the
2128corresponding statements are executed. Each match case statement is terminated with  ;; .
2129Once all the parameters are parsed in variables  percent ,  scale ,  source_dir ,  ext , and
2130dest_dir , a  for loop is used to iterate through path of each file in the source directory and
2131the corresponding action to convert file is performed.
2132If the variable  ext is defined (if  -ext is given in the command argument), the extension of the
2133destination file is changed from  source_file.extension to  source_file.$ext . In the
2134next statement it checks whether the  -dest parameter is provided. If the destination directory
2135is specified, the destination file path is crafted by replacing the directory in source path with
2136destination directory by using file name slicing. In the next statement, it crafts the parameter to
2137the  convert command for performing resize ( -resize widthx or  -resize perc% ). After
2138the parameters are crafted, the  convert command is executed with proper arguments.
2139See also
2140f Slicing filenames based on extension of Chapter 2, explains how to extract portion
2141of file name
2142www.it-ebooks.info
2143Index
2144Symbols
2145$RANDOM environment variable 81
2146-amin parameter 60
2147-atime parameter 59
2148-b option 77
2149^ character
2150tabs, displaying as 52
2151-cmin parameter 60
2152--complement option 144
2153%c parameter 274
2154%C parameter 274
2155-ctime parameter 59
2156-d argument 204
2157--date option 31
2158-delete flag 61
2159-delete option 209
2160-dest parameter 327
2161/dev/pts directory 310
2162/dev/zero 97
2163-d option 70, 77
2164%D parameter 274
2165-dump flag 183
2166-echo option 30
2167%E parameter 274
2168--exclude [PATTEN] 210
2169-exec parameter 61, 62
2170-ext parameter 327
2171<img> tag 193
2172-iname option 56
2173-iregex option 57
2174-k option 76
2175%k parameter 274
2176%K parameter 274
2177--limit-rate argument 181
2178-maxdepth parameter 57, 58
2179-max-filesize option 186
2180-mindepth parameter 57, 58
2181--mirror option 182
2182-mmin parameter 60
2183-mtime parameter 59
2184-name argument 56
2185-newer parameter 60
2186-n flag 52
2187-n option 77
2188-O option 181
2189-path argument 56
2190-percent parameter 327
2191-perm parameter 62
2192%P parameter 274
2193-print argument 55
2194/proc 312
2195--quota argument 182
2196-regex argument 56
2197-r option 76
2198-R option 108
2199--silent option 184
2200-s option 71
2201--sort parameter 299
2202-source parameter 327
2203-t flag 181
2204-T option 52
2205-traversal option 199
2206-type option 58
2207-u option 121
2208-wildcard argument 221
2209%w parameter 274
2210%W parameter 274
2211-x flag 33
2212%x parameter 274
2213%Z parameter 274
2214www.it-ebooks.info
2215330
2216A
2217access event 283
2218active user hours, on system
2219determining 292, 293
2220addgroup command 324
2221alias command 28
2222aliases 27
2223apropos 308
2224archive
2225files, appending to 206
2226files, deleting from 209
2227files, extracing from 207
2228folders, extracing from 207
2229archiving 205
2230arguments
2231about 35
2232negating 57
2233passing, to commands 37
2234arithmetic operations 17, 18
2235array indexes
2236listing 27
2237arrays 25
2238aspell command 89
2239aspell list command 90
2240associative arrays 25, 26
2241attrib event 283
2242automated FTP transfer 248
2243awk command
2244about 50, 150, 289
2245example 151
2246for loop, using 155
2247special variables 152, 153
2248working 147, 151
2249string manipulation functions 156
2250B
2251backups
2252scheduling, at regular intervals 226
2253bandwidth limit
2254specifying, on cURL 186
2255Base64 222
2256Bash
2257about 8
2258arguments 35
2259arithmetic operations 17, 18
2260array indexes 27
2261arrays 25
2262associative arrays 25, 26
2263about 132
2264MySQL database, reading from 316-320
2265MySQL database, writing from 316-320
2266parameter expansion short hands 177
2267text replacement techniques 177
2268filesystem related tests 45
2269functions 35
2270mathematical comparisions 44
2271string comparisions 46
2272tests 44
2273Bash hackers 64
2274Bash prompt string
2275modifying 16
2276BEGIN{} block 102
2277blank files
2278generating, in bulk 110, 111
2279blank lines
2280removing, sed command used 149
2281squeezing, in text files 51, 52
2282Block Size (BS) 97
2283bootable ISO files 119
2284Bourne Again Shell.  See Bash
2285broken links
2286searching, in website 199, 200
2287bunzip2
2288about 215
2289additional features 216
2290files, compressing with 215, 216
2291bytes
2292specifying, as fields 144, 145
2293C
2294case
2295ignoring, of pattern 139
2296cat command
2297about 50, 118
2298file content, concatenating with 50
2299options, for viewing files 51, 52
2300syntax 50
2301usage techniques 51
2302cd command 39, 126
2303cdrecord command 119
2304CD Rom tray
2305playing with 120
2306chage command 324
2307www.it-ebooks.info
2308331
2309character classes, tr command
2310about 72
2311alnum 72
2312alpha 72
2313cntrl 72
2314digit 72
2315graph 72
2316lower 72
2317print 72
2318punct 72
2319space 72
2320upper 72
2321xdigit 72
2322characters
2323counting, in files 128
2324deleting, with tr command 70
2325squeezing, with tr command 71
2326translating, with tr command 69
2327character set
2328complementing 71
2329chattr 110
2330checksum
2331about 73, 100
2332benefits 73
2333calculating, for dircetories 74
2334checksum verification 74
2335chmod command
2336about 107
2337permissions, setting for files 107, 108
2338chown command
2339about 108
2340file ownership, modifying 108
2341chsh command 324
2342close event 283
2343cmd parameter 298
2344coloured output
2345producing, on terminal 12
2346columns
2347multiple files, merging as 162, 163
2348command line interface (CLI) 126
2349command-line navigation
2350performing, popd command used 126, 127
2351performing, pushd command used 126, 127
2352command-line Twitter client
2353writing 196, 197
2354command-line utilities
2355interactive input, automating for 90, 92
2356command outputs
2357monitoring, watch command used 281
2358reading, from awk 155
2359commands
2360about 8
2361arguments, passing to 37
2362about 50
2363executing, with find 61, 62
2364running, on remote host with SSH 255-258
2365return value, obtaining 37
2366comma separated values.  See CSV data
2367comm command 97, 103
2368comm parameter 298
2369compression 205
2370compress parameter 284
2371Content-length parameter 187
2372context-based printing 141, 142
2373convert command 325
2374cookies
2375using, with cURL 185
2376cpio
2377about 211
2378files, archiving with 212
2379using 212
2380CPU 278
2381CPU consuming process
2382listing 278, 280
2383create 0600 root root parameter 284
2384create event 283
2385cron
2386scheduling with 313, 314
2387cron jobs 315
2388cron table
2389removing 316
2390crypt command 222
2391cryptographic tools
2392about 222
2393Base64 222
2394crypt 222
2395gpg 222
2396md5sum 223
2397salted hash 223
2398sha1sum 223
2399csplit utility 83
2400CSV data 41
2401cURL
2402about 182, 183
2403advanced resume download features 185
2404www.it-ebooks.info
2405332
2406bandwidth limit, specifying on 186
2407cookies, using with 185
2408data, posting in 204
2409FTP authentication, performing with 186, 187
2410HTTP authentication, performing with 186,
2411187
2412maximum download size, specifying for 186
2413referer string, setting with 185
2414used, for downloading 182
2415user agent string, setting with 186
2416working 184
2417current shell
2418displaying 15, 16
2419cut command
2420about 143
2421files, column-wise cutting 142-144
2422D
2423data
2424parsing, from website 189, 190
2425posting, in cURL 204
2426posting, to web page 203, 204
2427posting, wget command used 204
2428redirecting, into stdin 258, 259
2429data items
2430locating 136-138
2431mining, grep command used 136-138
2432searching, grep command used 136-138
2433date command 289
2434date format strings 31
2435dates
2436working with 30-32
2437dd command
2438about 96, 230
2439disks, cloning with 230, 231
2440example 96
2441hard drive, cloning with 230, 231
2442large size file, creating with given size 96, 97
2443syntax 230
2444working 97
2445debugging 33
2446default gateway
2447setting 239
2448define utility
2449writing 197-199
2450define:WORD query 197
2451delay
2452producing, in scripts 32
2453delete event 283
2454delgroup command 324
2455delimiter
2456setting, for fields 155
2457deluser command 323
2458df command 266
2459dictionary files
2460about 89
2461using 89
2462diff command
2463about 120, 122, 201
2464generating, against directories 122
2465difference operation 97
2466dir command 125
2467directories
2468checksum, calculating for 74
2469creating, for long path 103, 104
2470listing 125
2471directory depth based search 57, 58
2472directory tree
2473printing 129
2474disks
2475cloning, with dd command 230, 231
2476disk space 266
2477disk usage
2478calculating 266
2479disk free information 271
2480displaying, in KB, MB, or GB 267
2481files, excluding 269, 270
2482files, printing in specified units 269
2483grand total sum, displaying 268
2484large-size files, searching from directory 270
2485disk usage, of remote machines
2486monitoring 289-291
2487DNS 237, 238
2488DNS lookup
2489with fping command 246
2490Domain Name Service.  See DNS
2491du command 266
2492duplicate files
2493about 100
2494deleting 101-103
2495searching 101-103
2496www.it-ebooks.info
2497333
2498E
2499echo command
2500about 9, 152
2501newline, escaping in 12
2502echo packet count
2503limiting 242, 243
2504egrep command 289
2505egrepregex pattern 171
2506e-mail address
2507parsing, from text 171, 172
2508encryption 205
2509END{} block 102
2510environment variables
2511about 12
2512displaying, for process 302, 303
2513specifying 315, 316
2514env variable 14
2515epoch 31
2516Ethernet
2517about 250
2518setting up 251, 252
2519etime parameter 298
2520euid parameter 298
2521executable
2522running, as different user 109
2523execution time, for command
2524calculating 272-274
2525expect command 92
2526expect package 92
2527F
2528fields
2529delimiters, setting for 155
2530file command 113, 308
2531file content
2532concatenating, with cat command 50
2533file descriptors
2534about 19, 23, 24
2535redirecting with 19-22
2536stderr 19-21
2537stdin 19-21
2538stdout 19-21
2539filename-based search 56
2540filename prefix
2541specifying, for split files 82, 83
2542file names
2543slicing, based on extension 84, 85, 86
2544file ownership 104
2545file permissions 104, 105
2546files
2547about 96
2548appending, to archive 206
2549archiving, with cpio 212
2550archiving, with tar command 206
2551archiving, with zip 219, 220
2552characters, counting in 128
2553column-wise cutting, cut command
2554used 142-144
2555compressing, with bunzip2 215, 216
2556compressing, with gzip 212, 213
2557compressing, with lzma 217, 218
2558compressing, with zip 219, 220
2559deleting, from archive 209
2560downloading 180, 181
2561excluding, from archiving 210
2562extracing, from archive 207
2563frequency of words, detecting in 146, 147
2564generating, with random data 96, 97
2565iteration, through characters 161
2566iteration, through lines 161
2567iteration, through words 161
2568large size file, creating with given size 96, 97
2569lines, counting in 128
2570listing 55
2571making, immutable 109, 110
2572matching, based on file permissions 61
2573matching, based on ownership 61
2574moving, in bulk 86, 87, 88
2575ownership, modifying 108
2576permissions 105
2577renaming 86, 87, 88
2578searching 55
2579searching, recursively 138, 139
2580splitting 81
2581transferring 247
2582updating, with timestamp check 208, 209
2583words, counting in 128
2584files, archiving
2585with cpio 212
2586with tar command 206
2587with zip 219, 220
2588files, compressing
2589with bunzip2 215, 216
2590with gzip 212, 213
2591www.it-ebooks.info
2592334
2593with lzma 217, 218
2594with zip 219, 220
2595file sharing 247
2596files, in archive
2597comparing, with files in filesystem 209
2598file size based search 60
2599files ownership
2600modifying, chown command used 108
2601files timestamp based search 59, 60
2602filesystem related tests, Bash 45
2603File Transfer Protocol.  See FTP
2604file type based search 58, 59
2605file type statistics
2606enumerating 113-115
2607find command
2608about 50, 55, 114
2609example 55
2610finger USER command 324
2611first ten lines
2612printing, example 122
2613flow control 44
2614folders
2615extracting, from archive 207
2616fork bomb 36
2617for loop 43
2618format
2619converting, for images 325, 327
2620formatted arguments
2621passing, to command by reading stdin 65-67
2622formatted plain text
2623web page, downloading as 183
2624fping command
2625about 246
2626DNS lookup 246
2627frequency of words
2628detecting, in file 146, 147
2629Frequency parameter 253
2630frequently-used commands
2631printing 276, 278
2632FTP 247
2633FTP authentication
2634performing, cURL used 186, 187
2635ftp command 248
2636functions
2637about 35
2638exporting 36
2639recursive function 36
2640G
2641getline
2642line, reading explicitly 154
2643GET request 203
2644Git
2645about 227
2646used, for version control based
2647backup 227-229
2648Gmail
2649about 188
2650accessing, from command line 188, 189
2651GNU/Linux ecosystem 295
2652GNU privacy guard.  See gpg
2653gpg 222
2654grep command
2655about 50, 112, 136, 172
2656data items, mining 136-138
2657data items, searching in file 136-138
2658files, excluding for search 140
2659files, including for search 140
2660quiet condition 141
2661using, with xargs 140
2662group 105
2663group permissions 106
2664gzip
2665about 212
2666additional features 213, 214
2667files, compressing with 212, 213
2668using, with tarballs 213, 214
2669gzipped files
2670reading, without extracting 214
2671gzipped tarballs
2672creating 213
2673H
2674hard drive
2675cloning, with dd command 230, 231
2676head command
2677about 123, 176
2678example 123
2679implementing, with awk 175, 176
2680host command 287
2681HTML album page
2682generating 194, 195
2683HTML response
2684reading, from website 203, 204
2685www.it-ebooks.info
2686335
2687HTTP authentication
2688performing, cURL used 186, 187
2689hyperlinks 199
2690I
2691ICMP 241
2692ifconfig command 234
2693image crawlers 191, 192
2694image downloader script 192
2695image files
2696mounting 231
2697Imagemagick 325
2698images
2699format, converting 325, 327
2700resizing 325, 327
2701incremental backups 227
2702information
2703gathering, through processes 296-298
2704obtaining, about terminal 29
2705inotify-tools package 282
2706inotifywait command 282
2707interactive input
2708automating, for command-line utilities 90, 92
2709Internal Field Separator (IFS) 41-43, 320
2710Internet Control Message Protocol.  See ICMP
2711intersection operation
2712about 97
2713performing, on text files 97-100
2714intruder detection script
2715writing 287
2716intruders 286
2717intrusion detection system
2718designing 286
2719IP address
2720about 237
2721assigning 236
2722displaying 236
2723matching 135
2724ISO files 117
2725isohybrid command 119
2726ISO image
2727about 117
2728creating 118
2729iwconfig utility 250, 252
2730iwlist utility 250, 253
2731J
2732JavaScript
2733about 158
2734compressing 158, 159
2735decompressing 158, 160
2736K
2737killall command 305
2738kill command
2739about 305
2740using 304
2741L
2742lastb command 276
2743last command 276
2744Last-Modified parameter 187
2745last ten lines
2746printing, example 122
2747let command 17
2748lftp command 248
2749lines
2750counting, in files 128
2751filtering 155
2752printing, after pattern 172, 173
2753printing, before pattern 172, 173
2754printing, in reverse order 169, 170
2755load average 308
2756local mount point
2757remote driver, mounting 259
2758LOC (Lines of Code) 128
2759log events
2760access 283
2761attrib 283
2762close 283
2763create 283
2764delete 283
2765modify 283
2766move 283
2767open 283
2768logfiles
2769about 283
2770managing, logrotate command
2771used 283, 284
2772logfiles, in Linux
2773/var/log/auth.log 285
2774www.it-ebooks.info
2775336
2776/var/log/boot.log 285
2777/var/log/dmesg 285
2778/var/log/httpd 285
2779/var/log/mail.log 285
2780/var/log/messages 285
2781/var/log/Xorg.0.log 285
2782logging information
2783with syslog 285, 286
2784logrotate command 283
2785logrotate configuration file
2786compress parameter 284
2787create 0600 root root parameter 284
2788missingok parameter 284
2789notifempty parameter 284
2790rotate 5 parameter 284
2791size 30k parameter 284
2792weekly parameter 284
2793loopback filesystems 115
2794ls -l command 105
2795Lynx 183, 190, 198
2796lzma
2797about 217
2798additional features 218, 219
2799files, compressing with 217, 218
2800lzma tarball
2801extracting 218
2802M
2803MAC address
2804spoofing 237
2805machine information
2806obtaining 274, 276
2807machines
2808availability, verifying 243-245
2809matched sentence
2810removing 174, 175
2811matched string notation (&) 149
2812mathematical comparisions, Bash 44
2813md5sum
2814about 73, 102, 223
2815syntax 73
2816messages
2817sending, to user terminals 309, 310
2818meta characters
2819about 134
2820\b 134
2821\B 134
2822\d 134
2823\D 134
2824\n 134
2825\r 134
2826\s 134
2827\S 134
2828\w 134
2829\W 134
2830missingok parameter 284
2831mkdir command
2832about 103
2833example 103
2834mkfs command 116
2835mkisofs command 118
2836modify event 283
2837monitoring script
2838writing, for collecting details from remote
2839machines 289-291
2840mount command 96, 231
2841mount point 117
2842move event 283
2843multiple commands
2844combining 38
2845multiple expressions
2846combining 149
2847multiple files
2848merging, as columns 162, 163
2849multiple patterns
2850specifying, for matching 139
2851multiple tar files
2852merging 208
2853MX (Mail Exchanger) 238
2854MySQL 316
2855MySQL database
2856reading, from Bash 316-320
2857writing, from Bash 316-320
2858N
2859name servers 237
2860n characters
2861reading, without pressing Return 40
2862netstat command 263
2863networking 233
2864network interfaces
2865about 234, 235
2866list, printing 235
2867network ports 262, 263
2868www.it-ebooks.info
2869337
2870nice parameter 298
2871node 234
2872notifempty parameter 284
2873numeric characters
2874decrypting, tr command used 70
2875encrypting, tr command used 70
2876O
2877obfuscation tool 158
2878open event 283
2879ownership
2880applying, recursively to files 109
2881P
2882palindrome strings
2883verifying, with scripts 165-169
2884parameter expansion short hands 177
2885parameters, time command
2886%c 274
2887%C 274
2888%D 274
2889%E 274
2890%k 274
2891%K 274
2892%P 274
2893%w 274
2894%W 274
2895%x 274
2896%Z 274
2897passwd command 110, 324
2898paste command 162
2899patch
2900applying 122
2901patch file 120
2902pcpu parameter 298
2903Perl-style regular expressions 134
2904permissions
2905applying, recursively to files 108
2906permission strings
2907------rwx 106
2908---rwx--- 106
2909rwx------ 105
2910pgrep command 13, 296, 300
2911pid parameter 298
2912ping command
2913about 241, 244
2914echo packet count, limiting 242, 243
2915return status 243
2916RTT, finding 242
2917working 241
2918pipe operator 51
2919pkill command 305
2920pmem parameter 298
2921popd command
2922about 126
2923command-line navigation,
2924performing 126, 127
2925pop window
2926sending, with custom messages 260, 261
2927POSIX character class 134
2928POSIX classes
2929[:alnum:] 134
2930[:alpha:] 134
2931[:blank:] 134
2932[:digit:] 134
2933[:lower:] 134
2934[:punct:] 134
2935[:space:] 134
2936[:upper:] 134
2937POST request 203
2938ppid parameter 298
2939printf command 11
2940process
2941about 296
2942environment variables, displaying
2943for 302, 303
2944information, gathering through 296-298
2945termination 304
2946process ID
2947about 296
2948searching 299
2949process manipulation commands 298
2950process threads 301
2951ps command 278
2952about 296
2953filtering with 300
2954output, sorting 299
2955parameters 296-298
2956TTY filter 301
2957ps -eocomm,pcpu 280
2958pushd command
2959about 126
2960command-line navigation,
2961performing 126, 127
2962www.it-ebooks.info
2963338
2964pwd command 39
2965Q
2966quiet mode 141
2967R
2968random data
2969files, generating with 96, 97
2970range of characters
2971specifying, as fields 144, 145
2972rcp 249
2973read command 40
2974real time 272
2975recursive function 36
2976redirection
2977using 23
2978referer string
2979about 185
2980setting, with cURL 185
2981regular expressions
2982about 57, 132
2983components 133
2984examples 132, 133
2985special characters 135
2986regular expressions, components
2987^ 133
2988? 133
2989. 133
2990() 133
2991[^] 133
2992[-] 133
2993[] 133
2994* 133
2995\ 133
2996+ 133
2997| 133
2998$ 133
2999{n,} 133
3000{n} 133
3001{n, m} 133
3002relevant columns
3003printing 163
3004relevant words
3005printing 163
3006remote copy tool.  See rcp
3007remote drive
3008mounting, at local mount point 259
3009remote machines disk usage
3010monitoring 289-291
3011rename command 87
3012response headers
3013printing 187
3014rev command 168, 169
3015rm command 103
3016root 8
3017ROT13 70
3018rotate 5 parameter 284
3019Round Trip Time.  See RTT
3020route command 252
3021routing table information
3022displaying 239
3023rsync command
3024about 224, 249
3025additional features 226
3026working with 224, 225
3027RTT 242
3028S
3029salted hash 223
3030SCP
3031about 249, 250
3032recursive copying 250
3033script command
3034about 53
3035working 54
3036scripting 7
3037scriptreplay command 53
3038scripts
3039debugging 33, 34
3040delays, producing in 32
3041executing, ways 8, 9
3042palindrome strings, verifying with 165-169
3043search
3044directory depth based 57, 58
3045file name based 56
3046file size based 60
3047files timestamp based 59, 60
3048file type based 58, 59
3049Secure FTP.  See SFTP
3050Secure Shell (SSH) connection 208
3051sed command
3052about 50, 100, 147, 156, 165, 174
3053blank lines, removing 149
3054www.it-ebooks.info
3055339
3056options 148, 149
3057set difference operation
3058about 97
3059performing, on text files 97-100
3060setuid permission
3061about 105, 109
3062example 106
3063SFTP 249
3064SHA1 74
3065sha1sum 223
3066Shadowlike hash.  See salted hash
3067shebang 8, 35
3068Shell Scripting language 132
3069shell scripts 7, 8, 234
3070shift command 328
3071SIGNAL argument 304
3072signals
3073about 304
3074capturing 306
3075responding to 304, 306
3076sending 304
3077size 30k parameter 284
3078sort command
3079about 75, 289
3080usage techniques 75, 76
3081sorting