· 7 years ago · Nov 02, 2018, 01:42 PM
102.11.2018
2
3ps -elf
4 shows proccessess with parent PID
5changing niceness of currently running process:
6 renice +3 [pid]
7ps lf
8
9libraries, good if reused
10 static - loaded during compiling, changing it later doesn't affect running process
11 shared - loaded during runtime, changing it later affects running process. also called DDL (on Windows (?)). more efficient, memory usage is lower, exe size too, and they can be used by many apps at once.
12 big change in shared library can cause so called "DLL Hell". especially on 16-bit where all apps ran in shared address space
13
14ldd `which vim`
15 shows shared library dependencies
16 it's not safe since it can result in executing some arbitrary code
17
18ulimit -n 2048
19 change max no of opened files by 2048
20
21zombie process - has terminated but no other process has yet asked about its exit state
22
23
24
25ipcs
26 stary sposób na IPC - Inter Process Communication. System V IPC. Key of 0 means IPC_PRIVATE - they are only shared between processes in a parent/child relationship
27
28SIGNALS:
29 what they are?
30 know different types of signals in Linux
31 use signals from command line: kill, killall, pkill
32
33s. - used to notify processes asynchronously (so s. was not expected, or was expected but exact time was not expected)
342 paths:
35 kernel -> process when there is an exception
36 user process -> another (or the same) process
37
38s. can be sent only between processes owned by the same user or from a process owned by the superuser to any process
39
402 s. cannot be handled and just terminate the program:
41 SIGKILL
42 SIGSTOP
43
44SIGKILL kills a process and cannot be caught
45
46SIGTERM kills a process but can be caught to do a graceful exit
47
48SIGSTOP suspends the process until you do a SIGCONT
49
50kill -l
51 list all (?) signals
52
53man 7 signal
54 shows overview of signals
55
56examples of 'kill' command:
57 kill 1991
58 kill -9 1991
59 kill -SIGKILL 1991
60
61_______
62
63package managers:
64 Red Hat - Yum
65 Fedora - DNF
66 SUSE - Zypper
67 Debian - apt-get
68
692 levels of packaging system utilities:
70 low level, not resolving dependencies, like rpm, dpkg
71 high level, solving dependencies, like yum/dnf/zypper for rpm; apt-get/apt-cache for dpkg
72
73Process of installing/removing software:
74 creating symbolic links
75 creating dirs if needed
76 setting permissions
77 anything that can be scripted (?)
78
79in Debian based systems with source package comes:
80 tarball: "*.tar.gz"
81 Description: "*.dsc"
82 second tarball with patches or other files. "*.debian.tar.gz" or "*.diff.gz"
83
84
85
86
87apt-get source logrotate
88;
89https://lms.quickstart.com/custom/799658/LAB_7.1.pdf
90;
91/etc/apt/sources.list # poczytać!!!
92/etc/apt/sources.list.d/
93;
94/etc/apt/preferences # nadawanie preferencji. poczytać!
95;
96apt-get moo
97;
98sudo apt-get autoremove # get rid of older kernel versions
99sudo apt-get clean # cleans archived package files that have been installed
100;
101https://lms.quickstart.com/custom/799658/LAB_10.1.pdf
102!!!
103https://lms.quickstart.com/custom/799658/LAB_10.2.pdf
104https://lms.quickstart.com/custom/799658/LAB_10.3.pdf
105
106
107info about packages, examine contents, download
108packages.debian.org
109packages.ubuntu.org
110
111
112dpkg -l # list all installed packages
113dpkg -L wget # # list files in wget package
114dpkg -s wget # show info about installed package
115dpkg -I wget # show info about package file
116dpkg -c # list files in package file
117dpkg -S /etc/init/networking.conf # show what package owns /etc/init/networking.conf
118dpkg -S wget # list files in package file
119dpkg -V package # without arg. will verify all packages on the system. see man page to interpret output
120
121sudo dpkg -i foobar.deb # install/upgrade `foobar` package
122sudo dpkg -r foobar.deb # remove `foobar` package (except its configs)
123sudo dpkg -P foobar.deb # remove `foobar` package (with configs) (P is for Purge)
124
125apt-cache search apache2
126apt-cache show apache2 # show info about apache2
127apt-cache showpkg apache2 # show more detailed info about apache2
128apt-cache depends apache2 # list dependencies of apache2
129apt-cache search apache2.conf
130apt-cache list apache2 # list all files in apache2 package
131
132sudo apt-get update
133sudo apt-get install <package>
134sudo apt-get remove <package>
135sudo apt-get --purge remove <package> # remove package and its configs from system
136sudo apt-get upgrade # apply all available updates to packages already installed
137sudo apt-get dist-upgrade # smarter upgrade
138sudo apt-get autoremove # get rid of older kernel versions
139sudo apt-get clean # cleans archived package files that have been installed
140;
141notice that update without upgrade effectively does nothing
142
143
144
145monitoring - chapter 11
146 many system monitoring tools make use of pseudo-file systems, especially in /proc and /sys
147(/proc and /sys pseudo-filesystems)
148
149main process and load monitoring utilities:
150 top # process activity, dynamically updated
151 uptime # how long system is running and avg load
152 ps # detailed info about processes
153 pstree # a tree of processes and their connections
154 mpstat # multiple CPU usage
155 iostat # CPU utilization and I/O stats
156 sar # display and collect info about system activity
157 numastat # info about NUMA (Non-Uniform Memory-Architecture)
158 strace # info about all system calls a process makes
159
160memory monitoring utilities:
161 free # brief summary of memory usage
162 vmstat # detailed virtual memory stats and block I/O, dynamically updated
163 pmap # process memory map
164
165I/O monitoring utilities
166 iostat # CPU utilization and I/O statistics
167 sar # display and collect info about system activity
168 vmstat # detailed virtual memory stats and block I/O, dynamically updated
169
170network monitoring utilities
171 netstat # detailed networking stats
172 iptraf # gather info on network if-aces
173 tcpdump # detailed analysis of network packets and traffic
174 wireshark # detailed network traffic analysis
175
176/proc and /sys:
177 pseudo-filesystems with lot of info about system; many are writable and writing to them will change system behavior
178 pseudo-filesystems bcs:
179 when system is not running, they are empty
180 only when user looks at them. they are not updated periodically
181 most *tunable* system parameters are in '/proc/sys/*'
182 TODO: fs/ - file system; net/, vm/
183 modifying values:
184 sudo bash -c 'echo 100000 > /proc/sys/kernel/threads-max'
185
186/sys is based on Unified Device Model, conceptually device tree, with buses, devices, etc.. Most lines contain only 1 line of text.
187
188You might find the output from "man hier" fascinating
189
190network devices:
191ls -lF /sys/class/net
192
193
194sar - System Activity Reporter. just a command line tool. it's backend is SADC - SYstem Activity Data Collector which accumulates statistics.
195 > sar [options] [interval] [count]
196np.:
197 > sar 3 3
198ciekawe przełączki
199 -A almost all, ściana tekstu
200
201stress - tool to stress CPU
202np.:
203 > stress -c 8 -i 4 -m 6 -t 20s
204 fork off 8 CPU-intensive processes, each doing sqrt()
205 fork off 4 I/O-intensive processes, each doing sync()
206 fork off 6 memory-intensive processes, each doing malloc(), allocating 256MB by default. Size can be changes as in --vm-bytes 128M
207
208chapter 12 - process monitoring
209(by the end of this chapter: ps, pstree, top)
210
211ps has 3 formats of options (to wyjaśnia tą dziwną składnię)
212
213> ps aux
214// processes that exist totally within the kernel are surrounded by [] (like [kthreadd])
215if there is one per CPU, number tells us on which CPU it runs
216
217legend:
218VSZ - virtual memory size in KB
219RSS - resident set size
220STAT - describes state of the process. mostly sleeping or running.
221 < high prio (not nice)
222 N low prio (nice)
223 L having pages locked in memory
224 s session leader
225 l multi-threaded
226 + being in the foreground process group
227
228adding f option (ps auxf) shows ancestry, like pstree (?)
229
230> ps -elf #unix option format
231#shows NI(ceness) and Parent Process ID
232
233you can specify output format with "-o", like:
234ps -o pid,uid,cputime,pmem,command
235
236
237/// chapter 13 - memoty, monitoring usage, tuning ///
238by the end:
239 list the primary (inter-related) considerations and tasks involved in memory tuning (?)
240 know entries of /proc/sys/vm and
241 decipher /proc/meminfo
242 understand OOM-killer (which selects processess to exterminate to open up some memory)
243
244when tweaking /proc/sys/vm, you want to change 1 thing and look for effects.
245also:
246 control flushing (?)
247 control swap behaviour
248 control overcomission (?)
249
250utilities to use:
251 free - brief summary of memory usage
252 vmstat - detailed virtual memory stats and block I/O, dynamically updated (nie u mnie raczej)
253 pmap - processor map
254
255values in /proc/sys/vm can be changed by:
256 directly writing to the entry. almost all entries are writable (by root)
257 using sysctl utility
258you can find docs describing this directory in the kernel source (?). Usually under Documentation/sysctl/vm.txt
259
260> vmstat [options] [delay] [count]
261> vmstat 2 4 # jakies delaye i county
262> vmstat -s # summary fajne
263> vmstat -d # table of disk statistics
264> vmstat -p /dev/sdb1 2 4 # staty podanej partycji, i jeszcze jakieÅ› polle wykonywane
265
26613.7.b. /proc/meminfo II
267tą tabelkę z opisami pól nauczyć się, przeanalizować, zrobić screena, cokolwiek
268
269OOM-Killer I
270Linux overcommits memory, w praktyce się to sprawdza bo mało który program wykorzystuje 100% zaalokowanej pamięci
271Whenever a child process is forked, it receives a copy of entire memory space of parent
272Bcs Linux uses COW (Copy on Write) technique, no actual copy needs to be made unless one of the processess modifies memory. However, the kernel has to assume that the copy might need to be done (?).
273If mem is exhausted, Linux invokes OOM-Killer (Out Of Memory-Killer) which decides which processess should be exterminated.
274
275Order of killing is determined by badness (/proc/[pid]/oom_score). normal user can only increase the badness. negative value can be given only by root. note that /proc/[pid]/oom_adj is deprecated
276
277sudo swapoff -a # turn off a swap
278
279# !!!!!!!
280dmesg # kernel msgs
281
282disabling swap partitions increases the chanses of the system invoking the OOM-Killer
283
284algorytm heurystyczny - poczytać co to
285
286
287
288/// chapter 14 - I/O Monitoring and Tuning ///
289by the end:
290 use iostat to monitor system I/O device activity
291 use iotop to display a constantly updated table of current I/O usage
292 use ionice to set both the I/O schedulling class and the priority for a given process
293
294system is considered I/O-bound when the CPU is found sitting idle waiting for I/O to complete, or the network is waiting to clear buffers
295
296I/O is complex. we'll consider I/O scheduling later
297
298> iostat # generates general I/O reports
299#tps - I/O transactions per sec; logical requests can be merged into one actual request
300# block read or written per unit time, where block is most of the time 512B
301# total block read or written
302# dm - device mapper
303
304> iostat -k # kB instead of blocks. "-m" also works
305
306> iostat -xk # extended!!!
307
308> iotop # top dla I/O
309> iotop -o # shows only devices that are inputting/outputting now
310
311> ionice -p [pid] # checking scheduling class and priority for a given process
312
313
314
315/// chapter 15 - I/O scheduling ///
316(???)
317system depends heavily on optimizing the I/O scheduling strategy
318by the end:
319 explain the importance of I/O scheduling and describe the conflicting requirements that need to be satisfied
320 delineate and contrast the options available under Linux (?)
321 understand how CFQ (Completely Fair Queue) and Deadline algorithms work
322
3232 layers: VM (Virtual Memory) and VFS (Virtual File System) submit I/O requests to block devices. it is the job of the scheduling layer to prioritize and order there requests before they are given to the block devices
324
325at least 1 I/O scheduling algorithm must be compiled into the kernel:
326 CFQ
327 Deadline Scheduling
328 noop (A simple scheme)
329CFG and DS are default
330
331> cat /sys/block/<sda>/queue/rotational # checks if disk is SSD (0 - SSD)
332
333> echo noop > /sys/block/<sda>/queue/scheduler
334> cat /sys/block/<sda>/queue/scheduler
335
336things to change vary according to the particular I/O scheduler and can be found under:
337/sys/block/<device>/queue/iosched
338
339<bla bla, nie rozumiem tego>
340
341
342
343/// chapter 16 - Linux Filesystems and the VFS ///
344!!!!!!!
345by the end:
346 explain the basic filesystem organisation
347 understand the role of VFS
348 know which filessytems are available and which ones can be used on your actual system
349 know why journaling filesystems are better
350 discuss the sue of special filesystems in Linux
351
352VFS - Linux nie musi wiedzieć dokładnie na jakims systemie plików działa
353Modern Linux filesystems:
354 ext4
355 xfs
356 btrfs
357
358!!!
359Linux uses inverted tree hierarchy ("/"). Usually there are multiple partitions joined together at mount points. They can also include reomvable media, like USB drives and others.
360Also certain "virtual pseudo filesystems" will be mounted within the tree, things like /proc, /sys, /tmp, /run
361
362Each of the elements within tree may in facy have its own filesystem variety (!). But to the OS it all appears in one, unified tree structure.
363Linux implements VFS, like every modern OS (!). It translates all I/O system calls into specific code relevant to the particular actual filesystem. Therefore, filesystem needs to be considered by applications. Also, network filesystems (such as NFS) can be handled transparently
364
365/proc has filesystem named "proc" (!)
366
367!!!
368
369> cat /proc/filesystems # outputs all filesystems "understood' by our OS
370
371> dd if=/dev/zero of=junk bs=1M count=512
372> sudo mkfs.xfs junk
373> sudo mount junk /mnt
374> df -h # by sprawdzić nowo powstały dysk
375> lsmod | less # we can see that xfs is now used (it wasn't before)
376
377newer filesystems include full "journaling" capability, which allows to recover from system crashes. it comes with a price of more operations to do.
378In journaling filesystems operations are grouped into "transactions". Each t. must be completed without error, atomically, or will not be completed.
379Ext3 was 1st journaling filesystem for Linux (Ext3 was Ext2 + journaling)
380
381Some of Linux's filesystems have no mount point - user apps don't interact with them, but kernel uses them, taking the advantage of VFS layers and code.
382
383tmpfs - expands its size dynamically. starts at 0, expands as necessary up to a max. size it was mounted with
384
385
386/// chapter 17 - Disk partitioning - introduction ///
387After:
388 describe and contrast most the common types of hard disks and data buses
389 partitioning strategies
390 Use blkid and fdisk
391 back up and restore partitions (!)
392
393 disk geometry. get geometry:
394 fdisk -l /dev/sda
395
396up to 4 partitions -> disk
3971 of them may be subdivided into logical partitions
398
399reasons to divide disks:
400 separation
401 sharing - through /home
402 security - imposed quotas, permissions and settings
403 size
404
405also:
406 performance - data can be accesses faster if it is either closer to the center or on a quicker disk
407 swap - Linux prefer specific swap partitions. Hibernation also use it
408
409512 bytes - MBR, including:
410 first 446 bytes - program, usually GRUB
411 16*4 partition tables
412 2 left (?)
413
414Each of those 16 bytes:
415 active bit
416 beginning address in cylinder/head/sectors (CHS) format (ignored by Linux)
417 partition tpe code indicating: xfs, LVM, ext4, ...
418 ending address in CHS (also ignored by Linux)
419 start sector, counting linearly from 0 } in Linux those 2 is coded using Linear Block Addressing (LBA)
420 number of sectors in partition }
421
422Linux normally access device nodes in /dev directory through infrastructure of kernel's Virtual File System
423SCSI and SATA disks naming:
424 sda, sdb - 1st disk, 2nd disk
425 sda1, sdc4 - 1st partition of 1st disk, 4th partition of 3rd disk
426back in the days of IDE disks it could be also:
427 /dev/hda3, /dev/hdb
428
429blkid - utility to locate block devices and report on their attributes. it works with libblkid library.
430Identifying disks with /dev/* is not reliable. It can change after changing port. use UUID instead. Blkid shows UUIDs.
431> sudo blkid /dev/sda*
432;
433lsblk - related utility which presents results in a tree format
434
435Linux requires min. 2 partitions:
436 /
437 swap - can be >1. on single disk system try to center swap. on multiple disk system try to spread it over disks.
438
439backing up system!!!
440sudo dd if=/dev/sda of=mbrbackup bs=512 count=1 # backing up MBR on first disk including 64-bit partition table which is part of it
441restoring!!!
442sudo dd of=mbrbackup of=/dev/sda bs=512 count=1
443
444note that it only copies the primary partition table, not partition tables stored in the other partitions (for extended partition, etc.).
445
446
447utilities to manage partition tables:
448 fdisk - menu driven partition table editor.
449 sfdisk - non=interactive, useful for scripting. use it CAREFULLY
450 parted - GNU partition manipulation program. It can create, remove, resize and move partitions (including certain FSes)
451 gparted - GUI parted. Popular on live editions of systems. It's better (?) to use only CLI tools. E.G. RHEL no longer supports gparted.
452
453Fdisk is ALWAYS included in Linux distro, so it's good to know it.
454> sudo fdisk /dev/sdb
455 m - display the menu
456 p - list the partition table
457 n - new partition
458 d - delete the partition
459 t - change partition type
460 w - write new partition table information and exit. Before using it, no changes are made!!! So before you use `w`, use `p`
461 q - quit without making changes. It's safe to quit before using `w`.
462
463When it asks for last sector, you can input:
464> +5G
465to create 5 Gb partition
466
467System will not use new partition table until you reboot. But
468> sudo partprobe -s
469tries to read new partitino table. not always reliable
470> cat /proc/partitions (!!!)
471to examine partitions system is currently aware of
472
473Then:
474> mkfs.ext4 /dev/sda3
475Proceed?
476> y
477
478
479
480/// chapter 18 - Filesystems features - attributes, creating, checking, mounting ///
481By the end I should be able to:
482 Explain concepts as inodes (<3), directory files, extended attributes
483 create and format filesystems
484 check and fix errors on FSes
485 mount and unmount FSes
486
487
488Inode - data structure describing and storing file attributes, including location. Every file is associated with its own inode.
489Info contained:
490 permissions
491 user and group ownership
492 size
493 timestamps (nanosecond)
494 last access time
495 last modification time
496 change time
497 NOT filenames - they are stored in directory file
498All I/O activity concerning a file usually also involves the file's inode as information be updated
499
500
501Directory file: specific type of file. Associated file names and inodes. 2 ways of doing this:
502 Hard links point to an inode
503 Soft (symbolic) -links point to a file name which has an associated inode
504
505chattr [+|-|=mode] filename
506lsattr filename
507
508those are equivalent:
509> sudo mkfs -t ext4 /dev/sda10
510> sudo mkfs.ext4 /dev/sda10
511
512there's also
513> fsck - file system check (?) // TODO
514
515
516mount - attach an FS at any point in the tree structure
517umount - detach them
518
519mount point must exist before the FS is attached. It must exist before mount can use it (TODO). mkdir will help here. If files existed there before, they will disappear and appear again after unmounting (TODO).
520Each FS is mounted under a specific directory
521> sudo mount -t ext /dev/sdb4 /home
522 o mounts an ext4 FS
523 o FS is located on a specific partition of a hard drive (/dev/sdb4/)
524 o FS is mounted at the position /home in the current directory tree
525 o Now-existing files in /home will disappear for now
526
527More examples:
528> sudo mount /dev/sda2 /home
529> sudo mount LABEL=home /home // (?)
530> sudo mount -L home /home
531> sudo mount UUID=1234134-23542345-235235... /home
532> sudo mount U=1234134-23542345-235235... /home
533
534labels are assignedby specific utilities, e.g. e2label.
535
536remounts a filesystem with read-only attribute:
537sudo mount -o remount, ro /myfs
538
539unmounting:
540> sudo umount /dev/sda2
541device must not be used to unmount it
542> fuser (...) # use it to find user currently using the filesystem
543> lsof (...) # list open files to see which files are being used and blocking unmounting
544
545mounting network drives:
546sudo mount -t nfs myserver.com:/sharedir /mnt/sharedir
547you can put in /fstab:
548myserver.com:/sharedir /mnt/sharedir nfs rsize=8192,wsize=8192,timeo=14,intr 0 0
549
550the system may try to mount it before network is up. There options might help:
551_netdev, noauto
552for more info:
553> man nfs
554> man mount
555
556During system boot command "mount -a" is executed. It mounts all filesystems in the /etc/fstab configuration file. (TODO)
557
558mount and umount can use info from /etc/fstab. So this can work if you have related setting in /etc/fstab:
559> mount /usr/src
560instead of:
561> mount LABEL=src /usr/src
562
563setting up (e.g.) pendrive to be mounted only when used:
564
565> grep automount /etc/fstab
566LABEL=Sam128 /SAM ext4 noauto,x-systemd.automount,x-systemd.device-timeout=10,x-systemd.idle-timeout=30 0 0
567> df -h | grep SAM
568ls /SAM
569<output of ls>
570> sleep 40
571>df -h | grep SAM
572<empty output>
573
574list of currently mounted FSes: (TODO)
575> mount
576
577
578see if there are swaps:
579> cat /proc/swaps
580
581> df -h -T # T - type
582
583
584
585>dumpe2fs /dev/sda2 | less
586TODO!!!
587check "mount count", "maximum mount count"
588> fsck /dev/sda2 (TODO)
589
590mkfs - format filesystems
591fsck - checking and fixing filesystems
592lsattr - list extended attributes of a file
593chattr - change extended attributes of a file
594lsof - list open files
595
596
597/// chapter 19 - Filesystems features - swap, quotas, usage ///
598By the end:
599 o Expain the concept of swap and quotas
600 o use utilities: quotacheck, quotaon, quotaoff, edquota, quota
601 o use utilities: df, du
602
603the only commands involving swapping are:
604> mkswap # format a swap partition / file
605> swapon # activate a swap partition / file
606> deactivate a swap partition / file
607
608> sudo quotacheck -vu /home # (TODO)
609
610> df # (TODO) (-i - inode)
611> du # (TODO)
612
613
614/// chapter 20 - Filesystems ext2/3/4 ///
615By the end:
616 → describe main features of ext4 and how it's laid out on disk
617 → explain the concept of block groups, superblock, data blocks, inodes
618 → use dumpe2fs and tune2fs utilities
619 → list ext4 FS enhancements
620
621ext2 - rarely used today
622ext3 - ext2 + journalling. other than that it has the same on-disk layout as ext2
623ext4 - mainly used now and default on most systems (but not on RHEL7 using XFS)
624
625ext* was designed to cooperate with VFS (and the other way around).
626
627inode reservation - feature that creates several inodes when a dir is created, expecting them to be used in the future. Performance++.
628
629Fields on a disk are written in little-endian order - except the journal.
630
631ext2/3/4 - layout of one block groups:
632super block, group descriptors, data block bitmap, inode bitmap, inode table (n blocks), data blocks (n blocks)
633
634super block - redundantly stored
635data block bitmap, inode bitmap - bits contain 0 or 1 for each one used
636inode table - each inode is 128 bytes, so 4KB block can contain 32 inodes
637
638Ext3:
639 → backwards-compatible with ext2/3
640 → max. size of a FS: 1 EB (was: 16TB) } those limits case from 48-bit addressing used
641 → max. file size: 16TB (was: 2TB) }
642 → increases max. number of subdirectories to inifinite* (was: 32k)
643 → better large file performance
644 → preallocating. allocated space is guaranteed and contiguous.
645 → use checksums for journal. reliability++
646 → timestamps are in [ms] now
647 → snapshot support
648
649/// chapter 21 - Filesystems XFS, btrfs ///
650By the end:
651 → describe XFS
652 → maintain XFS
653 → describe btrfs
654
655XFS - most maintenance tasks can be done while system is fully mounted: defragmenting, enlarging, dumping/restoring.
656
657
658
659/// chapter 22 - disk encryption ///
660By the end:
661 → why to use encryption
662 → understand how LUKS operates through the use of "cryptsetup"
663 → be able to set up and use encrypted filesystems and partitions
664 → be able to configure system to mount encrypted partitions at boot
665
666LUKS - Linux Unified Key Setup - block device level encryption. LUKS is installed on top of "cryptsetup", an utility that can user other methods like "plain dm-crypt" volumes, "loop-AES", "TrueCrypt". LUKS is default.
667
668Encrypting is done using "cryptsetup". Encrypting partition:
669> sudo cryptsetup luksFormat /dev/sda7
670if your system doesn't support default encryption method used by "cryptsetup", you can choose different one:
671> cat /proc/crypto (TODO)
672> sudo cryptsetup luksFormat --cipher aes /dev/sda7
673
674make volume available at any time with:
675> sudo cryptsetup --verbose luksOpen /dev/sda7
676format partition:
677> sudo mkfs.ext4 /dev/mapper/SECRET
678mount it:
679> sudo mount /dev/mapper/SECRET /mnt
680use it. then unmount:
681> sudo umount /mnt
682remove the mapper association for now, the partition will always be available for later use:
683> sudo cryptsetup --verbose luksClose SECRET
684
685to mount an encrypted partition at boot:
686 → add an appropriate entry in /etc/fstab. nothing special about it and it doesn't refer to encryption in any way
687 → add an entry to /etc/crypttab, as:
688 SECRET /dev/mapper/MYSECRET
689
690> man crypttab (TODO)
691
692
693> dd if=/dev/zero of=loop-partition bs=1M count=1024
694> losetup -f
695 /dev/loop1
696> sudo losetup /dev/loop2 loop=partition
697> losetup -l
698 <we can see our loop2 added>
699> sudo cryptsetup luksFormat /dev/loop2
700> YES
701 <now it should be encrypted>
702
703> sudo cryptsetup luksOpen /dev/lopp2 crypt-partition
704> ls -l /dev/mapper
705> sudo mkfs.ext4 /dev/mapper/crypt-partition
706> sudo mount /dev/mapper/crypt-partition
707...
708> sudo umount /dev/mapper/crypt-partition
709> sudo cryptsetup luksClose /dev/mapper/crypt-partition
710> sudo losetup -d /dev/loop2
711> losetup -f
712 /dev/loop1
713> rm loop-partition
714
715so the order is:
7161. create a partition for the encrypted block device
7172. format with cryptsetup
7183. create the un-encrypted pass through device
7194. format with a standard FS such as ext4
7205. mount the filesystem on the encrypted block device
721
722
723
724/// chapter 23 - logical volume mgmt (LVM) ///
725By the end:
726 → explain the concept behind LVM
727 → create, display, resize logical volumes
728 → use LVM snapshots
729
730LVMs are similar to RAIDs, and actually can build on top of RAID device. LVMs are more scalable.
731
732> sudo lvdisplay # shows all physical volumes, volume groups, logical volumes
733
734resize2fs - app to resize ext4 partitions.
735
736LVM snapshots are useful for backups, apps testing and deploying VMs
737
738
739
740/// chapter 24 - RAID ///
741By the end:
742 → explain the concept of RAID
743 → summarize RAID lvls
744 → configure a RAID
745 → monitor RAID devices
746 → use hot spares (?)
747
748RAID - Redundant Array of Independent Disks spreads I/O over multiple disks. May be SW (it's mature part of The Kernel) or HW. HW'll propably be faster.
749
750
751Striping - better performance by spreading data so simultaneous writes are possible
752Mirroring - same data on multiple disks, safety++
753
754mdadm - tool to create RAIDs.
755one created, array name: /dev/mdX can be used just like any other device, like /dev/sda1
756
757> sudo mdadm -S # stops RAID
758> sudo mdadm -S /dev/md0 # stops RAID
759
760steps to create a RAID:
7611. create partitions on each disk (type fd in fdisk)
7622. create RAID device with mdadm
7633. format RAID device
7644. add device to /etc/fstab
7655. mount RAID device
7666. capture RAID details to ensure persistence
767
768E.g.:
769> sudo fdisk /dev/sdb
770> sudo fdisk /dev/sdc
771
772> sudo mdadm --create /dev/md0 --level=1 --raid-disks=2 /dev/sdbX /dev/sdcX
773> sudo mkfs.ext4 /dev/md0
774> sudo bash -c "mdadm --detail" --scan >> /etc/mdadm.conf"
775> sudo mkdir /myraid
776> sudo mount /dev/md0 /myraid
777
778Be sure to add a line in /etc/fstab for the mount point:
779/dev/md0 /myraid ext4 defaults 0 2
780
781
782
783monitoring:
784 > sudo mdadm --detail /dev/md0
785 > cat /proc/mdstat
786
787 or use mdmonitor service:
788 > echo << "MAILADDR your@mail.com" >> /etc/mdadm.conf
789 start it by typing:
790 > sudo systemctl start mdmonitor } on Ubuntu it's rather called mdadm
791 > sudo systemctl enable mdmonitor }
792
793
794
795/// chapter 25 ///
796by the end:
797 → grasp the main responsibilities that Kernel must fulfill and how it achieves them
798 → explain what params can be set on the kernel command line and how to make them effective for one or more systems - persistently
799 → know where to find detailed documentation on there parameters
800 → know how to use sysctl to set kernel parameters either after the system starts, or persistently across system reboots
801
802responsibilities of kernel:
803 → system initalization and boot up
804 → process scheduling
805 → memory mgmt
806 → controlling access to HW
807 → I/O between apps and storage devices
808 → implementation of local and network FSes
809 → security control, both locally (such as FS permissions) and over the network
810 → networking control
811
812
813params are passed to system at boot on the kernel cmd line. they can be modified at boot.
814to see what options were used to start this system:
815> cat /proc/cmdline
816
817documentation of available kernel parameters:
818kernel source: Documentation/kernel-parameters.txt
819or by typing:
820> man bootparam
821
822params are given in form:
823param=value, like:
824vmlinuz root=/dev/sda6 ... noapic ... crashkernel=256M
825
826!!!
827sysctl - app to read and tune kernel parameters at runtime
828
829show current values:
830> sysctl -a
831browsing /proc/sys will render the same information
832
833showing values:
834> sysctl kernel.pid_max
835
836changing values:
837sudo sysctl net.ipv4.ip_forward=1
838
839> man 8 sysctl
840
841
842(???)
843> sudo sysctl -p
844if settings are placed in /etc/sysctl.conf (!!!), this will read file at boot
845> man sysctl.conf # for details
846on newer systems setting file is in:
847/usr/lib/sysctl.d/00-system
848but the original file is still supported
849
850exercise:
851lower pid_max to 29000
852
853
854
855/// chapter 26 - kernel modules ///
856by the end:
857 → list advantages of utilizing kernel modules
858 → use insmod, rmmod, modprobe to load and unload kernel modules
859 → use modinfo to find out info about kernel modules
860
861some parts can be added (or removed) as modules when necessary. all but most central kernel modules are integrated in such a fashion.
862they may or may not be device drivers.
863even though usage of kernel modules is wastly widespread, Linux is monolithic architecture rather that microkernel one. This is bcs once a module is loaded, it becomes a fully functional part of the kernel, with few restrictions. It communicated with all kernel subsystems via shared resources, such as memory and locks, rather than through message passing as might a microkernel.
864Solaris also uses modules.
865
866apps for modules:
867 → lsmod - list loaded modules (!!!)
868 → insmod - directly load a module
869 → rmmod - directly remove a module
870 → modprobe - (un)load modules, using a pre-built module DB with dependency info
871 → depmod - rebuild the module depencency DB; needed by `modprobe` and `modinfo`
872 → modinfo - display info about a module
873
874location of modules: (!!!)
875 /lib/modules/module_name.ko
876
877kernel modules always have extension: *.ko
878
879kernel modules are kernel version specific and must match the running kernel or they cannot be loaded (!!!). they must be compiled either when the kernel itself is compiled, or later, on a system which retains enough of the kernel source and compilation configuration
880
881ciekawostka:
882/lib/modules/$(uname -r) # where uname -r is current kernel version, such as 4.14.2
883
884it's impossible to unload a module being used by on or more other modules
885
886many modules can be loaded while specifying parameter values, such as;
887 /sbin/insmod <path_to>/e1000e.ko debug=2 copybreak=256
888or for module already loaded:
889 /sbin/modprobe e1000e debug=2 copybreak=256
890
891files in /etc/modprobe.d control params important when loading with `modprobe`, like:
892 → module name aliases
893 → automatically supplied options
894 → blacklist of some modules
895format of files is simple. one command per line. # for comments. \ at the end - continuation of a line in new line
896
897
898dmesg !!!!!!!!!!
899
900
901
902/// chapter 27 - devices and udev ///
903udev - intelligent mechanism to DYNAMICALLY discover HW and peripherial devices during boot or later. Device Nodes are created automatically and then used by apps and OS subsystems to communicate with and transfer data to and from devices.
904Admins can control how udev operates and craft special udev rules to assure desired behaviour results.
905
906by the end:
907 → explain role of device nodes, major and minor numbers.
908 → understand the need for udev method and list its key components
909 → describe how udev device manager functions
910 → identify udev rule files and learn how to create custom rules
911
912character and block devices have FS entries associated with them; network devices don't.
913Device nodes can be used by programs to communicate with devices, using normal I/O system calls, such as open(), close(), read(), write() (!!!!!!!).
914Network devices work by transmitting and receiving packets, which must be constructed by breaking up streams of data, or reassembled into streams when received.
915
916A device driver may manage multiple device nodes, which are normally placed in /dev directory:
917> ls -l /dev
918
919udev runs as daemon (named (?) udevd or systemd-udevd) and monitors a netlink socket. when new device is initialized or removed, uevent kernel facility sends a message through the socket, which udev deceives and takes appriopriate action to create/remove device node of the right names according to the rules.
920
9213 components of udev:
922 → libudev - library which allows access to information about the devices
923 → udevd / systemd-udevd daemon that manages the /dev directory
924 → udevadm - utility for control and diagnostics
925
926path of rules: (!!!)
927 /etc/udev/rules.d/*.rules
928 /usr/lib/udev/rules.d/*.rules
929
930
931
932/// chapter 28 - virtualization overview ///
933by the end:
934 → understand concept of virtualization, hosts and guests
935 → discuss difference: emulation vs virtualization
936 → types of hypervisors
937 → know how linux distros use and depend on libvirt
938 → use `qemu` hypervisor
939 → install, use and manage KVM (!!!)
940
941outside world sees the VM as it were an actual physical machine, present somewhere on the network. apps running in VMs are generally unaware of their non-physical environment.
942
943other kinds of virtualization:
944 → network - details of actual physical network, like types of HW, routers, are abstracted and need not be known by software running on it and configuring it
945 → storage - multiple network storage devices are configured to look like one big storage unit
946 → application - is isolated in container
947
948still there are important differences between physical and virtual machines.
949
950virtualization has long history and started on mainframes.
951on PCs initially it was done using emulation
952
953host - underlying physical OS managing 1 or more VMs
954guest - VM which is an instance of a complete OS, running 1 or more apps. Also: client.
955
956emulator runs completely in software. HW constructs are replaced by software. it is useful for running virtual machines on different architectures, such as running a pretend ARM guest machine on an X86 host. Emulation is often used for developing an OS for a new CPU, even before HW is avalilable (!). Performance is relatively slow.
957
9582 types of virtualization:
959 → HW v. (Full v.) - does not need modifications.
960 → Para-v. - guest system is aware it is running in a virtualized environment and has been modified specifically to work with it.
961
962recent CPUs from Intel and AMD incorporate virtualization extensions to the x86 architecture that allow full v. with only minor performance penalty.
963 → Intel - Intel V. Technology
964 → AMD - AMD-V (code-name: Pacifica)
965
966checking if your CPU supports HW v.:
967> cat /proc/cpuinfo
968if your CPU is IVT-capable, you'll see `vmx` in the flags field. If AMD-V: `svm`. you may also ensure v. capability is turned on in yuor CMOS.
969
970
971Hypervisor can be:
972 → External to host OS - VMWare
973 → Internal to host OS - KVM - we'll use this one here, it's Open Source and requires no external hypervisor program
974
975KVM added hypervisor capabilities into Linux kernel.
976Libvirt - project designed to be a toolkit to interact with virtualization technologies. Provides mgmt for virtual machines, virtual networks, storage. Some of the apps using it:
977 → virt-manager
978 → virt-viewer
979 → virt-install
980 → virsh
981
982> ls -lF /usr/bin/virt* # (!!!)
983
984
985QEMU - Quick Emulator. It emulates CPUs by dynamically translating binary instructions between the host architecture and emulated one.
986
987Can be used to emulate apps, not just an entire OS. Can save, pause, restore a VM at any time. License: GPL.
988
989In fact, QEMU has often been used to develop CPUs which have not been physically produced or released.
990
991We recommand using virt-manager (!) to configure and run virtual machines.
992
993list of supported formats:
994> qemu-img --help | grep formats
995
996
997
998/// chapter 29 - containers ///
999by the end:
1000 → know and use docker
1001
1002container - emulate only app (usually) or set of apps. unlike virtual machines, multiple containers can be run on 1 system. common method of deploying containers is using docker.
1003
1004worth mentioning are orchestration systems, such as kubernetes or mesos, can decide on the proper quantity of containers needed, do load balancing, replicate images and remove them, etc. as needed.
1005
1006docker is app-lvl virtualization uses many images to build up necessary services to support target app. these images are packaged into containers. they can contain:
1007 → app code
1008 → runtime libs
1009 → system tools
1010 → and more...
1011
1012most docker commands have own help. exaples are:
1013 → docker
1014 → docker-search
1015 → docker-pull
1016 → docker-create
1017 → docker-run
1018
1019ps will list running containers, or all containers (with --all param).
1020
1021docker command has >40 sub-commands, some with >50 options.
1022
1023often confused are commands run, create, exec.
1024 docker run will start a new container and execute command within.
1025 docker create creates a container. it has many options for configuring settings and attachments.
1026 if the container is already running, "docker exec" will execute something inside of it. accepts -t and -d params
1027 docker images - shows images in various outputs
1028 docker rmi - remove images and delete untagged parents by default
1029
1030 you can also use shell to operate upon all containers. example:
1031 > docker rm $(ps -a -q)
1032
1033
1034
1035/// chapter 30 - user account mgmt ///
1036 → explain purpose of user accounts and their main attribute
1037 → create new accounts, modify properties, remove or lock accounts
1038 → manage user's passwords
1039 → explain restricted shell and restricted accout
1040 → understand root account
1041
1042purpose of individual user accounts:
1043 → individualized personal space
1044 → create accounts for specific purposes
1045 → distinguish privileges
1046
1047daemon account - it exists to allow processes to run as a user other than root
1048
1049each user has correcponding line in /etc/passwd that describes account attributes, in format:
1050 beav:x:1000:1000:John Smith:/home/beav:/bin/bash
1051 username:user_password:UID:GID:some_contact_info(?):home_dir_path:login_shell
1052
1053 password - it's 'x' when /etc/shadow is used
1054 login_shell - generally any executable. look also for: /sbin/nologin.
1055
1056nologin refuses to a user to log in, shows default message and returns 0. if /etc/nologin.txt exists, message is overwritten by its content.
1057
1058creating user using some predefined algorithm (described in course):
1059> sudo useradd stephanie
1060default options can be overrulled:
1061> sudo useradd -s /bin/csh -m -k /etc/skel -c "John Smith" jsmith
1062
1063> userdel stephanie
1064account will be deleted, all references will be erased from:
1065 → /etc/passwd
1066 → /etc/shadow
1067 → /etc/group
1068
1069/home/stephanie will not be deleted so the account may be reestablished. delete also home of a user with "userdel -r". however all other files on the system owned by removed user will remain
1070
1071usermod - change params of user account
1072
1073> sudo usermod -L stephanie
1074lock stephanie account. it stays in the system, but logging in is impossible. it's a good practice to lock user account whenever they leave organization or will absent for longer period of time.
1075> sudo chage -E 2011-01-01 stephanie
1076where date is a date in the past. effect is the same as usermod usage above
1077
1078don't modify /etc/passwd, /etc/group, nor /etc/shadow
1079
1080/etc/shadow format:
1081 daemon:*:16141:0:99999:7:::
1082 ...
1083 beav:$sdyubgy7asdfb77bgf7yb7fg/ngfdyuagnfysgdfugsunayuga:16316:0:99999:7:::
1084so colon-separated fields are:
1085 username:password:lastchange:mindays:maxdays:warn:grace:expire:reserved
1086
1087 username name must match that one from /etc/passwd, order also must match.
1088 password hash is the string "$6$" followed by an eight chars salt value, then '$' and an 88 chars (sha512).
1089
1090/etc/passwd permissions are 644 (-rw-r--r--)
1091/etc/shadow permissions are 400 (-r--------) (only root can access it)
1092
1093you should use /etc/shadow unless you have a good reason not to do so
1094
1095normal user can change only his password:
1096> passwd
1097root can change anyone's password:
1098> sudo passwd kevin
1099passwords are examined by pam_cracklib.so
1100when root changes a user's password, is not prompted for the current password
1101
1102it is important to change passwords periodically.
1103> chage -l <username> # list passwords data
1104
1105> bash -r # restricted mode, disallowing user to do some things
1106
1107root login via network is generally prohibited.
1108
1109
1110ssh'ing:
1111> whoami
1112student
1113> ssh farflung.com
1114student@farflung.com's password: (type here) #we assume there is 'student' account on farflung.com
1115> ssh root@farflung.com
1116
1117copy'ing files:
1118> scp file.txt farflung.com:/tmp
1119> scp file.txt student@farflung.com/home/student
1120> scp -r some_dir farflung.com:/tmp/some_dir
1121
1122to run command on multiple machines:
1123> for machines in node1 node2 node3
1124 do (ssh $machines some_command &)
1125 done
1126
1127permitting to log in without a password:
1128> ls -l ~/.ssh
1129id_rsa user's private encryption key; NEVER show it to anyone
1130id_rsa.pub user's public encryption key
1131authorized_keys list of public keys that are permitted to login; info about USERS AND NODES
1132known_hosts a list of hosts from which logins have been allowed in the past; ONLY info about computer NODES
1133config a configuration file for specifying various options
1134
1135
1136(!!!)
1137to log in to remote machine with full GUI, use VNC (Virtual Network Computing) client. a common implementation is "tigervnc".
1138> sudo apt-get install tigervnc tigervnc-server
1139#start server as normal user
1140> vncserver
1141> vncviewer localhost:2 #test it. you might use different number: 1, 3, 4 depending on how your machine is configured.
1142view from remote machine:
1143> vncviewer -via student@some_machine localhost:2
1144
1145if some "color profile" bug occurs, kill the "colord" daemon
1146> sudo systemctl stop colord
1147
1148
1149
1150working with accounts:
1151> less /etc/default/useradd #we can see env var, for example SHELL=/bin/sh
1152> sudo useradd -m "some name" -s /bin/bash someName666 # -m to make sure it creates a home dir
1153> sudo passwd someName666 #and type some password
1154> cat /etc/passwd /etc/group | grep someName666
1155/etc/passwd:someName666:...................
1156/etc/group:someName666:x:1001: