· 8 years ago · Sep 13, 2017, 01:18 AM
1#######################
2# VMs for this course #
3#######################
4https://infosecaddictsfiles.blob.core.windows.net/vms/Win7x64.zip
5 username: workshop
6 password: password
7
8https://infosecaddictsfiles.blob.core.windows.net/vms/InfoSecAddictsVM.zip
9user: infosecaddicts
10pass: infosecaddicts
11
12You don't have to, but you can do the updates in the Win7 VM (yes, it is a lot of updates).
13
14You'll need to create directory in the Win7 VM called "c:\ps"
15
16In this file you will also need to change the text '192.168.200.144' to the IP address of your Ubuntu host.
17
18
19
20
21
22##############################################
23# Log Analysis with Linux command-line tools #
24##############################################
25The following command line executables are found in the Mac as well as most Linux Distributions.
26
27cat – prints the content of a file in the terminal window
28grep – searches and filters based on patterns
29awk – can sort each row into fields and display only what is needed
30sed – performs find and replace functions
31sort – arranges output in an order
32uniq – compares adjacent lines and can report, filter or provide a count of duplicates
33
34
35##############
36# Cisco Logs #
37##############
38
39wget https://infosecaddictsfiles.blob.core.windows.net/files/cisco.log
40
41
42AWK Basics
43----------
44To quickly demonstrate the print feature in awk, we can instruct it to show only the 5th word of each line. Here we will print $5. Only the last 4 lines are being shown for brevity.
45
46cat cisco.log | awk '{print $5}' | tail -n 4
47
48
49
50
51Looking at a large file would still produce a large amount of output. A more useful thing to do might be to output every entry found in “$5â€, group them together, count them, then sort them from the greatest to least number of occurrences. This can be done by piping the output through “sort“, using “uniq -c†to count the like entries, then using “sort -rn†to sort it in reverse order.
52
53cat cisco.log | awk '{print $5}'| sort | uniq -c | sort -rn
54
55
56
57
58While that’s sort of cool, it is obvious that we have some garbage in our output. Evidently we have a few lines that aren’t conforming to the output we expect to see in $5. We can insert grep to filter the file prior to feeding it to awk. This insures that we are at least looking at lines of text that contain “facility-level-mnemonicâ€.
59
60cat cisco.log | grep %[a-zA-Z]*-[0-9]-[a-zA-Z]* | awk '{print $5}' | sort | uniq -c | sort -rn
61
62
63
64
65
66Now that the output is cleaned up a bit, it is a good time to investigate some of the entries that appear most often. One way to see all occurrences is to use grep.
67
68cat cisco.log | grep %LINEPROTO-5-UPDOWN:
69
70cat cisco.log | grep %LINEPROTO-5-UPDOWN:| awk '{print $10}' | sort | uniq -c | sort -rn
71
72cat cisco.log | grep %LINEPROTO-5-UPDOWN:| sed 's/,//g' | awk '{print $10}' | sort | uniq -c | sort -rn
73
74cat cisco.log | grep %LINEPROTO-5-UPDOWN:| sed 's/,//g' | awk '{print $10 " changed to " $14}' | sort | uniq -c | sort -rn
75
76
77
78
79#########
80# EGrep #
81#########
82
83
84
85
86
87#####################
88# Powershell Basics #
89#####################
90
91PowerShell is Microsoft’s new scripting language that has been built in since the release Vista.
92
93PowerShell file extension end in .ps1 .
94
95An important note is that you cannot double click on a PowerShell script to execute it.
96
97To open a PowerShell command prompt either hit Windows Key + R and type in PowerShell or Start -> All Programs -> Accessories -> Windows PowerShell -> Windows PowerShell.
98
99dir
100cd
101ls
102cd c:\
103
104
105To obtain a list of cmdlets, use the Get-Command cmdlet
106
107Get-Command
108
109
110
111You can use the Get-Alias cmdlet to see a full list of aliased commands.
112
113Get-Alias
114
115
116
117Don't worry you won't blow up your machine with Powershell
118Get-Process | stop-process Don't press [ ENTER ] What will this command do?
119Get-Process | stop-process -whatif
120
121
122To get help with a cmdlet, use the Get-Help cmdlet along with the cmdlet you want information about.
123
124Get-Help Get-Command
125
126Get-Help Get-Service –online
127
128Get-Service -Name TermService, Spooler
129
130Get-Service –N BITS
131
132
133
134PowerShell variables begin with the $ symbol. First lets create a variable
135
136$serv = Get-Service –N Spooler
137
138To see the value of a variable you can just call it in the terminal.
139
140$serv
141
142$serv.gettype().fullname
143
144
145Get-Member is another extremely useful cmdlet that will enumerate the available methods and properties of an object. You can pipe the object to Get-Member or pass it in
146
147$serv | Get-Member
148
149Get-Member -InputObject $serv
150
151
152
153
154
155Let’s use a method and a property with our object.
156
157$serv.Status
158$serv.Stop()
159$serv.Refresh()
160$serv.Status
161$serv.Start()
162$serv.Refresh()
163$serv.Status
164
165
166
167
168#############################
169# Simple Event Log Analysis #
170#############################
171
172Step 1: Dump the event logs
173---------------------------
174The first thing to do is to dump them into a format that facilitates later processing with Windows PowerShell.
175
176To dump the event log, you can use the Get-EventLog and the Exportto-Clixml cmdlets if you are working with a traditional event log such as the Security, Application, or System event logs.
177If you need to work with one of the trace logs, use the Get-WinEvent and the ExportTo-Clixml cmdlets.
178
179Get-EventLog -LogName application | Export-Clixml Applog.xml
180
181type .\Applog.xml
182
183$logs = "system","application","security"
184
185The % symbol is an alias for the Foreach-Object cmdlet. It is often used when working interactively from the Windows PowerShell console
186
187$logs | % { get-eventlog -LogName $_ | Export-Clixml "$_.xml" }
188
189
190
191
192
193Step 2: Import the event log of interest
194----------------------------------------
195To parse the event logs, use the Import-Clixml cmdlet to read the stored XML files.
196Store the results in a variable.
197Let's take a look at the commandlets Where-Object, Group-Object, and Select-Object.
198
199The following two commands first read the exported security log contents into a variable named $seclog, and then the five oldest entries are obtained.
200
201$seclog = Import-Clixml security.xml
202
203$seclog | select -Last 5
204
205
206Cool trick from one of our students named Adam. This command allows you to look at the logs for the last 24 hours:
207
208Get-EventLog Application -After (Get-Date).AddDays(-1)
209
210You can use '-after' and '-before' to filter date ranges
211
212One thing you must keep in mind is that once you export the security log to XML, it is no longer protected by anything more than the NFTS and share permissions that are assigned to the location where you store everything.
213By default, an ordinary user does not have permission to read the security log.
214
215
216
217
218Step 3: Drill into a specific entry
219-----------------------------------
220To view the entire contents of a specific event log entry, choose that entry, send the results to the Format-List cmdlet, and choose all of the properties.
221
222
223$seclog | select -first 1 | fl *
224
225The message property contains the SID, account name, user domain, and privileges that are assigned for the new login.
226
227
228($seclog | select -first 1).message
229
230(($seclog | select -first 1).message).gettype()
231
232
233
234In the *nix world you often want a count of something (wc -l).
235How often is the SeSecurityPrivilege privilege mentioned in the message property?
236To obtain this information, pipe the contents of the security log to a Where-Object to filter the events, and then send the results to the Measure-Object cmdlet to determine the number of events:
237
238$seclog | ? { $_.message -match 'SeSecurityPrivilege'} | measure
239
240If you want to ensure that only event log entries return that contain SeSecurityPrivilege in their text, use Group-Object to gather the matches by the EventID property.
241
242
243$seclog | ? { $_.message -match 'SeSecurityPrivilege'} | group eventid
244
245Because importing the event log into a variable from the stored XML results in a collection of event log entries, it means that the count property is also present.
246Use the count property to determine the total number of entries in the event log.
247
248$seclog.Count
249
250
251
252
253
254
255############################
256# Simple Log File Analysis #
257############################
258
259
260You'll need to create the directory c:\ps and download sample iss log http://pastebin.com/raw.php?i=LBn64cyA
261
262
263mkdir c:\ps
264cd c:\ps
265(new-object System.Net.WebClient).DownloadFile("http://pastebin.com/raw.php?i=LBn64cyA", "c:\ps\u_ex1104.log")
266(new-object System.Net.WebClient).DownloadFile("http://pastebin.com/raw.php?i=ysnhXxTV", "c:\ps\CiscoLogFileExamples.txt")
267
268Select-String 192.168.208.63 .\CiscoLogFileExamples.txt
269
270
271
272
273The Select-String cmdlet searches for text and text patterns in input strings and files. You can use it like Grep in UNIX and Findstr in Windows.
274
275Select-String 192.168.208.63 .\CiscoLogFileExamples.txt | select line
276
277
278
279
280To see how many connections are made when analyzing a single host, the output from that can be piped to another command: Measure-Object.
281
282Select-String 192.168.208.63 .\CiscoLogFileExamples.txt | select line | Measure-Object
283
284
285
286To select all IP addresses in the file expand the matches property, select the value, get unique values and measure the output.
287
288Select-String “\b(?:\d{1,3}\.){3}\d{1,3}\b†.\CiscoLogFileExamples.txt | select -ExpandProperty matches | select -ExpandProperty value | Sort-Object -Unique | Measure-Object
289
290
291
292Removing Measure-Object shows all the individual IPs instead of just the count of the IP addresses. The Measure-Object command counts the IP addresses.
293
294Select-String “\b(?:\d{1,3}\.){3}\d{1,3}\b†.\CiscoLogFileExamples.txt | select -ExpandProperty matches | select -ExpandProperty value | Sort-Object -Unique
295
296
297In order to determine which IP addresses have the most communication the last commands are removed to determine the value of the matches. Then the group command is issued on the piped output to group all the IP addresses (value), and then sort the objects by using the alias for Sort-Object: sort count –des.
298This sorts the IP addresses in a descending pattern as well as count and deliver the output to the shell.
299
300Select-String “\b(?:\d{1,3}\.){3}\d{1,3}\b†.\CiscoLogFileExamples.txt | select -ExpandProperty matches | select value | group value | sort count -des
301
302##############################################
303# Parsing Log files using windows PowerShell #
304##############################################
305
306Download the sample IIS log http://pastebin.com/LBn64cyA
307
308
309(new-object System.Net.WebClient).DownloadFile("http://pastebin.com/raw.php?i=LBn64cyA", "c:\ps\u_ex1104.log")
310
311Get-Content ".\*log" | ? { ($_ | Select-String "WebDAV")}
312
313
314
315The above command would give us all the WebDAV requests.
316
317To filter this to a particular user name, use the below command:
318
319Get-Content ".\*log" | ? { ($_ | Select-String "WebDAV") -and ($_ | Select-String "OPTIONS")}
320
321
322
323Some more options that will be more commonly required :
324
325For Outlook Web Access : Replace WebDAV with OWA
326
327For EAS : Replace WebDAV with Microsoft-server-activesync
328
329For ECP : Replace WebDAV with ECP
330
331
332
333
334#######################################
335# Regex Characters you might run into #
336#######################################
337
338^ Start of string, or start of line in a multiline pattern
339$ End of string, or start of line in a multiline pattern
340\b Word boundary
341\d Digit
342\ Escape the following character
343* 0 or more {3} Exactly 3
344+ 1 or more {3,} 3 or more
345? 0 or 1 {3,5} 3, 4 or 5
346
347
348
349####################################################################
350# Windows PowerShell: Extracting Strings Using Regular Expressions #
351####################################################################
352To build a script that will extract data from a text file and place the extracted text into another file, we need three main elements:
353
3541) The input file that will be parsed
355
356(new-object System.Net.WebClient).DownloadFile("http://pastebin.com/raw.php?i=rDN3CMLc", "c:\ps\emails.txt")
357(new-object System.Net.WebClient).DownloadFile("http://pastebin.com/raw.php?i=XySD8Mi2", "c:\ps\ip_addresses.txt")
358(new-object System.Net.WebClient).DownloadFile("http://pastebin.com/raw.php?i=v5Yq66sH", "c:\ps\URL_addresses.txt")
359
3602) The regular expression that the input file will be compared against
361
3623) The output file for where the extracted data will be placed.
363
364Windows PowerShell has a “select-string†cmdlet which can be used to quickly scan a file to see if a certain string value exists.
365Using some of the parameters of this cmdlet, we are able to search through a file to see whether any strings match a certain pattern, and then output the results to a separate file.
366
367To demonstrate this concept, below is a Windows PowerShell script I created to search through a text file for strings that match the Regular Expression (or RegEx for short) pattern belonging to e-mail addresses.
368
369$input_path = ‘c:\ps\emails.txt’
370$output_file = ‘c:\ps\extracted_addresses.txt’
371$regex = ‘\b[A-Za-z0-9._%-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,4}\b’
372select-string -Path $input_path -Pattern $regex -AllMatches | % { $_.Matches } | % { $_.Value } > $output_file
373
374In this script, we have the following variables:
375
3761) $input_path to hold the path to the input file we want to parse
377
3782) $output_file to hold the path to the file we want the results to be stored in
379
3803) $regex to hold the regular expression pattern to be used when the strings are being matched.
381
382The select-string cmdlet contains various parameters as follows:
383
3841) “-Path†which takes as input the full path to the input file
385
3862) “-Pattern†which takes as input the regular expression used in the matching process
387
3883) “-AllMatches†which searches for more than one match (without this parameter it would stop after the first match is found) and is piped to “$.Matches†and then “$_.Value†which represent using the current values of all the matches.
389
390Using “>†the results are written to the destination specified in the $output_file variable.
391
392Here are two further examples of this script which incorporate a regular expression for extracting IP addresses and URLs.
393
394IP addresses
395------------
396For the purposes of this example, I ran the tracert command to trace the route from my host to google.com and saved the results into a file called ip_addresses.txt. You may choose to use this script for extracting IP addresses from router logs, firewall logs, debug logs, etc.
397
398$input_path = ‘c:\ps\ip_addresses.txt’
399$output_file = ‘c:\ps\extracted_ip_addresses.txt’
400$regex = ‘\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b’
401select-string -Path $input_path -Pattern $regex -AllMatches | % { $_.Matches } | % { $_.Value } > $output_file
402
403
404URLs
405----
406For the purposes of this example, I created a couple of dummy web server log entries and saved them into URL_addresses.txt.
407You may choose to use this script for extracting URL addresses from proxy logs, network packet capture logs, debug logs, etc.
408
409$input_path = ‘c:\ps\URL_addresses.txt’
410$output_file = ‘c:\ps\extracted_URL_addresses.txt’
411$regex = ‘([a-zA-Z]{3,})://([\w-]+\.)+[\w-]+(/[\w- ./?%&=]*)*?’
412select-string -Path $input_path -Pattern $regex -AllMatches | % { $_.Matches } | % { $_.Value } > $output_file
413
414
415In addition to the examples above, many other types of strings can be extracted using this script.
416All you need to do is switch the regular expression in the “$regex†variable!
417In fact, the beauty of such a PowerShell script is its simplicity and speed of execution.
418
419
420
421
422
423
424
425
426###################
427# Regex in Python #
428###################
429
430
431
432
433**************************************************
434* What is Regular Expression and how is it used? *
435**************************************************
436
437
438Simply put, regular expression is a sequence of character(s) mainly used to find and replace patterns in a string or file.
439
440
441Regular expressions use two types of characters:
442
443a) Meta characters: As the name suggests, these characters have a special meaning, similar to * in wildcard.
444
445b) Literals (like a,b,1,2…)
446
447
448In Python, we have module "re" that helps with regular expressions. So you need to import library re before you can use regular expressions in Python.
449
450
451Use this code --> import re
452
453
454
455
456The most common uses of regular expressions are:
457--------------------------------------------------
458
459- Search a string (search and match)
460- Finding a string (findall)
461- Break string into a sub strings (split)
462- Replace part of a string (sub)
463
464
465
466Let's look at the methods that library "re" provides to perform these tasks.
467
468
469
470****************************************************
471* What are various methods of Regular Expressions? *
472****************************************************
473
474
475The ‘re' package provides multiple methods to perform queries on an input string. Here are the most commonly used methods, I will discuss:
476
477re.match()
478re.search()
479re.findall()
480re.split()
481re.sub()
482re.compile()
483
484Let's look at them one by one.
485
486
487re.match(pattern, string):
488-------------------------------------------------
489
490This method finds match if it occurs at start of the string. For example, calling match() on the string ‘AV Analytics AV' and looking for a pattern ‘AV' will match. However, if we look for only Analytics, the pattern will not match. Let's perform it in python now.
491
492Code
493
494import re
495result = re.match(r'AV', 'AV Analytics ESET AV')
496print result
497
498Output:
499<_sre.SRE_Match object at 0x0000000009BE4370>
500
501Above, it shows that pattern match has been found. To print the matching string we'll use method group (It helps to return the matching string). Use "r" at the start of the pattern string, it designates a python raw string.
502
503
504result = re.match(r'AV', 'AV Analytics ESET AV')
505print result.group(0)
506
507Output:
508AV
509
510
511Let's now find ‘Analytics' in the given string. Here we see that string is not starting with ‘AV' so it should return no match. Let's see what we get:
512
513
514Code
515
516result = re.match(r'Analytics', 'AV Analytics ESET AV')
517print result
518
519
520Output:
521None
522
523
524There are methods like start() and end() to know the start and end position of matching pattern in the string.
525
526Code
527
528result = re.match(r'AV', 'AV Analytics ESET AV')
529print result.start()
530print result.end()
531
532Output:
5330
5342
535
536Above you can see that start and end position of matching pattern ‘AV' in the string and sometime it helps a lot while performing manipulation with the string.
537
538
539
540
541
542re.search(pattern, string):
543-----------------------------------------------------
544
545
546It is similar to match() but it doesn't restrict us to find matches at the beginning of the string only. Unlike previous method, here searching for pattern ‘Analytics' will return a match.
547
548Code
549
550result = re.search(r'Analytics', 'AV Analytics ESET AV')
551print result.group(0)
552
553Output:
554Analytics
555
556Here you can see that, search() method is able to find a pattern from any position of the string but it only returns the first occurrence of the search pattern.
557
558
559
560
561
562
563re.findall (pattern, string):
564------------------------------------------------------
565
566
567It helps to get a list of all matching patterns. It has no constraints of searching from start or end. If we will use method findall to search ‘AV' in given string it will return both occurrence of AV. While searching a string, I would recommend you to use re.findall() always, it can work like re.search() and re.match() both.
568
569
570Code
571
572result = re.findall(r'AV', 'AV Analytics ESET AV')
573print result
574
575Output:
576['AV', 'AV']
577
578
579
580
581
582re.split(pattern, string, [maxsplit=0]):
583------------------------------------------------------
584
585
586
587This methods helps to split string by the occurrences of given pattern.
588
589
590Code
591
592result=re.split(r'y','Analytics')
593result
594
595Output:
596['Anal', 'tics']
597
598Above, we have split the string "Analytics" by "y". Method split() has another argument "maxsplit". It has default value of zero. In this case it does the maximum splits that can be done, but if we give value to maxsplit, it will split the string. Let's look at the example below:
599
600
601Code
602
603result=re.split(r's','Analytics eset')
604print result
605
606Output:
607['Analytic', 'e', 'et'] #It has performed all the splits that can be done by pattern "s".
608
609Code
610
611result=re.split(r's','Analytics eset',maxsplit=1)
612result
613
614Output:
615['Analytic', 'eset']
616
617Here, you can notice that we have fixed the maxsplit to 1. And the result is, it has only two values whereas first example has three values.
618
619
620
621
622re.sub(pattern, repl, string):
623----------------------------------------------------------
624
625It helps to search a pattern and replace with a new sub string. If the pattern is not found, string is returned unchanged.
626
627Code
628
629result=re.sub(r'Ruby','Python','Joe likes Ruby')
630result
631Output:
632'Joe likes Python'
633
634
635
636
637
638re.compile(pattern, repl, string):
639----------------------------------------------------------
640
641
642We can combine a regular expression pattern into pattern objects, which can be used for pattern matching. It also helps to search a pattern again without rewriting it.
643
644
645Code
646
647import re
648pattern=re.compile('XSS')
649result=pattern.findall('XSS is Cross Site Sripting, XSS')
650print result
651result2=pattern.findall('XSS is Cross Site Scripting, SQLi is Sql Injection')
652print result2
653Output:
654['XSS', 'XSS']
655['XSS']
656
657Till now, we looked at various methods of regular expression using a constant pattern (fixed characters). But, what if we do not have a constant search pattern and we want to return specific set of characters (defined by a rule) from a string? Don't be intimidated.
658
659This can easily be solved by defining an expression with the help of pattern operators (meta and literal characters). Let's look at the most common pattern operators.
660
661
662
663
664
665**********************************************
666* What are the most commonly used operators? *
667**********************************************
668
669
670Regular expressions can specify patterns, not just fixed characters. Here are the most commonly used operators that helps to generate an expression to represent required characters in a string or file. It is commonly used in web scrapping and text mining to extract required information.
671
672Operators Description
673. Matches with any single character except newline ‘\n'.
674? match 0 or 1 occurrence of the pattern to its left
675+ 1 or more occurrences of the pattern to its left
676* 0 or more occurrences of the pattern to its left
677\w Matches with a alphanumeric character whereas \W (upper case W) matches non alphanumeric character.
678\d Matches with digits [0-9] and /D (upper case D) matches with non-digits.
679\s Matches with a single white space character (space, newline, return, tab, form) and \S (upper case S) matches any non-white space character.
680\b boundary between word and non-word and /B is opposite of /b
681[..] Matches any single character in a square bracket and [^..] matches any single character not in square bracket
682\ It is used for special meaning characters like \. to match a period or \+ for plus sign.
683^ and $ ^ and $ match the start or end of the string respectively
684{n,m} Matches at least n and at most m occurrences of preceding expression if we write it as {,m} then it will return at least any minimum occurrence to max m preceding expression.
685a| b Matches either a or b
686( ) Groups regular expressions and returns matched text
687\t, \n, \r Matches tab, newline, return
688
689
690For more details on meta characters "(", ")","|" and others details , you can refer this link (https://docs.python.org/2/library/re.html).
691
692Now, let's understand the pattern operators by looking at the below examples.
693
694
695
696****************************************
697* Some Examples of Regular Expressions *
698****************************************
699
700******************************************************
701* Problem 1: Return the first word of a given string *
702******************************************************
703
704
705Solution-1 Extract each character (using "\w")
706---------------------------------------------------------------------------
707
708Code
709
710import re
711result=re.findall(r'.','Python is the best scripting language')
712print result
713
714Output:
715['P', 'y', 't', 'h', 'o', 'n', ' ', 'i', 's', ' ', 't', 'h', 'e', ' ', 'b', 'e', 's', 't', ' ', 's', 'c', 'r', 'i', 'p', 't', 'i', 'n', 'g', ' ', 'l', 'a', 'n', 'g', 'u', 'a', 'g', 'e']
716
717
718Above, space is also extracted, now to avoid it use "\w" instead of ".".
719
720
721Code
722
723result=re.findall(r'\w','Python is the best scripting language')
724print result
725
726Output:
727['P', 'y', 't', 'h', 'o', 'n', 'i', 's', 't', 'h', 'e', 'b', 'e', 's', 't', 's', 'c', 'r', 'i', 'p', 't', 'i', 'n', 'g', 'l', 'a', 'n', 'g', 'u', 'a', 'g', 'e']
728
729
730
731
732Solution-2 Extract each word (using "*" or "+")
733---------------------------------------------------------------------------
734
735Code
736
737result=re.findall(r'\w*','Python is the best scripting language')
738print result
739
740Output:
741['Python', '', 'is', '', 'the', '', 'best', '', 'scripting', '', 'language', '']
742
743
744Again, it is returning space as a word because "*" returns zero or more matches of pattern to its left. Now to remove spaces we will go with "+".
745
746Code
747
748result=re.findall(r'\w+','Python is the best scripting language')
749print result
750Output:
751['Python', 'is', 'the', 'best', 'scripting', 'language']
752
753
754
755
756
757Solution-3 Extract each word (using "^")
758-------------------------------------------------------------------------------------
759
760
761Code
762
763result=re.findall(r'^\w+','Python is the best scripting language')
764print result
765
766Output:
767['Python']
768
769If we will use "$" instead of "^", it will return the word from the end of the string. Let's look at it.
770
771Code
772
773result=re.findall(r'\w+$','Python is the best scripting language')
774print result
775Output:
776[‘language']
777
778
779
780
781
782**********************************************************
783* Problem 2: Return the first two character of each word *
784**********************************************************
785
786
787
788
789Solution-1 Extract consecutive two characters of each word, excluding spaces (using "\w")
790------------------------------------------------------------------------------------------------------
791
792Code
793
794result=re.findall(r'\w\w','Python is the best')
795print result
796
797Output:
798['Py', 'th', 'on', 'is,', 'th', 'eb', 'es']
799
800
801
802
803
804
805Solution-2 Extract consecutive two characters those available at start of word boundary (using "\b")
806------------------------------------------------------------------------------------------------------
807
808Code
809
810result=re.findall(r'\b\w.','Python is the best')
811print result
812
813Output:
814['Py', 'is,', 'th', 'be']
815
816
817
818
819
820
821********************************************************
822* Problem 3: Return the domain type of given email-ids *
823********************************************************
824
825
826To explain it in simple manner, I will again go with a stepwise approach:
827
828
829
830
831
832Solution-1 Extract all characters after "@"
833------------------------------------------------------------------------------------------------------------------
834
835Code
836
837result=re.findall(r'@\w+','abc.test@gmail.com, xyz@test.com, test.first@strategicsec.com, first.test@rest.biz')
838print result
839
840Output: ['@gmail', '@test', '@strategicsec', '@rest']
841
842
843
844Above, you can see that ".com", ".biz" part is not extracted. To add it, we will go with below code.
845
846
847result=re.findall(r'@\w+.\w+','abc.test@gmail.com, xyz@test.com, test.first@strategicsec.com, first.test@rest.biz')
848print result
849
850Output:
851['@gmail.com', '@test.com', '@strategicsec.com', '@rest.biz']
852
853
854
855
856
857
858Solution – 2 Extract only domain name using "( )"
859-----------------------------------------------------------------------------------------------------------------------
860
861
862Code
863
864result=re.findall(r'@\w+.(\w+)','abc.test@gmail.com, xyz@test.com, test.first@strategicsec.com, first.test@rest.biz')
865print result
866
867Output:
868['com', 'com', 'com', 'biz']
869
870
871
872
873
874
875********************************************
876* Problem 4: Return date from given string *
877********************************************
878
879
880Here we will use "\d" to extract digit.
881
882
883Solution:
884----------------------------------------------------------------------------------------------------------------------
885
886Code
887
888result=re.findall(r'\d{2}-\d{2}-\d{4}','Joe 34-3456 12-05-2007, XYZ 56-4532 11-11-2016, ABC 67-8945 12-01-2009')
889print result
890
891Output:
892['12-05-2007', '11-11-2016', '12-01-2009']
893
894If you want to extract only year again parenthesis "( )" will help you.
895
896
897Code
898
899
900result=re.findall(r'\d{2}-\d{2}-(\d{4})','Joe 34-3456 12-05-2007, XYZ 56-4532 11-11-2016, ABC 67-8945 12-01-2009')
901print result
902
903Output:
904['2007', '2016', '2009']
905
906
907
908
909
910*******************************************************************
911* Problem 5: Return all words of a string those starts with vowel *
912*******************************************************************
913
914
915
916
917Solution-1 Return each words
918-----------------------------------------------------------------------------------------------------------------
919
920Code
921
922result=re.findall(r'\w+','Python is the best')
923print result
924
925Output:
926['Python', 'is', 'the', 'best']
927
928
929
930
931
932Solution-2 Return words starts with alphabets (using [])
933------------------------------------------------------------------------------------------------------------------
934
935Code
936
937result=re.findall(r'[aeiouAEIOU]\w+','I love Python')
938print result
939
940Output:
941['I', 'ove', 'on']
942
943Above you can see that it has returned "ove" and "on" from the mid of words. To drop these two, we need to use "\b" for word boundary.
944
945
946
947
948
949Solution- 3
950------------------------------------------------------------------------------------------------------------------
951
952Code
953
954result=re.findall(r'\b[aeiouAEIOU]\w+','I love Python')
955print result
956
957Output:
958['I']
959
960
961In similar ways, we can extract words those starts with constant using "^" within square bracket.
962
963
964Code
965
966result=re.findall(r'\b[^aeiouAEIOU]\w+','I love Python')
967print result
968
969Output:
970[' love', ' Python']
971
972Above you can see that it has returned words starting with space. To drop it from output, include space in square bracket[].
973
974
975Code
976
977result=re.findall(r'\b[^aeiouAEIOU ]\w+','I love Python')
978print result
979
980Output:
981['love', 'Python']
982
983
984
985
986
987
988*************************************************************************************************
989* Problem 6: Validate a phone number (phone number must be of 10 digits and starts with 8 or 9) *
990*************************************************************************************************
991
992
993We have a list phone numbers in list "li" and here we will validate phone numbers using regular
994
995
996
997
998Solution
999-------------------------------------------------------------------------------------------------------------------------------------
1000
1001
1002Code
1003
1004import re
1005li=['9999999999','999999-999','99999x9999']
1006for val in li:
1007 if re.match(r'[8-9]{1}[0-9]{9}',val) and len(val) == 10:
1008 print 'yes'
1009 else:
1010 print 'no'
1011
1012
1013Output:
1014yes
1015no
1016no
1017
1018
1019
1020
1021
1022******************************************************
1023* Problem 7: Split a string with multiple delimiters *
1024******************************************************
1025
1026
1027
1028Solution
1029---------------------------------------------------------------------------------------------------------------------------
1030
1031
1032Code
1033
1034import re
1035line = 'asdf fjdk;afed,fjek,asdf,foo' # String has multiple delimiters (";",","," ").
1036result= re.split(r'[;,\s]', line)
1037print result
1038
1039Output:
1040['asdf', 'fjdk', 'afed', 'fjek', 'asdf', 'foo']
1041
1042
1043
1044We can also use method re.sub() to replace these multiple delimiters with one as space " ".
1045
1046
1047Code
1048
1049import re
1050line = 'asdf fjdk;afed,fjek,asdf,foo'
1051result= re.sub(r'[;,\s]',' ', line)
1052print result
1053
1054Output:
1055asdf fjdk afed fjek asdf foo
1056
1057
1058
1059
1060**************************************************
1061* Problem 8: Retrieve Information from HTML file *
1062**************************************************
1063
1064
1065
1066I want to extract information from a HTML file (see below sample data). Here we need to extract information available between <td> and </td> except the first numerical index. I have assumed here that below html code is stored in a string str.
1067
1068
1069
1070Sample HTML file (str)
1071
1072<tr align="center"><td>1</td> <td>Noah</td> <td>Emma</td></tr>
1073<tr align="center"><td>2</td> <td>Liam</td> <td>Olivia</td></tr>
1074<tr align="center"><td>3</td> <td>Mason</td> <td>Sophia</td></tr>
1075<tr align="center"><td>4</td> <td>Jacob</td> <td>Isabella</td></tr>
1076<tr align="center"><td>5</td> <td>William</td> <td>Ava</td></tr>
1077<tr align="center"><td>6</td> <td>Ethan</td> <td>Mia</td></tr>
1078<tr align="center"><td>7</td> <td HTML>Michael</td> <td>Emily</td></tr>
1079Solution:
1080
1081
1082
1083Code
1084
1085result=re.findall(r'<td>\w+</td>\s<td>(\w+)</td>\s<td>(\w+)</td>',str)
1086print result
1087
1088Output:
1089[('Noah', 'Emma'), ('Liam', 'Olivia'), ('Mason', 'Sophia'), ('Jacob', 'Isabella'), ('William', 'Ava'), ('Ethan', 'Mia'), ('Michael', 'Emily')]
1090
1091
1092
1093You can read html file using library urllib2 (see below code).
1094
1095
1096Code
1097
1098import urllib2
1099response = urllib2.urlopen('')
1100html = response.read()