· 9 years ago · Feb 09, 2017, 07:10 AM
1Jupyter Notebook Logout Control Panelhw03 Last Checkpoint: 38 minutes ago (autosaved)
2Python 3
3File
4Edit
5View
6Insert
7Cell
8Kernel
9Widgets
10Help
11CellToolbarMem: 735 / 2048 (MB)Verify Drive
12Homework 3: Tables and Charts
13Reading: Textbook chapters 5 and 6.
14Please complete this notebook by filling in the cells provided. Before you begin, execute the following cell to load the provided tests. Each time you start your server, you will need to execute this cell again to load the tests.
15In [93]:
16
17# Don't change this cell; just run it.
18​
19import numpy as np
20from datascience import *
21​
22%matplotlib inline
23import matplotlib.pyplot as plots
24plots.style.use('fivethirtyeight')
25​
26from client.api.notebook import Notebook
27ok = Notebook('hw03.ok')
28_ = ok.auth(inline=True)
29=====================================================================
30Assignment: Homework 3: Tables and Charts
31OK, version v1.9.5
32=====================================================================
33
34Successfully logged in as dkathuria@berkeley.edu
35Important: The ok tests don't usually tell you that your answer is correct. More often, they help catch careless mistakes. It's up to you to ensure that your answer is correct. If you're not sure, ask someone (not for the answer, but for some guidance about your approach).
36Once you're finished, select "Save and Checkpoint" in the File menu and then execute the submit cell below. The result will contain a link that you can use to check that your assignment has been submitted successfully. If you submit more than once before the deadline, we will only grade your final submission.
37In [94]:
38
39_ = ok.submit()
40Saving notebook... Saved 'hw03.ipynb'.
41Submit... 100% complete
42Submission successful for user: dkathuria@berkeley.edu
43URL: https://okpy.org/cal/data8/sp17/hw03/submissions/rkZvEK
44
451. Unemployment
46The Federal Reserve Bank of St. Louis publishes data about jobs in the US. Below we've loaded data on unemployment in the United States. There are many ways of defining unemployment, and our dataset includes two notions of the unemployment rate:
47Among people who are able to work and are looking for a full-time job, the percentage who can't find a job. This is called the Non-Employment Index, or NEI.
48Among people who are able to work and are looking for a full-time job, the percentage who can't find any job or are only working at a part-time job. The latter group is called "Part-Time for Economic Reasons", so the acronym for this index is NEI-PTER. (Economists are great at marketing.)
49The source of the data is here.
50Question 1. The data are in a CSV file called unemployment.csv. Load that file into a table called unemployment.
51In [95]:
52
53unemployment = Table.read_table("unemployment.csv")
54unemployment
55Out[95]:
56Date NEI NEI-PTER
571994-01-01 10.0974 11.172
581994-04-01 9.6239 10.7883
591994-07-01 9.3276 10.4831
601994-10-01 9.1071 10.2361
611995-01-01 8.9693 10.1832
621995-04-01 9.0314 10.1071
631995-07-01 8.9802 10.1084
641995-10-01 8.9932 10.1046
651996-01-01 9.0002 10.0531
661996-04-01 8.9038 9.9782
67... (80 rows omitted)
68In [5]:
69
70_ = ok.grade('q1_1')
71~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
72Running tests
73
74---------------------------------------------------------------------
75Test summary
76 Passed: 1
77 Failed: 0
78[ooooooooook] 100.0% passed
79
80Question 2. Sort the data in decreasing order by NEI, naming the sorted table by_nei. Create another table called by_nei_pter that's sorted in decreasing order by NEI-PTER instead.
81In [84]:
82
83by_nei = unemployment.sort("NEI", descending = True)
84by_nei_pter = unemployment.sort("NEI-PTER", descending = True)
85by_nei = unemployment.sort("NEI", descending = True)
86by_nei_pter = unemployment.sort("NEI-PTER", descending = True)
87In [85]:
88
89_ = ok.grade('q1_2')
90~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
91Running tests
92
93---------------------------------------------------------------------
94Test summary
95 Passed: 1
96 Failed: 0
97[ooooooooook] 100.0% passed
98
99Question 3. Use take to make a table containing the data for the 10 quarters when NEI was greatest. Call that table greatest_nei.
100In [18]:
101
102greatest_nei = by_nei.take(np.arange(10))
103greatest_nei
104Out[18]:
105Date NEI NEI-PTER
1062009-10-01 10.9698 12.8557
1072010-01-01 10.9054 12.7311
1082009-07-01 10.8089 12.7404
1092009-04-01 10.7082 12.5497
1102010-04-01 10.6597 12.5664
1112010-10-01 10.5856 12.4329
1122010-07-01 10.5521 12.3897
1132011-01-01 10.5024 12.3017
1142011-07-01 10.4856 12.2507
1152011-04-01 10.4409 12.247
116In [19]:
117
118_ = ok.grade('q1_3')
119~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
120Running tests
121
122---------------------------------------------------------------------
123Test summary
124 Passed: 1
125 Failed: 0
126[ooooooooook] 100.0% passed
127
128Question 4. It's believed that many people became PTER (recall: "Part-Time for Economic Reasons") in the "Great Recession" of 2008-2009. NEI-PTER is the percentage of people who are unemployed (and counted in the NEI) plus the percentage of people who are PTER. Compute an array containing the percentage of people who were PTER in each quarter. (The first element of the array should correspond to the first row of unemployment, and so on.)
129Note: Use the original unemployment table for this.
130In [31]:
131
132pter = unemployment.column("NEI-PTER") - unemployment.column("NEI")
133pter
134Out[31]:
135array([ 1.0746, 1.1644, 1.1555, 1.129 , 1.2139, 1.0757, 1.1282,
136 1.1114, 1.0529, 1.0744, 1.1004, 1.0747, 1.0705, 1.0455,
137 1.008 , 0.9734, 0.9753, 0.8931, 0.9451, 0.8367, 0.8208,
138 0.8105, 0.8248, 0.7578, 0.7251, 0.7445, 0.7543, 0.7423,
139 0.7399, 0.7687, 0.8418, 0.9923, 0.9181, 0.9629, 0.9703,
140 0.9575, 1.0333, 1.0781, 1.0675, 1.0354, 1.0601, 1.01 ,
141 1.0042, 1.0368, 0.9704, 0.923 , 0.9759, 0.93 , 0.889 ,
142 0.821 , 0.9409, 0.955 , 0.898 , 0.8948, 0.9523, 0.9579,
143 1.0149, 1.0762, 1.2873, 1.4335, 1.7446, 1.8415, 1.9315,
144 1.8859, 1.8257, 1.9067, 1.8376, 1.8473, 1.7993, 1.8061,
145 1.7651, 1.7927, 1.7286, 1.6387, 1.6808, 1.6805, 1.6629,
146 1.6253, 1.6477, 1.6298, 1.4796, 1.5131, 1.4866, 1.4345,
147 1.3675, 1.3097, 1.2319, 1.1735, 1.1844, 1.1746])
148In [32]:
149
150_ = ok.grade('q1_4')
151~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
152Running tests
153
154---------------------------------------------------------------------
155Test summary
156 Passed: 1
157 Failed: 0
158[ooooooooook] 100.0% passed
159
160Question 5. Add pter as a column to unemployment (named "PTER") and sort the resulting table by that column in decreasing order. Call the table by_pter.
161Try to do this with a single line of code, if you can.
162In [33]:
163
164by_pter = unemployment.with_column("PTER", pter).sort("PTER", descending = True)
165by_pter
166Out[33]:
167Date NEI NEI-PTER PTER
1682009-07-01 10.8089 12.7404 1.9315
1692010-04-01 10.6597 12.5664 1.9067
1702009-10-01 10.9698 12.8557 1.8859
1712010-10-01 10.5856 12.4329 1.8473
1722009-04-01 10.7082 12.5497 1.8415
1732010-07-01 10.5521 12.3897 1.8376
1742010-01-01 10.9054 12.7311 1.8257
1752011-04-01 10.4409 12.247 1.8061
1762011-01-01 10.5024 12.3017 1.7993
1772011-10-01 10.3287 12.1214 1.7927
178... (80 rows omitted)
179In [34]:
180
181_ = ok.grade('q1_5')
182~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
183Running tests
184
185---------------------------------------------------------------------
186Test summary
187 Passed: 1
188 Failed: 0
189[ooooooooook] 100.0% passed
190
191Question 6. Does it seem true that the PTER rate was very high during the Great Recession, compared to other periods in the dataset? Justify your answer by referring to specific values in the table or by generating a chart.
192In [102]:
193
194​
195Out[102]:
196numpy.str_
197
198*Write your answer here, replacing this text.*
1992. Birth Rates
200The following table gives census-based population estimates for each state on July 1, 2015 and July 1, 2016. The last four columns describe the components of the estimated change in population during this time interval. For all questions below, assume that the word "states" refers to all 52 rows including Puerto Rico & the District of Columbia.
201In [96]:
202
203# Don't change this cell; just run it.
204# From http://www2.census.gov/programs-surveys/popest/datasets/2010-2016/national/totals/nst-est2016-alldata.csv
205# See http://www2.census.gov/programs-surveys/popest/datasets/2010-2015/national/totals/nst-est2015-alldata.pdf
206# for column descriptions. (As of Feb 2017, no descriptions were posted for 2010-2016.)
207pop = Table.read_table('nst-est2016-alldata.csv').where('SUMLEV', 40).select([1, 4, 12, 13, 27, 34, 62, 69])
208pop = pop.relabeled(2, '2015').relabeled(3, '2016')
209pop = pop.relabeled(4, 'BIRTHS').relabeled(5, 'DEATHS')
210pop = pop.relabeled(6, 'MIGRATION').relabeled(7, 'OTHER')
211pop.set_format([2, 3, 4, 5, 6, 7], NumberFormatter(decimals=0)).show(5)
212REGION NAME 2015 2016 BIRTHS DEATHS MIGRATION OTHER
2133 Alabama 4,853,875 4,863,300 58,556 52,405 3,874 -600
2144 Alaska 737,709 741,894 11,255 4,511 -2,557 -2
2154 Arizona 6,817,565 6,931,071 87,204 56,564 76,405 6,461
2163 Arkansas 2,977,853 2,988,248 37,936 30,581 3,530 -490
2174 California 38,993,940 39,250,017 502,848 273,850 33,530 -6,451
218... (47 rows omitted)
219Question 1. Assign us_birth_rate to the total US annual birth rate during this time interval. The annual birth rate for a year-long period is the number of births in that period as a proportion of the population at the start of the period.
220In [41]:
221
222us_birth_rate = sum(pop.column("BIRTHS") / pop.column("2015"))
223us_birth_rate
224Out[41]:
2250.64106727268407149
226In [42]:
227
228_ = ok.grade('q2_1')
229~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
230Running tests
231
232---------------------------------------------------------------------
233Test summary
234 Passed: 1
235 Failed: 0
236[ooooooooook] 100.0% passed
237
238Question 2. Assign fastest_growth to an array of the names of the five states with the fastest population growth rates in descending order of growth rate.
239In [51]:
240
241fastest_growth = pop.with_column("test", (pop.column("2016") - pop.column("2015"))/pop.column("2015")).sort("test", descending = True).column("NAME") [0:5]
242fastest_growth
243Out[51]:
244array(['Utah', 'Nevada', 'Idaho', 'Florida', 'Washington'],
245 dtype='<U20')
246In [52]:
247
248_ = ok.grade('q2_2')
249~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
250Running tests
251
252---------------------------------------------------------------------
253Test summary
254 Passed: 1
255 Failed: 0
256[ooooooooook] 100.0% passed
257
258Question 3. Assign movers to the number of states for which the absolute annual rate of migration was higher than 1%. The annual rate of migration for a year-long period is the net number of migrations (in and out) as a proportion of the population at the start of the period. The MIGRATION column contains estimated annual net migration counts by state.
259In [66]:
260
261movers = pop.with_column("test", pop.column("MIGRATION")/ pop.column("2015")).where("test", lambda x: abs(x)>.01).num_rows
262movers
263Out[66]:
2649
265In [67]:
266
267_ = ok.grade('q2_3')
268~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
269Running tests
270
271---------------------------------------------------------------------
272Test summary
273 Passed: 1
274 Failed: 0
275[ooooooooook] 100.0% passed
276
277Question 4. Assign west_births to the total number of births that occurred in region 4 (the Western US).
278In [78]:
279
280west_births = sum(pop.where("REGION", lambda x: x == "4").column("BIRTHS"))
281west_births
282Out[78]:
283979657
284In [79]:
285
286_ = ok.grade('q2_4')
287~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
288Running tests
289
290---------------------------------------------------------------------
291Test summary
292 Passed: 1
293 Failed: 0
294[ooooooooook] 100.0% passed
295
296Question 5. Assign less_than_west_births to the number of states that had a total population in 2016 that was smaller than the number of babies born in region 4 (the Western US) during this time interval.
297In [81]:
298
299less_than_west_births = pop.where("2016", lambda x: x < west_births).num_rows
300less_than_west_births
301Out[81]:
3027
303In [82]:
304
305_ = ok.grade('q2_5')
306~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
307Running tests
308
309---------------------------------------------------------------------
310Test summary
311 Passed: 1
312 Failed: 0
313[ooooooooook] 100.0% passed
314
315Question 6. Was there an association between birth rate and death rate during this time interval? Use the code cell below to support your conclusion with a chart. If an association exists, what might explain it?
316There is some association between the birth rate and death rate in a slight inverted correlation. As the scatter plot shows, there is a decrease in the death rate as the birth rate increases. This could possibly be attributed to a younger population overall in the state; younger people imply more babies being born with less overall deaths.
317In [99]:
318
319Death
320# Generate a chart here to support your conclusion
321pop.with_columns("Birth Rate", pop.column("BIRTHS") / pop.column("2015"), "Death Rate", pop.column("DEATHS") / pop.column("2015")).scatter("Birth Rate", "Death Rate")
322
3233. Consumer Financial Protection Bureau Complaints
324The Consumer Financial Protection Bureau has collected and published consumer complaints against financial companies since 2011. The data are available here (or at this direct link. For this exercise, to make your code run faster, we've selected only the data from May 2016.
325Run the next cell to load the data. Each row represents one consumer's complaint.
326In [83]:
327
328# Just run this cell.
329complaints = Table.read_table("complaints.csv")
330complaints
331Out[83]:
332company company_public_response company_response complaint_id complaint_what_happened consumer_consent_provided consumer_disputed date_received date_sent_to_company issue product state sub_issue sub_product submitted_via tags timely zip_code
333TransUnion Intermediate Holdings, Inc. Company has responded to the consumer and the CFPB and c ... Closed with explanation 1920073 (None) (None) Yes 2016-05-11T15:39:07.000 2016-05-11T15:39:07.000 Credit reporting company's investigation Credit reporting VT Inadequate help over the phone (None) Phone (None) Yes 05035
334TransUnion Intermediate Holdings, Inc. Company has responded to the consumer and the CFPB and c ... Closed with explanation 1914777 (None) Consent not provided No 2016-05-08T00:53:47.000 2016-05-12T18:40:34.000 Incorrect information on credit report Credit reporting MO Information is not mine (None) Web (None) Yes 63020
335Bank of America Company has responded to the consumer and the CFPB and c ... Closed with explanation 1907306 I became aware of several charges on a Bank of America c ... Consent provided No 2016-05-03T16:49:33.000 2016-05-03T16:49:34.000 Other Credit card VA (None) (None) Web (None) Yes 239XX
336Finance of America Reverse LLC Company believes it acted appropriately as authorized by ... Closed with explanation 1919055 I applied for a reverse mortgage and everthing was going ... Consent provided No 2016-05-10T20:13:22.000 2016-05-10T20:13:23.000 Application, originator, mortgage broker Mortgage TX (None) Reverse mortgage Web Older American Yes 774XX
337Acceptance Solutions Group, INC Company believes it acted appropriately as authorized by ... Closed with explanation 1908628 Keeps calling numbers that are not mine. And talking to ... Consent provided No 2016-05-03T21:05:42.000 2016-05-06T13:42:45.000 Improper contact or sharing of info Debt collection OH Talked to a third party about my debt Payday loan Web (None) Yes 430XX
338Equifax (None) Closed with explanation 1909176 (None) (None) No 2016-05-04T20:08:06.000 2016-05-09T15:11:00.000 Incorrect information on credit report Credit reporting NC Information is not mine (None) Postal mail (None) Yes 28052
339TransUnion Intermediate Holdings, Inc. Company has responded to the consumer and the CFPB and c ... Closed with explanation 1914477 When I enter my personal information to receive my credi ... Consent provided No 2016-05-06T23:09:50.000 2016-05-08T22:40:19.000 Unable to get credit report/credit score Credit reporting OH Problem getting my free annual report (None) Web (None) Yes 450XX
340Encore Capital Group (None) Closed with non-monetary relief 1919937 (None) Consent not provided (None) 2016-05-11T18:58:25.000 2016-05-11T21:53:54.000 Cont'd attempts collect debt not owed Debt collection CT Debt is not mine Credit card Web Older American Yes 06801
341Nationstar Mortgage (None) Closed with explanation 1920517 I am livid with Nation Star for refusing to work with me ... Consent provided (None) 2016-05-11T20:38:09.000 2016-05-11T20:38:09.000 Application, originator, mortgage broker Mortgage IL (None) Conventional adjustable mortgage (ARM) Web (None) Yes 606XX
342Convergent Resources, Inc. (None) Closed with explanation 1920464 (None) Consent not provided No 2016-05-11T12:16:31.000 2016-05-11T12:16:32.000 Cont'd attempts collect debt not owed Debt collection TX Debt is not mine Other (i.e. phone, health club, etc.) Web (None) Yes 78109
343... (15021 rows omitted)
344Question 1. Financial companies offer a variety of products. How many complaints were made against each kind of product? Make a table called complaints_per_product with one row per product category and 2 columns: "product" (the name of the product) and "number of complaints" (the number of complaints made against that kind of product).
345In [88]:
346
347complaints_per_product = complaints.group("product").relabel("count", "number of complaints")
348complaints_per_product
349Out[88]:
350product number of complaints
351Bank account or service 1687
352Consumer Loan 775
353Credit card 1566
354Credit reporting 3820
355Debt collection 3022
356Money transfers 142
357Mortgage 3468
358Other financial service 16
359Payday loan 119
360Prepaid card 110
361... (1 rows omitted)
362In [89]:
363
364_ = ok.grade('q3_1')
365~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
366Running tests
367
368---------------------------------------------------------------------
369Test summary
370 Passed: 1
371 Failed: 0
372[ooooooooook] 100.0% passed
373
374Question 2. Make a bar chart showing how many complaints were made about each product category. Sort the bars from shortest to longest.
375In [ ]:
376
377...
378Question 3. Make a table of the number of complaints made against each company. Call it complaints_per_company. It should have one row per company and 2 columns: "company" (the name of the company) and "number of complaints" (the number of complaints made against that company).
379In [91]:
380
381number of complaints
382complaints_per_company = complaints.group("company").relabel("count", "number of complaints")
383complaints_per_company
384Out[91]:
385company number of complaints
3861st Preference Mortgage 2
38721st Mortgage Corporation 7
3882288984 Ontario Inc. 3
389360 Mortgage 1
3903rd Generation, Inc. 1
3914M Collections, LLC 1
392A.R.M. Solutions, Inc. 2
393AC Autopay, LLC 1
394ACE Cash Express Inc. 21
395ACS Education Services 8
396... (1131 rows omitted)
397In [92]:
398
399_ = ok.grade('q3_3')
400~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
401Running tests
402
403---------------------------------------------------------------------
404Test summary
405 Passed: 1
406 Failed: 0
407[ooooooooook] 100.0% passed
408
409Question 4. It wouldn't be a good idea to make a bar chart of that data. (Don't try it!) Why not?
410Write your answer here, replacing this text.
411Question 5. Make a bar chart of just the 10 companies with the most complaints.
412In [ ]:
413
414...
415Question 6. Make a bar chart like the one above, with one difference: The size of each company's bar should be the proportion (among all complaints made against any company in complaints) that were made against that company.
416Note: Graphs aren't very useful without accurate labels. Make sure that the text on the horizontal axis of the graph makes sense.
4174. Marginal Histograms
418Consider the following scatter plot:
419The axes of the plot represent values of two variables: xx and yy.
420Suppose we have a table called t that has two columns in it:
421x: a column containing the x-values of the points in the scatter plot
422y: a column containing the y-values of the points in the scatter plot
423Question 1: Match each of the following lines of code to the histograms they produce. Explain your reasoning.
424Line 1: t.hist('x')
425Line 2: t.hist('y')
426Histogram A: Histogram B:
427Histogram for Line 1:
428Explanation:
429Histogram for Line 2:
430Explanation: