· 8 years ago · Apr 18, 2018, 12:04 PM
1{"paragraphs":[{"text":"%md\n# Assignment 2\n\nThe goal of this assignment is to put into action the data manipulation techniques from the previous lab ([week 5](https://dslab2018.github.io/labs/week5/)).\n\n## Cluster Usage\n\nAs there will be approximately 45 groups working with the cluster, we encourage you to prototype your queries on small data samples before running them on whole datasets. This will be particularly relevant for part II, where the source dataset size is close to 180GB.\n\n## Documentation\n\nHive queries: <https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Select>\nHive functions: <https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF>\n\n## Displaying results in Zeppelin\n\nAs part of this homework, many questions ask to retrieve and display data. You should do so by using Zeppelin functionality to display tables and plots. It is part of the exercises to find out which representations are the most releveant.\n\n## Part I - Data from SBB/CFF/FFS (30 points)\n\nData source: <https://opentransportdata.swiss/en/dataset/istdaten>\n\nIn this part, you will leverage Hive to perform exploratory analysis of data published by the Open Data Platform Swiss Public Transport. (<https://opentransportdata.swiss>).\n\n### Dataset Description\n\nFormat: the dataset is presented a collection of textfiles with fields separated by ';' (semi-colon). There is one file per day, ranging from 13.09.2017 until 30.11.2017. The textfiles have been compressed using gzip.\n\nLocation: you can find the data on HDFS at the path `/datasets/sbb`.\n\nUnfortunately, the full description from opentransportdata.swiss is only provided in German (<https://opentransportdata.swiss/de/cookbook/ist-daten/>). You can use an automated translator to get more information, but here are the relevant column descriptions:\n\n- `BETRIEBSTAG`: date of the trip\n- `FAHRT_BEZEICHNER`: identifies the trip\n- `BETREIBER_ABK`, `BETREIBER_NAME`: operator (name will contain the full name, e.g. Schweizerische Bundesbahnen for SBB)\n- `PRODUCT_ID`: type of transport, e.g. train, bus\n- `LINIEN_ID`: for trains, this is the train number\n- `LINIEN_TEXT`,`VERKEHRSMITTEL_TEXT`: for trains, the service type (IC, IR, RE, etc.)\n- `ZUSATZFAHRT_TF`: boolean, true if this is an additional trip (not part of the regular schedule)\n- `FAELLT_AUS_TF`: boolean, true if this trip failed (cancelled or not completed)\n- `HALTESTELLEN_NAME`: name of the stop\n- `ANKUNFTSZEIT`: arrival time at the stop according to schedule\n- `AN_PROGNOSE`: actual arrival time (when `AN_PROGNOSE_STATUS` is `GESCHAETZT`)\n- `AN_PROGNOSE_STATUS`: look only at lines when this is `GESCHAETZT`. This indicates that `AN_PROGNOSE` is the measured time of arrival.\n- `ABFAHRTSZEIT`: departure time at the stop according to schedule\n- `AB_PROGNOSE`: actual departure time (when `AN_PROGNOSE_STATUS` is `GESCHAETZT`)\n- `AB_PROGNOSE_STATUS`: look only at lines when this is `GESCHAETZT`. This indicates that `AB_PROGNOSE` is the measured time of arrival.\n- `DURCHFAHRT_TF`: boolean, true if the transport does not stop there\n\nEach line of the file represents a stop and contains arrival and departure times. When the stop is the start or end of a journey, the corresponding columns will be empty (`ANKUNFTSZEIT`/`ABFAHRTSZEIT`).\nIn some cases, the actual times were not measured so the `AN_PROGNOSE_STATUS`/`AB_PROGNOSE_STATUS` will be empty or set to `PROGNOSE` and `AN_PROGNOSE`/`AB_PROGNOSE` will be empty.\n\nIn the repository you will also find the file `sbb_example.png` that shows how you can relate this dataset to the schedule you get from SBB's website.","user":"baetu","dateUpdated":"2018-04-13T19:48:26+0200","config":{"tableHide":false,"editorSetting":{"editOnDblClick":true,"language":"markdown"},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h1>Assignment 2</h1>\n<p>The goal of this assignment is to put into action the data manipulation techniques from the previous lab (<a href=\"https://dslab2018.github.io/labs/week5/\">week 5</a>).</p>\n<h2>Cluster Usage</h2>\n<p>As there will be approximately 45 groups working with the cluster, we encourage you to prototype your queries on small data samples before running them on whole datasets. This will be particularly relevant for part II, where the source dataset size is close to 180GB.</p>\n<h2>Documentation</h2>\n<p>Hive queries: <a href=\"https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Select\">https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Select</a>\n<br />Hive functions: <a href=\"https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF\">https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF</a></p>\n<h2>Displaying results in Zeppelin</h2>\n<p>As part of this homework, many questions ask to retrieve and display data. You should do so by using Zeppelin functionality to display tables and plots. It is part of the exercises to find out which representations are the most releveant.</p>\n<h2>Part I - Data from SBB/CFF/FFS (30 points)</h2>\n<p>Data source: <a href=\"https://opentransportdata.swiss/en/dataset/istdaten\">https://opentransportdata.swiss/en/dataset/istdaten</a></p>\n<p>In this part, you will leverage Hive to perform exploratory analysis of data published by the Open Data Platform Swiss Public Transport. (<a href=\"https://opentransportdata.swiss\">https://opentransportdata.swiss</a>).</p>\n<h3>Dataset Description</h3>\n<p>Format: the dataset is presented a collection of textfiles with fields separated by ';' (semi-colon). There is one file per day, ranging from 13.09.2017 until 30.11.2017. The textfiles have been compressed using gzip.</p>\n<p>Location: you can find the data on HDFS at the path <code>/datasets/sbb</code>.</p>\n<p>Unfortunately, the full description from opentransportdata.swiss is only provided in German (<a href=\"https://opentransportdata.swiss/de/cookbook/ist-daten/\">https://opentransportdata.swiss/de/cookbook/ist-daten/</a>). You can use an automated translator to get more information, but here are the relevant column descriptions:</p>\n<ul>\n<li><code>BETRIEBSTAG</code>: date of the trip</li>\n<li><code>FAHRT_BEZEICHNER</code>: identifies the trip</li>\n<li><code>BETREIBER_ABK</code>, <code>BETREIBER_NAME</code>: operator (name will contain the full name, e.g. Schweizerische Bundesbahnen for SBB)</li>\n<li><code>PRODUCT_ID</code>: type of transport, e.g. train, bus</li>\n<li><code>LINIEN_ID</code>: for trains, this is the train number</li>\n<li><code>LINIEN_TEXT</code>,<code>VERKEHRSMITTEL_TEXT</code>: for trains, the service type (IC, IR, RE, etc.)</li>\n<li><code>ZUSATZFAHRT_TF</code>: boolean, true if this is an additional trip (not part of the regular schedule)</li>\n<li><code>FAELLT_AUS_TF</code>: boolean, true if this trip failed (cancelled or not completed)</li>\n<li><code>HALTESTELLEN_NAME</code>: name of the stop</li>\n<li><code>ANKUNFTSZEIT</code>: arrival time at the stop according to schedule</li>\n<li><code>AN_PROGNOSE</code>: actual arrival time (when <code>AN_PROGNOSE_STATUS</code> is <code>GESCHAETZT</code>)</li>\n<li><code>AN_PROGNOSE_STATUS</code>: look only at lines when this is <code>GESCHAETZT</code>. This indicates that <code>AN_PROGNOSE</code> is the measured time of arrival.</li>\n<li><code>ABFAHRTSZEIT</code>: departure time at the stop according to schedule</li>\n<li><code>AB_PROGNOSE</code>: actual departure time (when <code>AN_PROGNOSE_STATUS</code> is <code>GESCHAETZT</code>)</li>\n<li><code>AB_PROGNOSE_STATUS</code>: look only at lines when this is <code>GESCHAETZT</code>. This indicates that <code>AB_PROGNOSE</code> is the measured time of arrival.</li>\n<li><code>DURCHFAHRT_TF</code>: boolean, true if the transport does not stop there</li>\n</ul>\n<p>Each line of the file represents a stop and contains arrival and departure times. When the stop is the start or end of a journey, the corresponding columns will be empty (<code>ANKUNFTSZEIT</code>/<code>ABFAHRTSZEIT</code>).\n<br />In some cases, the actual times were not measured so the <code>AN_PROGNOSE_STATUS</code>/<code>AB_PROGNOSE_STATUS</code> will be empty or set to <code>PROGNOSE</code> and <code>AN_PROGNOSE</code>/<code>AB_PROGNOSE</code> will be empty.</p>\n<p>In the repository you will also find the file <code>sbb_example.png</code> that shows how you can relate this dataset to the schedule you get from SBB's website.</p>\n"}]},"apps":[],"jobName":"paragraph_1522235709476_1432228720","id":"20180326-131038_1335607032","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-13T19:48:26+0200","dateFinished":"2018-04-13T19:48:26+0200","status":"FINISHED","progressUpdateIntervalMs":500,"focus":true,"$$hashKey":"object:9307"},{"text":"%md\n### Question I.a. (5/30)\n\nFirst, create an external table named `<your_gaspar_name>.sbb` for this dataset.\nThen create table `<your_gaspar_name>.sbb_17_10_2017` which contains data only for the day of 17.10.2017.\n\n### Answer\n\nA big rule for any data related project is to first look at the data. For this, we used the following bash command in order to read the first 5 lines from the gzip archived file\n\n~~~bash\nhdfs dfs -cat /datasets/sbb/2017/09/2017-09-16istdaten.csv.gz | zcat | head -n 5\n~~~\n\n\nThe command above generated the following output:\n\n BETRIEBSTAG;FAHRT_BEZEICHNER;BETREIBER_ID;BETREIBER_ABK;BETREIBER_NAME;PRODUKT_ID;LINIEN_ID;LINIEN_TEXT;UMLAUF_ID;VERKEHRSMITTEL_TEXT;ZUSATZFAHRT_TF;FAELLT_AUS_TF;BPUIC;HALTESTELLEN_NAME;ANKUNFTSZEIT;AN_PROGNOSE;AN_PROGNOSE_STATUS;ABFAHRTSZEIT;AB_PROGNOSE;AB_PROGNOSE_STATUS;DURCHFAHRT_TF\n 16.09.2017;80:06____:17010:000;80:06____;DB;DB Regio AG;Zug;17010;RE;;RE;false;false;8500090;Basel Bad Bf;;;PROGNOSE;16.09.2017 05:45;16.09.2017 05:46:00;PROGNOSE;false\n 16.09.2017;80:06____:17012:000;80:06____;DB;DB Regio AG;Zug;17012;RE;;RE;false;false;8500090;Basel Bad Bf;;;PROGNOSE;16.09.2017 06:34;16.09.2017 06:35:00;PROGNOSE;false\n 16.09.2017;80:06____:17014:000;80:06____;DB;DB Regio AG;Zug;17014;RE;;RE;false;false;8500090;Basel Bad Bf;;;PROGNOSE;16.09.2017 09:48;16.09.2017 09:49:00;PROGNOSE;false\n 16.09.2017;80:06____:17015:000;80:06____;DB;DB Regio AG;Zug;17015;RE;;RE;false;false;8500090;Basel Bad Bf;16.09.2017 08:06;16.09.2017 08:06:00;PROGNOSE;;;PROGNOSE;false\n \nThis helped us a lot in determining the right type for the columns in our **sbb** table and possible problems that might appear when creating the table as, for example, the **date** type in hive expects the format \"YYYY-ÂMM-ÂDD\" while the dates in our table are of the form \"DD.MM.YYYY\".\n\nIn our approach, we will first load the data into a table **sbb_temp** with appropriate columns but with data unparsed for problematic column types as date or timestamp. Upon this, we will create the requested **sbb** table with column values parsed.\n\n#### Analysis of the problematic column types\n\n- `BETRIEBSTAG` (**TripDate**) : has format \"DD.MM.YYYY\". In the [official hive documentation for date](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-date), the supported format for automatic conversion from string to date is \"YYYY-MM-DD\", hence in order to parse the TripDate to a date, we will use date_format functionality to specify the desired format.\n- `ANKUNFTSZEIT` (**ArrivalTimeScheduled**) : has format \"DD.MM.YYYY HH:MM\". In the [official hive documentation for timestamp](https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-TimestampstimestampTimestamps), the supported conversion from string accepts the following format \"YYYY-MM-DD HH:MM:SS.fffffffff\". We will use the technique as for **TripDate** to parse this to the appropriate type. \n- `AN_PROGNOSE` (**ArrivalTimeActual**) : has format \"DD.MM.YYYY HH:MM:SS\". The same rule as for **ArrivalTimeScheduled** will apply here also. \n- `Any other date/timestamp` falls into one of the categories above.\n\nThe inferred column types are also confirmed by the specification [full description](https://opentransportdata.swiss/de/cookbook/ist-daten/#Struktur_der_Daten).","user":"baetu","dateUpdated":"2018-04-12T09:39:31+0200","config":{"tableHide":false,"editorSetting":{"editOnDblClick":true,"language":"markdown"},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Question I.a. (5/30)</h3>\n<p>First, create an external table named <code><your_gaspar_name>.sbb</code> for this dataset.\n<br />Then create table <code><your_gaspar_name>.sbb_17_10_2017</code> which contains data only for the day of 17.10.2017.</p>\n<h3>Answer</h3>\n<p>A big rule for any data related project is to first look at the data. For this, we used the following bash command in order to read the first 5 lines from the gzip archived file</p>\n<pre><code class=\"bash\">hdfs dfs -cat /datasets/sbb/2017/09/2017-09-16istdaten.csv.gz | zcat | head -n 5\n</code></pre>\n<p>The command above generated the following output:</p>\n<pre><code>BETRIEBSTAG;FAHRT_BEZEICHNER;BETREIBER_ID;BETREIBER_ABK;BETREIBER_NAME;PRODUKT_ID;LINIEN_ID;LINIEN_TEXT;UMLAUF_ID;VERKEHRSMITTEL_TEXT;ZUSATZFAHRT_TF;FAELLT_AUS_TF;BPUIC;HALTESTELLEN_NAME;ANKUNFTSZEIT;AN_PROGNOSE;AN_PROGNOSE_STATUS;ABFAHRTSZEIT;AB_PROGNOSE;AB_PROGNOSE_STATUS;DURCHFAHRT_TF\n16.09.2017;80:06____:17010:000;80:06____;DB;DB Regio AG;Zug;17010;RE;;RE;false;false;8500090;Basel Bad Bf;;;PROGNOSE;16.09.2017 05:45;16.09.2017 05:46:00;PROGNOSE;false\n16.09.2017;80:06____:17012:000;80:06____;DB;DB Regio AG;Zug;17012;RE;;RE;false;false;8500090;Basel Bad Bf;;;PROGNOSE;16.09.2017 06:34;16.09.2017 06:35:00;PROGNOSE;false\n16.09.2017;80:06____:17014:000;80:06____;DB;DB Regio AG;Zug;17014;RE;;RE;false;false;8500090;Basel Bad Bf;;;PROGNOSE;16.09.2017 09:48;16.09.2017 09:49:00;PROGNOSE;false\n16.09.2017;80:06____:17015:000;80:06____;DB;DB Regio AG;Zug;17015;RE;;RE;false;false;8500090;Basel Bad Bf;16.09.2017 08:06;16.09.2017 08:06:00;PROGNOSE;;;PROGNOSE;false\n</code></pre>\n<p>This helped us a lot in determining the right type for the columns in our <strong>sbb</strong> table and possible problems that might appear when creating the table as, for example, the <strong>date</strong> type in hive expects the format “YYYY-ÂMM-ÂDD” while the dates in our table are of the form “DD.MM.YYYY”.</p>\n<p>In our approach, we will first load the data into a table <strong>sbb_temp</strong> with appropriate columns but with data unparsed for problematic column types as date or timestamp. Upon this, we will create the requested <strong>sbb</strong> table with column values parsed.</p>\n<h4>Analysis of the problematic column types</h4>\n<ul>\n<li><code>BETRIEBSTAG</code> (<strong>TripDate</strong>) : has format “DD.MM.YYYY”. In the <a href=\"https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-date\">official hive documentation for date</a>, the supported format for automatic conversion from string to date is “YYYY-MM-DD”, hence in order to parse the TripDate to a date, we will use date_format functionality to specify the desired format.</li>\n<li><code>ANKUNFTSZEIT</code> (<strong>ArrivalTimeScheduled</strong>) : has format “DD.MM.YYYY HH:MM”. In the <a href=\"https://cwiki.apache.org/confluence/display/Hive/LanguageManual+Types#LanguageManualTypes-TimestampstimestampTimestamps\">official hive documentation for timestamp</a>, the supported conversion from string accepts the following format “YYYY-MM-DD HH:MM:SS.fffffffff”. We will use the technique as for <strong>TripDate</strong> to parse this to the appropriate type.</li>\n<li><code>AN_PROGNOSE</code> (<strong>ArrivalTimeActual</strong>) : has format “DD.MM.YYYY HH:MM:SS”. The same rule as for <strong>ArrivalTimeScheduled</strong> will apply here also.</li>\n<li><code>Any other date/timestamp</code> falls into one of the categories above.</li>\n</ul>\n<p>The inferred column types are also confirmed by the specification <a href=\"https://opentransportdata.swiss/de/cookbook/ist-daten/#Struktur_der_Daten\">full description</a>.</p>\n"}]},"apps":[],"jobName":"paragraph_1522235709478_1432998218","id":"20180326-141315_427673736","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-12T09:39:31+0200","dateFinished":"2018-04-12T09:39:31+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9308"},{"text":"%jdbc(hive)\n-- Date : TripDate\n-- Timestamp : ArrivalTimeScheduler, ArrivalTimeActual, DepartureTimeScheduler, DepartureTimeActual\ncreate table if not exists musuroi.sbb_temp(TripDate string, TripId string, OperatorId string, OperatorAbbrv string, OperatorName string, ProductId string, LineId string, LineType string, UmlaufId string, \n TransportType string, AdditionalTrip boolean, FailedTrip boolean, BPUIC string, StopName string, ArrivalTimeScheduled string, ArrivalTimeActual string, ArrivalTimeActualStatus string, \n DepartureTimeScheduled string, DepartureTimeActual string, DepartureTimeActualStatus string, SkipStation boolean)\n row format delimited fields terminated by ';'\n stored as textfile\n location '/datasets/sbb/';","user":"baetu","dateUpdated":"2018-04-13T12:48:07+0200","config":{"tableHide":false,"editorSetting":{"editOnDblClick":false,"language":"sql"},"colWidth":12,"editorMode":"ace/mode/sql","editorHide":false,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"}]},"apps":[],"jobName":"paragraph_1522235709480_1430689724","id":"20180326-131158_1608768124","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-13T12:48:07+0200","dateFinished":"2018-04-13T12:48:07+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9309"},{"text":"%md\nNow that we have created the intermediary table and loaded it with the sbb data we can create the actual sbb table having the date or datetime columns properly parsed. \n\nAdditionally, we omit the row containing the column names by selecting only the rows that do not have the 'BETRIEBSTAG' value for the TripDate.","user":"musuroi","dateUpdated":"2018-03-31T19:11:43+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>Now that we have created the intermediary table and loaded it with the sbb data we can create the actual sbb table having the date or datetime columns properly parsed.</p>\n<p>Additionally, we omit the row containing the column names by selecting only the rows that do not have the 'BETRIEBSTAG' value for the TripDate.</p>\n"}]},"apps":[],"jobName":"paragraph_1522514643558_1343447073","id":"20180331-184403_2140832743","dateCreated":"2018-03-31T18:44:03+0200","dateStarted":"2018-03-31T19:11:43+0200","dateFinished":"2018-03-31T19:11:43+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9310"},{"text":"%jdbc(hive)\n\ncreate table if not exists musuroi.sbb \nstored as parquet\nas\nselect from_unixtime(unix_timestamp(TripDate,'dd.MM.yyyy')) as TripDate,\n TripId, OperatorId, OperatorAbbrv, OperatorName, ProductId, LineId, LineType, UmlaufId, TransportType, AdditionalTrip, FailedTrip, BPUIC, StopName,\n from_unixtime(unix_timestamp(ArrivalTimeScheduled, \"dd.MM.yyyy HH:mm\")) as ArrivalTimeScheduled,\n from_unixtime(unix_timestamp(ArrivalTimeActual, \"dd.MM.yyyy HH:mm:ss\")) as ArrivalTimeActual,\n ArrivalTimeActualStatus,\n from_unixtime(unix_timestamp(DepartureTimeScheduled, \"dd.MM.yyyy HH:mm\")) as DepartureTimeScheduled,\n from_unixtime(unix_timestamp(DepartureTimeActual, \"dd.MM.yyyy HH:mm:ss\")) as DepartureTimeActual,\n DepartureTimeActualStatus, SkipStation\nfrom musuroi.sbb_temp\nwhere TripDate != 'BETRIEBSTAG';","user":"baetu","dateUpdated":"2018-04-13T12:48:40+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"}]},"apps":[],"jobName":"paragraph_1522505836372_-1841463209","id":"20180331-161716_329010072","dateCreated":"2018-03-31T16:17:16+0200","dateStarted":"2018-04-13T12:48:41+0200","dateFinished":"2018-04-13T12:48:41+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9311"},{"text":"%md\nWhat is left to do for this exercise is to create the table **sbb_17_10_2017** only with the data from 17.10.2017.","user":"musuroi","dateUpdated":"2018-03-31T19:15:56+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>What is left to do for this exercise is to create the table <strong>sbb_17_10_2017</strong> only with the data from 17.10.2017.</p>\n"}]},"apps":[],"jobName":"paragraph_1522516503562_1794732636","id":"20180331-191503_2104765707","dateCreated":"2018-03-31T19:15:03+0200","dateStarted":"2018-03-31T19:15:56+0200","dateFinished":"2018-03-31T19:15:56+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9312"},{"text":"%jdbc(hive)\ncreate table if not exists musuroi.sbb_17_10_2017\nstored as parquet\nas \nselect * from musuroi.sbb\nwhere TripDate == '2017-10-17 00:00:00';","user":"baetu","dateUpdated":"2018-04-13T12:48:45+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"}]},"apps":[],"jobName":"paragraph_1522513469470_-1682271351","id":"20180331-182429_98717934","dateCreated":"2018-03-31T18:24:29+0200","dateStarted":"2018-04-13T12:48:45+0200","dateFinished":"2018-04-13T12:48:45+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9313"},{"text":"%jdbc(hive) \n\nselect * from musuroi.sbb limit 2;","user":"baetu","dateUpdated":"2018-04-13T19:56:38+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"sbb.tripdate\tsbb.tripid\tsbb.operatorid\tsbb.operatorabbrv\tsbb.operatorname\tsbb.productid\tsbb.lineid\tsbb.linetype\tsbb.umlaufid\tsbb.transporttype\tsbb.additionaltrip\tsbb.failedtrip\tsbb.bpuic\tsbb.stopname\tsbb.arrivaltimescheduled\tsbb.arrivaltimeactual\tsbb.arrivaltimeactualstatus\tsbb.departuretimescheduled\tsbb.departuretimeactual\tsbb.departuretimeactualstatus\tsbb.skipstation\n2017-09-14 00:00:00\t80:06____:17010:000\t80:06____\tDB\tDB Regio AG\tZug\t17010\tRE\t\tRE\tfalse\tfalse\t8500090\tBasel Bad Bf\tnull\tnull\tPROGNOSE\t2017-09-14 05:45:00\t2017-09-14 05:45:00\tPROGNOSE\tfalse\n2017-09-14 00:00:00\t80:06____:17012:000\t80:06____\tDB\tDB Regio AG\tZug\t17012\tRE\t\tRE\tfalse\tfalse\t8500090\tBasel Bad Bf\tnull\tnull\tPROGNOSE\t2017-09-14 06:34:00\t2017-09-14 06:37:00\tPROGNOSE\tfalse\n"}]},"apps":[],"jobName":"paragraph_1523549946486_-1078764407","id":"20180412-181906_1206804403","dateCreated":"2018-04-12T18:19:06+0200","dateStarted":"2018-04-13T19:56:27+0200","dateFinished":"2018-04-13T19:56:28+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9314"},{"text":"%md\n### Question I.b. (5/30)\n\ni) How many stops can you find for the date 17.10.2017? Display the information by transport type (bus, train, etc.).\nNote: you can keep the German labels like 'Zug' for train.\n\nii) Now do the same for the whole dataset. Show and document any pattern you see.\nHint: to properly order by date, you may have to parse them using the `unix_timestamp` function.\n\n### Answer\n#### i)\nAs it is specified in the Dataset Description paragraph, each line of the data corresponds to a stop. Taking this into consideration, we understand by number of stops, in question **i)**, the number of entries/rows in our database that are not marked as skipped (SkipStation = False or null) and are not failed (FailedTrip = False or null) groupped for each transport type (ProductId).","user":"musuroi","dateUpdated":"2018-04-01T12:14:50+0200","config":{"tableHide":false,"editorSetting":{"editOnDblClick":true,"language":"markdown"},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Question I.b. (5/30)</h3>\n<p>i) How many stops can you find for the date 17.10.2017? Display the information by transport type (bus, train, etc.).\n<br />Note: you can keep the German labels like 'Zug' for train.</p>\n<p>ii) Now do the same for the whole dataset. Show and document any pattern you see.\n<br />Hint: to properly order by date, you may have to parse them using the <code>unix_timestamp</code> function.</p>\n<h3>Answer</h3>\n<h4>i)</h4>\n<p>As it is specified in the Dataset Description paragraph, each line of the data corresponds to a stop. Taking this into consideration, we understand by number of stops, in question <strong>i)</strong>, the number of entries/rows in our database that are not marked as skipped (SkipStation = False or null) and are not failed (FailedTrip = False or null) groupped for each transport type (ProductId).</p>\n"}]},"apps":[],"jobName":"paragraph_1522235709481_1430304975","id":"20180326-142408_1931615887","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-01T12:14:50+0200","dateFinished":"2018-04-01T12:14:50+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9315"},{"text":"%jdbc(hive)\nselect count(*) from musuroi.sbb\nwhere SkipStation = True or FailedTrip = True;","user":"baetu","dateUpdated":"2018-04-13T13:07:08+0200","config":{"colWidth":12,"enabled":true,"results":{"0":{"graph":{"mode":"table","height":84,"optionOpen":false}}},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"_c0\n424072\n"}]},"apps":[],"jobName":"paragraph_1522571851057_-2059025578","id":"20180401-103731_420084757","dateCreated":"2018-04-01T10:37:31+0200","dateStarted":"2018-04-13T13:07:08+0200","dateFinished":"2018-04-13T13:16:28+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9316"},{"text":"%jdbc(hive)\nselect ProductId, count(*) stops\nfrom musuroi.sbb_17_10_2017\nwhere nvl(SkipStation, False) = False and nvl(FailedTrip, False) = False\ngroup by ProductId;","user":"baetu","dateUpdated":"2018-04-13T13:00:57+0200","config":{"colWidth":12,"editorMode":"ace/mode/sql","results":{"0":{"graph":{"mode":"multiBarChart","height":300,"optionOpen":false,"setting":{"multiBarChart":{"stacked":false}},"commonSetting":{},"keys":[{"name":"productid","index":0,"aggr":"sum"}],"groups":[],"values":[{"name":"stops","index":1,"aggr":"sum"}]},"helium":{}}},"enabled":true,"editorSetting":{"editOnDblClick":false,"language":"sql"}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"productid\tstops\nBUS\t81436\nBus\t473940\nSchiff\t403\nTram\t177713\nZug\t152662\n"}]},"apps":[],"jobName":"paragraph_1522235709482_1431459222","id":"20180326-143516_1075569998","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-13T13:00:57+0200","dateFinished":"2018-04-13T13:03:35+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9317"},{"text":"%md \nWe observe the two bars named **BUS** and **Bus** and decide to look further into the description of the dataset to see if there is any difference between them.\n\nIn the [Structure of data](https://opentransportdata.swiss/de/cookbook/ist-daten/#Struktur_der_Daten) part of the full description page, in the PRODUKT_ID row description, there is a link sending us to [VM category](https://opentransportdata.swiss/de/cookbook/ptmode/) where the transport modes are detalied and from where we can conclude there is no difference between **BUS** and **Bus**.\n\nHence, we decide to combine them into one group ignoring the case and redo the visualization.","user":"musuroi","dateUpdated":"2018-03-31T21:30:46+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>We observe the two bars named <strong>BUS</strong> and <strong>Bus</strong> and decide to look further into the description of the dataset to see if there is any difference between them.</p>\n<p>In the <a href=\"https://opentransportdata.swiss/de/cookbook/ist-daten/#Struktur_der_Daten\">Structure of data</a> part of the full description page, in the PRODUKT_ID row description, there is a link sending us to <a href=\"https://opentransportdata.swiss/de/cookbook/ptmode/\">VM category</a> where the transport modes are detalied and from where we can conclude there is no difference between <strong>BUS</strong> and <strong>Bus</strong>.</p>\n<p>Hence, we decide to combine them into one group ignoring the case and redo the visualization.</p>\n"}]},"apps":[],"jobName":"paragraph_1522524265153_1706854927","id":"20180331-212425_1146363366","dateCreated":"2018-03-31T21:24:25+0200","dateStarted":"2018-03-31T21:30:46+0200","dateFinished":"2018-03-31T21:30:46+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9318"},{"text":"%jdbc(hive)\nselect lower(ProductId) ProductId, count(TripDate) stops\nfrom musuroi.sbb_17_10_2017\nwhere nvl(SkipStation, False) = False and nvl(FailedTrip, False) = False\ngroup by lower(ProductId);","user":"musuroi","dateUpdated":"2018-04-01T11:48:17+0200","config":{"colWidth":12,"enabled":true,"results":{"0":{"graph":{"mode":"multiBarChart","height":300,"optionOpen":false,"setting":{"multiBarChart":{"stacked":false}},"commonSetting":{},"keys":[{"name":"productid","index":0,"aggr":"sum"}],"groups":[],"values":[{"name":"stops","index":1,"aggr":"sum"}]},"helium":{}}},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"productid\tstops\nbus\t555376\nschiff\t403\ntram\t177713\nzug\t152662\n"}]},"apps":[],"jobName":"paragraph_1522524507979_-282729252","id":"20180331-212827_232812720","dateCreated":"2018-03-31T21:28:27+0200","dateStarted":"2018-04-01T11:48:17+0200","dateFinished":"2018-04-01T12:02:49+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9319"},{"text":"%md\n#### ii)\nFor the second question we are asked to do the same visualization for the entire dataset. We understand by this that we should present the same statistics for each day in the dataset.\n\nWe do this by following the same procedure as for the table **sbb_17_19_2017**, but we first group by the TripDate and then by the ProductId. We then stack the results for each type of transport per day.","user":"musuroi","dateUpdated":"2018-04-01T21:54:56+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h4>ii)</h4>\n<p>For the second question we are asked to do the same visualization for the entire dataset. We understand by this that we should present the same statistics for each day in the dataset.</p>\n<p>We do this by following the same procedure as for the table <strong>sbb_17_19_2017</strong>, but we first group by the TripDate and then by the ProductId. We then stack the results for each type of transport per day.</p>\n"}]},"apps":[],"jobName":"paragraph_1522524665029_-1321704524","id":"20180331-213105_513388084","dateCreated":"2018-03-31T21:31:05+0200","dateStarted":"2018-04-01T21:54:56+0200","dateFinished":"2018-04-01T21:54:56+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9320"},{"text":"%jdbc(hive)\nselect TripDate, lower(ProductId) ProductId, count(*) stops\nfrom musuroi.sbb\ngroup by TripDate, lower(ProductId)\norder by TripDate;","user":"musuroi","dateUpdated":"2018-04-01T21:40:00+0200","config":{"colWidth":12,"enabled":true,"results":{"0":{"graph":{"mode":"lineChart","height":300,"optionOpen":false,"setting":{"lineChart":{"lineWithFocus":false}},"commonSetting":{},"keys":[{"name":"tripdate","index":0,"aggr":"sum"}],"groups":[{"name":"productid","index":1,"aggr":"sum"}],"values":[{"name":"stops","index":2,"aggr":"sum"}]},"helium":{}}},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"tripdate\tproductid\tstops\n2017-09-13 00:00:00\tzug\t155113\n2017-09-13 00:00:00\ttram\t174970\n2017-09-13 00:00:00\tschiff\t379\n2017-09-13 00:00:00\tbus\t558807\n2017-09-13 00:00:00\t\t5\n2017-09-14 00:00:00\tbus\t559398\n2017-09-14 00:00:00\ttram\t172946\n2017-09-14 00:00:00\tzug\t154714\n2017-09-14 00:00:00\t\t12\n2017-09-14 00:00:00\tschiff\t379\n2017-09-15 00:00:00\tschiff\t379\n2017-09-15 00:00:00\tzug\t155980\n2017-09-15 00:00:00\tbus\t565778\n2017-09-15 00:00:00\ttram\t175642\n2017-09-16 00:00:00\tzug\t148675\n2017-09-16 00:00:00\tschiff\t372\n2017-09-16 00:00:00\ttram\t159668\n2017-09-16 00:00:00\tbus\t455591\n2017-09-17 00:00:00\tzug\t137825\n2017-09-17 00:00:00\tschiff\t420\n2017-09-17 00:00:00\ttram\t121496\n2017-09-17 00:00:00\tbus\t333321\n2017-09-18 00:00:00\ttram\t172864\n2017-09-18 00:00:00\tbus\t560499\n2017-09-18 00:00:00\tschiff\t379\n2017-09-18 00:00:00\tzug\t153897\n2017-09-19 00:00:00\ttram\t172466\n2017-09-19 00:00:00\tschiff\t379\n2017-09-19 00:00:00\tzug\t154497\n2017-09-19 00:00:00\tbus\t558511\n2017-09-20 00:00:00\tschiff\t379\n2017-09-20 00:00:00\tbus\t560143\n2017-09-20 00:00:00\tzug\t154474\n2017-09-20 00:00:00\t\t2\n2017-09-20 00:00:00\ttram\t173937\n2017-09-21 00:00:00\ttram\t172428\n2017-09-21 00:00:00\t\t2\n2017-09-21 00:00:00\tschiff\t379\n2017-09-21 00:00:00\tbus\t561044\n2017-09-21 00:00:00\tzug\t154802\n2017-09-22 00:00:00\ttram\t178310\n2017-09-22 00:00:00\tzug\t155538\n2017-09-22 00:00:00\tbus\t556741\n2017-09-22 00:00:00\tschiff\t379\n2017-09-23 00:00:00\tbus\t439919\n2017-09-23 00:00:00\tschiff\t372\n2017-09-23 00:00:00\ttram\t167482\n2017-09-23 00:00:00\tzug\t145587\n2017-09-24 00:00:00\tzug\t133856\n2017-09-24 00:00:00\tschiff\t420\n2017-09-24 00:00:00\tbus\t318774\n2017-09-24 00:00:00\ttram\t121068\n2017-09-25 00:00:00\tschiff\t403\n2017-09-25 00:00:00\ttram\t170436\n2017-09-25 00:00:00\tbus\t549887\n2017-09-25 00:00:00\tzug\t152292\n2017-09-26 00:00:00\t\t6\n2017-09-26 00:00:00\ttram\t171391\n2017-09-26 00:00:00\tschiff\t403\n2017-09-26 00:00:00\tzug\t152242\n2017-09-26 00:00:00\tbus\t555105\n2017-09-27 00:00:00\ttram\t171994\n2017-09-27 00:00:00\t\t22\n2017-09-27 00:00:00\tbus\t556098\n2017-09-27 00:00:00\tschiff\t403\n2017-09-27 00:00:00\tzug\t152494\n2017-09-28 00:00:00\ttram\t170774\n2017-09-28 00:00:00\t\t7\n2017-09-28 00:00:00\tbus\t556503\n2017-09-28 00:00:00\tzug\t152031\n2017-09-28 00:00:00\tschiff\t403\n2017-09-29 00:00:00\t\t14\n2017-09-29 00:00:00\tbus\t563325\n2017-09-29 00:00:00\ttram\t174574\n2017-09-29 00:00:00\tzug\t153656\n2017-09-29 00:00:00\tschiff\t403\n2017-09-30 00:00:00\t\t11\n2017-09-30 00:00:00\tbus\t456732\n2017-09-30 00:00:00\ttram\t160823\n2017-09-30 00:00:00\tzug\t142908\n2017-09-30 00:00:00\tschiff\t396\n2017-10-01 00:00:00\ttram\t120465\n2017-10-01 00:00:00\tzug\t134072\n2017-10-01 00:00:00\tbus\t333932\n2017-10-01 00:00:00\tschiff\t444\n2017-10-02 00:00:00\tbus\t554460\n2017-10-02 00:00:00\ttram\t176719\n2017-10-02 00:00:00\tschiff\t403\n2017-10-02 00:00:00\t\t2\n2017-10-02 00:00:00\tzug\t152330\n2017-10-03 00:00:00\tbus\t551625\n2017-10-03 00:00:00\t\t5\n2017-10-03 00:00:00\ttram\t176103\n2017-10-03 00:00:00\tzug\t152311\n2017-10-03 00:00:00\tschiff\t403\n2017-10-04 00:00:00\tzug\t152717\n2017-10-04 00:00:00\tschiff\t403\n2017-10-04 00:00:00\t\t7\n2017-10-04 00:00:00\tbus\t555700\n2017-10-04 00:00:00\ttram\t176546\n2017-10-05 00:00:00\tbus\t555298\n2017-10-05 00:00:00\ttram\t176507\n2017-10-05 00:00:00\t\t5\n2017-10-05 00:00:00\tschiff\t403\n2017-10-05 00:00:00\tzug\t152418\n2017-10-06 00:00:00\t\t9\n2017-10-06 00:00:00\ttram\t166603\n2017-10-06 00:00:00\tbus\t526412\n2017-10-06 00:00:00\tschiff\t403\n2017-10-06 00:00:00\tzug\t153311\n2017-10-07 00:00:00\tbus\t455894\n2017-10-07 00:00:00\tzug\t144363\n2017-10-07 00:00:00\tschiff\t396\n2017-10-07 00:00:00\ttram\t164265\n2017-10-08 00:00:00\tzug\t134323\n2017-10-08 00:00:00\tbus\t333533\n2017-10-08 00:00:00\tschiff\t444\n2017-10-08 00:00:00\ttram\t124084\n2017-10-09 00:00:00\tbus\t554092\n2017-10-09 00:00:00\tzug\t152485\n2017-10-09 00:00:00\t\t2\n2017-10-09 00:00:00\tschiff\t403\n2017-10-09 00:00:00\ttram\t178876\n2017-10-10 00:00:00\tschiff\t403\n2017-10-10 00:00:00\tbus\t554721\n2017-10-10 00:00:00\ttram\t176510\n2017-10-10 00:00:00\tzug\t152188\n2017-10-11 00:00:00\tschiff\t403\n2017-10-11 00:00:00\tzug\t152160\n2017-10-11 00:00:00\tbus\t554275\n2017-10-11 00:00:00\ttram\t176582\n2017-10-12 00:00:00\tzug\t152091\n2017-10-12 00:00:00\tschiff\t403\n2017-10-12 00:00:00\tbus\t554286\n2017-10-12 00:00:00\ttram\t176159\n2017-10-13 00:00:00\t\t5\n2017-10-13 00:00:00\ttram\t181506\n2017-10-13 00:00:00\tschiff\t403\n2017-10-13 00:00:00\tzug\t153478\n2017-10-13 00:00:00\tbus\t560497\n2017-10-14 00:00:00\tzug\t146534\n2017-10-14 00:00:00\tschiff\t396\n2017-10-14 00:00:00\ttram\t163834\n2017-10-14 00:00:00\t\t4\n2017-10-14 00:00:00\tbus\t456034\n2017-10-15 00:00:00\ttram\t124251\n2017-10-15 00:00:00\tschiff\t444\n2017-10-15 00:00:00\tzug\t135672\n2017-10-15 00:00:00\tbus\t334001\n2017-10-16 00:00:00\t\t2\n2017-10-16 00:00:00\tzug\t154227\n2017-10-16 00:00:00\tbus\t557501\n2017-10-16 00:00:00\ttram\t181561\n2017-10-16 00:00:00\tschiff\t403\n2017-10-17 00:00:00\ttram\t178512\n2017-10-17 00:00:00\tbus\t558899\n2017-10-17 00:00:00\tschiff\t403\n2017-10-17 00:00:00\tzug\t154306\n2017-10-18 00:00:00\tbus\t556653\n2017-10-18 00:00:00\tzug\t154293\n2017-10-18 00:00:00\t\t20\n2017-10-18 00:00:00\tschiff\t403\n2017-10-18 00:00:00\ttram\t178151\n2017-10-19 00:00:00\tschiff\t403\n2017-10-19 00:00:00\ttram\t178674\n2017-10-19 00:00:00\tbus\t555728\n2017-10-19 00:00:00\t\t1\n2017-10-19 00:00:00\tzug\t154374\n2017-10-20 00:00:00\ttram\t182638\n2017-10-20 00:00:00\t\t8\n2017-10-20 00:00:00\tzug\t155257\n2017-10-20 00:00:00\tschiff\t403\n2017-10-20 00:00:00\tbus\t564096\n2017-10-21 00:00:00\tbus\t455411\n2017-10-21 00:00:00\t\t9\n2017-10-21 00:00:00\tschiff\t396\n2017-10-21 00:00:00\tzug\t144757\n2017-10-21 00:00:00\ttram\t166071\n2017-10-22 00:00:00\tbus\t331154\n2017-10-22 00:00:00\tschiff\t444\n2017-10-22 00:00:00\ttram\t125515\n2017-10-22 00:00:00\tzug\t133647\n2017-10-23 00:00:00\t\t11\n2017-10-23 00:00:00\tbus\t556472\n2017-10-23 00:00:00\ttram\t177372\n2017-10-23 00:00:00\tzug\t154186\n2017-10-23 00:00:00\tschiff\t168\n2017-10-24 00:00:00\tschiff\t168\n2017-10-24 00:00:00\tbus\t557857\n2017-10-24 00:00:00\tzug\t154182\n2017-10-24 00:00:00\t\t39\n2017-10-24 00:00:00\ttram\t178641\n2017-10-25 00:00:00\t\t68\n2017-10-25 00:00:00\tbus\t562440\n2017-10-25 00:00:00\tzug\t154196\n2017-10-25 00:00:00\ttram\t179699\n2017-10-25 00:00:00\tschiff\t168\n2017-10-26 00:00:00\t\t68\n2017-10-26 00:00:00\tbus\t558786\n2017-10-26 00:00:00\ttram\t178096\n2017-10-26 00:00:00\tschiff\t176\n2017-10-26 00:00:00\tzug\t154091\n2017-10-27 00:00:00\tbus\t568089\n2017-10-27 00:00:00\tschiff\t176\n2017-10-27 00:00:00\tzug\t155423\n2017-10-27 00:00:00\t\t8\n2017-10-27 00:00:00\ttram\t182633\n2017-10-28 00:00:00\ttram\t163474\n2017-10-28 00:00:00\tzug\t147156\n2017-10-28 00:00:00\tbus\t458906\n2017-10-28 00:00:00\t\t16\n2017-10-28 00:00:00\tschiff\t164\n2017-10-29 00:00:00\t\t10\n2017-10-29 00:00:00\tzug\t136662\n2017-10-29 00:00:00\tschiff\t172\n2017-10-29 00:00:00\tbus\t330684\n2017-10-29 00:00:00\ttram\t124994\n2017-10-30 00:00:00\t\t8\n2017-10-30 00:00:00\ttram\t178083\n2017-10-30 00:00:00\tbus\t559521\n2017-10-30 00:00:00\tzug\t150284\n2017-10-30 00:00:00\tschiff\t168\n2017-10-31 00:00:00\tbus\t562389\n2017-10-31 00:00:00\tzug\t149757\n2017-10-31 00:00:00\tschiff\t168\n2017-10-31 00:00:00\ttram\t180914\n2017-11-01 00:00:00\tschiff\t168\n2017-11-01 00:00:00\ttram\t178642\n2017-11-01 00:00:00\tzug\t149438\n2017-11-01 00:00:00\tbus\t498112\n2017-11-02 00:00:00\tbus\t555630\n2017-11-02 00:00:00\ttram\t177145\n2017-11-02 00:00:00\tschiff\t176\n2017-11-02 00:00:00\tzug\t150137\n2017-11-03 00:00:00\tzug\t151248\n2017-11-03 00:00:00\tbus\t566304\n2017-11-03 00:00:00\ttram\t182124\n2017-11-03 00:00:00\tschiff\t176\n2017-11-04 00:00:00\ttram\t165819\n2017-11-04 00:00:00\tzug\t141513\n2017-11-04 00:00:00\tbus\t455144\n2017-11-04 00:00:00\tschiff\t164\n2017-11-05 00:00:00\ttram\t125449\n2017-11-05 00:00:00\tbus\t333945\n2017-11-05 00:00:00\tschiff\t172\n2017-11-05 00:00:00\tzug\t131347\n2017-11-06 00:00:00\tbus\t560080\n2017-11-06 00:00:00\tzug\t149901\n2017-11-06 00:00:00\tschiff\t168\n2017-11-06 00:00:00\ttram\t178234\n2017-11-07 00:00:00\tbus\t561131\n2017-11-07 00:00:00\ttram\t179685\n2017-11-07 00:00:00\tzug\t150086\n2017-11-07 00:00:00\tschiff\t168\n2017-11-08 00:00:00\ttram\t178629\n2017-11-08 00:00:00\tbus\t563440\n2017-11-08 00:00:00\tschiff\t168\n2017-11-08 00:00:00\tzug\t150265\n2017-11-09 00:00:00\tschiff\t176\n2017-11-09 00:00:00\tzug\t150259\n2017-11-09 00:00:00\ttram\t181002\n2017-11-09 00:00:00\tbus\t553355\n2017-11-10 00:00:00\tschiff\t176\n2017-11-10 00:00:00\tzug\t151003\n2017-11-10 00:00:00\tbus\t568766\n2017-11-10 00:00:00\ttram\t184548\n2017-11-11 00:00:00\tzug\t142649\n2017-11-11 00:00:00\tschiff\t164\n2017-11-11 00:00:00\tbus\t457463\n2017-11-11 00:00:00\ttram\t167566\n2017-11-12 00:00:00\tbus\t335286\n2017-11-12 00:00:00\ttram\t129359\n2017-11-12 00:00:00\tzug\t132547\n2017-11-12 00:00:00\tschiff\t178\n2017-11-13 00:00:00\ttram\t178298\n2017-11-13 00:00:00\tbus\t561989\n2017-11-13 00:00:00\tzug\t150089\n2017-11-13 00:00:00\tschiff\t174\n2017-11-14 00:00:00\tschiff\t174\n2017-11-14 00:00:00\tzug\t150287\n2017-11-14 00:00:00\tbus\t557323\n2017-11-14 00:00:00\ttram\t175867\n2017-11-15 00:00:00\tbus\t562315\n2017-11-15 00:00:00\tzug\t150066\n2017-11-15 00:00:00\tschiff\t174\n2017-11-15 00:00:00\ttram\t178304\n2017-11-16 00:00:00\tbus\t563134\n2017-11-16 00:00:00\tschiff\t182\n2017-11-16 00:00:00\tzug\t150037\n2017-11-16 00:00:00\ttram\t178042\n2017-11-17 00:00:00\tschiff\t176\n2017-11-17 00:00:00\ttram\t183701\n2017-11-17 00:00:00\tzug\t150945\n2017-11-17 00:00:00\tbus\t567471\n2017-11-18 00:00:00\tbus\t458613\n2017-11-18 00:00:00\tschiff\t164\n2017-11-18 00:00:00\ttram\t169591\n2017-11-18 00:00:00\tzug\t141826\n2017-11-19 00:00:00\tzug\t132128\n2017-11-19 00:00:00\ttram\t128817\n2017-11-19 00:00:00\tschiff\t178\n2017-11-19 00:00:00\tbus\t333901\n2017-11-20 00:00:00\tbus\t562798\n2017-11-20 00:00:00\tzug\t151665\n2017-11-20 00:00:00\tschiff\t156\n2017-11-20 00:00:00\ttram\t179094\n2017-11-21 00:00:00\tbus\t561829\n2017-11-21 00:00:00\tzug\t151683\n2017-11-21 00:00:00\ttram\t178459\n2017-11-21 00:00:00\tschiff\t156\n2017-11-22 00:00:00\tschiff\t156\n2017-11-22 00:00:00\tzug\t151609\n2017-11-22 00:00:00\tbus\t560437\n2017-11-22 00:00:00\ttram\t179923\n2017-11-23 00:00:00\tschiff\t164\n2017-11-23 00:00:00\tbus\t519037\n2017-11-23 00:00:00\ttram\t164958\n2017-11-23 00:00:00\tzug\t151668\n2017-11-24 00:00:00\tbus\t568422\n2017-11-24 00:00:00\ttram\t184811\n2017-11-24 00:00:00\tschiff\t164\n2017-11-24 00:00:00\tzug\t152149\n2017-11-25 00:00:00\tschiff\t152\n2017-11-25 00:00:00\tbus\t457216\n2017-11-25 00:00:00\ttram\t163402\n2017-11-25 00:00:00\tzug\t142424\n2017-11-26 00:00:00\tzug\t132062\n2017-11-26 00:00:00\tschiff\t166\n2017-11-26 00:00:00\ttram\t127056\n2017-11-26 00:00:00\tbus\t335592\n2017-11-27 00:00:00\tbus\t564473\n2017-11-27 00:00:00\ttram\t178333\n2017-11-27 00:00:00\tzug\t150526\n2017-11-27 00:00:00\tschiff\t156\n2017-11-28 00:00:00\ttram\t178060\n2017-11-28 00:00:00\tbus\t561537\n2017-11-28 00:00:00\tschiff\t156\n2017-11-28 00:00:00\tzug\t150513\n2017-11-29 00:00:00\tschiff\t156\n2017-11-29 00:00:00\ttram\t180125\n2017-11-29 00:00:00\tzug\t150643\n2017-11-29 00:00:00\tbus\t563507\n2017-11-30 00:00:00\tschiff\t164\n2017-11-30 00:00:00\tbus\t562753\n2017-11-30 00:00:00\ttram\t177557\n2017-11-30 00:00:00\tzug\t150472\n"}]},"apps":[],"jobName":"paragraph_1522581092319_-1824333233","id":"20180401-131132_2088391393","dateCreated":"2018-04-01T13:11:32+0200","dateStarted":"2018-04-01T13:13:02+0200","dateFinished":"2018-04-01T13:13:17+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9321"},{"text":"%jdbc(hive)\nselect TripDate, lower(ProductId) ProductId, count(*) stops\nfrom musuroi.sbb\nwhere nvl(SkipStation, False) = False and nvl(FailedTrip, False) = False\ngroup by TripDate, lower(ProductId) \norder by TripDate;","user":"musuroi","dateUpdated":"2018-04-01T21:56:26+0200","config":{"colWidth":12,"enabled":true,"results":{"0":{"graph":{"mode":"multiBarChart","height":512,"optionOpen":false,"setting":{"multiBarChart":{"stacked":true},"scatterChart":{"xAxis":{"name":"tripdate","index":0,"aggr":"sum"},"yAxis":{"name":"stops","index":2,"aggr":"sum"},"group":{"name":"productid","index":1,"aggr":"sum"}}},"keys":[{"name":"tripdate","index":0,"aggr":"sum"}],"groups":[{"name":"productid","index":1,"aggr":"sum"}],"values":[{"name":"stops","index":2,"aggr":"sum"}],"commonSetting":{}},"helium":{}}},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"tripdate\tproductid\tstops\n2017-09-13 00:00:00\tbus\t555962\n2017-09-13 00:00:00\t\t5\n2017-09-13 00:00:00\ttram\t172771\n2017-09-13 00:00:00\tzug\t153767\n2017-09-13 00:00:00\tschiff\t379\n2017-09-14 00:00:00\tzug\t153731\n2017-09-14 00:00:00\t\t12\n2017-09-14 00:00:00\tbus\t556775\n2017-09-14 00:00:00\tschiff\t379\n2017-09-14 00:00:00\ttram\t171589\n2017-09-15 00:00:00\tbus\t563063\n2017-09-15 00:00:00\tschiff\t379\n2017-09-15 00:00:00\ttram\t175003\n2017-09-15 00:00:00\tzug\t155188\n2017-09-16 00:00:00\tbus\t453398\n2017-09-16 00:00:00\tschiff\t372\n2017-09-16 00:00:00\ttram\t159310\n2017-09-16 00:00:00\tzug\t146990\n2017-09-17 00:00:00\ttram\t120824\n2017-09-17 00:00:00\tbus\t332033\n2017-09-17 00:00:00\tschiff\t420\n2017-09-17 00:00:00\tzug\t135689\n2017-09-18 00:00:00\tschiff\t379\n2017-09-18 00:00:00\tbus\t557174\n2017-09-18 00:00:00\ttram\t172031\n2017-09-18 00:00:00\tzug\t153013\n2017-09-19 00:00:00\tbus\t554556\n2017-09-19 00:00:00\tschiff\t379\n2017-09-19 00:00:00\ttram\t171412\n2017-09-19 00:00:00\tzug\t153558\n2017-09-20 00:00:00\tbus\t556410\n2017-09-20 00:00:00\t\t2\n2017-09-20 00:00:00\tschiff\t379\n2017-09-20 00:00:00\ttram\t172880\n2017-09-20 00:00:00\tzug\t153532\n2017-09-21 00:00:00\t\t2\n2017-09-21 00:00:00\tbus\t557385\n2017-09-21 00:00:00\tschiff\t379\n2017-09-21 00:00:00\ttram\t171801\n2017-09-21 00:00:00\tzug\t153670\n2017-09-22 00:00:00\ttram\t176123\n2017-09-22 00:00:00\tbus\t552906\n2017-09-22 00:00:00\tschiff\t379\n2017-09-22 00:00:00\tzug\t154707\n2017-09-23 00:00:00\tschiff\t372\n2017-09-23 00:00:00\ttram\t163431\n2017-09-23 00:00:00\tzug\t143979\n2017-09-23 00:00:00\tbus\t435769\n2017-09-24 00:00:00\tbus\t314977\n2017-09-24 00:00:00\tschiff\t420\n2017-09-24 00:00:00\ttram\t120855\n2017-09-24 00:00:00\tzug\t133013\n2017-09-25 00:00:00\tbus\t547101\n2017-09-25 00:00:00\tschiff\t403\n2017-09-25 00:00:00\ttram\t170204\n2017-09-25 00:00:00\tzug\t151340\n2017-09-26 00:00:00\t\t6\n2017-09-26 00:00:00\tbus\t552309\n2017-09-26 00:00:00\tschiff\t403\n2017-09-26 00:00:00\ttram\t170889\n2017-09-26 00:00:00\tzug\t151426\n2017-09-27 00:00:00\ttram\t171149\n2017-09-27 00:00:00\t\t22\n2017-09-27 00:00:00\tbus\t553362\n2017-09-27 00:00:00\tschiff\t403\n2017-09-27 00:00:00\tzug\t151601\n2017-09-28 00:00:00\tbus\t553183\n2017-09-28 00:00:00\tschiff\t403\n2017-09-28 00:00:00\ttram\t170606\n2017-09-28 00:00:00\tzug\t151437\n2017-09-28 00:00:00\t\t7\n2017-09-29 00:00:00\tzug\t152343\n2017-09-29 00:00:00\t\t14\n2017-09-29 00:00:00\tbus\t559009\n2017-09-29 00:00:00\tschiff\t403\n2017-09-29 00:00:00\ttram\t173834\n2017-09-30 00:00:00\t\t11\n2017-09-30 00:00:00\tbus\t453572\n2017-09-30 00:00:00\tschiff\t396\n2017-09-30 00:00:00\ttram\t159650\n2017-09-30 00:00:00\tzug\t141677\n2017-10-01 00:00:00\tbus\t332640\n2017-10-01 00:00:00\tschiff\t444\n2017-10-01 00:00:00\ttram\t120285\n2017-10-01 00:00:00\tzug\t131989\n2017-10-02 00:00:00\tschiff\t403\n2017-10-02 00:00:00\tbus\t551067\n2017-10-02 00:00:00\ttram\t176041\n2017-10-02 00:00:00\tzug\t150896\n2017-10-03 00:00:00\t\t5\n2017-10-03 00:00:00\tbus\t548680\n2017-10-03 00:00:00\tschiff\t403\n2017-10-03 00:00:00\ttram\t175732\n2017-10-03 00:00:00\tzug\t150987\n2017-10-04 00:00:00\t\t7\n2017-10-04 00:00:00\tbus\t551395\n2017-10-04 00:00:00\tschiff\t403\n2017-10-04 00:00:00\ttram\t175995\n2017-10-04 00:00:00\tzug\t151153\n2017-10-05 00:00:00\t\t5\n2017-10-05 00:00:00\tbus\t551793\n2017-10-05 00:00:00\tschiff\t403\n2017-10-05 00:00:00\ttram\t175883\n2017-10-05 00:00:00\tzug\t151023\n2017-10-06 00:00:00\t\t9\n2017-10-06 00:00:00\tbus\t522666\n2017-10-06 00:00:00\tschiff\t403\n2017-10-06 00:00:00\ttram\t165747\n2017-10-06 00:00:00\tzug\t151796\n2017-10-07 00:00:00\tbus\t454017\n2017-10-07 00:00:00\tschiff\t396\n2017-10-07 00:00:00\ttram\t163984\n2017-10-07 00:00:00\tzug\t141339\n2017-10-08 00:00:00\tbus\t332667\n2017-10-08 00:00:00\ttram\t123969\n2017-10-08 00:00:00\tzug\t131155\n2017-10-08 00:00:00\tschiff\t444\n2017-10-09 00:00:00\tzug\t150664\n2017-10-09 00:00:00\t\t2\n2017-10-09 00:00:00\tbus\t551071\n2017-10-09 00:00:00\tschiff\t403\n2017-10-09 00:00:00\ttram\t176877\n2017-10-10 00:00:00\tbus\t551741\n2017-10-10 00:00:00\tschiff\t403\n2017-10-10 00:00:00\ttram\t175952\n2017-10-10 00:00:00\tzug\t150463\n2017-10-11 00:00:00\tbus\t551455\n2017-10-11 00:00:00\tschiff\t403\n2017-10-11 00:00:00\ttram\t175812\n2017-10-11 00:00:00\tzug\t150471\n2017-10-12 00:00:00\tbus\t550659\n2017-10-12 00:00:00\tschiff\t403\n2017-10-12 00:00:00\ttram\t175697\n2017-10-12 00:00:00\tzug\t150401\n2017-10-13 00:00:00\tbus\t558063\n2017-10-13 00:00:00\tschiff\t403\n2017-10-13 00:00:00\ttram\t179964\n2017-10-13 00:00:00\tzug\t151697\n2017-10-13 00:00:00\t\t5\n2017-10-14 00:00:00\tzug\t142813\n2017-10-14 00:00:00\t\t4\n2017-10-14 00:00:00\tbus\t454248\n2017-10-14 00:00:00\tschiff\t396\n2017-10-14 00:00:00\ttram\t163512\n2017-10-15 00:00:00\tbus\t332403\n2017-10-15 00:00:00\tschiff\t444\n2017-10-15 00:00:00\ttram\t124001\n2017-10-15 00:00:00\tzug\t131465\n2017-10-16 00:00:00\t\t2\n2017-10-16 00:00:00\tbus\t555375\n2017-10-16 00:00:00\tschiff\t403\n2017-10-16 00:00:00\ttram\t179537\n2017-10-16 00:00:00\tzug\t152448\n2017-10-17 00:00:00\tbus\t555376\n2017-10-17 00:00:00\tschiff\t403\n2017-10-17 00:00:00\ttram\t177713\n2017-10-17 00:00:00\tzug\t152662\n2017-10-18 00:00:00\tbus\t553311\n2017-10-18 00:00:00\tschiff\t403\n2017-10-18 00:00:00\t\t20\n2017-10-18 00:00:00\ttram\t177142\n2017-10-18 00:00:00\tzug\t152606\n2017-10-19 00:00:00\tschiff\t403\n2017-10-19 00:00:00\t\t1\n2017-10-19 00:00:00\tbus\t552881\n2017-10-19 00:00:00\ttram\t177799\n2017-10-19 00:00:00\tzug\t152716\n2017-10-20 00:00:00\t\t8\n2017-10-20 00:00:00\tbus\t561835\n2017-10-20 00:00:00\tschiff\t403\n2017-10-20 00:00:00\ttram\t181461\n2017-10-20 00:00:00\tzug\t153857\n2017-10-21 00:00:00\t\t9\n2017-10-21 00:00:00\tbus\t453609\n2017-10-21 00:00:00\tschiff\t396\n2017-10-21 00:00:00\ttram\t164813\n2017-10-21 00:00:00\tzug\t144128\n2017-10-22 00:00:00\tschiff\t444\n2017-10-22 00:00:00\ttram\t124806\n2017-10-22 00:00:00\tzug\t132543\n2017-10-22 00:00:00\tbus\t330406\n2017-10-23 00:00:00\t\t9\n2017-10-23 00:00:00\tbus\t554120\n2017-10-23 00:00:00\tschiff\t168\n2017-10-23 00:00:00\ttram\t176719\n2017-10-23 00:00:00\tzug\t152953\n2017-10-24 00:00:00\t\t12\n2017-10-24 00:00:00\tbus\t553280\n2017-10-24 00:00:00\tschiff\t168\n2017-10-24 00:00:00\ttram\t177720\n2017-10-24 00:00:00\tzug\t153107\n2017-10-25 00:00:00\t\t2\n2017-10-25 00:00:00\tbus\t558261\n2017-10-25 00:00:00\tschiff\t168\n2017-10-25 00:00:00\ttram\t177744\n2017-10-25 00:00:00\tzug\t153096\n2017-10-26 00:00:00\t\t9\n2017-10-26 00:00:00\tbus\t555793\n2017-10-26 00:00:00\tschiff\t176\n2017-10-26 00:00:00\ttram\t177409\n2017-10-26 00:00:00\tzug\t152823\n2017-10-27 00:00:00\tbus\t564248\n2017-10-27 00:00:00\tschiff\t176\n2017-10-27 00:00:00\t\t8\n2017-10-27 00:00:00\ttram\t181158\n2017-10-27 00:00:00\tzug\t154317\n2017-10-28 00:00:00\tschiff\t164\n2017-10-28 00:00:00\t\t16\n2017-10-28 00:00:00\tbus\t455852\n2017-10-28 00:00:00\ttram\t163158\n2017-10-28 00:00:00\tzug\t143635\n2017-10-29 00:00:00\t\t7\n2017-10-29 00:00:00\tbus\t330209\n2017-10-29 00:00:00\tschiff\t172\n2017-10-29 00:00:00\ttram\t124297\n2017-10-29 00:00:00\tzug\t132529\n2017-10-30 00:00:00\ttram\t177605\n2017-10-30 00:00:00\t\t8\n2017-10-30 00:00:00\tbus\t556503\n2017-10-30 00:00:00\tschiff\t168\n2017-10-30 00:00:00\tzug\t149110\n2017-10-31 00:00:00\tschiff\t168\n2017-10-31 00:00:00\ttram\t178504\n2017-10-31 00:00:00\tbus\t557785\n2017-10-31 00:00:00\tzug\t148458\n2017-11-01 00:00:00\tbus\t494087\n2017-11-01 00:00:00\tschiff\t168\n2017-11-01 00:00:00\ttram\t177751\n2017-11-01 00:00:00\tzug\t148090\n2017-11-02 00:00:00\ttram\t176475\n2017-11-02 00:00:00\tbus\t552539\n2017-11-02 00:00:00\tschiff\t176\n2017-11-02 00:00:00\tzug\t149020\n2017-11-03 00:00:00\tbus\t563183\n2017-11-03 00:00:00\tschiff\t176\n2017-11-03 00:00:00\ttram\t180900\n2017-11-03 00:00:00\tzug\t150298\n2017-11-04 00:00:00\tbus\t453666\n2017-11-04 00:00:00\tschiff\t164\n2017-11-04 00:00:00\ttram\t163999\n2017-11-04 00:00:00\tzug\t139704\n2017-11-05 00:00:00\tbus\t332758\n2017-11-05 00:00:00\tschiff\t172\n2017-11-05 00:00:00\ttram\t125097\n2017-11-05 00:00:00\tzug\t129181\n2017-11-06 00:00:00\tbus\t556924\n2017-11-06 00:00:00\tschiff\t168\n2017-11-06 00:00:00\ttram\t177671\n2017-11-06 00:00:00\tzug\t149148\n2017-11-07 00:00:00\tbus\t558029\n2017-11-07 00:00:00\tschiff\t168\n2017-11-07 00:00:00\ttram\t178040\n2017-11-07 00:00:00\tzug\t149059\n2017-11-08 00:00:00\tbus\t559589\n2017-11-08 00:00:00\tschiff\t168\n2017-11-08 00:00:00\ttram\t177787\n2017-11-08 00:00:00\tzug\t149244\n2017-11-09 00:00:00\tbus\t549040\n2017-11-09 00:00:00\tschiff\t176\n2017-11-09 00:00:00\ttram\t178767\n2017-11-09 00:00:00\tzug\t149114\n2017-11-10 00:00:00\tbus\t564428\n2017-11-10 00:00:00\tschiff\t176\n2017-11-10 00:00:00\ttram\t182636\n2017-11-10 00:00:00\tzug\t150600\n2017-11-11 00:00:00\tschiff\t164\n2017-11-11 00:00:00\ttram\t167092\n2017-11-11 00:00:00\tbus\t455453\n2017-11-11 00:00:00\tzug\t141190\n2017-11-12 00:00:00\tbus\t334124\n2017-11-12 00:00:00\tschiff\t178\n2017-11-12 00:00:00\ttram\t128562\n2017-11-12 00:00:00\tzug\t130580\n2017-11-13 00:00:00\ttram\t177663\n2017-11-13 00:00:00\tbus\t558129\n2017-11-13 00:00:00\tschiff\t174\n2017-11-13 00:00:00\tzug\t149194\n2017-11-14 00:00:00\tbus\t551462\n2017-11-14 00:00:00\tschiff\t174\n2017-11-14 00:00:00\ttram\t175060\n2017-11-14 00:00:00\tzug\t149281\n2017-11-15 00:00:00\tbus\t558843\n2017-11-15 00:00:00\tschiff\t174\n2017-11-15 00:00:00\ttram\t177499\n2017-11-15 00:00:00\tzug\t149496\n2017-11-16 00:00:00\tbus\t559946\n2017-11-16 00:00:00\tschiff\t182\n2017-11-16 00:00:00\ttram\t177585\n2017-11-16 00:00:00\tzug\t149419\n2017-11-17 00:00:00\tbus\t564632\n2017-11-17 00:00:00\tschiff\t176\n2017-11-17 00:00:00\ttram\t182265\n2017-11-17 00:00:00\tzug\t150396\n2017-11-18 00:00:00\tbus\t456790\n2017-11-18 00:00:00\tschiff\t164\n2017-11-18 00:00:00\ttram\t168810\n2017-11-18 00:00:00\tzug\t141441\n2017-11-19 00:00:00\tbus\t333135\n2017-11-19 00:00:00\tschiff\t178\n2017-11-19 00:00:00\ttram\t128482\n2017-11-19 00:00:00\tzug\t130673\n2017-11-20 00:00:00\tbus\t559663\n2017-11-20 00:00:00\tschiff\t156\n2017-11-20 00:00:00\ttram\t177934\n2017-11-20 00:00:00\tzug\t149971\n2017-11-21 00:00:00\tbus\t559004\n2017-11-21 00:00:00\tschiff\t156\n2017-11-21 00:00:00\ttram\t178130\n2017-11-21 00:00:00\tzug\t149971\n2017-11-22 00:00:00\tschiff\t156\n2017-11-22 00:00:00\ttram\t178750\n2017-11-22 00:00:00\tbus\t553108\n2017-11-22 00:00:00\tzug\t149946\n2017-11-23 00:00:00\tbus\t516852\n2017-11-23 00:00:00\tschiff\t164\n2017-11-23 00:00:00\ttram\t162789\n2017-11-23 00:00:00\tzug\t150089\n2017-11-24 00:00:00\tbus\t565252\n2017-11-24 00:00:00\tschiff\t164\n2017-11-24 00:00:00\ttram\t181880\n2017-11-24 00:00:00\tzug\t150699\n2017-11-25 00:00:00\tbus\t455774\n2017-11-25 00:00:00\tschiff\t152\n2017-11-25 00:00:00\ttram\t163128\n2017-11-25 00:00:00\tzug\t140290\n2017-11-26 00:00:00\ttram\t125735\n2017-11-26 00:00:00\tbus\t334501\n2017-11-26 00:00:00\tschiff\t166\n2017-11-26 00:00:00\tzug\t130142\n2017-11-27 00:00:00\tbus\t562015\n2017-11-27 00:00:00\tschiff\t156\n2017-11-27 00:00:00\ttram\t177720\n2017-11-27 00:00:00\tzug\t150156\n2017-11-28 00:00:00\tbus\t558769\n2017-11-28 00:00:00\tschiff\t156\n2017-11-28 00:00:00\ttram\t177491\n2017-11-28 00:00:00\tzug\t150127\n2017-11-29 00:00:00\tbus\t560072\n2017-11-29 00:00:00\tschiff\t156\n2017-11-29 00:00:00\ttram\t178674\n2017-11-29 00:00:00\tzug\t149919\n2017-11-30 00:00:00\tbus\t558965\n2017-11-30 00:00:00\tschiff\t164\n2017-11-30 00:00:00\ttram\t176384\n2017-11-30 00:00:00\tzug\t149898\n"}]},"apps":[],"jobName":"paragraph_1522524830394_-2116582072","id":"20180331-213350_1133108371","dateCreated":"2018-03-31T21:33:50+0200","dateStarted":"2018-04-01T11:48:46+0200","dateFinished":"2018-04-01T12:03:01+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9322"},{"text":"%md\nWe can clearly observe that there is a pattern that is repeating every seven bars, starting from the 6th one. Looking at the structure of the pattern made of five similar size bars and the last two with smaller sizes, it signals that it might fit the days of the week. Indeed, looking at 18-09-2017, the first bar corresponding to the first pattern, we can see that is a Monday, hence it is a weekly pattern. This explains the first five bars which also match the pattern as they correspond to the last 5 days of week 11-17.09.2017. Same argument applies for the last 4 bars.\n\nNow that we established that the pattern corresponds to the 7 days of the week, we can document additional observations. \n\nThere is hardly any difference between the first 5 days of the week in the number of stops, besides a small increase for Fridays. This is also valid at the finer level of type of transport where we can note that the number of bus, trams and trains stops are visually the same for the working days of the week.\n\nThe number of stops decreases significantly on Saturdays and Sundays comparing to the day before.\n\nWe want to look more in depth at possible exceptions to the pattern described above. For this, would be more helpful for us another visualization, where each type of transport is represented by a line. We show this below with the same query data.","user":"musuroi","dateUpdated":"2018-04-01T21:47:57+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>We can clearly observe that there is a pattern that is repeating every seven bars, starting from the 6th one. Looking at the structure of the pattern made of five similar size bars and the last two with smaller sizes, it signals that it might fit the days of the week. Indeed, looking at 18-09-2017, the first bar corresponding to the first pattern, we can see that is a Monday, hence it is a weekly pattern. This explains the first five bars which also match the pattern as they correspond to the last 5 days of week 11-17.09.2017. Same argument applies for the last 4 bars.</p>\n<p>Now that we established that the pattern corresponds to the 7 days of the week, we can document additional observations.</p>\n<p>There is hardly any difference between the first 5 days of the week in the number of stops, besides a small increase for Fridays. This is also valid at the finer level of type of transport where we can note that the number of bus, trams and trains stops are visually the same for the working days of the week.</p>\n<p>The number of stops decreases significantly on Saturdays and Sundays comparing to the day before.</p>\n<p>We want to look more in depth at possible exceptions to the pattern described above. For this, would be more helpful for us another visualization, where each type of transport is represented by a line. We show this below with the same query data.</p>\n"}]},"apps":[],"jobName":"paragraph_1522577678595_2092566054","id":"20180401-121438_114956265","dateCreated":"2018-04-01T12:14:38+0200","dateStarted":"2018-04-01T21:47:57+0200","dateFinished":"2018-04-01T21:47:57+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9323"},{"text":"%jdbc(hive)\nselect TripDate, lower(ProductId) ProductId, count(*) stops\nfrom musuroi.sbb\nwhere nvl(SkipStation, False) = False and nvl(FailedTrip, False) = False\ngroup by TripDate, lower(ProductId)\norder by TripDate;","user":"musuroi","dateUpdated":"2018-04-01T21:48:57+0200","config":{"colWidth":12,"enabled":true,"results":{"0":{"graph":{"mode":"lineChart","height":424,"optionOpen":false,"setting":{"lineChart":{}},"commonSetting":{},"keys":[{"name":"tripdate","index":0,"aggr":"sum"}],"groups":[{"name":"productid","index":1,"aggr":"sum"}],"values":[{"name":"stops","index":2,"aggr":"sum"}]},"helium":{}}},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"tripdate\tproductid\tstops\n2017-09-13 00:00:00\tbus\t555962\n2017-09-13 00:00:00\t\t5\n2017-09-13 00:00:00\ttram\t172771\n2017-09-13 00:00:00\tzug\t153767\n2017-09-13 00:00:00\tschiff\t379\n2017-09-14 00:00:00\tzug\t153731\n2017-09-14 00:00:00\t\t12\n2017-09-14 00:00:00\tbus\t556775\n2017-09-14 00:00:00\tschiff\t379\n2017-09-14 00:00:00\ttram\t171589\n2017-09-15 00:00:00\tbus\t563063\n2017-09-15 00:00:00\tschiff\t379\n2017-09-15 00:00:00\ttram\t175003\n2017-09-15 00:00:00\tzug\t155188\n2017-09-16 00:00:00\tbus\t453398\n2017-09-16 00:00:00\tschiff\t372\n2017-09-16 00:00:00\ttram\t159310\n2017-09-16 00:00:00\tzug\t146990\n2017-09-17 00:00:00\ttram\t120824\n2017-09-17 00:00:00\tbus\t332033\n2017-09-17 00:00:00\tschiff\t420\n2017-09-17 00:00:00\tzug\t135689\n2017-09-18 00:00:00\tschiff\t379\n2017-09-18 00:00:00\tbus\t557174\n2017-09-18 00:00:00\ttram\t172031\n2017-09-18 00:00:00\tzug\t153013\n2017-09-19 00:00:00\tbus\t554556\n2017-09-19 00:00:00\tschiff\t379\n2017-09-19 00:00:00\ttram\t171412\n2017-09-19 00:00:00\tzug\t153558\n2017-09-20 00:00:00\tbus\t556410\n2017-09-20 00:00:00\t\t2\n2017-09-20 00:00:00\tschiff\t379\n2017-09-20 00:00:00\ttram\t172880\n2017-09-20 00:00:00\tzug\t153532\n2017-09-21 00:00:00\t\t2\n2017-09-21 00:00:00\tbus\t557385\n2017-09-21 00:00:00\tschiff\t379\n2017-09-21 00:00:00\ttram\t171801\n2017-09-21 00:00:00\tzug\t153670\n2017-09-22 00:00:00\ttram\t176123\n2017-09-22 00:00:00\tbus\t552906\n2017-09-22 00:00:00\tschiff\t379\n2017-09-22 00:00:00\tzug\t154707\n2017-09-23 00:00:00\tschiff\t372\n2017-09-23 00:00:00\ttram\t163431\n2017-09-23 00:00:00\tzug\t143979\n2017-09-23 00:00:00\tbus\t435769\n2017-09-24 00:00:00\tbus\t314977\n2017-09-24 00:00:00\tschiff\t420\n2017-09-24 00:00:00\ttram\t120855\n2017-09-24 00:00:00\tzug\t133013\n2017-09-25 00:00:00\tbus\t547101\n2017-09-25 00:00:00\tschiff\t403\n2017-09-25 00:00:00\ttram\t170204\n2017-09-25 00:00:00\tzug\t151340\n2017-09-26 00:00:00\t\t6\n2017-09-26 00:00:00\tbus\t552309\n2017-09-26 00:00:00\tschiff\t403\n2017-09-26 00:00:00\ttram\t170889\n2017-09-26 00:00:00\tzug\t151426\n2017-09-27 00:00:00\ttram\t171149\n2017-09-27 00:00:00\t\t22\n2017-09-27 00:00:00\tbus\t553362\n2017-09-27 00:00:00\tschiff\t403\n2017-09-27 00:00:00\tzug\t151601\n2017-09-28 00:00:00\tbus\t553183\n2017-09-28 00:00:00\tschiff\t403\n2017-09-28 00:00:00\ttram\t170606\n2017-09-28 00:00:00\tzug\t151437\n2017-09-28 00:00:00\t\t7\n2017-09-29 00:00:00\tzug\t152343\n2017-09-29 00:00:00\t\t14\n2017-09-29 00:00:00\tbus\t559009\n2017-09-29 00:00:00\tschiff\t403\n2017-09-29 00:00:00\ttram\t173834\n2017-09-30 00:00:00\t\t11\n2017-09-30 00:00:00\tbus\t453572\n2017-09-30 00:00:00\tschiff\t396\n2017-09-30 00:00:00\ttram\t159650\n2017-09-30 00:00:00\tzug\t141677\n2017-10-01 00:00:00\tbus\t332640\n2017-10-01 00:00:00\tschiff\t444\n2017-10-01 00:00:00\ttram\t120285\n2017-10-01 00:00:00\tzug\t131989\n2017-10-02 00:00:00\tschiff\t403\n2017-10-02 00:00:00\tbus\t551067\n2017-10-02 00:00:00\ttram\t176041\n2017-10-02 00:00:00\tzug\t150896\n2017-10-03 00:00:00\t\t5\n2017-10-03 00:00:00\tbus\t548680\n2017-10-03 00:00:00\tschiff\t403\n2017-10-03 00:00:00\ttram\t175732\n2017-10-03 00:00:00\tzug\t150987\n2017-10-04 00:00:00\t\t7\n2017-10-04 00:00:00\tbus\t551395\n2017-10-04 00:00:00\tschiff\t403\n2017-10-04 00:00:00\ttram\t175995\n2017-10-04 00:00:00\tzug\t151153\n2017-10-05 00:00:00\t\t5\n2017-10-05 00:00:00\tbus\t551793\n2017-10-05 00:00:00\tschiff\t403\n2017-10-05 00:00:00\ttram\t175883\n2017-10-05 00:00:00\tzug\t151023\n2017-10-06 00:00:00\t\t9\n2017-10-06 00:00:00\tbus\t522666\n2017-10-06 00:00:00\tschiff\t403\n2017-10-06 00:00:00\ttram\t165747\n2017-10-06 00:00:00\tzug\t151796\n2017-10-07 00:00:00\tbus\t454017\n2017-10-07 00:00:00\tschiff\t396\n2017-10-07 00:00:00\ttram\t163984\n2017-10-07 00:00:00\tzug\t141339\n2017-10-08 00:00:00\tbus\t332667\n2017-10-08 00:00:00\ttram\t123969\n2017-10-08 00:00:00\tzug\t131155\n2017-10-08 00:00:00\tschiff\t444\n2017-10-09 00:00:00\tzug\t150664\n2017-10-09 00:00:00\t\t2\n2017-10-09 00:00:00\tbus\t551071\n2017-10-09 00:00:00\tschiff\t403\n2017-10-09 00:00:00\ttram\t176877\n2017-10-10 00:00:00\tbus\t551741\n2017-10-10 00:00:00\tschiff\t403\n2017-10-10 00:00:00\ttram\t175952\n2017-10-10 00:00:00\tzug\t150463\n2017-10-11 00:00:00\tbus\t551455\n2017-10-11 00:00:00\tschiff\t403\n2017-10-11 00:00:00\ttram\t175812\n2017-10-11 00:00:00\tzug\t150471\n2017-10-12 00:00:00\tbus\t550659\n2017-10-12 00:00:00\tschiff\t403\n2017-10-12 00:00:00\ttram\t175697\n2017-10-12 00:00:00\tzug\t150401\n2017-10-13 00:00:00\tbus\t558063\n2017-10-13 00:00:00\tschiff\t403\n2017-10-13 00:00:00\ttram\t179964\n2017-10-13 00:00:00\tzug\t151697\n2017-10-13 00:00:00\t\t5\n2017-10-14 00:00:00\tzug\t142813\n2017-10-14 00:00:00\t\t4\n2017-10-14 00:00:00\tbus\t454248\n2017-10-14 00:00:00\tschiff\t396\n2017-10-14 00:00:00\ttram\t163512\n2017-10-15 00:00:00\tbus\t332403\n2017-10-15 00:00:00\tschiff\t444\n2017-10-15 00:00:00\ttram\t124001\n2017-10-15 00:00:00\tzug\t131465\n2017-10-16 00:00:00\t\t2\n2017-10-16 00:00:00\tbus\t555375\n2017-10-16 00:00:00\tschiff\t403\n2017-10-16 00:00:00\ttram\t179537\n2017-10-16 00:00:00\tzug\t152448\n2017-10-17 00:00:00\tbus\t555376\n2017-10-17 00:00:00\tschiff\t403\n2017-10-17 00:00:00\ttram\t177713\n2017-10-17 00:00:00\tzug\t152662\n2017-10-18 00:00:00\tbus\t553311\n2017-10-18 00:00:00\tschiff\t403\n2017-10-18 00:00:00\t\t20\n2017-10-18 00:00:00\ttram\t177142\n2017-10-18 00:00:00\tzug\t152606\n2017-10-19 00:00:00\tschiff\t403\n2017-10-19 00:00:00\t\t1\n2017-10-19 00:00:00\tbus\t552881\n2017-10-19 00:00:00\ttram\t177799\n2017-10-19 00:00:00\tzug\t152716\n2017-10-20 00:00:00\t\t8\n2017-10-20 00:00:00\tbus\t561835\n2017-10-20 00:00:00\tschiff\t403\n2017-10-20 00:00:00\ttram\t181461\n2017-10-20 00:00:00\tzug\t153857\n2017-10-21 00:00:00\t\t9\n2017-10-21 00:00:00\tbus\t453609\n2017-10-21 00:00:00\tschiff\t396\n2017-10-21 00:00:00\ttram\t164813\n2017-10-21 00:00:00\tzug\t144128\n2017-10-22 00:00:00\tschiff\t444\n2017-10-22 00:00:00\ttram\t124806\n2017-10-22 00:00:00\tzug\t132543\n2017-10-22 00:00:00\tbus\t330406\n2017-10-23 00:00:00\t\t9\n2017-10-23 00:00:00\tbus\t554120\n2017-10-23 00:00:00\tschiff\t168\n2017-10-23 00:00:00\ttram\t176719\n2017-10-23 00:00:00\tzug\t152953\n2017-10-24 00:00:00\t\t12\n2017-10-24 00:00:00\tbus\t553280\n2017-10-24 00:00:00\tschiff\t168\n2017-10-24 00:00:00\ttram\t177720\n2017-10-24 00:00:00\tzug\t153107\n2017-10-25 00:00:00\t\t2\n2017-10-25 00:00:00\tbus\t558261\n2017-10-25 00:00:00\tschiff\t168\n2017-10-25 00:00:00\ttram\t177744\n2017-10-25 00:00:00\tzug\t153096\n2017-10-26 00:00:00\t\t9\n2017-10-26 00:00:00\tbus\t555793\n2017-10-26 00:00:00\tschiff\t176\n2017-10-26 00:00:00\ttram\t177409\n2017-10-26 00:00:00\tzug\t152823\n2017-10-27 00:00:00\tbus\t564248\n2017-10-27 00:00:00\tschiff\t176\n2017-10-27 00:00:00\t\t8\n2017-10-27 00:00:00\ttram\t181158\n2017-10-27 00:00:00\tzug\t154317\n2017-10-28 00:00:00\tschiff\t164\n2017-10-28 00:00:00\t\t16\n2017-10-28 00:00:00\tbus\t455852\n2017-10-28 00:00:00\ttram\t163158\n2017-10-28 00:00:00\tzug\t143635\n2017-10-29 00:00:00\t\t7\n2017-10-29 00:00:00\tbus\t330209\n2017-10-29 00:00:00\tschiff\t172\n2017-10-29 00:00:00\ttram\t124297\n2017-10-29 00:00:00\tzug\t132529\n2017-10-30 00:00:00\ttram\t177605\n2017-10-30 00:00:00\t\t8\n2017-10-30 00:00:00\tbus\t556503\n2017-10-30 00:00:00\tschiff\t168\n2017-10-30 00:00:00\tzug\t149110\n2017-10-31 00:00:00\tschiff\t168\n2017-10-31 00:00:00\ttram\t178504\n2017-10-31 00:00:00\tbus\t557785\n2017-10-31 00:00:00\tzug\t148458\n2017-11-01 00:00:00\tbus\t494087\n2017-11-01 00:00:00\tschiff\t168\n2017-11-01 00:00:00\ttram\t177751\n2017-11-01 00:00:00\tzug\t148090\n2017-11-02 00:00:00\ttram\t176475\n2017-11-02 00:00:00\tbus\t552539\n2017-11-02 00:00:00\tschiff\t176\n2017-11-02 00:00:00\tzug\t149020\n2017-11-03 00:00:00\tbus\t563183\n2017-11-03 00:00:00\tschiff\t176\n2017-11-03 00:00:00\ttram\t180900\n2017-11-03 00:00:00\tzug\t150298\n2017-11-04 00:00:00\tbus\t453666\n2017-11-04 00:00:00\tschiff\t164\n2017-11-04 00:00:00\ttram\t163999\n2017-11-04 00:00:00\tzug\t139704\n2017-11-05 00:00:00\tbus\t332758\n2017-11-05 00:00:00\tschiff\t172\n2017-11-05 00:00:00\ttram\t125097\n2017-11-05 00:00:00\tzug\t129181\n2017-11-06 00:00:00\tbus\t556924\n2017-11-06 00:00:00\tschiff\t168\n2017-11-06 00:00:00\ttram\t177671\n2017-11-06 00:00:00\tzug\t149148\n2017-11-07 00:00:00\tbus\t558029\n2017-11-07 00:00:00\tschiff\t168\n2017-11-07 00:00:00\ttram\t178040\n2017-11-07 00:00:00\tzug\t149059\n2017-11-08 00:00:00\tbus\t559589\n2017-11-08 00:00:00\tschiff\t168\n2017-11-08 00:00:00\ttram\t177787\n2017-11-08 00:00:00\tzug\t149244\n2017-11-09 00:00:00\tbus\t549040\n2017-11-09 00:00:00\tschiff\t176\n2017-11-09 00:00:00\ttram\t178767\n2017-11-09 00:00:00\tzug\t149114\n2017-11-10 00:00:00\tbus\t564428\n2017-11-10 00:00:00\tschiff\t176\n2017-11-10 00:00:00\ttram\t182636\n2017-11-10 00:00:00\tzug\t150600\n2017-11-11 00:00:00\tschiff\t164\n2017-11-11 00:00:00\ttram\t167092\n2017-11-11 00:00:00\tbus\t455453\n2017-11-11 00:00:00\tzug\t141190\n2017-11-12 00:00:00\tbus\t334124\n2017-11-12 00:00:00\tschiff\t178\n2017-11-12 00:00:00\ttram\t128562\n2017-11-12 00:00:00\tzug\t130580\n2017-11-13 00:00:00\ttram\t177663\n2017-11-13 00:00:00\tbus\t558129\n2017-11-13 00:00:00\tschiff\t174\n2017-11-13 00:00:00\tzug\t149194\n2017-11-14 00:00:00\tbus\t551462\n2017-11-14 00:00:00\tschiff\t174\n2017-11-14 00:00:00\ttram\t175060\n2017-11-14 00:00:00\tzug\t149281\n2017-11-15 00:00:00\tbus\t558843\n2017-11-15 00:00:00\tschiff\t174\n2017-11-15 00:00:00\ttram\t177499\n2017-11-15 00:00:00\tzug\t149496\n2017-11-16 00:00:00\tbus\t559946\n2017-11-16 00:00:00\tschiff\t182\n2017-11-16 00:00:00\ttram\t177585\n2017-11-16 00:00:00\tzug\t149419\n2017-11-17 00:00:00\tbus\t564632\n2017-11-17 00:00:00\tschiff\t176\n2017-11-17 00:00:00\ttram\t182265\n2017-11-17 00:00:00\tzug\t150396\n2017-11-18 00:00:00\tbus\t456790\n2017-11-18 00:00:00\tschiff\t164\n2017-11-18 00:00:00\ttram\t168810\n2017-11-18 00:00:00\tzug\t141441\n2017-11-19 00:00:00\tbus\t333135\n2017-11-19 00:00:00\tschiff\t178\n2017-11-19 00:00:00\ttram\t128482\n2017-11-19 00:00:00\tzug\t130673\n2017-11-20 00:00:00\tbus\t559663\n2017-11-20 00:00:00\tschiff\t156\n2017-11-20 00:00:00\ttram\t177934\n2017-11-20 00:00:00\tzug\t149971\n2017-11-21 00:00:00\tbus\t559004\n2017-11-21 00:00:00\tschiff\t156\n2017-11-21 00:00:00\ttram\t178130\n2017-11-21 00:00:00\tzug\t149971\n2017-11-22 00:00:00\tschiff\t156\n2017-11-22 00:00:00\ttram\t178750\n2017-11-22 00:00:00\tbus\t553108\n2017-11-22 00:00:00\tzug\t149946\n2017-11-23 00:00:00\tbus\t516852\n2017-11-23 00:00:00\tschiff\t164\n2017-11-23 00:00:00\ttram\t162789\n2017-11-23 00:00:00\tzug\t150089\n2017-11-24 00:00:00\tbus\t565252\n2017-11-24 00:00:00\tschiff\t164\n2017-11-24 00:00:00\ttram\t181880\n2017-11-24 00:00:00\tzug\t150699\n2017-11-25 00:00:00\tbus\t455774\n2017-11-25 00:00:00\tschiff\t152\n2017-11-25 00:00:00\ttram\t163128\n2017-11-25 00:00:00\tzug\t140290\n2017-11-26 00:00:00\ttram\t125735\n2017-11-26 00:00:00\tbus\t334501\n2017-11-26 00:00:00\tschiff\t166\n2017-11-26 00:00:00\tzug\t130142\n2017-11-27 00:00:00\tbus\t562015\n2017-11-27 00:00:00\tschiff\t156\n2017-11-27 00:00:00\ttram\t177720\n2017-11-27 00:00:00\tzug\t150156\n2017-11-28 00:00:00\tbus\t558769\n2017-11-28 00:00:00\tschiff\t156\n2017-11-28 00:00:00\ttram\t177491\n2017-11-28 00:00:00\tzug\t150127\n2017-11-29 00:00:00\tbus\t560072\n2017-11-29 00:00:00\tschiff\t156\n2017-11-29 00:00:00\ttram\t178674\n2017-11-29 00:00:00\tzug\t149919\n2017-11-30 00:00:00\tbus\t558965\n2017-11-30 00:00:00\tschiff\t164\n2017-11-30 00:00:00\ttram\t176384\n2017-11-30 00:00:00\tzug\t149898\n"}]},"apps":[],"jobName":"paragraph_1522612073323_21291622","id":"20180401-214753_796730872","dateCreated":"2018-04-01T21:47:53+0200","dateStarted":"2018-04-01T21:47:56+0200","dateFinished":"2018-04-01T21:48:18+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9324"},{"text":"%md\nThere are a few days that do not fit the pattern we described: **06.10.2017, 01.11.2017** and **23.11.2017**. Looking at the number of stops per category for these three exceptional days, we observe that \nthe number of train stops stays the same, while only the number of bus and tram stops change, but not for all three of them. \nChecking (public holidays in Switzerland)[https://en.wikipedia.org/wiki/Public_holidays_in_Switzerland] we observe that \n01.11.2017 is a national holiday, while the other two do not appear in the table. This one is the day in which only the number of bus stops drops, while the train and tram stops stays the same.\n\nA thing to note is that for **06.10.2017** and **23.11.2017**, the number of bus stops and tram stops both drop, which did not happen for **01.11.2017** that was a national holiday.","user":"musuroi","dateUpdated":"2018-04-16T22:11:48+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>There are a few days that do not fit the pattern we described: <strong>06.10.2017, 01.11.2017</strong> and <strong>23.11.2017</strong>. Looking at the number of stops per category for these three exceptional days, we observe that\n<br />the number of train stops stays the same, while only the number of bus and tram stops change, but not for all three of them.\n<br />Checking (public holidays in Switzerland)[https://en.wikipedia.org/wiki/Public_holidays_in_Switzerland] we observe that\n<br />01.11.2017 is a national holiday, while the other two do not appear in the table. This one is the day in which only the number of bus stops drops, while the train and tram stops stays the same.</p>\n<p>A thing to note is that for <strong>06.10.2017</strong> and <strong>23.11.2017</strong>, the number of bus stops and tram stops both drop, which did not happen for <strong>01.11.2017</strong> that was a national holiday.</p>\n"}]},"apps":[],"jobName":"paragraph_1522611930070_2055504506","id":"20180401-214530_1343729408","dateCreated":"2018-04-01T21:45:30+0200","dateStarted":"2018-04-16T22:11:48+0200","dateFinished":"2018-04-16T22:11:48+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9325"},{"text":"%md\n### Question I.c. (5/30)\n\ni) How many trips can you find for the date 17.10.2017? Display the information by transport type (bus, train, etc.). \nNote: trip=full journey.\n\nii) Now do the same for the whole dataset. Is 17.10.2017 a typical day?\n\n### Answers\n\n**Subquestion i)**\n\nIn the dataset description at the beginning of the notebook we have the following note:\n\n> When the stop is the start or end of a journey, the corresponding columns will be empty (ANKUNFTSZEIT/ABFAHRTSZEIT) (In our case ArrivalTimeScheduled/DepartureTimeScheduled)\n\nmeaning that a NULL value in column **ArrivalTimeScheduled** marks the beginning of a new journey. Now, as specified in the description of the dataset, some trips have failed and, as we understand from the note, we are asked for the number of **full journeys** which excludes the failed trips. Given this, the number of full journeys can be mapped directly to the number of stops that correspond to the end of a journey that did not failed. In our table, this translates to the number rows with NULL values in column DepartureTimeScheduled and False for FailedTrip.","user":"musuroi","dateUpdated":"2018-04-16T22:12:01+0200","config":{"tableHide":false,"editorSetting":{"editOnDblClick":true,"language":"markdown"},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Question I.c. (5/30)</h3>\n<p>i) How many trips can you find for the date 17.10.2017? Display the information by transport type (bus, train, etc.).\n<br />Note: trip=full journey.</p>\n<p>ii) Now do the same for the whole dataset. Is 17.10.2017 a typical day?</p>\n<h3>Answers</h3>\n<p><strong>Subquestion i)</strong></p>\n<p>In the dataset description at the beginning of the notebook we have the following note:</p>\n<blockquote><p>When the stop is the start or end of a journey, the corresponding columns will be empty (ANKUNFTSZEIT/ABFAHRTSZEIT) (In our case ArrivalTimeScheduled/DepartureTimeScheduled)</p>\n</blockquote>\n<p>meaning that a NULL value in column <strong>ArrivalTimeScheduled</strong> marks the beginning of a new journey. Now, as specified in the description of the dataset, some trips have failed and, as we understand from the note, we are asked for the number of <strong>full journeys</strong> which excludes the failed trips. Given this, the number of full journeys can be mapped directly to the number of stops that correspond to the end of a journey that did not failed. In our table, this translates to the number rows with NULL values in column DepartureTimeScheduled and False for FailedTrip.</p>\n"}]},"apps":[],"jobName":"paragraph_1522235709483_1431074473","id":"20180326-144303_347568851","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-16T22:12:01+0200","dateFinished":"2018-04-16T22:12:01+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9326"},{"text":"%jdbc(hive)\nselect lower(ProductId) as ProductId, count(*) as counts\nfrom musuroi.sbb_17_10_2017\nwhere DepartureTimeScheduled is NULL and FailedTrip = False\ngroup by lower(ProductId);","user":"musuroi","dateUpdated":"2018-04-02T11:25:00+0200","config":{"colWidth":12,"enabled":true,"results":{"0":{"graph":{"mode":"multiBarChart","height":300,"optionOpen":false,"setting":{"multiBarChart":{"stacked":false}},"commonSetting":{},"keys":[{"name":"productid","index":0,"aggr":"sum"}],"groups":[],"values":[{"name":"counts","index":1,"aggr":"sum"}]},"helium":{}}},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"productid\tcounts\nbus\t35554\nschiff\t71\ntram\t8308\nzug\t14096\n"}]},"apps":[],"jobName":"paragraph_1522660303433_-1898076292","id":"20180402-111143_1035889642","dateCreated":"2018-04-02T11:11:43+0200","dateStarted":"2018-04-02T11:24:46+0200","dateFinished":"2018-04-02T11:24:53+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9327"},{"text":"%md\n**Subquestion ii)**\nTo compute the same statistics for the entire dataset, we apply the same procedure as above with that addition that we group by TripDate first.","user":"musuroi","dateUpdated":"2018-04-02T11:30:35+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p><strong>Subquestion ii)</strong>\n<br />To compute the same statistics for the entire dataset, we apply the same procedure as above with that addition that we group by TripDate first.</p>\n"}]},"apps":[],"jobName":"paragraph_1522661110112_319565462","id":"20180402-112510_932941600","dateCreated":"2018-04-02T11:25:10+0200","dateStarted":"2018-04-02T11:30:23+0200","dateFinished":"2018-04-02T11:30:23+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9328"},{"text":"%jdbc(hive)\nselect TripDate, lower(ProductId) as ProductId, count(*) as counts\nfrom musuroi.sbb\nwhere DepartureTimeScheduled is NULL and FailedTrip = False\ngroup by TripDate, lower(ProductId);","user":"baetu","dateUpdated":"2018-04-13T15:22:47+0200","config":{"colWidth":12,"enabled":true,"results":{"0":{"graph":{"mode":"lineChart","height":300,"optionOpen":false,"setting":{"multiBarChart":{"stacked":true}},"commonSetting":{},"keys":[{"name":"tripdate","index":0,"aggr":"sum"}],"groups":[{"name":"productid","index":1,"aggr":"sum"}],"values":[{"name":"counts","index":2,"aggr":"sum"}]},"helium":{}}},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"tripdate\tproductid\tcounts\n2017-09-13 00:00:00\t\t1\n2017-09-13 00:00:00\tbus\t35629\n2017-09-13 00:00:00\tschiff\t67\n2017-09-13 00:00:00\ttram\t8803\n2017-09-13 00:00:00\tzug\t14145\n2017-09-14 00:00:00\tbus\t35690\n2017-09-14 00:00:00\tschiff\t67\n2017-09-14 00:00:00\ttram\t8720\n2017-09-14 00:00:00\tzug\t14122\n2017-09-15 00:00:00\tbus\t36011\n2017-09-15 00:00:00\tschiff\t67\n2017-09-15 00:00:00\ttram\t8856\n2017-09-15 00:00:00\tzug\t14216\n2017-09-16 00:00:00\tbus\t28097\n2017-09-16 00:00:00\tschiff\t65\n2017-09-16 00:00:00\ttram\t8098\n2017-09-16 00:00:00\tzug\t13137\n2017-09-17 00:00:00\tbus\t20796\n2017-09-17 00:00:00\tschiff\t76\n2017-09-17 00:00:00\ttram\t6103\n2017-09-17 00:00:00\tzug\t12208\n2017-09-18 00:00:00\tbus\t35706\n2017-09-18 00:00:00\tschiff\t67\n2017-09-18 00:00:00\ttram\t8704\n2017-09-18 00:00:00\tzug\t14021\n2017-09-19 00:00:00\tbus\t35600\n2017-09-19 00:00:00\tschiff\t67\n2017-09-19 00:00:00\ttram\t8686\n2017-09-19 00:00:00\tzug\t14107\n2017-09-20 00:00:00\tbus\t35714\n2017-09-20 00:00:00\tschiff\t67\n2017-09-20 00:00:00\ttram\t8829\n2017-09-20 00:00:00\tzug\t14101\n2017-09-21 00:00:00\tbus\t35728\n2017-09-21 00:00:00\tschiff\t67\n2017-09-21 00:00:00\ttram\t8692\n2017-09-21 00:00:00\tzug\t14121\n2017-09-22 00:00:00\tbus\t35282\n2017-09-22 00:00:00\tschiff\t67\n2017-09-22 00:00:00\ttram\t8982\n2017-09-22 00:00:00\tzug\t14207\n2017-09-23 00:00:00\tbus\t27044\n2017-09-23 00:00:00\tschiff\t65\n2017-09-23 00:00:00\ttram\t8393\n2017-09-23 00:00:00\tzug\t12938\n2017-09-24 00:00:00\tbus\t19696\n2017-09-24 00:00:00\tschiff\t76\n2017-09-24 00:00:00\ttram\t6104\n2017-09-24 00:00:00\tzug\t12112\n2017-09-25 00:00:00\tbus\t35062\n2017-09-25 00:00:00\tschiff\t71\n2017-09-25 00:00:00\ttram\t8621\n2017-09-25 00:00:00\tzug\t14003\n2017-09-26 00:00:00\t\t1\n2017-09-26 00:00:00\tbus\t35458\n2017-09-26 00:00:00\tschiff\t71\n2017-09-26 00:00:00\ttram\t8675\n2017-09-26 00:00:00\tzug\t14013\n2017-09-27 00:00:00\t\t2\n2017-09-27 00:00:00\tbus\t35587\n2017-09-27 00:00:00\tschiff\t71\n2017-09-27 00:00:00\ttram\t8679\n2017-09-27 00:00:00\tzug\t13973\n2017-09-28 00:00:00\tbus\t35468\n2017-09-28 00:00:00\tschiff\t71\n2017-09-28 00:00:00\ttram\t8640\n2017-09-28 00:00:00\tzug\t13948\n2017-09-29 00:00:00\t\t2\n2017-09-29 00:00:00\tbus\t35848\n2017-09-29 00:00:00\tschiff\t71\n2017-09-29 00:00:00\ttram\t8807\n2017-09-29 00:00:00\tzug\t14193\n2017-09-30 00:00:00\t\t3\n2017-09-30 00:00:00\tbus\t28179\n2017-09-30 00:00:00\tschiff\t69\n2017-09-30 00:00:00\ttram\t7832\n2017-09-30 00:00:00\tzug\t12988\n2017-10-01 00:00:00\tbus\t20837\n2017-10-01 00:00:00\tschiff\t80\n2017-10-01 00:00:00\ttram\t5956\n2017-10-01 00:00:00\tzug\t12142\n2017-10-02 00:00:00\tbus\t35322\n2017-10-02 00:00:00\tschiff\t71\n2017-10-02 00:00:00\ttram\t8510\n2017-10-02 00:00:00\tzug\t14067\n2017-10-03 00:00:00\t\t3\n2017-10-03 00:00:00\tbus\t35243\n2017-10-03 00:00:00\tschiff\t71\n2017-10-03 00:00:00\ttram\t8498\n2017-10-03 00:00:00\tzug\t14068\n2017-10-04 00:00:00\tbus\t35417\n2017-10-04 00:00:00\tschiff\t71\n2017-10-04 00:00:00\ttram\t8514\n2017-10-04 00:00:00\tzug\t14096\n2017-10-05 00:00:00\tbus\t35318\n2017-10-05 00:00:00\tschiff\t71\n2017-10-05 00:00:00\ttram\t8526\n2017-10-05 00:00:00\tzug\t14088\n2017-10-06 00:00:00\tbus\t33066\n2017-10-06 00:00:00\tschiff\t71\n2017-10-06 00:00:00\ttram\t8044\n2017-10-06 00:00:00\tzug\t14160\n2017-10-07 00:00:00\tbus\t28160\n2017-10-07 00:00:00\tschiff\t69\n2017-10-07 00:00:00\ttram\t7995\n2017-10-07 00:00:00\tzug\t12974\n2017-10-08 00:00:00\tbus\t20830\n2017-10-08 00:00:00\tschiff\t80\n2017-10-08 00:00:00\ttram\t6122\n2017-10-08 00:00:00\tzug\t12136\n2017-10-09 00:00:00\tbus\t35190\n2017-10-09 00:00:00\tschiff\t71\n2017-10-09 00:00:00\ttram\t8609\n2017-10-09 00:00:00\tzug\t13938\n2017-10-10 00:00:00\tbus\t35282\n2017-10-10 00:00:00\tschiff\t71\n2017-10-10 00:00:00\ttram\t8524\n2017-10-10 00:00:00\tzug\t13911\n2017-10-11 00:00:00\tbus\t35101\n2017-10-11 00:00:00\tschiff\t71\n2017-10-11 00:00:00\ttram\t8504\n2017-10-11 00:00:00\tzug\t13910\n2017-10-12 00:00:00\tbus\t35060\n2017-10-12 00:00:00\tschiff\t71\n2017-10-12 00:00:00\ttram\t8497\n2017-10-12 00:00:00\tzug\t13899\n2017-10-13 00:00:00\tbus\t35404\n2017-10-13 00:00:00\tschiff\t71\n2017-10-13 00:00:00\ttram\t8702\n2017-10-13 00:00:00\tzug\t14031\n2017-10-14 00:00:00\tbus\t28019\n2017-10-14 00:00:00\tschiff\t69\n2017-10-14 00:00:00\ttram\t7978\n2017-10-14 00:00:00\tzug\t12850\n2017-10-15 00:00:00\tbus\t20717\n2017-10-15 00:00:00\tschiff\t80\n2017-10-15 00:00:00\ttram\t6124\n2017-10-15 00:00:00\tzug\t11898\n2017-10-16 00:00:00\tbus\t35416\n2017-10-16 00:00:00\tschiff\t71\n2017-10-16 00:00:00\ttram\t8428\n2017-10-16 00:00:00\tzug\t14076\n2017-10-17 00:00:00\tbus\t35554\n2017-10-17 00:00:00\tschiff\t71\n2017-10-17 00:00:00\ttram\t8308\n2017-10-17 00:00:00\tzug\t14096\n2017-10-18 00:00:00\t\t1\n2017-10-18 00:00:00\tbus\t35513\n2017-10-18 00:00:00\tschiff\t71\n2017-10-18 00:00:00\ttram\t8314\n2017-10-18 00:00:00\tzug\t14104\n2017-10-19 00:00:00\t\t1\n2017-10-19 00:00:00\tbus\t35480\n2017-10-19 00:00:00\tschiff\t71\n2017-10-19 00:00:00\ttram\t8395\n2017-10-19 00:00:00\tzug\t14101\n2017-10-20 00:00:00\t\t4\n2017-10-20 00:00:00\tbus\t36056\n2017-10-20 00:00:00\tschiff\t71\n2017-10-20 00:00:00\ttram\t8496\n2017-10-20 00:00:00\tzug\t14205\n2017-10-21 00:00:00\t\t2\n2017-10-21 00:00:00\tbus\t28126\n2017-10-21 00:00:00\tschiff\t69\n2017-10-21 00:00:00\ttram\t7764\n2017-10-21 00:00:00\tzug\t12919\n2017-10-22 00:00:00\tbus\t20662\n2017-10-22 00:00:00\tschiff\t80\n2017-10-22 00:00:00\ttram\t5974\n2017-10-22 00:00:00\tzug\t11944\n2017-10-23 00:00:00\t\t3\n2017-10-23 00:00:00\tbus\t35559\n2017-10-23 00:00:00\tschiff\t44\n2017-10-23 00:00:00\ttram\t8259\n2017-10-23 00:00:00\tzug\t13917\n2017-10-24 00:00:00\t\t10\n2017-10-24 00:00:00\tbus\t35574\n2017-10-24 00:00:00\tschiff\t44\n2017-10-24 00:00:00\ttram\t8300\n2017-10-24 00:00:00\tzug\t13928\n2017-10-25 00:00:00\t\t1\n2017-10-25 00:00:00\tbus\t36014\n2017-10-25 00:00:00\tschiff\t44\n2017-10-25 00:00:00\ttram\t8312\n2017-10-25 00:00:00\tzug\t13939\n2017-10-26 00:00:00\t\t2\n2017-10-26 00:00:00\tbus\t35661\n2017-10-26 00:00:00\tschiff\t46\n2017-10-26 00:00:00\ttram\t8287\n2017-10-26 00:00:00\tzug\t13901\n2017-10-27 00:00:00\t\t2\n2017-10-27 00:00:00\tbus\t36060\n2017-10-27 00:00:00\tschiff\t46\n2017-10-27 00:00:00\ttram\t8519\n2017-10-27 00:00:00\tzug\t14027\n2017-10-28 00:00:00\t\t3\n2017-10-28 00:00:00\tbus\t28322\n2017-10-28 00:00:00\tschiff\t42\n2017-10-28 00:00:00\ttram\t7675\n2017-10-28 00:00:00\tzug\t12829\n2017-10-29 00:00:00\t\t2\n2017-10-29 00:00:00\tbus\t20983\n2017-10-29 00:00:00\tschiff\t41\n2017-10-29 00:00:00\ttram\t5946\n2017-10-29 00:00:00\tzug\t11921\n2017-10-30 00:00:00\t\t2\n2017-10-30 00:00:00\tbus\t35667\n2017-10-30 00:00:00\tschiff\t44\n2017-10-30 00:00:00\ttram\t8310\n2017-10-30 00:00:00\tzug\t13670\n2017-10-31 00:00:00\tbus\t35778\n2017-10-31 00:00:00\tschiff\t44\n2017-10-31 00:00:00\ttram\t8391\n2017-10-31 00:00:00\tzug\t13587\n2017-11-01 00:00:00\tbus\t32089\n2017-11-01 00:00:00\tschiff\t44\n2017-11-01 00:00:00\ttram\t8315\n2017-11-01 00:00:00\tzug\t13612\n2017-11-02 00:00:00\tbus\t35428\n2017-11-02 00:00:00\tschiff\t46\n2017-11-02 00:00:00\ttram\t8270\n2017-11-02 00:00:00\tzug\t13788\n2017-11-03 00:00:00\tbus\t36011\n2017-11-03 00:00:00\tschiff\t46\n2017-11-03 00:00:00\ttram\t8451\n2017-11-03 00:00:00\tzug\t13905\n2017-11-04 00:00:00\tbus\t28076\n2017-11-04 00:00:00\tschiff\t42\n2017-11-04 00:00:00\ttram\t7748\n2017-11-04 00:00:00\tzug\t12707\n2017-11-05 00:00:00\tbus\t20910\n2017-11-05 00:00:00\tschiff\t41\n2017-11-05 00:00:00\ttram\t5975\n2017-11-05 00:00:00\tzug\t11813\n2017-11-06 00:00:00\tbus\t35702\n2017-11-06 00:00:00\tschiff\t44\n2017-11-06 00:00:00\ttram\t8296\n2017-11-06 00:00:00\tzug\t13731\n2017-11-07 00:00:00\tbus\t35743\n2017-11-07 00:00:00\tschiff\t44\n2017-11-07 00:00:00\ttram\t8336\n2017-11-07 00:00:00\tzug\t13724\n2017-11-08 00:00:00\tbus\t35860\n2017-11-08 00:00:00\tschiff\t44\n2017-11-08 00:00:00\ttram\t8315\n2017-11-08 00:00:00\tzug\t13761\n2017-11-09 00:00:00\tbus\t35303\n2017-11-09 00:00:00\tschiff\t46\n2017-11-09 00:00:00\ttram\t8399\n2017-11-09 00:00:00\tzug\t13745\n2017-11-10 00:00:00\tbus\t36075\n2017-11-10 00:00:00\tschiff\t46\n2017-11-10 00:00:00\ttram\t8589\n2017-11-10 00:00:00\tzug\t13861\n2017-11-11 00:00:00\tbus\t28140\n2017-11-11 00:00:00\tschiff\t42\n2017-11-11 00:00:00\ttram\t7751\n2017-11-11 00:00:00\tzug\t12858\n2017-11-12 00:00:00\tbus\t20958\n2017-11-12 00:00:00\tschiff\t42\n2017-11-12 00:00:00\ttram\t6066\n2017-11-12 00:00:00\tzug\t11918\n2017-11-13 00:00:00\tbus\t35703\n2017-11-13 00:00:00\tschiff\t45\n2017-11-13 00:00:00\ttram\t8313\n2017-11-13 00:00:00\tzug\t13734\n2017-11-14 00:00:00\tbus\t35265\n2017-11-14 00:00:00\tschiff\t45\n2017-11-14 00:00:00\ttram\t8185\n2017-11-14 00:00:00\tzug\t13751\n2017-11-15 00:00:00\tbus\t35797\n2017-11-15 00:00:00\tschiff\t45\n2017-11-15 00:00:00\ttram\t8301\n2017-11-15 00:00:00\tzug\t13788\n2017-11-16 00:00:00\tbus\t35856\n2017-11-16 00:00:00\tschiff\t47\n2017-11-16 00:00:00\ttram\t8296\n2017-11-16 00:00:00\tzug\t13780\n2017-11-17 00:00:00\tbus\t36113\n2017-11-17 00:00:00\tschiff\t46\n2017-11-17 00:00:00\ttram\t8496\n2017-11-17 00:00:00\tzug\t13856\n2017-11-18 00:00:00\tbus\t28370\n2017-11-18 00:00:00\tschiff\t42\n2017-11-18 00:00:00\ttram\t7795\n2017-11-18 00:00:00\tzug\t12772\n2017-11-19 00:00:00\tbus\t20901\n2017-11-19 00:00:00\tschiff\t42\n2017-11-19 00:00:00\ttram\t6031\n2017-11-19 00:00:00\tzug\t11843\n2017-11-20 00:00:00\tbus\t35826\n2017-11-20 00:00:00\tschiff\t40\n2017-11-20 00:00:00\ttram\t8299\n2017-11-20 00:00:00\tzug\t13721\n2017-11-21 00:00:00\tbus\t35756\n2017-11-21 00:00:00\tschiff\t40\n2017-11-21 00:00:00\ttram\t8299\n2017-11-21 00:00:00\tzug\t13730\n2017-11-22 00:00:00\tbus\t35321\n2017-11-22 00:00:00\tschiff\t40\n2017-11-22 00:00:00\ttram\t8355\n2017-11-22 00:00:00\tzug\t13709\n2017-11-23 00:00:00\tbus\t32602\n2017-11-23 00:00:00\tschiff\t42\n2017-11-23 00:00:00\ttram\t7620\n2017-11-23 00:00:00\tzug\t13736\n2017-11-24 00:00:00\tbus\t36080\n2017-11-24 00:00:00\tschiff\t42\n2017-11-24 00:00:00\ttram\t8545\n2017-11-24 00:00:00\tzug\t13779\n2017-11-25 00:00:00\tbus\t28178\n2017-11-25 00:00:00\tschiff\t38\n2017-11-25 00:00:00\ttram\t7673\n2017-11-25 00:00:00\tzug\t12617\n2017-11-26 00:00:00\tbus\t20937\n2017-11-26 00:00:00\tschiff\t38\n2017-11-26 00:00:00\ttram\t5935\n2017-11-26 00:00:00\tzug\t11761\n2017-11-27 00:00:00\tbus\t36575\n2017-11-27 00:00:00\tschiff\t40\n2017-11-27 00:00:00\ttram\t9394\n2017-11-27 00:00:00\tzug\t13772\n2017-11-28 00:00:00\tbus\t35762\n2017-11-28 00:00:00\tschiff\t40\n2017-11-28 00:00:00\ttram\t8295\n2017-11-28 00:00:00\tzug\t13765\n2017-11-29 00:00:00\tbus\t35883\n2017-11-29 00:00:00\tschiff\t40\n2017-11-29 00:00:00\ttram\t8383\n2017-11-29 00:00:00\tzug\t13730\n2017-11-30 00:00:00\tbus\t35859\n2017-11-30 00:00:00\tschiff\t42\n2017-11-30 00:00:00\ttram\t8242\n2017-11-30 00:00:00\tzug\t13750\n"}]},"apps":[],"jobName":"paragraph_1522661167106_-1439229029","id":"20180402-112607_1544683628","dateCreated":"2018-04-02T11:26:07+0200","dateStarted":"2018-04-13T13:10:28+0200","dateFinished":"2018-04-13T13:18:13+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9329"},{"text":"%md\nFrom a visual analysis of the results above, we can conclude that 17.10.2017 is a typical day for the period corresponding to the data we have.","user":"musuroi","dateUpdated":"2018-04-13T13:38:48+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>From a visual analysis of the results above, we can conclude that 17.10.2017 is a typical day for the period corresponding to the data we have.</p>\n"}]},"apps":[],"jobName":"paragraph_1522661519353_1811648533","id":"20180402-113159_1212574746","dateCreated":"2018-04-02T11:31:59+0200","dateStarted":"2018-04-13T13:38:36+0200","dateFinished":"2018-04-13T13:38:36+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9330"},{"text":"%md\n### Question I.d. (5/30)\n\nHow many trains per day stop at the Lausanne train station? Display the information by date and train type. Find the 3 most represented train types.\nNote: Use `VERKEHRSMITTEL_TEXT` to get the train type.\n\nBonus points (2.5 points): find out what happened on the two outlier days","user":"baetu","dateUpdated":"2018-04-13T15:26:32+0200","config":{"tableHide":false,"editorSetting":{"editOnDblClick":true,"language":"markdown"},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Question I.d. (5/30)</h3>\n<p>How many trains per day stop at the Lausanne train station? Display the information by date and train type. Find the 3 most represented train types.\n<br />Note: Use <code>VERKEHRSMITTEL_TEXT</code> to get the train type.</p>\n<p>Bonus points (2.5 points): find out what happened on the two outlier days</p>\n"}]},"apps":[],"jobName":"paragraph_1522235709486_1429920226","id":"20180326-145338_262324116","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-13T15:26:32+0200","dateFinished":"2018-04-13T15:26:32+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9331"},{"text":"%md\n\n### Answer\n\nWe plot the daily number of train stops in Lausanne train station, grouped by train type below:","user":"baetu","dateUpdated":"2018-04-13T15:33:00+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Answer</h3>\n<p>We plot the daily number of train stops in Lausanne train station, grouped by train type below:</p>\n"}]},"apps":[],"jobName":"paragraph_1523626030870_1200739997","id":"20180413-152710_1273898852","dateCreated":"2018-04-13T15:27:10+0200","dateStarted":"2018-04-13T15:32:40+0200","dateFinished":"2018-04-13T15:32:40+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9332"},{"text":"%jdbc(hive)\n\nSELECT COUNT(*) as NumStops, TripDate, TransportType FROM musuroi.sbb WHERE StopName == 'Lausanne'\n GROUP BY TripDate, TransportType\n ORDER BY TripDate ASC, NumStops DESC;","user":"baetu","dateUpdated":"2018-04-13T15:24:11+0200","config":{"colWidth":12,"editorMode":"ace/mode/sql","results":{"0":{"graph":{"mode":"lineChart","height":300,"optionOpen":false,"setting":{"lineChart":{}},"commonSetting":{},"keys":[{"name":"tripdate","index":1,"aggr":"sum"}],"groups":[{"name":"transporttype","index":2,"aggr":"sum"}],"values":[{"name":"numstops","index":0,"aggr":"sum"}]},"helium":{}}},"enabled":true,"editorSetting":{"editOnDblClick":false,"language":"sql"}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"numstops\ttripdate\ttransporttype\n240\t2017-09-13 00:00:00\tS\n126\t2017-09-13 00:00:00\tIR\n88\t2017-09-13 00:00:00\tRE\n41\t2017-09-13 00:00:00\tICN\n35\t2017-09-13 00:00:00\tIC\n10\t2017-09-13 00:00:00\tTGV\n9\t2017-09-13 00:00:00\tEC\n238\t2017-09-14 00:00:00\tS\n126\t2017-09-14 00:00:00\tIR\n88\t2017-09-14 00:00:00\tRE\n41\t2017-09-14 00:00:00\tICN\n35\t2017-09-14 00:00:00\tIC\n10\t2017-09-14 00:00:00\tTGV\n8\t2017-09-14 00:00:00\tEC\n239\t2017-09-15 00:00:00\tS\n124\t2017-09-15 00:00:00\tIR\n96\t2017-09-15 00:00:00\tRE\n40\t2017-09-15 00:00:00\tICN\n35\t2017-09-15 00:00:00\tIC\n9\t2017-09-15 00:00:00\tTGV\n8\t2017-09-15 00:00:00\tEC\n234\t2017-09-16 00:00:00\tS\n118\t2017-09-16 00:00:00\tIR\n86\t2017-09-16 00:00:00\tRE\n40\t2017-09-16 00:00:00\tICN\n34\t2017-09-16 00:00:00\tIC\n8\t2017-09-16 00:00:00\tEC\n8\t2017-09-16 00:00:00\tTGV\n235\t2017-09-17 00:00:00\tS\n121\t2017-09-17 00:00:00\tIR\n78\t2017-09-17 00:00:00\tRE\n41\t2017-09-17 00:00:00\tICN\n36\t2017-09-17 00:00:00\tIC\n9\t2017-09-17 00:00:00\tTGV\n8\t2017-09-17 00:00:00\tEC\n237\t2017-09-18 00:00:00\tS\n127\t2017-09-18 00:00:00\tIR\n88\t2017-09-18 00:00:00\tRE\n42\t2017-09-18 00:00:00\tICN\n38\t2017-09-18 00:00:00\tIC\n10\t2017-09-18 00:00:00\tTGV\n8\t2017-09-18 00:00:00\tEC\n240\t2017-09-19 00:00:00\tS\n126\t2017-09-19 00:00:00\tIR\n88\t2017-09-19 00:00:00\tRE\n41\t2017-09-19 00:00:00\tICN\n38\t2017-09-19 00:00:00\tIC\n10\t2017-09-19 00:00:00\tTGV\n8\t2017-09-19 00:00:00\tEC\n238\t2017-09-20 00:00:00\tS\n131\t2017-09-20 00:00:00\tIR\n88\t2017-09-20 00:00:00\tRE\n41\t2017-09-20 00:00:00\tICN\n36\t2017-09-20 00:00:00\tIC\n10\t2017-09-20 00:00:00\tTGV\n8\t2017-09-20 00:00:00\tEC\n1\t2017-09-20 00:00:00\tEXT\n237\t2017-09-21 00:00:00\tS\n128\t2017-09-21 00:00:00\tIR\n88\t2017-09-21 00:00:00\tRE\n41\t2017-09-21 00:00:00\tICN\n39\t2017-09-21 00:00:00\tIC\n10\t2017-09-21 00:00:00\tTGV\n8\t2017-09-21 00:00:00\tEC\n236\t2017-09-22 00:00:00\tS\n130\t2017-09-22 00:00:00\tIR\n96\t2017-09-22 00:00:00\tRE\n40\t2017-09-22 00:00:00\tICN\n37\t2017-09-22 00:00:00\tIC\n9\t2017-09-22 00:00:00\tTGV\n8\t2017-09-22 00:00:00\tEC\n234\t2017-09-23 00:00:00\tS\n116\t2017-09-23 00:00:00\tIR\n89\t2017-09-23 00:00:00\tRE\n43\t2017-09-23 00:00:00\tICN\n34\t2017-09-23 00:00:00\tIC\n8\t2017-09-23 00:00:00\tTGV\n8\t2017-09-23 00:00:00\tEC\n236\t2017-09-24 00:00:00\tS\n119\t2017-09-24 00:00:00\tIR\n80\t2017-09-24 00:00:00\tRE\n42\t2017-09-24 00:00:00\tICN\n36\t2017-09-24 00:00:00\tIC\n9\t2017-09-24 00:00:00\tTGV\n8\t2017-09-24 00:00:00\tEC\n237\t2017-09-25 00:00:00\tS\n128\t2017-09-25 00:00:00\tIR\n88\t2017-09-25 00:00:00\tRE\n41\t2017-09-25 00:00:00\tICN\n37\t2017-09-25 00:00:00\tIC\n10\t2017-09-25 00:00:00\tTGV\n8\t2017-09-25 00:00:00\tEC\n237\t2017-09-26 00:00:00\tS\n126\t2017-09-26 00:00:00\tIR\n88\t2017-09-26 00:00:00\tRE\n41\t2017-09-26 00:00:00\tICN\n36\t2017-09-26 00:00:00\tIC\n10\t2017-09-26 00:00:00\tTGV\n8\t2017-09-26 00:00:00\tEC\n237\t2017-09-27 00:00:00\tS\n128\t2017-09-27 00:00:00\tIR\n88\t2017-09-27 00:00:00\tRE\n41\t2017-09-27 00:00:00\tICN\n36\t2017-09-27 00:00:00\tIC\n10\t2017-09-27 00:00:00\tTGV\n8\t2017-09-27 00:00:00\tEC\n1\t2017-09-27 00:00:00\tEXT\n237\t2017-09-28 00:00:00\tS\n126\t2017-09-28 00:00:00\tIR\n88\t2017-09-28 00:00:00\tRE\n41\t2017-09-28 00:00:00\tICN\n38\t2017-09-28 00:00:00\tIC\n10\t2017-09-28 00:00:00\tTGV\n8\t2017-09-28 00:00:00\tEC\n236\t2017-09-29 00:00:00\tS\n124\t2017-09-29 00:00:00\tIR\n96\t2017-09-29 00:00:00\tRE\n41\t2017-09-29 00:00:00\tICN\n35\t2017-09-29 00:00:00\tIC\n9\t2017-09-29 00:00:00\tTGV\n8\t2017-09-29 00:00:00\tEC\n234\t2017-09-30 00:00:00\tS\n115\t2017-09-30 00:00:00\tIR\n86\t2017-09-30 00:00:00\tRE\n40\t2017-09-30 00:00:00\tICN\n35\t2017-09-30 00:00:00\tIC\n8\t2017-09-30 00:00:00\tEC\n8\t2017-09-30 00:00:00\tTGV\n235\t2017-10-01 00:00:00\tS\n117\t2017-10-01 00:00:00\tIR\n78\t2017-10-01 00:00:00\tRE\n40\t2017-10-01 00:00:00\tICN\n34\t2017-10-01 00:00:00\tIC\n9\t2017-10-01 00:00:00\tTGV\n8\t2017-10-01 00:00:00\tEC\n239\t2017-10-02 00:00:00\tS\n127\t2017-10-02 00:00:00\tIR\n88\t2017-10-02 00:00:00\tRE\n40\t2017-10-02 00:00:00\tICN\n35\t2017-10-02 00:00:00\tIC\n10\t2017-10-02 00:00:00\tTGV\n8\t2017-10-02 00:00:00\tEC\n239\t2017-10-03 00:00:00\tS\n125\t2017-10-03 00:00:00\tIR\n88\t2017-10-03 00:00:00\tRE\n41\t2017-10-03 00:00:00\tICN\n35\t2017-10-03 00:00:00\tIC\n10\t2017-10-03 00:00:00\tTGV\n8\t2017-10-03 00:00:00\tEC\n239\t2017-10-04 00:00:00\tS\n126\t2017-10-04 00:00:00\tIR\n88\t2017-10-04 00:00:00\tRE\n42\t2017-10-04 00:00:00\tICN\n35\t2017-10-04 00:00:00\tIC\n10\t2017-10-04 00:00:00\tTGV\n8\t2017-10-04 00:00:00\tEC\n1\t2017-10-04 00:00:00\tEXT\n239\t2017-10-05 00:00:00\tS\n127\t2017-10-05 00:00:00\tIR\n92\t2017-10-05 00:00:00\tRE\n41\t2017-10-05 00:00:00\tICN\n37\t2017-10-05 00:00:00\tIC\n10\t2017-10-05 00:00:00\tTGV\n8\t2017-10-05 00:00:00\tEC\n238\t2017-10-06 00:00:00\tS\n124\t2017-10-06 00:00:00\tIR\n97\t2017-10-06 00:00:00\tRE\n41\t2017-10-06 00:00:00\tICN\n34\t2017-10-06 00:00:00\tIC\n9\t2017-10-06 00:00:00\tTGV\n8\t2017-10-06 00:00:00\tEC\n235\t2017-10-07 00:00:00\tS\n116\t2017-10-07 00:00:00\tIR\n87\t2017-10-07 00:00:00\tRE\n41\t2017-10-07 00:00:00\tICN\n34\t2017-10-07 00:00:00\tIC\n8\t2017-10-07 00:00:00\tEC\n8\t2017-10-07 00:00:00\tTGV\n237\t2017-10-08 00:00:00\tS\n121\t2017-10-08 00:00:00\tIR\n86\t2017-10-08 00:00:00\tRE\n40\t2017-10-08 00:00:00\tICN\n34\t2017-10-08 00:00:00\tIC\n10\t2017-10-08 00:00:00\tTGV\n9\t2017-10-08 00:00:00\tEC\n1\t2017-10-08 00:00:00\tEXT\n239\t2017-10-09 00:00:00\tS\n130\t2017-10-09 00:00:00\tIR\n93\t2017-10-09 00:00:00\tRE\n40\t2017-10-09 00:00:00\tICN\n35\t2017-10-09 00:00:00\tIC\n10\t2017-10-09 00:00:00\tTGV\n8\t2017-10-09 00:00:00\tEC\n239\t2017-10-10 00:00:00\tS\n129\t2017-10-10 00:00:00\tIR\n93\t2017-10-10 00:00:00\tRE\n40\t2017-10-10 00:00:00\tICN\n34\t2017-10-10 00:00:00\tIC\n10\t2017-10-10 00:00:00\tTGV\n8\t2017-10-10 00:00:00\tEC\n239\t2017-10-11 00:00:00\tS\n131\t2017-10-11 00:00:00\tIR\n93\t2017-10-11 00:00:00\tRE\n40\t2017-10-11 00:00:00\tICN\n34\t2017-10-11 00:00:00\tIC\n10\t2017-10-11 00:00:00\tTGV\n9\t2017-10-11 00:00:00\tEC\n241\t2017-10-12 00:00:00\tS\n127\t2017-10-12 00:00:00\tIR\n93\t2017-10-12 00:00:00\tRE\n40\t2017-10-12 00:00:00\tICN\n34\t2017-10-12 00:00:00\tIC\n10\t2017-10-12 00:00:00\tEC\n10\t2017-10-12 00:00:00\tTGV\n238\t2017-10-13 00:00:00\tS\n129\t2017-10-13 00:00:00\tIR\n97\t2017-10-13 00:00:00\tRE\n41\t2017-10-13 00:00:00\tICN\n39\t2017-10-13 00:00:00\tIC\n9\t2017-10-13 00:00:00\tTGV\n8\t2017-10-13 00:00:00\tEC\n234\t2017-10-14 00:00:00\tS\n151\t2017-10-14 00:00:00\tIR\n91\t2017-10-14 00:00:00\tRE\n67\t2017-10-14 00:00:00\tIC\n42\t2017-10-14 00:00:00\tICN\n8\t2017-10-14 00:00:00\tTGV\n8\t2017-10-14 00:00:00\tEC\n238\t2017-10-15 00:00:00\tS\n156\t2017-10-15 00:00:00\tIR\n85\t2017-10-15 00:00:00\tRE\n68\t2017-10-15 00:00:00\tIC\n42\t2017-10-15 00:00:00\tICN\n9\t2017-10-15 00:00:00\tTGV\n8\t2017-10-15 00:00:00\tEC\n239\t2017-10-16 00:00:00\tS\n129\t2017-10-16 00:00:00\tIR\n93\t2017-10-16 00:00:00\tRE\n40\t2017-10-16 00:00:00\tICN\n34\t2017-10-16 00:00:00\tIC\n11\t2017-10-16 00:00:00\tTGV\n8\t2017-10-16 00:00:00\tEC\n245\t2017-10-17 00:00:00\tS\n128\t2017-10-17 00:00:00\tIR\n93\t2017-10-17 00:00:00\tRE\n40\t2017-10-17 00:00:00\tICN\n35\t2017-10-17 00:00:00\tIC\n11\t2017-10-17 00:00:00\tTGV\n8\t2017-10-17 00:00:00\tEC\n242\t2017-10-18 00:00:00\tS\n128\t2017-10-18 00:00:00\tIR\n94\t2017-10-18 00:00:00\tRE\n40\t2017-10-18 00:00:00\tICN\n34\t2017-10-18 00:00:00\tIC\n11\t2017-10-18 00:00:00\tTGV\n9\t2017-10-18 00:00:00\tEC\n240\t2017-10-19 00:00:00\tS\n131\t2017-10-19 00:00:00\tIR\n94\t2017-10-19 00:00:00\tRE\n41\t2017-10-19 00:00:00\tICN\n34\t2017-10-19 00:00:00\tIC\n11\t2017-10-19 00:00:00\tTGV\n8\t2017-10-19 00:00:00\tEC\n238\t2017-10-20 00:00:00\tS\n125\t2017-10-20 00:00:00\tIR\n97\t2017-10-20 00:00:00\tRE\n40\t2017-10-20 00:00:00\tICN\n34\t2017-10-20 00:00:00\tIC\n9\t2017-10-20 00:00:00\tTGV\n8\t2017-10-20 00:00:00\tEC\n235\t2017-10-21 00:00:00\tS\n116\t2017-10-21 00:00:00\tIR\n91\t2017-10-21 00:00:00\tRE\n40\t2017-10-21 00:00:00\tICN\n34\t2017-10-21 00:00:00\tIC\n8\t2017-10-21 00:00:00\tEC\n8\t2017-10-21 00:00:00\tTGV\n236\t2017-10-22 00:00:00\tS\n120\t2017-10-22 00:00:00\tIR\n83\t2017-10-22 00:00:00\tRE\n40\t2017-10-22 00:00:00\tICN\n34\t2017-10-22 00:00:00\tIC\n9\t2017-10-22 00:00:00\tTGV\n8\t2017-10-22 00:00:00\tEC\n239\t2017-10-23 00:00:00\tS\n129\t2017-10-23 00:00:00\tIR\n93\t2017-10-23 00:00:00\tRE\n40\t2017-10-23 00:00:00\tICN\n35\t2017-10-23 00:00:00\tIC\n10\t2017-10-23 00:00:00\tTGV\n8\t2017-10-23 00:00:00\tEC\n241\t2017-10-24 00:00:00\tS\n128\t2017-10-24 00:00:00\tIR\n93\t2017-10-24 00:00:00\tRE\n39\t2017-10-24 00:00:00\tICN\n35\t2017-10-24 00:00:00\tIC\n10\t2017-10-24 00:00:00\tTGV\n8\t2017-10-24 00:00:00\tEC\n241\t2017-10-25 00:00:00\tS\n127\t2017-10-25 00:00:00\tIR\n94\t2017-10-25 00:00:00\tRE\n39\t2017-10-25 00:00:00\tICN\n34\t2017-10-25 00:00:00\tIC\n10\t2017-10-25 00:00:00\tTGV\n8\t2017-10-25 00:00:00\tEC\n241\t2017-10-26 00:00:00\tS\n128\t2017-10-26 00:00:00\tIR\n93\t2017-10-26 00:00:00\tRE\n39\t2017-10-26 00:00:00\tICN\n35\t2017-10-26 00:00:00\tIC\n10\t2017-10-26 00:00:00\tTGV\n8\t2017-10-26 00:00:00\tEC\n238\t2017-10-27 00:00:00\tS\n130\t2017-10-27 00:00:00\tIR\n97\t2017-10-27 00:00:00\tRE\n41\t2017-10-27 00:00:00\tICN\n39\t2017-10-27 00:00:00\tIC\n9\t2017-10-27 00:00:00\tTGV\n8\t2017-10-27 00:00:00\tEC\n254\t2017-10-28 00:00:00\tS\n243\t2017-10-28 00:00:00\tRE\n148\t2017-10-28 00:00:00\tIR\n67\t2017-10-28 00:00:00\tIC\n43\t2017-10-28 00:00:00\tICN\n12\t2017-10-28 00:00:00\tTGV\n8\t2017-10-28 00:00:00\tEC\n257\t2017-10-29 00:00:00\tS\n235\t2017-10-29 00:00:00\tRE\n151\t2017-10-29 00:00:00\tIR\n67\t2017-10-29 00:00:00\tIC\n42\t2017-10-29 00:00:00\tICN\n13\t2017-10-29 00:00:00\tTGV\n8\t2017-10-29 00:00:00\tEC\n240\t2017-10-30 00:00:00\tS\n128\t2017-10-30 00:00:00\tIR\n93\t2017-10-30 00:00:00\tRE\n40\t2017-10-30 00:00:00\tICN\n37\t2017-10-30 00:00:00\tIC\n10\t2017-10-30 00:00:00\tTGV\n8\t2017-10-30 00:00:00\tEC\n240\t2017-10-31 00:00:00\tS\n128\t2017-10-31 00:00:00\tIR\n93\t2017-10-31 00:00:00\tRE\n40\t2017-10-31 00:00:00\tICN\n35\t2017-10-31 00:00:00\tIC\n10\t2017-10-31 00:00:00\tTGV\n8\t2017-10-31 00:00:00\tEC\n239\t2017-11-01 00:00:00\tS\n131\t2017-11-01 00:00:00\tIR\n93\t2017-11-01 00:00:00\tRE\n40\t2017-11-01 00:00:00\tICN\n35\t2017-11-01 00:00:00\tIC\n10\t2017-11-01 00:00:00\tTGV\n8\t2017-11-01 00:00:00\tEC\n239\t2017-11-02 00:00:00\tS\n127\t2017-11-02 00:00:00\tIR\n93\t2017-11-02 00:00:00\tRE\n39\t2017-11-02 00:00:00\tICN\n34\t2017-11-02 00:00:00\tIC\n10\t2017-11-02 00:00:00\tTGV\n9\t2017-11-02 00:00:00\tEC\n238\t2017-11-03 00:00:00\tS\n125\t2017-11-03 00:00:00\tIR\n97\t2017-11-03 00:00:00\tRE\n40\t2017-11-03 00:00:00\tICN\n35\t2017-11-03 00:00:00\tIC\n9\t2017-11-03 00:00:00\tTGV\n8\t2017-11-03 00:00:00\tEC\n234\t2017-11-04 00:00:00\tS\n115\t2017-11-04 00:00:00\tIR\n86\t2017-11-04 00:00:00\tRE\n40\t2017-11-04 00:00:00\tICN\n36\t2017-11-04 00:00:00\tIC\n8\t2017-11-04 00:00:00\tEC\n8\t2017-11-04 00:00:00\tTGV\n236\t2017-11-05 00:00:00\tS\n121\t2017-11-05 00:00:00\tIR\n83\t2017-11-05 00:00:00\tRE\n41\t2017-11-05 00:00:00\tICN\n36\t2017-11-05 00:00:00\tIC\n9\t2017-11-05 00:00:00\tEC\n9\t2017-11-05 00:00:00\tTGV\n239\t2017-11-06 00:00:00\tS\n128\t2017-11-06 00:00:00\tIR\n94\t2017-11-06 00:00:00\tRE\n40\t2017-11-06 00:00:00\tICN\n34\t2017-11-06 00:00:00\tIC\n10\t2017-11-06 00:00:00\tTGV\n8\t2017-11-06 00:00:00\tEC\n239\t2017-11-07 00:00:00\tS\n127\t2017-11-07 00:00:00\tIR\n93\t2017-11-07 00:00:00\tRE\n40\t2017-11-07 00:00:00\tICN\n34\t2017-11-07 00:00:00\tIC\n10\t2017-11-07 00:00:00\tTGV\n8\t2017-11-07 00:00:00\tEC\n239\t2017-11-08 00:00:00\tS\n131\t2017-11-08 00:00:00\tIR\n93\t2017-11-08 00:00:00\tRE\n40\t2017-11-08 00:00:00\tICN\n36\t2017-11-08 00:00:00\tIC\n10\t2017-11-08 00:00:00\tTGV\n8\t2017-11-08 00:00:00\tEC\n239\t2017-11-09 00:00:00\tS\n127\t2017-11-09 00:00:00\tIR\n93\t2017-11-09 00:00:00\tRE\n40\t2017-11-09 00:00:00\tICN\n34\t2017-11-09 00:00:00\tIC\n10\t2017-11-09 00:00:00\tTGV\n8\t2017-11-09 00:00:00\tEC\n238\t2017-11-10 00:00:00\tS\n124\t2017-11-10 00:00:00\tIR\n97\t2017-11-10 00:00:00\tRE\n40\t2017-11-10 00:00:00\tICN\n34\t2017-11-10 00:00:00\tIC\n9\t2017-11-10 00:00:00\tTGV\n8\t2017-11-10 00:00:00\tEC\n235\t2017-11-11 00:00:00\tS\n119\t2017-11-11 00:00:00\tIR\n86\t2017-11-11 00:00:00\tRE\n40\t2017-11-11 00:00:00\tICN\n34\t2017-11-11 00:00:00\tIC\n9\t2017-11-11 00:00:00\tEC\n8\t2017-11-11 00:00:00\tTGV\n235\t2017-11-12 00:00:00\tS\n119\t2017-11-12 00:00:00\tIR\n82\t2017-11-12 00:00:00\tRE\n40\t2017-11-12 00:00:00\tICN\n35\t2017-11-12 00:00:00\tIC\n9\t2017-11-12 00:00:00\tTGV\n8\t2017-11-12 00:00:00\tEC\n239\t2017-11-13 00:00:00\tS\n128\t2017-11-13 00:00:00\tIR\n93\t2017-11-13 00:00:00\tRE\n42\t2017-11-13 00:00:00\tICN\n35\t2017-11-13 00:00:00\tIC\n10\t2017-11-13 00:00:00\tTGV\n9\t2017-11-13 00:00:00\tEC\n239\t2017-11-14 00:00:00\tS\n127\t2017-11-14 00:00:00\tIR\n93\t2017-11-14 00:00:00\tRE\n44\t2017-11-14 00:00:00\tICN\n35\t2017-11-14 00:00:00\tIC\n11\t2017-11-14 00:00:00\tEC\n10\t2017-11-14 00:00:00\tTGV\n240\t2017-11-15 00:00:00\tS\n127\t2017-11-15 00:00:00\tIR\n93\t2017-11-15 00:00:00\tRE\n41\t2017-11-15 00:00:00\tICN\n36\t2017-11-15 00:00:00\tIC\n10\t2017-11-15 00:00:00\tTGV\n8\t2017-11-15 00:00:00\tEC\n1\t2017-11-15 00:00:00\tEXT\n239\t2017-11-16 00:00:00\tS\n127\t2017-11-16 00:00:00\tIR\n93\t2017-11-16 00:00:00\tRE\n40\t2017-11-16 00:00:00\tICN\n35\t2017-11-16 00:00:00\tIC\n10\t2017-11-16 00:00:00\tTGV\n8\t2017-11-16 00:00:00\tEC\n244\t2017-11-17 00:00:00\tS\n131\t2017-11-17 00:00:00\tIR\n97\t2017-11-17 00:00:00\tRE\n40\t2017-11-17 00:00:00\tICN\n35\t2017-11-17 00:00:00\tIC\n9\t2017-11-17 00:00:00\tTGV\n8\t2017-11-17 00:00:00\tEC\n234\t2017-11-18 00:00:00\tS\n120\t2017-11-18 00:00:00\tIR\n86\t2017-11-18 00:00:00\tRE\n40\t2017-11-18 00:00:00\tICN\n35\t2017-11-18 00:00:00\tIC\n10\t2017-11-18 00:00:00\tTGV\n8\t2017-11-18 00:00:00\tEC\n236\t2017-11-19 00:00:00\tS\n119\t2017-11-19 00:00:00\tIR\n82\t2017-11-19 00:00:00\tRE\n40\t2017-11-19 00:00:00\tICN\n37\t2017-11-19 00:00:00\tIC\n11\t2017-11-19 00:00:00\tTGV\n8\t2017-11-19 00:00:00\tEC\n241\t2017-11-20 00:00:00\tS\n128\t2017-11-20 00:00:00\tIR\n93\t2017-11-20 00:00:00\tRE\n40\t2017-11-20 00:00:00\tICN\n35\t2017-11-20 00:00:00\tIC\n10\t2017-11-20 00:00:00\tTGV\n8\t2017-11-20 00:00:00\tEC\n244\t2017-11-21 00:00:00\tS\n127\t2017-11-21 00:00:00\tIR\n94\t2017-11-21 00:00:00\tRE\n40\t2017-11-21 00:00:00\tICN\n35\t2017-11-21 00:00:00\tIC\n10\t2017-11-21 00:00:00\tTGV\n8\t2017-11-21 00:00:00\tEC\n241\t2017-11-22 00:00:00\tS\n127\t2017-11-22 00:00:00\tIR\n93\t2017-11-22 00:00:00\tRE\n40\t2017-11-22 00:00:00\tICN\n34\t2017-11-22 00:00:00\tIC\n10\t2017-11-22 00:00:00\tTGV\n8\t2017-11-22 00:00:00\tEC\n242\t2017-11-23 00:00:00\tS\n128\t2017-11-23 00:00:00\tIR\n93\t2017-11-23 00:00:00\tRE\n40\t2017-11-23 00:00:00\tICN\n34\t2017-11-23 00:00:00\tIC\n12\t2017-11-23 00:00:00\tTGV\n8\t2017-11-23 00:00:00\tEC\n238\t2017-11-24 00:00:00\tS\n125\t2017-11-24 00:00:00\tIR\n99\t2017-11-24 00:00:00\tRE\n40\t2017-11-24 00:00:00\tICN\n34\t2017-11-24 00:00:00\tIC\n9\t2017-11-24 00:00:00\tTGV\n8\t2017-11-24 00:00:00\tEC\n236\t2017-11-25 00:00:00\tS\n115\t2017-11-25 00:00:00\tIR\n90\t2017-11-25 00:00:00\tRE\n40\t2017-11-25 00:00:00\tICN\n34\t2017-11-25 00:00:00\tIC\n8\t2017-11-25 00:00:00\tEC\n8\t2017-11-25 00:00:00\tTGV\n236\t2017-11-26 00:00:00\tS\n121\t2017-11-26 00:00:00\tIR\n84\t2017-11-26 00:00:00\tRE\n41\t2017-11-26 00:00:00\tICN\n36\t2017-11-26 00:00:00\tIC\n9\t2017-11-26 00:00:00\tTGV\n8\t2017-11-26 00:00:00\tEC\n239\t2017-11-27 00:00:00\tS\n129\t2017-11-27 00:00:00\tIR\n93\t2017-11-27 00:00:00\tRE\n40\t2017-11-27 00:00:00\tICN\n35\t2017-11-27 00:00:00\tIC\n10\t2017-11-27 00:00:00\tTGV\n8\t2017-11-27 00:00:00\tEC\n240\t2017-11-28 00:00:00\tS\n129\t2017-11-28 00:00:00\tIR\n93\t2017-11-28 00:00:00\tRE\n40\t2017-11-28 00:00:00\tICN\n34\t2017-11-28 00:00:00\tIC\n10\t2017-11-28 00:00:00\tTGV\n8\t2017-11-28 00:00:00\tEC\n240\t2017-11-29 00:00:00\tS\n128\t2017-11-29 00:00:00\tIR\n93\t2017-11-29 00:00:00\tRE\n40\t2017-11-29 00:00:00\tICN\n34\t2017-11-29 00:00:00\tIC\n10\t2017-11-29 00:00:00\tTGV\n8\t2017-11-29 00:00:00\tEC\n239\t2017-11-30 00:00:00\tS\n127\t2017-11-30 00:00:00\tIR\n93\t2017-11-30 00:00:00\tRE\n40\t2017-11-30 00:00:00\tICN\n36\t2017-11-30 00:00:00\tIC\n10\t2017-11-30 00:00:00\tTGV\n8\t2017-11-30 00:00:00\tEC\n"}]},"apps":[],"jobName":"paragraph_1522235709487_1429535478","id":"20180326-145706_1402422065","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-13T13:15:07+0200","dateFinished":"2018-04-13T13:19:12+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9333"},{"text":"%md\n\nAs we can see from a visual analysis, the three most predominant train classes that stop in Lausanne are `R (Regional)`, `IR (InterRegio)`, and `RE (RegioExpress)` (<a href=\"https://en.wikipedia.org/wiki/Train_categories_in_Europe#Switzerland[3]\"> Source </a>). \n\nHowever, there are two days in which the number of trains increases significantly, especially in the case of `RE` trains: *October 28* and *October 29*. Let's see how many trips which are not ordinary had been performed during those days:","user":"baetu","dateUpdated":"2018-04-13T16:51:06+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>As we can see from a visual analysis, the three most predominant train classes that stop in Lausanne are <code>R (Regional)</code>, <code>IR (InterRegio)</code>, and <code>RE (RegioExpress)</code> (<a href=\"https: //en.wikipedia.org/wiki/Train_categories_in_Europe#Switzerland[3]\"> Source </a>).</p>\n<p>However, there are two days in which the number of trains increases significantly, especially in the case of <code>RE</code> trains: <em>October 28</em> and <em>October 29</em>. Let's see how many trips which are not ordinary had been performed during those days:</p>\n"}]},"apps":[],"jobName":"paragraph_1523630937312_-203558504","id":"20180413-164857_1995192621","dateCreated":"2018-04-13T16:48:57+0200","dateStarted":"2018-04-13T16:51:06+0200","dateFinished":"2018-04-13T16:51:06+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9334"},{"text":"%jdbc(hive)\n\nSELECT tripdate, additionaltrip, count(*) FROM musuroi.sbb WHERE TripDate in ('2017-10-28 00:00:00', '2017-10-29 00:00:00') and StopName == 'Lausanne' GROUP BY TripDate, AdditionalTrip;","user":"baetu","dateUpdated":"2018-04-16T11:59:22+0200","config":{"colWidth":12,"enabled":true,"results":{"0":{"graph":{"mode":"multiBarChart","height":300,"optionOpen":false,"setting":{"lineChart":{},"multiBarChart":{"stacked":false},"pieChart":{}},"commonSetting":{},"keys":[{"name":"tripdate","index":0,"aggr":"sum"}],"groups":[{"name":"additionaltrip","index":1,"aggr":"sum"}],"values":[{"name":"_c2","index":2,"aggr":"sum"}]},"helium":{}}},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"tripdate\tadditionaltrip\t_c2\n2017-10-28 00:00:00\tfalse\t525\n2017-10-28 00:00:00\ttrue\t250\n2017-10-29 00:00:00\tfalse\t517\n2017-10-29 00:00:00\ttrue\t256\n"}]},"apps":[],"jobName":"paragraph_1523629705496_-563875587","id":"20180413-162825_879861371","dateCreated":"2018-04-13T16:28:25+0200","dateStarted":"2018-04-13T16:42:49+0200","dateFinished":"2018-04-13T16:42:58+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9335"},{"text":"%md \n\nAs we can see, there are around 250 additional trips in each day. If we search on the Internet for possible causes for this event to happen, we realize that during those two days of weekend, there have been\nperformed works for modernizing the Renens trainstation [(Source CFF)](https://company.sbb.ch/content/dam/sbb/fr/pdf/fr_sbb-konzern/fr_bunde-und-kantone/fr_regionalverkehr/fr_westschweiz/485ren_Broschuere_bf.pdf). That is why the trains on the route Lausanne - Genève-Aéroport (which were passing by Renens gare) had to be replaced by trains traveling in `RE` regime, but with a higher frequency, i.e. every 15 minutes [(Source 20 minutes)](http://www.20min.ch/ro/news/vaud/story/Il-faudra-prendre-son-mal-en-patience-ce-week-end-21098523), which explains the higher number of `RE` trains which appear in the dataset during those two days.","user":"baetu","dateUpdated":"2018-04-13T17:56:32+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>As we can see, there are around 250 additional trips in each day. If we search on the Internet for possible causes for this event to happen, we realize that during those two days of weekend, there have been\n<br />performed works for modernizing the Renens trainstation <a href=\"https://company.sbb.ch/content/dam/sbb/fr/pdf/fr_sbb-konzern/fr_bunde-und-kantone/fr_regionalverkehr/fr_westschweiz/485ren_Broschuere_bf.pdf\">(Source CFF)</a>. That is why the trains on the route Lausanne - Genève-Aéroport (which were passing by Renens gare) had to be replaced by trains traveling in <code>RE</code> regime, but with a higher frequency, i.e. every 15 minutes <a href=\"http://www.20min.ch/ro/news/vaud/story/Il-faudra-prendre-son-mal-en-patience-ce-week-end-21098523\">(Source 20 minutes)</a>, which explains the higher number of <code>RE</code> trains which appear in the dataset during those two days.</p>\n"}]},"apps":[],"jobName":"paragraph_1523626368113_563486905","id":"20180413-153248_326566491","dateCreated":"2018-04-13T15:32:48+0200","dateStarted":"2018-04-13T17:56:22+0200","dateFinished":"2018-04-13T17:56:22+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9336"},{"text":"%md\n### Question I.e. (5/30)\n\nGet the set of IC trains you can take to go from Genève (main station) to Lausanne on a typical week day (not Saturday, not Sunday, not a bank holiday). Display the train number (`LINIEN_ID`) as well as the schedule of the trains.\nNote: do not hesitate to create intermediary tables.\nYou can use the advanced search of SBB's website to check your answer, the schedule of IC1 from Genève to Lausanne has not changed since last year.","user":"baetu","dateUpdated":"2018-04-13T18:39:51+0200","config":{"tableHide":false,"editorSetting":{"editOnDblClick":true,"language":"markdown"},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Question I.e. (5/30)</h3>\n<p>Get the set of IC trains you can take to go from Genève (main station) to Lausanne on a typical week day (not Saturday, not Sunday, not a bank holiday). Display the train number (<code>LINIEN_ID</code>) as well as the schedule of the trains.\n<br />Note: do not hesitate to create intermediary tables.\n<br />You can use the advanced search of SBB's website to check your answer, the schedule of IC1 from Genève to Lausanne has not changed since last year.</p>\n"}]},"apps":[],"jobName":"paragraph_1522235709488_1439923698","id":"20180326-150827_1333006785","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-13T18:39:51+0200","dateFinished":"2018-04-13T18:39:51+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9337"},{"text":"%md\n\n### Answer \n\nThe only bank holiday in cantons of Vaud and Genève during the period November 13th to November 30th, which was not during weekend is on September 18th [(Source)](https://www.officeholidays.com/countries/switzerland/2017.php), thus we will hardcode this value in our queries.\n\nThe approach we will follow is simple:\n\n* we construct a table with all the trains that have a stop in Genève during the respecting days\n* we construct a table with all the trains that have a stop in Lausanne during the respecting days\n* we perform the join of the two tables, by the train number and the actual date, and filtering the results with the departure time from Genève less than the arrival time in Lausanne, resulting in the table of trains that can transport people from Genève to Lausanne each typical day of week.","user":"baetu","dateUpdated":"2018-04-13T18:46:13+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Answer</h3>\n<p>The only bank holiday in cantons of Vaud and Genève during the period November 13th to November 30th, which was not during weekend is on September 18th <a href=\"https://www.officeholidays.com/countries/switzerland/2017.php\">(Source)</a>, thus we will hardcode this value in our queries.</p>\n<p>The approach we will follow is simple:</p>\n<ul>\n<li>we construct a table with all the trains that have a stop in Genève during the respecting days</li>\n<li>we construct a table with all the trains that have a stop in Lausanne during the respecting days</li>\n<li>we perform the join of the two tables, by the train number and the actual date, and filtering the results with the departure time from Genève less than the arrival time in Lausanne, resulting in the table of trains that can transport people from Genève to Lausanne each typical day of week.</li>\n</ul>\n"}]},"apps":[],"jobName":"paragraph_1523634015860_764069059","id":"20180413-174015_280355240","dateCreated":"2018-04-13T17:40:15+0200","dateStarted":"2018-04-13T18:46:02+0200","dateFinished":"2018-04-13T18:46:02+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9338"},{"text":"%jdbc(hive)\n\nDROP TABLE IF EXISTS baetu.departed_geneve;\n\nCREATE TABLE baetu.departed_geneve STORED AS parquet AS\n SELECT tripdate, lineid, stopname, departuretimescheduled from musuroi.sbb where stopname == \"Genève\" and departuretimescheduled is not null and date_format(tripdate ,'u') not in (6, 7) and tripdate != '2017-09-18 00:00:00' and transporttype=='IC';\n \nDROP TABLE IF EXISTS baetu.arrived_lausanne;\n\nCREATE TABLE baetu.arrived_lausanne STORED AS parquet AS\n SELECT tripdate, lineid, stopname, arrivaltimescheduled from musuroi.sbb where stopname == \"Lausanne\" and arrivaltimescheduled is not null and date_format(tripdate ,'u') not in (6, 7) and tripdate != '2017-09-18 00:00:00' and transporttype == 'IC';","user":"baetu","dateUpdated":"2018-04-13T19:38:10+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"}]},"apps":[],"jobName":"paragraph_1523635733580_1929507938","id":"20180413-180853_1464607513","dateCreated":"2018-04-13T18:08:53+0200","dateStarted":"2018-04-13T19:38:10+0200","dateFinished":"2018-04-13T19:38:44+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9339"},{"text":"%jdbc(hive)\n\nDROP TABLE IF EXISTS baetu.geneve_lausanne;\n\nCREATE TABLE baetu.geneve_lausanne STORED AS parquet AS\n SELECT g.tripdate AS trip_date, g.stopname AS departure_station, l.stopname AS arrival_station, g.lineid AS train_number, SUBSTR(g.departuretimescheduled, 12) AS departure_time, SUBSTR(l.arrivaltimescheduled, 12) AS arrival_time \n FROM baetu.departed_geneve g INNER JOIN baetu.arrived_lausanne l ON (g.tripdate == l.tripdate AND g.lineid == l.lineid)\n WHERE g.departuretimescheduled < l.arrivaltimescheduled\n SORT BY departure_time;","user":"baetu","dateUpdated":"2018-04-13T19:38:51+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"}]},"apps":[],"jobName":"paragraph_1523636059669_356514872","id":"20180413-181419_323961328","dateCreated":"2018-04-13T18:14:19+0200","dateStarted":"2018-04-13T19:38:51+0200","dateFinished":"2018-04-13T19:38:57+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9340"},{"text":"%md \n\nNext, we want to see for each train how many occurences it has in the dataset, in order to decide if during each day we have exactly the same trains or not:","user":"baetu","dateUpdated":"2018-04-13T19:08:16+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>Next, we want to see for each train how many occurences it has in the dataset, in order to decide if during each day we have exactly the same trains or not:</p>\n"}]},"apps":[],"jobName":"paragraph_1523639144529_2122905819","id":"20180413-190544_86220133","dateCreated":"2018-04-13T19:05:44+0200","dateStarted":"2018-04-13T19:08:06+0200","dateFinished":"2018-04-13T19:08:06+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9341"},{"text":"%jdbc(hive)\n\nSELECT train_number, departure_time, arrival_time, COUNT(*) as trains_count FROM baetu.geneve_lausanne GROUP BY train_number, departure_time, arrival_time SORT BY trains_count;","user":"baetu","dateUpdated":"2018-04-13T19:39:01+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"train_number\tdeparture_time\tarrival_time\ttrains_count\n70723\t13:42:00\t14:18:00\t1\n70727\t15:42:00\t16:18:00\t1\n70713\t08:42:00\t09:18:00\t1\n70731\t17:42:00\t18:18:00\t1\n70719\t11:42:00\t12:18:00\t2\n30037\t20:42:00\t21:18:00\t2\n70717\t10:42:00\t11:18:00\t2\n70715\t09:42:00\t10:18:00\t2\n30735\t19:42:00\t20:18:00\t2\n70735\t19:42:00\t20:18:00\t3\n30737\t20:42:00\t21:18:00\t9\n737\t20:42:00\t21:18:00\t56\n707\t05:42:00\t06:18:00\t56\n709\t06:42:00\t07:18:00\t56\n715\t09:42:00\t10:18:00\t56\n717\t10:42:00\t11:18:00\t56\n719\t11:42:00\t12:18:00\t56\n725\t14:42:00\t15:18:00\t56\n729\t16:42:00\t17:18:00\t56\n731\t17:42:00\t18:18:00\t56\n735\t19:42:00\t20:18:00\t56\n711\t07:42:00\t08:18:00\t56\n721\t12:42:00\t13:18:00\t56\n723\t13:42:00\t14:18:00\t56\n733\t18:42:00\t19:18:00\t56\n713\t08:42:00\t09:18:00\t56\n727\t15:42:00\t16:18:00\t56\n"}]},"apps":[],"jobName":"paragraph_1523638396307_-40222017","id":"20180413-185316_1487854719","dateCreated":"2018-04-13T18:53:16+0200","dateStarted":"2018-04-13T19:39:01+0200","dateFinished":"2018-04-13T19:39:07+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9342"},{"text":"%md\n\nIf we compute the number of typical days between September 13th and November 30th, it will result 56, so most of the trains actually traveled every typical day.\n\nWe can deduce that there are several trains which happened to travel only during a strict subset of the days considered. Therefore, we will not include those trains in the set of trains which usually can transport people from Geneve to Lausanne in a normal day. We will show below the list of trains someone can take to go from Geneve to Lausanne on a typical day:","user":"baetu","dateUpdated":"2018-04-13T19:26:46+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>If we compute the number of typical days between September 13th and November 30th, it will result 56, so most of the trains actually traveled every typical day.</p>\n<p>We can deduce that there are several trains which happened to travel only during a strict subset of the days considered. Therefore, we will not include those trains in the set of trains which usually can transport people from Geneve to Lausanne in a normal day. We will show below the list of trains someone can take to go from Geneve to Lausanne on a typical day:</p>\n"}]},"apps":[],"jobName":"paragraph_1523639306786_312909075","id":"20180413-190826_383523911","dateCreated":"2018-04-13T19:08:26+0200","dateStarted":"2018-04-13T19:26:46+0200","dateFinished":"2018-04-13T19:26:46+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9343"},{"text":"%jdbc(hive)\n\nDROP TABLE IF EXISTS baetu.train_frequencies;\n\nCREATE TABLE baetu.train_frequencies STORED AS parquet AS\n SELECT train_number, departure_time, arrival_time, COUNT(*) AS trains_count FROM baetu.geneve_lausanne GROUP BY train_number, departure_time, arrival_time;\n \nSELECT train_number, departure_time, arrival_time FROM baetu.train_frequencies WHERE trains_count == 56 ORDER BY departure_time ASC;","user":"baetu","dateUpdated":"2018-04-13T19:41:19+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TABLE","data":"train_number\tdeparture_time\tarrival_time\n707\t05:42:00\t06:18:00\n709\t06:42:00\t07:18:00\n711\t07:42:00\t08:18:00\n713\t08:42:00\t09:18:00\n715\t09:42:00\t10:18:00\n717\t10:42:00\t11:18:00\n719\t11:42:00\t12:18:00\n721\t12:42:00\t13:18:00\n723\t13:42:00\t14:18:00\n725\t14:42:00\t15:18:00\n727\t15:42:00\t16:18:00\n729\t16:42:00\t17:18:00\n731\t17:42:00\t18:18:00\n733\t18:42:00\t19:18:00\n735\t19:42:00\t20:18:00\n737\t20:42:00\t21:18:00\n"}]},"apps":[],"jobName":"paragraph_1523640409492_-445556355","id":"20180413-192649_1671530049","dateCreated":"2018-04-13T19:26:49+0200","dateStarted":"2018-04-13T19:41:19+0200","dateFinished":"2018-04-13T19:41:29+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9344"},{"text":"%md\n### Question I.f. (5/30)\n\n\nLet's focus on train IC 711 and see if it tends to be on time.\n\ni) Display the distribution of delays for the IC 711 train at the Lausanne train station. Display the information as a histogram with bins of size equal to one minute. How many trains have arrived more than 5 minutes late?\nNote: when the train is ahead of schedule, count this as a delay of 0.\n\nii) Compute the 50th and 75th percentiles of delays for IC 702, 704, ..., 728, 730 (15 trains total) at Genève main station. Which trains are the most disrupted?","user":"baetu","dateUpdated":"2018-04-13T20:27:49+0200","config":{"tableHide":false,"editorSetting":{"editOnDblClick":true,"language":"markdown"},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Question I.f. (5/30)</h3>\n<p>Let's focus on train IC 711 and see if it tends to be on time.</p>\n<p>i) Display the distribution of delays for the IC 711 train at the Lausanne train station. Display the information as a histogram with bins of size equal to one minute. How many trains have arrived more than 5 minutes late?\n<br />Note: when the train is ahead of schedule, count this as a delay of 0.</p>\n<p>ii) Compute the 50th and 75th percentiles of delays for IC 702, 704, …, 728, 730 (15 trains total) at Genève main station. Which trains are the most disrupted?</p>\n"}]},"apps":[],"jobName":"paragraph_1522235709490_1440693196","id":"20180326-174330_288141471","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-13T20:27:49+0200","dateFinished":"2018-04-13T20:27:49+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9345"},{"text":"%md\n\n### Answer \n\n#### Subpoint i)\n\nFirst we create a table with the amount of minutes of delay of IC 711 at the Lausanne train station:","user":"baetu","dateUpdated":"2018-04-13T20:17:36+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Answer</h3>\n<h4>Subpoint i)</h4>\n<p>First we create a table with the amount of minutes of delay of IC 711 at the Lausanne train station:</p>\n"}]},"apps":[],"jobName":"paragraph_1523641421009_1585600963","id":"20180413-194341_306105472","dateCreated":"2018-04-13T19:43:41+0200","dateStarted":"2018-04-13T20:16:50+0200","dateFinished":"2018-04-13T20:16:50+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9346"},{"text":"%jdbc(hive)\n\nDROP TABLE IF EXISTS baetu.train_delays;\n\nCREATE TABLE baetu.train_delays STORED AS parquet AS\n SELECT IF(unix_timestamp(departuretimeactual) - unix_timestamp(departuretimescheduled) < 0, 0, int((unix_timestamp(departuretimeactual) - unix_timestamp(departuretimescheduled)) / 60)) AS minutes_late\n FROM musuroi.sbb \n WHERE transporttype == 'IC' AND lineid == '711' AND stopname == 'Lausanne' AND departuretimescheduled is not null AND departuretimeactual is not null;","user":"baetu","dateUpdated":"2018-04-13T20:12:58+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"}]},"apps":[],"jobName":"paragraph_1523642634989_-1570222839","id":"20180413-200354_1755723334","dateCreated":"2018-04-13T20:03:54+0200","dateStarted":"2018-04-13T20:12:47+0200","dateFinished":"2018-04-13T20:13:44+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9347"},{"text":"%md\n\nThen, we simply have to group by the minutes of delay and count the contents of each bin:","user":"baetu","dateUpdated":"2018-04-13T20:19:04+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>Then, we simply have to group by the minutes of delay and count the contents of each bin:</p>\n"}]},"apps":[],"jobName":"paragraph_1523643419194_-587058583","id":"20180413-201659_2145960482","dateCreated":"2018-04-13T20:16:59+0200","dateStarted":"2018-04-13T20:19:04+0200","dateFinished":"2018-04-13T20:19:04+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9348"},{"text":"%jdbc(hive)\n\nSELECT minutes_late, COUNT(*) FROM baetu.train_delays GROUP BY minutes_late;","user":"baetu","dateUpdated":"2018-04-13T20:22:00+0200","config":{"colWidth":12,"enabled":true,"results":{"0":{"graph":{"mode":"multiBarChart","height":300,"optionOpen":false},"helium":{}}},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"minutes_late\t_c1\n0\t13\n1\t31\n2\t13\n3\t8\n4\t3\n5\t3\n6\t2\n7\t1\n11\t1\n"}]},"apps":[],"jobName":"paragraph_1523643237352_-1747120845","id":"20180413-201357_1097616406","dateCreated":"2018-04-13T20:13:57+0200","dateStarted":"2018-04-13T20:19:55+0200","dateFinished":"2018-04-13T20:20:53+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9349"},{"text":"%md \n\nFrom the previous bins, we can see that only 4 trains arrived more than 5 minutes late (2 trains 6 minutes late, 1 train 7 minutes late and 1 train 11 minutes late).\n\n#### Subpoint ii)\n\nFirst, we create a table with the delays for all the specified trains:","user":"baetu","dateUpdated":"2018-04-13T20:26:52+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>From the previous bins, we can see that only 4 trains arrived more than 5 minutes late (2 trains 6 minutes late, 1 train 7 minutes late and 1 train 11 minutes late).</p>\n<h4>Subpoint ii)</h4>\n<p>First, we create a table with the delays for all the specified trains:</p>\n"}]},"apps":[],"jobName":"paragraph_1523643642789_-2043242870","id":"20180413-202042_1387846324","dateCreated":"2018-04-13T20:20:42+0200","dateStarted":"2018-04-13T20:26:52+0200","dateFinished":"2018-04-13T20:26:52+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9350"},{"text":"%jdbc(hive)\n\nDROP TABLE IF EXISTS baetu.train_delays_geneve;\n\nCREATE TABLE baetu.train_delays_geneve STORED AS parquet AS\n SELECT IF(unix_timestamp(departuretimeactual) - unix_timestamp(departuretimescheduled) < 0, 0, int((unix_timestamp(departuretimeactual) - unix_timestamp(departuretimescheduled)) / 60)) AS minutes_late, lineid\n FROM musuroi.sbb \n WHERE transporttype == 'IC' AND 701 < lineid AND lineid < 731 AND pmod(lineid, 2) == 0 AND stopname == 'Genève' AND departuretimescheduled is not null AND departuretimeactual is not null;","user":"baetu","dateUpdated":"2018-04-13T20:40:08+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"}]},"apps":[],"jobName":"paragraph_1523644015566_62074632","id":"20180413-202655_415765464","dateCreated":"2018-04-13T20:26:55+0200","dateStarted":"2018-04-13T20:40:08+0200","dateFinished":"2018-04-13T20:42:34+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9351"},{"text":"%jdbc(hive)\n\nSELECT * FROM baetu.train_delays_geneve limit 20;","user":"baetu","dateUpdated":"2018-04-13T20:58:17+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"train_delays_geneve.minutes_late\ttrain_delays_geneve.lineid\n0\t702\n10\t704\n1\t706\n2\t708\n2\t710\n5\t712\n7\t714\n2\t716\n1\t718\n2\t720\n5\t722\n5\t724\n4\t728\n2\t702\n8\t704\n0\t706\n2\t708\n10\t710\n1\t712\n1\t714\n"}]},"apps":[],"jobName":"paragraph_1523644645652_287224426","id":"20180413-203725_1528394015","dateCreated":"2018-04-13T20:37:25+0200","dateStarted":"2018-04-13T20:58:17+0200","dateFinished":"2018-04-13T20:58:17+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9352"},{"text":"%md\n\nThen, we extract the 50th and 75th percentiles of delays for each train:","user":"baetu","dateUpdated":"2018-04-13T20:33:51+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>Then, we extract the 50th and 75th percentiles of delays for each train:</p>\n"}]},"apps":[],"jobName":"paragraph_1523644391288_-197926969","id":"20180413-203311_1883058968","dateCreated":"2018-04-13T20:33:11+0200","dateStarted":"2018-04-13T20:33:51+0200","dateFinished":"2018-04-13T20:33:51+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9353"},{"text":"%jdbc(hive)\n\nSELECT int(percentile(minutes_late, 0.5)) AS percentile_50, int(percentile(minutes_late, 0.75)) AS percentile_75, lineid \n FROM baetu.train_delays_geneve \n GROUP BY lineid;","user":"baetu","dateUpdated":"2018-04-13T21:04:26+0200","config":{"colWidth":12,"enabled":true,"results":{"0":{"graph":{"mode":"multiBarChart","height":300,"optionOpen":false,"setting":{"pieChart":{},"multiBarChart":{}},"commonSetting":{},"keys":[{"name":"lineid","index":2,"aggr":"sum"}],"groups":[],"values":[{"name":"percentile_50","index":0,"aggr":"sum"},{"name":"percentile_75","index":1,"aggr":"sum"}]},"helium":{}}},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"percentile_50\tpercentile_75\tlineid\n1\t2\t702\n2\t4\t704\n2\t5\t706\n1\t2\t708\n1\t3\t710\n1\t3\t712\n1\t2\t714\n1\t2\t716\n1\t2\t718\n0\t1\t720\n1\t2\t722\n2\t3\t724\n1\t3\t726\n2\t4\t728\n2\t4\t730\n"}]},"apps":[],"jobName":"paragraph_1523644436179_-353472794","id":"20180413-203356_342701364","dateCreated":"2018-04-13T20:33:56+0200","dateStarted":"2018-04-13T21:01:02+0200","dateFinished":"2018-04-13T21:02:08+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9354"},{"text":"%md\n\nBy looking at the previous plot, we can deduce than the most distrupted trains are the IC 706, with the median delay of 2 minutes and the 75th percentile delay of 5 minutes.","user":"baetu","dateUpdated":"2018-04-13T21:06:08+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>By looking at the previous plot, we can deduce than the most distrupted trains are the IC 706, with the median delay of 2 minutes and the 75th percentile delay of 5 minutes.</p>\n"}]},"apps":[],"jobName":"paragraph_1523646308093_-160462951","id":"20180413-210508_1569230586","dateCreated":"2018-04-13T21:05:08+0200","dateStarted":"2018-04-13T21:06:08+0200","dateFinished":"2018-04-13T21:06:08+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9355"},{"text":"%md\n## Part II - Twitter Dataset (30 points)\n\nData source: <https://archive.org/details/twitterstream?sort=-publicdate>\n\nIn this part, you will leverage Hive to extract the hashtags from the source data, and then perform light exploration of the prepared data.\n\n### Dataset Description\n\nFormat: the dataset is presented as a collection of textfiles containing one JSON document per line. The data is organized in a hierarchy of folders, with one file per minute, ranging from 01.04.2016 until 31.07.2016. The textfiles have been compressed using bzip2.\n\nLocation: you can find the data on HDFS at the path `/datasets/twitter`.\n\nRelevant fields:\n\n- `created_at`, `timestamp_ms`: the first is a human-readable string representation of when the tweet was posted. The latter represents the same instant as a timestamp in seconds since UNIX epoch.\n- `lang`: the language of the tweet content\n- `entities`: parsed entities from the tweet, e.g. hashtags, user mentions, URLs\n\nIn the repository, you will also find the file `tweet-example.json` which contains a single tweet, nicely printed.\n\n### Disclaimer\n\nThis dataset contains unfiltered data from Twitter. As such, you may be exposed to tweets/hashtags containing vulgarities, references to sexual acts, drug usage, etc.","dateUpdated":"2018-03-28T13:15:09+0200","config":{"tableHide":false,"editorSetting":{"editOnDblClick":true,"language":"markdown"},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h2>Part II - Twitter Dataset (30 points)</h2>\n<p>Data source: <a href=\"https://archive.org/details/twitterstream?sort=-publicdate\">https://archive.org/details/twitterstream?sort=-publicdate</a></p>\n<p>In this part, you will leverage Hive to extract the hashtags from the source data, and then perform light exploration of the prepared data.</p>\n<h3>Dataset Description</h3>\n<p>Format: the dataset is presented as a collection of textfiles containing one JSON document per line. The data is organized in a hierarchy of folders, with one file per minute, ranging from 01.04.2016 until 31.07.2016. The textfiles have been compressed using bzip2.</p>\n<p>Location: you can find the data on HDFS at the path <code>/datasets/twitter</code>.</p>\n<p>Relevant fields:</p>\n<ul>\n<li><code>created_at</code>, <code>timestamp_ms</code>: the first is a human-readable string representation of when the tweet was posted. The latter represents the same instant as a timestamp in seconds since UNIX epoch.</li>\n<li><code>lang</code>: the language of the tweet content</li>\n<li><code>entities</code>: parsed entities from the tweet, e.g. hashtags, user mentions, URLs</li>\n</ul>\n<p>In the repository, you will also find the file <code>tweet-example.json</code> which contains a single tweet, nicely printed.</p>\n<h3>Disclaimer</h3>\n<p>This dataset contains unfiltered data from Twitter. As such, you may be exposed to tweets/hashtags containing vulgarities, references to sexual acts, drug usage, etc.</p>\n"}]},"apps":[],"jobName":"paragraph_1522235709493_1437999953","id":"20180326-182640_429144832","dateCreated":"2018-03-28T13:15:09+0200","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:9356"},{"text":"%md\n### Question II.a. (5/30)\n\nFirst, create an external table named `<your_gaspar_name>.twitter` for this dataset. \nThen create table `<your_gaspar_name>.twitter_11_06_2016` which contains data only for the day of 11.06.2017.\n\n### Answer\n\nFirst, we will create two temporary tables `baetu.twitter_raw` and `baetu.twitter_11_06_2016_raw` with the raw data: first from the full dataset and the latter containing only the data for the day 11.06.2017:","user":"baetu","dateUpdated":"2018-04-12T18:16:46+0200","config":{"tableHide":false,"editorSetting":{"editOnDblClick":true,"language":"markdown"},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Question II.a. (5/30)</h3>\n<p>First, create an external table named <code><your_gaspar_name>.twitter</code> for this dataset.\n<br />Then create table <code><your_gaspar_name>.twitter_11_06_2016</code> which contains data only for the day of 11.06.2017.</p>\n<h3>Answer</h3>\n<p>First, we will create two temporary tables <code>baetu.twitter_raw</code> and <code>baetu.twitter_11_06_2016_raw</code> with the raw data: first from the full dataset and the latter containing only the data for the day 11.06.2017:</p>\n"}]},"apps":[],"jobName":"paragraph_1522235709494_1439154200","id":"20180326-195201_1927949570","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-12T18:16:46+0200","dateFinished":"2018-04-12T18:16:46+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9357"},{"text":"%jdbc(hive)\n\ncreate temporary external table if not exists baetu.twitter_raw(json string)\n stored as textfile\n location '/datasets/twitter/';\n \ncreate temporary external table if not exists baetu.twitter_11_06_2016_raw(json string)\n stored as textfile\n location '/datasets/twitter/2016/06/11';","user":"baetu","dateUpdated":"2018-04-13T15:21:41+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql","editorHide":false,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"}]},"apps":[],"jobName":"paragraph_1523518517435_-163507776","id":"20180412-093517_879729664","dateCreated":"2018-04-12T09:35:17+0200","dateStarted":"2018-04-13T15:21:41+0200","dateFinished":"2018-04-13T15:21:41+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9358"},{"text":"%md \n\nNext, we create the desired tables from the ones previously created, using the `get_json_object` function to extract information from the JSONs:","user":"baetu","dateUpdated":"2018-04-12T18:12:19+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>Next, we create the desired tables from the ones previously created, using the <code>get_json_object</code> function to extract information from the JSONs:</p>\n"}]},"apps":[],"jobName":"paragraph_1523549396809_1104300328","id":"20180412-180956_55578907","dateCreated":"2018-04-12T18:09:56+0200","dateStarted":"2018-04-12T18:12:19+0200","dateFinished":"2018-04-12T18:12:19+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9359"},{"text":"%jdbc(hive)\n\nDROP TABLE IF EXISTS baetu.twitter_11_06_2016;\n\nCREATE TABLE baetu.twitter_11_06_2016 STORED AS parquet AS\n SELECT get_json_object(json, '$.created_at') created_at,\n get_json_object(json, '$.timestamp_ms') timestamp_ms,\n get_json_object(json, '$.lang') lang,\n get_json_object(json, '$.entities') entities\n FROM baetu.twitter_11_06_2016_raw;","user":"baetu","dateUpdated":"2018-04-14T14:15:02+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"}]},"apps":[],"jobName":"paragraph_1523549559355_198033302","id":"20180412-181239_1879556620","dateCreated":"2018-04-12T18:12:39+0200","dateStarted":"2018-04-13T13:01:57+0200","dateFinished":"2018-04-13T13:04:32+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9360"},{"text":"%jdbc(hive)\n\n--DROP TABLE IF EXISTS baetu.twitter;\n\nCREATE TABLE IF NOT EXISTS baetu.twitter STORED AS parquet AS\nSELECT get_json_object(json, '$.created_at') created_at,\n get_json_object(json, '$.timestamp_ms') timestamp_ms,\n get_json_object(json, '$.lang') lang,\n get_json_object(json, '$.entities') entities\nFROM baetu.twitter_raw;","user":"baetu","dateUpdated":"2018-04-14T12:30:17+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"}]},"apps":[],"jobName":"paragraph_1523548323151_-1840327500","id":"20180412-175203_569017808","dateCreated":"2018-04-12T17:52:03+0200","dateStarted":"2018-04-13T15:20:57+0200","dateFinished":"2018-04-13T16:30:31+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9361"},{"text":"%md\n\nFinally, we want to check if everything looks as expected in the tables:","user":"baetu","dateUpdated":"2018-04-12T18:15:59+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>Finally, we want to check if everything looks as expected in the tables:</p>\n"}]},"apps":[],"jobName":"paragraph_1523549741683_-556014890","id":"20180412-181541_528800892","dateCreated":"2018-04-12T18:15:41+0200","dateStarted":"2018-04-12T18:15:59+0200","dateFinished":"2018-04-12T18:15:59+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9362"},{"text":"%jdbc(hive)\nselect * from baetu.twitter_11_06_2016 limit 50;","user":"baetu","dateUpdated":"2018-04-12T19:04:57+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"twitter_11_06_2016.created_at\ttwitter_11_06_2016.timestamp_ms\ttwitter_11_06_2016.lang\ttwitter_11_06_2016.entities\nSat Jun 11 06:30:00 +0000 2016\t1465626600657\ten\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"blvnkheart\",\"name\":\".\",\"id\":722275385258278913,\"id_str\":\"722275385258278913\",\"indices\":[3,14]}],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600661\ten\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"blvnkheart\",\"name\":\".\",\"id\":722275385258278913,\"id_str\":\"722275385258278913\",\"indices\":[3,14]}],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600664\ten\t{\"hashtags\":[],\"urls\":[{\"url\":\"https://t.co/IX2XkGJJiw\",\"expanded_url\":\"http://fb.me/1gaS6xEmT\",\"display_url\":\"fb.me/1gaS6xEmT\",\"indices\":[120,143]}],\"user_mentions\":[],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600662\tja\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"evance3205\",\"name\":\"ã‚ŠãŠ@6/12ストレイテナーBe0\",\"id\":248151154,\"id_str\":\"248151154\",\"indices\":[0,11]}],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600661\tja\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600660\tes\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600663\tth\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"BUBBLETEAHOLIC\",\"name\":\"`ë²„ë¸”í‹°ì„¸í›„ë‹ˆâ™¡ì• ê¸° ×0412\",\"id\":1065730652,\"id_str\":\"1065730652\",\"indices\":[3,18]}],\"symbols\":[]}\nnull\tnull\tnull\tnull\nSat Jun 11 06:30:00 +0000 2016\t1465626600658\ten\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"h4iasi\",\"name\":\"blu\",\"id\":304867544,\"id_str\":\"304867544\",\"indices\":[1,8]}],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600664\ten\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600657\ten\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"jwilphotos\",\"name\":\"Jerren Willis\",\"id\":20498232,\"id_str\":\"20498232\",\"indices\":[0,11]},{\"screen_name\":\"kellyndyann\",\"name\":\"Kellyn\",\"id\":201943660,\"id_str\":\"201943660\",\"indices\":[12,24]}],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600666\ten\t{\"hashtags\":[{\"text\":\"StopBoknal\",\"indices\":[17,28]},{\"text\":\"EndDogMeat\",\"indices\":[83,94]}],\"urls\":[{\"url\":\"https://t.co/W30Um3pjI6\",\"expanded_url\":\"http://www.ambafrance-kr.org/\",\"display_url\":\"ambafrance-kr.org\",\"indices\":[59,82]},{\"url\":\"https://t.co/wUlDkDhJv8\",\"expanded_url\":\"https://twitter.com/mohwpr/status/739074102565036032\",\"display_url\":\"twitter.com/mohwpr/status/…\",\"indices\":[139,140]}],\"user_mentions\":[{\"screen_name\":\"GoLLLion\",\"name\":\"#INDIGNEZ-VOUS\",\"id\":2977588989,\"id_str\":\"2977588989\",\"indices\":[3,12]},{\"screen_name\":\"gouvernementFR\",\"name\":\"Gouvernement\",\"id\":951480480,\"id_str\":\"951480480\",\"indices\":[95,110]},{\"screen_name\":\"fhollande\",\"name\":\"François Hollande\",\"id\":18814998,\"id_str\":\"18814998\",\"indices\":[111,121]},{\"screen_name\":\"Elysee\",\"name\":\"Élysée\",\"id\":16717501,\"id_str\":\"16717501\",\"indices\":[122,129]}],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600660\ten\t{\"hashtags\":[],\"urls\":[{\"url\":\"https://t.co/EHgJXQ791T\",\"expanded_url\":\"https://twitter.com/newton_mikayla/status/741516712202231810\",\"display_url\":\"twitter.com/newton_mikayla…\",\"indices\":[62,85]}],\"user_mentions\":[{\"screen_name\":\"_ariannalac\",\"name\":\"Ms. Third Ward\",\"id\":1482777674,\"id_str\":\"1482777674\",\"indices\":[3,15]}],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600662\tja\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"dsrknown\",\"name\":\"大森林(ﾅｯｺï¾8/13土 æ±ï¼35b)\",\"id\":100233597,\"id_str\":\"100233597\",\"indices\":[3,12]}],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600664\tja\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[],\"symbols\":[]}\nnull\tnull\tnull\tnull\nSat Jun 11 06:30:00 +0000 2016\t1465626600662\tar\t{\"hashtags\":[],\"urls\":[{\"url\":\"https://t.co/R1vk6hRJxJ\",\"expanded_url\":\"http://du3a.org\",\"display_url\":\"du3a.org\",\"indices\":[77,100]}],\"user_mentions\":[],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600664\tja\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"ariyoshihiroiki\",\"name\":\"有å‰å¼˜è¡Œ\",\"id\":114446939,\"id_str\":\"114446939\",\"indices\":[3,19]}],\"symbols\":[],\"media\":[{\"id\":741513349284777984,\"id_str\":\"741513349284777984\",\"indices\":[25,48],\"media_url\":\"http://pbs.twimg.com/media/CkpidmRUUAAMhPc.jpg\",\"media_url_https\":\"https://pbs.twimg.com/media/CkpidmRUUAAMhPc.jpg\",\"url\":\"https://t.co/Iwjl7JCz49\",\"display_url\":\"pic.twitter.com/Iwjl7JCz49\",\"expanded_url\":\"http://twitter.com/ariyoshihiroiki/status/741513358717788160/photo/1\",\"type\":\"photo\",\"sizes\":{\"medium\":{\"w\":681,\"h\":908,\"resize\":\"fit\"},\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"large\":{\"w\":681,\"h\":908,\"resize\":\"fit\"},\"small\":{\"w\":510,\"h\":680,\"resize\":\"fit\"}},\"source_status_id\":741513358717788160,\"source_status_id_str\":\"741513358717788160\",\"source_user_id\":114446939,\"source_user_id_str\":\"114446939\"}]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600657\ten\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"JaiWaetford\",\"name\":\"Jai Waetford\",\"id\":1170311498,\"id_str\":\"1170311498\",\"indices\":[0,12]}],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600661\tja\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600665\tzh\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"osoul1214\",\"name\":\"陵言語\",\"id\":216505360,\"id_str\":\"216505360\",\"indices\":[0,10]}],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600663\tzh\t{\"hashtags\":[],\"urls\":[{\"url\":\"https://t.co/HZOxR6yNdH\",\"expanded_url\":\"http://fb.me/34veheLCZ\",\"display_url\":\"fb.me/34veheLCZ\",\"indices\":[47,70]}],\"user_mentions\":[],\"symbols\":[]}\nnull\tnull\tnull\tnull\nSat Jun 11 06:30:00 +0000 2016\t1465626600658\tja\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"ohBWg25d0Il7t8M\",\"name\":\"ãµã¿ã‚„☺︎\",\"id\":3230353506,\"id_str\":\"3230353506\",\"indices\":[3,19]}],\"symbols\":[],\"media\":[{\"id\":741507305397616640,\"id_str\":\"741507305397616640\",\"indices\":[42,65],\"media_url\":\"http://pbs.twimg.com/ext_tw_video_thumb/741507305397616640/pu/img/fcsofqhZ5P4XsUtn.jpg\",\"media_url_https\":\"https://pbs.twimg.com/ext_tw_video_thumb/741507305397616640/pu/img/fcsofqhZ5P4XsUtn.jpg\",\"url\":\"https://t.co/2TSjiB1bUM\",\"display_url\":\"pic.twitter.com/2TSjiB1bUM\",\"expanded_url\":\"http://twitter.com/ohBWg25d0Il7t8M/status/741507489296834560/video/1\",\"type\":\"photo\",\"sizes\":{\"small\":{\"w\":340,\"h\":604,\"resize\":\"fit\"},\"large\":{\"w\":720,\"h\":1280,\"resize\":\"fit\"},\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"medium\":{\"w\":600,\"h\":1067,\"resize\":\"fit\"}},\"source_status_id\":741507489296834560,\"source_status_id_str\":\"741507489296834560\",\"source_user_id\":3230353506,\"source_user_id_str\":\"3230353506\"}]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600665\ten\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"ddill_\",\"name\":\"dd\",\"id\":2477382324,\"id_str\":\"2477382324\",\"indices\":[0,7]}],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600662\tko\t{\"hashtags\":[],\"urls\":[{\"url\":\"https://t.co/2XyCQuPY6h\",\"expanded_url\":\"http://blog.naver.com/regularshowmabs\",\"display_url\":\"blog.naver.com/regularshowmabs\",\"indices\":[118,141]}],\"user_mentions\":[{\"screen_name\":\"RegularShowMabs\",\"name\":\"ë ˆê·¤ëŸ¬ì‡¼íŽ˜ì–´\",\"id\":731331046378115074,\"id_str\":\"731331046378115074\",\"indices\":[3,19]}],\"symbols\":[],\"media\":[{\"id\":741516122902519809,\"id_str\":\"741516122902519809\",\"indices\":[143,144],\"media_url\":\"http://pbs.twimg.com/tweet_video_thumb/Ckpk_CzUYAEfsE0.jpg\",\"media_url_https\":\"https://pbs.twimg.com/tweet_video_thumb/Ckpk_CzUYAEfsE0.jpg\",\"url\":\"https://t.co/Ey7noUcQkh\",\"display_url\":\"pic.twitter.com/Ey7noUcQkh\",\"expanded_url\":\"http://twitter.com/RegularShowMabs/status/741517666901331969/photo/1\",\"type\":\"photo\",\"sizes\":{\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"small\":{\"w\":340,\"h\":190,\"resize\":\"fit\"},\"medium\":{\"w\":500,\"h\":280,\"resize\":\"fit\"},\"large\":{\"w\":500,\"h\":280,\"resize\":\"fit\"}},\"source_status_id\":741517666901331969,\"source_status_id_str\":\"741517666901331969\",\"source_user_id\":731331046378115074,\"source_user_id_str\":\"731331046378115074\"}]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600660\tko\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"spsp_A\",\"name\":\"세필\",\"id\":841488350,\"id_str\":\"841488350\",\"indices\":[3,10]}],\"symbols\":[],\"media\":[{\"id\":740524519580803076,\"id_str\":\"740524519580803076\",\"indices\":[46,69],\"media_url\":\"http://pbs.twimg.com/media/CkbfIIkUoAQUea4.jpg\",\"media_url_https\":\"https://pbs.twimg.com/media/CkbfIIkUoAQUea4.jpg\",\"url\":\"https://t.co/tagr3RsnvZ\",\"display_url\":\"pic.twitter.com/tagr3RsnvZ\",\"expanded_url\":\"http://twitter.com/spsp_A/status/740524522491645953/photo/1\",\"type\":\"photo\",\"sizes\":{\"small\":{\"w\":540,\"h\":374,\"resize\":\"fit\"},\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"medium\":{\"w\":540,\"h\":374,\"resize\":\"fit\"},\"large\":{\"w\":540,\"h\":374,\"resize\":\"fit\"}},\"source_status_id\":740524522491645953,\"source_status_id_str\":\"740524522491645953\",\"source_user_id\":841488350,\"source_user_id_str\":\"841488350\"}]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600662\ten\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600660\ten\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"mkenzie4\",\"name\":\"kenz.\",\"id\":517424609,\"id_str\":\"517424609\",\"indices\":[0,9]},{\"screen_name\":\"apuschi\",\"name\":\"puschpop\",\"id\":419290742,\"id_str\":\"419290742\",\"indices\":[10,18]}],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600661\tja\t{\"hashtags\":[],\"urls\":[{\"url\":\"https://t.co/5FG9FhkEIz\",\"expanded_url\":\"http://www.ironmannet.com/buynyushi/juniorhigh/2017/13/book/atomi_gakuen/koekyo.htm\",\"display_url\":\"ironmannet.com/buynyushi/juni…\",\"indices\":[54,77]}],\"user_mentions\":[{\"screen_name\":\"ironmannet\",\"name\":\"アイアンマンãƒãƒƒãƒˆï½œé‰„人ãƒãƒƒãƒˆ\",\"id\":414600829,\"id_str\":\"414600829\",\"indices\":[78,89]}],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600661\ten\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"Mourinholic\",\"name\":\"Mourinholic\",\"id\":159412087,\"id_str\":\"159412087\",\"indices\":[3,15]}],\"symbols\":[],\"media\":[{\"id\":741372929133416449,\"id_str\":\"741372929133416449\",\"indices\":[50,73],\"media_url\":\"http://pbs.twimg.com/ext_tw_video_thumb/741372929133416449/pu/img/7Z8tnkTA9Zna7JpT.jpg\",\"media_url_https\":\"https://pbs.twimg.com/ext_tw_video_thumb/741372929133416449/pu/img/7Z8tnkTA9Zna7JpT.jpg\",\"url\":\"https://t.co/X3Limc0a33\",\"display_url\":\"pic.twitter.com/X3Limc0a33\",\"expanded_url\":\"http://twitter.com/Paulius98/status/741372988948373504/video/1\",\"type\":\"photo\",\"sizes\":{\"small\":{\"w\":312,\"h\":180,\"resize\":\"fit\"},\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"medium\":{\"w\":312,\"h\":180,\"resize\":\"fit\"},\"large\":{\"w\":312,\"h\":180,\"resize\":\"fit\"}},\"source_status_id\":741372988948373504,\"source_status_id_str\":\"741372988948373504\",\"source_user_id\":169006925,\"source_user_id_str\":\"169006925\"}]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600665\tar\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[],\"symbols\":[]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600666\tja\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"00machi\",\"name\":\"町蔵\",\"id\":159475463,\"id_str\":\"159475463\",\"indices\":[3,11]}],\"symbols\":[],\"media\":[{\"id\":741333584984739841,\"id_str\":\"741333584984739841\",\"indices\":[20,43],\"media_url\":\"http://pbs.twimg.com/media/Ckm-98EVEAErT27.jpg\",\"media_url_https\":\"https://pbs.twimg.com/media/Ckm-98EVEAErT27.jpg\",\"url\":\"https://t.co/rNma3yQPxI\",\"display_url\":\"pic.twitter.com/rNma3yQPxI\",\"expanded_url\":\"http://twitter.com/00machi/status/741333589481033728/photo/1\",\"type\":\"photo\",\"sizes\":{\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"large\":{\"w\":450,\"h\":650,\"resize\":\"fit\"},\"medium\":{\"w\":450,\"h\":650,\"resize\":\"fit\"},\"small\":{\"w\":450,\"h\":650,\"resize\":\"fit\"}},\"source_status_id\":741333589481033728,\"source_status_id_str\":\"741333589481033728\",\"source_user_id\":159475463,\"source_user_id_str\":\"159475463\"}]}\nnull\tnull\tnull\tnull\nnull\tnull\tnull\tnull\nSat Jun 11 06:30:00 +0000 2016\t1465626600659\ten\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"_nsykr\",\"name\":\"ï¸ â€\",\"id\":198808091,\"id_str\":\"198808091\",\"indices\":[3,10]}],\"symbols\":[],\"media\":[{\"id\":664794517854294016,\"id_str\":\"664794517854294016\",\"indices\":[45,68],\"media_url\":\"http://pbs.twimg.com/ext_tw_video_thumb/664794517854294016/pu/img/i1f28VKQ0hJl1DOi.jpg\",\"media_url_https\":\"https://pbs.twimg.com/ext_tw_video_thumb/664794517854294016/pu/img/i1f28VKQ0hJl1DOi.jpg\",\"url\":\"https://t.co/COWJHd4T6F\",\"display_url\":\"pic.twitter.com/COWJHd4T6F\",\"expanded_url\":\"http://twitter.com/NSyakiraa/status/664803605749239808/video/1\",\"type\":\"photo\",\"sizes\":{\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"medium\":{\"w\":600,\"h\":338,\"resize\":\"fit\"},\"small\":{\"w\":340,\"h\":191,\"resize\":\"fit\"},\"large\":{\"w\":1024,\"h\":576,\"resize\":\"fit\"}},\"source_status_id\":664803605749239808,\"source_status_id_str\":\"664803605749239808\",\"source_user_id\":198808091,\"source_user_id_str\":\"198808091\"}]}\nSat Jun 11 06:30:00 +0000 2016\t1465626600665\ttl\t{\"hashtags\":[],\"urls\":[],\"user_mentions\":[{\"screen_name\":\"HistoryToLearn\",\"name\":\"Lost In History\",\"id\":149760844,\"id_str\":\"149760844\",\"indices\":[3,18]}],\"symbols\":[],\"media\":[{\"id\":741492078014824448,\"id_str\":\"741492078014824448\",\"indices\":[37,60],\"media_url\":\"http://pbs.twimg.com/media/CkpPHcnXAAApTxr.jpg\",\"media_url_https\":\"https://pbs.twimg.com/media/CkpPHcnXAAApTxr.jpg\",\"url\":\"https://t.co/RHIa5VbujI\",\"display_url\":\"pic.twitter.com/RHIa5VbujI\",\"expanded_url\":\"http://twitter.com/HistoryToLearn/status/741492082972491776/photo/1\",\"type\":\"photo\",\"sizes\":{\"large\":{\"w\":500,\"h\":767,\"resize\":\"fit\"},\"thumb\":{\"w\":150,\"h\":150,\"resize\":\"crop\"},\"small\":{\"w\":443,\"h\":680,\"resize\":\"fit\"},\"medium\":{\"w\":500,\"h\":767,\"resize\":\"fit\"}},\"source_status_id\":741492082972491776,\"source_status_id_str\":\"741492082972491776\",\"source_user_id\":149760844,\"source_user_id_str\":\"149760844\"}]}\nnull\tnull\tnull\tnull\nnull\tnull\tnull\tnull\nnull\tnull\tnull\tnull\nnull\tnull\tnull\tnull\nnull\tnull\tnull\tnull\nnull\tnull\tnull\tnull\nnull\tnull\tnull\tnull\nnull\tnull\tnull\tnull\nnull\tnull\tnull\tnull\nnull\tnull\tnull\tnull\nnull\tnull\tnull\tnull\nnull\tnull\tnull\tnull\nnull\tnull\tnull\tnull\n"}]},"apps":[],"jobName":"paragraph_1523519142660_-1191688177","id":"20180412-094542_431263891","dateCreated":"2018-04-12T09:45:42+0200","dateStarted":"2018-04-12T19:04:57+0200","dateFinished":"2018-04-12T19:04:57+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9363"},{"text":"%md\n### Question II.b. (10/30)\n\nWork on the `twitter_11_06_2016` table.\n\ni) Extract the hastags array from each tweet. Filter out the rows which do not have any hashtag. Include `timestamp_ms` and `lang` in the resulting table.\nii) Normalize the table obtained from the previous step. This means that each row should contain exactly one hashtag. Include `timestamp_ms` and `lang` in the resulting table.\n\nHint: your intermediary table should look like `intermediary_hashtags_example` and the final table should look like `hashtags_example` (see below).\nHint: in (ii), you can use the following Hive functions:\n- `substr` function: <https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-StringFunctions>\n- `split` function: <https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-StringFunctions>\n- `explode` function: <https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-explode>","dateUpdated":"2018-03-28T13:15:09+0200","config":{"tableHide":false,"editorSetting":{"editOnDblClick":true,"language":"markdown"},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Question II.b. (10/30)</h3>\n<p>Work on the <code>twitter_11_06_2016</code> table.</p>\n<p>i) Extract the hastags array from each tweet. Filter out the rows which do not have any hashtag. Include <code>timestamp_ms</code> and <code>lang</code> in the resulting table.\n<br />ii) Normalize the table obtained from the previous step. This means that each row should contain exactly one hashtag. Include <code>timestamp_ms</code> and <code>lang</code> in the resulting table.</p>\n<p>Hint: your intermediary table should look like <code>intermediary_hashtags_example</code> and the final table should look like <code>hashtags_example</code> (see below).\n<br />Hint: in (ii), you can use the following Hive functions:</p>\n<ul>\n<li><code>substr</code> function: <a href=\"https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-StringFunctions\">https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-StringFunctions</a></li>\n<li><code>split</code> function: <a href=\"https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-StringFunctions\">https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-StringFunctions</a></li>\n<li><code>explode</code> function: <a href=\"https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-explode\">https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-explode</a></li>\n</ul>\n"}]},"apps":[],"jobName":"paragraph_1522235709496_1436845707","id":"20180326-201126_894808921","dateCreated":"2018-03-28T13:15:09+0200","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:9364"},{"text":"%md\n ### Answers\n \n #### Subquestion i)\n \n For extracting the hashtags from each tweet, we simply write a regex which replaces each object in the array corresponding to the field `hashtags` with the value of the field `text`, which represents the actual value of the hashtag:","user":"musuroi","dateUpdated":"2018-04-13T14:19:27+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Answers</h3>\n<h4>Subquestion i)</h4>\n<p>For extracting the hashtags from each tweet, we simply write a regex which replaces each object in the array corresponding to the field <code>hashtags</code> with the value of the field <code>text</code>, which represents the actual value of the hashtag:</p>\n"}]},"apps":[],"jobName":"paragraph_1523550967573_781348216","id":"20180412-183607_1364115037","dateCreated":"2018-04-12T18:36:07+0200","dateStarted":"2018-04-13T14:19:27+0200","dateFinished":"2018-04-13T14:19:27+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9365"},{"text":"%jdbc(hive)\n\nDROP TABLE IF EXISTS baetu.intermediary_hashtags;\n\nCREATE TABLE baetu.intermediary_hashtags STORED AS parquet AS\n SELECT timestamp_ms,\n lang,\n regexp_replace(get_json_object(entities, '$.hashtags'),'\\\\{\\\"text\\\":(\\\"[^\\\"]+\\\"),\\\"indices\\\":\\\\[[0-9]+,[0-9]+\\\\]\\\\}','$1') as hashtags\n FROM baetu.twitter_11_06_2016\n WHERE get_json_object(entities, '$.hashtags') IS NOT NULL AND get_json_object(entities, '$.hashtags') != '[]';\n ","user":"baetu","dateUpdated":"2018-04-12T21:20:03+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"}]},"apps":[],"jobName":"paragraph_1523551140398_1189595803","id":"20180412-183900_223198892","dateCreated":"2018-04-12T18:39:00+0200","dateStarted":"2018-04-12T21:20:03+0200","dateFinished":"2018-04-12T21:20:19+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9366"},{"text":"%md \n\nLet's see if the result is satisfactory:","user":"baetu","dateUpdated":"2018-04-12T21:27:44+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>Let's see if the result is satisfactory:</p>\n"}]},"apps":[],"jobName":"paragraph_1523561251984_-1175670003","id":"20180412-212731_35577444","dateCreated":"2018-04-12T21:27:31+0200","dateStarted":"2018-04-12T21:27:44+0200","dateFinished":"2018-04-12T21:27:44+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9367"},{"text":"%jdbc(hive)\n\nSELECT * FROM baetu.intermediary_hashtags limit 10;","user":"baetu","dateUpdated":"2018-04-12T21:22:42+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"intermediary_hashtags.timestamp_ms\tintermediary_hashtags.lang\tintermediary_hashtags.hashtags\n1465626600666\ten\t[\"StopBoknal\",\"EndDogMeat\"]\n1465626601660\ttr\t[\"OrangeSponsorsYou\",\"TUR\"]\n1465626601662\ten\t[\"PrayForChristina\"]\n1465626601660\tund\t[\"MasterCommands\",\"CheckThisOut\"]\n1465626601666\ten\t[\"PrayForChristina\"]\n1465626601666\ttl\t[\"ALDUBEBTrumpets\"]\n1465626601662\ttl\t[\"tbtfeels\"]\n1465626601662\tja\t[\"券売機\",\"券売機valtec\"]\n1465626600665\ten\t[\"bigbang\",\"foshan\",\"fanmeeting\",\"gd\",\"bangbangbang\",\"ifyou\"]\n1465626602663\tja\t[\"çµµæãã•ã‚“ã¨ç¹‹ãŒã‚ŠãŸã„\"]\n"}]},"apps":[],"jobName":"paragraph_1523551875942_1965478328","id":"20180412-185115_762982088","dateCreated":"2018-04-12T18:51:15+0200","dateStarted":"2018-04-12T21:22:42+0200","dateFinished":"2018-04-12T21:22:43+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9368"},{"text":"%md \n\nIt is indeed satisfactory, as it is similar to the example given below:","user":"baetu","dateUpdated":"2018-04-12T21:35:03+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>It is indeed satisfactory, as it is similar to the example given below:</p>\n"}]},"apps":[],"jobName":"paragraph_1523561673855_-411577301","id":"20180412-213433_1858966415","dateCreated":"2018-04-12T21:34:33+0200","dateStarted":"2018-04-12T21:35:03+0200","dateFinished":"2018-04-12T21:35:03+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9369"},{"text":"%jdbc(hive)\n-- Display example 1\ndrop table if exists baetu.intermediary_hashtags_example;\ncreate external table baetu.intermediary_hashtags_example(timestamp_ms string, lang string, hashtags string)\nstored as parquet\nlocation '/datasets/intermediary_hashtags_example';\nselect *\nfrom baetu.intermediary_hashtags_example;","user":"baetu","dateUpdated":"2018-04-12T21:35:05+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TABLE","data":"intermediary_hashtags_example.timestamp_ms\tintermediary_hashtags_example.lang\tintermediary_hashtags_example.hashtags\n1510434840661\tar\t[\"المسابقة_الوطنية_للغة_العربية_٢٠١٧\"]\n1510434840658\ten\t[\"COYBIG\"]\n1510434840658\tar\t[\"رابطه_الاتØاد_للاتØاد_Ùقط\"]\n1510434840665\tar\t[\"العلوم_تنتÙض_ضد_الظلم\"]\n1510434840664\tes\t[\"cólera\",\"Yemen\",\"MSF\"]\n1510434840658\tar\t[\"اقاله_العيسي_مطلب_الشعب\"]\n1510434840665\tund\t[\"SindiChorros\"]\n1510434840663\ten\t[\"Eisenhower\"]\n1510434840666\tes\t[\"11Nov2017\",\"SaharaLibre\"]\n1510434840664\tund\t[\"NewProfilePic\"]\n1510434841664\tpt\t[\"EMABiggestFansKatyPerry\"]\n1510434841658\ten\t[\"EMABiggestFansArianaGrande\"]\n1510434841664\ten\t[\"EMABiggestFansJustinBieber\"]\n1510434841663\tund\t[\"reputaylurking\",\"reputation\"]\n1510434842658\ten\t[\"blog\",\"bloggerswanted\",\"blogging\"]\n1510434842659\ten\t[\"MINHYUK\"]\n1510434842661\ten\t[\"Textiles\",\"Crafters\",\"fabric\",\"homedecor\",\"Handmade\"]\n1510434843661\ten\t[\"Llibertatpresopolitics\"]\n1510434843657\tes\t[\"startups\",\"empresas\"]\n1510434843659\ten\t[\"wcw\"]\n1510434843659\tfr\t[\"ah\"]\n1510434843659\tes\t[\"CNCOwners\"]\n1510434843663\tund\t[\"الشتاء\"]\n1510434844658\ten\t[\"COYBIG\",\"DENvIRE\"]\n1510434844666\ten\t[\"AMAs\"]\n1510434844665\ten\t[\"AMAs\"]\n1510434842663\tes\t[\"CulturaPrevención\"]\n1510434844657\ten\t[\"USNavy\",\"Marines\",\"USAirforce\"]\n1510434844659\ten\t[\"Flights\"]\n1510434844658\tja\t[\"トレクル\"]\n1510434844662\ten\t[\"tvtime\"]\n1510434845661\ten\t[\"GoDinos\",\"HardyCup81\"]\n1510434845660\ten\t[\"Switch\"]\n1510434845658\ten\t[\"LestWeForget\"]\n1510434845660\tar\t[\"بوØ_المساء\"]\n1510434846657\tund\t[\"HeartbreakOnAFullMoon\"]\n1510434846659\tja\t[\"æ—¥æœ¬æ ª\"]\n1510434846658\ttr\t[\"Erzurum\",\"VatanAÅŸkı\"]\n1510434846664\tar\t[\"زد_رصيدك40\"]\n1510434846657\tvi\t[\"shop\"]\n1510434846664\tde\t[\"EMABiggestFansJustinBieber\"]\n1510434846658\ten\t[\"ForTheBrand\"]\n1510434846658\tja\t[\"水樹日ã®å‡ºãƒœãƒƒãƒˆ\"]\n1510434846660\tes\t[\"Fútbol\"]\n1510434847662\tda\t[\"CardFraud\",\"IDtheft\"]\n1510434847664\tin\t[\"AggieEagleKAOS\",\"NCAT\",\"NCCU\"]\n1510434847666\ten\t[\"EU\",\"NHS\"]\n1510434847661\tel\t[\"stinigiamas\"]\n1510434847666\ten\t[\"people\",\"life\"]\n1510434841660\ten\t[\"OneTreeHill\"]\n1510434847666\tes\t[\"liberenaCarlosVelasco\"]\n1510434847660\tpt\t[\"EMABiggestFansShawnMendes\"]\n1510434846661\ten\t[\"flickr\"]\n1510434848662\ten\t[\"iphone\",\"photovideo\"]\n1510434848665\tes\t[\"EMABiggestFansJustinBieber\"]\n1510434848660\tja\t[\"booklog\"]\n1510434848664\ten\t[\"AMAs\"]\n1510434848664\ten\t[\"GoDogs\"]\n1510434848663\tes\t[\"AvilaTeToca\"]\n1510434848660\ten\t[\"teamwoowee\"]\n1510434848664\ten\t[\"InternetOfThings\",\"DigitalConnectIN\",\"data\",\"Cloud\",\"ML\",\"blockchain\",\"mobile\"]\n1510434849664\ten\t[\"ConorMcGregor\",\"ConorMcGregor\"]\n1510434849665\tes\t[\"SabadoDeGanarSeguidores\"]\n1510434849663\ten\t[\"hiring\",\"job\",\"Retail\",\"Wrentham\"]\n1510434850659\tja\t[\"followmeJP\",\"followmejp\"]\n1510434850665\ten\t[\"Mysteries\"]\n1510434850665\tund\t[\"MPN\",\"SorryNotSorryClipe\"]\n1510434850660\tpt\t[\"EMABiggestFansShawnMendes\"]\n1510434850661\tar\t[\"كاس_العالم_ØيوØشنا_Ù¢\"]\n1510434850660\tes\t[\"CopaChileXel13\"]\n1510434850657\ten\t[\"MAGA\",\"ClintonNewsNetwork\",\"FakeNews\"]\n1510434850662\tund\t[\"SafitürkKastamonuyaGeliyor\"]\n1510434851664\tfr\t[\"FRAvNZL\"]\n1510434851665\tes\t[\"Perú\",\"MachuPicchu\"]\n1510434852658\tar\t[\"كاس_العالم_ØيوØشنا_Ù£\"]\n1510434852665\ttr\t[\"silagderki\"]\n1510434852665\tin\t[\"EMABiggestFansJustinBieber\"]\n1510434852661\ten\t[\"RoyMoore\"]\n1510434852662\ten\t[\"Luxury\",\"Food\",\"Vlog\",\"Foodie\",\"PestoChampion\",\"Eat\",\"Foodporn\",\"Dinner\"]\n1510434852666\ten\t[\"FakeNews\"]\n1510434852658\tar\t[\"قروب_أنÙـاس_Ù…Øبوبي\"]\n1510434852663\tar\t[\"كاس_العالم_ØيوØشنا_Ù£\"]\n1510434853666\ten\t[\"NoMoore\"]\n1510434853666\ten\t[\"ARTPOP\"]\n1510434854665\ttr\t[\"Bendtner\"]\n1510434854664\tar\t[\"قرطاج\",\"روسيا\",\"WorldCup\",\"تونس\",\"السعودية\",\"مصر\",\"المغرب\",\"مونديال2022_Ùخر_للعرب\"]\n1510434854665\tund\t[\"PhrasesDeLibros\"]\n1510434855660\ten\t[\"PIZZA\"]\n1510434855662\tar\t[\"زد_رصيدك40\"]\n1510434855659\ten\t[\"HOW_I_DRAW_CREEK\",\"SouthPark\"]\n1510434855658\tund\t[\"EMABiggestFansKatyPerry\"]\n1510434855659\ten\t[\"Eastcote\"]\n1510434855659\tde\t[\"Nachrichten\"]\n1510434855666\ten\t[\"Preacher\"]\n1510434856663\tja\t[\"ç‹ã®ç‹¬ã‚Šè¨€\"]\n1510434856666\tar\t[\"قطر\"]\n1510434856665\ten\t[\"nonsense\",\"nonsenseengine\"]\n1510434856658\ttr\t[\"yaghaz\",\"aşıkoldum\"]\n1510434856661\tes\t[\"SiliconValley\"]\n1510434856659\ten\t[\"GetAHalo\",\"StayCharged\",\"TheBest\",\"PortablePower\",\"HaveFun\"]\n"}]},"apps":[],"jobName":"paragraph_1523561657051_-1327895939","id":"20180412-213417_1741548387","dateCreated":"2018-04-12T21:34:17+0200","dateStarted":"2018-04-12T21:35:05+0200","dateFinished":"2018-04-12T21:35:06+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9370"},{"text":"%md\n\n#### Subquestion ii)\n \n For normalizing the table obtained in the previous step, we follow the steps:\n * first, call the function `substr` for stripping the first and the last two characters of the `hashtags` column (which is of type `string`)\n * then, call the function `split` to split the string by the pattern `\",\"`, thus resulting an array of plain hashtags\n * lastly, call the function `explode` to transform the newly created column with array type into table","user":"musuroi","dateUpdated":"2018-04-13T17:58:04+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h4>Subquestion ii)</h4>\n<p>For normalizing the table obtained in the previous step, we follow the steps:</p>\n<ul>\n<li>first, call the function <code>substr</code> for stripping the first and the last two characters of the <code>hashtags</code> column (which is of type <code>string</code>)</li>\n<li>then, call the function <code>split</code> to split the string by the pattern <code>\",\"</code>, thus resulting an array of plain hashtags</li>\n<li>lastly, call the function <code>explode</code> to transform the newly created column with array type into table</li>\n</ul>\n"}]},"apps":[],"jobName":"paragraph_1523560917426_-2137645710","id":"20180412-212157_221894308","dateCreated":"2018-04-12T21:21:57+0200","dateStarted":"2018-04-13T17:57:53+0200","dateFinished":"2018-04-13T17:57:53+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9371"},{"text":"%jdbc(hive)\n\nDROP TABLE IF EXISTS baetu.hashtags;\n\nCREATE TABLE baetu.hashtags STORED AS parquet AS\n SELECT timestamp_ms,\n lang,\n hashtag\n FROM baetu.intermediary_hashtags\n LATERAL VIEW explode(split(substr(hashtags, 3, length(hashtags) - 4), '\\\",\\\"')) hash_table as hashtag;","user":"baetu","dateUpdated":"2018-04-12T21:44:15+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"}]},"apps":[],"jobName":"paragraph_1523561605311_-176788527","id":"20180412-213325_1825053002","dateCreated":"2018-04-12T21:33:25+0200","dateStarted":"2018-04-12T21:44:15+0200","dateFinished":"2018-04-12T21:44:25+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9372"},{"text":"%md\n\nLet's check if the result is satisfactory:","user":"baetu","dateUpdated":"2018-04-12T21:40:25+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>Let's check if the result is satisfactory:</p>\n"}]},"apps":[],"jobName":"paragraph_1523562009623_-1858726828","id":"20180412-214009_542616226","dateCreated":"2018-04-12T21:40:09+0200","dateStarted":"2018-04-12T21:40:25+0200","dateFinished":"2018-04-12T21:40:25+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9373"},{"text":"%jdbc(hive)\n\nSELECT * FROM baetu.hashtags limit 10;","user":"baetu","dateUpdated":"2018-04-12T21:46:07+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"hashtags.timestamp_ms\thashtags.lang\thashtags.hashtag\n1465626600666\ten\tStopBoknal\n1465626600666\ten\tEndDogMeat\n1465626601660\ttr\tOrangeSponsorsYou\n1465626601660\ttr\tTUR\n1465626601662\ten\tPrayForChristina\n1465626601660\tund\tMasterCommands\n1465626601660\tund\tCheckThisOut\n1465626601666\ten\tPrayForChristina\n1465626601666\ttl\tALDUBEBTrumpets\n1465626601662\ttl\ttbtfeels\n"}]},"apps":[],"jobName":"paragraph_1523562031570_1301646610","id":"20180412-214031_1298784447","dateCreated":"2018-04-12T21:40:31+0200","dateStarted":"2018-04-12T21:46:07+0200","dateFinished":"2018-04-12T21:46:08+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9374"},{"text":"%jdbc(hive)\n-- Display example 2\ndrop table if exists baetu.hashtags_example;\ncreate external table baetu.hashtags_example(timestamp_ms string, lang string, hashtag string)\nstored as parquet\nlocation '/datasets/hashtags_example';\nselect *\nfrom baetu.hashtags_example;","user":"baetu","dateUpdated":"2018-04-12T21:46:31+0200","config":{"colWidth":12,"editorMode":"ace/mode/sql","results":{},"enabled":true,"editorSetting":{"editOnDblClick":false,"language":"sql"}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TABLE","data":"hashtags_example.timestamp_ms\thashtags_example.lang\thashtags_example.hashtag\n1510434840661\tar\tالمسابقة_الوطنية_للغة_العربية_٢٠١٧\n1510434840658\ten\tCOYBIG\n1510434840658\tar\tرابطه_الاتØاد_للاتØاد_Ùقط\n1510434840665\tar\tالعلوم_تنتÙض_ضد_الظلم\n1510434840664\tes\tcólera\n1510434840664\tes\tYemen\n1510434840664\tes\tMSF\n1510434840658\tar\tاقاله_العيسي_مطلب_الشعب\n1510434840665\tund\tSindiChorros\n1510434840663\ten\tEisenhower\n1510434840666\tes\t11Nov2017\n1510434840666\tes\tSaharaLibre\n1510434840664\tund\tNewProfilePic\n1510434841664\tpt\tEMABiggestFansKatyPerry\n1510434841658\ten\tEMABiggestFansArianaGrande\n1510434841664\ten\tEMABiggestFansJustinBieber\n1510434841663\tund\treputaylurking\n1510434841663\tund\treputation\n1510434842658\ten\tblog\n1510434842658\ten\tbloggerswanted\n1510434842658\ten\tblogging\n1510434842659\ten\tMINHYUK\n1510434842661\ten\tTextiles\n1510434842661\ten\tCrafters\n1510434842661\ten\tfabric\n1510434842661\ten\thomedecor\n1510434842661\ten\tHandmade\n1510434843661\ten\tLlibertatpresopolitics\n1510434843657\tes\tstartups\n1510434843657\tes\tempresas\n1510434843659\ten\twcw\n1510434843659\tfr\tah\n1510434843659\tes\tCNCOwners\n1510434843663\tund\tالشتاء\n1510434844658\ten\tCOYBIG\n1510434844658\ten\tDENvIRE\n1510434844666\ten\tAMAs\n1510434844665\ten\tAMAs\n1510434842663\tes\tCulturaPrevención\n1510434844657\ten\tUSNavy\n1510434844657\ten\tMarines\n1510434844657\ten\tUSAirforce\n1510434844659\ten\tFlights\n1510434844658\tja\tトレクル\n1510434844662\ten\ttvtime\n1510434845661\ten\tGoDinos\n1510434845661\ten\tHardyCup81\n1510434845660\ten\tSwitch\n1510434845658\ten\tLestWeForget\n1510434845660\tar\tبوØ_المساء\n1510434846657\tund\tHeartbreakOnAFullMoon\n1510434846659\tja\tæ—¥æœ¬æ ª\n1510434846658\ttr\tErzurum\n1510434846658\ttr\tVatanAÅŸkı\n1510434846664\tar\tزد_رصيدك40\n1510434846657\tvi\tshop\n1510434846664\tde\tEMABiggestFansJustinBieber\n1510434846658\ten\tForTheBrand\n1510434846658\tja\t水樹日ã®å‡ºãƒœãƒƒãƒˆ\n1510434846660\tes\tFútbol\n1510434847662\tda\tCardFraud\n1510434847662\tda\tIDtheft\n1510434847664\tin\tAggieEagleKAOS\n1510434847664\tin\tNCAT\n1510434847664\tin\tNCCU\n1510434847666\ten\tEU\n1510434847666\ten\tNHS\n1510434847661\tel\tstinigiamas\n1510434847666\ten\tpeople\n1510434847666\ten\tlife\n1510434841660\ten\tOneTreeHill\n1510434847666\tes\tliberenaCarlosVelasco\n1510434847660\tpt\tEMABiggestFansShawnMendes\n1510434846661\ten\tflickr\n1510434848662\ten\tiphone\n1510434848662\ten\tphotovideo\n1510434848665\tes\tEMABiggestFansJustinBieber\n1510434848660\tja\tbooklog\n1510434848664\ten\tAMAs\n1510434848664\ten\tGoDogs\n1510434848663\tes\tAvilaTeToca\n1510434848660\ten\tteamwoowee\n1510434848664\ten\tInternetOfThings\n1510434848664\ten\tDigitalConnectIN\n1510434848664\ten\tdata\n1510434848664\ten\tCloud\n1510434848664\ten\tML\n1510434848664\ten\tblockchain\n1510434848664\ten\tmobile\n1510434849664\ten\tConorMcGregor\n1510434849664\ten\tConorMcGregor\n1510434849665\tes\tSabadoDeGanarSeguidores\n1510434849663\ten\thiring\n1510434849663\ten\tjob\n1510434849663\ten\tRetail\n1510434849663\ten\tWrentham\n1510434850659\tja\tfollowmeJP\n1510434850659\tja\tfollowmejp\n1510434850665\ten\tMysteries\n1510434850665\tund\tMPN\n1510434850665\tund\tSorryNotSorryClipe\n1510434850660\tpt\tEMABiggestFansShawnMendes\n1510434850661\tar\tكاس_العالم_ØيوØشنا_Ù¢\n1510434850660\tes\tCopaChileXel13\n1510434850657\ten\tMAGA\n1510434850657\ten\tClintonNewsNetwork\n1510434850657\ten\tFakeNews\n1510434850662\tund\tSafitürkKastamonuyaGeliyor\n1510434851664\tfr\tFRAvNZL\n1510434851665\tes\tPerú\n1510434851665\tes\tMachuPicchu\n1510434852658\tar\tكاس_العالم_ØيوØشنا_Ù£\n1510434852665\ttr\tsilagderki\n1510434852665\tin\tEMABiggestFansJustinBieber\n1510434852661\ten\tRoyMoore\n1510434852662\ten\tLuxury\n1510434852662\ten\tFood\n1510434852662\ten\tVlog\n1510434852662\ten\tFoodie\n1510434852662\ten\tPestoChampion\n1510434852662\ten\tEat\n1510434852662\ten\tFoodporn\n1510434852662\ten\tDinner\n1510434852666\ten\tFakeNews\n1510434852658\tar\tقروب_أنÙـاس_Ù…Øبوبي\n1510434852663\tar\tكاس_العالم_ØيوØشنا_Ù£\n1510434853666\ten\tNoMoore\n1510434853666\ten\tARTPOP\n1510434854665\ttr\tBendtner\n1510434854664\tar\tقرطاج\n1510434854664\tar\tروسيا\n1510434854664\tar\tWorldCup\n1510434854664\tar\tتونس\n1510434854664\tar\tالسعودية\n1510434854664\tar\tمصر\n1510434854664\tar\tالمغرب\n1510434854664\tar\tمونديال2022_Ùخر_للعرب\n1510434854665\tund\tPhrasesDeLibros\n1510434855660\ten\tPIZZA\n1510434855662\tar\tزد_رصيدك40\n1510434855659\ten\tHOW_I_DRAW_CREEK\n1510434855659\ten\tSouthPark\n1510434855658\tund\tEMABiggestFansKatyPerry\n1510434855659\ten\tEastcote\n1510434855659\tde\tNachrichten\n1510434855666\ten\tPreacher\n1510434856663\tja\tç‹ã®ç‹¬ã‚Šè¨€\n1510434856666\tar\tقطر\n1510434856665\ten\tnonsense\n1510434856665\ten\tnonsenseengine\n1510434856658\ttr\tyaghaz\n1510434856658\ttr\taşıkoldum\n1510434856661\tes\tSiliconValley\n1510434856659\ten\tGetAHalo\n1510434856659\ten\tStayCharged\n1510434856659\ten\tTheBest\n1510434856659\ten\tPortablePower\n1510434856659\ten\tHaveFun\n"}]},"apps":[],"jobName":"paragraph_1522235709499_1437230456","id":"20180326-205541_1698608500","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-12T21:46:32+0200","dateFinished":"2018-04-12T21:46:32+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9375"},{"text":"%md\n\nWe can see that indeed, we did the normalization as expected.\n","user":"baetu","dateUpdated":"2018-04-12T21:47:23+0200","config":{"colWidth":12,"editorMode":"ace/mode/markdown","results":{},"enabled":true,"editorSetting":{"editOnDblClick":true,"language":"markdown"},"editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>We can see that indeed, we did the normalization as expected.</p>\n"}]},"apps":[],"jobName":"paragraph_1522235709500_1435306711","id":"20180326-210211_615760584","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-12T21:47:23+0200","dateFinished":"2018-04-12T21:47:23+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9376"},{"text":"%md\n### Question II.c. (5/30)\n\ni) From the hashtags table you obtained in the previous question, find out the 10 most represented languages.\nii) Get the top 20 most mentioned hashtags. Display the results with the contribution of each language to the hashtags.","dateUpdated":"2018-03-28T13:15:09+0200","config":{"tableHide":false,"editorSetting":{"editOnDblClick":true,"language":"markdown"},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Question II.c. (5/30)</h3>\n<p>i) From the hashtags table you obtained in the previous question, find out the 10 most represented languages.\n<br />ii) Get the top 20 most mentioned hashtags. Display the results with the contribution of each language to the hashtags.</p>\n"}]},"apps":[],"jobName":"paragraph_1522235709501_1434921962","id":"20180326-210155_1341471366","dateCreated":"2018-03-28T13:15:09+0200","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:9377"},{"text":"%md\n### Answers\n\n#### Subquestion i)\n\nTo obtain the 10 most represented languages, we simply group the rows in the table hashtags by lang values and we use the count function for aggregating the groups.","user":"musuroi","dateUpdated":"2018-04-13T18:38:03+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Answers</h3>\n<h4>Subquestion i)</h4>\n<p>To obtain the 10 most represented languages, we simply group the rows in the table hashtags by lang values and we use the count function for aggregating the groups.</p>\n"}]},"apps":[],"jobName":"paragraph_1523621955089_1055794468","id":"20180413-141915_2085023549","dateCreated":"2018-04-13T14:19:15+0200","dateStarted":"2018-04-13T18:37:53+0200","dateFinished":"2018-04-13T18:37:53+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9378"},{"text":"%jdbc(hive)\nSELECT lang, count(*) as counts FROM baetu.hashtags\n GROUP BY lang\n ORDER BY counts DESC\n LIMIT 10;","user":"musuroi","dateUpdated":"2018-04-13T19:17:14+0200","config":{"colWidth":12,"editorMode":"ace/mode/sql","results":{"0":{"graph":{"mode":"multiBarChart","height":300,"optionOpen":false},"helium":{}}},"enabled":true,"editorSetting":{"editOnDblClick":false,"language":"sql"}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"lang\tcounts\nen\t461057\nja\t161624\nes\t140883\nund\t131245\nar\t110069\nko\t108531\nth\t53890\ntr\t31212\nfr\t24797\nin\t24171\n"}]},"apps":[],"jobName":"paragraph_1522235709503_1435691460","id":"20180326-210601_623712001","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-13T14:25:38+0200","dateFinished":"2018-04-13T14:25:48+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9379"},{"text":"%md\n#### Subquestion ii)\n\nFor the answer to this subquestion, we will use nested select in the following way. First we select the 20 most popular hashtags from baetu.hashtags table using group by hashtag. Then we select the hashtag and the language using group by hashtag and lang also from baetu.hashtags table, but this time we add the condition that the hashtag is among the hashtags of the first select. We additionally keep the total count of the hashtag in order to sort them according to the total counts.","user":"musuroi","dateUpdated":"2018-04-16T21:08:58+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h4>Subquestion ii)</h4>\n<p>For the answer to this subquestion, we will use nested select in the following way. First we select the 20 most popular hashtags from baetu.hashtags table using group by hashtag. Then we select the hashtag and the language using group by hashtag and lang also from baetu.hashtags table, but this time we add the condition that the hashtag is among the hashtags of the first select. We additionally keep the total count of the hashtag in order to sort them according to the total counts.</p>\n"}]},"apps":[],"jobName":"paragraph_1523622381671_-372612921","id":"20180413-142621_1459412978","dateCreated":"2018-04-13T14:26:21+0200","dateStarted":"2018-04-16T21:08:58+0200","dateFinished":"2018-04-16T21:08:58+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9380"},{"text":"%jdbc(hive)\nWITH popular_hashtags AS\n (\n SELECT LOWER(hashtag) as hashtag, COUNT(*) AS counts_all FROM baetu.hashtags\n GROUP BY LOWER(hashtag)\n ORDER BY counts_all DESC\n LIMIT 20\n ),\n h AS\n (\n SELECT * from baetu.hashtags\n )\nSELECT LOWER(h.hashtag) as hashtag, h.lang, COUNT(*) AS counts, p.counts_all FROM h\n JOIN popular_hashtags p ON lower(h.hashtag) = lower(p.hashtag) \n GROUP BY LOWER(h.hashtag), h.lang, p.counts_all\n ORDER BY p.counts_all DESC, counts DESC;\n","user":"musuroi","dateUpdated":"2018-04-16T22:10:43+0200","config":{"colWidth":12,"enabled":true,"results":{"0":{"graph":{"mode":"multiBarChart","height":300,"optionOpen":false,"setting":{"multiBarChart":{"stacked":true}},"commonSetting":{},"keys":[{"name":"hashtag","index":0,"aggr":"sum"}],"groups":[{"name":"h.lang","index":1,"aggr":"sum"}],"values":[{"name":"counts","index":2,"aggr":"sum"}]},"helium":{}}},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql","lineNumbers":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"hashtag\th.lang\tcounts\tp.counts_all\nmtvawardsstar\tit\t13407\t29288\nmtvawardsstar\ten\t6460\t29288\nmtvawardsstar\tund\t2079\t29288\nmtvawardsstar\tes\t1299\t29288\nmtvawardsstar\tpt\t1232\t29288\nmtvawardsstar\tru\t1185\t29288\nmtvawardsstar\tfr\t672\t29288\nmtvawardsstar\tht\t539\t29288\nmtvawardsstar\tet\t331\t29288\nmtvawardsstar\tin\t330\t29288\nmtvawardsstar\tcs\t308\t29288\nmtvawardsstar\ttr\t250\t29288\nmtvawardsstar\tfi\t221\t29288\nmtvawardsstar\ttl\t192\t29288\nmtvawardsstar\tde\t135\t29288\nmtvawardsstar\tlv\t118\t29288\nmtvawardsstar\tpl\t99\t29288\nmtvawardsstar\tlt\t96\t29288\nmtvawardsstar\teu\t49\t29288\nmtvawardsstar\tro\t42\t29288\nmtvawardsstar\thi\t38\t29288\nmtvawardsstar\tnl\t36\t29288\nmtvawardsstar\tcy\t29\t29288\nmtvawardsstar\tsv\t25\t29288\nmtvawardsstar\tja\t23\t29288\nmtvawardsstar\tno\t20\t29288\nmtvawardsstar\tda\t19\t29288\nmtvawardsstar\tsl\t14\t29288\nmtvawardsstar\tis\t13\t29288\nmtvawardsstar\tbg\t7\t29288\nmtvawardsstar\tuk\t7\t29288\nmtvawardsstar\thu\t6\t29288\nmtvawardsstar\tth\t3\t29288\nmtvawardsstar\tvi\t2\t29288\nmtvawardsstar\tar\t1\t29288\nmtvawardsstar\tzh\t1\t29288\nripchristina\ten\t21873\t28884\nripchristina\tes\t1993\t28884\nripchristina\tund\t1572\t28884\nripchristina\tpt\t1355\t28884\nripchristina\tth\t844\t28884\nripchristina\tfr\t370\t28884\nripchristina\tit\t296\t28884\nripchristina\tpl\t185\t28884\nripchristina\ttl\t140\t28884\nripchristina\tde\t75\t28884\nripchristina\tru\t41\t28884\nripchristina\tja\t41\t28884\nripchristina\tin\t27\t28884\nripchristina\tar\t23\t28884\nripchristina\tnl\t17\t28884\nripchristina\ttr\t9\t28884\nripchristina\tfi\t4\t28884\nripchristina\tcs\t4\t28884\nripchristina\tsv\t3\t28884\nripchristina\tht\t3\t28884\nripchristina\tet\t2\t28884\nripchristina\tda\t1\t28884\nripchristina\tcy\t1\t28884\nripchristina\thu\t1\t28884\nripchristina\tis\t1\t28884\nripchristina\tko\t1\t28884\nripchristina\tvi\t1\t28884\nripchristina\tno\t1\t28884\nmtvactappjaramillo\tes\t21999\t23081\nmtvactappjaramillo\tund\t1065\t23081\nmtvactappjaramillo\tit\t7\t23081\nmtvactappjaramillo\ten\t5\t23081\nmtvactappjaramillo\tpt\t2\t23081\nmtvactappjaramillo\tin\t1\t23081\nmtvactappjaramillo\tnl\t1\t23081\nmtvactappjaramillo\ttr\t1\t23081\nmtvmiaw\tes\t12793\t20002\nmtvmiaw\tund\t4098\t20002\nmtvmiaw\ten\t1778\t20002\nmtvmiaw\tpt\t229\t20002\nmtvmiaw\tin\t169\t20002\nmtvmiaw\tcy\t122\t20002\nmtvmiaw\tpl\t99\t20002\nmtvmiaw\ttr\t87\t20002\nmtvmiaw\tht\t78\t20002\nmtvmiaw\tfr\t75\t20002\nmtvmiaw\tit\t64\t20002\nmtvmiaw\ttl\t62\t20002\nmtvmiaw\tfi\t47\t20002\nmtvmiaw\tde\t44\t20002\nmtvmiaw\tet\t35\t20002\nmtvmiaw\tnl\t30\t20002\nmtvmiaw\tsv\t27\t20002\nmtvmiaw\teu\t24\t20002\nmtvmiaw\thu\t22\t20002\nmtvmiaw\tlt\t19\t20002\nmtvmiaw\tda\t17\t20002\nmtvmiaw\tlv\t16\t20002\nmtvmiaw\tvi\t16\t20002\nmtvmiaw\tcs\t16\t20002\nmtvmiaw\tno\t11\t20002\nmtvmiaw\thi\t10\t20002\nmtvmiaw\tro\t7\t20002\nmtvmiaw\tsl\t3\t20002\nmtvmiaw\tis\t3\t20002\nmtvmiaw\tja\t1\t20002\nflyinbangkok\tth\t13946\t18020\nflyinbangkok\ten\t2178\t18020\nflyinbangkok\tund\t874\t18020\nflyinbangkok\tko\t637\t18020\nflyinbangkok\tin\t206\t18020\nflyinbangkok\ttl\t42\t18020\nflyinbangkok\tja\t30\t18020\nflyinbangkok\tes\t17\t18020\nflyinbangkok\tsv\t11\t18020\nflyinbangkok\tde\t11\t18020\nflyinbangkok\tar\t10\t18020\nflyinbangkok\tet\t8\t18020\nflyinbangkok\tvi\t8\t18020\nflyinbangkok\tht\t6\t18020\nflyinbangkok\ttr\t5\t18020\nflyinbangkok\thu\t5\t18020\nflyinbangkok\tfr\t4\t18020\nflyinbangkok\tnl\t4\t18020\nflyinbangkok\tpt\t4\t18020\nflyinbangkok\tis\t3\t18020\nflyinbangkok\tno\t3\t18020\nflyinbangkok\tda\t2\t18020\nflyinbangkok\tcy\t2\t18020\nflyinbangkok\tit\t1\t18020\nflyinbangkok\thi\t1\t18020\nflyinbangkok\tzh\t1\t18020\nflyinbangkok\tlt\t1\t18020\neuro2016\ten\t6747\t13332\neuro2016\tfr\t1780\t13332\neuro2016\tes\t894\t13332\neuro2016\tar\t663\t13332\neuro2016\ttr\t657\t13332\neuro2016\tund\t582\t13332\neuro2016\tru\t328\t13332\neuro2016\tde\t327\t13332\neuro2016\tin\t270\t13332\neuro2016\tit\t202\t13332\neuro2016\tpt\t197\t13332\neuro2016\tja\t123\t13332\neuro2016\tnl\t110\t13332\neuro2016\tpl\t76\t13332\neuro2016\tet\t61\t13332\neuro2016\tth\t48\t13332\neuro2016\tel\t34\t13332\neuro2016\tsv\t26\t13332\neuro2016\ttl\t23\t13332\neuro2016\teu\t19\t13332\neuro2016\tsl\t16\t13332\neuro2016\tuk\t16\t13332\neuro2016\tcs\t14\t13332\neuro2016\tcy\t14\t13332\neuro2016\tno\t12\t13332\neuro2016\tvi\t11\t13332\neuro2016\tfi\t11\t13332\neuro2016\tht\t10\t13332\neuro2016\tlt\t10\t13332\neuro2016\tro\t10\t13332\neuro2016\tda\t9\t13332\neuro2016\tlv\t6\t13332\neuro2016\tne\t4\t13332\neuro2016\tis\t4\t13332\neuro2016\tfa\t4\t13332\neuro2016\tbg\t4\t13332\neuro2016\tsr\t3\t13332\neuro2016\thu\t3\t13332\neuro2016\tko\t2\t13332\neuro2016\tiw\t1\t13332\neuro2016\tzh\t1\t13332\ntur\ttr\t8681\t12750\ntur\tund\t3205\t12750\ntur\tfr\t216\t12750\ntur\ten\t180\t12750\ntur\tin\t88\t12750\ntur\tes\t36\t12750\ntur\tde\t36\t12750\ntur\teu\t31\t12750\ntur\tro\t25\t12750\ntur\tht\t24\t12750\ntur\thu\t23\t12750\ntur\ttl\t20\t12750\ntur\tvi\t19\t12750\ntur\tpt\t17\t12750\ntur\tet\t15\t12750\ntur\tfi\t15\t12750\ntur\tda\t14\t12750\ntur\tit\t12\t12750\ntur\tar\t12\t12750\ntur\tsv\t11\t12750\ntur\tpl\t11\t12750\ntur\tlv\t10\t12750\ntur\tnl\t9\t12750\ntur\tcy\t8\t12750\ntur\tlt\t6\t12750\ntur\tno\t5\t12750\ntur\thi\t5\t12750\ntur\tis\t5\t12750\ntur\tsl\t4\t12750\ntur\tcs\t4\t12750\ntur\tel\t1\t12750\ntur\tja\t1\t12750\ntur\tth\t1\t12750\nmtvpopcd9\tund\t6922\t12332\nmtvpopcd9\tes\t4310\t12332\nmtvpopcd9\tin\t397\t12332\nmtvpopcd9\tpt\t288\t12332\nmtvpopcd9\ten\t239\t12332\nmtvpopcd9\tfr\t30\t12332\nmtvpopcd9\tet\t19\t12332\nmtvpopcd9\tsv\t18\t12332\nmtvpopcd9\tpl\t17\t12332\nmtvpopcd9\tit\t17\t12332\nmtvpopcd9\ttr\t16\t12332\nmtvpopcd9\tde\t13\t12332\nmtvpopcd9\teu\t12\t12332\nmtvpopcd9\tro\t10\t12332\nmtvpopcd9\tlt\t8\t12332\nmtvpopcd9\thu\t7\t12332\nmtvpopcd9\tno\t4\t12332\nmtvpopcd9\ttl\t3\t12332\nmtvpopcd9\tlv\t1\t12332\nmtvpopcd9\tht\t1\t12332\nmtvsnapmexdanna\ten\t6393\t9200\nmtvsnapmexdanna\tund\t882\t9200\nmtvsnapmexdanna\tes\t276\t9200\nmtvsnapmexdanna\ttr\t225\t9200\nmtvsnapmexdanna\tin\t218\t9200\nmtvsnapmexdanna\tde\t122\t9200\nmtvsnapmexdanna\tit\t102\t9200\nmtvsnapmexdanna\tcy\t90\t9200\nmtvsnapmexdanna\tfi\t90\t9200\nmtvsnapmexdanna\tht\t88\t9200\nmtvsnapmexdanna\ttl\t85\t9200\nmtvsnapmexdanna\tnl\t84\t9200\nmtvsnapmexdanna\tvi\t70\t9200\nmtvsnapmexdanna\tda\t70\t9200\nmtvsnapmexdanna\tcs\t64\t9200\nmtvsnapmexdanna\tpl\t45\t9200\nmtvsnapmexdanna\tno\t45\t9200\nmtvsnapmexdanna\tet\t45\t9200\nmtvsnapmexdanna\thi\t35\t9200\nmtvsnapmexdanna\tsv\t35\t9200\nmtvsnapmexdanna\tlt\t30\t9200\nmtvsnapmexdanna\tpt\t26\t9200\nmtvsnapmexdanna\tlv\t25\t9200\nmtvsnapmexdanna\teu\t15\t9200\nmtvsnapmexdanna\tfr\t15\t9200\nmtvsnapmexdanna\tis\t10\t9200\nmtvsnapmexdanna\thu\t5\t9200\nmtvsnapmexdanna\tro\t5\t9200\nmtvsnapmexdanna\tsl\t5\t9200\nripchristinagrimmie\ten\t6206\t7949\nripchristinagrimmie\tund\t717\t7949\nripchristinagrimmie\tth\t469\t7949\nripchristinagrimmie\tes\t235\t7949\nripchristinagrimmie\tpt\t76\t7949\nripchristinagrimmie\ttl\t69\t7949\nripchristinagrimmie\tfr\t63\t7949\nripchristinagrimmie\tit\t25\t7949\nripchristinagrimmie\tde\t20\t7949\nripchristinagrimmie\tin\t17\t7949\nripchristinagrimmie\tja\t16\t7949\nripchristinagrimmie\tpl\t13\t7949\nripchristinagrimmie\tnl\t5\t7949\nripchristinagrimmie\ttr\t4\t7949\nripchristinagrimmie\tet\t3\t7949\nripchristinagrimmie\tar\t2\t7949\nripchristinagrimmie\tis\t2\t7949\nripchristinagrimmie\tru\t2\t7949\nripchristinagrimmie\tne\t1\t7949\nripchristinagrimmie\tzh\t1\t7949\nripchristinagrimmie\tcs\t1\t7949\nripchristinagrimmie\tfi\t1\t7949\nripchristinagrimmie\tko\t1\t7949\norangesponsorsyou\ttr\t5387\t6800\norangesponsorsyou\tund\t973\t6800\norangesponsorsyou\ten\t77\t6800\norangesponsorsyou\tfr\t56\t6800\norangesponsorsyou\tin\t47\t6800\norangesponsorsyou\tde\t28\t6800\norangesponsorsyou\teu\t28\t6800\norangesponsorsyou\thu\t18\t6800\norangesponsorsyou\tes\t18\t6800\norangesponsorsyou\tvi\t15\t6800\norangesponsorsyou\tda\t15\t6800\norangesponsorsyou\tht\t14\t6800\norangesponsorsyou\tpl\t13\t6800\norangesponsorsyou\tfi\t13\t6800\norangesponsorsyou\tpt\t12\t6800\norangesponsorsyou\ttl\t10\t6800\norangesponsorsyou\tet\t10\t6800\norangesponsorsyou\tit\t9\t6800\norangesponsorsyou\tlv\t8\t6800\norangesponsorsyou\tnl\t8\t6800\norangesponsorsyou\tcy\t6\t6800\norangesponsorsyou\tsv\t5\t6800\norangesponsorsyou\tlt\t4\t6800\norangesponsorsyou\tcs\t4\t6800\norangesponsorsyou\tno\t4\t6800\norangesponsorsyou\tro\t4\t6800\norangesponsorsyou\tsl\t4\t6800\norangesponsorsyou\thi\t4\t6800\norangesponsorsyou\tis\t3\t6800\norangesponsorsyou\tar\t2\t6800\norangesponsorsyou\tel\t1\t6800\nprayforchristina\ten\t4072\t5986\nprayforchristina\tund\t625\t5986\nprayforchristina\tth\t580\t5986\nprayforchristina\tes\t185\t5986\nprayforchristina\tfr\t133\t5986\nprayforchristina\tpt\t129\t5986\nprayforchristina\tpl\t124\t5986\nprayforchristina\tit\t36\t5986\nprayforchristina\tja\t35\t5986\nprayforchristina\ttl\t32\t5986\nprayforchristina\tar\t5\t5986\nprayforchristina\ttr\t5\t5986\nprayforchristina\tde\t5\t5986\nprayforchristina\tru\t4\t5986\nprayforchristina\tin\t4\t5986\nprayforchristina\tnl\t3\t5986\nprayforchristina\tda\t2\t5986\nprayforchristina\tet\t2\t5986\nprayforchristina\teu\t1\t5986\nprayforchristina\tlt\t1\t5986\nprayforchristina\tvi\t1\t5986\nprayforchristina\tko\t1\t5986\nprayforchristina\tfi\t1\t5986\neng\ten\t4001\t5594\neng\tes\t309\t5594\neng\tfr\t267\t5594\neng\tund\t263\t5594\neng\tar\t189\t5594\neng\ttr\t104\t5594\neng\tin\t72\t5594\neng\tde\t63\t5594\neng\tru\t55\t5594\neng\tit\t46\t5594\neng\tja\t34\t5594\neng\tpt\t32\t5594\neng\tnl\t27\t5594\neng\tth\t18\t5594\neng\tpl\t15\t5594\neng\ttl\t14\t5594\neng\tfi\t8\t5594\neng\tsv\t8\t5594\neng\tfa\t7\t5594\neng\tno\t6\t5594\neng\tro\t6\t5594\neng\tet\t6\t5594\neng\tsl\t5\t5594\neng\tis\t5\t5594\neng\tcy\t4\t5594\neng\tsr\t4\t5594\neng\tda\t4\t5594\neng\tlt\t3\t5594\neng\tuk\t3\t5594\neng\tht\t3\t5594\neng\tbg\t2\t5594\neng\tlv\t2\t5594\neng\tel\t2\t5594\neng\thu\t1\t5594\neng\teu\t1\t5594\neng\tiw\t1\t5594\neng\tne\t1\t5594\neng\tzh\t1\t5594\neng\tcs\t1\t5594\neng\thi\t1\t5594\nmtvsnapcolmartinez\tes\t4383\t5364\nmtvsnapcolmartinez\tund\t965\t5364\nmtvsnapcolmartinez\ten\t10\t5364\nmtvsnapcolmartinez\tpt\t2\t5364\nmtvsnapcolmartinez\tnl\t1\t5364\nmtvsnapcolmartinez\tin\t1\t5364\nmtvsnapcolmartinez\tit\t1\t5364\nmtvsnapcolmartinez\ttr\t1\t5364\naldubebtrumpets\ten\t2363\t4902\naldubebtrumpets\ttl\t2177\t4902\naldubebtrumpets\tund\t106\t4902\naldubebtrumpets\tin\t73\t4902\naldubebtrumpets\tnl\t23\t4902\naldubebtrumpets\tht\t23\t4902\naldubebtrumpets\tet\t19\t4902\naldubebtrumpets\tes\t19\t4902\naldubebtrumpets\tfr\t13\t4902\naldubebtrumpets\tsv\t12\t4902\naldubebtrumpets\teu\t11\t4902\naldubebtrumpets\tit\t9\t4902\naldubebtrumpets\tro\t9\t4902\naldubebtrumpets\tpt\t8\t4902\naldubebtrumpets\tde\t7\t4902\naldubebtrumpets\tlt\t6\t4902\naldubebtrumpets\tar\t5\t4902\naldubebtrumpets\tfi\t5\t4902\naldubebtrumpets\tpl\t3\t4902\naldubebtrumpets\tda\t3\t4902\naldubebtrumpets\tcs\t2\t4902\naldubebtrumpets\ttr\t2\t4902\naldubebtrumpets\tno\t2\t4902\naldubebtrumpets\thu\t1\t4902\naldubebtrumpets\tja\t1\t4902\nmtvmatchnavarro\tund\t2105\t4440\nmtvmatchnavarro\tes\t1950\t4440\nmtvmatchnavarro\tpt\t164\t4440\nmtvmatchnavarro\ten\t85\t4440\nmtvmatchnavarro\tin\t69\t4440\nmtvmatchnavarro\tet\t12\t4440\nmtvmatchnavarro\tfr\t11\t4440\nmtvmatchnavarro\tht\t8\t4440\nmtvmatchnavarro\tit\t7\t4440\nmtvmatchnavarro\teu\t5\t4440\nmtvmatchnavarro\ttr\t5\t4440\nmtvmatchnavarro\tde\t4\t4440\nmtvmatchnavarro\tno\t3\t4440\nmtvmatchnavarro\ttl\t3\t4440\nmtvmatchnavarro\tlt\t2\t4440\nmtvmatchnavarro\tro\t2\t4440\nmtvmatchnavarro\tpl\t2\t4440\nmtvmatchnavarro\tlv\t1\t4440\nmtvmatchnavarro\tsv\t1\t4440\nmtvmatchnavarro\tda\t1\t4440\ngameinsight\ten\t1867\t3950\ngameinsight\tru\t1126\t3950\ngameinsight\tde\t330\t3950\ngameinsight\tfr\t158\t3950\ngameinsight\tes\t144\t3950\ngameinsight\tund\t77\t3950\ngameinsight\tzh\t66\t3950\ngameinsight\tit\t59\t3950\ngameinsight\tko\t53\t3950\ngameinsight\ttr\t35\t3950\ngameinsight\tja\t21\t3950\ngameinsight\tpt\t8\t3950\ngameinsight\tcs\t5\t3950\ngameinsight\tnl\t1\t3950\nرمضان\tar\t3748\t3911\nرمضان\tund\t131\t3911\nرمضان\tfa\t13\t3911\nرمضان\tur\t8\t3911\nرمضان\ten\t8\t3911\nرمضان\thi\t1\t3911\nرمضان\tja\t1\t3911\nرمضان\tfr\t1\t3911\nteenchoice\ten\t3396\t3645\nteenchoice\tund\t100\t3645\nteenchoice\tes\t52\t3645\nteenchoice\tpt\t46\t3645\nteenchoice\tfr\t16\t3645\nteenchoice\ttr\t12\t3645\nteenchoice\tpl\t9\t3645\nteenchoice\tcy\t4\t3645\nteenchoice\tar\t3\t3645\nteenchoice\tit\t2\t3645\nteenchoice\tde\t2\t3645\nteenchoice\tno\t1\t3645\nteenchoice\tht\t1\t3645\nteenchoice\tcs\t1\t3645\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tja\t3321\t3633\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tund\t171\t3633\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\ten\t101\t3633\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tda\t9\t3633\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tpt\t9\t3633\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\thu\t8\t3633\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tnl\t6\t3633\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tin\t5\t3633\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tet\t2\t3633\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tko\t1\t3633\n"}]},"apps":[],"jobName":"paragraph_1523635872367_-2147090548","id":"20180413-181112_509102729","dateCreated":"2018-04-13T18:11:12+0200","dateStarted":"2018-04-16T21:11:02+0200","dateFinished":"2018-04-16T21:23:11+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9381"},{"text":"%md\n### Question II.d. (5/30)\n\nNow, working on the whole twitter dataset, perform the same as in questions II.b. and II.c.","dateUpdated":"2018-03-28T13:15:09+0200","config":{"tableHide":false,"editorSetting":{"editOnDblClick":true,"language":"markdown"},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Question II.d. (5/30)</h3>\n<p>Now, working on the whole twitter dataset, perform the same as in questions II.b. and II.c.</p>\n"}]},"apps":[],"jobName":"paragraph_1522235709504_1421455751","id":"20180326-213348_208498637","dateCreated":"2018-03-28T13:15:09+0200","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:9382"},{"text":"%md\n### Answers\n\nFor this question, we take the same approach as for the II.b and II.c.\n\nMeaning, we first create an intermediary table with the values of the hashtags.","user":"musuroi","dateUpdated":"2018-04-13T18:02:54+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Answers</h3>\n<p>For this question, we take the same approach as for the II.b and II.c.</p>\n<p>Meaning, we first create an intermediary table with the values of the hashtags.</p>\n"}]},"apps":[],"jobName":"paragraph_1523635151065_-1082303686","id":"20180413-175911_1829843412","dateCreated":"2018-04-13T17:59:11+0200","dateStarted":"2018-04-13T18:02:54+0200","dateFinished":"2018-04-13T18:02:54+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9383"},{"text":"%jdbc(hive)\n\nDROP TABLE IF EXISTS musuroi.intermediary_hashtags;\n\nCREATE TABLE musuroi.intermediary_hashtags STORED AS parquet AS\n SELECT timestamp_ms,\n lang,\n regexp_replace(get_json_object(entities, '$.hashtags'),'\\\\{\\\"text\\\":(\\\"[^\\\"]+\\\"),\\\"indices\\\":\\\\[[0-9]+,[0-9]+\\\\]\\\\}','$1') as hashtags\n FROM baetu.twitter\n WHERE get_json_object(entities, '$.hashtags') IS NOT NULL AND get_json_object(entities, '$.hashtags') != '[]';\n ","user":"musuroi","dateUpdated":"2018-04-13T18:08:41+0200","config":{"colWidth":12,"editorMode":"ace/mode/sql","results":{},"enabled":true,"editorSetting":{"editOnDblClick":false,"language":"sql"}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"},{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"}]},"apps":[],"jobName":"paragraph_1522235709505_1421071002","id":"20180326-213452_1466399305","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-13T18:08:42+0200","dateFinished":"2018-04-13T18:43:41+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9384"},{"text":"%md\nThen we create the actual hashtags table with the exploded hashtag values.","user":"musuroi","dateUpdated":"2018-04-13T19:11:34+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>Then we create the actual hashtags table with the exploded hashtag values.</p>\n"}]},"apps":[],"jobName":"paragraph_1523638670018_-1015179542","id":"20180413-185750_445607395","dateCreated":"2018-04-13T18:57:50+0200","dateStarted":"2018-04-13T19:11:34+0200","dateFinished":"2018-04-13T19:11:34+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9385"},{"text":"%jdbc(hive)\nCREATE TABLE musuroi.hashtags STORED AS parquet AS\n SELECT timestamp_ms,\n lang,\n hashtag\n FROM musuroi.intermediary_hashtags\n LATERAL VIEW explode(split(substr(hashtags, 3, length(hashtags) - 4), '\\\",\\\"')) hash_table as hashtag;","user":"musuroi","dateUpdated":"2018-04-13T18:55:12+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Query executed successfully. Affected rows : -1"}]},"apps":[],"jobName":"paragraph_1523635617302_1944746297","id":"20180413-180657_1542656944","dateCreated":"2018-04-13T18:06:57+0200","dateStarted":"2018-04-13T18:55:05+0200","dateFinished":"2018-04-13T18:56:08+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9386"},{"text":"%jdbc(hive)\nSELECT * FROM musuroi.hashtags\n LIMIT 10;\n ","user":"musuroi","dateUpdated":"2018-04-13T19:21:11+0200","config":{"colWidth":12,"enabled":true,"results":{"0":{"graph":{"mode":"table","height":300,"optionOpen":false},"helium":{}}},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"hashtags.timestamp_ms\thashtags.lang\thashtags.hashtag\n1459519918660\tth\tDescendantsOfTheSun\n1459519918659\tin\txolaykepo\n1459519918660\tja\tæ¤ç‰©å›³é‘‘\n1459519918662\ten\tJFCShawnMendes\n1459519918662\ten\tJUNOS\n1459519918663\ten\tStarWarsEpisodeVIII\n1459519918662\tja\tRTã•ã‚ŒãŸæ•°ã ã‘創作ã‚ャラを紹介ã™ã‚‹\n1459519918664\tko\tì°½ì„\n1459519918659\tund\tShugetsu\n1459519918659\tund\tTsukemen\n"}]},"apps":[],"jobName":"paragraph_1523639646784_388595695","id":"20180413-191406_1702113862","dateCreated":"2018-04-13T19:14:06+0200","dateStarted":"2018-04-13T19:18:04+0200","dateFinished":"2018-04-13T19:18:05+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9387"},{"text":"%md\nNow, we can move to counting.\n\nFirst, we count the most represented languages.","user":"musuroi","dateUpdated":"2018-04-13T19:11:51+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>Now, we can move to counting.</p>\n<p>First, we count the most represented languages.</p>\n"}]},"apps":[],"jobName":"paragraph_1523638632074_-910405248","id":"20180413-185712_387509581","dateCreated":"2018-04-13T18:57:12+0200","dateStarted":"2018-04-13T19:11:51+0200","dateFinished":"2018-04-13T19:11:51+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9388"},{"text":"%jdbc(hive)\nSELECT lang, COUNT(*) as counts\nFROM musuroi.hashtags\nGROUP BY lang\nORDER BY counts DESC\nLIMIT 10;\n","user":"musuroi","dateUpdated":"2018-04-13T19:13:07+0200","config":{"colWidth":12,"enabled":true,"results":{"0":{"graph":{"mode":"multiBarChart","height":300,"optionOpen":false},"helium":{}}},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"lang\tcounts\nen\t55944928\nja\t17273220\nko\t12769746\nes\t12713258\nund\t12600487\nar\t12064416\nth\t4304887\npt\t3431333\nfr\t2789404\nin\t2516291\n"}]},"apps":[],"jobName":"paragraph_1523638715354_74121215","id":"20180413-185835_226009742","dateCreated":"2018-04-13T18:58:35+0200","dateStarted":"2018-04-13T19:11:57+0200","dateFinished":"2018-04-13T19:13:04+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9389"},{"text":"%md \nNow, we move to displaying the top 20 most popular hashtags with the contributions per language.","user":"musuroi","dateUpdated":"2018-04-16T21:11:12+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown"},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1523905803033_1856272110","id":"20180416-211003_1588949929","dateCreated":"2018-04-16T21:10:03+0200","status":"READY","progressUpdateIntervalMs":500,"$$hashKey":"object:9390"},{"text":"%jdbc(hive)\nWITH popular_hashtags AS\n (\n SELECT LOWER(hashtag) as hashtag, COUNT(*) AS counts_all FROM musuroi.hashtags\n GROUP BY LOWER(hashtag)\n ORDER BY counts_all DESC\n LIMIT 20\n ),\n h AS\n (\n SELECT * from musuroi.hashtags\n )\nSELECT LOWER(h.hashtag) as hashtag, h.lang, COUNT(*) AS counts, p.counts_all FROM h\n JOIN popular_hashtags p ON lower(h.hashtag) = lower(p.hashtag) \n GROUP BY LOWER(h.hashtag), h.lang, p.counts_all\n ORDER BY p.counts_all DESC, counts DESC;","user":"musuroi","dateUpdated":"2018-04-16T22:10:47+0200","config":{"colWidth":12,"enabled":true,"results":{"0":{"graph":{"mode":"multiBarChart","height":300,"optionOpen":false,"setting":{"multiBarChart":{"stacked":true}},"commonSetting":{},"keys":[{"name":"hashtag","index":0,"aggr":"sum"}],"groups":[{"name":"h.lang","index":1,"aggr":"sum"}],"values":[{"name":"counts","index":2,"aggr":"sum"}]},"helium":{}}},"editorSetting":{"language":"sql","editOnDblClick":false},"editorMode":"ace/mode/sql"},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"hashtag\th.lang\tcounts\tp.counts_all\nmtvhottest\ten\t800829\t1308625\nmtvhottest\tes\t165822\t1308625\nmtvhottest\tpt\t132616\t1308625\nmtvhottest\tpl\t32785\t1308625\nmtvhottest\tit\t17333\t1308625\nmtvhottest\tis\t17133\t1308625\nmtvhottest\tund\t15364\t1308625\nmtvhottest\tfi\t15140\t1308625\nmtvhottest\ttr\t13514\t1308625\nmtvhottest\tht\t13024\t1308625\nmtvhottest\tin\t12605\t1308625\nmtvhottest\tet\t11252\t1308625\nmtvhottest\tro\t9485\t1308625\nmtvhottest\tfr\t9392\t1308625\nmtvhottest\thu\t5893\t1308625\nmtvhottest\tde\t5588\t1308625\nmtvhottest\ttl\t4956\t1308625\nmtvhottest\tcs\t4432\t1308625\nmtvhottest\tda\t4341\t1308625\nmtvhottest\tsv\t3193\t1308625\nmtvhottest\tnl\t2831\t1308625\nmtvhottest\tno\t2409\t1308625\nmtvhottest\tja\t1914\t1308625\nmtvhottest\teu\t1549\t1308625\nmtvhottest\tcy\t1473\t1308625\nmtvhottest\tlt\t956\t1308625\nmtvhottest\tsl\t772\t1308625\nmtvhottest\tru\t553\t1308625\nmtvhottest\tth\t466\t1308625\nmtvhottest\thi\t405\t1308625\nmtvhottest\tvi\t307\t1308625\nmtvhottest\tlv\t241\t1308625\nmtvhottest\tar\t18\t1308625\nmtvhottest\tbg\t13\t1308625\nmtvhottest\tuk\t8\t1308625\nmtvhottest\tko\t8\t1308625\nmtvhottest\tzh\t5\t1308625\niheartawards\ten\t447651\t781171\niheartawards\tund\t137987\t781171\niheartawards\tpt\t72399\t781171\niheartawards\tes\t47700\t781171\niheartawards\tlv\t24400\t781171\niheartawards\tru\t7451\t781171\niheartawards\tfr\t4612\t781171\niheartawards\ttl\t4471\t781171\niheartawards\tin\t3839\t781171\niheartawards\tit\t3566\t781171\niheartawards\tpl\t3365\t781171\niheartawards\tht\t2750\t781171\niheartawards\ttr\t2645\t781171\niheartawards\tno\t2435\t781171\niheartawards\tth\t1746\t781171\niheartawards\tcy\t1696\t781171\niheartawards\tsv\t1656\t781171\niheartawards\tnl\t1603\t781171\niheartawards\tde\t1530\t781171\niheartawards\tet\t1079\t781171\niheartawards\tro\t979\t781171\niheartawards\tda\t868\t781171\niheartawards\tfi\t770\t781171\niheartawards\teu\t703\t781171\niheartawards\tcs\t690\t781171\niheartawards\tlt\t533\t781171\niheartawards\thu\t388\t781171\niheartawards\tsl\t360\t781171\niheartawards\tvi\t294\t781171\niheartawards\tiw\t276\t781171\niheartawards\thi\t208\t781171\niheartawards\tis\t179\t781171\niheartawards\tja\t120\t781171\niheartawards\tuk\t99\t781171\niheartawards\tbg\t87\t781171\niheartawards\tsr\t14\t781171\niheartawards\tar\t9\t781171\niheartawards\tel\t5\t781171\niheartawards\tzh\t4\t781171\niheartawards\tko\t3\t781171\niheartawards\tfa\t1\t781171\nbestfanarmy\ten\t429739\t738116\nbestfanarmy\tund\t118512\t738116\nbestfanarmy\tpt\t69526\t738116\nbestfanarmy\tes\t47415\t738116\nbestfanarmy\tlv\t24419\t738116\nbestfanarmy\tru\t7481\t738116\nbestfanarmy\tfr\t4224\t738116\nbestfanarmy\ttl\t4219\t738116\nbestfanarmy\tin\t3641\t738116\nbestfanarmy\tit\t3368\t738116\nbestfanarmy\tpl\t3168\t738116\nbestfanarmy\tht\t2668\t738116\nbestfanarmy\ttr\t2517\t738116\nbestfanarmy\tno\t2418\t738116\nbestfanarmy\tth\t1715\t738116\nbestfanarmy\tcy\t1670\t738116\nbestfanarmy\tsv\t1564\t738116\nbestfanarmy\tnl\t1487\t738116\nbestfanarmy\tde\t1407\t738116\nbestfanarmy\tro\t963\t738116\nbestfanarmy\tet\t959\t738116\nbestfanarmy\tda\t816\t738116\nbestfanarmy\tfi\t729\t738116\nbestfanarmy\teu\t674\t738116\nbestfanarmy\tcs\t605\t738116\nbestfanarmy\tsl\t376\t738116\nbestfanarmy\thu\t338\t738116\nbestfanarmy\tlt\t282\t738116\nbestfanarmy\tvi\t281\t738116\nbestfanarmy\tiw\t276\t738116\nbestfanarmy\thi\t196\t738116\nbestfanarmy\tis\t162\t738116\nbestfanarmy\tuk\t100\t738116\nbestfanarmy\tbg\t87\t738116\nbestfanarmy\tja\t82\t738116\nbestfanarmy\tsr\t13\t738116\nbestfanarmy\tar\t6\t738116\nbestfanarmy\tel\t5\t738116\nbestfanarmy\tzh\t4\t738116\nbestfanarmy\tko\t3\t738116\nbestfanarmy\tfa\t1\t738116\nveranomtv2016\ten\t539028\t705633\nveranomtv2016\tpt\t47643\t705633\nveranomtv2016\tes\t45506\t705633\nveranomtv2016\tpl\t15795\t705633\nveranomtv2016\ttr\t8382\t705633\nveranomtv2016\tin\t7297\t705633\nveranomtv2016\tit\t7008\t705633\nveranomtv2016\tde\t6981\t705633\nveranomtv2016\tund\t5297\t705633\nveranomtv2016\tcs\t3162\t705633\nveranomtv2016\tfi\t2590\t705633\nveranomtv2016\tno\t2166\t705633\nveranomtv2016\tda\t2010\t705633\nveranomtv2016\tro\t1814\t705633\nveranomtv2016\tnl\t1779\t705633\nveranomtv2016\ttl\t1773\t705633\nveranomtv2016\tsv\t1614\t705633\nveranomtv2016\tfr\t1443\t705633\nveranomtv2016\tcy\t1037\t705633\nveranomtv2016\teu\t769\t705633\nveranomtv2016\tet\t739\t705633\nveranomtv2016\tht\t646\t705633\nveranomtv2016\tlt\t301\t705633\nveranomtv2016\tru\t249\t705633\nveranomtv2016\tsl\t174\t705633\nveranomtv2016\thu\t152\t705633\nveranomtv2016\tis\t92\t705633\nveranomtv2016\tvi\t64\t705633\nveranomtv2016\tth\t42\t705633\nveranomtv2016\thi\t32\t705633\nveranomtv2016\tja\t27\t705633\nveranomtv2016\tuk\t13\t705633\nveranomtv2016\tlv\t6\t705633\nveranomtv2016\tar\t1\t705633\nveranomtv2016\tko\t1\t705633\nkcamexico\tes\t210170\t520387\nkcamexico\tund\t141867\t520387\nkcamexico\tpt\t88699\t520387\nkcamexico\ten\t52645\t520387\nkcamexico\tit\t5009\t520387\nkcamexico\tfr\t4178\t520387\nkcamexico\tin\t3628\t520387\nkcamexico\ttr\t1477\t520387\nkcamexico\ttl\t1429\t520387\nkcamexico\tru\t1148\t520387\nkcamexico\tht\t992\t520387\nkcamexico\tet\t925\t520387\nkcamexico\tde\t856\t520387\nkcamexico\tcy\t819\t520387\nkcamexico\tpl\t803\t520387\nkcamexico\tnl\t735\t520387\nkcamexico\teu\t616\t520387\nkcamexico\thu\t525\t520387\nkcamexico\tsv\t523\t520387\nkcamexico\tlt\t439\t520387\nkcamexico\tro\t421\t520387\nkcamexico\tlv\t387\t520387\nkcamexico\tja\t338\t520387\nkcamexico\tcs\t325\t520387\nkcamexico\tfi\t297\t520387\nkcamexico\tda\t291\t520387\nkcamexico\tis\t283\t520387\nkcamexico\tno\t205\t520387\nkcamexico\tsl\t120\t520387\nkcamexico\tvi\t112\t520387\nkcamexico\thi\t79\t520387\nkcamexico\tuk\t18\t520387\nkcamexico\tth\t15\t520387\nkcamexico\tbg\t4\t520387\nkcamexico\tzh\t3\t520387\nkcamexico\tar\t2\t520387\nkcamexico\tsr\t1\t520387\nkcamexico\tko\t1\t520387\nkcamexico\tiw\t1\t520387\nkcamexico\tel\t1\t520387\n방탄소년단\tko\t162337\t492506\n방탄소년단\ten\t76471\t492506\n방탄소년단\tpt\t48565\t492506\n방탄소년단\ttr\t44879\t492506\n방탄소년단\tund\t44677\t492506\n방탄소년단\tru\t39146\t492506\n방탄소년단\tja\t18511\t492506\n방탄소년단\tth\t16085\t492506\n방탄소년단\tes\t8166\t492506\n방탄소년단\tin\t6388\t492506\n방탄소년단\ttl\t4795\t492506\n방탄소년단\tuk\t2680\t492506\n방탄소년단\tfr\t1941\t492506\n방탄소년단\tzh\t1765\t492506\n방탄소년단\tet\t1725\t492506\n방탄소년단\tht\t1495\t492506\n방탄소년단\tit\t1461\t492506\n방탄소년단\tar\t1078\t492506\n방탄소년단\tpl\t1046\t492506\n방탄소년단\tro\t1015\t492506\n방탄소년단\tsv\t862\t492506\n방탄소년단\tnl\t843\t492506\n방탄소년단\tde\t785\t492506\n방탄소년단\tda\t673\t492506\n방탄소년단\teu\t610\t492506\n방탄소년단\tfi\t508\t492506\n방탄소년단\tvi\t482\t492506\n방탄소년단\tbg\t478\t492506\n방탄소년단\tcs\t459\t492506\n방탄소년단\thu\t380\t492506\n방탄소년단\tsl\t379\t492506\n방탄소년단\tno\t365\t492506\n방탄소년단\tlt\t337\t492506\n방탄소년단\tcy\t318\t492506\n방탄소년단\thi\t221\t492506\n방탄소년단\tsr\t191\t492506\n방탄소년단\tlv\t146\t492506\n방탄소년단\tis\t118\t492506\n방탄소년단\thy\t67\t492506\n방탄소년단\tbn\t36\t492506\n방탄소년단\tel\t11\t492506\n방탄소년단\tfa\t8\t492506\n방탄소년단\tta\t1\t492506\n방탄소년단\tur\t1\t492506\n방탄소년단\tiw\t1\t492506\ngameinsight\ten\t213772\t463485\ngameinsight\tru\t139552\t463485\ngameinsight\tde\t38909\t463485\ngameinsight\tfr\t19771\t463485\ngameinsight\tes\t17250\t463485\ngameinsight\tko\t7252\t463485\ngameinsight\tzh\t6843\t463485\ngameinsight\tit\t6746\t463485\ngameinsight\tund\t6253\t463485\ngameinsight\tja\t3239\t463485\ngameinsight\ttr\t2366\t463485\ngameinsight\tpt\t1008\t463485\ngameinsight\tcs\t323\t463485\ngameinsight\tar\t72\t463485\ngameinsight\tnl\t70\t463485\ngameinsight\tbg\t17\t463485\ngameinsight\tht\t16\t463485\ngameinsight\tin\t11\t463485\ngameinsight\tro\t3\t463485\ngameinsight\ttl\t3\t463485\ngameinsight\tlt\t2\t463485\ngameinsight\tno\t2\t463485\ngameinsight\tet\t1\t463485\ngameinsight\tvi\t1\t463485\ngameinsight\tlv\t1\t463485\ngameinsight\tda\t1\t463485\ngameinsight\thu\t1\t463485\nteenchoice\ten\t345250\t396993\nteenchoice\tes\t19641\t396993\nteenchoice\tpt\t11706\t396993\nteenchoice\tund\t10624\t396993\nteenchoice\tar\t1872\t396993\nteenchoice\tet\t1064\t396993\nteenchoice\tin\t959\t396993\nteenchoice\tfr\t772\t396993\nteenchoice\tth\t756\t396993\nteenchoice\tit\t491\t396993\nteenchoice\tpl\t483\t396993\nteenchoice\ttl\t457\t396993\nteenchoice\tko\t426\t396993\nteenchoice\ttr\t407\t396993\nteenchoice\tht\t340\t396993\nteenchoice\tru\t227\t396993\nteenchoice\tnl\t224\t396993\nteenchoice\tvi\t209\t396993\nteenchoice\thu\t178\t396993\nteenchoice\tde\t149\t396993\nteenchoice\tro\t132\t396993\nteenchoice\tja\t111\t396993\nteenchoice\tcs\t96\t396993\nteenchoice\tfi\t77\t396993\nteenchoice\tcy\t65\t396993\nteenchoice\tsv\t62\t396993\nteenchoice\tno\t51\t396993\nteenchoice\tda\t44\t396993\nteenchoice\teu\t26\t396993\nteenchoice\tlt\t21\t396993\nteenchoice\tis\t18\t396993\nteenchoice\tzh\t18\t396993\nteenchoice\tlv\t13\t396993\nteenchoice\thi\t12\t396993\nteenchoice\tuk\t4\t396993\nteenchoice\tsl\t4\t396993\nteenchoice\tel\t1\t396993\nteenchoice\tbg\t1\t396993\nteenchoice\tiw\t1\t396993\nteenchoice\tfa\t1\t396993\nandroid\ten\t177983\t386722\nandroid\tru\t99093\t386722\nandroid\tde\t33250\t386722\nandroid\tes\t18624\t386722\nandroid\tfr\t16438\t386722\nandroid\tpt\t8760\t386722\nandroid\tund\t7474\t386722\nandroid\tit\t5937\t386722\nandroid\tko\t5756\t386722\nandroid\tzh\t5107\t386722\nandroid\tja\t4335\t386722\nandroid\ttr\t2103\t386722\nandroid\tin\t522\t386722\nandroid\tcs\t357\t386722\nandroid\tnl\t154\t386722\nandroid\tar\t151\t386722\nandroid\tet\t105\t386722\nandroid\tda\t88\t386722\nandroid\tht\t71\t386722\nandroid\ttl\t59\t386722\nandroid\tsv\t47\t386722\nandroid\tpl\t47\t386722\nandroid\tro\t41\t386722\nandroid\tcy\t30\t386722\nandroid\thu\t26\t386722\nandroid\tlt\t24\t386722\nandroid\thi\t20\t386722\nandroid\tno\t18\t386722\nandroid\tth\t16\t386722\nandroid\tfi\t14\t386722\nandroid\tbg\t10\t386722\nandroid\tuk\t9\t386722\nandroid\tvi\t9\t386722\nandroid\tlv\t9\t386722\nandroid\tsl\t8\t386722\nandroid\teu\t7\t386722\nandroid\tfa\t5\t386722\nandroid\tgu\t4\t386722\nandroid\tel\t4\t386722\nandroid\tis\t2\t386722\nandroid\tta\t1\t386722\nandroid\tml\t1\t386722\nandroid\tbn\t1\t386722\nandroid\tsr\t1\t386722\nandroid\tmr\t1\t386722\nandroidgames\ten\t164998\t365687\nandroidgames\tru\t101047\t365687\nandroidgames\tde\t33699\t365687\nandroidgames\tfr\t15952\t365687\nandroidgames\tes\t14133\t365687\nandroidgames\tpt\t8652\t365687\nandroidgames\tko\t5999\t365687\nandroidgames\tund\t5807\t365687\nandroidgames\tit\t5323\t365687\nandroidgames\tzh\t5217\t365687\nandroidgames\tja\t2349\t365687\nandroidgames\ttr\t2034\t365687\nandroidgames\tcs\t288\t365687\nandroidgames\tar\t40\t365687\nandroidgames\tet\t38\t365687\nandroidgames\tin\t33\t365687\nandroidgames\tnl\t29\t365687\nandroidgames\tht\t19\t365687\nandroidgames\tro\t7\t365687\nandroidgames\ttl\t6\t365687\nandroidgames\tbg\t5\t365687\nandroidgames\tda\t3\t365687\nandroidgames\tpl\t3\t365687\nandroidgames\tno\t2\t365687\nandroidgames\tcy\t1\t365687\nandroidgames\thu\t1\t365687\nandroidgames\tlv\t1\t365687\nandroidgames\tlt\t1\t365687\nmtvmiaw\tes\t196163\t353608\nmtvmiaw\tund\t100015\t353608\nmtvmiaw\ten\t20800\t353608\nmtvmiaw\tpt\t7490\t353608\nmtvmiaw\tin\t7234\t353608\nmtvmiaw\tde\t6624\t353608\nmtvmiaw\tit\t2922\t353608\nmtvmiaw\tfr\t1984\t353608\nmtvmiaw\tht\t1210\t353608\nmtvmiaw\ttl\t1111\t353608\nmtvmiaw\tpl\t989\t353608\nmtvmiaw\ttr\t988\t353608\nmtvmiaw\tet\t759\t353608\nmtvmiaw\tcy\t719\t353608\nmtvmiaw\tnl\t589\t353608\nmtvmiaw\teu\t510\t353608\nmtvmiaw\tlt\t478\t353608\nmtvmiaw\tfi\t469\t353608\nmtvmiaw\tsv\t432\t353608\nmtvmiaw\tlv\t354\t353608\nmtvmiaw\tro\t287\t353608\nmtvmiaw\thu\t260\t353608\nmtvmiaw\tcs\t228\t353608\nmtvmiaw\tja\t191\t353608\nmtvmiaw\tda\t162\t353608\nmtvmiaw\tno\t148\t353608\nmtvmiaw\tis\t139\t353608\nmtvmiaw\tvi\t136\t353608\nmtvmiaw\thi\t84\t353608\nmtvmiaw\tru\t77\t353608\nmtvmiaw\tsl\t49\t353608\nmtvmiaw\tar\t2\t353608\nmtvmiaw\tbg\t1\t353608\nmtvmiaw\tzh\t1\t353608\nmtvmiaw\tko\t1\t353608\nmtvmiaw\tiw\t1\t353608\nmtvmiaw\tuk\t1\t353608\nnowplaying\ten\t240373\t337412\nnowplaying\tja\t26527\t337412\nnowplaying\tes\t13907\t337412\nnowplaying\tund\t9769\t337412\nnowplaying\tin\t6233\t337412\nnowplaying\tfr\t5910\t337412\nnowplaying\tde\t4104\t337412\nnowplaying\tht\t3842\t337412\nnowplaying\tet\t3789\t337412\nnowplaying\tpt\t3581\t337412\nnowplaying\ttl\t2749\t337412\nnowplaying\tit\t2638\t337412\nnowplaying\tnl\t1866\t337412\nnowplaying\ttr\t1072\t337412\nnowplaying\tpl\t1064\t337412\nnowplaying\tfi\t1034\t337412\nnowplaying\tda\t1016\t337412\nnowplaying\tko\t923\t337412\nnowplaying\tcy\t823\t337412\nnowplaying\tro\t793\t337412\nnowplaying\tar\t745\t337412\nnowplaying\tlt\t702\t337412\nnowplaying\tsv\t642\t337412\nnowplaying\tzh\t608\t337412\nnowplaying\tno\t522\t337412\nnowplaying\tcs\t519\t337412\nnowplaying\thi\t329\t337412\nnowplaying\teu\t273\t337412\nnowplaying\tru\t256\t337412\nnowplaying\thu\t241\t337412\nnowplaying\tth\t168\t337412\nnowplaying\tis\t143\t337412\nnowplaying\tsl\t107\t337412\nnowplaying\tlv\t59\t337412\nnowplaying\tvi\t37\t337412\nnowplaying\tuk\t21\t337412\nnowplaying\tel\t8\t337412\nnowplaying\tfa\t7\t337412\nnowplaying\tne\t4\t337412\nnowplaying\tbg\t4\t337412\nnowplaying\tkn\t1\t337412\nnowplaying\tnull\t1\t337412\nnowplaying\tsr\t1\t337412\nnowplaying\tur\t1\t337412\ngot7\tko\t147998\t315389\ngot7\ten\t62921\t315389\ngot7\tth\t54798\t315389\ngot7\tund\t25850\t315389\ngot7\tin\t7010\t315389\ngot7\tja\t4832\t315389\ngot7\tit\t2096\t315389\ngot7\ttl\t1437\t315389\ngot7\tzh\t1354\t315389\ngot7\tes\t740\t315389\ngot7\tsl\t685\t315389\ngot7\tis\t548\t315389\ngot7\tfr\t518\t315389\ngot7\tnl\t514\t315389\ngot7\tet\t507\t315389\ngot7\tht\t450\t315389\ngot7\tda\t361\t315389\ngot7\tde\t351\t315389\ngot7\tpt\t348\t315389\ngot7\tsv\t345\t315389\ngot7\tvi\t254\t315389\ngot7\teu\t199\t315389\ngot7\thu\t183\t315389\ngot7\tcy\t177\t315389\ngot7\tro\t169\t315389\ngot7\tpl\t156\t315389\ngot7\ttr\t119\t315389\ngot7\tfi\t104\t315389\ngot7\thi\t92\t315389\ngot7\tlt\t84\t315389\ngot7\tru\t61\t315389\ngot7\tno\t48\t315389\ngot7\tcs\t34\t315389\ngot7\tlv\t20\t315389\ngot7\tar\t17\t315389\ngot7\tel\t7\t315389\ngot7\tmy\t1\t315389\ngot7\tsr\t1\t315389\nrt\ten\t161109\t310673\nrt\tfr\t27553\t310673\nrt\tund\t27158\t310673\nrt\tes\t25257\t310673\nrt\tja\t17972\t310673\nrt\tpt\t9777\t310673\nrt\tru\t7789\t310673\nrt\tpl\t6955\t310673\nrt\tit\t6236\t310673\nrt\tko\t5516\t310673\nrt\tin\t2608\t310673\nrt\tnl\t1402\t310673\nrt\ttl\t1270\t310673\nrt\tda\t1220\t310673\nrt\thu\t1123\t310673\nrt\tth\t1067\t310673\nrt\tde\t997\t310673\nrt\tht\t963\t310673\nrt\ttr\t785\t310673\nrt\tar\t696\t310673\nrt\tlv\t572\t310673\nrt\tet\t378\t310673\nrt\tno\t356\t310673\nrt\tuk\t344\t310673\nrt\tsv\t302\t310673\nrt\thi\t199\t310673\nrt\tcs\t172\t310673\nrt\tro\t135\t310673\nrt\tlt\t109\t310673\nrt\tfi\t95\t310673\nrt\teu\t91\t310673\nrt\tcy\t91\t310673\nrt\tsl\t81\t310673\nrt\tbg\t63\t310673\nrt\tur\t49\t310673\nrt\tta\t43\t310673\nrt\tne\t33\t310673\nrt\tis\t23\t310673\nrt\tzh\t23\t310673\nrt\tvi\t18\t310673\nrt\tsr\t12\t310673\nrt\tel\t8\t310673\nrt\tbn\t7\t310673\nrt\tfa\t5\t310673\nrt\tml\t3\t310673\nrt\tnull\t2\t310673\nrt\tiw\t2\t310673\nrt\tmr\t2\t310673\nrt\tte\t1\t310673\nrt\thy\t1\t310673\nmtvawardsstar\ten\t109633\t299397\nmtvawardsstar\tit\t99608\t299397\nmtvawardsstar\tpt\t17967\t299397\nmtvawardsstar\tund\t14373\t299397\nmtvawardsstar\tes\t13046\t299397\nmtvawardsstar\tru\t6888\t299397\nmtvawardsstar\tfr\t6024\t299397\nmtvawardsstar\tht\t5543\t299397\nmtvawardsstar\ttl\t4745\t299397\nmtvawardsstar\tin\t4644\t299397\nmtvawardsstar\tet\t3156\t299397\nmtvawardsstar\ttr\t2855\t299397\nmtvawardsstar\tcs\t1927\t299397\nmtvawardsstar\tja\t1400\t299397\nmtvawardsstar\tde\t1188\t299397\nmtvawardsstar\tpl\t1077\t299397\nmtvawardsstar\tfi\t1007\t299397\nmtvawardsstar\tlv\t684\t299397\nmtvawardsstar\tno\t548\t299397\nmtvawardsstar\tlt\t493\t299397\nmtvawardsstar\tro\t446\t299397\nmtvawardsstar\tnl\t412\t299397\nmtvawardsstar\tda\t347\t299397\nmtvawardsstar\teu\t277\t299397\nmtvawardsstar\tsv\t198\t299397\nmtvawardsstar\tcy\t187\t299397\nmtvawardsstar\tis\t186\t299397\nmtvawardsstar\tsl\t137\t299397\nmtvawardsstar\thi\t121\t299397\nmtvawardsstar\thu\t107\t299397\nmtvawardsstar\tuk\t56\t299397\nmtvawardsstar\tth\t41\t299397\nmtvawardsstar\tvi\t33\t299397\nmtvawardsstar\tbg\t22\t299397\nmtvawardsstar\tar\t15\t299397\nmtvawardsstar\tzh\t4\t299397\nmtvawardsstar\tsr\t2\t299397\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tja\t263364\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tund\t16300\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\ten\t7510\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tnl\t1685\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tda\t1422\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\thu\t974\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\ttl\t855\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tin\t834\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tpt\t654\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tsv\t271\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tfi\t150\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tes\t120\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tht\t118\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tit\t117\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tet\t71\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tno\t47\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tzh\t19\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tko\t18\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tcs\t17\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tsl\t16\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tde\t16\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tro\t13\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tfr\t11\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\teu\t9\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tlv\t9\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tvi\t8\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\ttr\t7\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tcy\t7\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tpl\t4\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tis\t2\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tel\t2\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\tlt\t1\t294652\nrtã—ãŸäººå…¨å“¡ãƒ•ã‚©ãƒãƒ¼ã™ã‚‹\thi\t1\t294652\nトレクル\tja\t287743\t287960\nトレクル\ten\t96\t287960\nトレクル\tund\t93\t287960\nトレクル\tht\t10\t287960\nトレクル\tde\t7\t287960\nトレクル\tpt\t6\t287960\nトレクル\tfi\t1\t287960\nトレクル\tpl\t1\t287960\nトレクル\ttl\t1\t287960\nトレクル\tar\t1\t287960\nトレクル\tit\t1\t287960\n5sosfam\ten\t170984\t286210\n5sosfam\tund\t32139\t286210\n5sosfam\tpt\t29859\t286210\n5sosfam\tlv\t24102\t286210\n5sosfam\tes\t11961\t286210\n5sosfam\tno\t2102\t286210\n5sosfam\tht\t1799\t286210\n5sosfam\tin\t1735\t286210\n5sosfam\tfr\t1439\t286210\n5sosfam\tcy\t1273\t286210\n5sosfam\ttr\t1131\t286210\n5sosfam\ttl\t1114\t286210\n5sosfam\tpl\t932\t286210\n5sosfam\tit\t886\t286210\n5sosfam\tth\t780\t286210\n5sosfam\tsv\t613\t286210\n5sosfam\tro\t593\t286210\n5sosfam\tde\t478\t286210\n5sosfam\tet\t434\t286210\n5sosfam\tda\t399\t286210\n5sosfam\tnl\t361\t286210\n5sosfam\tsl\t267\t286210\n5sosfam\tfi\t238\t286210\n5sosfam\thu\t109\t286210\n5sosfam\thi\t95\t286210\n5sosfam\tcs\t87\t286210\n5sosfam\teu\t71\t286210\n5sosfam\tis\t59\t286210\n5sosfam\tru\t50\t286210\n5sosfam\tja\t46\t286210\n5sosfam\tlt\t36\t286210\n5sosfam\tvi\t33\t286210\n5sosfam\tar\t2\t286210\n5sosfam\tzh\t1\t286210\n5sosfam\tko\t1\t286210\n5sosfam\tsr\t1\t286210\nharmonizers\ten\t162953\t277425\nharmonizers\tund\t62863\t277425\nharmonizers\tpt\t22146\t277425\nharmonizers\tes\t15801\t277425\nharmonizers\tit\t1509\t277425\nharmonizers\tin\t1314\t277425\nharmonizers\tfr\t1296\t277425\nharmonizers\ttr\t1106\t277425\nharmonizers\tnl\t831\t277425\nharmonizers\tsv\t825\t277425\nharmonizers\tpl\t660\t277425\nharmonizers\tde\t587\t277425\nharmonizers\teu\t527\t277425\nharmonizers\tcs\t507\t277425\nharmonizers\tfi\t499\t277425\nharmonizers\tht\t485\t277425\nharmonizers\tet\t402\t277425\nharmonizers\ttl\t379\t277425\nharmonizers\tda\t369\t277425\nharmonizers\tcy\t330\t277425\nharmonizers\tro\t310\t277425\nharmonizers\tno\t305\t277425\nharmonizers\tlv\t279\t277425\nharmonizers\tiw\t276\t277425\nharmonizers\tlt\t216\t277425\nharmonizers\tvi\t207\t277425\nharmonizers\thu\t190\t277425\nharmonizers\thi\t90\t277425\nharmonizers\tis\t76\t277425\nharmonizers\tsl\t69\t277425\nharmonizers\tja\t10\t277425\nharmonizers\tzh\t3\t277425\nharmonizers\tel\t2\t277425\nharmonizers\tru\t1\t277425\nharmonizers\tbg\t1\t277425\nharmonizers\tuk\t1\t277425\neuro2016\ten\t124870\t274839\neuro2016\tfr\t45245\t274839\neuro2016\tes\t28093\t274839\neuro2016\tpt\t11742\t274839\neuro2016\tund\t10122\t274839\neuro2016\tit\t8332\t274839\neuro2016\tde\t8290\t274839\neuro2016\tar\t8075\t274839\neuro2016\ttr\t7727\t274839\neuro2016\tin\t4548\t274839\neuro2016\tru\t2850\t274839\neuro2016\tja\t2760\t274839\neuro2016\tpl\t2642\t274839\neuro2016\tnl\t1657\t274839\neuro2016\tth\t1271\t274839\neuro2016\tht\t1177\t274839\neuro2016\tro\t627\t274839\neuro2016\tsv\t545\t274839\neuro2016\tel\t471\t274839\neuro2016\tet\t397\t274839\neuro2016\tcs\t339\t274839\neuro2016\tcy\t338\t274839\neuro2016\ttl\t327\t274839\neuro2016\tis\t289\t274839\neuro2016\tda\t269\t274839\neuro2016\tfi\t248\t274839\neuro2016\teu\t228\t274839\neuro2016\thu\t222\t274839\neuro2016\tlv\t190\t274839\neuro2016\tno\t180\t274839\neuro2016\tuk\t178\t274839\neuro2016\tsl\t142\t274839\neuro2016\tlt\t85\t274839\neuro2016\tne\t70\t274839\neuro2016\tfa\t66\t274839\neuro2016\thi\t54\t274839\neuro2016\tbg\t32\t274839\neuro2016\tsr\t30\t274839\neuro2016\tvi\t23\t274839\neuro2016\tzh\t17\t274839\neuro2016\tiw\t16\t274839\neuro2016\tko\t13\t274839\neuro2016\tur\t12\t274839\neuro2016\tml\t9\t274839\neuro2016\tta\t8\t274839\neuro2016\thy\t4\t274839\neuro2016\tbn\t4\t274839\neuro2016\tmr\t3\t274839\neuro2016\tka\t1\t274839\neuro2016\tsi\t1\t274839\n"}]},"apps":[],"jobName":"paragraph_1523905872620_775051079","id":"20180416-211112_661553321","dateCreated":"2018-04-16T21:11:12+0200","dateStarted":"2018-04-16T21:44:04+0200","dateFinished":"2018-04-16T22:05:52+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9391"},{"text":"%md\n### Question II.e. (5/30)\n\nDisplay the evolution of the hashtags `#EURO2016` and `#GameofThrones`. That is, display the number of mentions for each day over the range of the whole data set.\nYou should use the hastag table from the previous question.","user":"musuroi","dateUpdated":"2018-04-13T19:49:33+0200","config":{"tableHide":false,"editorSetting":{"editOnDblClick":true,"language":"markdown"},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Question II.e. (5/30)</h3>\n<p>Display the evolution of the hashtags <code>#EURO2016</code> and <code>#GameofThrones</code>. That is, display the number of mentions for each day over the range of the whole data set.\n<br />You should use the hastag table from the previous question.</p>\n"}]},"apps":[],"jobName":"paragraph_1522235709506_1422225248","id":"20180326-213500_169173111","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-13T19:49:33+0200","dateFinished":"2018-04-13T19:49:33+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9392"},{"text":"%md\n### Answers\n\nTo display the number of mentions of the two hashtags for each day, we first group the data by date - after we transformed the timestamp to date - and also group by hashtag and use count as the aggregation function.","user":"musuroi","dateUpdated":"2018-04-13T19:48:14+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown","editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Answers</h3>\n<p>To display the number of mentions of the two hashtags for each day, we first group the data by date - after we transformed the timestamp to date - and also group by hashtag and use count as the aggregation function.</p>\n"}]},"apps":[],"jobName":"paragraph_1523641299231_795510613","id":"20180413-194139_203771828","dateCreated":"2018-04-13T19:41:39+0200","dateStarted":"2018-04-13T19:48:14+0200","dateFinished":"2018-04-13T19:48:14+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9393"},{"text":"%jdbc(hive)\nSELECT from_unixtime(CAST(timestamp_ms/1000 as BIGINT), 'yyyy-MM-dd') as day, lower(hashtag) as hashtag, COUNT(*) counts FROM musuroi.hashtags \n WHERE lower(hashtag) = \"euro2016\" OR lower(hashtag) = \"gameofthrones\"\n GROUP BY from_unixtime(CAST(timestamp_ms/1000 as BIGINT), 'yyyy-MM-dd'), lower(hashtag);","user":"musuroi","dateUpdated":"2018-04-13T19:41:35+0200","config":{"colWidth":12,"editorMode":"ace/mode/sql","results":{"0":{"graph":{"mode":"lineChart","height":300,"optionOpen":false,"setting":{"multiBarChart":{"stacked":true},"scatterChart":{"xAxis":{"name":"day","index":0,"aggr":"sum"},"yAxis":{"name":"counts","index":2,"aggr":"sum"},"group":{"name":"hashtag","index":1,"aggr":"sum"}}},"keys":[{"name":"day","index":0,"aggr":"sum"}],"groups":[{"name":"hashtag","index":1,"aggr":"sum"}],"values":[{"name":"counts","index":2,"aggr":"sum"}]},"helium":{}}},"enabled":true,"editorSetting":{"editOnDblClick":false,"language":"sql"},"lineNumbers":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TABLE","data":"day\thashtag\tcounts\n2016-04-01\teuro2016\t40\n2016-04-01\tgameofthrones\t79\n2016-04-02\teuro2016\t32\n2016-04-02\tgameofthrones\t55\n2016-04-03\teuro2016\t39\n2016-04-03\tgameofthrones\t67\n2016-04-04\teuro2016\t52\n2016-04-04\tgameofthrones\t78\n2016-04-05\teuro2016\t38\n2016-04-05\tgameofthrones\t71\n2016-04-06\teuro2016\t40\n2016-04-06\tgameofthrones\t51\n2016-04-07\teuro2016\t40\n2016-04-07\tgameofthrones\t76\n2016-04-08\teuro2016\t43\n2016-04-08\tgameofthrones\t121\n2016-04-09\teuro2016\t31\n2016-04-09\tgameofthrones\t77\n2016-04-10\teuro2016\t39\n2016-04-10\tgameofthrones\t71\n2016-04-11\teuro2016\t41\n2016-04-11\tgameofthrones\t410\n2016-04-12\teuro2016\t53\n2016-04-12\tgameofthrones\t221\n2016-04-13\teuro2016\t131\n2016-04-13\tgameofthrones\t137\n2016-04-14\teuro2016\t79\n2016-04-14\tgameofthrones\t115\n2016-04-15\teuro2016\t75\n2016-04-15\tgameofthrones\t130\n2016-04-16\teuro2016\t45\n2016-04-16\tgameofthrones\t94\n2016-04-17\teuro2016\t61\n2016-04-17\tgameofthrones\t117\n2016-04-18\teuro2016\t64\n2016-04-18\tgameofthrones\t148\n2016-04-19\teuro2016\t56\n2016-04-19\tgameofthrones\t210\n2016-04-20\teuro2016\t66\n2016-04-20\tgameofthrones\t204\n2016-04-21\teuro2016\t128\n2016-04-21\tgameofthrones\t249\n2016-04-22\teuro2016\t50\n2016-04-22\tgameofthrones\t232\n2016-04-23\teuro2016\t5\n2016-04-23\tgameofthrones\t68\n2016-04-24\teuro2016\t31\n2016-04-24\tgameofthrones\t2188\n2016-04-25\teuro2016\t41\n2016-04-25\tgameofthrones\t9918\n2016-04-26\teuro2016\t58\n2016-04-26\tgameofthrones\t1104\n2016-04-27\teuro2016\t61\n2016-04-27\tgameofthrones\t425\n2016-04-28\teuro2016\t70\n2016-04-28\tgameofthrones\t291\n2016-04-29\teuro2016\t56\n2016-04-29\tgameofthrones\t234\n2016-04-30\teuro2016\t37\n2016-04-30\tgameofthrones\t146\n2016-05-01\teuro2016\t47\n2016-05-01\tgameofthrones\t313\n2016-05-02\teuro2016\t33\n2016-05-02\tgameofthrones\t2438\n2016-05-03\teuro2016\t39\n2016-05-03\tgameofthrones\t969\n2016-05-04\teuro2016\t64\n2016-05-04\tgameofthrones\t462\n2016-05-05\teuro2016\t58\n2016-05-05\tgameofthrones\t309\n2016-05-06\teuro2016\t107\n2016-05-06\tgameofthrones\t253\n2016-05-07\teuro2016\t91\n2016-05-07\tgameofthrones\t219\n2016-05-08\teuro2016\t58\n2016-05-08\tgameofthrones\t412\n2016-05-09\teuro2016\t157\n2016-05-09\tgameofthrones\t3288\n2016-05-10\teuro2016\t239\n2016-05-10\tgameofthrones\t549\n2016-05-11\teuro2016\t189\n2016-05-11\tgameofthrones\t329\n2016-05-12\teuro2016\t665\n2016-05-12\tgameofthrones\t508\n2016-05-13\teuro2016\t253\n2016-05-13\tgameofthrones\t378\n2016-05-14\teuro2016\t136\n2016-05-14\tgameofthrones\t283\n2016-05-15\teuro2016\t112\n2016-05-15\tgameofthrones\t352\n2016-05-16\teuro2016\t652\n2016-05-16\tgameofthrones\t4878\n2016-05-17\teuro2016\t823\n2016-05-17\tgameofthrones\t947\n2016-05-18\teuro2016\t331\n2016-05-18\tgameofthrones\t395\n2016-05-19\teuro2016\t199\n2016-05-19\tgameofthrones\t311\n2016-05-20\teuro2016\t196\n2016-05-20\tgameofthrones\t300\n2016-05-21\teuro2016\t183\n2016-05-21\tgameofthrones\t314\n2016-05-22\teuro2016\t249\n2016-05-22\tgameofthrones\t616\n2016-05-23\teuro2016\t285\n2016-05-23\tgameofthrones\t6768\n2016-05-24\teuro2016\t364\n2016-05-24\tgameofthrones\t1485\n2016-05-25\teuro2016\t373\n2016-05-25\tgameofthrones\t664\n2016-05-26\teuro2016\t317\n2016-05-26\tgameofthrones\t540\n2016-05-27\teuro2016\t330\n2016-05-27\tgameofthrones\t274\n2016-05-28\teuro2016\t346\n2016-05-28\tgameofthrones\t336\n2016-05-29\teuro2016\t358\n2016-05-29\tgameofthrones\t765\n2016-05-30\teuro2016\t655\n2016-05-30\tgameofthrones\t3179\n2016-05-31\teuro2016\t2001\n2016-05-31\tgameofthrones\t582\n2016-06-01\teuro2016\t1364\n2016-06-01\tgameofthrones\t372\n2016-06-02\teuro2016\t155\n2016-06-02\tgameofthrones\t110\n2016-06-07\teuro2016\t371\n2016-06-07\tgameofthrones\t117\n2016-06-08\teuro2016\t1132\n2016-06-08\tgameofthrones\t252\n2016-06-09\teuro2016\t3621\n2016-06-09\tgameofthrones\t353\n2016-06-10\teuro2016\t15750\n2016-06-10\tgameofthrones\t302\n2016-06-11\teuro2016\t13300\n2016-06-11\tgameofthrones\t260\n2016-06-12\teuro2016\t13880\n2016-06-12\tgameofthrones\t322\n2016-06-13\teuro2016\t11887\n2016-06-13\tgameofthrones\t3192\n2016-06-14\teuro2016\t7885\n2016-06-14\tgameofthrones\t666\n2016-06-15\teuro2016\t7954\n2016-06-15\tgameofthrones\t432\n2016-06-16\teuro2016\t2559\n2016-06-16\tgameofthrones\t187\n2016-06-18\teuro2016\t6476\n2016-06-18\tgameofthrones\t309\n2016-06-19\teuro2016\t5048\n2016-06-19\tgameofthrones\t803\n2016-06-20\teuro2016\t5696\n2016-06-20\tgameofthrones\t11539\n2016-06-21\teuro2016\t7831\n2016-06-21\tgameofthrones\t2061\n2016-06-22\teuro2016\t9999\n2016-06-22\tgameofthrones\t889\n2016-06-23\teuro2016\t3359\n2016-06-23\tgameofthrones\t700\n2016-06-24\teuro2016\t1645\n2016-06-24\tgameofthrones\t529\n2016-06-25\teuro2016\t8398\n2016-06-25\tgameofthrones\t507\n2016-06-26\teuro2016\t10255\n2016-06-26\tgameofthrones\t948\n2016-06-27\teuro2016\t15020\n2016-06-27\tgameofthrones\t15850\n2016-06-28\teuro2016\t4834\n2016-06-28\tgameofthrones\t2319\n2016-06-29\teuro2016\t1659\n2016-06-29\tgameofthrones\t1084\n2016-06-30\teuro2016\t6140\n2016-06-30\tgameofthrones\t764\n2016-07-01\teuro2016\t8185\n2016-07-01\tgameofthrones\t593\n2016-07-02\teuro2016\t10132\n2016-07-02\tgameofthrones\t473\n2016-07-03\teuro2016\t10835\n2016-07-03\tgameofthrones\t560\n2016-07-04\teuro2016\t3476\n2016-07-04\tgameofthrones\t538\n2016-07-05\teuro2016\t1833\n2016-07-05\tgameofthrones\t376\n2016-07-06\teuro2016\t7753\n2016-07-06\tgameofthrones\t337\n2016-07-07\teuro2016\t12066\n2016-07-07\tgameofthrones\t319\n2016-07-08\teuro2016\t5024\n2016-07-08\tgameofthrones\t273\n2016-07-09\teuro2016\t2657\n2016-07-09\tgameofthrones\t210\n2016-07-10\teuro2016\t17872\n2016-07-10\tgameofthrones\t253\n2016-07-11\teuro2016\t13681\n2016-07-11\tgameofthrones\t205\n2016-07-12\teuro2016\t1346\n2016-07-12\tgameofthrones\t211\n2016-07-13\teuro2016\t517\n2016-07-13\tgameofthrones\t225\n2016-07-14\teuro2016\t342\n2016-07-14\tgameofthrones\t683\n2016-07-15\teuro2016\t205\n2016-07-15\tgameofthrones\t303\n2016-07-16\teuro2016\t134\n2016-07-16\tgameofthrones\t162\n2016-07-17\teuro2016\t109\n2016-07-17\tgameofthrones\t156\n2016-07-18\teuro2016\t104\n2016-07-18\tgameofthrones\t308\n2016-07-19\teuro2016\t93\n2016-07-19\tgameofthrones\t260\n2016-07-20\teuro2016\t88\n2016-07-20\tgameofthrones\t208\n2016-07-21\teuro2016\t108\n2016-07-21\tgameofthrones\t181\n2016-07-22\teuro2016\t79\n2016-07-22\tgameofthrones\t227\n2016-07-23\teuro2016\t34\n2016-07-23\tgameofthrones\t393\n2016-07-24\teuro2016\t63\n2016-07-24\tgameofthrones\t186\n2016-07-25\teuro2016\t81\n2016-07-25\tgameofthrones\t193\n2016-07-26\teuro2016\t59\n2016-07-26\tgameofthrones\t153\n2016-07-27\teuro2016\t56\n2016-07-27\tgameofthrones\t180\n2016-07-28\teuro2016\t44\n2016-07-28\tgameofthrones\t143\n2016-07-29\teuro2016\t38\n2016-07-29\tgameofthrones\t106\n2016-07-30\teuro2016\t38\n2016-07-30\tgameofthrones\t237\n2016-07-31\teuro2016\t41\n2016-07-31\tgameofthrones\t193\n2016-08-01\teuro2016\t16\n2016-08-01\tgameofthrones\t36\n"}]},"apps":[],"jobName":"paragraph_1522235709507_1421840500","id":"20180326-213716_295075867","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-13T19:37:50+0200","dateFinished":"2018-04-13T19:38:46+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9394"},{"text":"%md\nWe can nicely observe the spikes in `#GameofThrones` mentions associated with the release of each episode.\nAccordingly, we can observe the spikes in `#EURO2016` mentions associated with the important games.","user":"musuroi","dateUpdated":"2018-04-13T19:53:22+0200","config":{"colWidth":12,"editorMode":"ace/mode/markdown","results":{},"enabled":true,"editorSetting":{"editOnDblClick":true,"language":"markdown"},"editorHide":true,"tableHide":false},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>We can nicely observe the spikes in <code>#GameofThrones</code> mentions associated with the release of each episode.\n<br />Accordingly, we can observe the spikes in <code>#EURO2016</code> mentions associated with the important games.</p>\n"}]},"apps":[],"jobName":"paragraph_1522235709508_1419916755","id":"20180326-222158_386750263","dateCreated":"2018-03-28T13:15:09+0200","dateStarted":"2018-04-13T19:53:22+0200","dateFinished":"2018-04-13T19:53:22+0200","status":"FINISHED","progressUpdateIntervalMs":500,"$$hashKey":"object:9395"},{"text":"%md\n","user":"musuroi","dateUpdated":"2018-04-13T19:53:22+0200","config":{"colWidth":12,"enabled":true,"results":{},"editorSetting":{"language":"markdown","editOnDblClick":true},"editorMode":"ace/mode/markdown"},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1523642002117_1907850855","id":"20180413-195322_1859699621","dateCreated":"2018-04-13T19:53:22+0200","status":"READY","progressUpdateIntervalMs":500,"$$hashKey":"object:9396"}],"name":"Assignment 2 Cip Doru","id":"2DBN8PVD1","angularObjects":{"2CHS8UYQQ:shared_process":[],"2CK8A9MEG:shared_process":[],"2C4U48MY3_spark2:shared_process":[],"2CKAY1A8Y:shared_process":[],"2CKEKWY8Z:shared_process":[]},"config":{"looknfeel":"default","personalizedMode":"false"},"info":{}}