· 8 years ago · Apr 15, 2018, 10:26 PM
1%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
2% Stylish Article
3% LaTeX Template
4% Version 2.1 (1/10/15)
5%
6% This template has been downloaded from:
7% http://www.LaTeXTemplates.com
8%
9% Original author:
10% Mathias Legrand (legrand.mathias@gmail.com)
11% With extensive modifications by:
12% Vel (vel@latextemplates.com)
13%
14% License:
15% CC BY-NC-SA 3.0 (http://creativecommons.org/licenses/by-nc-sa/3.0/)
16%
17%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
18
19%----------------------------------------------------------------------------------------
20% PACKAGES AND OTHER DOCUMENT CONFIGURATIONS
21%----------------------------------------------------------------------------------------
22
23\documentclass[fleqn,10pt]{SelfArx} % Document font size and equations flushed left
24
25\usepackage[english]{babel} % Specify a different language here - english by default
26
27\usepackage{lipsum} % Required to insert dummy text. To be removed otherwise
28\usepackage{subcaption}
29\usepackage{float}
30\usepackage{algorithmic}
31
32%----------------------------------------------------------------------------------------
33% COLUMNS
34%----------------------------------------------------------------------------------------
35
36\setlength{\columnsep}{0.55cm} % Distance between the two columns of text
37\setlength{\fboxrule}{0.75pt} % Width of the border around the abstract
38
39%----------------------------------------------------------------------------------------
40% COLORS
41%----------------------------------------------------------------------------------------
42
43\definecolor{color1}{RGB}{0,0,90} % Color of the article title and sections
44\definecolor{color2}{RGB}{0,20,20} % Color of the boxes behind the abstract and headings
45
46%----------------------------------------------------------------------------------------
47% HYPERLINKS
48%----------------------------------------------------------------------------------------
49
50\usepackage{hyperref} % Required for hyperlinks
51\hypersetup{hidelinks,colorlinks,breaklinks=true,urlcolor=color2,citecolor=color1,linkcolor=color1,bookmarksopen=false,pdftitle={Title},pdfauthor={Author}}
52
53%----------------------------------------------------------------------------------------
54% ARTICLE INFORMATION
55%----------------------------------------------------------------------------------------
56
57\JournalInfo{Introduction to Data Analysis and Mining 2018} % Journal information
58\Archive{} % Additional notes (e.g. copyright, DOI, review/research article)
59
60\PaperTitle{Semester Project} % Article title
61
62\Authors{Charles Frank, Ryan Hartford, Sam Madden\textsuperscript{1}*} % Authors
63\affiliation{\textsuperscript{1}\textit{Computer Science, School of Informatics , Computing and Engineering, Indiana University, Bloomington, IN, USA}} % Author affiliation
64
65
66\Keywords{Keyword1 --- Keyword2 --- Keyword3} % Keywords - if you don't want any simply remove all the text between the curly brackets
67\newcommand{\keywordname}{Keywords} % Defines the keywords heading name
68
69%----------------------------------------------------------------------------------------
70% ABSTRACT
71%----------------------------------------------------------------------------------------
72
73\Abstract{Include abstract here -- A summary of your work}
74
75%----------------------------------------------------------------------------------------
76
77\begin{document}
78
79\flushbottom % Makes all text pages the same height
80
81\maketitle % Print the title and abstract box
82
83\tableofcontents % Print the contents section
84
85\thispagestyle{empty} % Removes page numbering from the first page
86
87
88
89
90%----------------------------------------------------------------------------------------
91%Problem and Data Description
92%----------------------------------------------------------------------------------------
93
94
95\section{Problem and Data Description} % The \section*{} command stops section numbering
96%Briefly describe the data mining problem and the data
97
98Porto Seguro is one of the largest car insurance companies in Brazil. They are hoping to better serve their customers by more accurately pricing their car insurance based on a predictive model. Porto Seguro has gracefully provided a good dataset. There is little noise, the data is already mostly clean, and there are minimal missing values. Missing values are represented with a $-1$. Training and test .csv files have been provided. The data is ordinal, binary, nominal, or interval. There are 59 variables and $595,212$ rows. Excluding \textit{id} and \textit{target}, which are not an input type, there are 57 features describing the attributes of each record.
99
100\bigskip
101\bigskip
102
103%----------------------------------------------------------------------------------------
104% Data Preprocessing $\&$ Exploratory Data Analysis
105%----------------------------------------------------------------------------------------
106\section{Data Preprocessing $\&$ Exploratory Data Analysis} % The \section*{} command stops section numbering
107
108
109\subsection{Handling Missing Values}
110
111The provided training set has dimensions of $595,212 X 59$ with a mixture of columns containing
112int64 and float64 numbers. Upon initial evaluation of the set, no missing values exist from .info() output.
113This is because NaN values are instead given the value of -1. After replacing -1 values with NaN the
114missing values become apparent. Notably the columns \textit{ps\_reg\_03} missing $107,772$ values,
115\textit{ps\_car\_03\_cat} missing $411,231$ values, and \textit{ps\_car\_05\_cat} missing $266,551$. Using the package
116missingno to visualize the data columns containing missing values are quickly identified.
117
118 \begin{figure}[h]
119 \includegraphics[width=\linewidth]{Figure_1.png}
120 \caption{MissingNo representaion}
121 \label{fig:missingno}
122\end{figure}
123 Columns \textit{ps\_reg\_03}, \textit{ps\_car\_03\_cat}, \textit{ps\_car\_05\_cat} are dropped due to high number of missing values.
124The remaining missing values are replaced with the mean of the respective column.
125\begin{center}
126\[\bar{x} = \frac{1}{n} \sum\ x\]
127\end{center}
128After reassessing the data, no NaN values are identified and the dataset is complete.
129
130\subsection{Scaling using Standardization}
131
132The standardization approach of scaling is used on this dataset. Normalization(min-max scaling) is declined in this instance due to its susceptibility to outliers. Using standardization, the feature columns are centered at mean 0 with standard deviation 1 meaning a Gaussian distribution. This scaling will retain information about outliers while also making the classifiers less sensitive to large deviations from the mean. The equation for standardization is as follows:
133
134\[x^{(i)}_{std} = \frac{x^{(i)} - \mu_x}{\sigma_x}\]
135
136Where \(\mu_x\) is the sample mean of a particular feature column and \(\sigma_x\) is the corresponding standard deviation.
137
138Due to this transformation, the X\_test will also be transformed but not fitted using the scaler.
139
140\bigskip
141\bigskip
142%----------------------------------------------------------------------------------------
143 % Algorithm and Methodology
144%----------------------------------------------------------------------------------------
145
146
147\section{Algorithm and Methodology}
148
149%Briefly explain the algorithms in this section, i.e., linear regressions. You can add more subsections if needed, i.e., regression trees etc.
150
151Informal plan: Optimize models for 3 different algorithms and use the best model as the final model. The 3 algorithms we are using a Decision tree, a Naive Bayes Classifier, and a Support Vector Machine. We will also work to reduce the dimensionality of the features of the records using either an L1 or an L2 regularization.
152
153\subsection{Naive Bayes Algorithm}
154[Add puesdocode for Naive Bayes]
155
156\subsection{Decision Tree Classifiers}
157\subparagraph{LightGBM Classifier} I will use the LightGBM Classifier. This is a fast, distributed, high performance gradient boosting framework based on decision tree learning algorithms. This classifier is used to for ranking, classifying and general machine learning. LightGBM performs similar to XGBoost. The two differ in speed, and LightGBM builds trees vertically (leaf-wise) while XGBoost builds level-wise. LightGBM is slightly faster than XGBoost and pulls slightly higher accuracy. LightGBM grows on leaves with maximum delta loss and experiences sensitivity due to over-fitting on smaller data sets. This can be found in the package\textbf{\textit{ lightgbm}} in python.
158
159
160\subparagraph{Generic Decision Tree} A decision tree is a standard tree that uses branches to classify a dataset. Their algorithms follow a greedy search style through all possible branches. They are implemented top-down, which means implementation begins at a point and works its way to the bottom.
161
162\subparagraph{Information gain} Decision trees gain information with a decrease of entropy after splitting attributes of a dataset. Decision trees first-and-foremost build and find attributes that return the most homogeneous branches.
163
164Entropy is used in decision trees to define the structure. To calculate the frequency of two attributes. We are familiar with this equation from class as:
165$E(X, Y) = \sum\limits_{c \in X} P(c)E(c)$
166
167\subparagraph{Algorithm} The Algorithm begins with calculating entropy. Next we utilize our information gains to create decision nodes. The dataset is divided by the number of branches and the process is repeated. We always ensure the leaf node of a branch has entropy of 0, if this term is not met, then we will perform further splitting. Alrogithm is run recursively on all non-leaf branches until all data is classified.
168
169The LGBMClassifier is a very useful package we can find in package lightgbm. The algorithm already exists, so we are just required to define parameters and decide how to manage our outputs.
170
171To run this alrogithm we will take the cross-folded data as our inputs, plug in our parameters included with a gini index, and receive our scores.
172The Gini index is very simple to calculate.
173
174\bigskip
175\bigskip
176%----------------------------------------------------------------------------------------
177 % Experiments and Results
178%----------------------------------------------------------------------------------------
179\section{Experiments and Results}
180
181We have conducted some preliminary experiments using the Naive Bayes Algorithm and 3 assumptions for the distribution of the data. The assumptions we tested were Gaussian, Multinomial, and Bernoulli. We did one experiment for each assumption. In each trail for in each experiment, we identified the best column to drop from the data by creating a model with all the other present columns, defining best as lowest model error rate. The lowest error rate achieved was $3.645\%$ and all three experiments reached this error rate. The all required different columns to be dropped to achieve it. The Gaussian assumption, seen in table \ref{table:exp1}, required 14 columns to be dropped to achieve an error rate of $3.645\%$. The Multinomial assumption, seen in table \ref{table:exp2}, required 3 columns to be dropped to achieve an error rate of $3.645\%$. The Bernoulli assumption, seen in table \ref{table:exp3}, required 1 column to be dropped to achieve an error rate of $3.645\%$ but was able to drop an additional two columns while maintaining an error rate of $3.645\%$.
182
183\begin{center}
184 \begin{table}[H]
185 \begin{tabular}{|l|r|}
186 \hline
187 Column & Error Rate Without Column \\ \hline
188 ps\_car\_04\_cat & 7.380\% \\
189 ps\_car\_13 & 6.080\% \\
190 ps\_car\_07\_cat & 5.583\% \\
191 ps\_ind\_05\_cat & 5.167\% \\
192 ps\_car\_12 & 4.788\% \\
193 ps\_ind\_17\_bin & 4.628\% \\
194 ps\_reg\_02 & 4.612\% \\
195 ps\_car\_14 & 4.609\% \\
196 ps\_ind\_12\_bin & 4.609\% \\
197 ps\_ind\_14 & 3.889\% \\
198 ps\_ind\_11\_bin & 3.759\% \\
199 ps\_ind\_13\_bin & 3.679\% \\
200 ps\_ind\_10\_bin & 3.647\% \\
201 ps\_car\_08\_cat & 3.645\% \\ \hline
202 \end{tabular}
203 \caption {Output for Gaussian Assumption}\label{table:exp1}
204 \end{table}
205\end{center}
206
207\begin{center}
208 \begin{table}[H]
209 \begin{tabular}{|l|r|}
210 \hline
211 Column & Error Rate Without Column \\ \hline
212 ps\_car\_04\_cat & 3.806\% \\
213 ps\_ind\_05\_cat & 3.648\% \\
214 ps\_ind\_14 & 3.645\% \\ \hline
215 \end{tabular}
216 \caption {Output for Multinomial Assumption}\label{table:exp2}
217 \end{table}
218\end{center}
219
220\begin{center}
221 \begin{table}[H]
222 \begin{tabular}{|l|r|}
223 \hline
224 Column & Error Rate Without Column \\ \hline
225 ps\_ind\_14 & 3.645\% \\
226 ps\_ind\_03 & 3.645\% \\
227 ps\_ind\_10\_bin & 3.645\% \\ \hline
228 \end{tabular}
229 \caption {Output for Bernoulli Assumption}\label{table:exp3}
230 \end{table}
231\end{center}
232
233\bigskip
234\bigskip
235%----------------------------------------------------------------------------------------
236 % Summary and Conclusions
237%----------------------------------------------------------------------------------------
238\section{Summary and Conclusions}
239\bigskip
240\bigskip
241\bigskip
242
243
244
245\phantomsection
246\section*{Acknowledgments} % The \section*{} command stops section numbering
247
248\addcontentsline{toc}{section}{Acknowledgments} % Adds this section to the table of contents
249
250
251
252%----------------------------------------------------------------------------------------
253% REFERENCE LIST
254%----------------------------------------------------------------------------------------
255\phantomsection
256\bibliographystyle{unsrt}
257\bibliography{sample}
258
259%----------------------------------------------------------------------------------------
260
261\end{document}