Libraries¶
We need four main libraries for this lecture: Scikit learn, numpy, scipy, and Pandas. An easy way to install all of them in one go is to get Anaconda: https://anaconda.org/anaconda/python. When you install Anaconda you will also get access to Jupyter notebook using which you can run all the below commands. An alternative, which I highly recommend is to first install Anaconda and run your code using editors such as Text Wrangle, Sublime Text, Notepad ++, or Vim editor. Doing the later will highly be beneficial for you after this course is over (but it does have a learning curve).
Data¶
For this practice class lets take the data from the first assignment.
Using our libraries to import the data¶
In [321]:
import pandas as pd
import numpy as np
import scipy
In [322]:
print("Hello world!")
In [41]:
df = pd.read_csv("Location_of_you_folder/train.csv",encoding='ISO-8859-1') # there are different encoding and for HW1 this particular encoding works well.
In [327]:
df.head(2)
Out[327]:
In [328]:
df.columns ## To print out the column names
Out[328]:
In [325]:
df.shape ## To know the dimensions of the data frame. The output should be read as [rows,columns]
Out[325]:
In [23]:
[rows,columns] = df.shape # Print both values out to check the output yourself.
In [329]:
df.describe()
Out[329]:
In [339]:
df['class'].unique() ## To know the unique labels.
Out[339]:
In [331]:
df['class'].value_counts() ## To know the distribution of the labels.
Out[331]:
Convert labels into a machine readable format¶
In [332]:
from sklearn import preprocessing ### Importing a preprocessor to convert the labels in the target class.
In [333]:
train_class_y = ['negative','positive']
In [334]:
le = preprocessing.LabelEncoder() ## Label encoder does the trick.
In [335]:
le.fit(train_class_y) ## We are fitting the categories now.
Out[335]:
In [337]:
train_y = le.transform(df['class'])
In [340]:
train_y ## These are out labels now.
Out[340]:
In [341]:
le.transform(['positive','negative','positive']) ### Just to check
Out[341]:
In [342]:
le.inverse_transform([0,1,1]) ### Doing an inverse of the transformation
Out[342]:
Tackling our text data¶
In [343]:
train_x = df['text']
In [344]:
train_x.shape ## Just making sure that we have what we want.
Out[344]:
Rather than doing bag of words, we are goint to convert our data into tf.idf's¶
In [345]:
from sklearn.feature_extraction.text import TfidfVectorizer ## Importing the library that will help us do this.
In [424]:
tf = TfidfVectorizer(min_df=1,stop_words='english',max_features=5000) ## Ask yourself, why min_df =1? We are using english stopwords.
####max_features=3000
In [425]:
train_x_tfidf = tf.fit_transform(train_x)
In [426]:
tf.get_feature_names() ## Be careful to check your feature names with tf and not with train_x_tfidf
Out[426]:
In [427]:
train_x_tfidf_array = train_x_tfidf.toarray()
In [428]:
train_x_tfidf_array[0]
Out[428]:
In [429]:
tf.inverse_transform(train_x_tfidf_array[0]) ## just to check what all features are there.
Out[429]:
Importing a learning model¶
Let us start with a Multinomial Naive Bayes
In [354]:
from sklearn.naive_bayes import MultinomialNB
In [471]:
mnb = MultinomialNB(alpha=1.0) # Check what this alpha value is. You have already learnt most of the math to understand this.
In [376]:
mnb.fit(train_x_tfidf_array,train_y)
Out[376]:
Lets get out test data now¶
In [430]:
test_df = pd.read_csv("/Users/boY/Desktop/inls613_fall/class_exercise_Data/test.csv",encoding='ISO-8859-1')
In [431]:
test_x_tfidf = tf.transform(test_df['text'])
In [432]:
test_x_tfidf_array = test_x_tfidf.toarray()
In [433]:
test_y = le.transform(test_df['class'])
In [434]:
test_y.shape
Out[434]:
In [435]:
test_x_tfidf_array.shape
Out[435]:
In [382]:
predictions = mnb.predict(test_x_tfidf_array)
In [383]:
predictions.shape
Out[383]:
In [384]:
count = 0
for i in range (len(predictions)):
if predictions[i]==test_y[i]:
count=count+1
In [385]:
count
Out[385]:
In [386]:
count/2000
Out[386]:
Feature engineering: Pick the top tf idf features¶
In [114]:
feature_array = np.array(tf.get_feature_names())
In [115]:
tfidf_sorting = np.argsort(train_x_tfidf.toarray()).flatten()[::-1]
In [116]:
tfidf_sorting
Out[116]:
In [117]:
n = 500
In [126]:
top_n = feature_array[tfidf_sorting][:n]
In [127]:
top_n
Out[127]:
Different classification algorithm: Logistic Regression¶
In [436]:
from sklearn.linear_model import LogisticRegression # load the library
In [443]:
log_reg = LogisticRegression(C=4.0)
In [444]:
log_reg.fit(train_x_tfidf_array,train_y)
Out[444]:
In [445]:
log_reg.score(train_x_tfidf_array,train_y) # running it on the train set itself.
Out[445]:
In [447]:
log_reg.score(test_x_tfidf_array,test_y) # running it on the test set.
Out[447]:
Different classification algorithm: SVM¶
In [448]:
from sklearn import svm
In [449]:
clf = svm.SVC(C=2.0,degree=1,kernel='linear')
In [450]:
clf.fit(train_x_tfidf_array,train_y)
Out[450]:
In [451]:
predicted = clf.predict(test_x_tfidf_array)
In [452]:
count = 0
for i in range (len(predicted)):
if predicted[i]==test_y[i]:
count=count+1
count
Out[452]:
In [453]:
len(predicted)
Out[453]:
In [472]:
1636/2000 # Computing accurarcy here.
Out[472]:
Different classification algorithm: Random Forest¶
In [455]:
from sklearn.ensemble import RandomForestClassifier
In [463]:
forest = RandomForestClassifier(max_depth=10,n_estimators=100,min_samples_leaf=2)
#max_depth=10,n_estimators=100,min_samples_leaf=2
In [464]:
forest.fit(train_x_tfidf_array,train_y)
Out[464]:
In [465]:
forest.score(train_x_tfidf_array,train_y)
Out[465]:
In [466]:
forest.score(test_x_tfidf_array,test_y)
Out[466]:
In [312]:
forest_predictions = forest.predict(test_x_tfidf_array)
In [467]:
from sklearn.metrics import confusion_matrix
In [468]:
confusion_matrix(test_y, forest_predictions)
Out[468]:
In [315]:
(766+819)/(766+819+234+181)
Out[315]:
In [469]:
import matplotlib.pyplot as plt
In [470]:
plt.matshow(confusion_matrix(test_y, forest_predictions),cmap='seismic')
Out[470]:
In [ ]: