SGDClassifier geben verschiedene Genauigkeit jedes Mal für die Textklassifizierung

Question

Ich benutze den SVM Classifier für die Klassifizierung von Text als guten Text und Kauderwelsch. Ich verwende Python Scikit-Learn und tun es wie folgt:

''' 
Created on May 5, 2017 
''' 

import re 
import random 
import numpy as np 
from sklearn.feature_extraction.text import CountVectorizer 
from sklearn.linear_model import SGDClassifier 
from sklearn import metrics 

# Prepare data 

def prepare_data(data): 
    """ 
    data is expected to be a list of tuples of category and texts. 
    Returns a tuple of a list of lables and a list of texts 
    """ 
    random.shuffle(data) 
    return zip(*data) 

# Format training data 

training_data = [ 
    ("good", "rain a lot the packs maybe damage."), 
    ("good", "15107 Lane Pflugerville, TX customer called me and his phone number and my phone numbers were not masked. thank you customer has had a stroke and items were missing from his delivery the cleaning supplies for his wet vacuum steam cleaner. he needs a call back from customer support "), 
    ("gibber", "wh. screen"), 
    ("gibber", "How will I know if I"), 
    ("good", "I have problems scheduling blocks they are never any available. Can I do full time? Can I get scheduled more than one day a month?"), 
    ("good", "Suggestion: easier way to sign in due alleviate the tediousness of periodically having to sign back in to the app to check for blocks."), 
    ("good", "I am so glad to hear from you. "), 
    ("good", "loading on today's itinerary takes ages!!!!!! time consuming when you have 150+ packages to deliver!!!!!"), 
    ("good", "due to the new update that makes hours available at 10 pm. if you worked 8 hours that day you can't see next day hours due to 8 hour limit. please fix this"), 
    ("good", "omg, PLEASE make it so we don't have to sign in every time we need to go into the app. At least make it good for a week. Thanks."), 
    ("good", "Constantly being logged out of app, if we could have a continuous login so we could receive notifications if blocks are available that would be ideal."), 
    ("good", "I am having problems with the App. Every time I exit the App and reopen it asks for my login info."), 
    ("good", "15 minute service time due to 33rd floor and 20l lbs of cargo"), 
    ("good", "I have been sceduled 1 block in 3 weeks. I check for new block availability multiple times a day and have not seen 1 available in three weeks. is there any way to get more blocks."), 
    ("good", "When will delivery jobs be available? Everytime I open this app, it says nothing is available. Have deliveries in Cincinnati started yet?"), 
    ("good", "During delivery had to call customer support and after 10 minutes support person couldn't find my pick up location Kirkland /Bellevue and told me to hang up and call different support team. Support person were unprofessional and rude, which is not acceptable."), 
    ("good", "can you please remove the pick up from my phone"), 
    ("good", "Dear friends: I'm very very happy it's a big oportunitt"), 
    ("good", "THANK YOU so much for the block you assigned me for next week. If you have an additional 5 blocks please go ahead and assign them to me for next week. My availability is updated and current. You guys are awesome!!!"), 
    ("good", "after update every time I open app I have too log in! I used to be able to stay logged in unless I logged out, can you return stay logged in option."), 
    ("good", "It looks like my app is not installed properly on my android phone, Note 5. I cannot access or do not see the tab to swipe to start delivering and the map or help button that should be visible for me to work today 5/6 at rpm"), 
    ("gibber", "AF0000"), 
    ("good", "awesome app, awesome hiring process, awesome delivery warehouse , awesome team and help in the field! lets deliver I would like more more more delivers , looking forward to the future ! I just bought a new delivery vehical !"), 
    ("good", "I will like to ask why I can't get more delivery's only one in two weeks"), 
    ("good", "device too slow software crashing all day"), 
    ("good", "it doesn't work sometimes."), 
    ("good", "can you please remove the old sprouts pick up from my phone"), 
    ("good", "They ability to zoom in on text screens would be very helpful. Am example would be customer notes when viewing in certain lighting conditions can be difficult."), 
    ("good", "I missed out on a delivery day when I clicked check in and waited for my turn to get an order only to find out that not only did my check in not register but the gps showed me down the street. I encountered this issue again when one of the warehouse employees placed an order for that location and the app wanted me to drive in a big circle to get back to where I was standing."), 
    ("good", "i am a little concerned that i didn't receive any blocks of time for this coming week, even though i had a perfect delivery score from this past pay period. Did the Cincinnati market over hire drivers where there are many people being shut completely out of any delivery blocks for an entire week? i really enjoy this type of work and the app makes it quite convenient."), 
    ("good", "I've arrived at the pick up restaurant but the staff did not have the barecode for me to scan, however I pick up the package and deliver but my is still not let me move on"), 
    ("good", "might want to check my assigned hours for next week. 5am to 1pm??"), 
    ("good", "hi team--just want to give some positive feedback. I have had nothing but positive feedback from customers. Great support when calling help line. Thank you for this opportunity and if there is ever a situation where you need drivers immediately I will drop what I'm doing and help. You guys are the best."), 
    ("good", "Allow days or blocks throughout the day to be modified after General availability is set up for time off like doctors appointments."), 
    ("gibber", "AL0"), 
    ("good", "Please, enlight me."), 
    ("good", "it only shows my schedule starting in two weeks. when will we be able to start work"), 
    ("good", "include more packages for one block, if the packages can be fitted into the car, so driver don't have to come back and pickup every two hours. 25% of the time is wasted coming back for pick up."), 
    ("gibber", "BBB h"), 
    ("gibber", "AG0003006033SDgCJ12344"), 
    ("gibber", "How will I know if I"), 
    ("good", "please bring back some sort of hours cap! or possibly stagger the hour drops from 1200 to 1203 so that people with slower internet/slower phone arent at a disadvantage!"), 
    ("good", "when the hours released tonight all of the people who didn't have 40 hours could see them. however the drivers that are capped at 40 were unable to see them due to a flawed system. please fix the system so that we are not continually treated unfairly like all of the drivers that whined so much and got us in to this mess. the cap system is unfair to people that want to work and it caused problems with a lack of drivers to deliver today at the hub. obviously this is not a good system and benefits no one."), 
    ("good", "You have seriously messed up the whole scheduling process. Why can't I get any blocks at 10 even if I wait exactly until 10? Midnight was much better. So now that scheduling is a huge random pain in the ass, why would people want to keep doing this? I haven't been able to schedule work for three days now, it's quite frustrating when I don't get a chance to sign up, even when I'm diligent with timing."), 
    ("good", "Seriously, that's all I'm going to get is one lousy day? Tell me again why you need drivers if all we get is one day. I'm not sure this is gonna work out for me. I waited forever to get my background check back and this is what I get? smh"), 
    ("good", "doesn't save updated access codes"), 
    ("good", "the scheduling of my route is nor done very accurately. it keeps me driving back and forth"), 
    ("good", "can't understand how to pick up a block. my availability is wide open. when you guys send the alerts about blocks available I open it real quick and there is nothing there. I do it in a matter of seconds"), 
    ("good", "My availability keeps disappearing from my calender. I set my availability for three weeks in advance. The gray dots are visible but disappear on Wednesday or Thursday. This makes it impossible for me to see and choose available blocks for the upcoming week. How can I get it fix. Mike"), 
    ("good", "GPS blank screen"), 
    ("gibber", "sea swq"), 
    ("gibber", "hiw o"), 
    ("gibber", "Dr a"), 
    ("gibber", "quick to quick to u uhu wu just us"), 
    ("gibber", "Awa what's"), 
    ("gibber", "wxdfcs"), 
    ("gibber", "7k9opu"), 
    ("gibber", "o.m.day day"), 
    ("gibber", "GGT part his h"), 
    ("gibber", "aawfhg"), 
    ("gibber", "seesaw 2s"), 
    ("gibber", "wawaa"), 
    ("gibber", "of ll"), 
    ("gibber", "rewards"), 
    ("gibber", "mmqqm5my"), 
    ("gibber", ".in w"), 
    ("gibber", "play r"), 
    ("gibber", "was wwnw www www n"), 
    ("gibber", "wqq2fwqq2fz22"), 
    ("gibber", "not"), 
    ("gibber", "I by yu I"), 
    ("gibber", "Hi just wanted to let you know that it's bee"), 
    ("gibber", "I erroneously v"), 
    ("gibber", "I find it"), 
    ("gibber", "bqyyx I a"), 
    ("gibber", "are are"), 
    ("gibber", "wawi waarnnnkwn"), 
    ("gibber", "t Petey ueteu he"), 
    ("gibber", "ews ri"), 
    ("gibber", "bd xd"), 
    ("gibber", "hatpa"), 
    ("gibber", "se wests tasgt"), 
    ("gibber", "wa vgcx azc Jo of"), 
    ("gibber", "2w222"), 
    ("gibber", "her u t b"), 
    ("gibber", "ddddedc"), 
    ("gibber", "just juju in hiking"), 
    ("gibber", "wew2ww2wwwew2i2wkkk"), 
    ("gibber", "meleeee"), 
    ("gibber", "Aaq wqXD"), 


] 
training_labels, training_texts = prepare_data(training_data) 


# Format test data 

test_data = [ 

("gibber", "an quality"), 
    ("good", "Can't check in. Time was 4:06. I didn't drive out here for no reason."), 
    ("good", "can you do view all full address including postal code how it's in old app that helps do correctly delivery and not waist customer time"), 
    ("good", "i am available again starting at 10am to 10pm. thanks"), 
    ("gibber", "Hello, I encountered"), 
    ("good", "I want to know how we are notified if there is a block I have been signed in and haven't been given a block yet"), 
    ("gibber", "aawaaw"), 
    ("gibber", "eeeeeeeeene"), 
    ("good", "I am not getting enough shifts"), 
    ("gibber", "hey e75k"), 
    ("good", "my screen had went black or inverted"), 
    ("good", "maps packed up again in sr20ls"), 
    ("good", "how to clear my itinerary from old pickup address ?"), 
    ("good", "keep signing me out."), 
    ("good", "For alcohol delivery, where does customer sign?"), 
    ("gibber", "t Petey ueteu he"), 
    ("good", "can't get blocks. too many drivers ??"), 
    ("good", "got a new phone how do i download to new phone") 



] 
test_labels, test_texts = prepare_data(test_data) 


# Create feature vectors 

""" 
Convert a collection of text documents to a matrix of token counts. 
See: http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html 
""" 
vectorizer = CountVectorizer() 
X = vectorizer.fit_transform(training_texts) 
y = training_labels 


# Train the classifier 


clf = SGDClassifier() 
clf.fit(X, y) 


# Test performance 

X_test = vectorizer.transform(test_texts) 
y_test = test_labels 

# Generates a list of labels corresponding to the samples 
test_predictions = clf.predict(X_test) 

# Convert back to the usual format 
annotated_test_data = list(zip(test_predictions, test_texts)) 
print(annotated_test_data) 

# evaluate predictions 
y_test = np.array(test_labels) 
print(metrics.classification_report(y_test, test_predictions)) 
print("Accuracy: %0.4f" % metrics.accuracy_score(y_test, test_predictions)) 

Aber ich halte unterschiedliche Genauigkeit immer jedes Mal, wenn ich es laufen. Warum passiert dies?

UPDATE: Also zog ich die training_data in eine Textdatei, und ich bin es in dem obigen Code wie folgt zu lesen:

lines = [line.rstrip('\n') for line in open("file.txt")] 
training_data=[] 
for i in lines: 
    result = i.rstrip(',') 
    l = literal_eval(result) 
    training_data.append(l) 

training_labels, training_texts = prepare_data(training_data) 

Und ich änderte auch dies in meinem obigen Code:

clf = SGDClassifier(random_state=5000) 

Also, jetzt ist der random_state nicht None. Aber ich bekomme immer noch andere Genauigkeiten !!

Answer 1

Alle Klassifizierer, die Daten teilen oder mischen (h/t zu Vivek), haben im Konstruktor optional eine Variable random_state mit einem Standardwert von None. Wenn das random_state übergeben wird, wird es von einer internen check_random_state Funktion überprüft.

Aus der Dokumentation:

check_random_state: Erstellen Sie ein np.random.RandomState Objekt aus einem Parameter random_state. Wenn random_stateNone oder ist, wird ein nach dem Zufallsprinzip initialisiertes Objekt RandomState zurückgegeben. Wenn random_state eine Ganzzahl ist, wird sie verwendet, um ein neues RandomState-Objekt zu bilden. Wenn random_state ein RandomState Objekt ist, wird es durchlaufen.

Da Sie den Standard None verwenden, haben Sie einige unkontrollierte stochastischen Rauschen in Ihrem Code.

Eine Saat zur Reproduzierbarkeit übergeben.

Answer 2

Dies ist, weil Sie in Ihrer prepare_data() Methode die Daten zufällig mischen. Dies ist, was Sie tun:

random.shuffle(data) 

So beeinflusst es das Training des Schätzers und damit die Ergebnisse.

Versuchen Sie, diese Zeile mit der random_state im SGDClassifier zu kommentieren oder zu entfernen. Sie erhalten jedes Mal exakt dieselben Ergebnisse.

Vorschlag: Verwenden Sie verschiedene Schätzfunktionen, um zu sehen, welche am besten funktioniert. Wenn Sie das SGDClassifier verwenden möchten, würde ich empfehlen, den n_iter Parameter zu sehen und zu verstehen. Versuchen Sie, einen größeren Wert zu wählen, und Sie werden sehen, dass der Unterschied in der Genauigkeit immer geringer wird (selbst wenn Sie Daten mischen).

Sie können für weitere Details über sie zu dieser Antwort suchen:

• https://datascience.stackexchange.com/a/9794