Agfdhyk

I'm trying to use a spam classification application in Python but i'm getting the errors below. I don't understand it though since i'm using the .keys method to delete items from the dictionary so that shouldn't be a problem?
I've tried removing all the functionars bar the dictionary function to try and find a cause but I can't seem to wrap my head around this

Python code

    import os

    import numpy as np

    from collections import Counter

    from sklearn.naive_bayes import MultinomialNB

    from sklearn.svm import LinearSVC

    from sklearn.metrics import confusion_matrix



    def make_Dictionary(train_dir):

        emails = [os.path.join(train_dir,f) for f in os.listdir(train_dir)]    

        all_words =        

        for mail in emails:    

            with open(mail) as m:

                for i,line in enumerate(m):

                    if i == 2:

                        words = line.split()

                        all_words += words



        dictionary = Counter(all_words)



        list_to_remove = dictionary.keys()

        for item in list_to_remove:

            if item.isalpha() == False: 

                del dictionary[item]

            elif len(item) == 1:

                del dictionary[item]

        dictionary = dictionary.most_common(3000)

        return dictionary



    def extract_features(mail_dir): 

        files = [os.path.join(mail_dir,fi) for fi in os.listdir(mail_dir)]

        features_matrix = np.zeros((len(files),3000))

        docID = 0;

        for fil in files:

          with open(fil) as fi:

            for i,line in enumerate(fi):

              if i == 2:

                words = line.split()

                for word in words:

                  wordID = 0

                  for i,d in enumerate(dictionary):

                    if d[0] == word:

                      wordID = i

                      features_matrix[docID,wordID] = words.count(word)

            docID = docID + 1     

        return features_matrix



    # Create a dictionary of words with its frequency



    train_dir = r'.train-mails'

    dictionary = make_Dictionary(train_dir)



    # Prepare feature vectors per training mail and its labels



    train_labels = np.zeros(702)

    train_labels[351:701] = 1

    train_matrix = extract_features(train_dir)



    # Training SVM and Naive bayes classifier and its variants



    model1 = LinearSVC()





    model1.fit(train_matrix,train_labels)





    # Test the unseen mails for Spam



    test_dir = r'.test-mails'

    test_matrix = extract_features(test_dir)

    test_labels = np.zeros(260)

    test_labels[130:260] = 1



    result1 = model1.predict(test_matrix)





    print (confusion_matrix(test_labels,result1))

    print (confusion_matrix(test_labels,result2))

Error

RuntimeError: dictionary changed size during iteration

This doesn't work in Python 3.x because keys returns an iterator instead of a list.

Another way is to use list to force a copy of the keys to be made. This one also works in Python 3.x:

for i in list(list_to_remove):

dictionary.keys() is actually returning a reference to the original dictionary keys.

You can check this by doing the following:

 a_dict = {'a': 1}

 keys = a_dict.keys() # keys is dict_keys(['a'])

 a_dict['b'] = 2 # keys is dict_keys(['a', 'b'])

That's why you are getting the error: with del dictionary[item] you are actually affecting list_to_remove, which is forbidden during a loop.

You can avoid this by creating a copy of the original keys before looping through them. The easiest way to achieve this is using the list constructor. So changing your line

list_to_remove = dictionary.keys()

with:

list_to_remove = list(dictionary.keys())

solves the issue.

Edition after comment

Note that this behaviour only happens in python 3, in python 2, the .keys() method returned a normal list, with no reference to the dictionary:

a_dict = {'a': 1}

keys = a_dict.keys() # keys is ['a']

a_dict['b'] = 2 # keys is still ['a']

About this on the Python 3.0 changelog:

Some well-known APIs no longer return lists:

• dict methods dict.keys(), dict.items() and dict.values() return “views” instead of lists.