Agfdhyk

I have a dataframe, which contains 599 tokenized texts, one per row. Also i have these lists:

grwoth = ['growth', 'grow', 'growing', 'grows']

syergies = ['synergies', 'synergy' ,'accretive', 'accretion','efficiencies' ,'efficient', 'efficiently' ]

intangibles = ['brand','branded','branding','brands','goodwill','patent','patents','goodwil']

customers = ['customer', 'customers' ,'consumer' ,'consumers' ]

technology = ['technological', 'technologically', 'technologies', 'technology', 'innovate', 'innovation', 'innovations', 'innovative', 'innovator', 'innovators']

human = ['employee', 'employees', 'employees', 'team', 'teamed', 'teaming', 'teams', 'Expertise' ]

I want to create a new column in my dataframe for each list and to count how often the words from the lists has been counted in each text.

I tried to input them into my orginal dataframe (without tokenization) but this didnt work either. I Had the following code:

%%time



    growth = ['growth', 'grow', 'growing', 'grows']

    synergies = ['synergies', 'synergy' ,'accretive', 'accretion','efficiencies' ,'efficient', 'efficiently' ]

    intangibles = ['brand','branded','branding','brands','goodwill','patent','patents','goodwil']

    customers = ['customer', 'customers' ,'consumer' ,'consumers' ]

    technology = ['technological', 'technologically', 'technologies', 'technology', 'innovate', 'innovation', 'innovations', 'innovative', 'innovator', 'innovators']

    human = ['employee', 'employees', 'employees', 'team', 'teamed', 'teaming', 'teams', 'expertise' ]

    the = 'Wire'



    result_list=

    count_growth = 0

    count_human = 0

    count_technology= 0

    count_customers = 0

count_intagibles = 0

count_synergies = 0

count_the = 0

for file in file_list:

    name = file[len(input_path):]

    date = name[11:17]

    type_1 = name[17:20]

    with open(file, "r", encoding="utf-8", errors="surrogateescape") as rfile:

            # We need to encode/decode as some text files are not in utf-8 format

            text = rfile.read()

            text = text.encode('utf-8', 'ignore')

            text = text.decode('utf-8', 'ignore')



    for word in text.split():

        if word in growth:

            count_growth = count_growth +1

        if word in synergies:

            count_synergies = count_synergies +1 

        if word in intagibles:

            count_intagibles = count_intagibles+1

        if word in customers:

            count_customers = count_customers +1

        if word in technology:

            count_technology = count_technology +1

        if word in human:

            count_human = count_human +1

        if word == 'The':

            count_the = count_the +1

    length = len(text.split())









    a={"File": name, "Text": text,'the':count_the, 'Datum': date, 'File_type': type_1, 'length':length, 'grwoth':count_growth, 'synergies': count_synergies,'intagibles':count_intagibles,'customers':count_customers, 'technology':count_technology,'human':count_human,}

    result_list.append(a)

The problem here was, that it creates a total sum but not a sum for each row as it does for length.

Thanks in advance for any solutions!

You simply need to clear the variables inside the for loop. This way it outputs the count for the various files, as it does for the length.

I hope I understood correctly what you wanted to do.

Code below:

for file in file_list:

    count_growth = 0

    count_human = 0

    count_technology= 0

    count_customers = 0

    count_intagibles = 0

    count_synergies = 0

    count_the = 0

    name = file[len(input_path):]

    date = name[11:17]

    type_1 = name[17:20]

    with open(file, "r", encoding="utf-8", errors="surrogateescape") as rfile:

            # We need to encode/decode as some text files are not in utf-8 format

            text = rfile.read()

            text = text.encode('utf-8', 'ignore')

            text = text.decode('utf-8', 'ignore')



    for word in text.split():

        if word in growth:

            count_growth = count_growth +1

        if word in synergies:

            count_synergies = count_synergies +1 

        if word in intagibles:

            count_intagibles = count_intagibles+1

        if word in customers:

            count_customers = count_customers +1

        if word in technology:

            count_technology = count_technology +1

        if word in human:

            count_human = count_human +1

        if word == 'The':

            count_the = count_the +1

    length = len(text.split())    

    a={"File": name, "Text": text,'the':count_the, 'Datum': date, 'File_type': type_1, 'length':length, 'grwoth':count_growth, 'synergies': count_synergies,'intagibles':count_intagibles,'customers':count_customers, 'technology':count_technology,'human':count_human,}

    result_list.append(a)