Agfdhyk

I was testing if i could get the same results for the tf-idf matrix returned by sklearn's TfidfVectorizer by computing tf and idf separatly and then multiplying both results. However, i didn't succeed.

Here's how implemented it:

corpus = ["Hello what day is today","I have no idea what day it is"]



#IDF vector

vectorizer = TfidfVectorizer(smooth_idf=False)

tf_idf = vectorizer.fit_transform(corpus)

idf = vectorizer.idf_

pd.DataFrame(np.reshape(idf,((1,-1))),columns=vectorizer.get_feature_names(),index=["IDF"])

#TF matrix

vectorizer = CountVectorizer()

tf_idf = vectorizer.fit_transform(corpus)

pd.DataFrame(tf_idf.todense(),columns=vectorizer.get_feature_names(),index=["doc1","doc2"])

But this is sklearn's return of the tf-idf, which clearly gives different values than what i would get if i multiplied my tf with my idf.

#sklearn's tf-idf

vectorizer = TfidfVectorizer(smooth_idf=False)

tf_idf = vectorizer.fit_transform(corpus)

pd.DataFrame(tf_idf.todense(),columns=vectorizer.get_feature_names(),index=["doc1","doc2"])

I don't see what am i doing differently. I even looked at sklearn's documentation on how they implemented this, but i don't see the difference:

The formula that is used to compute the tf-idf of term t is
tf-idf(d, t) = tf(t) * idf(d, t), and the idf is computed as
idf(d, t) = log [ n / df(d, t) ] + 1 (if smooth_idf=False),
where n is the total number of documents and df(d, t) is the
document frequency; the document frequency is the number of documents d
that contain term t.