> ## Documentation Index
> Fetch the complete documentation index at: https://docs.exploratory.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Calculate TF-IDF

## How to Access This Feature

### From + (plus) Button

You can access it from 'Add' (Plus) button. "Text Mining..." -> "Calculate TF-IDF".

![](https://mintcdn.com/exploratory-inc/VL_CUSf6Xsk2Ztlz/en/text/images/do_tfidf_add.png?fit=max\&auto=format\&n=VL_CUSf6Xsk2Ztlz\&q=85\&s=1b3572f32c7d139f41457e087ff3b7c4)

## How to Use?

![](https://mintcdn.com/exploratory-inc/VL_CUSf6Xsk2Ztlz/en/text/images/do_tfidf_param.png?fit=max\&auto=format\&n=VL_CUSf6Xsk2Ztlz\&q=85\&s=1ebc2df2ccb2393c4fc8e15d40d5358b)

* Select a column as document id - A column considered as document id. If you run [do\_tokenize](https://docs.exploratory.io/en/text/do_tokenize.md) beforehand, this can be document\_id.
* Select a column that has tokenized text - Set a column that has tokens. This is "token" column if it's tokenized by [do\_tokenize](https://docs.exploratory.io/en/text/do_tokenize.md) function.
* TF Weight (Optional) - The default is "raw".
  * "raw" is count of a term in a document.
  * "binary" is if it exists or not. If it exists, it is 1 and if not, it is 0.
  * "log\_scale" is `1+log(count of a term in a document)`.
* IDF Log Scale Function (Optional) - The default is log. This is a function to suppress the increase of idf value. Idf is calculated by `log_scale_function((the total number of documents)/(the number of documents which have the token))`. It's how rare the token is in the set of documents. It might be worth trying log2 or log10. log2 increases the value more easily and log10 increases it more slowly.
* Normalization (Optional) - The default is l2. How to normalize the tfidf vector.
  * "l2" is normalization that Euclidean distance of the tfidf vector for a document becomes 1.
  * "l1" is normalization that Manhattan distance (sum of values) of the tfidf vector for a document becomes 1.
  * FALSE doesn't normalize the result.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.