Wi^ddlmZddlmZddlmZddlmZddlZ ddl m Z GddeZ y))List)TfidfTransformer) normalize) check_arrayNc eZdZdZ d dededeedzdeffd Zd de jd e jfd Z de jfd ZxZS)ClassTfidfTransformera'A Class-based TF-IDF procedure using scikit-learns TfidfTransformer as a base. ![](../algorithm/c-TF-IDF.svg) c-TF-IDF can best be explained as a TF-IDF formula adopted for multiple classes by joining all documents per class. Thus, each class is converted to a single document instead of set of documents. The frequency of each word **x** is extracted for each class **c** and is **l1** normalized. This constitutes the term frequency. Then, the term frequency is multiplied with IDF which is the logarithm of 1 plus the average number of words per class **A** divided by the frequency of word **x** across all classes. Arguments: bm25_weighting: Uses BM25-inspired idf-weighting procedure instead of the procedure as defined in the c-TF-IDF formula. It uses the following weighting scheme: `log(1+((avg_nr_samples - df + 0.5) / (df+0.5)))` reduce_frequent_words: Takes the square root of the bag-of-words after normalizing the matrix. Helps to reduce the impact of words that appear too frequently. seed_words: Specific words that will have their idf value increased by the value of `seed_multiplier`. NOTE: This will only increase the value of words that have an exact match. seed_multiplier: The value with which the idf values of the words in `seed_words` are multiplied. Examples: ```python transformer = ClassTfidfTransformer() ``` Nbm25_weightingreduce_frequent_words seed_wordsseed_multipliercb||_||_||_||_tt |yN)r r r r superr__init__)selfr r r r __class__s h/home/sietch6/trending-topics-pipeline/venv/lib/python3.12/site-packages/bertopic/vectorizers/_ctfidf.pyrzClassTfidfTransformer.__init__)s2-%:"$. #T35X multiplierctt|d}tj|stj|}tj }|j r|j\}}t jt j|jd}t|jdj}|jr%t jd||z dz|dzz z}nt j||z dz}|||z}tj|d||fd||_|S) zLearn the idf vector (global term weights). Arguments: X: A matrix of term/token counts. multiplier: A multiplier for increasing/decreasing certain IDF scores )csrcsc) accept_sparser)axisg?r)offsetsshapeformatdtype)rspissparse csr_matrixnpfloat64use_idfrsqueezeasarraysumintmeanr logdiags _idf_diag) rrrr _ n_featuresdfavg_nr_samplesidfs rfitzClassTfidfTransformer.fit6s  8{{1~ a A  <<GGMAzBJJquu!u}56B!A!3!3!56N""ffQ>B#6#<c"JKL ffnr1Q67%J&XX!:. DN rc|jrNt|ddd}|jr$tj|j |_||j z}|S)zTransform a count-based matrix to c-TF-IDF. Arguments: X (sparse matrix): A matrix of term/token counts. Returns: X (sparse matrix): A c-TF-IDF matrix rl1F)rnormcopy)r&rr r$sqrtdatar.)rrs r transformzClassTfidfTransformer.transformbsJ <<!!$U;A))DNN"Ar)FFNr)__name__ __module__ __qualname____doc__boolrstrfloatrr!r#r$ndarrayr4r; __classcell__)rs@rrr swB %&+'+!" 6 6 $ 6I$ 6  6*R]]* *X2==rr)typingrsklearn.feature_extraction.textrsklearn.preprocessingr sklearn.utilsrnumpyr$ scipy.sparsesparser!rrrrNs&<+%j,jr