Wi$^ddlZddlmZddlmZddlmZddlm Z ddl m Z Gdde Z y) N)chain)List)sparse) csr_matrix)CountVectorizercneZdZdZd dedzdedzffd ZdeeddfdZdeede fd Z d d Z xZ S) OnlineCountVectorizera An online variant of the CountVectorizer with updating vocabulary. At each `.partial_fit`, its vocabulary is updated based on any OOV words it might find. Then, `.update_bow` can be used to track and update the Bag-of-Words representation. These functions are separated such that the vectorizer can be used in iteration without updating the Bag-of-Words representation can might speed up the fitting process. However, the `.update_bow` function is used in BERTopic to track changes in the topic representations and allow for decay. This class inherits its parameters and attributes from: `sklearn.feature_extraction.text.CountVectorizer` Arguments: decay: A value between [0, 1] to weight the percentage of frequencies the previous bag-of-words should be decreased. For example, a value of `.1` will decrease the frequencies in the bag-of-words matrix with 10% at each iteration. delete_min_df: Delete words at each iteration from its vocabulary that are below a minimum frequency. This will keep the resulting bag-of-words matrix small such that it does not explode in size with increasing vocabulary. If `decay` is None then this equals `min_df`. **kwargs: Set of parameters inherited from: `sklearn.feature_extraction.text.CountVectorizer` In practice, this means that you can still use parameters from the original CountVectorizer, like `stop_words` and `ngram_range`. Attributes: X_ (scipy.sparse.csr_matrix) : The Bag-of-Words representation Examples: ```python from bertopic.vectorizers import OnlineCountVectorizer vectorizer = OnlineCountVectorizer(stop_words="english") for index, doc in enumerate(my_docs): vectorizer.partial_fit(doc) # Update and clean the bow every 100 iterations: if index % 100 == 0: X = vectorizer.update_bow() ``` To use the model in BERTopic: ```python from bertopic import BERTopic from bertopic.vectorizers import OnlineCountVectorizer vectorizer_model = OnlineCountVectorizer(stop_words="english") topic_model = BERTopic(vectorizer_model=vectorizer_model) ``` References: Adapted from: https://github.com/idoshlomo/online_vectorizers Ndecay delete_min_dfc H||_||_tt|di|y)N)r r superr __init__)selfr r kwargs __class__s k/home/sietch6/trending-topics-pipeline/venv/lib/python3.12/site-packages/bertopic/vectorizers/_online_cv.pyrzOnlineCountVectorizer.__init__Gs% * #T3=f= raw_documentsreturnc*t|ds|j|S|j}|Dcgc] }|| }}tt j |}|j t|jj}|ryt|jj}tt|tt|dz|dzt|zd}|jj!||Scc}w)zPerform a partial fit and update vocabulary with OOV tokens. Arguments: raw_documents: A list of documents vocabulary_)hasattrfitbuild_analyzersetr from_iterable differencerkeysmaxvaluesdictziplistrangelenupdate) rranalyzerdocanalyzed_documents new_tokens oov_tokens max_indexoov_vocabularys r partial_fitz!OnlineCountVectorizer.partial_fitLs t]+88M* *&&(7DEhsmEE,,-?@A **3t/?/?/D/D/F+GH D,,3356I!y1}i!mc*o.MqQRN    # #N 3 FsDct|drB|j|}t|jjd|jd|jjdz ft }t j|j|g|_t|jd|jjdz |jjdft }t j|j|g|_|j!|jd|jz z|_|xj|z c_n|j||_|j|j|jS)asCreate or update the bag-of-words matrix. Update the bag-of-words matrix by adding the newly transformed documents. This may add empty columns if new words are found and/or add empty rows if new topics are found. During this process, the previous bag-of-words matrix might be decayed if `self.decay` has been set during init. Similarly, words that do not exceed `self.delete_min_df` are removed from its vocabulary and bag-of-words matrix. Arguments: raw_documents: A list of documents Returns: X_: Bag-of-words matrix X_rr)dtype) r transformrr2shapeintrhstackvstackr r _clean_bow)rrXcolumnsrowss r update_bowz OnlineCountVectorizer.update_bowfs#$ 4 }-A!$''--"2AGGAJqAQ4Q!RZ]^GmmTWWg$67DGqwwqzDGGMM!,<NOWZ[DmmTWWdO4DGzz%''Q^4 GGqLGnn]3DG    ) OO wwrctj|jjd|jk\d}|Dcic]}||}}|jdd|f|_i}|j j Dcic]\}}|| }}}t|D]\}}|j|||||<!||_ycc}wcc}}w)z5Remove words that do not exceed `self.delete_min_df`.rrN) npwherer2sumr ritems enumerateget) rindicesindex indices_dict new_vocabkvvocabulary_dictis rr9z OnlineCountVectorizer._clean_bows((477;;q>T-?-??@C29:u : :''!W*% ,0,<,<,B,B,DEDAq1a4EE!'* 6HAu&245 /%01 6%; Fs  C  C)NN)rN) __name__ __module__ __qualname____doc__floatrrstrr0rr=r9 __classcell__)rs@rr r sX9v>edl>%$,> ct4(S (j(T%rr ) numpyr? itertoolsrtypingrscipyr scipy.sparsersklearn.feature_extraction.textrr r rrrZs%#;S%OS%r