Wi-&~ddlZddlZddlmZddlmZddlm Z m Z m Z m Z ddl mZddlmZddlmZGdd eZy) N)version) csr_matrix)MappingListTupleUnion)cosine_similarity)BaseRepresentation) __version__ceZdZ ddededededef dZ ddej d ed ee e e e e ffd e jd ee e e e e fff d Zd ed ee e e e e ffd ee e e e e fffdZ dd ee e e e e ffde e de e ede jd ee je e ff dZde e d ee e e e e ffde jd ee e e e e fffdZy)KeyBERTInspired top_n_words nr_repr_docs nr_samplesnr_candidate_words random_statecJ||_||_||_||_||_y)aUse a KeyBERT-like model to fine-tune the topic representations. The algorithm follows KeyBERT but does some optimization in order to speed up inference. The steps are as follows. First, we extract the top n representative documents per topic. To extract the representative documents, we randomly sample a number of candidate documents per cluster which is controlled by the `nr_samples` parameter. Then, the top n representative documents are extracted by calculating the c-TF-IDF representation for the candidate documents and finding, through cosine similarity, which are closest to the topic c-TF-IDF representation. Next, the top n words per topic are extracted based on their c-TF-IDF representation, which is controlled by the `nr_repr_docs` parameter. Then, we extract the embeddings for words and representative documents and create topic embeddings by averaging the representative documents. Finally, the most similar words to each topic are extracted by calculating the cosine similarity between word and topic embeddings. Arguments: top_n_words: The top n words to extract per topic. nr_repr_docs: The number of representative documents to extract per cluster. nr_samples: The number of candidate documents to extract per cluster. nr_candidate_words: The number of candidate words per cluster. random_state: The random state for randomly sampling candidate documents. Usage: ```python from bertopic.representation import KeyBERTInspired from bertopic import BERTopic # Create your representation model representation_model = KeyBERTInspired() # Use the representation model in BERTopic on top of the default pipeline topic_model = BERTopic(representation_model=representation_model) ``` N)rrrrr)selfrrrrrs l/home/sietch6/trending-topics-pipeline/venv/lib/python3.12/site-packages/bertopic/representation/_keybert.py__init__zKeyBERTInspired.__init__ s,b'($"4(N documentsc_tf_idftopics embeddingsreturnc:|j||||j|j\}}}}d} |%tj|D cgc]} ||  } } |j |||}|j ||||| \} } |j| || } | Scc} w)aExtract topics. Arguments: topic_model: A BERTopic model documents: All input documents c_tf_idf: The topic c-TF-IDF representation topics: The candidate topics as calculated with c-TF-IDF embeddings: Pre-trained document embeddings. These can be used instead of an embedding model Returns: updated_topics: Updated topic representations N)_extract_representative_docsrrnp concatenate_extract_candidate_words_extract_embeddings_extract_top_words)r topic_modelrrrr_representative_docsrepr_doc_indicesrepr_embeddingsindex sim_matrixwordsupdated_topicss rextract_topicszKeyBERTInspired.extract_topicsDs,7B6^6^ i$:K:K7 3  0!   !>@nnM]>^_Uz%0_O_..{HfM!44 !46F  E00 K`s Bctt|jDcgc] }t|}}t j t t j dk\r|jj}n|jj}|j||j}|j||}tj|d} tj|| d}tj|| d}t!|D  cic]E\} }|t#|| ddd|| dddD cgc]\} } | | dkDr|| | fndc} } G}} }} } |j%D cic]$\}} |t't#| d|j&}}} |Scc}wcc} } wcc} } }} wcc} }w)aGFor each topic, extract candidate words based on the c-TF-IDF representation. Arguments: topic_model: A BERTopic model c_tf_idf: The topic c-TF-IDF representation topics: The top words per topic Returns: topics: The `self.top_n_words` per topic z1.0.0axisNr)gh㈵>)sortedlistkeysintrparsesklearn_versionvectorizer_modelget_feature_names_outget_feature_names_top_n_idx_sparser_top_n_values_sparserargsorttake_along_axis enumeratezipitemsnext)rr$rrlabellabelsr+indicesscoressorted_indicesr) word_indexscorevaluess rr!z(KeyBERTInspired._extract_candidate_wordsps"+1fkkm1D*EF#e*FF == )W]]7-C C00FFHE00BBDE//$:Q:QR11(GDFA.$$Wn1E##FNC!*& 1   u ),WU^DbD-A6%=QUSUQUCV)W%J/9.DQRz"E*Xee   ]c\h\h\jk=5RX%c6*CD,C,C#DEFFkk 3G" ls#F1!*F< F6%F< )G6F< r&r'r(c ^||j|dd}|Dcgc]%}tj||d|ddzd'}}tt |j D cgc] }|D]} |  c} }} |j| dd} t || } | | fScc}wcc} }w)a;Extract the representative document embeddings and create topic embeddings. Then extract word embeddings and calculate the cosine similarity between topic embeddings and the word embeddings. Topic embeddings are the average of representative document embeddings. Arguments: topic_model: A BERTopic model topics: The top words per topic representative_docs: A flat list of representative documents repr_doc_indices: The indices of representative documents that belong to each topic repr_embeddings: Embeddings of respective representative_docs Returns: sim: The similarity matrix between word and topic embeddings vocab: The complete vocabulary of input documents documentF)methodverboserr2r/r0)r"rmeanr5setrLr ) rr$rr&r'r(itopic_embeddingsr+wordvocabword_embeddingssims rr"z#KeyBERTInspired._extract_embeddingss4  ")==>QZdns=tOXhiSTBGGOAaD1R519$EANiiSFMMON5N$N$NOP%99% \a9b 0/BEzjOs *B$$B)rVrXc&tt|jDcgc] }t|}}i}t |D]\}}||D cgc]} |j | } } |dd| f|} t j| |j dD cgc]} | |  } } tt j| |j d| D cgc] \}} || |fc} }ddd||<|Scc}wcc} wcc} wcc} }w)aExtract the top n words per topic based on the similarity matrix between topics and words. Arguments: vocab: The complete vocabulary of input documents labels: All topic labels topics: The top words per topic sim: The similarity matrix between word and topic embeddings Returns: updated_topics: The updated topic representations Nr2) r4r5r6r7rAr)rr?rrBsort)rrVrrXrErFr,rStopicrUrGrLr) word_indicesvals rr#z"KeyBERTInspired._extract_top_wordss$+1fkkm1D*EF#e*FF!&) HAu5;E]CTu{{4(CGCG_Q'F8: 68JDL\L\K\K^8_`uGEN`L`69"''&/4K[K[J[J]:^`l6m%(2Uus#%d%N5 !  GD`%sC>D DD ) id*)N)__name__ __module__ __qualname__r7rpd DataFramerrstrrrfloatrndarrayr-r!rr"r#rrr r s"% 5)5)5) 5)  5)  5)z"& *<<* * T%U "3445 * JJ * d5e,-- .*X**T%U "3445 * d5e,-- . *d'+ $T%U "3445$"#Y $ tCy/ $  $ rzz49$ %$LCyT%U "3445ZZ  d5e,-- . rr )numpyrpandasre packagingr scipy.sparsertypingrrrrsklearn.metrics.pairwiser bertopic.representation._baser sklearnr r9r rjrrrss/#..6<2R(Rr