WiVddlZddlmZmZddlmZddlmZddl m Z Gdde Z y)N)ListUnion) StaticModel)CountVectorizer) BaseEmbedderc eZdZdZdidfdeeefdedededzffd Z dd e ed ed e jfd Z d ZxZS)Model2VecBackendaLModel2Vec embedding model. Arguments: embedding_model: Either a model2vec model or a string pointing to a model2vec model distill: Indicates whether to distill a sentence-transformers compatible model. The distillation will happen during fitting of the topic model. NOTE: Only works if `embedding_model` is a string. distill_kwargs: Keyword arguments to pass to the distillation process of `model2vec.distill.distill` distill_vectorizer: A CountVectorizer used for creating a custom vocabulary based on the same documents used for topic modeling. NOTE: If "vocabulary" is in `distill_kwargs`, this will be ignored. Examples: To create a model, you can load in a string pointing to a model2vec model: ```python from bertopic.backend import Model2VecBackend sentence_model = Model2VecBackend("minishlab/potion-base-8M") ``` or you can instantiate a model yourself: ```python from bertopic.backend import Model2VecBackend from model2vec import StaticModel embedding_model = StaticModel.from_pretrained("minishlab/potion-base-8M") sentence_model = Model2VecBackend(embedding_model) ``` If you want to distill a sentence-transformers model with the vocabulary of the documents, run the following: ```python from bertopic.backend import Model2VecBackend sentence_model = Model2VecBackend("sentence-transformers/all-MiniLM-L6-v2", distill=True) ``` FNembedding_modeldistilldistill_kwargsdistill_vectorizerct|||_||_||_d|_|jrN|j |jst|_t|tr||_ ytdt|tr||_ yt|trtj||_ ytd)NFzNPlease pass a string pointing to a sentence-transformer model when distilling.zPlease select a correct Model2Vec model: `from model2vec import StaticModel` `model = StaticModel.from_pretrained('minishlab/potion-base-8M')`)super__init__r r r _has_distilled_check_model2vec_installationr isinstancestrr ValueErrorrfrom_pretrained)selfr r r r __class__s g/home/sietch6/trending-topics-pipeline/venv/lib/python3.12/site-packages/bertopic/backend/_model2vec.pyrzModel2VecBackend.__init__6s  ,"4# <<  . . 0***9*;'/3/'6$ !qrr 5#2D  -#.#>#>#OD T  documentsverbosereturncT|jr|jsddlm}|jj ds|j j |}tj|jdj}|j j}tt||ddDcgc]\}}| } }}| |jd<||jfi|j|_d|_|jj||} | Scc}}w) aEmbed a list of n documents/words into an n-dimensional matrix of embeddings. Arguments: documents: A list of documents or words to be embedded verbose: Controls the verbosity of the process Returns: Document/words embeddings with shape (n, m) with `n` documents/words that each have an embeddings size of `m` rr vocabulary)axisc |dS)N)xs rz(Model2VecBackend.embed..os `abc`drT)keyreverse)show_progress_bar)r rmodel2vec.distillr getr fit_transformnparraysumflattenget_feature_names_outsortedzipr encode) rrrr X word_countswordsword_r embeddingss rembedzModel2VecBackend.embedZs << 3 3 1&&**<8++99)D hhquu!u}5==? //EEG28UK9PVdnr2stwtQdt t4>##L1$+4+?+?#W4CVCV#WD #'D ))00g0V us> D$cB ddlm}y#t$r tdwxYw)NrrzTTo distill a model using model2vec, you need to run `pip install model2vec[distill]`)r*r ImportError)rr s rrz.Model2VecBackend._check_model2vec_installation}s( v 1 vtu u vs )F)__name__ __module__ __qualname____doc__rrrbooldictrrr-ndarrayr;r __classcell__)rs@rr r sw*^!)- "sK/0"" "  $J "H!tCy!4!BJJ!Fvrr ) numpyr-typingrr model2vecrsklearn.feature_extraction.textrbertopic.backendrr r$rrrKs$!;)xv|xvr