WiF d dl Z d dlZd dlZd dlmZ d dlmZ d dlm Z m
Z
m Z m Z m
Z
mZ dZd dlmZ d dlZdZ d dlmZ dZd Zd
Zd Zd Zd
Z dZ dZ dZ d/de de de dz de dz de!de!de!de dee e!f de!fdZ"d Z#d Z$de fd Z%de fd Z&d e de fd!Z'd" Z(d#e fd$Z)d% Z*d#e fd&Z+d#e fd'Z,d(ee e jZ f fd)Z. G d* d+ej^ Z0d, Z1d- Z2d. Z3y# e$ r dZY w xY w# e$ r dZY w xY w# e$ r dZY w xY w)0 N)Path)TemporaryDirectory) create_repoget_hf_file_metadatahf_hub_download
hf_hub_urlrepo_type_and_id_from_hf_id
upload_folderTF)Union)Imagez topics.jsonz config.jsonztopic_embeddings.binz topic_embeddings.safetensorsz
ctfidf.binzctfidf.safetensorszctfidf_config.jsona3
---
tags:
- bertopic
library_name: bertopic
pipeline_tag: {PIPELINE_TAG}
---
# {MODEL_NAME}
This is a [BERTopic](https://github.com/MaartenGr/BERTopic) model.
BERTopic is a flexible and modular topic modeling framework that allows for the generation of easily interpretable topics from large datasets.
## Usage
To use this model, please install BERTopic:
```
pip install -U bertopic
```
You can use the model as follows:
```python
from bertopic import BERTopic
topic_model = BERTopic.load("{PATH}")
topic_model.get_topic_info()
```
## Topic overview
* Number of topics: {NR_TOPICS}
* Number of training documents: {NR_DOCUMENTS}
Click here for an overview of all topics.
{TOPICS}
## Training hyperparameters
{HYPERPARAMS}
## Framework versions
{FRAMEWORKS}
repo_idcommit_messagetokenrevisionprivate create_pr
model_card
serializationsave_embedding_model save_ctfidfc t s t d t |||d } t | \ } }
}|
d| }t 5 }| j ||| |
t
t |d| t |||||
cd d d S # |r-t | | }t | dz }|j |d Y LxY w# 1 sw Y y xY w) af Push your BERTopic model to a HuggingFace Hub.
Arguments:
model: The BERTopic model to push
repo_id: The name of your HuggingFace repository
commit_message: A commit message
token: Token to add if not already logged in
revision: Repository revision
private: Whether to create a private repository
create_pr: Whether to upload the model as a Pull Request
model_card: Whether to automatically create a modelcard
serialization: The type of serialization.
Either `safetensors` or `pytorch`
save_embedding_model: A pointer towards a HuggingFace model to be loaded in with
SentenceTransformers. E.g.,
`sentence-transformers/all-MiniLM-L6-v2`
save_ctfidf: Whether to save c-TF-IDF information
z\Make sure you have the huggingface hub installed via `pip install --upgrade huggingface_hub`T)r r exist_ok/)r r r z README.md)r
filenamer utf8encoding)r
folder_pathr r r N) _has_hf_hub
ValueErrorr r r saver r generate_readmer
write_textr
)modelr
r r r r r r r r r repo_url_
repo_owner repo_nametmpdir readme_text readme_paths `/home/sietch6/trending-topics-pipeline/venv/lib/python3.12/site-packages/bertopic/_save_utils.pypush_to_hf_hubr- j s > wxx 7%4PH :8 D Az9 Ai[)G
'!5#
E Gk\d!e f )
'
E -eW = "6l[ 8 &&{V&D!
s$ B>B/B>1B;9B>>Cc t | t z }t | t z }| t z }|j r t | }na| t z }|j rHt j |d }|j D ci c] \ }}||j }}} d}| t z }|j r t | }na| t z }|j rHt j |d }|j D ci c] \ }}||j }}}t | t z } d}
t rl t j | dz d} | rOt# |d j% } i }
| D ], }
t j | d |
d
z }||
t' |
< . |||| |
fS c c}}w c c}}w # d\ } }Y xY w# d} Y wxY w) zLoad local BERTopic files.cpu map_locationNNN images/0.jpgTFtopic_representationsimages/.jpg)load_cfg_from_json TOPICS_NAME CONFIG_NAMEHF_SAFE_WEIGHTS_NAMEis_fileload_safetensorsHF_WEIGHTS_NAMEtorchloaditemsnumpyCTFIDF_SAFE_WEIGHTS_NAMECTFIDF_WEIGHTS_NAMECTFIDF_CFG_NAME _has_visionr openlistkeysint)pathtopicsparamssafetensor_pathtensors
torch_pathkvctfidf_tensors
ctfidf_configimages _has_images
topic_listtopicimages r, load_local_filesrY s { 2
3F
{ 2
3F 11O "?3O+
jj%@G07
@1q!'')|@G@ 3 !99 " " $ -o >N 33J!!#!&JU!K;I;O;O;Q!R41a!QWWY,!R!R*4/+AB
F JJtn, - K f%< = B B DEJF#
+
4GE7$*?#?@%*s5z"
+ 67NM6 IIC A "S3(2%
~ Ks1
G,A*G G1G G G GG c t | } t t | t d }t t | t d } t | t
d }t
| } t t | t d } t | t d }t
| }d}t rk t | dd d}|rXt |d j }i }|D ]5 } t! j" t | d
| d d }
|
|t% | < 7 ||||||fS # t | t d }t j |d }Y xY w# t | t d }t j |d }Y xY w# d\ }}Y xY w# d}Y xY w) z Load files from HuggingFace.N)r r/ r0 r2 r3 TFr4 r5 r6 )strr7 r r8 r9 r: r< r= r> r? rD rB rC rE rG rH r rF rI ) rJ rK rL rN rS rR rT rU rV rW rX s r, load_files_from_hfr\ s t9D kD Q
RF
kD Q
RF:!$(Fc
* t }d}|j d d }| j j D ci c]
\ }}d|vs || }}}dj |j D cg c]
\ }}d| d| c}} }t t
t | j }t t t | j } | j -t t | j j }
nd }
|D cg c]4 } d
j t t | j! | dd 6 } } |D cg c] } | j# | }
} | j$ r | j$ n |D cg c] } | j& | c} }t) | D cg c] \ }} d | d
| | d
|
| d
|| d }}} |d j | z }dj t+ j D cg c]
\ }}d| d| c}} }|j- d| }|j- d| }|j- d| }|j- d|j/ }|j- d|
}|j- d| }|j- d| }t1 | }|s|j- dd }|S |j- dd }|S c c}}w c c}}w c c} w c c} w c c} w c c} }w c c}}w )z+Generate README for HuggingFace model card.zr| Topic ID | Topic Keywords | Topic Frequency | Label |
|----------|----------------|-----------------|-------|
r r$
z* z: N z - z| z | z |
z {MODEL_NAME}z{PATH}z {NR_TOPICS}z{TOPICS}z{NR_DOCUMENTS}z
{HYPERPARAMS}z {FRAMEWORKS}z{PIPELINE_TAG}ztext-classificationz pipeline_tag: {PIPELINE_TAG}
)MODEL_CARD_TEMPLATEsplit
get_paramsr@ joinsortedrG settopics_r[ len topic_sizes_sumvaluesnextzip get_topicget_topic_freqcustom_labels_
topic_labels_ enumerateget_package_versionsreplacestripcheck_has_visual_aspect)r$ r
r topic_table_head
model_nameparamvaluerL rK nr_topics nr_documentsrW topic_keywords
topic_freq topic_labelsindex
frameworkshas_visual_aspects r, r" r" s $J N s#B'J/4/?/?/A/G/G/I
b|ueW\aMaeUl
bF
b
YY&,,.Q,%"UG2eW-Q
RF
DU]]+,
-FCEMM*+,I %3u 1 1 8 8 :;< W]]Uejjc5??5+A&B!CBQ!GH]N];AB%%&&u-BJB+0+?+?5''ntEuejeFYFYZ_F`EuL &f - E5
UG3~e,-SE1B0C3|TYGZF[[`aF
/FFZF\FbFbFdeleUbr% 1efJ ##NJ?J##Hg6J##M9=J##J ?J##$4lCJ##OV
from_numpyr! r= )r$ save_directoryr rN s r, save_hfr > s hhu..bjjAG
%%w/ *>>H !AAAz%u'7'7'@A
7N_ <= " r c | j j }| j j }| j j }t j
| j j }t j
| j j j }|dk( r|||||d}t |t z | |dk( rt sJ d t j | t j | t j | t j | t j | d}t j ||t z yy)z Save c-TF-IDF sparse matrix.r )indptrindicesdatashapediagr z%`pip install pytorch` to save as .binN) c_tf_idf_r r r r r r ctfidf_model _idf_diagr rB r r> r r! rC ) r$ r r r r r r r rN s r, r r K s
__
#
#Foo%%G ?? D HHU__** +E
88E&&0055 6D
%
*BBGL !BBBz&&v. ''0$$T*%%e,$$T*
7N-@ @A "r c i }| j j | j j d|d< | j j }|d= |d= |d= t |d t s|d= || j j d|d< |j d 5 }t j ||d
d d d y # 1 sw Y y xY w)
z9Save parameters to recreate CountVectorizer and c-TF-IDF.)bm25_weightingreduce_frequent_wordsr tokenizer preprocessorr analyzer)rL vocabvectorizer_modelw indentN) r r r r rd
isinstancer[ vocabulary_rF jsondump)r$ rJ config cv_paramsfs r, save_ctfidf_configr h s
F ,,;;!&!3!3!I!I F> &&113I+ . 99W;M i
+S 1 j ! ''33"F
3 '1 &!A&' ' 's B;;CrJ c @ t | }| j }|j D ci c]
\ }}d|vs || }}}t |t r||d< |j d 5 }t
j ||d ddd |S c c}}w # 1 sw Y |S xY w)z Save BERTopic configuration.r$ embedding_modelr r r N)r rd r@ r r[ rF r r )r$ rJ r rL rz r{ r r s r, save_configr s :D
F/5||~
V|uePUAUeUl
VF
V /3'$3 !
3 '1 &!A&' M W ' Ms B
B
*BB c t rB| j j D ]$ \ }}t |d t j s$ y yy)z!Check if model has visual aspect.r TN)rE topic_aspects_r@ r r )r$ aspectr{ s r, rw rw sA " 1 1 7 7 9 MFE%(EKK0 r c @ t rd}| j j D ]3 \ }}t |d t j s$| j | } n |C|j dd |j D ] \ }}|j
|| dz yyy)zSave topic images.Nr T)r parentsr6 )rE r r@ r r mkdirr! )r$ rJ visual_aspectsr r{ rW rX s r, save_imagesr s " 1 1 7 7 9 MFE%(EKK0!&!5!5f!=
% JJdJ 3 . 4 4 6
2 u
4UG4.01
2 & r c t | }t rMi }| j j D ]- \ }}t |d t
j
s|||< )d|d< / n | j }| j | j D cg c]
}t | c}| j t j | j j t j | j | j t | j" |d}|j% d 5 }t' j( ||dt* d d d y c c}w # 1 sw Y y xY w)
z Save Topic-specific information.r T
Visual_Aspectr )r4 rK topic_sizes topic_mapperr
custom_labels _outliers
topic_aspectsr r )r clsN)r rE r r@ r r topic_representations_rh rI rj r r
topic_mapper_ mappings_tolistrr rq r rF r r NumpyEncoder)r$ rJ selected_topic_aspectsr r{ rW rK r s r, save_topicsr s! :D!# " 1 1 7 7 9 ?MFE eAh 416&v.:>&7 ? "'!5!5 "'!=!=+0==9%3u:9 ))!4!4!>!>cJQQS ++ -- )/ F
3 91 &!A<89 9 :9 9s E E
E json_filec t | dd 5 }|j }ddd t j S # 1 sw Y xY w)z Load configuration from json.rzutf-8r N)rF readr loads)r readertexts r, r7 r7 s?
iw / 6{{} ::d s =Ac e Zd Z fdZ xZS )r c t |t j r t | S t |t j r t | S t t | # | S )N) r r integerrI floatingfloatsuperr default)selfobj __class__s r, r zNumpyEncoder.default sD c2:: &s8O c2;; ': \4055r )__name__
__module__ __qualname__r
__classcell__)r s @r, r r s
6 6r r c ddl } ddlm} ddlm} ddlm} ddlm} ddlm} |d } ddl
m} ddl m} ddl m} ddl
m}
|||||||
| || j d
S # t t f$ r d}Y Jw xY w# t t f$ r d}Y Zw xY w# t t f$ r d}Y jw xY w# t t f$ r d} Y zw xY w# t t f$ r d}
Y w xY w# t $ r } | cY d} ~ S d} ~ ww xY w)z.Get versions of main dependencies of BERTopic.r N) __version__)versionhdbscan)
NumpyHDBSCANUMAPPandasz Scikit-LearnzSentence-transformers TransformersNumbaPlotlyPython)platformrA r pandassklearnplotlyimportlib.metadatar ImportErrorModuleNotFoundErrorumapsentence_transformersnumba transformerspython_version Exception) r
np_versionpandas_versionsklearn_versionplotly_versionr hdbscan_version umap_version
sbert_version
numba_versiontransformers_versiones r, rt rt s /38:8 # 2 %i 0O 8 ! J ! : ( H
& $ +%2 0"$ --/
/ 01 # "O #
01 L
01 ! M !
01 ! M !
01 (#' ( s C A$ A; B B) C C $A85C 7A88C ;B C BC B&#C %B&&C )B=:C <B==C CC CC C, C'!C,'C,c r ddl }|j j | S # t $ r t d w xY w)z3Load safetensors and check whether it is installed.r Nz.`pip install safetensors` to load .safetensors)safetensors.numpyrA load_filer r )rJ r s r, r< r< s? K **400 KIJJKs ! 6c v ddl }|j j || y# t $ r t d w xY w)z3Save safetensors and check whether it is installed.r Nz1`pip install safetensors` to save as .safetensors)r rA save_filer r )rJ rN r s r, r r s> N ##GT2 NLMMNs # 8) zAdd BERTopic modelNNFFTr TF)4osr rA r pathlibr tempfiler huggingface_hubr r r r r r
r r typingr r> r PILr rE r8 r9 r= r: rC rB rD rb r[ boolr- rY r\ r" r r r r rw r r PathLiker7 JSONEncoderr rt r< r r r, r s ' K
J
K
(5 " / &1 n / &-1 A
A
A
: A
Dj A
A
A
A
A
T *A
A
H/Jd,J^,C ,^
>#
>Bs B3 B:'2
S
2S 2 9S 9: %R[[(8"9 64## 61hKN} K J Ks3 C% C2 C? %C/.C/2C<;C<?D D