WiddlZddlmZddlmZddlmZmZddlm Z ddl m Z m Z m Z mZmZmZddlmZddlmZmZd ZGd d eZy) N)tqdm) csr_matrix)pipelineset_seed)Pipeline)MappingListTupleAnyUnionCallable)BaseRepresentation)truncate_document%validate_truncate_document_parametersz\ I have a topic described by the following keywords: [KEYWORDS]. The name of this topic is: ceZdZdZdidddddfdeeefdedzdeeefde d e d e dzd e dzd eee fdzfd Z de jdedeeeeee ffdeeeeee fffdZdZy)TextGenerationaw Text2Text or text generation with transformers. Arguments: model: A transformers pipeline that should be initialized as "text-generation" for gpt-like models or "text2text-generation" for T5-like models. For example, `pipeline('text-generation', model='gpt2')`. If a string is passed, "text-generation" will be selected by default. prompt: The prompt to be used in the model. If no prompt is given, `self.default_prompt_` is used instead. NOTE: Use `"[KEYWORDS]"` and `"[DOCUMENTS]"` in the prompt to decide where the keywords and documents need to be inserted. pipeline_kwargs: Kwargs that you can pass to the transformers.pipeline when it is called. random_state: A random state to be passed to `transformers.set_seed` nr_docs: The number of documents to pass to OpenAI if a prompt with the `["DOCUMENTS"]` tag is used. diversity: The diversity of documents to pass to OpenAI. Accepts values between 0 and 1. A higher values results in passing more diverse documents whereas lower values passes more similar documents. doc_length: The maximum length of each document. If a document is longer, it will be truncated. If None, the entire document is passed. tokenizer: The tokenizer used to calculate to split the document into segments used to count the length of a document. * If tokenizer is 'char', then the document is split up into characters which are counted to adhere to `doc_length` * If tokenizer is 'whitespace', the document is split up into words separated by whitespaces. These words are counted and truncated depending on `doc_length` * If tokenizer is 'vectorizer', then the internal CountVectorizer is used to tokenize the document. These tokens are counted and truncated depending on `doc_length` * If tokenizer is a callable, then that callable is used to tokenize the document. These tokens are counted and truncated depending on `doc_length` Usage: To use a gpt-like model: ```python from bertopic.representation import TextGeneration from bertopic import BERTopic # Create your representation model generator = pipeline('text-generation', model='gpt2') representation_model = TextGeneration(generator) # Use the representation model in BERTopic on top of the default pipeline topic_model = BERTo pic(representation_model=representation_model) ``` You can use a custom prompt and decide where the keywords should be inserted by using the `[KEYWORDS]` or documents with thte `[DOCUMENTS]` tag: ```python from bertopic.representation import TextGeneration prompt = "I have a topic described by the following keywords: [KEYWORDS]. Based on the previous keywords, what is this topic about?"" # Create your representation model generator = pipeline('text2text-generation', model='google/flan-t5-base') representation_model = TextGeneration(generator) ``` N*modelpromptpipeline_kwargs random_statenr_docs diversity doc_length tokenizerc |||_t|t|trt d||_n#t|t r||_n td||nt|_ t|_ ||_ ||_ ||_ ||_||_t!|j|jg|_y)Nztext-generation)rzwMake sure that the HF model that youpass is either a string referring to aHF model or a `transformers.pipeline` object.)rr isinstancestrrrr ValueErrorDEFAULT_PROMPTrdefault_prompt_rrrrrrprompts_) selfrrrrrrrrs s/home/sietch6/trending-topics-pipeline/venv/lib/python3.12/site-packages/bertopic/representation/_textgeneration.py__init__zTextGeneration.__init__Us) eS !!"35ADJ x (DJ@  !' 2f -. "$"-dnndooN documentsc_tf_idftopicsreturnc (|jtk7r>d|jvr0|j|||d|j|j\}}}}n|j Dcic]}|d}}i}t |j|j D]\}} | 0| D cgc]$} t||j|j| &c} n| } |j| ||} |jj| |j| fi|j } | Dcgc]}|dj#| ddf} }t%| dkr(| t'dt%| z Dcgc]}d c}z } | ||<|Scc}wcc} wcc}wcc}w) aOExtract topic representations and return a single label. Arguments: topic_model: A BERTopic model documents: Not used c_tf_idf: Not used topics: The candidate topics as calculated with c-TF-IDF Returns: updated_topics: Updated topic representations [DOCUMENTS]iN)disablegenerated_text )r0r)rr!_extract_representative_docsrrkeysritemsverboserrr_create_promptr#appendrrreplacelenrange)r$ topic_modelr(r)r*repr_docs_mappings_topicupdated_topicsdocsdoctruncated_docsrtopic_description descriptions r%extract_topicszTextGeneration.extract_topicsws& ;;. (]dkk-I*5*R*R)VS$,,+ ' 1ar^s6#+0==<c k'kr'