WiK"ddlZddlZddlmZddlmZddlmZmZm Z m Z m Z ddl m Z ddlmZmZdZdZGd d e Zy) N)tqdm) csr_matrix)MappingListTupleUnionCallable)BaseRepresentation)truncate_document%validate_truncate_document_parametersa This is a list of texts where each collection of texts describe a topic. After each collection of texts, the name of the topic they represent is mentioned as a short-highly-descriptive title --- Topic: Sample texts from this topic: - Traditional diets in most cultures were primarily plant-based with a little meat on top, but with the rise of industrial style meat production and factory farming, meat has become a staple food. - Meat, but especially beef, is the word food in terms of emissions. - Eating meat doesn't make you a bad person, not eating meat doesn't make you a good one. Keywords: meat beef eat eating emissions steak food health processed chicken Topic name: Environmental impacts of eating meat --- Topic: Sample texts from this topic: - I have ordered the product weeks ago but it still has not arrived! - The website mentions that it only takes a couple of days to deliver but I still have not received mine. - I got a message stating that I received the monitor but that is not true! - It took a month longer to deliver than was advised... Keywords: deliver weeks product shipping long delivery received arrived arrive week Topic name: Shipping and delivery issues --- Topic: Sample texts from this topic: [DOCUMENTS] Keywords: [KEYWORDS] Topic name:z@You are an assistant that extracts high-level topics from texts.ceZdZdZ ddededzdedzdedzdededzd edzd eeefdzfd Z d e jd e de eeeeeffde eeeeefffdZdZedZy)Cohereav Use the Cohere API to generate topic labels based on their generative model. Find more about their models here: https://docs.cohere.ai/docs Arguments: client: A `cohere.Client` model: Model to use within Cohere, defaults to `"xlarge"`. prompt: The prompt to be used in the model. If no prompt is given, `self.default_prompt_` is used instead. NOTE: Use `"[KEYWORDS]"` and `"[DOCUMENTS]"` in the prompt to decide where the keywords and documents need to be inserted. system_prompt: The system prompt to be used in the model. If no system prompt is given, `self.default_system_prompt_` is used instead. delay_in_seconds: The delay in seconds between consecutive prompts in order to prevent RateLimitErrors. nr_docs: The number of documents to pass to OpenAI if a prompt with the `["DOCUMENTS"]` tag is used. diversity: The diversity of documents to pass to OpenAI. Accepts values between 0 and 1. A higher values results in passing more diverse documents whereas lower values passes more similar documents. doc_length: The maximum length of each document. If a document is longer, it will be truncated. If None, the entire document is passed. tokenizer: The tokenizer used to calculate to split the document into segments used to count the length of a document. * If tokenizer is 'char', then the document is split up into characters which are counted to adhere to `doc_length` * If tokenizer is 'whitespace', the document is split up into words separated by whitespaces. These words are counted and truncated depending on `doc_length` * If tokenizer is 'vectorizer', then the internal CountVectorizer is used to tokenize the document. These tokens are counted and truncated depending on `doc_length` * If tokenizer is a callable, then that callable is used to tokenize the document. These tokens are counted and truncated depending on `doc_length` Usage: To use this, you will need to install cohere first: `pip install cohere` Then, get yourself an API key and use Cohere's API as follows: ```python import cohere from bertopic.representation import Cohere from bertopic import BERTopic # Create your representation model co = cohere.Client(my_api_key) representation_model = Cohere(co) # Use the representation model in BERTopic on top of the default pipeline topic_model = BERTopic(representation_model=representation_model) ``` You can also use a custom prompt: ```python prompt = "I have the following documents: [DOCUMENTS]. What topic do they contain?" representation_model = Cohere(co, prompt=prompt) ``` Nmodelprompt system_promptdelay_in_secondsnr_docs diversity doc_length tokenizerc ||_||_||nt|_||nt|_t|_t|_||_||_ ||_ ||_ | |_ t|j|jg|_y)N)clientrDEFAULT_PROMPTrDEFAULT_SYSTEM_PROMPTrdefault_prompt_default_system_prompt_rrrrrr prompts_) selfrrrrrrrrrs k/home/sietch6/trending-topics-pipeline/venv/lib/python3.12/site-packages/bertopic/representation/_cohere.py__init__zCohere.__init__os}  & 2f .;.G]Mb-&;# 0 "$"-dnndooN documentsc_tf_idftopicsreturnc |j|||d|j|j\}}}}i}t|j |j D]\}} | D cgc]$} t ||j|j| &} } |j| ||} |jj| |jrtj|j|jj!|j"|j$| ddg} | j&j)}|dfgt+dDcgc]}dc}z||<|Scc} wcc}w) aExtract topics. Arguments: topic_model: Not used documents: Not used c_tf_idf: Not used topics: The candidate topics as calculated with c-TF-IDF Returns: updated_topics: Updated topic representations i)disable2 )rpreamblemessage max_tokensstop_sequences )r)_extract_representative_docsrrritemsverboser rr_create_promptrappendrtimesleeprchatrrtextstriprange)r topic_modelr"r#r$repr_docs_mappings_updated_topicstopicdocsdoctruncated_docsrrequestlabels rextract_topicszCohere.extract_topicssU&'2&N&N idllDNN' #Aq!  2 8 8 : H[H[D[\ OKE4nrsgj/ T__dnn^absNs((GF MM  ($$ 4001kk&&jj++ $v 'GLL&&(E&+QZLU1X3NG3N$NN5 !# O&%t 4Os ")E E!c`tt||}|jtk(r?|jj ddj |}|j ||}|S|j}d|vr!|j ddj |}d|vr|j ||}|S)Nz [KEYWORDS]z, [DOCUMENTS])nextziprrreplacejoin_replace_documents)rrAr@r$keywordsrs rr4zCohere._create_promptsVE]+, ;;. ([[((tyy7JKF,,VT:F [[Fv% dii6IJ&00> r!cNd}|D] }|d|dz } |jd|}|S)Nr0z- r)rH)rK)rrA to_replacerBs rrMzCohere._replace_documentss=  'C Bse2, &J ' z: r!)z command-rNNNNNN)__name__ __module__ __qualname____doc__strfloatintrr r pd DataFramerrrrrFr4 staticmethodrMr!rrr)sCP!!$()-"&!%15d  Tz   $, 4<$Jh'$.6,<<, , T%U "3445 , d5e,-- . ,\&r!r)r6pandasrYr scipy.sparsertypingrrrrr bertopic.representation._baser bertopic.representation._utilsr r rrrr\r!rrbs< #88<c8[h hr!