WiF,ddlZddlZddlZddlmZddlmZddlmZm Z m Z m Z m Z m Z ddlmZddlmZmZmZdZdZGd d eZd Zy) N)tqdm) csr_matrix)MappingListTupleAnyUnionCallable)BaseRepresentation)retry_with_exponential_backofftruncate_document%validate_truncate_document_parametersa+You will extract a short topic label from given documents and keywords. Here are two examples of topics you created before: # Example 1 Sample texts from this topic: - Traditional diets in most cultures were primarily plant-based with a little meat on top, but with the rise of industrial style meat production and factory farming, meat has become a staple food. - Meat, but especially beef, is the worst food in terms of emissions. - Eating meat doesn't make you a bad person, not eating meat doesn't make you a good one. Keywords: meat beef eat eating emissions steak food health processed chicken topic: Environmental impacts of eating meat # Example 2 Sample texts from this topic: - I have ordered the product weeks ago but it still has not arrived! - The website mentions that it only takes a couple of days to deliver but I still have not received mine. - I got a message stating that I received the monitor but that is not true! - It took a month longer to deliver than was advised... Keywords: deliver weeks product shipping long delivery received arrived arrive week topic: Shipping and delivery issues # Your task Sample texts from this topic: [DOCUMENTS] Keywords: [KEYWORDS] Based on the information above, extract a short topic label (three words at most) in the following format: topic: z@You are an assistant that extracts high-level topics from texts.ceZdZdZdddiddddddf dededzdedzd eeefd edzd ed e d edzde dzde ee fdzfdZ de jdedeeeeeeffdeeeeeefffdZdZedZy)OpenAIa}Using the OpenAI API to generate topic labels based on one of their Completion of ChatCompletion models. For an overview see: https://platform.openai.com/docs/models Arguments: client: A `openai.OpenAI` client model: Model to use within OpenAI, defaults to `"gpt-4o-mini"`. generator_kwargs: Kwargs passed to `openai.Completion.create` for fine-tuning the output. prompt: The prompt to be used in the model. If no prompt is given, `self.default_prompt_` is used instead. NOTE: Use `"[KEYWORDS]"` and `"[DOCUMENTS]"` in the prompt to decide where the keywords and documents need to be inserted. system_prompt: The system prompt to be used in the model. If no system prompt is given, `self.default_system_prompt_` is used instead. delay_in_seconds: The delay in seconds between consecutive prompts in order to prevent RateLimitErrors. exponential_backoff: Retry requests with a random exponential backoff. A short sleep is used when a rate limit error is hit, then the requests is retried. Increase the sleep length if errors are hit until 10 unsuccessful requests. If True, overrides `delay_in_seconds`. nr_docs: The number of documents to pass to OpenAI if a prompt with the `["DOCUMENTS"]` tag is used. diversity: The diversity of documents to pass to OpenAI. Accepts values between 0 and 1. A higher values results in passing more diverse documents whereas lower values passes more similar documents. doc_length: The maximum length of each document. If a document is longer, it will be truncated. If None, the entire document is passed. tokenizer: The tokenizer used to calculate to split the document into segments used to count the length of a document. * If tokenizer is 'char', then the document is split up into characters which are counted to adhere to `doc_length` * If tokenizer is 'whitespace', the document is split up into words separated by whitespaces. These words are counted and truncated depending on `doc_length` * If tokenizer is 'vectorizer', then the internal CountVectorizer is used to tokenize the document. These tokens are counted and truncated depending on `doc_length` * If tokenizer is a callable, then that callable is used to tokenize the document. These tokens are counted and truncated depending on `doc_length` Usage: To use this, you will need to install the openai package first: `pip install openai` Then, get yourself an API key and use OpenAI's API as follows: ```python import openai from bertopic.representation import OpenAI from bertopic import BERTopic # Create your representation model client = openai.OpenAI(api_key=MY_API_KEY) representation_model = OpenAI(client, delay_in_seconds=5) # Use the representation model in BERTopic on top of the default pipeline topic_model = BERTopic(representation_model=representation_model) ``` You can also use a custom prompt: ```python prompt = "I have the following documents: [DOCUMENTS] \nThese documents are about the following topic: '" representation_model = OpenAI(client, prompt=prompt, delay_in_seconds=5) ``` To choose a model: ```python representation_model = OpenAI(client, model="gpt-4o-mini", delay_in_seconds=10) ``` z gpt-4o-miniNFmodelprompt system_promptgenerator_kwargsdelay_in_secondsexponential_backoffnr_docs diversity doc_length tokenizerc  r||_||_| t|_n||_| t|_n||_t|_t|_||_||_ ||_ | |_ | |_ | |_ t|j|jg|_||_|j j#dr#|j#d|_|j d=|j j#dr |j d=|j j#dsd|j d<yy)Nrrstop )clientrDEFAULT_CHAT_PROMPTrDEFAULT_SYSTEM_PROMPTrdefault_prompt_default_system_prompt_rrrrrrrprompts_rget) selfrrrrrrrrrrrkwargss k/home/sietch6/trending-topics-pipeline/venv/lib/python3.12/site-packages/bertopic/representation/_openai.py__init__zOpenAI.__init__s  >-DK DK  !6D !.D 2&;# 0#6  "$"-dnndooN 0  $ $W -)--g6DJ%%g.  $ $X .%%h/$$((0,0D ! !& )1 documentsc_tf_idftopicsreturnc |j|||d|j|j\}}}}i}t|j |j D]\}} | D cgc]$} t ||j|j| &} } |j| ||} |jj| |jrtj|jd|jdd| dg} |j | d|j"}|j$rt'|j(fi|}n0|j(j*j,j.d i|}|ret1|j2dj4drB|j2dj4j6j9j;d d }nd }|d fg||<|Scc} w)aJExtract topics. Arguments: topic_model: A BERTopic model documents: All input documents c_tf_idf: The topic c-TF-IDF representation topics: The candidate topics as calculated with c-TF-IDF Returns: updated_topics: Updated topic representations i)disablesystem)rolecontentuser)rmessagesrr3ztopic: zNo label returned)_extract_representative_docsrrritemsverboser rr_create_promptr$appendrtimesleeprrrrchat_completions_with_backoffrchat completionscreatehasattrchoicesmessager3stripreplace)r& topic_modelr+r,r-repr_docs_mappings_updated_topicstopicdocsdoctruncated_docsrr5r'responselabels r(extract_topicszOpenAI.extract_topicss&'2&N&N idllDNN' #Aq!  2 8 8 : H[H[D[\ 1KE4nrsgj/ T__dnn^absNs((GF MM  ($$ 4001"d.@.@AF3H $''F ''8OO>4;;++77>>HH GH$4$4Q$7$?$?K ((+33;;AACKKIWYZ+&+QZLN5 !? 1BAts#)Gc`tt||}|jtk(r?|jj ddj |}|j ||}|S|j}d|vr!|j ddj |}d|vr|j ||}|S)Nz [KEYWORDS]z, [DOCUMENTS])nextziprr rHjoin_replace_documents)r&rNrMr-keywordsrs r(r<zOpenAI._create_promptsVE]+, ;;- -[[((tyy7JKF,,VT:F [[Fv% dii6IJ&00> r*cNd}|D] }|d|dz } |jd|}|S)Nr6z- rrU)rH)rrN to_replacerOs r(rYzOpenAI._replace_documentss=  'C Bse2, &J ' z: r*)__name__ __module__ __qualname____doc__strrrfloatboolintr r r)pd DataFramerrrrSr< staticmethodrYr8r*r(rr2s9Pj#!$(.0)-$)"&!%15/1/1d /1 Tz /1 "#s(+ /1 $,/1"/1/14</1$J/1h'$./1b:<<: : T%U "3445 : d5e,-- . :x&r*rc t|jjjtj fdi|S)N)errorsr8)r rArBrCopenaiRateLimitError)rr's r(r@r@sB  ) &&%%'    r*)r>rjpandasrer scipy.sparsertypingrrrrr r bertopic.representation._baser bertopic.representation._utilsr r rr r!rr@r8r*r(rqsN #==<@[Y Yxr*