WiR#vddlZddlmZddlmZddlmZmZm Z m Z m Z ddl m Z ddlmZmZdZGdd e Zy) N)Document) csr_matrix)CallableMappingListTupleUnion)BaseRepresentation)truncate_document%validate_truncate_document_parametersz;What are these documents about? Please give a single label.c eZdZdZ ddedzdededzdedzdeeefdzf dZ d e jd e d e eeeeeffd e eeeeefffd Zy) LangChainaUsing chains in langchain to generate topic labels. The classic example uses `langchain.chains.question_answering.load_qa_chain`. This returns a chain that takes a list of documents and a question as input. You can also use Runnables such as those composed using the LangChain Expression Language. Arguments: chain: The langchain chain or Runnable with a `batch` method. Input keys must be `input_documents` and `question`. Output key must be `output_text`. prompt: The prompt to be used in the model. If no prompt is given, `self.default_prompt_` is used instead. NOTE: Use `"[KEYWORDS]"` in the prompt to decide where the keywords need to be inserted. Keywords won't be included unless indicated. Unlike other representation models, Langchain does not use the `"[DOCUMENTS]"` tag to insert documents into the prompt. The load_qa_chain function formats the representative documents within the prompt. nr_docs: The number of documents to pass to LangChain diversity: The diversity of documents to pass to LangChain. Accepts values between 0 and 1. A higher values results in passing more diverse documents whereas lower values passes more similar documents. doc_length: The maximum length of each document. If a document is longer, it will be truncated. If None, the entire document is passed. tokenizer: The tokenizer used to calculate to split the document into segments used to count the length of a document. * If tokenizer is 'char', then the document is split up into characters which are counted to adhere to `doc_length` * If tokenizer is 'whitespace', the document is split up into words separated by whitespaces. These words are counted and truncated depending on `doc_length` * If tokenizer is 'vectorizer', then the internal CountVectorizer is used to tokenize the document. These tokens are counted and truncated depending on `doc_length`. They are decoded with whitespaces. * If tokenizer is a callable, then that callable is used to tokenize the document. These tokens are counted and truncated depending on `doc_length` chain_config: The configuration for the langchain chain. Can be used to set options like max_concurrency to avoid rate limiting errors. Usage: To use this, you will need to install the langchain package first. Additionally, you will need an underlying LLM to support langchain, like openai: `pip install langchain` `pip install openai` Then, you can create your chain as follows: ```python from langchain.chains.question_answering import load_qa_chain from langchain.llms import OpenAI chain = load_qa_chain(OpenAI(temperature=0, openai_api_key=my_openai_api_key), chain_type="stuff") ``` Finally, you can pass the chain to BERTopic as follows: ```python from bertopic.representation import LangChain # Create your representation model representation_model = LangChain(chain) # Use the representation model in BERTopic on top of the default pipeline topic_model = BERTopic(representation_model=representation_model) ``` You can also use a custom prompt: ```python prompt = "What are these documents about? Please give a single label." representation_model = LangChain(chain, prompt=prompt) ``` You can also use a Runnable instead of a chain. The example below uses the LangChain Expression Language: ```python from bertopic.representation import LangChain from langchain.chains.question_answering import load_qa_chain from langchain.chat_models import ChatAnthropic from langchain.schema.document import Document from langchain.schema.runnable import RunnablePassthrough from langchain_experimental.data_anonymizer.presidio import PresidioReversibleAnonymizer prompt = ... llm = ... # We will construct a special privacy-preserving chain using Microsoft Presidio pii_handler = PresidioReversibleAnonymizer(analyzed_fields=["PERSON"]) chain = ( { "input_documents": ( lambda inp: [ Document( page_content=pii_handler.anonymize( d.page_content, language="en", ), ) for d in inp["input_documents"] ] ), "question": RunnablePassthrough(), } | load_qa_chain(representation_llm, chain_type="stuff") | (lambda output: {"output_text": pii_handler.deanonymize(output["output_text"])}) ) representation_model = LangChain(chain, prompt=representation_prompt) ``` Npromptnr_docs diversity doc_length tokenizerc||_||nt|_t|_||_||_||_||_||_t|j|jy)N) chainDEFAULT_PROMPTrdefault_prompt_ chain_configrrrrr )selfrrrrrrrs n/home/sietch6/trending-topics-pipeline/venv/lib/python3.12/site-packages/bertopic/representation/_langchain.py__init__zLangChain.__init__sW & 2f -( "$"-dnndooN documentsc_tf_idftopicsreturnc|j|||d|j|j\}}}}|jDcgc]<}|Dcgc].}t t ||j |j|0c}>} }}d|jvr{g} |D]R} tt|| } |jjddj| } | j| Tt| | D cgc] \}} || d }}} n| Dcgc]}||jd}}|jj||j }|Dcgc]}|dj#}}t|j%|D cic]%\} }| |d fgt'd Dcgc]}d c}z'}}} }|Scc}wcc}}wcc} }wcc}wcc}wcc}wcc}}} w) aJExtract topics. Arguments: topic_model: A BERTopic model documents: All input documents c_tf_idf: The topic c-TF-IDF representation topics: The candidate topics as calculated with c-TF-IDF Returns: updated_topics: Updated topic representations i)rrr nr_samples nr_repr_docsr) page_contentz [KEYWORDS]z, )input_documentsquestion)inputsconfig output_text )r)_extract_representative_docsrrvaluesrr rrrnextzipreplacejoinappendrbatchrstripkeysrange)r topic_modelrrrrepr_docs_mappings_docsdoc chain_docspromptstopickeywordsrr'outputsoutputlabelslabelupdated_topicss rextract_topicszLangChain.extract_topicss &'2&N&Nnn 'O' #Aq!+113 ,   &7 T__VZVdVdfi&jk , ,  4;; &G 'VE] 34,,\499X;NOv& ' Y\\fhoXpq f$FCqFqV``T$DKKH`F`**""&9J9J"K>EFF&'--/FFSVVhVmVmVoqwRx  BN%EUAJ sB G 3G G>G GGG", G5G"GG")NNNNN)__name__ __module__ __qualname____doc__strintfloatr rrpd DataFramerrrrrFrrrr svv""&!%15Od O O 4< O $J Oh'$.O(<<<< < T%U "3445 < d5c?++ , <rr)pandasrOlangchain.docstore.documentr scipy.sparsertypingrrrrr bertopic.representation._baser bertopic.representation._utilsr r rrrQrrrXs.0#88<cNI"Ir