Wi ddlZddlZddlmZdeedfdeeefdedefdZdedeffdZ dd e d e d e d ed e dzf dZ y)N)Union doc_length tokenizerdocumentreturnch||dk(r|d|}|S|dk(r$dj|jd|}|S|dk(r6|jj}dj||d|}|St |dr1t |dr%|j |}|j |d|}S|S)ahTruncate a document to a certain length. If you want to add a custom tokenizer, then it will need to have a `decode` and `encode` method. An example would be the following custom tokenizer: ```python class Tokenizer: 'A custom tokenizer that splits on commas' def encode(self, doc): return doc.split(",") def decode(self, doc_chunks): return ",".join(doc_chunks) ``` You can use this tokenizer by passing it to the `tokenizer` parameter. Arguments: topic_model: A BERTopic model doc_length: The maximum length of each document. If a document is longer, it will be truncated. If None, the entire document is passed. tokenizer: The tokenizer used to calculate to split the document into segments used to count the length of a document. * If tokenizer is 'char', then the document is split up into characters which are counted to adhere to `doc_length` * If tokenizer is 'whitespace', the document is split up into words separated by whitespaces. These words are counted and truncated depending on `doc_length` * If tokenizer is 'vectorizer', then the internal CountVectorizer is used to tokenize the document. These tokens are counted and truncated depending on `doc_length`. They are decoded with whitespaces. * If tokenizer is a callable, then that callable is used to tokenize the document. These tokens are counted and truncated depending on `doc_length` document: A single document Returns: truncated_document: A truncated document Nchar whitespace vectorizerencodedecode)joinsplitvectorizer_modelbuild_tokenizerhasattrr r) topic_modelrrrtruncated_documentencoded_documents j/home/sietch6/trending-topics-pipeline/venv/lib/python3.12/site-packages/bertopic/representation/_utils.pytruncate_documentrsR  !)+:!6 "!, &!$(..*:;J*G!H "! , &#44DDFI!$)H*=kz*J!K "!Y )gi.J(//9 !*!1!12B;J2O!P !! OcB| | td|| tdyy)zGValidates parameters that are used in the function `truncate_document`.NzPlease select from one of the valid options for the `tokenizer` parameter: {'char', 'whitespace', 'vectorizer'} If `tokenizer` is of type callable ensure it has methods to encode and decode a document zNIf `tokenizer` is provided, `doc_length` of type int must be provided as well.) ValueError)rrs r%validate_truncate_document_parametersr>sAZ3 j   :#5ijj$6 r initial_delayexponential_basejitter max_retrieserrorsc&fd}|S)z*Retry a function with exponential backoff.cd}} |i|S#$rQ|dz }| kDrtd d|d tjzzzz}tj|Ynt$r}|d}~wwxYwq)NrzMaximum number of retries (z ) exceeded.) Exceptionrandomtimesleep) argskwargs num_retriesdelayer!rfuncrrr s rwrapperz/retry_with_exponential_backoff..wrapperTs  T,V,, "q ,#&A+k$Z[[)Q&--/1I-IJJ 5!  )sAA4%A4-A//A4)r.rrrr r!r/s`````` rretry_with_exponential_backoffr1Js: Nr)r$T N)r&r'typingrintstrcallablerrrfloatbooltupler1r0rrr;s 5uS$Y/?5ERUW_R_L`5lo5tw5p kE$PZJZD[ k '''  '  ' DL 'r