Wi ^ ddlmZddlmZddlmZddlmZddlZddl m Z ddl m Z d d l mZd d lmZmZd e d efdZ ddej(deedeeded df dZeGddZGddZy)) dataclass)partial)count)AnyN)PretrainedConfig)traced)PagedAttentionCache) TMP_TOKEN_ID RequestStateconfigreturnc|jdvS)z:Checks if attention mask is needed for the given (config).)z paged|eagerz paged|sdpa)_attn_implementation)rs /home/sietch6/trending-topics-pipeline/venv/lib/python3.12/site-packages/transformers/generation/continuous_batching/input_ouputs.pyattn_mask_is_neededrs  & &*G GGattention_maskcumulative_seqlens_qcumulative_seqlens_ksliding_windowcJtj|jj}t t |dz D]}||dz||z }||dz||z }||kr|dk\r ||z dz}nd}t ||||dz} t ||||dz} tj|d| | fj||j|j} tj| |} |dkDr"||z |z } | tj| | z } | |d| | f<y)uBuilds an attention mask inplace using the cumulative seqlens of the query and key. If given a sliding window, it will also apply a sliding window mask on top. The attention mask is not boolean, it uses zeroes and -inf (or its equivalent) so it's more of an attention score bias tensor. The attention mask is a block-diagonal matrix, with each block an attention mask for a single query-key pair. Each of those block is built from a causal mask and, if there is a sliding window, a sliding window mask. An example is represented below, with seqlen_k = 8, seqlen_q = 4 and sliding_window = 6: CAUSAL MASK: █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ █ █ █ █ █ █ █ ░ █ █ █ █ █ █ █ █ SLIDING WINDOW MASK: ┌──────────────────────── seqlen_k - seqlen_q - sliding_window = 8 - 4 - 6 = -2 offset to the left <─┴─> ░ █ | █ █ █ █ █ █ █ █ ░ ░ | █ █ █ █ █ █ █ █ ░ ░ | ░ █ █ █ █ █ █ █ ░ ░ | ░ ░ █ █ █ █ █ █ ATTENTION MASK (sum of causal and sliding window masks): █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ Another example with seqlen_k = 5, seqlen_q = 3 and sliding_window = 2: CAUSAL MASK: █ █ █ ░ ░ █ █ █ █ ░ █ █ █ █ █ SLIDING WINDOW MASK: ┌──────────────────────── seqlen_k - seqlen_q - sliding_window = 5 - 3 - 2 = 0 offset to the left <┴> | ░ █ █ █ █ | ░ ░ █ █ █ | ░ ░ ░ █ █ ATTENTION MASK (sum of causal and sliding window masks): ░ █ █ ░ ░ ░ ░ █ █ ░ ░ ░ ░ █ █ r .dtypedevice)diagonalN) torchfinforminrangelenslicefullshapertriutril)rrrr min_valueiseqlen_qseqlen_kcausal_diagonal query_range key_range minus_infmaskedsliding_diagonals rbuild_attention_maskr2!sSt N00155I 3+,q0 1='A.1Ea1HH'A.1Ea1HH h 8q=&1A5OO035I!a%5PQ .q13GA3NO JJ 3 Y6 7 = =  &&!((  I@ A '(2^C  ejj5EF FF6<sK23-=rceZdZUdZej ed<ej eeej fzdzed<ej ed<ej ed<ej eeej fzed<e ed<e eee fzed <e ej ed <e ej ed <ej ed <e ed <dZ e ed<deeeffdZy)PagedAttentionArgsaDataclass containing the keyword arguments for a forward pass using paged attention. Attributes: input_ids: Input token IDs tensor of shape `(1, total_query_tokens)`. attention_mask: Attention mask tensor or dictionary mapping layer types to masks. Can be `None` if the attention implementation doesn't require explicit masks. position_ids: Position IDs tensor of shape `(1, total_query_tokens)`. cu_seq_lens_q: Cumulative sequence lengths for queries, used for variable-length batching. cu_seq_lens_k: Cumulative sequence lengths for keys/values. Can be a tensor or dictionary mapping layer types (e.g., "full_attention", "sliding_attention") to tensors for hybrid models. max_seqlen_q: Maximum query sequence length in the batch. max_seqlen_k: Maximum key/value sequence length. Can be an int or dictionary for hybrid models. write_index: List of tensors indicating where to write new KV states in the cache, one per attention group. read_index: List of tensors indicating which cache positions to read from, one per attention group. logits_indices: Tensor indicating which positions in the output should be used for next-token prediction. cache: The [`PagedAttentionCache`] instance managing the KV cache. use_cache: Whether to use caching (always `False` in continuous batching as the cache is managed externally). input_idsNr position_ids cu_seq_lens_q cu_seq_lens_k max_seqlen_q max_seqlen_k write_index read_indexlogits_indicescacheF use_cacherc |j|j|j|j|j|j |j |j|j|j|j|jd S)N r5rr6r7r8r9r:r;r<r=r>r?rA)selfs rasdictzPagedAttentionArgs.asdictsp"11 --!//!// -- --++//"11ZZ  r)__name__ __module__ __qualname____doc__rTensor__annotations__dictstrintlistr r?boolrrCrrr4r4us&||LL4U\\(9#::TAA,,<<<<$sELL'8"999S#X&&ell##U\\""LL It S#X rr4c eZdZdZdededejdejddf dZ e d dd Z e ejdd e ddfd Ze deeddfdZddededeeeffdZy)ContinuousBatchingIOsa?Manages input/output tensors for continuous batching generation. This class handles the allocation and management of static tensors used during generation steps in continuous batching mode. Allocation is done once at init time. The class is responsible for: - Setting up static tensor storage for all generation inputs/outputs - Preparing batch tensors from a list of request states before each forward pass - Building model keyword arguments with optional padding for CUDA graphs/torch.compile - Resetting tensors between batches while minimizing memory operations It keeps track of the requests in the current batch as well as the actual number of tokens (Q and KV), sequences in the batch and sizes of indices. This is useful when using padded inputs, for CUDA graphs and/or torch.compile. r>rr model_dtyperNcR||_||_||_||_t |dddn |j |_g|_d|_d|_d|_ t|jDcgc]}dc}|_ |j|jdycc}w)aFInitialize the continuous batching I/O manager. Args: cache: The [`PagedAttentionCache`] instance managing the KV cache. config: The model's pretrained configuration. device: The device to allocate tensors on. model_dtype: The data type for model computations. rNr rrrT) full_reset)r>rrrRgetattrrrequests_in_batchactual_query_lengthactual_key_lengthactual_batch_sizer! num_groupsactual_index_sizessetup_static_tensorsreset_static_tensors)rBr>rrrR_s r__init__zContinuousBatchingIOs.__init__s   &#*63CT#J#RaX^XmXm57#$ !"!"389I9I3J"Ka6"K !!# !!T!2#Ls0 B$T) standalonec|jj|jjz}tjd|jj ftj |j|_tjd|jj ftj |j|_ tj|jj dzftj |j|_ d|_ tj|jj ftj |j|_ tj|jj ftj |j|_ i|_|jjrUtj|jj dzftj |j|jd<|jj rUtj|jj dzftj |j|jd<t"j%|jj'd|_t+|j,ri|_|jj'D]j}tjdd|jj ||jj zf|j0|j|j.|<lnd|_t3|jj4Dcgc]G}tj|jj ftj |jIc}|_t3|jj4Dcgc]I}tj||jj ztj |jKc}|_ycc}wcc}w)aAllocates static tensors for generation inputs and outputs. This is called only once at init time, to avoid repeated allocations and enable CUDA graphs. All tensors are allocated with maximum possible sizes. The allocated tensors are: - `input_ids` and `position_ids`: Query token information - `cumulative_seqlens_q` and `cumulative_seqlens_k`: Sequence length tracking for FlashAttention-style batching - `attention_mask`: Optional attention masks (only for eager/SDPA implementations) - `write_index` and `read_index` storage: Cache indexing tensors for each attention group - `output_ids`: Storage for generated token IDs r rrfull_attentionsliding_attention)sizerrN)r> num_blocks block_sizeremptymax_batch_tokensint32rr5r6rr9r= output_idsrnum_full_attention_groupsnum_sliding_attention_groupsrJfromkeyskeysr:rrrrRr!r[write_index_storageread_index_storage)rB num_pages layer_typer_s rr]z*ContinuousBatchingIOs.setup_static_tensorssJJ))DJJ,A,AA a)D)D%EU[[aealalm!KKDJJ,G,G(HPUP[P[dhdodop$)KK ZZ ( (1 , .ekk$++% !#kk4::+F+F*HPUP[P[dhdodop++tzz'B'B&DEKK`d`k`kl>@! :: / /:?++,,q02%++dkk;D % %&6 7 :: 2 2=B[[,,q02%++dkk>D % %&9 :!MM$*C*C*H*H*JAN t{{ +"$D "77<<>  27++Q ; ;YIdId=de**;;3##J/ #'D  4::001$  KK446ekkRVR]R] ^$  4::001#  KKTZZ%@%@@]a]h]h i#  $ # sA Q=AQrUcx|r|jdjdn |j}|r|jdjdn |j}|r|jdjdn |j }|j ddd|fj|jddd|fj|jd|dzjd|_ |jd|jd|jd|jd|jD]}|j|d|dzjd|j|<|j B|j |ddddd|d|fjt#j$|j&j(t+|j,j.D]G}|j|d|jd|j|d||zjdIy)aAReset static tensors for the next batch. For efficiency, this only resets the portions of tensors that were actually used in the previous batch, using the attributes actual_query_length, actual_key_length, and actual_batch_size. If a (full_reset) is requested, the entire tensor storage is reset. rNr )rprerXrqrYrZr5zero_r6rr9r=fill_rkrr:rrrrRr r!r>r[)rBrUq_lenk_lenb_sizersr)s rr^z*ContinuousBatchingIOs.reset_static_tensors s 9C((+004H`H`7A''*//3tG]G]8B))!,11!4H^H^ q&5&y!'') !VeV)$**, !!,FQJ/557 FU#))"- %%b)33 oJ  % %j 1,FQJ ? E E G,-D  j )"".##J/1fuffuf0DEKKEKKX\XhXhLiLmLmn  otzz,,- BA  $ $Q ' / 5 5b 9  # #A & 7 = =b A BrrWc$ ||_|js td|jd|_d|_d|_g}g}dg}g}|j jDcic]}|dg}}t|jjDcgc]}g} }t|jjDcgc]}g} }|jD]} | j} t| j} |jj| | }|xj| z c_|xjt|j!z c_|xj dz c_| xj| z c_ |j#| j|j#t| | | z|j%|d| zt|j&| |_|j)D]H\}}||j%||d|zt|j*|||j*|<J|jj-| j.| | | | | j0r|j%|ddz | j2j%t4t7t8j:t8j<|j>}|||j@dddt|f<|||jBdddt|f<|||jDdt||||jFdt||d|_$|j)D]`\}}|||j |dt||jJ4tM|jJ||||dk(r |jNndbg|_(g|_)tUtW| | D]k\}}}|||jX|dt||||jZ|dt|t|t|f|j\|<mycc}wcc}wcc}w) aPrepare tensors and metadata for the next model forward pass, using the given requests as data. This method: 1. Resets the static tensors from the previous batch 2. Iterates through requests to accumulate input_ids, position_ids, and sequence lengths 3. Extends read/write indices for cache management 4. Builds attention masks if needed (for eager/SDPA implementations) 5. Converts accumulated lists to tensors and copies them to static storage This method also modifies the `position_offset` attribute of each request to track progress and adds a temporary token at the end of the requests for which there will a new token. zNo requests in batchrr rurNrd)rrrr)/rW ValueErrorr^rXrYrZrror!r>r[position_offsetr"tokens_to_process get_seqlens_kmaxvaluesextendappendr9itemsr:extend_read_and_write_indices request_idremaining_prefill_tokensgenerated_tokensr rrtensorrjrr5r6rr=total_seqlen_qrr2rr<r;ziprrqrpr\)rBrWr5r6rr=rsrr_r<r;state past_length query_length seqlens_klayer_type_seqlen_k to_tensorlayer_type_seqlens_kr)group_read_indicesgroup_write_indicess rprepare_batch_tensorsz+ContinuousBatchingIOs.prepare_batch_tensors+sj"3%%34 4 !!##$ !"!"  !sBFB[B[B`B`BbcJ QCcc"' (=(=">?Qb? ?#()>)>#?@ar@ @++! &BC&&--l;C! 5>~5N1c.122261E0J0J0L  ,J,QZ[oQpD % %j 12MC8L4M N"".$#'#6#6z#B)=)=:DH[:[4#6#6ab  :=egzS^:_ ] 6A!#6DMN`DaD # #A &'@-?)@ AFOPcFdD $ $Q '(B#.A*B C*-.@*A3GZC[)\D # #A & ]C d?@s) R R R  padded_q_sizepadded_kv_cache_sizec|dkDxr|dkD}|r|n |j}|r|n |j}||z}t|jddd|f|jddd|f|j d|dz|j |jd|iiigg|jd }|rUt|j ||jz |_|j |_||j |jdzdt|jD]j\}\} } |r|n| } |r|n| } |jj|j|d| |j j|j"|d| lt%|j&j)} t+| dkDri|_i|_i|_|j&j3D]r\} } | d|dz|j.| <|j,| |j,| <|j0D|r|n| |}|j0| dd|d|f|j0| <tns| d} |j&| d|dz|_|j,| |_|j04|r|n|j&| |}|j0| dd|d|f|_|j0d|_|j5S)a6Get model keyword arguments for the current batch, eventually padding the query dimension to (padded_q_size) and the keys/values dimension to (padded_kv_cache_size). The padding is only useful if we want static shapes, like when using cuda graphs AND only activated if both Q and KV are padded.rNr F) r5r6r7r9r=r8r:rr<r;r>r?.)rXrZr4r5r6rr9r=r>rr enumerater\r<rrqr;rprMrror"r:r8rrrC)rBrr use_paddingryr{padded_kv_sizekwargsr)read_index_sizewrite_index_size layer_typesrsrrzs rget_model_kwargsz&ContinuousBatchingIOs.get_model_kwargss5 $a'D,@1,D !, $2J2J"-43I3I&)==$nnQY/**1fuf9533LfqjA**..v6**  $  #D$5$5ut?R?R7R SD "&"3"3F FKD % %d&<&&>q&ABSCS&T U  V44499;< { a 24F <>F =?F !)-)B)B)H)H)J m% I3<\vz3J$$Z0262C2CJ2O##J/&&2.9Ny?PE8<8K8KJ8WX[]c^c]cekfkekXk8lF))*5  m%QJ#'#<#r requestsr r rNrrHrMrLr2r4rQrOrrrs" =#&0H 0HTH Q=LLQ=s)Q=s)Q= Q=  Q=h / /  / dhhr