WiddlZddlZddlmZddlmZddlmZddlm Z ddl m Z ddl Z ddl m Z ddlmZdd lmZd d lmZd d lmZmZd d lmZd dlmZd dlmZmZmZddlmZddl m!Z!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)m*Z*m+Z+ dZ,dZ-de.de.de.de.fdZ/Gdde j`Z1eGddZ2eGd d!Z3Gd"d#Z4y)$N)abstractmethod) Generator)contextmanager)ceil) perf_counter)nn)tqdm)logging_redirect_tqdm)PretrainedConfig) CompileConfigGenerationConfig)LogitsProcessorList)logging)ContinuousBatchProcessorMetrics attach_tracertraced)PagedAttentionCache)ContinuousBatchingIOsattn_mask_is_needed)GenerationOutput RequestState RequestStatuslogger)SCHEDULER_MAPPING FIFOScheduler Schedulersize max_value nb_intervalsreturncb||z}|dk(r|S|dkDrt||z |zn|}t||S)zRReturn the smallest multiple of (max_value) // (nb_intervals) greater than (size).r)rmin)r!r"r# interval_sizepaddeds /home/sietch6/trending-topics-pipeline/venv/lib/python3.12/site-packages/transformers/generation/continuous_batching/continuous_api.pypad_by_intervalsr*>sC-M;?!8T$& '- 7F vy !!ceZdZUeed<ej ed<ejed<ede ddfdZ ede de fd Z y) ProtoPretrainedModelconfigdtypedeviceattn_implementationr$NcyN)selfr1s r)set_attn_implementationz,ProtoPretrainedModel.set_attn_implementationM r+generation_configcyr3r4)r5r8s r)_get_logits_processorz*ProtoPretrainedModel._get_logits_processorQr7r+)__name__ __module__ __qualname__r __annotations__torchr/r0rstrr6rrr:r4r+r)r-r-Hs`  ;; LL 3 4   7G L_  r+r-ceZdZdedededejdejdejde jde jd e d ed ed ed eddfdZdefdZed,dZedededdfdZd,dZedefdZededdfdZed,dZedefdZedZededdfdZee j>de jBde"deddfd Z#ede jBd!e$de"deddf d"Z%ed#$de jBd!e$de jLfd%Z'ed&$d!e$d'e jLde"de jLfd(Z(ed)$d*e jLd!e$deddfd+Z)y)-ContinuousBatchProcessorcacher.r8 input_queue output_queue stop_event model_device model_dtype schedulermanual_evictionuse_cuda_graphq_padding_intervalskv_padding_intervalsr$Nc||_||_||_||_||_||_||_||_| |_| |_ t|dddn |j|_ g|_ | |_ | |_| rind|_t|dd|_d|_| xs'|j duxr|j j$ |_|j(|_t+|j(|_t/|||||_y)aInitialize the continuous batch processor. Args: cache: A [`PagedAttentionCache`] object config: The model configuration generation_config: The generation configuration input_queue: Queue for incoming requests output_queue: Queue for outgoing results stop_event: Event to signal processing should stop model_device: Device for model inputs/outputs model_dtype: Data type for model inputs/outputs scheduler: The [`Scheduler`] to use manual_eviction: Whether to manually evict blocks from the cache use_cuda_graph: Whether to use cuda graphs or not during CB. Check the docstring at the top of the file for more details. sliding_windowNrcompile_configF)rCr.r8rDrErFrGrHrIrJgetattrrOrequests_in_batchrLrM_graphsrP'_forward_process_and_sample_is_compileddynamic _pad_inputsmax_batch_tokensrmetricsrinputs_and_outputs)r5rCr.r8rDrErFrGrHrIrJrKrLrMs r)__init__z!ContinuousBatchProcessor.__init__Ys@  !2&($(&".$+63CT#J#RaX^XmXm57#6 $8!Q_2ei 4;>QRVR`R`RqRqQrrs u%%668AAC D r+c|jjsU |jj}|8|jj ||jjsTyy#t j $rYyt$rQ}tjd|dtjd}||j||Yd}~d}~wwxYw)z?Pull new requests from the input queue and add to waiting list.NzError processing new request: Texc_infostate) rDempty get_nowaitrIadd_waiting_requestqueueEmpty Exceptionrerrorlocalsget_handle_request_error)r5rees r)_get_new_requestsz*ContinuousBatchProcessor._get_new_requestss""((* 9((335=2259 ""((*;;  9 =aSADQ&,hll7&;$..q%8  9s#A1A11CCACCrlrectj|_t||_t |j tr+|jj|j |_ ng|_ |jj|j|j |jj|jy)z(Handle general request processing error.N)rFAILEDstatusr@rl isinstance request_idrI!get_active_request_static_outputsgenerated_tokensrXrecord_request_completion created_timerEputto_generation_output)r5rlres r)roz.ContinuousBatchProcessor._handle_request_errors%++ %j  e&& ,%)^^%U%UV[VfVf%gE "%'E " ..u/A/A5CSCST e88:;r+c .|jjr(|jjj\}}n9t t |jjj \}}tjd|dt|jdt|jd|j}|jj|d|jj|d|j_y)aSoft resets one active request by removing it from active requests and re-adding it to the waiting queue. The generated tokens are kept as part of the new request's initial prompt. When `block_new_requests` is False, the oldest request is offloaded; when True, the newest request is offloaded. This method also sets `block_new_requests` to True to prevent infinite loops of offloading and re-scheduling requests. zSoft resetting request z with z initial tokens and z generated tokensTevict_from_cacheN)rIblock_new_requestsr]popitemnextiteritemsrinfoleninitial_tokensrx!create_equivalent_initial_requestfinish_requestrh)r5rvre new_states r)soft_reset_one_requestz/ContinuousBatchProcessor.soft_reset_one_requests >> , , $ > > F F H J $T$..*H*H*N*N*P%Q R J %j\E ;;=  %%j4%H **95-1)r+c|j|jj|jjsy|jj t |jjt |jj|jj|j|jj}|>t |jjdkDr|jytd||_|jsy|jj!|j|j"j%||jj'|jt)j*t,j.r|j"j0}|j"j2t5fdD}t)j6dt |jdt |jjdt |jjd|d d |d |jj9 y ) zPrepare tensors and metadata for the next model forward pass. Returns True if there are requests to process, False otherwise.Frz=No requests can be scheduled and no request can be offloaded.c3.K|] }|dyw)Nr4).0 layer_typecumulative_seqlens_ks r) z>ContinuousBatchProcessor.prepare_next_batch..saj)*5b9az Scheduled: z , Waiting: z , Active: z . cum Q: rz . cum KV: z, free blocks: T)rqrIclear_cancelled_requestshas_pending_requestsrXrecord_queue_metricsrr]r^schedule_batchrWrC num_pagesr RuntimeErrorrRrecord_batch_metricsrYprepare_batch_tensorsrecord_kv_cache_memory_metricsr isEnabledForrDEBUGcumulative_seqlens_qrmaxdebugget_num_free_blocks)r5rRrckrs @r)prepare_next_batchz+ContinuousBatchProcessor.prepare_next_batchs   //1~~224 ))#dnn.L.L*MsSWSaSaSrSrOst!NN99$:O:OQUQ[Q[QeQef  $4>>112Q6++-"#bcc!2%% ))$*@*@A 556GH 33DJJ?   w}} -#'#:#:#O#O #'#:#:#O#O aL`aaB LLc$"8"89:+c$..JiJiFjEklt~~==>?yI]^`IaHbc$odjj.L.L.N-OQ  r+c|js|jtjk(r*|jj |j yy)zCSend output to the queue based on streaming mode and request state.N) streamingrtrFINISHEDrEr{r|)r5res r)_maybe_send_outputz+ContinuousBatchProcessor._maybe_send_outputs? ??ellm.D.DD    ! !%"<"<"> ?Er+cN|jjdt|jj }d}|jD]}t|j dk(r|j dk(rE|jj|j|jtj|_ ||}|g|_|dz }|j|}|j j#||rs|jj%|j|j|j&j)|j|j* d|j&_|j/|;|jtj0k(r2|j j#|tj2|_ t5d|jd|jgg}}|j&j6r|j&j6j9}|j:} d|_t=| D cgc]} |jd| } } | D]*} |j?| |j&j@| <,|j jC|j| \} }|jE| |jE||j&j6r|r|j jG||yycc} w) z0Update request states based on generated tokens.Nrrr~FzRequest z is in an unexpected state: z__child#)$rY output_idsrrRtolistremaining_prefill_tokens generated_lenrXrecord_ttft_metricrzrvrDECODINGrttokens_to_processupdate_and_check_completionrC!mark_shareable_blocks_as_completeryrIrrJrrPREFILLING_SPLITSPLIT_PENDING_REMAINDER ValueError_requests_to_forkpop num_childrenrangeforkr] fork_requestextend copy_cache)r5 new_tokenscurrent_logits_indexretoken is_finished copy_sourcecopy_destination state_to_forkrinew_request_idsnew_request_idcopy_srccopy_dsts r) update_batchz%ContinuousBatchProcessor.update_batchs,,778U#d>T>T:UV]]_  ++ jE5112a7&&(A-LL33E4F4FHXHXY#0#9#9EL"#78+0''$)$$??F  < d"&!8!89Q9QSb!c Hh   x (  # #H -nn..  JJ ! !+/? @ fsL"c6|jjS)z2Check if there are any active or waiting requests.)rIrras r)rz-ContinuousBatchProcessor.has_pending_requestsDs~~2244r+c|j}|D]9}|j|||jj|j;y)z&Handle errors during batch processing.N)rRrorIrrv)r5rl failed_reqsreqs r)handle_batch_errorz+ContinuousBatchProcessor.handle_batch_errorIsD,,  :C  & &uc 2 NN ) )#.. 9 :r+ct|jjj}|D]9}|j |||jj |j ;t|jjjD]9}|jjj|}|j ||;|jjjy)zFail all active requests with the given error. Args: error: The error to report in the failure message N) listrIr]valuesrorrvr^keysrwaiting_requests_orderclear)r5rlrequestsrereq_ids r)fail_all_requestsz*ContinuousBatchProcessor.fail_all_requestsQs66==?@ >::??AB 5FNN3377?E  & &ue 4 5 --335r+modellogit_processor do_samplecn |j|jstj|j|jj |jj |jj|jj|jj|_d|_|jr|jj}|jj t||j|j }t# fdt%|j&j(D}t||j&j*|j,}nd\}}|jj/||}|j0|j ||||y|j0j3||f} | | j5yt7j8d||ftj:j=|j>} | jAtj:jCtj:jE| 5|j ||||dddtj:jCjA| tj:jG} tj:jI| | 5|j ||||ddd| |j0||f<y#1swYxYw#1swY'xYw) z!Perform a single generation step.N) fullgraphmoderUbackendoptionsTc3.K|] }|dyw)rNr4)rractual_index_sizess r)rz.|s%e1&8&;A&>%er)rrz8Creating graph for (padded_q, padded_read_index_size) = )r0)stream)%rPrTr?compile_forward_process_and_samplerrrUrrrVrYactual_query_lengthrr*rWrLrrrC num_groupsrrMr_rSrnreplayrrcudaStreamr0 wait_streamcurrent_streamr CUDAGraphgraph) r5rrrrpadded_qmax_read_index_sizepadded_read_index_size batch_datarrrs @r)_generation_stepz)ContinuousBatchProcessor._generation_stepfs    *43_3_/4}}00--77((--++33++33++33 0D ,<@D 8   "&"9"9"M"M !%!8!8!K!K '(;T=R=RTXTlTlmH"%%edjjNcNcHd%e"e %5#TZZ%9%94;T;T& "04 ,H,,,==hH^_  <<   , ,UJQZ [   (,B!CD   LLN  O85K*L)PQR""%,,"75::4467 ZZ  v & \  , ,UJQZ [ \ !!#//7 $$& ZZ  eF  3 \  , ,UJQZ [ \;@ h 678 \ \  \ \s'L0L+L(+L4rct|j||}|j|||}|j|||y)zThis function performs the forward pass, logits processing, and sampling; which are broken down into smaller function to be easier to trace with OpenTelemetry.N)_model_forward_process_logit_sample)r5rrrrlogitsprobss r)rz4ContinuousBatchProcessor._forward_process_and_samples: $$UJ7##JH UJ 2r+ model_forward span_namec&|di|jS)Nr4)r)r5rrs r)rz'ContinuousBatchProcessor._model_forwards"z")))r+logit_processingrct|dr|j|d|d|j\}}}|j||z|}|dj||z}|||} | j|||S)Nset_continuous_batching_contextlogits_indices cu_seq_lens_q input_ids)hasattrrshapeview) r5rrr batch_sizeseq_len vocab_size logits_2d input_ids_2dprocessed_logits_2ds r)rz'ContinuousBatchProcessor._process_logits ?$E F  ; ;JGWb">b!>b, >b [[ >b kk >bOO>bll>b[[>b>b>b>b!>b">b >b@ #  9 9"  <9 <\ |_ |j,r tCdy)aSInitialize the continuous batching manager. Args: model: The language model for generation generation_config: Configuration for generation parameters max_queue_size: Maximum size of the request queue (0 = unlimited) num_q_padding_intervals: (optional) Number of intervals used to pad the query dimension num_kv_padding_intervals: (optional) Number of intervals used to pad the keys/values dimension allow_block_sharing: (optional) Whether to allow block sharing if the model has some full attention layers zpaged|)maxsizeNrlog_prob_generationFrTnum_return_sequencesrrKrP)rKr'r(rPz(log_prob_generation is not supported yet)"r._attn_implementationr6evalrrJ_allow_block_sharing_use_prefix_sharingrirrDrErrrFbatch_processor_generation_thread_request_counterLock _request_lockr8rQr,rr:rr-_decide_use_cuda_graphsrKNUM_Q_PADDING_INTERVALSrLNUM_KV_PADDING_INTERVALSrMNotImplementedError) r5rr8rJr&r'r(r)r-s r)rZz"ContinuousBatchingManager.__init__s* 5<<<< <  ) )F5<<3T3T2U*V WZZ\ .$7!#6  ;;~>!KKM#//+@D"& !&^^-8I7PE33Vg!2#*+<>SUZ#[  !2KF48JJ4T4TUf4g&'8:PRVWY#:_v (@1(D $Jb !  # #%&PQ Q $r+rKrPctjjs8|r5tjdtjjdy||S|dkDs|dkDry||tj j j|j|j}|jdd}|r#tjd|jd| St|jj }tjd |d |jjjd |S) aReturns whether or not to use cuda graphs for continuous batching, depending on the following criteria: - (use_cuda_graph) which is the user choice - (num_q_padding_intervals) or (num_kv_padding_intervals) which is used to pad inputs: if it was specified by the user, it's probable they want to use cuda graphs so inputs need to be padded - (compile_config): if compile is on, turn on cuda graphs unless the compile mode uses its own cudagraphs If none of the above criteria are met, we use a default heuristic based on the attention implementation: we turn on cuda graphs if and only if no attention mask is needed. z7use_cuda_graph is True but torch.cuda.is_available() = z: turning off cuda graphs.FrTztriton.cudagraphsz%Compile config compile_config.mode = z uses cudagraphs, which usually does not work well with continuous batching. We recommend using mode 'default' or 'max-autotune-no-cudagraphs' instead.zINo behavior specified for use_cuda_graph, defaulting to use_cuda_graph = z2 because self.model.config._attn_implementation = zc. If you want to save memory, turn off cuda graphs, but they tend to improve performances by a lot.)r?r is_availablerwarning _inductorlist_mode_optionsrnrrrrr.r.)r5rKr'r(rPrcompile_uses_cudagraphss r)r7z1ContinuousBatchingManager._decide_use_cuda_graphss1 zz&&(!YUZZ=T=T=VQ>QSaSiSijG&-kk2Eu&M #&vv/. .11B1BCCX~FYZ9zz  559:: : r+c|j0|jjrtjdyt j |j |_|jjy)z'Start the background generation thread.Nz"Manager thread is already running.)target)r3is_aliverr=rThread_run_generation_loopstartras r)rFzContinuousBatchingManager.startKs\  " " .43J3J3S3S3U NN? @ "+"2"2$:S:S"T %%'r+cV|jduxr|jjS)z5Check if the background generation thread is running.N)r3rCras r) is_runningz$ContinuousBatchingManager.is_runningUs'&&d2Yt7N7N7W7W7YYr+blocktimeoutc|jtjdnV|jjjr6tj d|jjj |jtjdyt}|jjs/|jjtj d|r|j||d|_y)zSignal the background thread to stop. Args: block: Whether to wait for the thread to stop timeout: Maximum time to wait for the thread to stop Nz% Batch processor was not initialized.z- Prefix sharing was on. Total prefix length: zManager not started.z'Stopping continuous batching manager...) r2rr=rCuse_prefix_sharingr_total_prefix_lengthr3rrFis_setsetjoin)r5rIrJstop_trigger_times r)stopzContinuousBatchingManager.stopZs    ' NNC D##))<< DTEYEYE_E_EtEtDuv  " " * NN1 2 (N%%' OO   ! KKA B  II' 1#r+rQc|j|jj||jjrtjd|dyt }tj d||z ddd|_yy)zWait for the background thread to finish. Args: timeout: Maximum time to wait for the thread to stop NrJz3Generation thread did not exit after join timeout (z).z*Continuous Batching Manager stopped after z.2fzs.)r3rPrCrr=rr)r5rQrJends r)rPzContinuousBatchingManager.joinws  " " .  # # ( ( ( 9&&//1!TU\T]]_`a"n HO`I`adHeeghi*.' /r+rrvmax_new_tokensrrecord_timestampsc |9|j5d|j}|xjdz c_ddd||jjn|}t |t ||j dz |t |||jj|}|jj|dd|S#1swYxYw)a/Add a new generation request to the queue. Args: input_ids: Input token IDs to use as prompt request_id: Optional custom request ID (auto-generated if None) **kwargs: Additional generation parameters Returns: str: The request ID Nreq_r)rvrrrWrrV eos_token_idrT rIrJ) r6r4r8rVrrr-rZrDr{)r5rrvrVrrWres r) add_requestz%ContinuousBatchingManager.add_requests$  ## +#D$9$9#:; %%*% +CQBX//>>^l! ?22Q6/"9o)//<<   U$;) + +s %B<<Cinputsc|j5t|j|jt|zDcgc]}d| }}|xjt|z c_dddt t |}|j rt|dd}|D]\}} |j| ||||ycc}w#1swY^xYw)NrYc |dS)Nrr4)xs r)z8ContinuousBatchingManager.add_requests..s !A$r+T)keyreverse) r6rr4rrzipr1sortedr]) r5r^rVrrWr request_idsids_and_inputsrvrs r) add_requestsz&ContinuousBatchingManager.add_requestss   1/4T5J5JDLaLadghndoLo/pq!T!:qKq  ! !S[ 0 ! 1c+v67  # ##NPTUN%3 b !J   Y NIO` a br 1 1s/C C CCCch|j&|jjj|yy)zkCancel a request by its ID. Args: request_id: The ID of the request to cancel N)r2rIset_request_cancellationr5rvs r)cancel_requestz(ContinuousBatchingManager.cancel_requests/    +  * * C CJ O ,r+c|j|jjry |jjd|}|+|j|k7r|jj |y|S#t j$rYywxYw)zRetrieve one result from the output queue. Args: timeout: Maximum time to wait for a result Returns: Optional[GenerationOutput]: The result data or None if timeout NTr\)r3rErfrnrvr{rirj)r5rvrJresults r) get_resultz$ContinuousBatchingManager.get_results  " " *t/@/@/F/F/H &&**w*GF%&*;*;z*I!!%%f-M{{  sA A53A55B  B c#K|j\|jjrA|jd}|||j|jjr?yyyyw)z.Iterate over results as they become available.N皙?rT)r3rCrp)r5ros r)__iter__z"ContinuousBatchingManager.__iter__sf%%1d6M6M6V6V6X__S_1F! %%1d6M6M6V6V6X16X1s A%A,(A,c#PKd}|j|jjry|sv|j|d}|||j%|jjj |}|j|jjr|ssyyyyyyw)zMIterate over results matching a specific request id as they become available.FNrr)rvrJ)r3rCrpr2rIrequest_is_cancelled)r5rvrequest_cancelledros r)request_id_iterz)ContinuousBatchingManager.request_id_iters!%%1d6M6M6V6V6Xar__ C_HF! ##/$($8$8$B$B$W$WXb$c! %%1d6M6M6V6V6Xar6X1ar6X1s BB& B&c|j td|jj|j|j|j y)z=Perform a single generation step. This is mostly cuda graphedNzNTried to perform a generation step before the batch processor was initialized.)r2rrrrrras r)rz*ContinuousBatchingManager._generation_stepsE    'op p --djj$:N:NPTP^P^_r+cd} t}t|jj|j|jj |jj t|jdd|j}|j|_ tjdt|z dd}t|jdrLtj|jj d}|&tj"d|dt$}nt$}t}t'||jj|j|j(|j*|j,|jj |jj |||j.|j.|j0|j2|j4 }||_d |_tjd t|z d|j,j;r|j=rR|j?||xj8d z c_|j,j;sA|j=rRtjFdy#t@$r6}tjBd |d|jE||Yd}~Pd}~wwxYw#tjFdwxYw)z6Main processing loop running in the background thread.N_tp_size)tp_sizer)zPagedAttentionCache created in z secondsrIz Scheduler 'z ' not found. Defaulting to FIFO.) rCr.r8rDrErFrGrHrIrJrKrLrMrzbatch_processor created in rzError in generation loop: TrczGeneration loop finished.)$rrrr.r8r0r/rQr0rLr1rrrrrnrIr=rrBrDrErFrJrKrLrMr2 current_batchrNr_inner_generation_looprkrl_handle_critical_errorr)r5r2t0paged_attention_cacherIt1rps r)rEz.ContinuousBatchingManager._run_generation_loopsd;?2 5B$7 !!&& !!    J=$($=$= % !(='O'OD $ LL:<>B;N:OxX YIt--{;-11$2H2H2R2RTXY $NN[ ;[#\] -I* B6+zz(("&"8"8 ,,!..??!ZZ.. JJ,,#$94;O;OP $ 4 4#22$($<$<%)%>%>O$3D !"D  LL6|~7J6K8T U--/O4X4X4Z++O<""a'"--/O4X4X4Z KK3 4  < LL5aS9D I  ' '? ; ; < KK3 4s0I:J&?J&& K%/,K K( K%%K((K?generation_looprr2ctjjrtjj|j sy|j |j yr3)r?rr< synchronizerrr)r5r2s r)r}z0ContinuousBatchingManager._inner_generation_loop1sK :: " " $ JJ " " $113  $$&r+rlc|jj |jj}||j ||/#t j $rYnwxYw||j|yy)z:Handle critical errors that terminate the generation loop.N)rFrOrDrgrorirjr)r5rlr2req_datas r)r~z0ContinuousBatchingManager._handle_critical_error<s~  ++668".#99%J{{     &  - -e 4 's0A A"!A"c|js td|j&|jjj |yy)zSEvict a request from the cache. It is assumed that the request is already finished.z0Manual eviction is not enabled for this manager.N)rJrr2rIrrls r)evict_request_from_cachez2ContinuousBatchingManager.evict_request_from_cacheOsD##QR R    +  * * 9 9* E ,r+)FrrrTr)TNr3)NNFF)NFF)NN)"r;r<r=__doc__r-rrrrZr r7rrFrHfloatrRrPrr@r]rirmrrprsrrwrrErBr}rkr~rr4r+r)r%r%s!&'(()$(AR#AR,AR AR  AR "% AR#&AR"AR ARF-t -"%-#& - &, -  -^ ( (ZDZ $$$ $$: /e /edl /d /$"&%)"' '9'$J'd '  '  ' 'X&*"' bT#Ybd b b  b  b&PPPS4ZYilpYp*d#d) +F84 %%L>3   S7J S S4 S$$!&**=M2Nbs%"#$|3$///:F!'!2!2!--/.4GFO*a/N KKN&113"LL)UV!"gh!%|3  S S S0s6{# GA[[4s,F!06!D*- |A3.DEF  G!  S >qcBTRR S+ S S S S S Ssa6H 9H<H>B,G+H3H  H !H ;HH HH HH H  H))NFrrrTTN)NFrrrT)NrrTFT)r;r<r=rrr>rrrrrr%rrrr?inference_moderr"r@rrr4r+r)rrXsI''6: %!""#$( $7+d277 7  7  7"777 , -77>6: %'(()$(* +d2* *  * "% * #& * "*  #* Z U6:'(()$("'!U!T#YU!,d2U!"% U! #& U! " U! U!U! c## $U! U!r+r)5rirabcrcollections.abcr contextlibrmathrtimerr?rr tqdm.contrib.loggingr configuration_utilsr generation.configuration_utilsr rgeneration.logits_processr utils.loggingr utils.metricsrrrrCr input_ouputsrrrrrrrrIrrrr8r9rr*r!r-rBr%rr4r+r)rs %% 63M<$SS&DKKBB&"3"3"c"c"  299  uGuGuGr CFCFCFL g!g!r+