v_i dZddlmZmZddlmZmZmZddlm Z efde de de fd Z d ee dee fd Zd ee deeee ffd Zd ee dee fdZy)z5Cross-source merging and scoring for trending topics.)ListDict)MAX_ITEMS_PER_SOURCECROSS_SOURCE_BOOSTMIN_SOURCES_FOR_BOOST)deduplicate_topicsrankmax_rankreturncd|dz |z z S)z Convert rank to a 0-1 score (higher = better). Args: rank: Position in source list (1 = best) max_rank: Maximum possible rank Returns: Normalized score between 0 and 1 g?r)r r s ;/home/sietch6/trending-topics-pipeline/aggregator/merger.pynormalize_scorer s 4!8x' ((itemsc6|sgS|D]}t|d|d<t|}|D]>}|d}|jdd}|tk\r|tz|d<d|d<5||d<d|d<@|j d d t |d D] \}}||d < |S) a Merge items from all sources with equal weighting and cross-source boosting. 1. Normalize ranks to 0-1 scores per source 2. Deduplicate with fuzzy matching 3. Apply cross-source boost for topics in 2+ sources 4. Sort by final score Args: items: List of trend item dictionaries from all sources Returns: Merged, deduplicated, and scored list r normalized_score source_countr final_scoreTboostedFc |dS)Nrr)xs rz!merge_and_score..>s q/r)keyreverse)start final_rank)rr getrrsort enumerate)ritemdeduped base_scoreris rmerge_and_scorer&s  A#24<#@  A!'G $,- xx2  0 0",/A"AD "DO",D #DO $ LL/L>WA.4\ Nr all_itemsci}|D]$}|d}||vrg||<||j|&|jDcic]\}}|t|c}}Scc}}w)z Merge items grouped by region. Args: all_items: List of all trend items from all sources and regions Returns: Dictionary mapping region to merged/scored trend list region)appendrr&)r' by_regionr"r)rs rmerge_by_regionr,Gsu(*I'h  " "If &  & 'AJ@Q R}vuFOE* * RR RsAct|S)a Merge all items across all regions into a single list. Useful for seeing overall trending topics regardless of region. Args: all_items: List of all trend items from all sources and regions Returns: Single merged/scored trend list )r&)r's rmerge_all_regionsr.^s 9 %%rN)__doc__typingrrconfigrrr deduplicatorr intfloatrdictr&strr,r.rrrr7s;RR,0D )# ) ) )-4:-$t*-`StDzSd3T ?.CS. &d &T &r