Ei'ZdZddlZddlmZddlmZddlmZmZddlm Z Gdd eZ y) zGoogle News RSS feed collector.N)List)datetime) BaseCollector TrendItem)GOOGLE_NEWS_TOPICSc leZdZdZfdZdedeefdZdedededee fdZ dedee fd Z xZ S) GoogleNewsCollectorzQCollects top headlines from Google News RSS feeds including topic-specific feeds.c$t|dy)N google_news)super__init__)self __class__s @/home/sietch6/trending-topics-pipeline/collectors/google_news.pyrzGoogleNewsCollector.__init__s 'regionreturnc |dk(rd}d}nd}d}g}|jd|d|d|d ftjD]$\}}|jd |d |d|d||f&|S) zk Get all RSS feed URLs for a region. Returns list of (url, category_name) tuples. CAzen-CAzCA:enzen-USzUS:enzhttps://news.google.com/rss?hl=z&gl=z&ceid= Headlinesz#https://news.google.com/rss/topics/z?hl=)appendritems)rrlangceidfeeds topic_nametopic_ids r _get_feedszGoogleNewsCollector._get_feedss T>DDDD  -dV4xvdV L   %7$<$<$>  J LL5hZtD6fXU[\`[ab    rurlcategorycg} tj|}|jr9|js-|jj d|d|j gSt|jdD]\}}d}t|dr|jjdd}|j t|jd||tjj|jd |jd d||d   |S#t $r5} |jj d |dt#| Yd} ~ |Sd} ~ wwxYw)z.Parse a single RSS feed and return TrendItems.zFailed to parse z feed: rstartsourcetitler link published)r)source_publicationr!)rr&rankr timestampr metadatazError parsing N) feedparserparsebozoentrieserrorsrbozo_exception enumeratehasattrr&getrr'rutcnow isoformat Exceptionstr) rr rr!rfeedr+entry source_namees r _parse_feedzGoogleNewsCollector._parse_feed/s; K##C(Dyy ""%5hZwtGZGZF[#\] (Q?  e 5(+"',,"2"27B"?K #(;;,!%"*//"3"="="?!IIf-).;)C2=(0"   0  K KK  zQI J J  KsADB7D E!*EEc^g}t}|j|}|D]n\}}|j|||}|D]Q}|jj j } | |vs0|j | |j|Spt|dD] \} }| |_ |S)aM Collect headlines from all Google News RSS feeds. Fetches main headlines plus topic-specific feeds (Technology, Science, Business, Entertainment, Health) for broader coverage. Args: region: Two-letter region code (US, CA) Returns: List of TrendItem objects rr#) setrr?rlowerstripaddrr4r+) rr all_items seen_titlesrr r!ritem title_keyis rcollectzGoogleNewsCollector.collectTs e '" +MC$$S&(;E + OO11399; K/OOI.$$T*  +  +!!4 GAtDI r) __name__ __module__ __qualname____doc__rr:rtuplerrr?rJ __classcell__)rs@rr r s_[(e<#s#C#3#4 ?#Jcd9orr ) rNr.typingrrbaserrconfigrr rrrUs%%*%h-hr