i i#dZddlZddlZddlZddlmZddlmZmZm Z m Z ddl m Z m Z ddl mZmZddlmZmZmZmZeej,j,d z d z ZGd d eZy) aReddit subreddit scraper - PRIMARY collector for trending questions. Scrapes "explain it to me" subreddits where people ask about trending topics. Uses direct HTTP calls to old.reddit.com JSON endpoints - NO PRAW, NO auth. Supports incremental collection via reddit_seen table. N)Path)ListOptionalSetTuple)datetimetimezone) BaseCollector TrendItem)REDDIT_SUBREDDITSREDDIT_POSTS_PER_SUBREDDIT_REQUEST_DELAYREDDIT_USER_AGENToutputztrends_history.dbceZdZdZddeeffd ZdeefdZdee eeffdZ dded e de e fd Z dd ed ed e deede eeee eefff dZddedeefdZxZS)RedditScraperCollectorzTScrapes Reddit subreddits for trending questions/topics with incremental collection. subredditsct|ddti|_gd|_ddid|_|r||_yt |_y)a4 Initialize the Reddit scraper. Args: subreddits: Optional list of subreddit names to scrape. If None, uses the default REDDIT_SUBREDDITS from config.py. When a channel is specified, pass the channel's configured subreddits. redditz User-Agent)<xi,rnewseen by_subredditN)super__init__rheaders backoff_timesstatsr r)selfr __class__s C/home/sietch6/trending-topics-pipeline/collectors/reddit_scraper.pyr zRedditScraperCollector.__init__sE "$&78 42> (2*8Ireturncb tjt}|j}|j d|j Dchc]}|d }}|j |Scc}w#t$r2}|jjd|tcYd}~Sd}~wwxYw)z0Load all previously seen post IDs from database.zSELECT post_id FROM reddit_seenrzError loading seen posts: N) sqlite3connectDB_PATHcursorexecutefetchallclose Exceptionerrorsappendset)r$connr-rowres r&_get_seen_post_idsz)RedditScraperCollector._get_seen_post_ids-s ??7+D[[]F NN< =&,oo&78sCF8D8 JJLK9  KK  !;A3? @5L s0A A3 A.A3.A33 B.<'B)#B.)B.postsc |sy tjt}|j}t j t jj}|jd|Dcgc] }|d|d|fc}|j|jycc}w#t$r(}|jjd|Yd}~yd}~wwxYw)zqMark post IDs as seen in database. Args: posts: List of (post_id, subreddit) tuples NzVINSERT OR IGNORE INTO reddit_seen (post_id, subreddit, first_seen_at) VALUES (?, ?, ?)rr zError marking posts seen: )r*r+r,r-rnowr utc isoformat executemanycommitr0r1r2r3)r$r9r5r-r;pr7s r&_mark_posts_seenz'RedditScraperCollector._mark_posts_seen:s   A??7+D[[]F,,x||,668C   h,12q!A$!c"2  KKM JJL3 A KK  !;A3? @ @ As*A*B./B) 'B.)B.. C7CCurlretriesc Vt|D]} tj||jd}|jdk(rT|j t |t|j dz }td|dtj||j|jcSy #tjj$rR}|jj!d|dt#|||dz krtjd Yd }~d }~wwxYw) z Fetch JSON from Reddit with exponential backoff on 429s. Args: url: The Reddit JSON endpoint retries: Number of retry attempts Returns: Parsed JSON dict or None on failure )r!timeoutir z Rate limited, waiting zs...z Fetch error z: N)rangerequestsgetr! status_coder"minlenprinttimesleepraise_for_statusjson exceptionsRequestExceptionr2r3str)r$rBrCattemptresponsebackoffr7s r& _fetch_jsonz"RedditScraperCollector._fetch_jsonOsW~ "G "#<< LL ''3."00Wc$BTBT>UXY>Y1Z[G6witDEJJw'))+}}&! ". &&77 " ""\#bQ#ABWq[(JJqM "sBB9B99D(AD##D( subredditsortparamsseen_idsc2| t}d|d|d}dti}|r|j|djd|j D}|d|}|j |} | sggfSg} g} | j d ij d g} t| d D]i\} }|j d i}|j d d}|j dd}|r|sB||vr|jdxxd z cc<^|j dr|j ddddnd}| jt|d| dtjtjjd|j dd|||j dd|j dd|j d|||j d|j dd | j||f|jdxxd z cc<l| | fS) a Scrape posts from a subreddit, filtering out already-seen posts. Args: subreddit: Subreddit name (without r/) sort: Sort type (hot, top, rising) params: Additional URL params (e.g., t=month for top) seen_ids: Set of post IDs to skip (already collected) Returns: Tuple of (List of TrendItem objects, List of (post_id, subreddit) for new posts) Nzhttps://old.reddit.com/r//z.jsonlimit&c30K|]\}}|d|yw)=N).0kvs r& z;RedditScraperCollector._scrape_subreddit..sGDAqs!A3ZGs?datachildrenr )startidtitlerselftextirglobalzhttps://reddit.com permalinkscorer num_comments created_utc upvote_ratioauthor) rZpost_idrsrtrurpr[rvrw) topic_namesourcerankregion timestamprBmetadatar)r4rupdatejoinitemsrYrJ enumerater#r3r rr;r r<r=)r$rZr[r\r]base_url url_params param_strrBrjr new_post_idsr9r{post post_datarxrorps r&_scrape_subredditz(RedditScraperCollector._scrape_subredditss#  uH.yk4&F34    f %HHGJ4D4D4FGG  !I;'$r6M $((R8#E3$ #JD$,ImmD"-GMM'2.E(" 6"a'"?HmmJ>Wy}}Z4Tc:]_H LL ",,x||4>>@({B)G(HI!*&&]]7A6$-MM.!$D#,==#? ( $-MM.$A'mmH5  &   ) 4 5 JJu  " I$ #Ll""r'r|cnddid|_|j}tdt|dg}g}|jD]}td|ddg}g}|j |d | \}} |j ||j | |jd | Dtjt|j |d d di| \} } |j | |j | |jd| Dtjt|j |dddi| \} } |j | |j | |jd| Dtjtt||jd|<tt|d|j ||j ||j|td|jdd|jdd|S)a^ Collect trending questions from all target subreddits (incremental). Fetches 'hot', 'top/month', and 'rising' for each subreddit. Skips posts already seen in previous runs. Args: region: Ignored for Reddit (always global) Returns: List of TrendItem objects (only NEW posts) rrz Loaded z previously seen post IDsz Scraping r/z... )endhot)r]c3&K|] \}}| ywNrdrerm_s r&rhz1RedditScraperCollector.collect..452qB4toptmonthc3&K|] \}}| ywrrdrs r&rhz1RedditScraperCollector.collect..rrrisingr`2c3&K|] \}}| ywrrdrs r&rhz1RedditScraperCollector.collect..s752qB7rrz new postsz Reddit collection: rz new, rz skipped (already seen)) r#r8rNrMrrextendrrOrPrrA)r$r|r] all_itemsall_new_post_idsrZ sub_items sub_new_ids hot_itemshot_ids top_itemstop_ids rising_items rising_idss r&collectzRedditScraperCollector.collects!2> **, CM?*CDE ! 1I OI;c2 <IK"&!7!7 5S[!7!\ Iw   Y '   w ' OO4G4 4 JJ+ ,"&!7!7 53PW.ck!7!l Iw   Y '   w ' OO4G4 4 JJ+ ,(,'='=8gr]X(>( $L*   \ *   z * OO7J7 7 JJ+ ,47 NDJJ~ &y 1 S^$J/ 0   Y '  # #K 0C! 1H ./ ' 5(9':&FAS@TTklmr'r))NN)rq)__name__ __module__ __qualname____doc__rrUr rr8rrAintrdictrYr rr __classcell__)r%s@r&rrs^ J49 J CH Ad5c?&;A*"s"S"$"JTXH#H#$'H#15H#HKCH# tIU38_ 55 6H#T?c?i?r'r)rr*rOrIpathlibrtypingrrrrrr baser r configr rrr__file__parentr,rrdr'r&rs^ --'* x.   & & 14G Ga]ar'