FEijdZddlZddlZddlmcmZddlmZddlm Z m Z m Z m Z m Z mZmZerddlZddlZddlmZmZmZddlmZmZmZddlmZe d Zd ed ed ed e def dZ!de d ededdfdZ"de#d ed ed ed e de e f dZ$de e ded ed ede e e edff dZ%d edefdZ&edddddfd eded ed ed e dede e e edffdZ'eddddddfd eded ed ed e de d dede e e edffd!Z( d(d"e ed ed ed e d#ed$e)de ee e ffd%Z* d)d"e ed ed ed e d#ed&e de ee e ffd'Z+y)*a/ Google Trends RSS Feed Downloader - Fast, Rich Media Data This module provides fast access to Google Trends RSS feed data, including images, news articles, and headlines. Perfect for: - Real-time monitoring - Qualitative research (news context) - Visual content (images for presentations) - Fast data collection (0.2s vs 10s for CSV) Use Cases: - Journalism: Breaking news validation with sources - Research: Mixed methods (combine with CSV for complete picture) - Trading: Fast alerts with news context - Marketing: Quick trend checks with visual content N)datetime)DictListOptionalUnionLiteral TYPE_CHECKINGcast) COUNTRIES US_STATES DEFAULT_GEO)InvalidParameterError DownloadErrorRateLimitError) get_rss_cache)csvjson dataframedictgeoinclude_imagesinclude_articlesmax_articles_per_trendreturnc|d|d|d|S)z-Generate a cache key from request parameters.:)rrrrs d/home/sietch6/trending-topics-pipeline/venv/lib/python3.12/site-packages/trendspyg/rss_downloader.py_make_cache_keyr %s&U!N#1%5$6a8N7O PP status_codeurlc |dk(s|dk(rtd|d|d||dk(rtd|d||d k\rtd |d |td |d |d|d)z Raise appropriate exception based on HTTP status code. Provides specific, actionable error messages for common HTTP errors. iizRate limit exceeded (HTTP u) Google is temporarily blocking requests. Solutions: • Wait 1-2 minutes before trying again • Reduce request frequency (add delays between calls) • Use caching: results are cached for 5 minutes by default • For batch operations, use max_concurrent=5 or delay=0.5 Geo: z | URL: iz-RSS feed not found (HTTP 404) The geo code 'zt' may not have an RSS feed available. Try a different country code or check if the code is correct. URL attempted: iz!Google Trends server error (HTTP u) Google's servers are having issues. This is temporary. • Wait a few minutes and try again • Check https://trends.google.com to verify it's working URL: z HTTP error z when fetching RSS feed Geo:  URL: zQ If this persists, please report at: https://github.com/flack0x/trendspyg/issues)rr)r"rr#s r_handle_http_errorr&/s c[C/( 6 5  '     E"!U $   / }=5    +'55: :  r! xml_contentc . tj|}ddi}g}|j dD]} | j d} | | j nd} | j d|} | | j nd} | j d }| |j nd}d}|r tj|d }| | |d | d |d d}|rH| j d|}| j d|}| |j nd| |j ndd|d<|rg}| j d|d|}|D]}|j d|}|j d|}|j d|}|j d|}| |j nd| |j nd| |j nd| |j ndd}|j|||d<|j||S#tj$r}td|d}~wwxYw#t$r|}YbwxYw)a Parse RSS XML content into list of trend dictionaries. Shared parsing logic used by both sync and async downloaders. Args: xml_content: Raw XML bytes from RSS feed geo: Geographic code for explore links include_images: Include image URLs and sources include_articles: Include news articles data max_articles_per_trend: Max news articles per trend Returns: List of trend dictionaries Raises: DownloadError: If XML parsing fails zFailed to parse RSS XML: Nhtz&https://trends.google.com/trending/rssz.//itemtitlezN/Azht:approx_trafficpubDatez%a, %d %b %Y %H:%M:%S %zz+https://trends.google.com/trends/explore?q=z&geo=z &hl=en-UStrendtraffic published explore_linkz ht:picturezht:picture_source)r#sourceimagez ht:news_itemzht:news_item_titlezht:news_item_urlzht:news_item_sourcezht:news_item_picture)headliner#r1r2 news_articles) ET fromstring ParseErrorrfindallfindtextrstrptime ValueErrorappend)r'rrrrrootenstrendsitemr*r- traffic_elemr. pub_date_elem pub_date_strr/ trend_data picture_elempicture_source_elemarticles news_itemsnews headline_elemurl_elem source_elem image_elemarticles r_parse_rss_xmlrQXsn2=}}[) 8 9BF Y'<" '"#/ Uyy!4b9 '3'?,##U ), -:-F}))D 15  )$--l C 99%7<"ii(=rB !YY'=rB 7D6O 2 2UY,4,@8==d2=2Ik..t0:0FZ__D  ( )+3J ' j!y<"| MO ===7s;<<=4 )(  )s)GHH/G==H HHrA output_formatz pd.DataFramec|dk(r|S|dk(r ddl}g}|D]}|d|d|d|d d }|rd |vr|d d |d <|d d|d<|rDd|vr@t|d|d<|dr*|ddd|d<|ddd |d<|ddd|d<|j||j|S|dk(rvddl}g} |D]X}|j } t| jdtr| dj| d<| j| Z|j| dS|dk(r(ddl } ddl m} | } |sygd }|r|jd dg|r|jgd| j!| |}|j#|D]}|d|dt|dtr|djn|d|d d }|rd |vr|d d |d <|d d|d<|rDd|vr@t|d|d<|dr*|ddd|d<|ddd |d<|ddd|d<|j%|| j'St)d|d#t$r tdwxYw) a5 Format trends data into requested output format. Shared formatting logic used by both sync and async downloaders. Args: trends: List of trend dictionaries output_format: Output format ('dict', 'dataframe', 'json', 'csv') include_images: Whether images were included include_articles: Whether articles were included Returns: Formatted output in requested format Raises: InvalidParameterError: If output_format is invalid ImportError: If pandas not installed for 'dataframe' format rrrNzXpandas is required for 'dataframe' format. Install with: pip install trendspyg[analysis]r-r.r/r0r,r2r# image_urlr1 image_sourcer4 article_countr3top_article_headlinetop_article_urltop_article_sourcer)indentr)StringIO)rVrWrXrY) fieldnameszInvalid output_format: 'z5'. Must be one of: 'dict', 'dataframe', 'json', 'csv')pandas ImportErrorlenr= DataFramercopy isinstancegetr isoformatdumpsrior\extend DictWriter writeheaderwriterowgetvaluer)rArRrrpd flattenedr-flatr json_trends trend_copyrr\outputr^writerrows r_format_outputrvsd0 + %    #Ew +";/ %n 5 D'U"2$)'N5$9[!',W~h'?^$Ou$<(+E/,B(C_%)383I!3LZ3XD/0.3O.DQ.G.ND*+161G1J81TD-.   T ") #,r||I&& &   +EJ*..5x@*4[*A*K*K*M ;'   z *  + zz+az00 % G    {N; <    p q:> !Ew +?I%P[J\^f?gU;/99;mrs~m %n 5 C'U"2#(>%#8K &+GnX&>N#Ou$<'*5+A'BO$)272H2KJ2WC./-2?-CA-Fu-MC)*05o0Fq0I(0SC,- OOC ' !*  $&}o6A A  o @  s I++Jc|j}|tvs|tvr|SttjttjzDcgc]&}|j |drt |dkDr|(c}dd}d|d}|r|ddj|dz }|d t td t td z }|d z }t|cc}w) zValidate geo parameter for RSS. Args: geo: Country code (e.g., 'US', 'GB') Returns: Validated geo code (uppercase) Raises: InvalidParameterError: If geo is invalid rNzInvalid geo code 'z'.z Did you mean: z, ?z Available: z countries, z US statesz> See trendspyg.config.COUNTRIES and trendspyg.config.US_STATES) upperr r listkeys startswithrajoinr)rcodesimilar error_msgs r_validate_geo_rssr8s ))+C i3)+ !%Y^^%5 6inn>N9O O;//#a&)SX\;;=A?G%SE,Ityy'9&:!<<  ?3y>"2,s9~>NjYYI RRI  **;s +C'rTrxcachec t|}t||||}|r)tj|}|t ||||Sd|} t j|d} | j t# j$|||| } |rtj'|| t | |||S#t j$r+} t| jj||Yd} ~ }d} ~ wt j$r tdt j$r tdt j$r,} tdt| j d| d |d} ~ wwxYw) u6 Download Google Trends RSS feed data with rich media content. **Fast alternative to CSV download** (0.2s vs 10s): - Returns ~10-20 current trending topics - Includes images, news articles, headlines - No time filtering (always current trends) - Perfect for real-time monitoring and qualitative research **Caching:** Results are cached for 5 minutes by default to reduce API calls and improve performance for repeated requests. Use `cache=False` to bypass the cache and always fetch fresh data. **Data Provided (RSS-specific):** - ✅ News article headlines and URLs - ✅ Article images and sources - ✅ Trend image (thumbnail) - ✅ Publication timestamp - ✅ Traffic volume **NOT Provided (use CSV for these):** - ❌ Start/end timestamps - ❌ Related search breakdown - ❌ Time period filtering - ❌ Category filtering - ❌ Large dataset (480 trends) **When to use RSS:** - Journalism: Need news articles + sources quickly - Research: Qualitative analysis (article content) - Monitoring: Real-time alerts (fast, frequent polling) - Visual content: Images for presentations/articles **When to use CSV instead:** - Need >20 trends (CSV has 480) - Need time filtering (4h, 24h, 48h, 7d) - Need category filtering (sports, tech, etc.) - Need historical context (start/end times) - Statistical analysis (large dataset) Args: geo: Country/region code (e.g., 'US', 'GB', 'US-CA') Supports 125 countries + 51 US states output_format: Output format - 'dict' (default): List of dictionaries (Python native) - 'dataframe': pandas DataFrame (requires pandas) - 'json': JSON string - 'csv': CSV string include_images: Include image URLs and sources include_articles: Include news articles data max_articles_per_trend: Max news articles to include per trend (default: 5) cache: Use cached results if available (default: True) Set to False to always fetch fresh data Returns: Depending on output_format: - 'dict': List[Dict] - List of trend dictionaries - 'dataframe': pd.DataFrame - pandas DataFrame - 'json': str - JSON string - 'csv': str - CSV string Raises: InvalidParameterError: If parameters are invalid DownloadError: If RSS fetch fails Examples: >>> # Basic usage - Fast data for monitoring >>> trends = download_google_trends_rss(geo='US') >>> print(f"Found {len(trends)} trending topics") >>> print(trends[0]['trend']) # First trend title >>> # For research - Get news articles for qualitative analysis >>> trends = download_google_trends_rss( ... geo='US', ... output_format='dataframe', # pandas for analysis ... include_articles=True ... ) >>> # Access news articles for each trend >>> for trend in trends: ... print(f"{trend['trend']}: {len(trend['news_articles'])} articles") >>> # For journalism - Quick check with sources >>> trends = download_google_trends_rss(geo='US', output_format='json') >>> # Use in API or save to file >>> # For presentations - Get images >>> trends = download_google_trends_rss(geo='US', include_images=True) >>> image_url = trends[0]['image']['url'] >>> image_source = trends[0]['image']['source'] >>> # Bypass cache for fresh data >>> trends = download_google_trends_rss(geo='US', cache=False) Performance: - Speed: ~0.2 seconds (50x faster than CSV), instant if cached - Trends: ~10-20 items - Data size: ~50-100KB - Update frequency: ~9 times per hour - Cache TTL: 5 minutes (configurable via set_rss_cache_ttl) Data Structure (dict format): { 'trend': str, # Search term 'traffic': str, # '200+', '2000+', etc. 'published': datetime, # Publication time 'image': { # Trend thumbnail (if include_images=True) 'url': str, 'source': str }, 'news_articles': [ # Related articles (if include_articles=True) { 'headline': str, 'url': str, 'source': str, 'image': str } ], 'explore_link': str # Google Trends explore URL } N+https://trends.google.com/trending/rss?geo= timeoutConnection failed - cannot reach Google Trends Possible causes: • No internet connection • DNS resolution failed • Firewall blocking the request Check your internet connection and try again.Request timed out after 10 seconds Possible causes: • Slow internet connection • Google Trends is experiencing delays Try again in a moment.Network error: Details: r%r'rrrr)rr rrervrequestsraise_for_status HTTPErrorr&responser"ConnectionErrorrTimeoutRequestExceptiontype__name__rQcontentset) rrRrrrr cache_key cached_trendsr#rr?rAs rdownload_google_trends_rssrVsB C C ^5EG]^I %++I6  $!-P`a a 8u =C! <<R0!!#B$$ %)5 F Iv. &-AQ RRY   =1::113<<  # #  <       %    $ $ d1g../0s5    s% 'B66E+ !C//AE+?'E&&E+sessionaiohttp.ClientSessionc tK ddl}t|}t||||}|r)t j |} | t | |||Sd|} |du} ||j} |j | |jd4d{} | jdk7rt| j|| | jd{} dddd{| r||j)d{ t+ ||||}|rt j-||t ||||S#t$r tdwxYw777w#1d{7swYxYw#|j$r!}t|j|| Yd}~d}~w|j$r td tj $r td |j"$r,}td t%|j&d |d | d}~wwxYw7#| r||j)d{7wwwxYww)aY Async version of download_google_trends_rss for concurrent fetching. **Why use async?** - Fetch multiple countries/regions in parallel - 3x-100x faster for batch operations - Non-blocking for web applications (FastAPI, Django async) - Resource efficient - one thread handles many connections **Caching:** Results are cached for 5 minutes by default (shared with sync version). Use `cache=False` to bypass the cache and always fetch fresh data. **Performance comparison:** ```python # Sync (sequential) - 125 countries takes ~25 seconds for geo in COUNTRIES: trends = download_google_trends_rss(geo=geo) # Async (parallel) - 125 countries takes ~0.5 seconds results = await asyncio.gather(*[ download_google_trends_rss_async(geo=geo) for geo in COUNTRIES ]) ``` Args: geo: Country/region code (e.g., 'US', 'GB', 'US-CA') Supports 125 countries + 51 US states output_format: Output format - 'dict' (default): List of dictionaries (Python native) - 'dataframe': pandas DataFrame (requires pandas) - 'json': JSON string - 'csv': CSV string include_images: Include image URLs and sources include_articles: Include news articles data max_articles_per_trend: Max news articles to include per trend (default: 5) session: Optional aiohttp.ClientSession for connection reuse. If not provided, a new session is created per call. For best performance with multiple requests, create and reuse a single session. cache: Use cached results if available (default: True) Set to False to always fetch fresh data Returns: Depending on output_format: - 'dict': List[Dict] - List of trend dictionaries - 'dataframe': pd.DataFrame - pandas DataFrame - 'json': str - JSON string - 'csv': str - CSV string Raises: InvalidParameterError: If parameters are invalid DownloadError: If RSS fetch fails ImportError: If aiohttp is not installed Examples: >>> # Basic async usage >>> import asyncio >>> trends = asyncio.run(download_google_trends_rss_async(geo='US')) >>> # Fetch multiple countries in parallel (50x faster than sync) >>> async def fetch_all(): ... countries = ['US', 'GB', 'CA', 'AU', 'DE'] ... results = await asyncio.gather(*[ ... download_google_trends_rss_async(geo=geo) ... for geo in countries ... ]) ... return dict(zip(countries, results)) >>> all_trends = asyncio.run(fetch_all()) >>> # With session reuse for better performance >>> import aiohttp >>> async def fetch_with_session(): ... async with aiohttp.ClientSession() as session: ... tasks = [ ... download_google_trends_rss_async(geo=geo, session=session) ... for geo in ['US', 'GB', 'JP', 'DE', 'FR'] ... ] ... return await asyncio.gather(*tasks) >>> # Use with FastAPI >>> from fastapi import FastAPI >>> app = FastAPI() >>> @app.get("/trends/{country}") ... async def get_trends(country: str): ... return await download_google_trends_rss_async(geo=country) >>> # Bypass cache for fresh data >>> trends = await download_google_trends_rss_async(geo='US', cache=False) Note: Requires aiohttp: pip install trendspyg[async] rNzTaiohttp is required for async operations. Install with: pip install trendspyg[async]rr)totalrrrrrr%r)aiohttpr`rr rrerv ClientSession ClientTimeoutstatusr&readClientResponseErrorClientConnectorErrorrasyncio TimeoutError ClientErrorrrcloserQr)rrRrrrrrrrrr# close_sessionrrr?rAs r download_google_trends_rss_asyncrsuL  C C ^5EG]^I %++I6  $!-P`a a 8u =CtOM)" ?+g++-G! {{30E0E0EB0O{P 0 0T\??c)&xSA ( / 0 0F W0--/ ! ! %)5 F Iv. &-AQ RRa   9   8 00 0 0 0 0 ** 3 qxxc 2 2++ @ ## ) "" !$q'"2"2!343u   " W0--/ ! !1=sH8D<A H8H*(E/EE/:EEE E/ E!E/%H8<H=?H8<EH8E/EE/E, E# !E,(E//H>FHAH"'H  HHH8H5,H/-H55H8geos show_progressdelayc2 ddlm}d}i} |} |r |r |dd } | D]G} t | d ||| } t tt| | | <|dkDs3ddl } | j|I| S#t$r"d}|rddl}td|jYwxYw) um Download RSS trends for multiple countries/regions with progress tracking. **Use this for batch operations** instead of looping manually. Shows a progress bar and returns results as a dictionary. Args: geos: List of geo codes (e.g., ['US', 'GB', 'CA', 'AU']) include_images: Include image URLs and sources include_articles: Include news articles data max_articles_per_trend: Max news articles per trend (default: 5) show_progress: Show tqdm progress bar (default: True) delay: Optional delay between requests in seconds (default: 0) Use 0.5-1.0 if you're fetching many countries to avoid rate limits Returns: Dict mapping geo code to list of trends: {'US': [...], 'GB': [...]} Raises: InvalidParameterError: If any geo code is invalid DownloadError: If any RSS fetch fails Examples: >>> # Fetch 5 countries with progress bar >>> results = download_google_trends_rss_batch(['US', 'GB', 'CA', 'AU', 'DE']) Fetching trends: 100%|██████████| 5/5 [00:01<00:00, 4.2 geo/s] >>> print(f"US has {len(results['US'])} trends") >>> # Fetch all countries (with small delay to be safe) >>> from trendspyg.config import COUNTRIES >>> results = download_google_trends_rss_batch( ... list(COUNTRIES.keys()), ... delay=0.5 # 0.5s between requests ... ) Fetching trends: 100%|██████████| 125/125 [01:05<00:00, 1.9 geo/s] Warning: Fetching many countries (>50) without delay may trigger Google rate limits. If you get blocked, wait a few minutes and add delay=0.5 or delay=1.0. Note: For maximum speed with many countries, use the async version: `download_google_trends_rss_batch_async()` which fetches in parallel. rtqdmTFN?Note: Install tqdm for progress bar: pip install trendspyg[cli]fileFetching trendsr)descunitr)rrRrrr) rr`sysprintstderrrr rrtimesleep)rrrrrrrhas_tqdmrresultsiteratorrrArs r download_google_trends_rss_batchrsh#&(GH#45A + )-#9  DJ/  19  JJu   N; #   Szz # #sA++(BBmax_concurrentcK ddl}ddl} ddlm}d} i} |j|d d d td tffd } |j4d{} |Dcgc] }| | | }}|rC| rA|j|t|dd 23d{}|d{\}}|| |<|j|D]}|d{\}}|| |<dddd{| S#t$r tdwxYw#t$r#d} |rddl} t d| jYwxYw7cc}w776f7t7\#1d{7swY| SxYww)u* Download RSS trends for multiple countries/regions in parallel with progress. **Fastest way to fetch multiple countries** - uses async/parallel requests. Shows a progress bar and returns results as a dictionary. Args: geos: List of geo codes (e.g., ['US', 'GB', 'CA', 'AU']) include_images: Include image URLs and sources include_articles: Include news articles data max_articles_per_trend: Max news articles per trend (default: 5) show_progress: Show tqdm progress bar (default: True) max_concurrent: Maximum concurrent requests (default: 10) Lower this if you get rate limited (try 5 or 3) Returns: Dict mapping geo code to list of trends: {'US': [...], 'GB': [...]} Raises: InvalidParameterError: If any geo code is invalid DownloadError: If any RSS fetch fails ImportError: If aiohttp is not installed Examples: >>> # Fetch 5 countries in parallel (~0.3s total) >>> import asyncio >>> results = asyncio.run( ... download_google_trends_rss_batch_async(['US', 'GB', 'CA', 'AU', 'DE']) ... ) Fetching trends: 100%|██████████| 5/5 [00:00<00:00, 15.2 geo/s] >>> # Fetch ALL 125 countries in parallel (~2-5 seconds) >>> from trendspyg.config import COUNTRIES >>> results = asyncio.run( ... download_google_trends_rss_batch_async( ... list(COUNTRIES.keys()), ... max_concurrent=10 # Limit concurrent requests ... ) ... ) Fetching trends: 100%|██████████| 125/125 [00:03<00:00, 38.5 geo/s] Performance: - 5 countries: ~0.3 seconds (vs ~1s sync) - 125 countries: ~3-5 seconds (vs ~25s sync) - Uses connection pooling for efficiency Warning: High concurrency (>20) may trigger Google rate limits. If you get errors, reduce max_concurrent to 5 or 3. Note: Requires aiohttp: pip install trendspyg[async] rNzZaiohttp is required for async batch operations. Install with: pip install trendspyg[async]rTFrrrrrrc K4d{t|d|d{}||fcdddd{S7277 #1d{7swYyxYww)Nr)rrRrrrr)r)rrrArrr semaphores r fetch_onez9download_google_trends_rss_batch_async..fetch_one}si  ;$-!1'= F;       sNA=AA?A AAAAAA A AAr)rrr)rrr` tqdm.asynciorrrr Semaphorestrtupler as_completedra)rrrrrrrratqdmrrrrrrtasksresultrAcorors ``` @r&download_google_trends_rss_batch_asyncr*sz #.&(G!!!.1I !8 s u  %w$$&&&'489S7C(99 X %$$$U+%j& ! & &f %+l V% -,,U3 &$(j V%  &&&& Ne   9    #   Szz # #,&9 & + !)!&&&&& NsE"C4D AE"D;E"E "D=2+E E!E"E%E +E,'E EE " E"-E .E"4D  E" (D84E"7D88E"=E EE E  E" EE EE")TTrxTg)TTrxTr),__doc__rrxml.etree.ElementTreeetree ElementTreer5rtypingrrrrrr r r_rnrconfigr r r exceptionsrrrutilsr OutputFormatrboolintr r&bytesrQrvrrrfloatrrrr!rrs>"""LLL55LL 9: Q QQQ Q  Q& C& c& & & Rbb bb b  b  $Z bJx Jx x x  x  4:sN *+ x v+3+3+>"(!"# S SSS S  S  S 4:sN *+SF"(!"#15xS xSxSxS xS  xS - . xS xS 4:sN *+xSz !"# T s)TTT T  T  T #tDz/Tr !"# r s)rrr r  r  r #tDz/rr!