;iE dZddlZddlZddlZddlZddlZddlmZmZddlm Z ddl m Z m Z m Z mZddlZddlmZddlmZmZmZmZmZmZGdd ej4Zd e ee fd efd Zd e ee fd efd Zde e d e e fdZ ee e!jDdz Z#e#dz Z$d?ded e fdZ%d?ded ee fdZ&d?ded e ee ffdZ' d@de ee fde e ee fde(d e e ee ffdZ)dZ*dZ+d ee,fdZ- dAde,de,ded e e ee ffdZ.d e e ee fd e ee ffd!Z/d"e ee fd e ee ffd#Z0dBd$e ee fded%e,de(d e ee ff d&Z1d'Z2d(Z3d)Z4d e(fd*Z5dCd+e ee fd,ed-e,d efd.Z6d?d/ed0ed eefd1Z7d2ed ee ee ffd3Z8 dDd4e ee fded5e,d6e(d e ee ff d7Z9d?d8eded e(fd9Z:dEd:e,d efd;Z;d4e ee ffd<Zd>k(re=yy)Fa Multi-Layer Trend Analysis Pipeline Layer 1: EMBED + CLUSTER (local) - sentence-transformers, UMAP, HDBSCAN Layer 2: FAST LLM CLASSIFICATION (Groq Llama 3.3 70B) Layer 3: DEEP ANALYSIS (Claude CLI via Max subscription) Layer 3.5: DEEP RESEARCH (Hybrid Research Service API) Outputs: - output/analysis_clusters.json - output/analysis_classified.json - output/analysis_opportunities.json (includes research data) N)datetimetimezone)Path)ListDictAnyOptional) load_dotenv) run_migrationcreate_analysis_runupdate_analysis_runinsert_opportunities_batch&update_opportunity_research_by_clusterget_latest_analysis_runc"eZdZdZfdZxZS) NumpyEncoderz&JSON encoder that handles numpy types.c t|tjr t|St|tjr t |St|tj r|jSt|%|SN) isinstancenpintegerintfloatingfloatndarraytolistsuperdefault)selfobj __class__s 1/home/sietch6/trending-topics-pipeline/analyze.pyrzNumpyEncoder.default)s[ c2:: &s8O c2;; ':  c2:: &::< ws##)__name__ __module__ __qualname____doc__r __classcell__)r!s@r"rr's0$$r#ranalysisreturnclddl}|jd}t|ttfr|dkDr t |St|t r5dD]0}||vst||ttfs"t ||cSt|t r t |}|dkDr|S d|vr4 t |jddj}|dkDr|S |jd|}|r) t |jd}d|cxkrdkr|Sd D]}|j|}|st|ttfr|dkDr t |cSt|t sP|jd |}|se t |jd}|dkDr|cS|jd i}|rd d d d ddd} ddd d ddd} d d d d ddd} d|vr| } n d|vr| } n| } d} d}| jD]e\}}|j|i}t|t r|jdd}nt|ttfr|}nd}|dkDs\d}| ||zz } g|r| dkDr t| dSy#t$rYwxYw#t$rYwxYw#t$rYwxYw#t$rYwxYw)an Robustly extract weighted_score from Claude's response. Handles various formats Claude might return: - Direct float: 4.65 - String: "4.65" - String with denominator: "4.65/5.0" - Dict: {"score": 4.65} or {"value": 4.65} - Narrative: "The weighted score is 4.65" Falls back to calculating from dimension scores if all else fails. rNweighted_score)scorevaluetotalr,/z!(\d+\.?\d*)\s*(?:/\s*5|out of 5)?)weighted_score_calculationweighted_score_math final_score total_scorez=\s*(\d+\.?\d*)scoresg?333333?皙?)scroll_stop_powercompletion_probabilityshare_save_potential demand_signalvisual_potentialevergreen_potentialg333333?皙?)curiosity_intensityr= content_gapdepth_explainabilityr?audience_breadth)r=rBexplainabilityr?rDcompetition_gapr:rAFr-T)regetrrrdictstr ValueErrorsplitstripsearchgroupitemsround)r)rIrawkeyvalmatch alt_fieldalt_rawr7dimension_map_v4dimension_map_v3dimension_map_v1 active_mapr/ found_anydim_nameweightdim_data dim_scores r"extract_weighted_scorerc3s" ,,' (C#U|$qSz#t@ 'CczjSC<@SX& ' #s *CQw  #: CIIcN1-33567J  >D  EKKN+s#EKKN37#&J#"\\(B 'F "&&*$(! $#'  $(!$(#' $  ""#' $#   & ()J "f ,)J)J * 0 0 2 , Hfzz(B/H(D)$LL!4 HsEl3$  1} V++ , ? " A       "&sNI6*1J2&JJ J&6 JJ JJ J#"J#& J32J3c|jdd}|dk7r|S|jdijdd}|dk(rrgd}tfd|D}t|}|d ks|d krtd |d |d yy)z Validate and potentially correct the verdict based on MAKE_NOW gates. MAKE_NOW requires ALL 6 gates to pass AND weighted_score >= 4.0. If verdict is MAKE_NOW but gates don't all pass, downgrade to WORTH_MAKING. verdictSKIPMAKE_NOWmake_now_gates gates_passedr)scroll_stop_testuniversal_access_test depth_test share_testsatisfaction_testchannel_fit_testc3JK|]}j|dusdyw)Tr1NrJ).0ggatess r" z#validate_verdict..s IEIIaLD4H1Is##g@z< [VALIDATE] Downgrading MAKE_NOW to WORTH_MAKING (gates: z /6, score: ) WORTH_MAKING)rJsumrcprint)r)reri gate_namesr,rts @r"validate_verdictr|sll9f-G* LL)2 .E99^Q/LqU IjII ,H5Na>C/ L\NZefteuuvwx r# opportunitiesc |Dcgc]}|jddk(s|}}|Dcgc]}|jddk7s|}}|jddt|}|dk(r|Stdt |dz}|tdt |d zz}|tdt |d zz}d}t |D]K\} } | jdd } | |krd } n| |krd } n | |krd} nd} | | k7s8d| d<| | d<| | d<|dz }Mt d|dt d|d||z d||z d||z t|z||zScc}wcc}w)u Reassign verdicts based on relative ranking rather than absolute scores. This enforces the target distribution mechanically. Distribution: - Top 15% by weighted_score → MAKE_NOW - Next 35% → WORTH_MAKING - Next 20% → NEEDS_RESEARCH - Bottom 30% → SKIP Exception: Any opportunity the model already marked SKIP stays SKIP regardless of score (the model identified a structural reason it shouldn't be produced). rerfct|Srrcxs r"z)assign_verdicts_by_rank..s !7!:r#TrUreverserr1r8gffffff?r@UNKNOWNrgrxNEEDS_RESEARCHverdict_adjustedoriginal_verdictz [RANK] Adjusted z# verdicts based on relative rankingz" [RANK] Distribution: MAKE_NOW=z, WORTH_MAKING=z, NEEDS_RESEARCH=z, SKIP=)rJsortlenmaxr enumeraterz) r}o model_skips scoreablenmake_now_cutoffworth_making_cutoffneeds_research_cutoffadjusted_countiopporiginal new_verdicts r"assign_verdicts_by_rankrs",JquuY/?6/I1JKJ)HqQUU9-=-GHIHNN:DNI IAAv!ST]+O)C3q4x=,AA/#aQX2GGNI& 3779i0  $K $ $(K & &*K K ( "&*C" #&.C" #(C N a N! $  00S TU ..?ObetOtNuvGH]`sHsGtt{|}@U|UX[\gXh|h{i jk { ""KKHsEEEEoutputztrends_history.db channel_idc2|r td|dz Stdz S)a Get the path for opportunities JSON file. Args: channel_id: If specified, returns channel-specific path. If None, returns the generic path (backward compatibility). Returns: Path to the opportunities JSON file analysis_opportunities_z.jsonzanalysis_opportunities.json) OUTPUT_DIRrs r"get_opportunities_pathrs(5j\GGG 5 55r#cddl}t|}|jsy|jd}|j |||S)z Create a backup of the existing opportunities file before writing. Args: channel_id: Channel ID for channel-specific file Returns: Path to backup file, or None if no file existed to backup rNz .json.bak)shutilrexists with_suffixcopy2)rr source_path backup_paths r"backup_opportunities_filer+sD(4K    ))+6K LLk* r#ct|}|jr+t|d5}tj|cdddSg|dS#1swYxYw)z Load existing opportunities file if it exists. Args: channel_id: Channel ID for channel-specific file Returns: Existing data dict, or empty structure if file doesn't exist rN)r}r)rropenjsonload)rpathfs r"load_existing_opportunitiesr@sR "* -D {{} $_ 99Q<  z ::  s A  A existing_datanew_opportunities replace_modecb|r|S|jdg}i}|D]}|jd}||||<t}|D]'}|jd}||j|)g}|D]} | jd}|e||vra||} | j} gd} | D]} | | vs| | vs | | | | <gd}|D]} | | vs| | vs | | | | <|j | {|j | |j D]\}} ||vs |j | |S)a Merge new opportunities with existing ones. Matching is done by cluster_id. New analysis updates existing entries. Opportunities not in the new batch are preserved. Args: existing_data: Existing opportunities data structure new_opportunities: Newly analyzed opportunities replace_mode: If True, completely replace (old behavior). If False, merge. Returns: Merged list of opportunities r} cluster_id) research_completedresearch_report_pathresearch_report_contentresearch_word_countresearch_source_countresearch_duration_secondsresearch_generated_atresearch_queryresearch_error) video_conceptslayer4a_metadata layer4a_error video_scriptproduction_speclayer4b_metadata layer4b_errorvisual_direction asset_specslayer4c_metadata layer4c_error)rJsetaddcopyappendrR)rrr existing_oppsexisting_by_clusterrrnew_cluster_idsmergednew_opp existing_opp merged_oppresearch_fieldsfield video_fieldss r"merge_opportunitiesrQs&  !%%or:M2WW\*  !.1  +2 eO ,WW\*  !    +, F$#[[.  !j4G&G.z:L JO ) <L(U*-D(4U(;Ju% < L & <L(U*-D(4U(;Ju% < MM* % MM' ";#@%8$=$=$?( L _ , MM, '( Mr#cgd}g}|D]} t||r/td|tdt j dyy#t$r|j|Y_wxYw)z+Verify all required packages are available.)sentence_transformersumaphdbscangroqz[ERROR] Missing packages: z>Run: pip install sentence-transformers umap-learn hdbscan groqr1N) __import__ ImportErrorrrzsysexit)requiredmissingpkgs r"check_dependenciesrsvH G  sO  *7)45 NO   NN3  s A  A)(A)ctjd}|s tdtjd t j gddddd}|jd k7r5td t|jtjdtd y#t$r#td tjdY7t j$r#td tjdYkwxYw)zVerify API keys are configured. GROQ_API_KEYz:[ERROR] GROQ_API_KEY not found in environment or .env filer1)claude--print-pzSay OKTZinputcapture_outputtexttimeoutrz[ERROR] Claude CLI not workingzT[ERROR] Claude CLI not found. Install with: npm install -g @anthropic-ai/claude-codez[ERROR] Claude CLI timed outz[OK] All API keys verifiedN) osgetenvrzrr subprocessrun returncodestderrFileNotFoundErrorTimeoutExpired)groq_keyresults r"check_api_keysrsyy(H  JK   1dB     ! 2 3 &-- HHQK &'  de   $ $ ,-  sA B%%)D2DDctjt}|j}|j d|j }|j |r|dSdS)z&Get the most recent collection run ID.z#SELECT MAX(id) FROM collection_runsrN)sqlite3connectDB_PATHcursorexecutefetchoneclose)connrrs r"get_last_run_idrsN ??7 #D [[]F NN89 __ FJJL6!9(D(r#limit since_run_idctjt}tj|_|j }|r?d}|g}|r|dz }|j ||dz }|r|d|z }|j||n,d}|r|d|z }|dz }|r|d|z }|j||j}|jg}t} |D]} | dj} | j| vs)| j| j|j | | d | d | d xsd | d xsd| dxsd | dxsd| dxsd| dxsd| dxsd| dd |S)aLoad trend items from SQLite database. Args: limit: Maximum number of items to load since_run_id: If set, only load items from runs >= this ID (for incremental analysis) channel_id: If set, only load items tagged to this channel a SELECT DISTINCT ti.topic_name, ti.source, ti.region, ti.final_score, ti.selftext, ti.num_comments, ti.subreddit, ti.category, ti.url, ti.signal_type, ti.run_id, ic.confidence as channel_confidence FROM trend_items ti JOIN item_channels ic ON ti.id = ic.item_id WHERE ti.topic_name IS NOT NULL AND ti.topic_name != '' AND ic.channel_id = ? z AND ti.run_id >= ?z1 ORDER BY ic.confidence DESC, ti.final_score DESCz LIMIT a SELECT DISTINCT topic_name, source, region, final_score, selftext, num_comments, subreddit, category, url, signal_type, run_id FROM trend_items WHERE topic_name IS NOT NULL AND topic_name != '' z AND run_id >= z ORDER BY final_score DESC topic_namesourceregionr5rselftextr num_comments subredditcategoryurl signal_typeunknownrun_id) r r r r-rrrrrrr)rrrRow row_factoryrrrfetchallrrrOlowerr) rr rrrqueryparamsrowstrends seen_topicsrowtopics r"load_trends_from_dbr!s ??7 #D{{D [[]F   * *E MM, ' DD  wug& &Euf%   |n5 5E --  wug& &Eu ?? DJJL F%KL!'') ;;= + OOEKKM * MM#h-h-]+0q O1r #N 3 8q -3 O1r5z'R"=1>Yh-   $ Mr#rc Hddlm}ddl}ddl}t dt dt dg}|D]+}|d}|dr|d |ddd zz }|j |-t d t |d |d }|j|d}t d|jt d|jddddd} | j|} t d| jt d|jdddd} | j| } t t| d| vrd ndz } t| jd}t d!| d"|d#i}t!t#|| D]\}\}}t%|}||vr$|ggtttd$||<|d|d%|d&|d'|d(|j'd)d*d+}||d,j |||d-j)|j'd)d*|d%d.k(r:||d/j |d|d0|d1|d'|dr|ddd2nd3|d(d4||d5j)|d%||d6j)|d&|j+D]}t|d5|d5<t|d6|d6<t|d-|d-<t |d,|d7<t |d5|d8<t |d/|d9<t-d:|d,D}||d;<|dkD|d<<t/|j+Dcgc] }|d=dk7s |c}d>d?}d@|vr|d@}d|d=<dA|dB<t || ||t1j2t4j6j9dC}t:dDz }t=|dE5}t?j@||dFtBGdddt dH||Scc}w#1swYxYw)Izc Layer 1: Embed topics with sentence-transformers, reduce with UMAP, cluster with HDBSCAN. r)SentenceTransformerN= ============================================================zLAYER 1: EMBEDDING + CLUSTERING<============================================================r r z[1.1] Embedding z topics with all-MiniLM-L6-v2...zall-MiniLM-L6-v2T)show_progress_barz Embedding shape: z2[1.2] Reducing dimensions with UMAP (384 -> 10)... r9cosine*) n_components n_neighborsmin_distmetric random_statez Reduced shape: z [1.3] Clustering with HDBSCAN...r2 euclideaneom)min_cluster_size min_samplesr0cluster_selection_methodr1z Found z clusters, z noise points)rtopics reddit_postssourcesregions signal_typesr r r-rrr)r r r r-rrr9r=redditr:rrir)titlerrr-rrr;r< topic_countcross_source_countreddit_post_countc3JK|]}|jddk(sdyw)rquestionr1Nrq)rrts r"ruz+layer1_embed_and_cluster..s Z1quu]7Kz7YQZs##question_signal_counthas_question_signalrc |dS)Nr@rs r"rz*layer1_embed_and_cluster..s a &r#rz-1zUnclustered/Noiselabel) total_topicstotal_clusters noise_pointsclusters timestampzanalysis_clusters.jsonwrHindentclsz [1.4] Saved: )"rr#rrrzrrencodeshapeUMAP fit_transformHDBSCAN fit_predictrlistcountrziprLrJrvaluesrysortedrnowrutc isoformatrrrdumpr)rr#rr topic_textsrErmodel embeddingsreducerreduced clusterercluster_labels n_clustersn_noiserNrtrendrJ label_str topic_datacquestion_countsorted_clusters noise_clusterr output_pathrs r"layer1_embed_and_clusterrt3s: / +, (OK ! Z= C!J--- -D4 ! S-..N OP  2 3EkTBJ #J$4$4#5 67 >?ii G##J/G !'-- 12 ,-!&  I **73NS()"2FQANJ>"((,G L Ky FGH&s6>'BC$<>E5J H $# "55 # #HY  -HoHo7^< 99]I>   H%,,Z8N+// -0ST ?h & Y  / 6 6|,";/ %n 5w7!23.q{+-"%a l"3 !$Q~%6!7 Z( ZZ%3 !"#1A#5   6OO%?q<B)>? &O x  &( l#!4 gF $#\\(,,/99; F77K k3 91 &!A<89 OK= )* M3 @*99s P&PPP! clusters_datac  ddlm}ddl}tdtdtd|t j d}g}g}g}|d }t |dd D]\}} t| d d d dd} dj| D cgc]} d| dd| ddc} } d} | jdrct| ddd dd}g}|D]3}|jd|dd|dd|jddd5d d!j|z} d"d#j| jd$g}| jd%r |d&| d'd(z }d#j| d)}d*| d+|d|d| d, } |jjjd-d.|d/gd0d12}|jdjj j#}d} t%j&|}|d4dd5dd6}|jd7d4j1}|d8vrd4}||d7<i| ||jd9d:}|j||jd;d}d<}|d=k(r|d>k\rd }n |d?k(r|d@k\rd }|jd9d}|r dA|dd dBnd}|rB|j|tdC|dDzdEt3d t5|dF|d&|dG| nV|j||jdHdddI}tdC|dDzdEt3d t5|dJ|d&|dK| i} |D]<}!|!jdNijd7dO}| j|ddDz| |<>tdPtdQt5|tdRt5|tdSt5|tdT| t5|t5|t5|| ||t;j<t>j@jCdU}"tDdVz }#tG|#dW5}$t%jH|"|$dXtJYdddtdZ|#|"Scc} w#t$j($ra|j+d3||j,}|r> t%j&|j/}n#t$j($rYnwxYwYwxYw#t6$rS}tdC|dDzdLt9|d4ddMt9|dd6}|ji| |dd:Yd}~d}~wwxYw#1swYxYw)[z Layer 2: Use Groq Llama 3.3 70B to classify and FILTER clusters. 3-category system: EXPLAINER_OPPORTUNITY, CURIOSITY_MAGNET, KILL. r)GroqNr$z(LAYER 2: GROQ CLASSIFICATION + FILTERINGr%r)api_keyrN2r9c |dS)Nr-rIrs r"rz,layer2_classify_and_filter..s 7r#Trr2 - r z (source: r rwrr:c&|jddS)Nrrrqrs r"rz,layer2_classify_and_filter..s~_`Iar#- r/r: "r?z" (rz comments)zReddit: z | zSignal types: , r=rG (rFz question signals)r;zBClassify this topic cluster. Pick ONE category. CLUSTER: Topics: z Sources: u CATEGORIES: EXPLAINER_OPPORTUNITY = People want something EXPLAINED. Look for: - Questions starting with "Why...", "How does...", "What is...", "ELI5..." - Reddit posts asking for understanding, not just news - Topics where the answer requires more than one sentence - Examples that ARE explainer opportunities: "Why does every power plant boil water?", "How do undersea cables handle all internet traffic?", "What are the Epstein files?" - Examples that are NOT: "Who won the Super Bowl?", "What time is the game?", "Breaking: CEO resigns" CURIOSITY_MAGNET = Not a question, but fascinating. Look for: - "What is this thing?" identification posts with high engagement - Counterintuitive facts people love sharing - "I never knew that" moments - Examples: mystery object identification, surprising animal behavior, bizarre historical facts KILL = Everything else. This includes ALL of the following: - Sports scores, game results, player stats, team matchups - Stock prices, market moves, financial tickers - News headlines about events (elections, arrests, speeches, disasters) with no deeper "why" angle - Celebrity gossip, entertainment news, movie/TV releases - YouTube video titles (these are someone else's content) - Music releases, concert announcements - Weather, schedules, product launch dates - Political news that is about WHAT HAPPENED not WHY IT WORKS THAT WAY - Any topic that will be irrelevant in 2 weeks - Topics where a Google search gives you the complete answer in 3 seconds Ask: "Could someone make a 10-minute video explaining this that people would still watch 6 months from now?" If YES → EXPLAINER_OPPORTUNITY or CURIOSITY_MAGNET If NO → KILL Respond in this exact JSON format: { "classification": "EXPLAINER_OPPORTUNITY" or "CURIOSITY_MAGNET" or "KILL", "score": number from 1 to 10, "reason": "one sentence why", "the_question": "the specific question people want answered, or null if KILL" }zllama-3.3-70b-versatileuser)rolecontentr'r@)rdmessages max_tokens temperature\{.*\}KILLzCould not parse Groq response)classificationr-reason the_questionr)EXPLAINER_OPPORTUNITYCURIOSITY_MAGNETrr)groq_classificationrr-Frrvr → ... [r1r0u] ✓ z/10)r(u] ✗ z/10) - z ] Error: zClassification failed: rrz [2.1] FILTER RESULTS:z Total evaluated: z PASSED: z KILLED: z Kill breakdown: )total_evaluated total_passed total_killedkill_breakdownpassed_clusterskilled_clustersrOzanalysis_classified.jsonrPrHrQz [2.2] Saved: )&rrwrIrzrrrr^joinrJrchat completionscreatechoicesmessagerrOrloadsJSONDecodeErrorrPDOTALLrQupperminr ExceptionrLrr_rr`rarrrbr)%rurwrIclientall_classifiedrrrNrclusterr9rE topic_listreddit_context reddit_items reddit_linesrsignal_contextr;promptresponse response_textr json_matchcls_typeclassified_clusterr- passes_filterr q_displayre kill_reasonsrorrsrs% r"layer2_classify_and_filterrs/   / 45 (O "))N3 4FNOOZ(H" .Om 7)/CTRSUTUVYYX^_ST"Q|_$5Z( }A N_`  ;;~ &!'."9?akopqsrstLL! u##d1[>*:$qzl$quuUcefOgNhhr$st u'%**\*BBN*$))GKKPR4S*T)UV ;;, - 7+B#C"DDVW WN))GI./    ' -^K m{{..55/#)f=> 6H%,,Q/77??EEGM"N !%M!:%&,=$( "&))*:FCIIKHTT!/7N+ ,""'5 . 2 2> B"   ! !"4 5 #&&w2E!M22uz $ //EQJ $ )--nbAL:F% Sb 12#6BI&&'9:AaC5#b#h-"8!9zE7RVW`Vabc&&'9:'++Hb9#2>AaC5#b#h-"8!9zE7RYZ`YabcKOmdL ?,b1556F R+//:Q> V? #% #C$7#8 9: N3/0 12 N3/0 12 "<. 12~.O,O,&**\\(,,/99;F99K k3 91 &!A<89 M+ '( MU `^'' YYy-K )-J4D4D4F)G//  j m C!uIc!fX. /"(3CF8< $ N  " "#kg#knfj#k l l mD99su R A S<%R:ES< U2S98#SS9S2/S91S22S95S<8S99S<< UAUUU$classified_data max_clustersctdtdtdd}d}d}|rddlm}||}|r|j}|jxsd} |j xsd} |j xsd} |jxsd}d|jd |jd }| r |d | d z }| r |d | d z }| r |d | d z }|dz }td|j|jdg} d} |rY t|t|jdgt|jdgt| } td| | s@tddgtjtj j#dSt| }|dkDr&t| |kDr| d|} td|d|dntdt| dg}t%| D]\}}|jdi}|jdxs(|jdxs|jdxsd }td!|d"zd#t'|dd$d%|d&Dcgc]}|d' }}d j)|Dcgc]}d(| c}}d)}|jd*rg}|d*D]}|jd+d}|jd,d}|jd-d.}|jd/d}|jd0ddd1}|j+d2|d3|d4|d5|d6 |s|j+d7|d%d j)|}d8j)|d9} |d:}!|jd;d}"|jd|d |%d?|d@|dA| dB|!dC|"dD}& t-j.gdE|&dFdFdGH}'|'j0dk7rtdI|'j2ddJ)|'j4j7}( ddl})|)j;dK|(|)j<}*|*r$t?j@|*jC}+nt?j@|(}+|dL|+dL<|dM|+dM<|d9|+d9<|d:|+d:<|jd<|+d<|jd,|+dN<||+d<|r||+dO<tE|+},|,|+dP<dQ|+vr dR|+vr |+dQ|+dR<n dR|+vrd|+dR<|+jdSdT}-tG|+}.|.|-k7r|.|+dS<dF|+dU<|-|+dV<|j+|+|+jdSd }/|+jdRd ddW}tdX|/dY|,dZd[|d%|D0cgc] }0dP|0vs|0 }1}0|1jMdbdFc|D0cgc] }0dP|0vs|0 }2}0tO|1}1|1|2z}3tQ|}4|s tS|nddgi}5tU|5|3|}6|6jMdedFct|5jddg}7t|3}8t|6}9|s|7dkDrtdf|7dg|8dh|9dintdj|9dk|t| |9|6tjtj j#dl}'tW|}:|:rtdm|:tY|4dn5};t?jZ|'|;dot\pdddtdq|4| rD|6rB t_|| |6drs}jH$rK}td\|j+|jd]d |(dd^t'||dLd_Yd}~Yd}~wwxYw#t,jJ$rtd`Yt$r"}tdat'|Yd}~Dd}~wwxYwcc}0wcc}0w#1swYMxYw#t$r2}tdx|| rta| dyt'|zYd}~2d}~wwxYw)|a Layer 3: Use Claude Opus for deep analysis of each surviving cluster. Args: classified_data: Output from Layer 2 channel_id: Optional channel ID for channel-specific analysis context max_clusters: Limit analysis to N clusters (0 = unlimited, for testing use 3-5) replace_mode: If True, replace entire file. If False (default), merge with existing. r$z"LAYER 3: CLAUDE OPUS DEEP ANALYSISr%rNr get_channelzK CHANNEL CONTEXT: You are analyzing opportunities for the YouTube channel "z". Channel Identity: r{z,The Itch (what viewer need this scratches): zFormat Pattern: z2Competitive Gap (what makes this channel unique): a IMPORTANT: Score topics higher if they directly exploit this channel's competitive gap. Score topics lower if they don't fit the channel identity, even if they're otherwise good opportunities. A great topic for another channel is a bad topic for THIS channel. z[3.0] Channel context loaded: r all_trendsrN)ritems_analyzedclusters_formed layer2_passedz[3.0] Analysis run created: ID z-[WARN] Failed to create analysis run record: z)[WARN] No clusters passed Layer 2 filter!)total_analyzedr}rOz[3.0] LIMITED: Analyzing r0z clusters (--max-clusters flag)z[3.0] Analyzing z clusters that passed filter...rrrUnknownz [3.r1z ] Analyzing: <rr9r r|z No Reddit posts in this cluster.r:rr-rrr?rr'r~rz " (score: z , comments: rwz Context: rr;rArFru ═══════════════════════════════════════════════════════════ CHANNEL-SPECIFIC HOOK CRITERIA ═══════════════════════════════════════════════════════════ uYou are a short-form video opportunity scout. Your job is to find the ONE specific detail in this cluster that would stop someone mid-scroll — then build a complete video concept around it. u ═══════════════════════════════════════════════════════════ RAW MATERIAL ═══════════════════════════════════════════════════════════ CLUSTER POSTS: z REDDIT SIGNALS: z SOURCES: z CROSS-SOURCE COUNT: z QUESTION SIGNALS: u+ ═══════════════════════════════════════════════════════════ PHASE 1: IDENTIFY THE HOOK MECHANISM ═══════════════════════════════════════════════════════════ Every viral hook uses one of these 5 mechanisms. Identify which ONE is strongest in this cluster: 1. CURIOSITY_GAP — "Wait, what?" A specific detail that contradicts what the viewer assumes to be true. They cannot scroll past without resolving it. Example: "CAPTCHAs Don't Actually Care About Your Answers" 2. SELF_RECOGNITION — "That's so me." Viewer sees their own behavior named in the first 1.5 seconds. Creates belonging. Example: "You're Not a Night Owl. You're Afraid of Tomorrow." 3. THREAT_INDIGNATION — "They're doing WHAT to me?" Reveals a specific manipulation the viewer experiences daily. Creates righteous awareness. Example: "Why the 'X' Button on Ads is Designed to Be Impossible to Tap" 4. CONSEQUENCE_CHAIN — "Then what happens?" A simple starting point with cascading non-obvious consequences. Creates domino-effect curiosity. Example: "What Actually Happens If You Never Pay a Parking Ticket" 5. NARRATIVE_SURPRISE — "That actually happened?" A real event that sounds like fiction. Lead with the most unbelievable detail. Example: "The Pope Who Put a Corpse on Trial" Search this cluster for up to 3 candidate hooks. For each one: - Quote the exact source post title - Identify the hook mechanism - Articulate the specific "wait, what?" moment in one sentence - Rate scroll-stop strength 1-5 (5 = physically cannot scroll past this) If NO hook with scroll-stop strength 4+ exists in this cluster, say so honestly. Not every cluster contains a video. ═══════════════════════════════════════════════════════════ PHASE 2: BUILD THE CONCEPT ═══════════════════════════════════════════════════════════ Take your strongest hook and build a video concept. The concept is about that ONE specific detail, not the general theme. A. PREMISE (the video title) The viewer must encounter something they cannot explain and need to resolve. Rules: - Must contain the specific detail that creates the information gap - Must make someone with ZERO prior interest in this topic stop scrolling - Under 15 words. The hook is in specificity, not length. - NO listicle formats ("7 Ways...", "10 Things...") - NO "Why You..." followed by something that makes viewer feel broken - The viewer should feel CURIOUS, not lectured CRITICAL FORMAT RULE: Do NOT default to the "You're Not X. You're Y." template. This format has been overused across short-form content and triggers pattern fatigue in algorithmic feeds. The orienting response fires on NOVELTY — a premise that follows a predictable syntactic pattern fails at Phase 1 regardless of how good the insight is. Each premise must have a distinct sentence structure. If you find yourself starting with "You're Not..." — stop and rewrite. The same insight can always be expressed differently: Instead of: "You're Not Lazy. Your Brain Is Missing a Start Button." Try: "Your Brain Has No Start Button. That's Why You're Stuck." Instead of: "You're Not a Night Owl. You're Afraid of Tomorrow." Try: "The Real Reason You Stay Up Late Has Nothing to Do With Sleep." The insight is the same. The format is fresh. B. FIRST_FRAME What text/image appears in the first frame to maximize scroll-stop? This is the thumbnail + opening visual moment. C. TRIGGER_MAP How this concept triggers the hook mechanism: - Which of the 5 mechanisms does this use? - What specific word/phrase/image creates the trigger? - Why can't the viewer scroll past this? D. OPENING HOOK (first 10 seconds) The spoken words that OPEN the curiosity loop wider. Do NOT explain here — intensify the question. E. CORE REVEAL The satisfying explanation that closes the loop. Must be: - Genuinely surprising (not "it's complicated") - Explainable in 60+ seconds (if shorter, it's a fun fact, not a video) - Leave viewer feeling smarter, not anxious F. DEPTH CHECK Can this sustain a 90-120 second video? Must have: - A reveal that takes at least 60 seconds to properly explain - At least one additional layer of surprise beyond the initial hook - Enough substance that viewer learns something real, not just a fun fact ═══════════════════════════════════════════════════════════ PHASE 3: SCORING & VERDICT ═══════════════════════════════════════════════════════════ SCORING (1-5 each, be brutally honest): SCROLL_STOP_POWER (25%): Can this premise make someone physically stop scrolling? - 5: Impossible to scroll past. "Wait, WHAT?" - 4: Very strong. Most people would stop. - 3: Interesting but skippable. DANGER ZONE — this is where mediocre content lives. - 2: Mild interest. "Huh, cool I guess." - 1: No hook. Educational but not compelling. COMPLETION_PROBABILITY (25%): Will viewers watch to the end? - 5: The reveal is so satisfying they'll watch twice - 4: Strong narrative pull, clear payoff - 3: Might lose people in the middle - 2: Hook is better than the content - 1: No reason to keep watching SHARE_SAVE_POTENTIAL (15%): Will viewers share this or save for later? - 5: "I need to text this to someone RIGHT NOW" - 4: Will send to specific friend who would care - 3: Might share if asked - 2: Unlikely to share - 1: No share impulse DEMAND_SIGNAL (15%): Is there evidence people care about this? - 5: High engagement on the specific detail (not just the theme) - 4: Strong Reddit/cross-source signals - 3: Moderate engagement - 2: Weak signals - 1: No evidence of demand VISUAL_POTENTIAL (10%): Can this be shown, not just told? - 5: Rich visual reveal (mechanism, cross-section, transformation) - 4: Good visual support available - 3: Talking head viable - 2: Hard to visualize - 1: Pure audio content EVERGREEN_POTENTIAL (10%): Will this work in 6 months? - 5: Permanent truth about how the world works - 4: Long-lasting relevance - 3: Moderate shelf life - 2: Tied to current events - 1: Expires quickly WEIGHTED SCORE = (scroll_stop * 0.25) + (completion * 0.25) + (share_save * 0.15) + (demand * 0.15) + (visual * 0.10) + (evergreen * 0.10) ═══════════════════════════════════════════════════════════ MAKE_NOW GATES (ALL must pass for MAKE_NOW verdict) ═══════════════════════════════════════════════════════════ Before giving a MAKE_NOW verdict, answer these 6 yes/no questions: 1. SCROLL_STOP_TEST: Does the premise contain a specific detail that creates an unresolved tension in under 3 seconds? (Not a theme — a detail) 2. UNIVERSAL_ACCESS_TEST: Would someone with zero interest in this topic still click? (Transcends subject matter) 3. DEPTH_TEST: Does the reveal require 60+ seconds to properly explain? (Not a fun fact — a video) 4. SHARE_TEST: Would someone text this premise to a friend unprompted? (Social currency) 5. SATISFACTION_TEST: Does the reveal leave the viewer feeling smarter, empowered, or validated — NOT anxious, guilty, or lectured? (Positive emotional payoff) 6. CHANNEL_FIT_TEST: Does this directly exploit this channel's specific itch and competitive gap? (Not just "good content" — good content FOR THIS CHANNEL) VERDICT RULES: - MAKE_NOW: All 6 gates pass. Weighted score 4.0+. Expected rate: 10-15% of clusters. - WORTH_MAKING: 4-5 gates pass. Weighted score 3.0-3.9. Expected rate: 30-40%. - NEEDS_RESEARCH: Hook is promising but facts need verification. Expected rate: 15-25%. - SKIP: Fewer than 4 gates pass OR no genuine hook exists. Expected rate: 25-35%. AUTOMATIC SKIP — Emotional Framing Guard: Assign SKIP verdict immediately if ANY of these apply: - Premise frames the viewer's existing behavior as pathological or broken - Viewer would feel WORSE about themselves after reading the title (not during — after) - Title uses shame as the hook mechanism (e.g., "You're Not Intense. You're Empty.") - Title suggests the viewer is fundamentally defective rather than curious ═══════════════════════════════════════════════════════════ OUTPUT FORMAT (JSON) ═══════════════════════════════════════════════════════════ Respond in JSON with these exact fields: { "hook_candidates": [ { "source_post_title": "exact Reddit post title", "hook_mechanism": "CURIOSITY_GAP|SELF_RECOGNITION|THREAT_INDIGNATION|CONSEQUENCE_CHAIN|NARRATIVE_SURPRISE", "the_detail": "the specific surprising detail in one sentence", "scroll_stop_strength": 1-5 } ], "best_hook_index": 0, "honest_assessment": "Is this cluster genuinely compelling or just analytically viable?", "premise": "the video title built around the specific detail", "first_frame": "what text/image appears in first frame", "trigger_map": { "mechanism": "which of the 5 mechanisms", "trigger_phrase": "the specific word/phrase/image", "why_irresistible": "why viewer can't scroll past" }, "opening_hook": "first 10 seconds spoken", "core_reveal": "the satisfying explanation", "depth_check": "can this sustain 90-120 seconds? why?", "emotional_payoff": "smarter|empowered|validated|relieved|amused", "target_audience": "specific moment when someone would watch this", "scores": { "scroll_stop_power": {"score": 0, "reasoning": "..."}, "completion_probability": {"score": 0, "reasoning": "..."}, "share_save_potential": {"score": 0, "reasoning": "..."}, "demand_signal": {"score": 0, "reasoning": "..."}, "visual_potential": {"score": 0, "reasoning": "..."}, "evergreen_potential": {"score": 0, "reasoning": "..."} }, "weighted_score": 0.0, "make_now_gates": { "scroll_stop_test": true|false, "universal_access_test": true|false, "depth_test": true|false, "share_test": true|false, "satisfaction_test": true|false, "channel_fit_test": true|false, "gates_passed": 0 }, "verdict": "MAKE_NOW|WORTH_MAKING|NEEDS_RESEARCH|SKIP", "verdict_reasoning": "one sentence on why", "source_post_title": "the specific post that inspired this concept", "suggested_title": "same as premise (backward compat)", "structure": "3-4 section outline for the video" })rrz--modelopusr-Ti,rz [ERROR] Claude CLI failed: drrr@ groq_scorechannelr,premisesuggested_titlererfverdict_validatedrryu → z (score: .1fz) - z( [WARN] Could not parse JSON responsethemei)r raw_response parse_errorrz [ERROR] Claude CLI timed outz [ERROR] ct|Srrrs r"rz&layer3_deep_analysis..As #9!#<r#rr}ct|Srrrs r"rz&layer3_deep_analysis..Ss ,B1,Er#z [3.X] Merge: z existing + u new/updated → z totalz [3.X] Generated z opportunities)rrtotal_opportunitiesr}rOz[3.X] Backup: rPrHrQz [3.X] Saved: v3)ranalysis_run_idr}prompt_versionz[3.X] DB: Inserted z opportunities (run_id: completed)ropportunities_foundstatusz[WARN] DB insert failed: failed)rr error_messager)1rzchannel_configrnamecompetitive_gapitch_descriptionformat_description hook_criteria descriptionrJr rrrr_rr`rarrLrrrrrrstdoutrOrIrPrrrrQrcr|rrrrrrrrrrbrrr )=rrrrchannel_context channel_namerrrrrrrrrrr}rrgroq_clsrrE all_titles titles_text reddit_textrrcommentsr-rr?rr;rArFrrhook_criteria_sectionrrrrIrr)weightedrvalidated_verdictrer scored_opps unscored_oppsrrsrmerged_opportunitiesexisting_count new_count final_countrropp_idss= r"layer3_deep_analysisrs  / ./ (OOLM.j) "<2yhll>6RyV^VbVbckVlypy  acU-L(9#2(>'?sCD07x/@A!a oA Aii: >a2aS >? 9 ;;~ &L^, E553gq)EE+y9 gr*55R0#6##d9+T% E7R^_g^hhi$jk ''+hZs(CD E))L1K))GI./$%9: ' ,CQ G&ll+;YG\\'1- !# )  % !   '()()b*#sjH +^^C# F  A%7 ds8K7LMN"MM//1M2 YYy-K #zz**:*:*<=H#zz-8H*1)>&*1-*@'&-i&8#189M1N-.2:,,?O2P./)1g)>&+7(*4HY'2(;-5)*(->h-N2:92EH./&h624H./$,<< 6#B $4X$>!$(88*;HY'48H013CH/0$$X.",,y)< %6 B3BG 8C.UG3OPs g+T ,E/?1/D1EKEOQ]^"EtT**?B?@N%&I*+K NQ. /|I;FWXcWddjkl ";-~>?!o.*-\\(,,/99; F,J7K {m,- k3 91 &!A<89 M+ '(/ 0% /2# G 'G ~5MoM^^_` a &$'(<$=" !0F  M[ G A!E F F G8B >f '' @B$$%\\'9=$1%4$8#&q6"),"7 & (( 7 4 6 + LQ) * * +FLR99*  -aS1 2#*#"%a& s8A]) ^ ^ A_+_+1E^  `9`99 `>`>aAa ]=$]88]= _(A_#_+#_((_++`6 `6`11`6a  b 'bb zhttp://127.0.0.1:8100ic|ddl} |jtdd}|jdk(S#t$rYywxYw)z)Check if the research service is running.rNz/healthr2rr'F)requestsrJRESEARCH_SERVICE_URL status_coder)rrs r"check_research_servicersH<<#7"8 @!<L##s** s (/ ;; opportunitychannel_description max_lengthc jt|jd|jdddd}t|jdddd}t|jd |jd ddd }|}|r |d vr |d |z }n|r |d vr|d |z }t||kDr |d|dz dz}|S)z Construct a web search query from Layer 3 output. Goes to Tavily (web search), so needs actual search terms, not instructions or creative titles. rrrNP core_revealrxthe_itch opening_hookr)rN/ANoner&r2r)rLrJr)rrrrrevealitchrs r"construct_research_queryr s+//)[__=NPY-Z[\]`^`aG 3 4Tc :F {z;??>2+NO PQURU VD E & 33 1VH $11 1TF 5zJozA~&. Lr#r thesiscNddl} d|i}|r||d<|jtd|d}|jdk(r|j j d St d |jd |jdd y#t$r}t d |Yd}~yd}~wwxYw)z@Submit an async research job. Returns job_id or None on failure.rNr r z/research/asyncr)rrr'job_idz& [ERROR] Research service returned : rz+ [ERROR] Failed to submit research job: ) rpostrrrrJrzrr)r r rpayloadrrs r"submit_research_jobrsE"  &GH ==#$O 4!    3 &==?&&x0 0 :8;O;O:PPRS[S`S`aebeSfRgh i  ;A3?@sAB(B B$ BB$rcPddl}ddl}|j} |j|z }|tkDrtd|dtdy |j t d|d}|j d k7rtd |j y|j}|j d }|d k(r|j d iS|dk(rtd|j ddy|jt#t$r}td|Yd}~yd}~wwxYw)zLPoll for research job completion. Returns result or None on failure/timeout.rNz [TIMEOUT] Research job z exceeded sz /research/rrr'z [ERROR] Poll failed: rrrrz" [FAILED] Research job failed: errorz Unknown errorz [ERROR] Poll error: ) rtimeRESEARCH_MAX_POLL_TIMErzrJrrrsleepRESEARCH_POLL_INTERVALr) rrr start_timeelapsedrdatarrs r"poll_research_jobrs,J ))+ * + + /xzBXAYYZ[ \ ||';&|s| jd?% t#|| d?d@| jd| jd| jd| jd | jd;| jd!dABd | d <dD| d<tdE|s| jd? t#|| d?dFdGdDiB| | z|d<t|||t'j(t*j,j/dH|dI<tdJ|d%|dKt1|}t3|dL5}t5j6||d"t8MdddtdN||S#t$$r}tdC|Yd}~d}~wwxYw#t$$rYwxYw#1swYNxYw)Oa Layer 3.5: Deep Research Integration. Automatically triggers deep research for the top N opportunities using the Hybrid Research Service API. Args: opportunities_data: Output from Layer 3 channel_id: Channel ID for context top_n: Number of top opportunities to research (default 5) skip_research: If True, skip research and just mark all as not researched Returns: Updated opportunities_data with research fields added r$z$LAYER 3.5: DEEP RESEARCH INTEGRATIONr%rr}z#[WARN] No opportunities to researchrrrz[3.5.0] Channel: z0[3.5.0] Skipping research (--skip-research flag)FrTresearch_skippedz)[WARN] Research service not available at zb Run: cd /home/sietch6/research-sandbox/option1-gpt-researcher && python research_service.pyz= Marking all opportunities as research_completed: falsezResearch service not availablerz&[3.5.1] Research service available at z[3.5.2] Will research top z opportunities (of z total)c&|jddS)Nr,rrqrs r"rz'layer35_deep_research..9saee,a0r#rNrrrrrrrHr1z [3.5.3] Processing batch r0rz topics)rrryz Submitting: rrr u → Job ID: zFailed to submit research jobz Polling: z (job z)...success report_pathreport_content word_count source_countduration_seconds generated_atru ✓ Completed: z words, z sourcesrr)r'r(r)r*rr+)rrresearch_status research_dataz* [WARN] DB update for research failed: z Research job failed or timed outu ✗ Failedrr)rresearched_counttop_n_requestedresearch_timestampresearch_statsz [3.5.4] Research complete: z successfulrPrQz[3.5.5] Saved: )rzrJrrrrrrrr^rranger rLrrrrrrr_rr`rarrrrbr)rrrr rr}rrrrr to_researchnot_researchedr batch_sizer. batch_startbatch batch_num total_batchesjobsr?rr rrrrsrs r"layer35_deep_researchr;s/* / 01 (O),,->?O&**?B?M  34!!.j) ")"5"5  %gll^4 5 @A  +C(-C$ %&*C" # +"! " # 9E N5'- .,S2EFECGG4Er,JKLF(v>F S&%01(12,1()(G$% H #'4  CGG-y9#2>E KwfVHD9 :&v.F&**Y/,0().4jj.K*+17PRS3T/0/5zz."/M+,(-$% A% +FJJ|Q,G+HQWQ[Q[\jlmQnPoowxy#sww|'<'HP>,;'*<'8,725'':S2T.1gg6K.L038O0P47GG,;'*<'8,4+24V*W [4 3M`+6*F'"=1, &ll8<<8BBD ,'( )*:);1UG; OP)4K k3 E1 $a|DE OK= )* I%P J1#NOOP % EEs7 A3S2/T<T'2 T;TT T$#T$'T0opportunity_idc |t|}|jstd|yt|d5}t j |}dddj dg}d}d}t|D]I\}} t| j d|t|k(st|t|k(sE| }|}n|td|dy|j d r td y d } |rd dl m } | |} | r | j} ts tdy|j dd} td| t|| }t|j d|j dd }t||}|sytd|t|}|r|j drd |d <|j dd |d<|j dd |d<|j dd |d<|j dd |d<|j dd |d <|j d!d |d"<||d#<|||<||d<t|d$5}t j ||d%&dddtd'|j dd d(y td)y#1swYZxYw#1swYBxYw)*a Manually trigger research for a specific opportunity. Args: opportunity_id: The ID of the opportunity to research channel_id: Optional channel ID for context Returns: True if research was successful, False otherwise z'[ERROR] No opportunities file found at FrNr}idz[ERROR] Opportunity z not foundrz1[INFO] Opportunity already has research completedTrrrz&[ERROR] Research service not availablerrz Researching: rr$zJob submitted: r%r&rr'rr(rr)rr*rr+rrrPrH)rRu✓ Research completed: z wordsu✗ Research failed)rrrzrrrrJrrLrrrrr rrrb)r<rrsrrr} target_opp target_idxidxrrrrr?rr rrs r"run_research_for_opportunityrBs)4K     7 }EF k3 1yy|HH_b1MJJm,S swwtS! "c.&9 9SX^I\=\JJ    $^$4J?@~~*+ AC.j) ")"5"5  " # 68 NN,i 8E M% !" $Z1D EE  :>>:KR+PQ RF v 6F  OF8 $% v &F &**Y'+/ '(-3ZZ r-J )*06 ;KR0P ,-,2JJ|Q,G ().4jj.K *+28**=OQR2S ./.4jj.L *+', #$%/ j! -_ +s # )q IIdAa ( ) (L!)D(EVLM #%M~ ) )sJ%J2%J/2J;max_lenc |yt|tr*dD]}||vst|||cSt|d|St|tr|rt|dtrtg}|ddD]T}|j d|j d|j dd }|s8|j t|dd V|rd j|d|Sd jd |ddDd|St|d|S)zFSafely convert a value to a truncated string, handling dicts and None.Nr)r.rrsummaryrformattypersectionr?rrrrc38K|]}t|ddyw)N)rL)rrvs r"ruzsafe_str..&s6Q6s)rrKsafe_strrLrZrJrr)rVrCrUsectionsrMr?s r"rNrNs {#t[ 3CczC'22 33x!!#t :c!fd+H!W 5iwfb8I)JKOOCJsO4 5||H-hw77yy6c"1g66x@@ s8HW r#c h tdtdtd|jdg}|s tdy|Dcgc]0}dt|jdd jvs/|2}}|Dcgc]0}d t|jdd jvs/|2}}|Dcgc]0}d t|jdd jvs/|2}}|Dcgc]0}d t|jdd jvs/|2}}t |}td tdt |dt ||z dzddtdt |dt ||z dzddtdt |dt ||z dzddtdt |dt ||z dzdd|rtdtdtdt |ddD]X\}} d| vrtd|dzdt | jdd} t | jd d!} | jd"d#} t| tr"| jd$| jd%d#} t| } t | jd'd(} t | jd)d}t | jd*d+}td|dzd,| d-| d.d/td0| td1| |r|d2k7rtd3|d4|sD|d2k7sKtd5|[|rtdtd6tdt |ddD]\}} d| vr t | jd d!} | jd"d#} t| tr"| jd$| jd%d#} t| } td7|dzd8| d.d9| |rtdtd:tdt |dd;D]\}} t | jd d} t | jd<d+}|d2k(rt | jd=d+}| jd"d#} t| tr"| jd$| jd%d#} t| } td7|dzd8| d.d9| td>||rtdtd tdt |dd;D]]\}} t | jd d} t | jd=d+}td7|dzd,| td?|_yycc}wcc}wcc}wcc}w#ttf$rd&} Y:wxYw#ttf$rd&} YwxYw#ttf$rd&} YwxYw)@z8Print a human-readable summary of deep analysis results.r$zDEEP ANALYSIS SUMMARYr%r}zNo opportunities analyzed.Nrgrerrxrrfz VERDICT BREAKDOWN (4 tiers):z MAKE_NOW: rrz.0fz%)z WORTH_MAKING: z NEEDS_RESEARCH: z SKIP: z= ------------------------------------------------------------z MAKE NOW (immediate opportunity)z<------------------------------------------------------------r2rz [r1z"] Parse error - raw response savedrryrFr,rr-r.rGsuggested_formatrr structurerz] z (Score: rz/5.0)z Title: z Format: rz Hook: rz Structure: zWORTH MAKING (no rush)rz] (z) zNEEDS RESEARCHr2research_neededverdict_reasoningz NEEDS: z Why: ) rzrJrLrrrrNrrKrrM TypeError)rr}rmake_now worth_makingneeds_researchskipr/rrrr?r-fmthookrSrTrs r" print_summaryr]*s / !" (O&**?B?M  *+)\aJ#aeeIr>R:S:Y:Y:[,[\H\,d!#aeeIWYFZB[BaBaBc0cAdLd!.hA2Bc!%%PY[]J^F_FeFeFg2gahNh$ T!#aeeIr6J2K2Q2Q2S(SA TD T  E *, s8}oRH e0CC0G/LB OP s<01C 4Ee4KC4OPS3TTV WX s>232c.6I%6OPS6STW5XXZ [\ s4ykCIeOC,?+DB GH o 01 h! - 5FAs$AaC5 BCDSWWW-r2ESWW%67|jn-td?|j"tjdtd0td@t?j@tBjDjGtItK tMd}|jPrtS}tdB|dCn&|jTr|jT}tdD|tdEtW|jX||j"F}tdGt[|dH|j"rtdI|j"dCi}|D],}|j]dJdK}|j]|ddz||<.tdL|t[|dMkr tdNtjdt_|}ta|}tc||j"|jd|jfO}ti||j"|jj|j8P}tm|tdQ dRdSg}|j"r|jod|j"gtqjr|dTdTdUV}|jtdk(r tdWntdX|jvtdZtd[td0td\td]txd^td]txd_td]t;|j"td]txd`tdatxdby#1swYKxYw#tN$r}tdA|Yd}~d}~wwxYw#tN$r}tdY|Yd}~d}~wwxYw)cz"Run the 3-layer analysis pipeline.rNz3-Layer Trend Analysis Pipeline)rz--since-last-run store_truez6Only analyze items from the most recent collection run)actionhelpz --since-runz+Only analyze items from runs >= this run ID)rGraz--limitz Maximum number of topics to loadz --channelz)Only analyze items tagged to this channelz--list-channelsz List available channels and exitz--top-nr1zFNumber of top opportunities to auto-research in Layer 3.5 (default: 1))rGrraz--skip-researchz!Skip Layer 3.5 deep research stepz --replacezTReplace entire opportunities file instead of merging (destructive, use with caution)z--research-onlyOPPORTUNITY_IDzARun research for a specific opportunity ID only (skip Layers 1-3))rGmetavarraz--max-clusterszILimit Layer 3 analysis to N clusters (0 = unlimited, for testing use 3-5)z--generate-videosuIRun Layers 4a→4b→4c video generation for all researched opportunitiesz--generate-videoIDuERun Layers 4a→4b→4c for a specific opportunity by index (0-based)z--prompt-versionv1z1Prompt version for video generation (default: v1))get_enabled_channelsget_channel_statszAvailable channels:z rz Items tagged: item_count last_analysisz Last analysis: r%zMANUAL RESEARCH TRIGGERr)&run_video_generation_for_opportunitiesget_latest_opportunities_filesave_opportunities_timestampedz1[ERROR] No opportunities file found for channel 'z'. Run analysis first.z[INFO] Loading from: r)rr specific_idrr z [OK] Video generation complete.z Saved: z Latest: zTREND ANALYSIS PIPELINErz Channel: z[ERROR] Unknown channel: z Started: z[WARN] Migration check failed: z0 [MODE] Analyzing only from latest run (run_id: rwz! [MODE] Analyzing from run_id >= z$ [0] Loading trends from database...)rr rz Loaded z unique topicsz (filtered to channel: rrz Signal types: r)zF[ERROR] Not enough data to analyze. Run the collection pipeline first.)rrr)rrr z! [4] Generating dashboard data...python3zgenerate_dashboard_data.pyTr)rrrz' Dashboard data updated successfullyz* Warning: Dashboard generation failed: z0 Warning: Could not generate dashboard data: r$zANALYSIS COMPLETEz Output files:z - z/analysis_clusters.jsonz/analysis_classified.jsonz/dashboard_data.jsonz Dashboard: file://z/dashboard.html)=argparseArgumentParser add_argumentrrL parse_args list_channelsrrfrgrzrRrrr research_onlyrBrgenerate_videosgenerate_videovideo_generationrjrkrlrrrrr rrrr_rr`rarrr rsince_last_runr since_runr!rrrJrtrrrreplacer;rr]extendrrrrr)roparserargsrfrgch_idchstatsr%rjrkrl input_pathrr saved_pathrrr r signal_countsrEstrurcmdrs r"mainrs2  $ $1R $ SF  E    :    /    8   /    U    0   c    P     X    X    T     @     D J #$-/557 FIE2%e,E BugRy) * &u\':&;< =_%+E/,B+CDE  F    h '( h.t/A/Adll[ g1% t22>  34<<@  Edll^Sij k HHQK %j\23 *c " .a!%1  .D1||++..,,  44F U  13 ZL)* 4T\\BCDE   (O #$ ||.  & IbggY' ( -dll^< = HHQK (O Ihll8<<0::<= >?5 L &(  A,qQR ~~  2<.AB 12 jj!<<F  KF }N 34 || *4<<.:;M 9 UU=) ,)--b!4q8 b9 }o ./ 6{R VW  -V4M1?O.<<&&\\ /<<jj(( $% ./F67 << JJ T\\2 3       ! ; < >v}}oN O /  (O / D 3 45 D 5 67 D' 56 78 D 0 12  O <=} . .P 5 /s3445V F @DEEFs=\1& \>A9]#1\;> ] ]] # ^,]??^__main__r)F)NNN)NrF)i|)Nr2F)r)?r'rrrrrrrpathlibrtypingrrrr numpyrdotenvr db_opportunitiesr r r rrr JSONEncoderrrLrrcr|r__file__parentrrrrrboolrrrrrr!rtrrrrrrr rrr;rBrNr]rr$rIr#r"rs   ',, $4## $@T#s(^@@F"tCH~"#"J6#4:6#$t*6#t (^ " "X - * * 6s 6d 6 #$*;C;4S>;(LS>LDcN+LL $sCx. L^*(8)#)Q QQQ $sCx. QpET$sCx.%9Ed38nEX@d38n@c3h@NH$sCx.HcH`cHw{HIMNQSVNVIWH^/$sCx.s`cnq0sC8C=.!c!htCH~&>!L xS#Xxx x x  #s(^ xvWW#WQUW|34f+d38nf+Rn>b zFr#