Tech News Digest

Automated tech news digest system with unified data source model, quality scoring pipeline, and template-based output generation.

Quick Start

1. Configuration Setup: Default configs are in config/defaults/. Copy to workspace for customization:

   mkdir -p workspace/config
   cp config/defaults/sources.json workspace/config/tech-news-digest-sources.json
   cp config/defaults/topics.json workspace/config/tech-news-digest-topics.json
   

2. Environment Variables:

   • TWITTERAPI_IO_KEY - twitterapi.io API key (optional, preferred)
   • X_BEARER_TOKEN - Twitter/X official API bearer token (optional, fallback)
   • TAVILY_API_KEY - Tavily Search API key, alternative to Brave (optional)
   • WEB_SEARCH_BACKEND - Web search backend: auto|brave|tavily (optional, default: auto)
   • BRAVE_API_KEYS - Brave Search API keys, comma-separated for rotation (optional)
   • BRAVE_API_KEY - Single Brave key fallback (optional)
   • GITHUB_TOKEN - GitHub personal access token (optional, improves rate limits)

3. Generate Digest:

   # Unified pipeline (recommended) — runs all 6 sources in parallel + merge
   python3 scripts/run-pipeline.py \
     --defaults config/defaults \
     --config workspace/config \
     --hours 48 --freshness pd \
     --archive-dir workspace/archive/tech-news-digest/ \
     --output /tmp/td-merged.json --verbose --force
   

4. Use Templates: Apply Discord, email, or PDF templates to merged output

Configuration Files

sources.json - Unified Data Sources

{
  "sources": [
    {
      "id": "openai-rss",
      "type": "rss",
      "name": "OpenAI Blog",
      "url": "https://openai.com/blog/rss.xml",
      "enabled": true,
      "priority": true,
      "topics": ["llm", "ai-agent"],
      "note": "Official OpenAI updates"
    },
    {
      "id": "sama-twitter",
      "type": "twitter", 
      "name": "Sam Altman",
      "handle": "sama",
      "enabled": true,
      "priority": true,
      "topics": ["llm", "frontier-tech"],
      "note": "OpenAI CEO"
    }
  ]
}

topics.json - Enhanced Topic Definitions

{
  "topics": [
    {
      "id": "llm",
      "emoji": "🧠",
      "label": "LLM / Large Models",
      "description": "Large Language Models, foundation models, breakthroughs",
      "search": {
        "queries": ["LLM latest news", "large language model breakthroughs"],
        "must_include": ["LLM", "large language model", "foundation model"],
        "exclude": ["tutorial", "beginner guide"]
      },
      "display": {
        "max_items": 8,
        "style": "detailed"
      }
    }
  ]
}

Scripts Pipeline

run-pipeline.py - Unified Pipeline (Recommended)

python3 scripts/run-pipeline.py \
  --defaults config/defaults [--config CONFIG_DIR] \
  --hours 48 --freshness pd \
  --archive-dir workspace/archive/tech-news-digest/ \
  --output /tmp/td-merged.json --verbose --force

• Features: Runs all 6 fetch steps in parallel, then merges + deduplicates + scores
• Output: Final merged JSON ready for report generation (~30s total)
• Metadata: Saves per-step timing and counts to *.meta.json
• GitHub Auth: Auto-generates GitHub App token if $GITHUB_TOKEN not set
• Fallback: If this fails, run individual scripts below

Individual Scripts (Fallback)

fetch-rss.py - RSS Feed Fetcher

python3 scripts/fetch-rss.py [--defaults DIR] [--config DIR] [--hours 48] [--output FILE] [--verbose]

• Parallel fetching (10 workers), retry with backoff, feedparser + regex fallback
• Timeout: 30s per feed, ETag/Last-Modified caching

fetch-twitter.py - Twitter/X KOL Monitor

python3 scripts/fetch-twitter.py [--defaults DIR] [--config DIR] [--hours 48] [--output FILE] [--backend auto|official|twitterapiio]

• Backend auto-detection: uses twitterapi.io if TWITTERAPI_IO_KEY set, else official X API v2 if X_BEARER_TOKEN set
• Rate limit handling, engagement metrics, retry with backoff

fetch-web.py - Web Search Engine

python3 scripts/fetch-web.py [--defaults DIR] [--config DIR] [--freshness pd] [--output FILE]

• Auto-detects Brave API rate limit: paid plans → parallel queries, free → sequential
• Without API: generates search interface for agents

fetch-github.py - GitHub Releases Monitor

python3 scripts/fetch-github.py [--defaults DIR] [--config DIR] [--hours 168] [--output FILE]

• Parallel fetching (10 workers), 30s timeout
• Auth priority: $GITHUB_TOKEN → GitHub App auto-generate → gh CLI → unauthenticated (60 req/hr)

fetch-github.py --trending - GitHub Trending Repos

python3 scripts/fetch-github.py --trending [--hours 48] [--output FILE] [--verbose]

• Searches GitHub API for trending repos across 4 topics (LLM, AI Agent, Crypto, Frontier Tech)
• Quality scoring: base 5 + daily_stars_est / 10, max 15

fetch-reddit.py - Reddit Posts Fetcher

python3 scripts/fetch-reddit.py [--defaults DIR] [--config DIR] [--hours 48] [--output FILE]

• Parallel fetching (4 workers), public JSON API (no auth required)
• 13 subreddits with score filtering

enrich-articles.py - Article Full-Text Enrichment

python3 scripts/enrich-articles.py --input merged.json --output enriched.json [--min-score 10] [--max-articles 15] [--verbose]

• Fetches full article text for high-scoring articles
• Cloudflare Markdown for Agents (preferred) → HTML extraction (fallback) → Skip (paywalled/social)
• Blog domain whitelist with lower score threshold (≥3)
• Parallel fetching (5 workers, 10s timeout)

merge-sources.py - Quality Scoring & Deduplication

python3 scripts/merge-sources.py --rss FILE --twitter FILE --web FILE --github FILE --reddit FILE

• Quality scoring, title similarity dedup (85%), previous digest penalty
• Output: topic-grouped articles sorted by score

validate-config.py - Configuration Validator

python3 scripts/validate-config.py [--defaults DIR] [--config DIR] [--verbose]

• JSON schema validation, topic reference checks, duplicate ID detection

generate-pdf.py - PDF Report Generator

python3 scripts/generate-pdf.py --input report.md --output digest.pdf [--verbose]

• Converts markdown digest to styled A4 PDF with Chinese typography (Noto Sans CJK SC)
• Emoji icons, page headers/footers, blue accent theme. Requires weasyprint.

sanitize-html.py - Safe HTML Email Converter

python3 scripts/sanitize-html.py --input report.md --output email.html [--verbose]

• Converts markdown to XSS-safe HTML email with inline CSS
• URL whitelist (http/https only), HTML-escaped text content

source-health.py - Source Health Monitor

python3 scripts/source-health.py --rss FILE --twitter FILE --github FILE --reddit FILE --web FILE [--verbose]

• Tracks per-source success/failure history over 7 days
• Reports unhealthy sources (>50% failure rate)

summarize-merged.py - Merged Data Summary

python3 scripts/summarize-merged.py --input merged.json [--top N] [--topic TOPIC]

• Human-readable summary of merged data for LLM consumption
• Shows top articles per topic with scores and metrics

User Customization

Workspace Configuration Override

Place custom configs in workspace/config/ to override defaults:

• Sources: Append new sources, disable defaults with "enabled": false
• Topics: Override topic definitions, search queries, display settings
• Merge Logic:
  • Sources with same id → user version takes precedence
  • Sources with new id → appended to defaults
  • Topics with same id → user version completely replaces default

Example Workspace Override

// workspace/config/tech-news-digest-sources.json
{
  "sources": [
    {
      "id": "simonwillison-rss",
      "enabled": false,
      "note"