Research pipeline

LLM call

Function

suggestResearchPlanWithLlm

Model

planner

Params

temp 0.15, 6k tokens, 180s provider / 300s outer

Schema

structured_research_plan

Output

research_plan_v2 — objective, collection unit, query plan, coverage lanes, source priorities, discovery channels, extraction targets, success criteria, follow-up rules, stop conditions.

What happens

• Fetches same-customer library context as routing signal (not factual proof)
• Deduplicates known fresh URLs out of new acquisition
• Builds research contract binding the run to plan + customer intent

Tools

Provider

Postgres — customer library tables

Query

Same-customer packs, sources, extractions by topic similarity

Output

library_recall_v1, research_contract_v1

Search

Primary

Exa — semantic search with topic, instructions, source policy, coverage lanes, query plan

Fallback

Firecrawl search

Budget

Planned queries + reserved refinement budget by depth

LLM — Candidate triage

Function

triageCandidatesWithLlm

Model

triage

Params

temp 0.12, 3k tokens, 90s

Schema

candidate_triage

Failure

Skip; deterministic selection continues

Scrape

Provider

Firecrawl

Output

Normalized SourceDocument — markdown, URL, domain, media type, metadata, hash, quality score

Dedup

Content hash + near-duplicate demotion

Video transcripts

Provider

Supadata API

Routing

YouTube URLs split from scrape candidates via canTranscribe()

Modes

native | auto | generate

Metadata

oEmbed — title, channel, thumbnail, duration

LLM — Speaker labeling

Function

restructureTranscript

Model

analyst

Method

36-turn chunks → infer speaker labels (Host, Guest, named)

Gate

55% labeled turns required; graceful fallback

Video artifacts

• video_transcript_v1 — segments with speaker labels, timestamps, confidence
• video_quotes_v1 — high-signal quotes, concept inference, entity mentions
• video_metadata_v1 — title, channel, thumbnail, duration
• 30–90s readable transcript windows for synthesis

No pre-filtering

Every acquired source goes directly to per-source extraction. There is no portfolio selection or deterministic ranking step — the extraction LLM sees the full content of each source and makes its own relevance judgment. Sources that are junk, off-topic, or too thin are marked ignored by the extraction LLM and excluded from the corpus. This eliminates the single-point-of-failure risk of a large batch triage call and ensures no source is silently discarded by topic-blind heuristics.

LLM — Source extraction

Function

extractSourceWithLlm

Model

extraction

Params

temp 0.1, 9k tokens, 120s

Schema

source_extraction

Sections

48 for transcripts, 18 for web

Reuse

Same-customer extractions by content hash or canonical URL

LLM — Extraction audit

Function

auditSourceExtractionWithLlm

Model

extraction

When

Per accepted extraction

Failure

Keep original if audit unavailable

Output

source_extraction_v1 — claims, facts, dates, evidence, classification.
source_analysis_record_v1 — durable per-source record for corpus metasynthesis.

LLM — Evidence coverage

Function

reviewEvidenceCoverageWithLlm

Model

coverage

Params

temp 0.1, 4k tokens, 120s provider / 150s outer

Schema

evidence_coverage_review

Failure

Deterministic coverage review baseline

Follow-up loop

If lanes are weak → targeted follow-up queries → reacquire + extract while loop budget remains (1 / 3 / 6 by depth).

Output

research_state_v1 — loop, phase, source counts, gap counts, next queries, history.

Stage 0 — Foundation

Deterministic aggregation of raw evidence from all extractions into structured candidate lists (claims, chronology, source rankings). Data wrangling that seeds LLM passes, never published as final output.

Stage 1 — Pattern Identification (4 parallel LLM passes)

1A: Claim consolidation

Semantic dedup + theme/pattern detection across all claims

1B: Chronology

Causal sequencing — what led to what, not just date sorting

1C: Comparisons

Analytical comparison tables that reveal insight

1D: Media concepts

Transcript runs only — named ideas, frameworks, models

Model

corpus

Schemas

consolidated_claims, enriched_chronology, comparison_tables

Stage 2 — Insight Synthesis (1 sequential LLM pass)

Input

Claim patterns, chronology, comparisons, deliverables, coverage lanes — NOT raw source data

Output

Competing theses with evidence threads, deliverable mappings, tensions, coverage gaps

Schema

insight_synthesis

Failure

Critical — retries 3 times, fails merge if exhausted

Stage 3 — Evidence Mapping (1 sequential LLM pass)

Input

Theses, claim patterns, source metadata

Output

Best sources mapped to theses, contradictions as analytical tensions

Schema

evidence_mapping

Failure

Non-critical — omitted on failure

Output

corpus_analysis_v1 — competing theses, consolidated claims with evidence threads, causal chronology, analytical comparisons, contradictions as tensions, best sources mapped to theses, coverage gaps, media concepts.

LLM — Research quality

Function

reviewResearchQualityWithLlm

Model

quality

Params

temp 0.1, 4k tokens, 120s provider / 150s outer

Schema

research_quality_review

Failure

Merges with deterministic baseline

Quality loop

High-severity issues → targeted acquisition → extraction → evidence review → corpus merge → re-evaluation. Stops on budget exhaustion, source cap, synthesis reserve depletion, or stagnation.

Output

research_quality_review_v1

LLM — Structured synthesis

Function

synthesizePackWithLlm

Model

synthesis

Params

temp 0.1, 7k tokens, 180s / 210s outer

Schema

research_pack_synthesis

LLM — Report writer

Function

writeResearchReportWithLlm

Model

synthesis, fallback gpt-4.1

Params

temp 0.2, 24k tokens, 300s

Format

Markdown text

LLM — Report contract review

Function

reviewResearchReportWithLlm

Model

synthesis

Params

temp 0.05, 4k tokens, 150s

Schema

report_contract_review

Rewrite flow

Medium/high issues → one rewrite (240s) → revised review (180s). Must pass: no high issues, score ≥ 0.75.

Output

research_report_markdown_v1, agent_brief_markdown_v1, claim_evidence_map_v1

Steps

1. Build cluster index + record index for Run Analyst retrieval
2. Generate action manifest from synthesis
3. Persist all artifacts to Postgres
4. Settle billing — close MPP budget session, record final usage
5. Mark job completed

LLM — Persona

Function

generateRunPersona

Model

analyst

Failure

Best-effort; ignored

Run Analyst

Function

answerRunAnalystQuestion

Model

analyst, fallback extraction

Scope

Run-scoped retrieval only — returns gaps instead of generic web

Final artifacts

cluster_index_v1, record_index_v1, action_manifest_v1