Fine‑tuning large language models for financial NLP is hampered by limited, imbalanced real‑world data: earnings and stock movement stories dominate, while rarer events such as credit‑rating changes, product approvals, and labor issues are underrepresented. A single large synthetic generation run produces many near‑duplicates, so the authors built an iterative pipeline that produced 502,536 unique financial news headlines across 12 topics plus an "Other" category.
What the workflow does
The pipeline combines NVIDIA NeMo Data Designer for structured generation, NVIDIA NeMo Curator for scalable semantic deduplication, and NVIDIA Nemotron models for high‑throughput headline synthesis. Generation is served by vLLM and deduplication uses sentence‑transformers embeddings (all‑MiniLM‑L6‑v2). A central steering mechanism is selecting few‑shot examples that are farthest from their cluster centroids each iteration, which helps guide new generations toward novel content.
Reproducibility — versions and hardware
Software and model versions used in the guide:
- NeMo Curator: 1.0.0rc0.dev0
- NeMo Data Designer: 0.1.5
- vLLM: 0.12.0
- Model: nvidia/NVIDIA‑Nemotron‑3‑Nano‑30B‑A3B‑FP8
- sentence‑transformers: >= 2.2.0
Hardware: a single 8‑way NVIDIA B200 node. GPUs 0–3 handled vLLM inference (4‑way tensor parallelism, 448 concurrent requests); GPUs 4–7 ran NeMo Curator with Ray. The end‑to‑end run required roughly six days of compute with checkpointing, crash recovery, and SLURM job chaining.
Pipeline overview
A single pass at large scale fails because real feeds are imbalanced: an initial naive 50,000‑headline run retained only 17,348 unique headlines after semantic deduplication. The iterative pipeline works as a closed loop: generate a category‑weighted batch, filter malformed outputs, deduplicate against the full accumulated corpus, select diverse few‑shot examples, adjust category weights, and repeat until the target corpus size is reached.
A key design decision is global deduplication: each batch is compared with all previously retained headlines, preventing cross‑batch duplicates and preserving semantic uniqueness across the target corpus.
Step 1 — Generate with NeMo Data Designer and Nemotron
The authors serve Nemotron‑3‑Nano (30B parameters, 3B active) via vLLM with 4‑way tensor parallelism and 448 concurrent requests. NeMo Data Designer orchestrates generation via a declarative config: a weighted category sampler (12 topics + Other) and an LLM text column that generates headlines conditioned on the sampled category and current few‑shot examples.
Prompts include a system role prompt ("You are a financial news headline writer."), few‑shot examples grouped by category (refreshed each iteration), and instructions to produce a single 10–25 word professional headline for the sampled category. Sampling favors diversity (temperature=0.95, top_p=0.95) while downstream steps remove low‑quality or redundant outputs.
Step 2 — Rule‑based quality filtering
Before semantic deduplication, a lightweight rule pass discards malformed outputs. Filters and thresholds included:
- Minimum words: >= 5
- Minimum characters: >= 25
- Maximum words: <= 100
- Maximum characters: <= 1,000
- Bracket count: < 5
- Special character ratio: <= 25%
These filters removed under 1% of headlines per batch in practice; semantic deduplication does the heavy lifting.
Step 3 — Semantic deduplication with NeMo Curator
After each batch, new headlines are concatenated with the accumulated corpus and processed by NeMo Curator's TextSemanticDeduplicationWorkflow. Headlines are embedded with sentence‑transformers/all‑MiniLM‑L6‑v2, clustered with K‑means, and pairwise cosine similarities are computed within clusters. Any headline above the 90% similarity threshold is removed; among duplicates the most representative (closest to centroid) copy is kept.
The workflow used 500 clusters. At the ~500K endpoint, 500 clusters keep average cluster size near ~1,000 and total pairwise comparisons near ~500 million; far fewer clusters would dramatically increase the quadratic comparison cost.
Monitoring included ECDF plots of pairwise cosine similarities per iteration: early iterations show right‑skewed curves (many near‑duplicates), while later iterations shift left and smooth, indicating broader coverage as few‑shot prompts evolve.
Step 4 — Extract diverse few‑shot examples (farthest‑from‑centroid)
Post‑deduplication, the pipeline selects few‑shot examples to steer the next generation. Using K‑means cluster centroids, headlines are ranked by cosine distance from their centroid and the farthest examples (most atypical) are chosen. Three examples per category are selected (39 total per iteration).
A cross‑iteration semantic filter rejects any candidate >= 80% cosine similarity to any previously used few‑shot example to avoid repeating prompt signals across iterations. If needed, fallbacks include a random sample from the category pool or seed manual examples.
Selected examples gradually become more specific over the run, moving from generic headlines to ones with concrete entities, niche instruments, and longer structures by iteration 82.
Step 5 — Dynamic distribution correction
Weighted sampling alone does not keep category proportions on target because the model prefers easier classes (e.g., Other, Stock Movement). After each iteration the pipeline compares target vs. actual category proportions, boosts underrepresented categories (with clamping to avoid extremes), and normalizes weights for the next batch. Missing categories receive a stronger boost. By iteration 82, rare categories such as Credit Ratings and Product Approval approached their ~1% targets, though Other stayed overrepresented.
Results
Run summary:
- Final unique headlines: 502,536 (target 500,000; excess 2,536 trimmed from Other)
- Iterations: 82
- Batch size: 35,000 per iteration (early iterations used 50,000)
- Total generated pre‑deduplication: ~2.87 million
- Cumulative deduplication rate: ~82%
- Net new per iteration (late run): ~5,000–6,000
- Categories: 12 topics + Other
- Similarity threshold: 90% cosine
- Few‑shot examples per iteration: 39 (3 per category)
Deduplication behavior: in iteration 1 a 50K batch yielded 17,348 retained headlines (~35% yield); 65% were near‑duplicates. Iterations 2–3 produced 15K–17K new headlines per 50K batch after adopting farthest‑from‑centroid examples. Once the deduplication drop rate passed ~80%, batch size was reduced from 50K to 35K, which saved compute while keeping net new yield similar. After ~100K corpus size, yields stabilized at ~15% per 35K batch (~5–6K new headlines).
Downstream application: model distillation
The FinHeadlineMix dataset supports model distillation. In a developer example, a 49B–70B teacher model labeled synthetic headlines and smaller 1B, 3B, and 8B student models were fine‑tuned with LoRA. With 25K labeled examples, a 3B student reached 95% of the teacher's F1 on the headline classification task. The authors note that semantic uniqueness and balanced categories are critical for exposing students to rare events and edge cases.
Practical takeaways
Four operational principles emerged:
- Iterate rather than one large batch — many duplicates result from a single pass.
- Deduplicate globally across the accumulated corpus to avoid cross‑batch collisions.
- Few‑shot examples steer generation; farthest‑from‑centroid selection plus cross‑iteration filtering is highly effective.
- Dynamic distribution correction is required to keep rare categories represented.
Getting started and resources
The FinHeadlineMix dataset is available on Hugging Face for direct use in classification, fine‑tuning, or distillation workflows. The recipe and components can be reproduced or extended using NeMo Data Designer for structured synthetic generation and NeMo Curator for scalable semantic deduplication. Nemotron‑3‑Nano‑30B‑A3B (the MoE model used) is approved by the authors for distillation workflows.
The authors provide this pipeline and dataset as a practical example for building license‑compliant synthetic data pipelines, scalable semantic deduplication, and efficient financial data workflows using model distillation.
