Summarizing New arXiv Papers with a 0.5B Model
The first experiment is a catalog of new AI papers from arXiv. Every paper gets a short summary written by a small model running locally. The summary is checked against the paper, and the paper is compared with other recent papers so you can see whether something similar was already written. The results are on Paper Radar, which grows as new papers are added.
The point isn’t the catalog itself. It’s a way to test how far a very small model can get on a real task, and what it takes to check its work.
Hardware and models
- Machine: Mac mini, Apple M1 with 8 GB of unified memory.
- Runtime: MLX and
mlx-lm. Everything runs locally; no API calls. - Small model: Qwen2.5-0.5B-Instruct, 4-bit quantized (
mlx-community/Qwen2.5-0.5B-Instruct-4bit, about 280 MB). - Larger model: Qwen2.5-3B-Instruct, 4-bit. Used as a teacher during training and as a fallback in the pipeline.
- Embeddings:
sentence-transformers/all-MiniLM-L6-v2, 384 dimensions. - PDF text extraction: PyMuPDF.
The pipeline
Each run does the following for every paper:
- Fetch. New cs.AI papers come from the arXiv API, newest first. The default run takes 20.
- Read the paper. The PDF is downloaded and converted to text. The model sees only the first 10,000 characters, about 2,850 tokens. That usually covers the abstract, introduction and the start of the method section, but the median paper is about 74,000 characters, so the model reads roughly 13% of it. The cutoff was chosen, not measured. (See the limitations below.)
- Summarize. The 0.5B model, with a LoRA adapter (described below), writes a 3–5 sentence summary covering the problem, what’s proposed, and what’s found. Temperature 0.3, at most 200 tokens.
- Check. A faithfulness check pulls every number and proper noun out of the summary and confirms each one appears in the full paper text, not just the part the model saw. Spelled-out numbers count (“ten” matches “10”). Matching is whole-word, so “four” doesn’t match “fourteen”. The summary passes only if every item is found.
- Fall back. If the summary fails, the 3B model gets one try. Its prompt names the items that weren’t found and asks for a new summary that only uses names and numbers from the paper. The better-scoring of the two summaries is kept.
- Withhold. A summary that still fails isn’t shown. The paper is still listed, with a link to its abstract on arXiv.
- Find similar papers. The paper’s title and abstract are embedded and compared with a store of recent arXiv abstracts. The three closest are listed. (Details below.)
- Publish. Each paper is written as a Markdown file with frontmatter, then copied into this site.
Training the small model: distillation
Out of the box, the 0.5B model wrote summaries that often weren’t faithful to the paper. The 3B model wrote better ones, but it’s about four times slower. The fix was distillation: have the 3B write summaries and train the 0.5B to copy them.
- Data: 1,165 recent arXiv papers. For each one, the 3B model summarized the same 10,000-character text the 0.5B would see.
- Filter: a pair was kept only if the 3B’s summary scored at least 0.9 on the faithfulness check and was at least 100 characters long. 1,000 pairs were kept and 165 dropped.
- Split: 900 training, 50 validation, 50 test.
- Training: a LoRA adapter on the top 8 of the model’s 24 transformer blocks. That’s 1.47M trainable parameters, 0.3% of the model. Batch size 1, learning rate 1e-4, max sequence length 2,560 tokens, 1,350 iterations.
- Result: best validation loss 0.926, at iteration 800.
- Time: building the dataset took about 7.3 hours overnight. Training ran under memory caps and low CPU priority so the Mac stayed usable.
Comparison on six papers the adapter hadn’t seen:
| Model | Faithfulness | Time per paper |
|---|---|---|
| 0.5B, no adapter | 0.878 | 8.2 s |
| 0.5B + distillation adapter | 0.944 | 3.7 s |
| 3B (the teacher) | 0.979 | 14.2 s |
The adapter closed most of the gap to the 3B at a quarter of the 3B’s time. Six papers is a small sample, so treat the differences as rough.
First catalog run
The run published on the summaries page covered 20 cs.AI papers submitted on Nov 10, 2026:
- Passed the check: 19 of 20, 18 of them on the 0.5B’s first try.
- Fell back to the 3B: 2. The 3B’s summary passed for one (“Quantifying the Sampling Frequency Invariance…”). For the other (“Precision on Demand”) it did no better, so that summary is withheld. The only item the check couldn’t find was “95%”, likely a formatting difference in the extracted PDF text. But the summary had real problems the check missed: it’s cut off mid-sentence, and it says the method uses the top 4 bits of the output logits when the paper uses the top 4 bits of the weights.
- Total time: about 80 seconds for all 20.
Finding similar papers
A summary says what a paper does. It doesn’t say whether the idea is new. To give some sense of that, each paper is compared with other recent papers using embeddings, a form of retrieval-augmented lookup (RAG):
- Corpus: title and abstract metadata for cs.AI, cs.LG and cs.CL papers, pulled from arXiv’s bulk metadata feed (OAI-PMH). The first harvest covered papers announced since Nov 2, 2026: 9,087 papers, embedded in 78 seconds. It includes a few hundred older papers whose records were updated that week.
- Lookup: each catalog paper’s title + abstract is embedded the same way, and the three most similar papers submitted on or before its date are listed, with a cosine similarity score.
- Updates: re-running the harvest adds only papers announced since the last run, so the corpus grows a little each day.
The scores so far range from 0.46 to 0.73. Some matches are good: for “ReTrace,” which turns logged agent runs into fixed workflows, one match is “How Much Skill Orchestration Can LLM Agents Compile Away?”, a closely related idea. Others only share a topic. A high score means similar topic and wording, not the same idea.
What didn’t work
Several approaches were tried and dropped before the pipeline above:
- A one-sentence summary model. The first LoRA was trained on SciTLDR, a dataset of one-sentence paper summaries. It learned to write one sentence, about 22 tokens. Too short to be useful.
- Summarizing abstracts only. For the first two days the pipeline summarized the abstract, not the paper, because abstracts were easier to train on. An abstract is already a summary, so that tested very little.
- The wrong full-paper dataset. A LoRA trained on a public paper-to-abstract dataset (ccdv/arxiv-summarization) produced fluent nonsense. That dataset’s text is lowercased and has citations replaced with placeholders, so the model learned to write like that. The faithfulness check passed some of this nonsense.
- A 0.5B model as a judge. A 0.5B model was trained on 1,000 labeled pairs to answer “is this summary faithful?” It answered “yes” to everything. The rule-based check stayed.
- A model as a novelty judge. The 3B model was asked whether an earlier paper already did the same thing as a new one. It never said “same” and missed overlaps that were clear on reading. Similar papers are listed without a verdict instead.
- Summarizing the whole paper in chunks. The paper was split into ~7,000-character pieces (15 on average), each piece summarized in one sentence, and those sentences combined into the final summary. On the 20 catalog papers it did reach later sections (12 of 20 summaries used something past the first 10,000 characters, versus none before), but it was worse overall: 17 of 20 passed the check instead of 19, it took 22.5 seconds per paper instead of about 4, and it summarized reference lists and appendices as if they were content. One summary built a “key innovation” out of the reference list; another opened by calling the paper “a precise scientific summarizer,” wording from the model’s own instructions.
- Training crashes. Three training runs failed with the loss going to NaN. The cause each time was estimating text length too loosely, so some examples were longer than the training limit.
Limitations
- Document size is unsolved. The summary is written from the first ~13% of a typical paper, so results and conclusions are often missing. Reading the whole paper in chunks made things worse (above). Better options, such as cutting references and appendices first or adding the conclusion section, haven’t been tried yet.
- The check is narrow. It confirms that names and numbers appear in the paper. It doesn’t check that they’re used correctly, or catch a claim that’s negated or attributed to the wrong thing. A summary can pass and still be wrong.
- Small samples. The model comparisons above use 6 to 20 papers.
- A short similarity window. Only papers since Nov 2, 2026 are searched, so older related work won’t show up and a paper can look newer than it is.
- Coverage. Around 400–500 AI papers appear on arXiv each weekday. The catalog summarizes 20 per run.
Next
- Grow the similarity corpus each day, and possibly backfill to the start of 2026.
- Try an embedding model trained on scientific papers (SPECTER2) and compare its matches with MiniLM’s.
- Hand-label a set of papers as worth reading or not, to measure whether the summaries and similarity scores help pick them. That labeled set is the kind of validation dataset the series is about.