The release is a model, two datasets, and the receipts

COPA published three connected artifacts. ArmWeb has 4.37 million Armenian news documents, or about 3.3 billion Gemma-4 tokens. ArmSTEM has 373,000 parallel English-Armenian mathematics and science problems. The arm-gemma-e4b model is Gemma-4-E4B after 10 billion continued-pretraining tokens.

Its training mixture is 69% ArmWeb, 6% ArmSTEM in both languages, 20% English web replay, and 5% code. COPA released the data, weights, evaluation configurations, logs, and pipeline code. A reader can trace a score back to the corpus and recipe instead of accepting an unexplained leaderboard row.

arm-gemma-e4b scored 0.50 on the six-task likelihood suite. The unadapted base scored 0.48. Three earlier open Armenian models ranged from 0.35 to 0.47. Each checkpoint came from one training run, not an average across seeds, and the paper estimates a 95% half-width of plus or minus 0.010 on the mean. The result is evidence for this recipe, not a universal model ranking.

The mean also needs unpacking. ARC-hy and HellaSwag-hy stayed near four-way chance for every model, while MultiBLiMP was near its ceiling. m-MMLU-hy, INCLUDE-Armenian, and Belebele supplied most of the useful separation.

News improved fluency and damaged knowledge

At a learning rate of 1e-4, the news-only run reduced the six-task mean from 0.477 to 0.412. Belebele lost 21.2 points and m-MMLU-hy lost 7.1. Checkpoint inspections showed that the decline was complete by mid-training, so picking an earlier final checkpoint would not fix the recipe.

Lowering the rate to 3e-5 preserved the fluency improvement and returned the suite mean to 0.462. About two-thirds of the Belebele loss came back, but the model still ended below its base.

The ArmSTEM mixture changed the outcome. Swapping 6% of the news share for verified STEM data put the released model at 0.500. Belebele ended 9.7 points above the base, INCLUDE gained 4.0 points, and m-MMLU-hy returned to within its confidence interval of the base. Every run included the 20% English replay stream, so replay by itself did not stop the forgetting in the news-only conditions.

Six percent is not a portable setting for every language. The study changed subject matter and QA format together, and every run used one seed. The narrower finding is still useful: "more native-language tokens" does not specify what a model will retain.

ArmSTEM checks a translation by solving it

A translated math problem can read smoothly and still be wrong. COPA masked numbers, LaTeX spans, and question-solution separators with placeholders before translation. A missing or duplicated placeholder failed the item. The pipeline then checked the language and gave the Armenian problem to an independent model, which had to recover the gold answer without seeing the English source.

When that solver failed, a control attempted the original English problem. Items failed by both were labelled solver_limited instead of being automatically treated as translation errors. After repair, 96.6% of items survived. Two native speakers independently rated 299 of 300 sampled problems valid and agreed on every verdict.

This is a functional check: can the translated problem still produce the intended answer? It says less about whether every item is pedagogically good, culturally suitable, or safe for instruction tuning. Those require different reviews.

Public crawl size did not guarantee a clean test

ArmWeb comes from a structured crawl of Armenian news published from 2011 through 2026. The pipeline used language identification, global exact and near deduplication, time-aware splits, leakage checks, and 13-gram decontamination against ten Armenian evaluation sets.

With the same scanner, the authors found evaluation overlap in 3.3% of ArmWeb's documents before removal and excluded those rows from the release. They measured 7.9% in CulturaX-hy, 10.9% in HPLT-v2-hy, and 17.4% in FineWeb-2-hy. These are document-level rates from COPA's scanner, not a general ranking of corpus quality.

Most overlap came from web-derived perplexity sets. Short, hand-built multiple-choice items were often too small for a 13-gram detector to see. A model can therefore have contaminated perplexity evaluation while its knowledge benchmark has a different exposure. Calling the whole corpus clean or dirty would erase that distinction.

An order of operations for another language

Document provenance should survive the cleaning pipeline. Split the corpus before training, keep a temporal holdout, and scan each evaluation set whose examples are long enough for a meaningful overlap test.

Small ablations can prevent an expensive mistake. Compare target-language text alone with a content-diverse mixture, keep the unadapted base in the table, and measure fluency separately from knowledge. Tokenizer fertility changes the effective training budget too. COPA measured 4.15 Gemma tokens per Armenian word, against 5.31 for Qwen3.5 and 12.2 for Llama-3.1, which informed its base-model choice.

Publish the recipe beside the weights. The arm-gemma-e4b model card describes a base checkpoint for continuation, likelihood scoring, or later supervised fine-tuning. It is not a chat model, has no safety tuning, and its part-of-speech score fell from 0.18 to 0.01. Dropping the raw checkpoint into ish.chat or another interactive product would require instruction tuning and a separate safety evaluation first.

The project publishes its failure as carefully as its winning score. The Hindi ASR leaderboard ran into its own measurement problem: adding a language changes what an evaluation must notice. COPA's ablations caught a model becoming more fluent and less knowledgeable, then identified the data change that reversed it.