AI Jam Sessions · Research note

Do jam-actions traces train a better tool-grounded musical-QA model?

A preregistered confirmatory evaluation on frozen fine-tuned artifacts.

An open dataset + fine-tuning research program · preregistered, LLM-assisted analysis with an external deterministic verifier

Dataset Interactive explorer Live demo (GPU) Code Collection Dataset DOI

On-platform research note & project page — not an arXiv-indexed paper.

Update, 2026-09-25. Version 0.6.0 of jam-actions-v0 withdrew 58 records from four songs (Chopin Op. 9 No. 2 and Op. 28 No. 4, Beethoven Op. 13 II, Schumann Op. 15 No. 7) whose source arrangements have no established licence. The jam-ft-v1-qwen25 adapters were trained on records that included them, so they have been withdrawn. Results shown here are the historical record of what was measured. Read the licence audit.

AI Jam Sessions — piano keys dissolving into a tool-node network

Verdict · frozen wording class, bar met

Powered win.

The jam-actions v1 recipe trains a model that beats the prompted baseline at tool-grounded musical QA on a preregistered 36-record cohort dominated by held-out material. Five frozen fine-tunes (no retraining, no reselection, artifacts sha-pinned before this cohort existed), scored once each against a fresh sealed baseline, move the primary condition from 0.678 to 0.890.

+0.212

mean Δ, primary condition

29/36

paired wins vs a 24/34 bar

p < 0.0001

exact sign test (p = 0.000039)

10/12

wins on never-trained music

Abstract

jam-actions is an open dataset of executable, tool-grounded reasoning traces over symbolic music. We ask a narrow question: does fine-tuning on these traces produce a model that answers musical questions better than a strong prompted baseline?

Across three preregistered arcs the answer sharpened. v0 was an honest negative. v1 was directionally better but underpowered — 12/16 paired wins, one short of a frozen bar that a 16-record cohort could barely resolve. This arc, B-1, widens the sealed cohort to 36 records dominated by never-trained material, mints a fresh sealed baseline on the published v0.5.0 records, and re-scores the frozen v1 artifacts. Nothing was retrained or reselected. On the primary tool-grounded condition the fine-tunes move accuracy from 0.678 to 0.890 (Δ +0.212; song-cluster CI95 [0.128, 0.305], the interval that grazed zero at n=16), winning 29 of 36 paired records against a preregistered 24/34 bar (p < 0.0001), and clearing significance on the twelve held-out clair-de-lune records the models never saw. They remain below baseline when answering from prose alone — reported here with equal weight.

The result

Four evaluation conditions, all-seeds means over the 36-record cohort (n=3 per condition). The fine-tunes win decisively where their tools are — the tool_inspected surface — and lose on the prose-only and no-tools surfaces.

Prompted baseline — qwen2.5:7b Fine-tuned — v1 recipe, all-seeds mean

Figure 1. Primary and secondary conditions. The fine-tunes answer better by inspecting (tool_inspected: +0.212) and worse by recalling prose (full, text_only, random_midi all below baseline). Only tool_inspected carries the preregistered claim; the controls are shown to bound it.

Condition	Baseline	Fine-tuned	Δ	Wins	Record CI95	Sign p
tool_inspected (primary)	0.678	0.890	+0.212	29/36 2 ties	[+0.148, +0.275]	0.000039
full (secondary)	0.396	0.313	−0.083	9/36 3 ties	[−0.136, −0.033]	0.0135
text_only (control)	0.381	0.307	−0.074	13/36 1 tie	[−0.123, −0.026]	0.175
random_midi (control)	0.397	0.304	−0.093	9/36 —	[−0.150, −0.034]	0.0039

Three arcs, one dataset

The claim is the endpoint of a documented sequence, not a single lucky run. Each arc was preregistered before any model was called; each reports all five seeds, no best-of-seeds.

Honest negative

Fine-tuning did not beat the prompted baseline. Reported as-is.

Directionally better, underpowered

12/16 paired wins, Δ +0.202, p = 0.0043 — one win short of a frozen ≥13/16 bar a 16-record cohort could barely resolve.

B-1 · this note

Powered win

Same frozen artifacts, wider preregistered cohort. 29/36 wins, Δ +0.212, p < 0.0001. The effect holds at proper power.

The question B-1 existed to answer — was v1's 12/16 miss a power artifact or a real ceiling? — is answered: power artifact.

Method

The confirmatory cohort (36 records, three strata)

The cohort is frozen by a seeded, byte-replayable derivation; its receipt is the authority (sha 7ea74fc9…). It is dominated at the margin by material the models never trained on.

Stratum	n	What it is	Baseline → FT	Δ	Wins	Exact p
CL	12	Every clair-de-lune test record — never in any training corpus, any arc, any form.	0.729 → 0.906	+0.176	10/12	0.039
LG	15	The sealed-history train-song cohort (v0/v1 continuity).	0.711 → 0.887	+0.176	11/15 (1t)	0.057
NW	9	Seeded-blind new train records, drawn before any B-1 output existed.	0.556 → 0.876	+0.320	8/9 (1t)	0.0078

Prompted baseline — qwen2.5:7b Fine-tuned — v1 recipe, all-seeds mean

Figure 2. Primary condition, unpooled by stratum. CL is the confirmatory headline: on twelve clair-de-lune records the models never saw in training — eleven of which had never been evaluated by anything — the frozen fine-tunes win 10/12 and the stratum clears significance on its own. LG reproduces v1's seen-song outcome exactly (11/15, 1 tie) under a different baseline on corrected records — the measurement is stable; the added power came from widening the cohort.

What was pinned, what never moved

Frozen artifacts. The five v1 PEFT adapters (Qwen2.5-7B base) were frozen before this cohort existed; the pre-run gate asserts each on-disk GGUF sha256 matches the pinned p4-receipt.json. No retraining, no reselection — selection at v1 used only inner-validation records and never saw any cohort record.
Fresh sealed baseline. A single sealed run of the prompted qwen2.5:7b on the published v0.5.0 records — because erratum-002 corrected the Bach records' prose that the prompts embed. Both arms saw identical prompts.
One eval per model. Six sealed runs total (baseline + five seeds), abort-on-first-failure, a completed results file never regenerated. All five seeds report.
Frozen victory bar. Set ex-ante as an exact-number table at α = 0.05 two-sided: 25/36 at n_eff = 36; with 2 ties, n_eff = 34 → 24/34. The result cleared it by five wins.
Deterministic external verifier. The eval harness and MCQ generators are deterministic TypeScript, byte-identical to the sealed v0/v1 pins — never the models under test. Statistics are seeded (mulberry32, seed 20260713; 10k bootstrap, 10k permutations).

What did not improve — reported with equal weight

The prose-only and no-tools surfaces remain below baseline, consistent with both prior arcs: full −0.083, text_only −0.074, random_midi −0.093 (all record-CIs exclude zero; text_only's sign test does not reach significance). The v1 fine-tunes' competence is concentrated where their tools are — they answer better by inspecting and worse by recalling prose. The preregistered claim does not extend to prose surfaces; that is the target of a future prose-surface retrain, not this arc.

Outcome-dependence, disclosed

This arc exists because v1 missed its bar by one win, and a confirmatory re-test designed after seeing a near-miss carries selection risk. The mitigations are all mechanical: the artifacts were frozen before this cohort existed; the added material is dominated by never-trained records (all 12 CL records structurally excluded from every training corpus; the 9 NW records drawn blind by a fixed seed); the bar was frozen ex-ante as an exact-number table at the same α the v1 bar used, recomputed for the new n, not loosened; one eval per model, no reruns, all five seeds report; and both possible outcomes had pre-committed wordings — a miss would have been a publishable result, not a retry trigger.

The claim, and the gate that opened

jam-actions traces, augmented with execution-verified grounding-shaped examples (the v1 recipe), train a model that beats the prompted baseline at tool-grounded musical QA — +0.21 mean, 29/36 paired wins against a preregistered 24/34 bar, p < 0.0001, strongest on never-trained music — while remaining below baseline when answering from prose alone.

Per the preregistered claim class, a passing result opens a director gate before anything ships. The gate fired 2026-07-11 with this result in view: publish all five seed adapters, with the claim tied to the all-seeds mean, per-seed numbers disclosed, no best-of-seeds. The published artifact was mcp-tool-shop/jam-ft-v1-qwen25 (withdrawn on 2026-09-25; see the notice at the top of this page) — the five selected-epoch PEFT adapters, byte-identical to the frozen artifacts this note evaluated (adapter_model.safetensors sha256s on the card and in the publish receipt).

Reproducibility & receipts

Preregistration: experiments/finetune-arc-v2/P0-LOCK.md, frozen at commit 8866608 (c9dcd17 before the repository's history was rewritten on 2026-09-26) before any model call.
Cohort + frozen bar table: data/b1-cohort.json (sha 7ea74fc9…, sampler seed 20260712).
Harness: pin 8bef493d…, eval-logic files byte-identical to the sealed v0/v1 pins.
Sealed artifacts: one results file per model, never regenerated; pre-run gate receipt (package checksums 274/274, cohort↔harness equality, five GGUF sha256 == receipt, six ollama tags with modelfile evidence).
Statistics: evals/b1-stats.json — seeded RNG, pooled + per-stratum + per-record table.
Dataset under eval: jam-actions-v0 v0.5.0, tag jam-actions-v0-0.5.0-cut-2026-07-11, DOI 10.5281/zenodo.21313954.
Cost & timing: six sealed runs, 36.2 minutes wall-clock total, $0 — local evals on frozen artifacts, no pod.

Citation

This note presents an evaluation of the jam-actions dataset. Please cite the dataset:

@misc{jam_actions_2026,
  title        = {jam-actions-v0: executable tool-grounded reasoning traces over symbolic music},
  author       = {{mcp-tool-shop} and {AI Jam Sessions}},
  year         = {2026},
  howpublished = {Hugging Face Datasets},
  doi          = {10.5281/zenodo.21313954},
  url          = {https://huggingface.co/datasets/mcp-tool-shop/jam-actions-v0},
  note         = {Confirmatory fine-tuning evaluation (arc B-1): +0.212 on tool-grounded
                  musical QA, 29/36 paired wins vs a preregistered 24/34 bar, p < 0.0001.}
}