AI Jam Sessions · Research note

Do jam-actions traces train a better tool-grounded musical-QA model?

A preregistered confirmatory evaluation on frozen fine-tuned artifacts.

An open dataset + fine-tuning research program · preregistered, LLM-assisted analysis with an external deterministic verifier

Dataset Model (5 adapters) Interactive explorer Live demo (GPU) Code Collection Dataset DOI

On-platform research note & project page — not an arXiv-indexed paper.

AI Jam Sessions — piano keys dissolving into a tool-node network

Verdict · frozen wording class, bar met

Powered win.

The jam-actions v1 recipe trains a model that beats the prompted baseline at tool-grounded musical QA on a preregistered 36-record cohort dominated by held-out material. Five frozen fine-tunes (no retraining, no reselection, artifacts sha-pinned before this cohort existed), scored once each against a fresh sealed baseline, move the primary condition from 0.678 to 0.890.

+0.212

mean Δ, primary condition

29/36

paired wins vs a 24/34 bar

p < 0.0001

exact sign test (p = 0.000039)

10/12

wins on never-trained music

Abstract

jam-actions is an open dataset of executable, tool-grounded reasoning traces over symbolic music. We ask a narrow question: does fine-tuning on these traces produce a model that answers musical questions better than a strong prompted baseline?

Across three preregistered arcs the answer sharpened. v0 was an honest negative. v1 was directionally better but underpowered — 12/16 paired wins, one short of a frozen bar that a 16-record cohort could barely resolve. This arc, B-1, widens the sealed cohort to 36 records dominated by never-trained material, mints a fresh sealed baseline on the published v0.5.0 records, and re-scores the frozen v1 artifacts. Nothing was retrained or reselected. On the primary tool-grounded condition the fine-tunes move accuracy from 0.678 to 0.890 (Δ +0.212; song-cluster CI95 [0.128, 0.305], the interval that grazed zero at n=16), winning 29 of 36 paired records against a preregistered 24/34 bar (p < 0.0001), and clearing significance on the twelve held-out clair-de-lune records the models never saw. They remain below baseline when answering from prose alone — reported here with equal weight.

The result

Four evaluation conditions, all-seeds means over the 36-record cohort (n=3 per condition). The fine-tunes win decisively where their tools are — the tool_inspected surface — and lose on the prose-only and no-tools surfaces.

Prompted baseline — qwen2.5:7b Fine-tuned — v1 recipe, all-seeds mean

Figure 1. Primary and secondary conditions. The fine-tunes answer better by inspecting (tool_inspected: +0.212) and worse by recalling prose (full, text_only, random_midi all below baseline). Only tool_inspected carries the preregistered claim; the controls are shown to bound it.

Condition	Baseline	Fine-tuned	Δ	Wins	Record CI95	Sign p
tool_inspected (primary)	0.678	0.890	+0.212	29/36 2 ties	[+0.148, +0.275]	0.000039
full (secondary)	0.396	0.313	−0.083	9/36 3 ties	[−0.136, −0.033]	0.0135
text_only (control)	0.381	0.307	−0.074	13/36 1 tie	[−0.123, −0.026]	0.175
random_midi (control)	0.397	0.304	−0.093	9/36 —	[−0.150, −0.034]	0.0039

Three arcs, one dataset

The claim is the endpoint of a documented sequence, not a single lucky run. Each arc was preregistered before any model was called; each reports all five seeds, no best-of-seeds.

Honest negative

Fine-tuning did not beat the prompted baseline. Reported as-is.

Directionally better, underpowered

12/16 paired wins, Δ +0.202, p = 0.0043 — one win short of a frozen ≥13/16 bar a 16-record cohort could barely resolve.

B-1 · this note

Powered win

Same frozen artifacts, wider preregistered cohort. 29/36 wins, Δ +0.212, p < 0.0001. The effect holds at proper power.

The question B-1 existed to answer — was v1's 12/16 miss a power artifact or a real ceiling? — is answered: power artifact.

Method

The confirmatory cohort (36 records, three strata)

The cohort is frozen by a seeded, byte-replayable derivation; its receipt is the authority (sha 7ea74fc9…). It is dominated at the margin by material the models never trained on.

Stratum	n	What it is	Baseline → FT	Δ	Wins	Exact p
CL	12	Every clair-de-lune test record — never in any training corpus, any arc, any form.	0.729 → 0.906	+0.176	10/12	0.039
LG	15	The sealed-history train-song cohort (v0/v1 continuity).	0.711 → 0.887	+0.176	11/15 (1t)	0.057
NW	9	Seeded-blind new train records, drawn before any B-1 output existed.	0.556 → 0.876	+0.320	8/9 (1t)	0.0078

Prompted baseline — qwen2.5:7b Fine-tuned — v1 recipe, all-seeds mean

Figure 2. Primary condition, unpooled by stratum. CL is the confirmatory headline: on twelve clair-de-lune records the models never saw in training — eleven of which had never been evaluated by anything — the frozen fine-tunes win 10/12 and the stratum clears significance on its own. LG reproduces v1's seen-song outcome exactly (11/15, 1 tie) under a different baseline on corrected records — the measurement is stable; the added power came from widening the cohort.

What was pinned, what never moved

Frozen artifacts. The five v1 PEFT adapters (Qwen2.5-7B base) were frozen before this cohort existed; the pre-run gate asserts each on-disk GGUF sha256 matches the pinned p4-receipt.json. No retraining, no reselection — selection at v1 used only inner-validation records and never saw any cohort record.
Fresh sealed baseline. A single sealed run of the prompted qwen2.5:7b on the published v0.5.0 records — because erratum-002 corrected the Bach records' prose that the prompts embed. Both arms saw identical prompts.
One eval per model. Six sealed runs total (baseline + five seeds), abort-on-first-failure, a completed results file never regenerated. All five seeds report.
Frozen victory bar. Set ex-ante as an exact-number table at α = 0.05 two-sided: 25/36 at n_eff = 36; with 2 ties, n_eff = 34 → 24/34. The result cleared it by five wins.
Deterministic external verifier. The eval harness and MCQ generators are deterministic TypeScript, byte-identical to the sealed v0/v1 pins — never the models under test. Statistics are seeded (mulberry32, seed 20260713; 10k bootstrap, 10k permutations).

What did not improve — reported with equal weight

The prose-only and no-tools surfaces remain below baseline, consistent with both prior arcs: full −0.083, text_only −0.074, random_midi −0.093 (all record-CIs exclude zero; text_only's sign test does not reach significance). The v1 fine-tunes' competence is concentrated where their tools are — they answer better by inspecting and worse by recalling prose. The preregistered claim does not extend to prose surfaces; that is the target of a future prose-surface retrain, not this arc.

Outcome-dependence, disclosed

This arc exists because v1 missed its bar by one win, and a confirmatory re-test designed after seeing a near-miss carries selection risk. The mitigations are all mechanical: the artifacts were frozen before this cohort existed; the added material is dominated by never-trained records (all 12 CL records structurally excluded from every training corpus; the 9 NW records drawn blind by a fixed seed); the bar was frozen ex-ante as an exact-number table at the same α the v1 bar used, recomputed for the new n, not loosened; one eval per model, no reruns, all five seeds report; and both possible outcomes had pre-committed wordings — a miss would have been a publishable result, not a retry trigger.

The claim, and the gate that opened

jam-actions traces, augmented with execution-verified grounding-shaped examples (the v1 recipe), train a model that beats the prompted baseline at tool-grounded musical QA — +0.21 mean, 29/36 paired wins against a preregistered 24/34 bar, p < 0.0001, strongest on never-trained music — while remaining below baseline when answering from prose alone.

Per the preregistered claim class, a passing result opens a director gate before anything ships. The gate fired 2026-07-11 with this result in view: publish all five seed adapters, with the claim tied to the all-seeds mean, per-seed numbers disclosed, no best-of-seeds. The published artifact is mcp-tool-shop/jam-ft-v1-qwen25 — the five selected-epoch PEFT adapters, byte-identical to the frozen artifacts this note evaluated (adapter_model.safetensors sha256s on the card and in the publish receipt).

Reproducibility & receipts

Preregistration: experiments/finetune-arc-v2/P0-LOCK.md, frozen at commit c9dcd17 before any model call.
Cohort + frozen bar table: data/b1-cohort.json (sha 7ea74fc9…, sampler seed 20260712).
Harness: pin 8bef493d…, eval-logic files byte-identical to the sealed v0/v1 pins.
Sealed artifacts: one results file per model, never regenerated; pre-run gate receipt (package checksums 274/274, cohort↔harness equality, five GGUF sha256 == receipt, six ollama tags with modelfile evidence).
Statistics: evals/b1-stats.json — seeded RNG, pooled + per-stratum + per-record table.
Dataset under eval: jam-actions-v0 v0.5.0, tag jam-actions-v0-0.5.0-cut-2026-07-11, DOI 10.5281/zenodo.21313954.
Cost & timing: six sealed runs, 36.2 minutes wall-clock total, $0 — local evals on frozen artifacts, no pod.

Citation

This note presents an evaluation of the jam-actions dataset. Please cite the dataset:

@misc{jam_actions_2026,
  title        = {jam-actions-v0: executable tool-grounded reasoning traces over symbolic music},
  author       = {{mcp-tool-shop} and {AI Jam Sessions}},
  year         = {2026},
  howpublished = {Hugging Face Datasets},
  doi          = {10.5281/zenodo.21313954},
  url          = {https://huggingface.co/datasets/mcp-tool-shop/jam-actions-v0},
  note         = {Confirmatory fine-tuning evaluation (arc B-1): +0.212 on tool-grounded
                  musical QA, 29/36 paired wins vs a preregistered 24/34 bar, p < 0.0001.}
}