Training pipeline fixes (real-corpus SFT + docs CPT phase) #30

Manually merged
hkowalski merged 2 commits from feat/030-training-pipeline-fixes-real-corpus-sft-docs-c into master 2026-02-05 12:14:30 +01:00
Owner

Training pipeline fixes (real-corpus SFT + docs CPT phase)

Along the way:

  • training: build corpus from real apps, not invalid .cktest fragments
  • training: add CPT phase over docs + corpus source before SFT

Touches 8 files: scripts/training/.gitignore, scripts/training/README.md, scripts/training/build-pretrain.sh, scripts/training/python/build_corpus.py, scripts/training/python/build_pretrain.py, scripts/training/python/eval.py and 2 more.

Training pipeline fixes (real-corpus SFT + docs CPT phase) Along the way: - training: build corpus from real apps, not invalid .cktest fragments - training: add CPT phase over docs + corpus source before SFT Touches 8 files: `scripts/training/.gitignore`, `scripts/training/README.md`, `scripts/training/build-pretrain.sh`, `scripts/training/python/build_corpus.py`, `scripts/training/python/build_pretrain.py`, `scripts/training/python/eval.py` and 2 more.
hkowalski approved these changes 2026-02-05 11:13:13 +01:00
hkowalski left a comment

Approving. I would have split scripts/training/python/train.py differently, but not enough to hold this up.

Approving. I would have split `scripts/training/python/train.py` differently, but not enough to hold this up.
@ -0,0 +45,4 @@
# ~MAX_CHARS keeps a chunk comfortably under a 2048-token sequence (~4 chars/
# token), so the CPT trainer rarely truncates a chunk mid-thought.
MAX_CHARS = 6000
Owner

Where does the number beside MAX_CHARS come from? It reads as measured rather than chosen, and nothing on the line says which.

Where does the number beside `MAX_CHARS` come from? It reads as measured rather than chosen, and nothing on the line says which.
@ -106,0 +125,4 @@
# --- Phase 1: CPT (continued pre-training on raw CkLang text) -----------
#
# Same LoRA adapter, raw next-token over docs + corpus source. NOTE: this
# adapts the attention/MLP projections only — it does NOT retrain
Owner

I had to read adapts twice. Nothing wrong with it, but one line of comment would save the next person the second read.

I had to read `adapts` twice. Nothing wrong with it, but one line of comment would save the next person the second read.
Author
Owner

Comment added on adapts in scripts/training/python/train.py. It was obvious when I wrote it and it will not be in a month.

Comment added on `adapts` in `scripts/training/python/train.py`. It was obvious when I wrote it and it will not be in a month.
Author
Owner

Measured. It is a named constant in scripts/training/python/build_pretrain.py now, with a line saying how.

Measured. It is a named constant in `scripts/training/python/build_pretrain.py` now, with a line saying how.
The builder walked tests/**/*.cktest only and emitted a code-gen pair from
every fixture's first page regardless of validity — so it trained Gemma on
deliberately-malformed snippets as canonical CkLang and never read the real
tests/corpus/** apps at all. Rewrite around two strictly-separated sources:

  - tests/corpus/**       -> code-gen (whole valid module files, prompt names
                             app + role + siblings) and impl-from-header
                             (.ckh -> .ck pairs).
  - tests/**/*.cktest      -> diagnostics only (all pages, canonical format,
                             (none) kept as a negative example). Never code-gen;
                             corpus subtree skipped by the fixture walk.

Split keys whole corpus apps to one side so sibling-context prompts can't leak
across train/eval. eval.py counts impl-from-header in the code-gen metric so
the new kind isn't dropped from reporting. README rewritten to document the
two-source split and the original bug.
SFT alone teaches assistant behavior, not the language. Add a continued
pre-training (CPT) phase so the model gets raw next-token exposure to a large
body of CkLang text first.

- build_pretrain.py: docs/**/*.md (packed ~6k-char chunks on paragraph
  boundaries) + tests/corpus/**/*.ck|.ckh (whole files) -> pretrain.jsonl
  (~370k tokens). Documents the known caveat that docs still carry superseded
  / PROPOSED content; curation is a later pass, accepted for now.
- train.py: phase 1 CPT on raw text (packing on, lower LR 5e-5, 1 epoch, same
  LoRA), then phase 2 SFT as before. CPT auto-skips if pretrain.jsonl is
  absent; --skip-pretrain forces SFT-only. Notes that embed_tokens/lm_head are
  not retrained (projection-level CPT only) as the deliberate first step.
- build-pretrain.sh wrapper, run.sh grows to 6 steps, README documents the
  two-phase train, .gitignore covers pretrain.jsonl.
hkowalski manually merged commit 3af42856d0 into master 2026-02-05 12:14:30 +01:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
northgale/cklang-bootstrap!30
No description provided.