Training pipeline fixes (real-corpus SFT + docs CPT phase) #30
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/030-training-pipeline-fixes-real-corpus-sft-docs-c"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Training pipeline fixes (real-corpus SFT + docs CPT phase)
Along the way:
Touches 8 files:
scripts/training/.gitignore,scripts/training/README.md,scripts/training/build-pretrain.sh,scripts/training/python/build_corpus.py,scripts/training/python/build_pretrain.py,scripts/training/python/eval.pyand 2 more.Approving. I would have split
scripts/training/python/train.pydifferently, but not enough to hold this up.@ -0,0 +45,4 @@# ~MAX_CHARS keeps a chunk comfortably under a 2048-token sequence (~4 chars/# token), so the CPT trainer rarely truncates a chunk mid-thought.MAX_CHARS = 6000Where does the number beside
MAX_CHARScome from? It reads as measured rather than chosen, and nothing on the line says which.@ -106,0 +125,4 @@# --- Phase 1: CPT (continued pre-training on raw CkLang text) -----------## Same LoRA adapter, raw next-token over docs + corpus source. NOTE: this# adapts the attention/MLP projections only — it does NOT retrainI had to read
adaptstwice. Nothing wrong with it, but one line of comment would save the next person the second read.Comment added on
adaptsinscripts/training/python/train.py. It was obvious when I wrote it and it will not be in a month.Measured. It is a named constant in
scripts/training/python/build_pretrain.pynow, with a line saying how.The builder walked tests/**/*.cktest only and emitted a code-gen pair from every fixture's first page regardless of validity — so it trained Gemma on deliberately-malformed snippets as canonical CkLang and never read the real tests/corpus/** apps at all. Rewrite around two strictly-separated sources: - tests/corpus/** -> code-gen (whole valid module files, prompt names app + role + siblings) and impl-from-header (.ckh -> .ck pairs). - tests/**/*.cktest -> diagnostics only (all pages, canonical format, (none) kept as a negative example). Never code-gen; corpus subtree skipped by the fixture walk. Split keys whole corpus apps to one side so sibling-context prompts can't leak across train/eval. eval.py counts impl-from-header in the code-gen metric so the new kind isn't dropped from reporting. README rewritten to document the two-source split and the original bug.