Lex: a literal ends where its own pattern says, not where a byte class does #220
Loading…
Reference in a new issue
No description provided.
Delete branch "feat/220-lex-a-literal-ends-where-its-own-pattern-says"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Lex: a literal ends where its own pattern says, not where a byte class does
Along the way:
Touches 6 files:
src/rust/crates/ck_driver/src/literal_lex.rs,src/rust/crates/ck_driver/tests/capability.rs,src/rust/crates/ck_driver/tests/typed_literal_lex.rs,tests/capability/geo_literal_parks_on_its_fold_body.cktest,tests/capability/money_eur_literal.cktest,tests/capability/money_usd_grouped_literal.cktest.Reads well. One question on
src/rust/crates/ck_driver/tests/typed_literal_lex.rs, nothing blocking.@ -137,0 +135,4 @@let from = match marker.position.as_str() {"Prefix" if src[start..].starts_with(&marker.text) => start + marker.text.len(),"Embedded" | "Suffix" if numeric_led => start,_ => continue,The catch-all arm means a variant added later compiles and lands here without a word. Can the match on
_name every variant so the compiler tells us?, at src/rust/crates/ck_driver/src/literal_lex.rs:138.@ -73,0 +111,4 @@let text = &app_file.src[rec.span.start.0 as usize..rec.span.end.0 as usize];assert_eq!(text, spelling, "the TypedLit spans the WHOLE lexeme");let claim = &table.claims()[rec.claim as usize];The
ascast truncates silently ifclaimis ever wider than the target. Is the bound guaranteed, or should this betry_from?Not guaranteed.
claimgoes through try_from now and the overflow is a typed error insrc/rust/crates/ck_driver/tests/typed_literal_lex.rs.Named them all. A new variant is a compile error in
src/rust/crates/ck_driver/src/literal_lex.rsnow, which is where we want to find out.Read it too. No objection to
src/rust/crates/ck_driver/tests/typed_literal_lex.rs.A claimed literal's scan window ended at an ordinary token boundary. Two spellings in the roster name a character that boundary excludes, so they could never be scanned as one lexeme at all: `Money<USD>`'s grouping `,` and `Money<EUR>`'s decimal `,` — the whole EUR claim was unreachable, and the USD grouping is the spelling `Types.ck` writes beside the claim. The window is now widened by the characters the CLAIM'S OWN pattern spells (every byte inside a quoted segment, `sep ','` included) and the pattern walks the widened tail longest-first. One mechanism, read off the pattern SOURCE — no branch keyed on a type, and a claim whose pattern names only ordinary body characters (Color, every temporal claim) scans byte-for-byte as before. The descending walk is what keeps `[$1.50, $2.50]` correct: the widened `1.50,` is offered first, the pattern refuses it, `1.50` is next. That is the point — the pattern decides where a literal ends, which is what it was always supposed to do. money_usd_grouped_literal_does_not_lex -> money_usd_grouped_literal runs: 42 money_eur_literal_does_not_lex -> money_eur_literal runs: 42 BROKEN_CELL_COUNT 54 -> 52 `geo_literal_does_not_lex` moved with them — its pattern names `' '`, the one space-bearing claim — but only one stage: it now lexes and typechecks and parks in the fold, so it stays BROKEN and pays no ratchet. It is renamed `geo_literal_parks_on_its_fold_body` and its declaration is `codegen: Unsupported`. The park is NOT float arithmetic, as the old comment guessed: `Types::signed` runs clean through the bounded evaluator; the fold body's named-tuple construction is `NotEvaluable("expr")`. Four cells' comments carried a cause that measurement contradicts, so they are corrected in place (no outcome moves, none was BROKEN for the stated reason): - `color_literal_*` / `timespan_*_parks` do not park on `decode_capture`'s missing optional arm; they park a step earlier. `tag_to_ty` types an optional capture `Ty::Unknown`, `Ty::named_tuple` then poisons the whole synthesized captures record, and `fold` has no field symbols to build anything from. - the two `timespan_*_parks` cells are additionally hiding a WRONG VALUE: `LiteralDSL::inner_match` matches only the leading class-and-quantifier of a capture's inner pattern and drops the rest, so `<h: digit+ 'h'>` accepts bare digits and `24s` is captured as twenty-four HOURS. Closing debt #71 alone would flip those cells green over a wrong fold. Same defect, one stage earlier, is why `timespan_multi_group_literal_does_not_lex` fails outright. - `datetime_literal_does_not_lex` fails in the LEXER, not the decode: `Date` and `Time` are other CLAIMS, and a capture's bareword roster is the three shipped `const Pattern`s. The fix is not a decision — `from_literal` reads `x.d.value`, so a capture naming a claim folds through that claim — only unbuilt. The remaining literal-DSL work is CkLang content (`LiteralDSL.ck`'s `inner_match`, its capture tags, `Types.ck`'s duration group order), not Rust, and belongs to the sibling repo that ships the same files. Verified: `ckc check` over all 437 capability cells, before and after, differs on exactly the three cells above and nothing else; the 36 corpus apps are byte-identical. `cargo fmt` clean, `cargo clippy --workspace --all-targets -D warnings` 0/0, `--test architecture` 38/38. The folded VALUE is unobservable from CkLang (`.value` on a tag-parameterised newtype does not lower), so `typed_literal_lex` gains the two gates that can see it: `$1,000.50` and `€1.000,50` both fold to 100050 minor units, and the aggregate spelling still ends at its comma.