Implement our parser, . field access, comment preservation and line number tracking for source-level debugging #26

alex-eg · 2026-09-09T17:42:39+02:00

alex-eg commented

2026-09-09 17:42:39 +02:00

Closes #16

alex-eg added 6 commits 2026-09-09 17:42:39 +02:00

replace Scheme read with our tokenizer and parser 1ca77aaf59

Also implement . as field access operator and preserve ;-comments in
generated C

fix tests/Makefile not cleaning up .o files 543a333165

fix sex-tests and sextest not rebuilding f6ec07a4b5

reuse sex reader in sextest 9903cae3f7

migrate to Chicken 6 528f26b7ef

add source line number preservation

Sex CI / build-macos (pull_request) Has been cancelled

Details

Sex CI / build-linux (pull_request) Failing after 3m34s

Details

d83ee32f12

For source-level debug. Add --line-directives option, defaulted to
statement. Disabled when used with -C. Explicit non-none value
overrides disabling when -C is present

alex-eg requested review from pkulev 2026-09-09 17:42:39 +02:00

pkulev requested changes 2026-09-14 23:40:08 +02:00

Dismissed

pkulev left a comment

Built and ran this on CHICKEN 6.0.0 (worktree at d83ee32). Unit tests are green; the issue #16 success program is not.

What I ran

Check	Result
`make sexc`	ok
`./sex-tests` (incl. line-directives, unicode `unkebabify`)	138/138 pass
`hello-world.sex`, `lists.sex`, `unicode.sex` via `sextest`	pass
`make run-tests`	fails: `tests/sex-programs/comments.sex` is not in the tree
issue #16 golden file via `sexc -C`	C that `cc` rejects
`sexc` compile (no `-C`) of a tiny program	runs

Chicken 6.0.0 names process ports from the child (process-input-port is writable). That part of the migration is correct.

vs issue #16

Field access and the plus macro work: a.pos.x + a.pos.y. Struct / enum / vectors look right. The rest of the golden sample does not.

| / || are not C operators. Actual -C output:

struct vertex a = {{10, 20}, {0.3, 0.5, 0.7}, |\||(B, A, R)};
if (|\|\||(0, 1)){

cc: expected expression before '|' / stray '\'. Old c-bit-or still emits 1 | 2 | 4. The reader can produce the symbols; sex-fmt-c still only special-cases c-or / c-bit-or and otherwise goes through c-apply. The (string->symbol ".") cond is the pattern to extend for "|", "||", "|=".

In-arg comments insert a comma:

printf("%d %f %d\n",
       /* the printf*/,
       a.pos.x + a.pos.y,

Comments are not packed; extra ;s are kept:

/*;; Main entry point*/
/*;; Multi line comments should be packed*/
    /*; the vertex*/

A ; in an if arm shifts clauses. Source:

(if 1
    ; comment
    (puts "then")
    (puts "else"))

became if (1) { /* comment*/ } else if (puts("then")) { puts("else"); }.

A ; between for clauses does the same: for (int i = 0; /* between*/; i < 1) with ++i moved into the body.

Fn headers already strip comments so positional accessors do not shift. That needs to apply to if / for / while / calls / switch as well — or comments should be spliced as whitespace, not list elements, except in statement position.

Merge-blocking

Missing comments.sex — make run-tests cannot succeed.
Issue #16 success program does not compile.
Comment forms are data, not whitespace, except where they happen to sit as extra statements.
CI still only runs make sex-tests (make run-tests is commented out). Unit tests alone would look green; they never compiled the golden sample.

--line-directives works as --line-directives=statement (getopt-long 4.0). Space-separated form errors. Help already shows =ARG. -C without that flag emits no #line, as intended.

What looks solid

The reader, #line tests, Chicken 6 build, unicode identifiers / octal string escapes, and sextest sharing reader.scm are in good shape. I would not merge until | / || lower to operators, comments splice as comments, the golden sample is a test, and comments.sex exists or is dropped from SEX_TEST_PROGRAMS.

Built and ran this on CHICKEN 6.0.0 (worktree at `d83ee32`). Unit tests are green; the issue #16 success program is not. ## What I ran | Check | Result | |---|---| | `make sexc` | ok | | `./sex-tests` (incl. line-directives, unicode `unkebabify`) | 138/138 pass | | `hello-world.sex`, `lists.sex`, `unicode.sex` via `sextest` | pass | | `make run-tests` | fails: `tests/sex-programs/comments.sex` is not in the tree | | issue #16 golden file via `sexc -C` | C that `cc` rejects | | `sexc` compile (no `-C`) of a tiny program | runs | Chicken 6.0.0 names process ports from the child (`process-input-port` is writable). That part of the migration is correct. ## vs issue #16 Field access and the `plus` macro work: `a.pos.x + a.pos.y`. Struct / enum / vectors look right. The rest of the golden sample does not. **`|` / `||` are not C operators.** Actual `-C` output: ```c struct vertex a = {{10, 20}, {0.3, 0.5, 0.7}, |\||(B, A, R)}; if (|\|\||(0, 1)){ ``` `cc`: `expected expression before '|'` / `stray '\'`. Old `c-bit-or` still emits `1 | 2 | 4`. The reader can produce the symbols; `sex-fmt-c` still only special-cases `c-or` / `c-bit-or` and otherwise goes through `c-apply`. The `(string->symbol ".")` cond is the pattern to extend for `"|"`, `"||"`, `"|="`. **In-arg comments insert a comma:** ```c printf("%d %f %d\n", /* the printf*/, a.pos.x + a.pos.y, ``` **Comments are not packed;** extra `;`s are kept: ```c /*;; Main entry point*/ /*;; Multi line comments should be packed*/ /*; the vertex*/ ``` **A `;` in an `if` arm shifts clauses.** Source: ```scheme (if 1 ; comment (puts "then") (puts "else")) ``` became `if (1) { /* comment*/ } else if (puts("then")) { puts("else"); }`. **A `;` between `for` clauses does the same:** `for (int i = 0; /* between*/; i < 1)` with `++i` moved into the body. Fn headers already strip comments so positional accessors do not shift. That needs to apply to `if` / `for` / `while` / calls / `switch` as well — or comments should be spliced as whitespace, not list elements, except in statement position. ## Merge-blocking 1. Missing `comments.sex` — `make run-tests` cannot succeed. 2. Issue #16 success program does not compile. 3. Comment forms are data, not whitespace, except where they happen to sit as extra **statements**. 4. CI still only runs `make sex-tests` (`make run-tests` is commented out). Unit tests alone would look green; they never compiled the golden sample. `--line-directives` works as `--line-directives=statement` (getopt-long 4.0). Space-separated form errors. Help already shows `=ARG`. `-C` without that flag emits no `#line`, as intended. ## What looks solid The reader, `#line` tests, Chicken 6 build, unicode identifiers / octal string escapes, and `sextest` sharing `reader.scm` are in good shape. I would not merge until `|` / `||` lower to operators, comments splice as comments, the golden sample is a test, and `comments.sex` exists or is dropped from `SEX_TEST_PROGRAMS`.

Makefile

						
				@@ -59,3 +59,3 @@

					cp ./tools/sextest/sextest .

				SEX_TEST_PROGRAMS = hello-world lists

				SEX_TEST_PROGRAMS = hello-world lists comments unicode

pkulev commented

2026-09-14 23:40:08 +02:00

comments is listed here, but tests/sex-programs/comments.sex is not in the tree. make run-tests dies on (open-input-file) … comments.sex after hello-world, so unicode never runs either. Add the file or drop it from SEX_TEST_PROGRAMS.

`comments` is listed here, but `tests/sex-programs/comments.sex` is not in the tree. `make run-tests` dies on `(open-input-file) … comments.sex` after hello-world, so unicode never runs either. Add the file or drop it from `SEX_TEST_PROGRAMS`.

alex-eg commented

2026-09-15 00:09:20 +02:00

Oops

fmt-c-writer.scm Outdated

						
				@@ -19,0 +61,4 @@

				        `(%begin ,(line-directive src) ,(walk-expr s))

				        (walk-expr s))))

				(define (walk-if-clauses clauses)

pkulev commented

2026-09-14 23:40:08 +02:00

This is strictly test stmt test stmt …. A ; comment is a list element, so it is treated as an arm and shifts the rest.

(if 1
    ; comment
    (puts "then")
    (puts "else"))

became if (1) { /* comment*/ } else if (puts("then")) { puts("else"); }. Same class of bug as stripping comments from fn headers — those accessors would otherwise shift too.

This is strictly `test stmt test stmt …`. A `;` comment is a list element, so it is treated as an arm and shifts the rest. ```scheme (if 1 ; comment (puts "then") (puts "else")) ``` became `if (1) { /* comment*/ } else if (puts("then")) { puts("else"); }`. Same class of bug as stripping comments from fn headers — those accessors would otherwise shift too.

fmt-c-writer.scm Outdated

						
				@@ -88,6 +137,28 @@

				    (('c-or . rest) (apply c-or (map walk-expr rest)))

pkulev commented

2026-09-14 23:40:08 +02:00

Still only c-or / c-bit-or / c-bit-or=. After the new reader, (| B A R) and (|| 0 1) are those symbols, not these names. sexc -C of the #16 sample emitted \|\\||(B, A, R) and cc rejected it. c-bit-or still works (1 | 2 | 4).

Still only `c-or` / `c-bit-or` / `c-bit-or=`. After the new reader, `(| B A R)` and `(|| 0 1)` are those symbols, not these names. `sexc -C` of the #16 sample emitted `\|\\||(B, A, R)` and `cc` rejected it. `c-bit-or` still works (`1 | 2 | 4`).

fmt-c-writer.scm Outdated

						
				@@ -91,0 +146,4 @@

				    (('while test . body)

				     (cons* 'while (walk-expr test) (walk-body body)))

				    (('for init test step . body)

				     (cons* 'for (walk-expr init) (walk-expr test) (walk-expr step)

pkulev commented

2026-09-14 23:40:08 +02:00

Same positional trap: a ; between init / test / step binds to the wrong slot. Observed:

for (int i = 0; /* between*/; i < 1) {
    ++i;
    puts("x");
}

c-apply also joins args with ", ", so (printf "%d" ; c\n x) becomes printf("%d", /* c */, x) — extra comma, and cc fails on the #16 sample.

Same positional trap: a `;` between `init` / `test` / `step` binds to the wrong slot. Observed: ```c for (int i = 0; /* between*/; i < 1) { ++i; puts("x"); } ``` `c-apply` also joins args with `", "`, so `(printf "%d" ; c\n x)` becomes `printf("%d", /* c */, x)` — extra comma, and `cc` fails on the #16 sample.

alex-eg added 6 commits 2026-09-15 15:55:18 +02:00

add `comments' test program 0db4a11a2b

pass all args after '--' to C compiler verbatim af6e778185

begin -> do d609a0c591

Shorter and doesn't imply existence of `end'

add codegen and args tests, fix couple of bugs in C generation 36932fb480

fix comments inside statements shifting their meaning fa4ad5acee

fix stray commas and add comment packing

Sex CI / build-linux (pull_request) Failing after 3m1s

Details

Sex CI / build-macos (pull_request) Has been cancelled

Details

7e6e32488b

pkulev requested changes 2026-09-15 18:11:50 +02:00

Dismissed

pkulev left a comment

Re-review of 7e6e324 on CHICKEN 6.0.0. Previous items on comments and make run-tests are fixed. The issue #16 success program still does not compile.

What I ran

Check	Result
`make run-tests`	pass (`....` — hello-world, lists, comments, unicode)
unit tests incl. new `codegen` / `args`	pass
issue #16 golden file via `sexc -C`	still rejected by `cc`

Fixed since last review

tests/sex-programs/comments.sex exists.
Consecutive ;;; lines pack, and leading ;s are stripped: /* Main entry point / Multi line comments should be packed */.
Comments in if / for / while / var / cast no longer steal positional slots. The new codegen tests pin this, including the silent else if case.
Stray commas in calls are gone ((g 1 ;; c\n 2) → g(1, 2)).
Useful extras: -- now passes non-dash compiler args; c-cast parenthesises binary operands; c-switch runs the scrutinee through c-expr.

Still blocking Closes #16

| / || are still not C operators. Same golden file as last time:

struct vertex a = {{10, 20}, {0.3, 0.5, 0.7}, |\||(B, A, R)};

and (|| 0 1) still emits |\|\||(0, 1). cc dies on expected expression before '|' / stray '\'.

The reader already produces those symbols. walk-expr still only special-cases c-or / c-bit-or / c-bit-or= (and those still work: 1 | 2 | 4). c-expr/sexp already has the (string->symbol ".") cond — "|", "||", "|=" belong there. There is still no codegen test for this, which is why the new suite is green while #16 is not.

Everything else in the sample is right: packed comments, . field access, plus, struct/enum/vectors. The printf comment is dropped rather than kept as /* the printf */ between arguments; that matches the new tests, but it is a remaining gap vs the issue's sketched C.

CI still runs only make sex-tests, so this | hole would not show up there either.

I would not merge until (| B A R) and (|| …) emit B | A | R / a || b and that is pinned next to the other codegen cases.

Re-review of `7e6e324` on CHICKEN 6.0.0. Previous items on comments and `make run-tests` are fixed. The issue #16 success program still does not compile. ## What I ran | Check | Result | |---|---| | `make run-tests` | pass (`....` — hello-world, lists, comments, unicode) | | unit tests incl. new `codegen` / `args` | pass | | issue #16 golden file via `sexc -C` | still rejected by `cc` | ## Fixed since last review - `tests/sex-programs/comments.sex` exists. - Consecutive `;;;` lines pack, and leading `;`s are stripped: `/* Main entry point` / `Multi line comments should be packed */`. - Comments in `if` / `for` / `while` / `var` / `cast` no longer steal positional slots. The new `codegen` tests pin this, including the silent `else if` case. - Stray commas in calls are gone (`(g 1 ;; c\n 2)` → `g(1, 2)`). - Useful extras: `--` now passes non-dash compiler args; `c-cast` parenthesises binary operands; `c-switch` runs the scrutinee through `c-expr`. ## Still blocking Closes #16 **`|` / `||` are still not C operators.** Same golden file as last time: ```c struct vertex a = {{10, 20}, {0.3, 0.5, 0.7}, |\||(B, A, R)}; ``` and `(|| 0 1)` still emits `|\|\||(0, 1)`. `cc` dies on `expected expression before '|'` / `stray '\'`. The reader already produces those symbols. `walk-expr` still only special-cases `c-or` / `c-bit-or` / `c-bit-or=` (and those still work: `1 | 2 | 4`). `c-expr/sexp` already has the `(string->symbol ".")` cond — `"|"`, `"||"`, `"|="` belong there. There is still no codegen test for this, which is why the new suite is green while #16 is not. Everything else in the sample is right: packed comments, `.` field access, `plus`, struct/enum/vectors. The printf comment is dropped rather than kept as `/* the printf */` between arguments; that matches the new tests, but it is a remaining gap vs the issue's sketched C. CI still runs only `make sex-tests`, so this `|` hole would not show up there either. I would not merge until `(| B A R)` and `(|| …)` emit `B | A | R` / `a || b` and that is pinned next to the other `codegen` cases.

fmt-c-writer.scm Outdated

						
				@@ -85,4 +192,4 @@

				    (('enum . _) (walk-enum form))

				    ;; | is problematic... And c-or/bit-or/etc are actually

				    ;; procedures, so we have to call the procedure itself

				    (('c-or . rest) (apply c-or (map walk-expr rest)))

pkulev commented

2026-09-15 18:11:50 +02:00

This is the remaining #16 hole. After the new reader, (| B A R) is the symbol |, not c-bit-or. sexc -C of the issue sample still emits \|\\||(B, A, R) and cc rejects it. Same for ||. The (string->symbol ".") cond in c-expr/sexp is the pattern; c-bit-or can stay as an alias.

This is the remaining #16 hole. After the new reader, `(| B A R)` is the symbol `|`, not `c-bit-or`. `sexc -C` of the issue sample still emits `\|\\||(B, A, R)` and `cc` rejects it. Same for `||`. The `(string->symbol ".")` cond in `c-expr/sexp` is the pattern; `c-bit-or` can stay as an alias.

tests/codegen.scm

						
				@@ -0,0 +77,4 @@

				  ;; A comment among a call's arguments used to become an argument,

				  ;; and c-apply put a comma on each side of it -- which does not

				  ;; compile. It is dropped, as in any other expression context.

				  (test-group "comments among arguments"

pkulev commented

2026-09-15 18:11:50 +02:00

These tests are why the suite is green while the #16 sample still fails: they pin comments and casts, but not (| B A R) → B | A | R or (|| a b) → a || b. A couple of emits? cases here would have caught it.

These tests are why the suite is green while the #16 sample still fails: they pin comments and casts, but not `(| B A R)` → `B | A | R` or `(|| a b)` → `a || b`. A couple of `emits?` cases here would have caught it.

alex-eg added 1 commit 2026-09-15 19:24:12 +02:00

support || | |= operators, since we now have our own parser

Sex CI / build-linux (pull_request) Failing after 3m1s

Details

Sex CI / build-linux (push) Failing after 3m1s

Details

Sex CI / build-macos (push) Has been cancelled

Details

Sex CI / build-macos (pull_request) Has been cancelled

Details

5f9f90ef37

pkulev approved these changes 2026-09-15 19:40:32 +02:00

pkulev left a comment

Re-review of 5f9f90e on CHICKEN 6.0.0. The last blocker is gone.

What I ran

Check	Result
`make run-tests`	pass, including new `
issue #16 golden file via `sexc -C` then `cc`	compiles
running it	`30 1.500000 3`
`(

(| B A R) now emits B | A | R. Renaming | / || / \|= in atom-to-fmt-c to heads fmt-c can actually spell is the right fix, and the tests pin both the new spelling and the old c-or / c-bit-or aliases.

Remaining nits, not blocking

A ; among call arguments is still dropped, so the issue sketch's /* the printf */ does not appear. That matches the tests; it is the one leftover vs the #16 C sample.
Packed block comments indent the continuation line (/* Main entry point / Multi line… */). Harmless.
CI still runs only make sex-tests. sextest and the golden sample would not run there.

Looks good to merge for Closes #16.

Re-review of `5f9f90e` on CHICKEN 6.0.0. The last blocker is gone. ## What I ran | Check | Result | |---|---| | `make run-tests` | pass, including new `|` / `||` / `\|=` codegen tests | | issue #16 golden file via `sexc -C` then `cc` | compiles | | running it | `30 1.500000 3` | | `(|| 0 1)` | `if (0 \|\| 1)` and runs | `(| B A R)` now emits `B | A | R`. Renaming `|` / `||` / `\|=` in `atom-to-fmt-c` to heads fmt-c can actually spell is the right fix, and the tests pin both the new spelling and the old `c-or` / `c-bit-or` aliases. ## Remaining nits, not blocking - A `;` among call arguments is still dropped, so the issue sketch's `/* the printf */` does not appear. That matches the tests; it is the one leftover vs the #16 C sample. - Packed block comments indent the continuation line (`/* Main entry point` / ` Multi line… */`). Harmless. - CI still runs only `make sex-tests`. `sextest` and the golden sample would not run there. Looks good to merge for Closes #16.

alex-eg merged commit 5f9f90ef37 into main

2026-09-15 19:43:14 +02:00

alex-eg deleted branch our-parser

2026-09-15 19:43:15 +02:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: alex-eg/sex#26

				`@@ -88,6 +137,28 @@`
				`(('c-or . rest) (apply c-or (map walk-expr rest)))`