Add type db, some nice macro features and a brand new SDL3 example #27

Built and ran this on CHICKEN 6.0.0 (worktree at 85bf1c1). This is the real thing: a type database the macros can actually ask questions of, plus an SDL3 program that only exists because the earlier codegen holes got closed.

What I ran

Check	Result
`make run-tests`	pass (unit + hello-world/lists/comments/unicode/serialize + `tests/modules`)
`example/serialize.sex` compile and run	`point { x=0 y=0 }` / `point { x=640 y=-480 }` / `person { name=Alex age=34 height=1.82 }`
`serialize-struct` on an imported struct	`serialize-struct: no such struct: point`
`(pub enum color …)` same file or via import	`pub must be followed by a definition`
nested `(* (* char))`	rejected, with `file:line:`
unknown toplevel form	`bad.sex:2: unknown top level form: (wat 1 2)`
`sexc -C` of `example/sdl3-triangle.sex`	emits `switch (event.type)`, `(void )(2 sizeof(GLfloat))`, `SDL_GLContext`
documented triangle build on Linux	`cc` dies on `-framework`; `sexc` still exits 0
cc argv after the temp-file change	`cc -o out /tmp/temp….c` (extension is `.c`, so dropping `-x c` is correct)

What is good

The type database is the feature. map-fields + type-match turn a struct name into a printer; example/serialize.sex is the proof, and tests/sex-programs/serialize.sex pins it. Macros get cat, a no-op comment, and read access to the DB, which is the right surface.

Module linking was actually wrong before: an imported pub fn lost its pub and became static, and an imported pub var became a private copy with an initializer. Keeping pub on prototypes and rewriting vars to extern is the correct C model, and tests/modules would have caught the old behaviour (it links and checks that greet-count is shared).

sex-error citing file:line is what the location map was for. Rejecting nested pointer chains instead of silently dropping a * is the same kind of fix. Writing a temp .c rather than stdin is simpler and lets #line / -g point at a real file; the .c suffix is why -x c can go.

The triangle is why this compiler is worth it. The generated C is what you would have written by hand: core-profile 3.3, uniforms for centre/spin/radius, mouse follow, LMB to spin, Esc to quit. switch (. event type) surviving as switch (event.type) is the previous PR paying off.

Remaining, not blocking

Imported types never enter the DB. process-imports registers macros, then conses everything else. register-aggregate! never runs for a pub struct that arrived through import, so the serialize example does not work across modules. The TODO on process-public-interface-form already says this; match-sex-form on those pasted forms is the obvious next step.
(pub enum …) is rejected both as a public interface and by walk-public (struct/union/typedef are handled; enum is not). A non-pub enum compiles. The error from the writer also has no file:line:, because it is handed the cdr of the pub form.
get-fields / map-fields do not chase typedefs. (typedef point-t point) plus (serialize-struct point-t) fails; get-underlying-type returns point and the macro would have to take that extra step. Fine if documented; easy to trip over.
sexc still exits 0 when cc fails. compile-to-file now returns process-wait's values, but main drops them. sextest treats a 0 as "compiled".
Temp .c is deleted after cc returns, but leaks if emit-c throws (nested-pointer error left /tmp/temp….c behind).
The triangle is Darwin-shaped: OpenGL/gl3.h and -framework OpenGL. On Linux that include is missing and -framework is not a gcc flag. Worth a one-line "this one is macOS" in the header, or SDL3/SDL_opengl.h plus a GL loader if you want it to build where CI runs.
tests/types.module.scm exports *, so the test build warns that semen.scm redefines imported comment-form?. Production types.module.scm is explicit and does not.

Looks good to merge. The module/typedef holes are the natural follow-up, not a reason to sit on this.

Built and ran this on CHICKEN 6.0.0 (worktree at `85bf1c1`). This is the real thing: a type database the macros can actually ask questions of, plus an SDL3 program that only exists because the earlier codegen holes got closed. ## What I ran | Check | Result | |---|---| | `make run-tests` | pass (unit + hello-world/lists/comments/unicode/serialize + `tests/modules`) | | `example/serialize.sex` compile and run | `point { x=0 y=0 }` / `point { x=640 y=-480 }` / `person { name=Alex age=34 height=1.82 }` | | `serialize-struct` on an **imported** struct | `serialize-struct: no such struct: point` | | `(pub enum color …)` same file or via import | `pub must be followed by a definition` | | nested `(* (* char))` | rejected, with `file:line:` | | unknown toplevel form | `bad.sex:2: unknown top level form: (wat 1 2)` | | `sexc -C` of `example/sdl3-triangle.sex` | emits `switch (event.type)`, `(void *)(2 * sizeof(GLfloat))`, `SDL_GLContext` | | documented triangle build on Linux | `cc` dies on `-framework`; `sexc` still exits 0 | | cc argv after the temp-file change | `cc -o out /tmp/temp….c` (extension is `.c`, so dropping `-x c` is correct) | ## What is good The type database is the feature. `map-fields` + `type-match` turn a struct name into a printer; `example/serialize.sex` is the proof, and `tests/sex-programs/serialize.sex` pins it. Macros get `cat`, a no-op `comment`, and read access to the DB, which is the right surface. Module linking was actually wrong before: an imported `pub fn` lost its `pub` and became `static`, and an imported `pub var` became a private copy with an initializer. Keeping `pub` on prototypes and rewriting vars to `extern` is the correct C model, and `tests/modules` would have caught the old behaviour (it links and checks that `greet-count` is shared). `sex-error` citing `file:line` is what the location map was for. Rejecting nested pointer chains instead of silently dropping a `*` is the same kind of fix. Writing a temp `.c` rather than stdin is simpler and lets `#line` / `-g` point at a real file; the `.c` suffix is why `-x c` can go. The triangle is why this compiler is worth it. The generated C is what you would have written by hand: core-profile 3.3, uniforms for centre/spin/radius, mouse follow, LMB to spin, Esc to quit. `switch (. event type)` surviving as `switch (event.type)` is the previous PR paying off. ## Remaining, not blocking - **Imported types never enter the DB.** `process-imports` registers macros, then `cons`es everything else. `register-aggregate!` never runs for a `pub struct` that arrived through `import`, so the serialize example does not work across modules. The TODO on `process-public-interface-form` already says this; `match-sex-form` on those pasted forms is the obvious next step. - **`(pub enum …)` is rejected** both as a public interface and by `walk-public` (struct/union/typedef are handled; enum is not). A non-`pub` enum compiles. The error from the writer also has no `file:line:`, because it is handed the cdr of the `pub` form. - **`get-fields` / `map-fields` do not chase typedefs.** `(typedef point-t point)` plus `(serialize-struct point-t)` fails; `get-underlying-type` returns `point` and the macro would have to take that extra step. Fine if documented; easy to trip over. - **`sexc` still exits 0 when `cc` fails.** `compile-to-file` now returns `process-wait`'s values, but `main` drops them. `sextest` treats a 0 as "compiled". - Temp `.c` is deleted after `cc` returns, but **leaks if `emit-c` throws** (nested-pointer error left `/tmp/temp….c` behind). - The triangle is Darwin-shaped: `OpenGL/gl3.h` and `-framework OpenGL`. On Linux that include is missing and `-framework` is not a gcc flag. Worth a one-line "this one is macOS" in the header, or `SDL3/SDL_opengl.h` plus a GL loader if you want it to build where CI runs. - `tests/types.module.scm` exports `*`, so the test build warns that `semen.scm` redefines imported `comment-form?`. Production `types.module.scm` is explicit and does not. Looks good to merge. The module/typedef holes are the natural follow-up, not a reason to sit on this.

example/sdl3-triangle.sex Outdated

						
				@@ -0,0 +13,4 @@

				(define GL-SILENCE-DEPRECATION 1)       ; OpenGL is deprecated on macOS

				(include SDL3/SDL.h)

pkulev commented

2026-09-15 21:15:56 +02:00

This, plus -framework OpenGL in the build line, is Apple's GL 3.3. On Linux OpenGL/gl3.h is missing and gcc does not know -framework; swapping in SDL3/SDL_opengl.h is not enough either (no glCreateShader). Fine as a macOS example — worth saying so in the header, since CI is Ubuntu.

This, plus `-framework OpenGL` in the build line, is Apple's GL 3.3. On Linux `OpenGL/gl3.h` is missing and gcc does not know `-framework`; swapping in `SDL3/SDL_opengl.h` is not enough either (no `glCreateShader`). Fine as a macOS example — worth saying so in the header, since CI is Ubuntu.

pkulev marked this conversation as resolved

fmt-c-writer.scm

pkulev commented

2026-09-15 21:15:56 +02:00

(pub enum …) dies here: struct/union/typedef are forwarded, enum is not. Same-file (pub enum color (red green blue)) is pub must be followed by a definition with no file:line: (this is the cdr of the pub form, which has no location of its own). Non-pub enums emit fine.

`(pub enum …)` dies here: struct/union/typedef are forwarded, enum is not. Same-file `(pub enum color (red green blue))` is `pub must be followed by a definition` with no `file:line:` (this is the cdr of the `pub` form, which has no location of its own). Non-`pub` enums emit fine.

semen.scm

						
				@@ -212,2 +218,4 @@

				  (register-aggregate! sex-struct)

				  (cons sex-struct acc))

				(define (register-aggregate! form)

pkulev commented

2026-09-15 21:15:56 +02:00

This is what makes serialize-struct work — but only for aggregates that go through match-sex-form. process-imports still just conses public structs, so a pub struct living in another module is visible as C and invisible to map-fields. I compiled that: serialize-struct: no such struct: point. The TODO on the module reader is this.

This is what makes `serialize-struct` work — but only for aggregates that go through `match-sex-form`. `process-imports` still just `cons`es public structs, so a `pub struct` living in another module is visible as C and invisible to `map-fields`. I compiled that: `serialize-struct: no such struct: point`. The TODO on the module reader is this.

pkulev marked this conversation as resolved

sex-modules.scm Outdated

						
				@@ -76,0 +75,4 @@

				       ;; A variable becomes an `extern' declaration

				       ((var)

				        (cons (copy-form-source! form (cons 'extern (take (cdr form) 3))) acc))

				       ((define defmacro import include struct typedef union)

pkulev commented

2026-09-15 21:15:56 +02:00

Same hole on import: struct typedef union are pasted, enum is not, so --public-interface on a module that exports an enum is pub must be followed by a definition and cites the location correctly. Adding enum next to union is the whitelist half; walk-public is the other.

Same hole on import: `struct typedef union` are pasted, `enum` is not, so `--public-interface` on a module that exports an enum is `pub must be followed by a definition` and cites the location correctly. Adding `enum` next to `union` is the whitelist half; `walk-public` is the other.

pkulev marked this conversation as resolved

sexc.scm Outdated

						
				@@ -140,0 +134,4 @@

				                                                (list))

				                                            (list c-file)

				                                            cc-args))))

				        (call-with-values (lambda () (process-wait proc))

pkulev commented

2026-09-15 21:15:56 +02:00

These wait values never become the process exit code: main ignores what compile-to-file returns, so sexc file.sex -- -bogus-flag prints the cc error and exits 0. sextest then treats that as a successful compile. Also delete-file* only runs after wait; if emit-c throws, the temp .c stays in /tmp.

These wait values never become the process exit code: `main` ignores what `compile-to-file` returns, so `sexc file.sex -- -bogus-flag` prints the cc error and exits 0. `sextest` then treats that as a successful compile. Also `delete-file*` only runs after wait; if `emit-c` throws, the temp `.c` stays in `/tmp`.

alex-eg added 8 commits 2026-09-16 17:01:28 +02:00

process imported forms as toplevel, not as text d0d0ea8e84

1. register imported types
2. protect against multiple imports (diamond, circular)

accept pub enum, and enums used as types 847bb14340

Implement pub support for enums, and also anonymous enums declared
in-place.

Also give the cdr of a `pub' form the form's own location, so a
error about what follows `pub' can say where it was written

resolve typedefs in get-fields and map-fields 97ef62b7f4

Now these macro helpers take in account the possibility of typedefing
one type to another, and correctly find the one intended

export explicitly from the test build's types module 9763f5fa8d

fail when the C compiler fails, and clean up when we do 294a275905

compile-to-file returned process-wait's values and main dropped them,
so cc errors were printed, but then main compiler exited 0.

Also cleanup tmp C files when compilation failed.

make sextest's (compilation ...) form work 6e4cb80424

1. Look for `compilation', not `compile' among the source
2. Read the file with provided --features from the (compilation ...)
form, then handle resulting file to sexc

read-time feature expressions bf83baa508

'#+' and '#-' introduce conditional compilation: the form that follows
is kept only when the feature expression is true, and otherwise is read
and thrown away.  An expression is a feature name, or and / or / not
of them.

They are read time, not compile time.

Default features are the host's software-version, software-type and
machine-type as CHICKEN reports them, plus what --features flag adds.

build the triangle on Linux too

Sex CI / build-linux (pull_request) Successful in 4m44s

Details

Sex CI / build-linux (push) Successful in 5m18s

Details

a221e0f8ea

<OpenGL/gl3.h> does not exist there and -framework is not a gcc flag,
so the documented build line failed with no hint why. #+macosx picks
the header now, and both build lines are in the file -- pkg-config
carries the flags on either platform, apart from Apple's GL framework,
which has no pkg-config file to carry.

alex-eg requested review from pkulev 2026-09-16 22:28:21 +02:00

alex-eg commented

2026-09-16 23:35:11 +02:00

Thanks — every one of these reproduced, and all seven are fixed, each with the test that would have caught it.

#	Item	Fix
1	imported types never enter the DB	`d0d0ea8`
2	`(pub enum …)` rejected, no `file:line:`	`847bb14`
3	`get-fields`/`map-fields` don't chase typedefs	`97ef62b`
4	exit 0 after `cc` fails	`294a275`
5	temp `.c` leaks when emission throws	`294a275`
6	triangle is Darwin-shaped	`bf83baa`, `a221e0f`
7	`tests/types.module.scm` exports `*`	`9763f5f`

1 — imported types. You were right that match-sex-form on the pasted forms is the next step; it turned out to be the whole step. process-imports is now (fold match-sex-form acc module-public-forms), and the hand-rolled defmacro case is gone, since match-sex-form already had one. That also fixed a latent one: a transitive pub import used to be pasted as an (import …) form and reached the writer as if it were a function call. Following imports for real made an include guard necessary — a diamond would otherwise paste the shared module twice into one translation unit, and a cycle would not terminate — so a module is now pasted at most once per unit, keyed by absolute path.

2 — pub enum. walk-public and the module whitelist both have it now. Fixing that exposed a layer underneath: an enum can be written as a type, not only as a definition, and neither side could render it. (var m (enum color) red) crashed in c-enum, and (var e (enum (up down)) up) emitted enum e { up, down } — c-type was passing the declarator's name in as the enum's tag, so the variable vanished. Both now do what struct already did: c-enum renders a bare tag the way c-struct/aux renders struct point p, and c-type lays the type out before the name being declared. Marked MODIFIED FROM UPSTREAM, next to the cast fix. The pub cdr carries the form's location now too — though with enum handled, that branch is only reachable if semen and the writer disagree again.

3 — typedefs. Resolved rather than documented: get-fields and map-fields follow the alias, through (typedef point-t point) and (typedef point-t (struct point)) alike, and through chains. get-type-info still returns what was written, for a caller that wants to see the typedef. One consequence worth naming: an enumerator's type now comes back named as the enum was declared — enum mood, never enum mood_t, which is not a C type. get-underlying-type also stops on a typedef that leads back to itself; nothing prevents one from being written, and it used to spin.

4 and 5. compile-to-file returns cc's status and main exits with it. The temp file needed handle-exceptions, not dynamic-wind — an unhandled error ends the process without unwinding, which is exactly why the file survived. Both are failure-path behaviour that no unit test can reach, so they live in tests/exit-code/, next to the module-linking test, and make run-tests runs them.

6 — the triangle. This one grew a feature. Rather than document the header swap, Sex now reads #+ and #- feature expressions the way Common Lisp does (bf83baa), and the example picks its GL header with them:

#+macosx (include OpenGL/gl3.h)
#-macosx (define GL-GLEXT-PROTOTYPES 1)
#-macosx (include GL/gl.h)
#-macosx (include GL/glext.h)

Read time, not compile time, which is the whole difference from cpp's #ifdef: what does not apply never reaches macro expansion, the type database or the writer, so a guarded form may name things that do not exist on the other platform at all. Features are the host's — software-version, software-type, machine-type as CHICKEN reports them — plus --features. Both build lines are in the file; pkg-config carries the flags on either platform, apart from Apple's GL framework, which has no pkg-config file to carry.

Since this machine cannot compile the Linux branch, --no-platform-features reads a file as another platform would, which is how that branch was checked:

$ ./sexc example/sdl3-triangle.sex -C --no-platform-features --features=linux,unix,x86-64
#include <SDL3/SDL.h>
#define GL_GLEXT_PROTOTYPES 1
#include <GL/gl.h>
#include <GL/glext.h>

SDL3/SDL_opengl.h stayed out of it: on Apple it resolves to the 2.1 header, which has no glGenVertexArrays and so cannot bind the VAO this program needs.

That work turned up one more thing. sextest's (compilation …) form never worked — it looked for compile among the settings while the reader collects compilation, so the flags were always #f and appending them built an improper argv. Fixed in 6e4cb80, along with a subtler problem it exposed: sextest reads the program and prints it back to sexc, so #+/#- were resolved on its side with its own feature list. It reads the file again with whatever --features the compilation flags ask for, and the reader now defaults to the host's features, so anything reading Sex sees what the compiler would.

7. Same export list as production, so the suite tests what ships. No warning in the test build any more.

Verification, on top of make run-tests (unit + six programs + tests/modules + the new exit-code check):

$ ./sexc app.sex -o app && ./app      # struct, typedef and enum, all imported
{ x=3 y=4 }
{ x=5 y=6 }
hue=1

$ ./sexc shapes.sex --public-interface
(struct point ((x int) (y int)))
(typedef point-t (struct point))
(enum hue (red green blue))

$ ./sexc app.sex -o app -- -no-such-flag; echo $?
1

tests/modules now also expands a macro that reads an imported struct's fields — directly and through the imported typedef — so the cross-module hole cannot come back silently, and tests/sex-programs/features.sex pins the feature expressions end to end, including a whole main dropped by #-.

Thanks — every one of these reproduced, and all seven are fixed, each with the test that would have caught it. | # | Item | Fix | |---|---|---| | 1 | imported types never enter the DB | `d0d0ea8` | | 2 | `(pub enum …)` rejected, no `file:line:` | `847bb14` | | 3 | `get-fields`/`map-fields` don't chase typedefs | `97ef62b` | | 4 | exit 0 after `cc` fails | `294a275` | | 5 | temp `.c` leaks when emission throws | `294a275` | | 6 | triangle is Darwin-shaped | `bf83baa`, `a221e0f` | | 7 | `tests/types.module.scm` exports `*` | `9763f5f` | **1 — imported types.** You were right that `match-sex-form` on the pasted forms is the next step; it turned out to be the whole step. `process-imports` is now `(fold match-sex-form acc module-public-forms)`, and the hand-rolled `defmacro` case is gone, since `match-sex-form` already had one. That also fixed a latent one: a transitive `pub import` used to be pasted as an `(import …)` form and reached the writer as if it were a function call. Following imports for real made an include guard necessary — a diamond would otherwise paste the shared module twice into one translation unit, and a cycle would not terminate — so a module is now pasted at most once per unit, keyed by absolute path. **2 — `pub enum`.** `walk-public` and the module whitelist both have it now. Fixing that exposed a layer underneath: an enum can be written as a *type*, not only as a definition, and neither side could render it. `(var m (enum color) red)` crashed in `c-enum`, and `(var e (enum (up down)) up)` emitted `enum e { up, down }` — `c-type` was passing the declarator's name in as the enum's tag, so the variable vanished. Both now do what `struct` already did: `c-enum` renders a bare tag the way `c-struct/aux` renders `struct point p`, and `c-type` lays the type out before the name being declared. Marked MODIFIED FROM UPSTREAM, next to the cast fix. The `pub` cdr carries the form's location now too — though with enum handled, that branch is only reachable if semen and the writer disagree again. **3 — typedefs.** Resolved rather than documented: `get-fields` and `map-fields` follow the alias, through `(typedef point-t point)` and `(typedef point-t (struct point))` alike, and through chains. `get-type-info` still returns what was written, for a caller that wants to see the typedef. One consequence worth naming: an enumerator's type now comes back named as the enum was *declared* — `enum mood`, never `enum mood_t`, which is not a C type. `get-underlying-type` also stops on a typedef that leads back to itself; nothing prevents one from being written, and it used to spin. **4 and 5.** `compile-to-file` returns `cc`'s status and `main` exits with it. The temp file needed `handle-exceptions`, not `dynamic-wind` — an unhandled error ends the process without unwinding, which is exactly why the file survived. Both are failure-path behaviour that no unit test can reach, so they live in `tests/exit-code/`, next to the module-linking test, and `make run-tests` runs them. **6 — the triangle.** This one grew a feature. Rather than document the header swap, Sex now reads `#+` and `#-` feature expressions the way Common Lisp does (`bf83baa`), and the example picks its GL header with them: ```scheme #+macosx (include OpenGL/gl3.h) #-macosx (define GL-GLEXT-PROTOTYPES 1) #-macosx (include GL/gl.h) #-macosx (include GL/glext.h) ``` Read time, not compile time, which is the whole difference from cpp's `#ifdef`: what does not apply never reaches macro expansion, the type database or the writer, so a guarded form may name things that do not exist on the other platform at all. Features are the host's — `software-version`, `software-type`, `machine-type` as CHICKEN reports them — plus `--features`. Both build lines are in the file; pkg-config carries the flags on either platform, apart from Apple's GL framework, which has no pkg-config file to carry. Since this machine cannot compile the Linux branch, `--no-platform-features` reads a file as another platform would, which is how that branch was checked: ``` $ ./sexc example/sdl3-triangle.sex -C --no-platform-features --features=linux,unix,x86-64 #include <SDL3/SDL.h> #define GL_GLEXT_PROTOTYPES 1 #include <GL/gl.h> #include <GL/glext.h> ``` `SDL3/SDL_opengl.h` stayed out of it: on Apple it resolves to the 2.1 header, which has no `glGenVertexArrays` and so cannot bind the VAO this program needs. That work turned up one more thing. sextest's `(compilation …)` form never worked — it looked for `compile` among the settings while the reader collects `compilation`, so the flags were always `#f` and appending them built an improper argv. Fixed in `6e4cb80`, along with a subtler problem it exposed: sextest reads the program and prints it back to sexc, so `#+`/`#-` were resolved on its side with its own feature list. It reads the file again with whatever `--features` the compilation flags ask for, and the reader now defaults to the host's features, so anything reading Sex sees what the compiler would. **7.** Same export list as production, so the suite tests what ships. No warning in the test build any more. Verification, on top of `make run-tests` (unit + six programs + `tests/modules` + the new exit-code check): ``` $ ./sexc app.sex -o app && ./app # struct, typedef and enum, all imported { x=3 y=4 } { x=5 y=6 } hue=1 $ ./sexc shapes.sex --public-interface (struct point ((x int) (y int))) (typedef point-t (struct point)) (enum hue (red green blue)) $ ./sexc app.sex -o app -- -no-such-flag; echo $? 1 ``` `tests/modules` now also expands a macro that reads an imported struct's fields — directly and through the imported typedef — so the cross-module hole cannot come back silently, and `tests/sex-programs/features.sex` pins the feature expressions end to end, including a whole `main` dropped by `#-`.

pkulev referenced this pull request

2026-09-17 23:05:51 +02:00

feature/project-local-deps #29

pkulev approved these changes 2026-09-18 00:09:38 +02:00

pkulev left a comment

LGTM

Makefile

						
				@@ -62,0 +69,4 @@

				# The failure paths are checked end to end; see tests/exit-code/Makefile.

				check-exit-code: sexc

					@$(MAKE) --no-print-directory -C ./tests/exit-code check SEXC=../../sexc

pkulev commented

2026-09-17 20:15:30 +02:00

Strange place to for that definition btw. But I see, the tests/ layout is kinda per-subproject, so probably that makes sense, but lookin' alien to me.

reader.scm

						
				@@ -176,0 +198,4 @@

				   ((symbol? test) (and (memq test (current-features)) #t))

				   ((pair? test)

				    (case (car test)

				      ((and) (every feature-true? (cdr test)))