Our reader/tokenizer #16

New Issue

alex-eg · 2025-10-24T12:39:11+02:00

alex-eg commented

2025-10-24 12:39:11 +02:00

(Migrated from github.com)

Chicken reader has two issues: reserved characters and comments.

= Reserved characters =
| and .. Due to this, we cannot have operators for bitwise or, bitwise or with assignment, logical or, and field reference. I.e the following is not currently possible:

(|| (is-integer n) (is-float n)) ; now must be (c-or (is-integer n)
                                 ; (is-float n))
(| MOVABLE SCALABLE MINIMIZABLE) ; (c-bit-or MOVABLE SCALABLE
                                 ; MINIMIZABLE)
(. settings window scale x) ; settings.window.scale.x -- particularly
                            ; inconvenient for writing macros, since
                            ; it requires creating new symbols

Additionally, while we can just ignore |s original meaning, forbidding symbols that require |, we can't ignore ., since we still need the original reader at least for defmacros, and . is very important in constructions like match, and in general in list processing.

= Comments =
Comments are lesser issue, but it would be nice to preserve them in generated C code.

= Success criteria =
So the success criterion for this feature is to be able to correctly read and process Sex source containing both Scheme and Sex code:

(include stdio.h)

(defmacro (plus . rest)
  `(+ ,@rest))

(struct vertex
  ((pos (struct ((x u32) (y u32))))
   (color (struct ((r float) (g float) (b float))))
   (mask u32)))

(enum
 (B A R))

;;; Main entry point
;;; Multi line comments should be packed
(pub fn main () int
  ;; the vertex
  (var a (struct vertex) #(#(10 20)
                           #(0.3 0.5 0.7)
                           (| B A R)))
  (printf "%d %f %d\n" ; the printf
          (plus (. a pos x) (. a pos y))
          (plus (. a color r) (. a color g) (. a color b))
          (. a mask))
  (return 0))

after processing should become:

struct vertex {
    struct{
        u32 x;
        u32 y;
    } pos;
    struct{
        float r;
        float g;
        float b;
    } color;
    u32 mask;
};

enum {
    B, A, R
};

/* Main entry point
Multi line comments should be packed */
int main (void) {
    /* the vertex */
    struct vertex a = {{10, 20}, {0.3, 0.5, 0.7}, B | A | R};
    printf("%d %f %d\n", /* the printf */
           a.pos.x + a.pos.y,
           a.color.r + a.color.g + a.color.b,
           a.mask);
    return 0;
}

Chicken reader has two issues: reserved characters and comments. **= Reserved characters =** `|` and `.`. Due to this, we cannot have operators for bitwise or, bitwise or with assignment, logical or, and field reference. I.e the following is not currently possible: ```scheme (|| (is-integer n) (is-float n)) ; now must be (c-or (is-integer n) ; (is-float n)) (| MOVABLE SCALABLE MINIMIZABLE) ; (c-bit-or MOVABLE SCALABLE ; MINIMIZABLE) (. settings window scale x) ; settings.window.scale.x -- particularly ; inconvenient for writing macros, since ; it requires creating new symbols ``` Additionally, while we can just ignore `|`s original meaning, forbidding symbols that require `|`, we can't ignore `.`, since we still need the original reader at least for `defmacro`s, and `.` is very important in constructions like `match`, and in general in list processing. **= Comments =** Comments are lesser issue, but it would be nice to preserve them in generated C code. **= Success criteria =** So the success criterion for this feature is to be able to correctly read and process Sex source containing both Scheme and Sex code: ```scheme (include stdio.h) (defmacro (plus . rest) `(+ ,@rest)) (struct vertex ((pos (struct ((x u32) (y u32)))) (color (struct ((r float) (g float) (b float)))) (mask u32))) (enum (B A R)) ;;; Main entry point ;;; Multi line comments should be packed (pub fn main () int ;; the vertex (var a (struct vertex) #(#(10 20) #(0.3 0.5 0.7) (| B A R))) (printf "%d %f %d\n" ; the printf (plus (. a pos x) (. a pos y)) (plus (. a color r) (. a color g) (. a color b)) (. a mask)) (return 0)) ``` after processing should become: ```c struct vertex { struct{ u32 x; u32 y; } pos; struct{ float r; float g; float b; } color; u32 mask; }; enum { B, A, R }; /* Main entry point Multi line comments should be packed */ int main (void) { /* the vertex */ struct vertex a = {{10, 20}, {0.3, 0.5, 0.7}, B | A | R}; printf("%d %f %d\n", /* the printf */ a.pos.x + a.pos.y, a.color.r + a.color.g + a.color.b, a.mask); return 0; } ```

alex-eg referenced a pull request that will close this issue

2026-09-09 18:38:16 +02:00

Implement our parser, . field access, comment preservation and line number tracking for source-level debugging #26

pkulev referenced this issue

2026-09-14 23:40:08 +02:00

Implement our parser, . field access, comment preservation and line number tracking for source-level debugging #26

pkulev referenced this issue

2026-09-14 23:40:08 +02:00

Implement our parser, . field access, comment preservation and line number tracking for source-level debugging #26

pkulev referenced this issue

2026-09-14 23:40:08 +02:00

Implement our parser, . field access, comment preservation and line number tracking for source-level debugging #26