tur/lexer

(nil-value)

(list-nil? [lst :int])

(cons [value :int next :int])

(head [lst :int])

(tail [lst :int])

(lex-ok [x :int])

(lex-err [e :int])

(lex-ok? [r :ptr<void>])

(lex-err? [r :ptr<void>])

(lex-ok-val [r :ptr<void>])

(lex-err-val [r :ptr<void>])

(scscm-tokenize-raw [source :cstr])

scan all tokens from source into a heap array.

(scscm-rtl-error? [tl :int])

true if the raw token list contains a scan error.

(scscm-rtl-error [tl :int])

return the error message from the raw token list.

(scscm-rtl-count [tl :int])

return the number of tokens in the raw token list.

(scscm-rtl-get [tl :int i :int])

return the i-th token handle from the raw token list.

(scscm-rtl-free [tl :int])

free the raw token list struct and all token structs.

(scscm-rtl-free-shallow [tl :int])

free the STL wrapper only, not the STok structs.

(token-type [tok :int])

return the type keyword string for a token handle.

(token-type (head (lex-ok-val (tokenize "(+ 1 2)"))))  ; => ":lparen"

Since: SC0

(token-value [tok :int])

return the raw text value of a token.

(token-value tok)  ; => "foo-bar"

Since: SC0

(token-line [tok :int])

return the 1-based source line of a token.

(token-line tok)  ; => 1

Since: SC0

(token-col [tok :int])

return the 1-based source column of a token.

(token-col tok)  ; => 1

Since: SC0

(scscm-tokens-build [tl :int i :int acc :int])

(scscm-list-rev [lst :int acc :int])

(scscm-cstr-as-int [s :cstr])

(scscm-cstr-eq? [a :cstr b :cstr])

(tokenize [source :cstr])

scan scscm source text and return a cons list of token handles.

(let [res (tokenize "(+ 1 2)")]
    (if (lex-ok? res) (lex-ok-val res) (lex-err-val res)))

Since: SC0

(scscm-token-free-one [tok :int])

free one STok struct (value string + struct itself).

(tokens-free [ts :int])

free all token STok structs in a cons list.

(tokens-free my-tokens)

Since: SC0