refactor tokenizer #19
Description
Refactor should include a complete overhaul, better utilizing cursor and AST.
For example, maybe remove repetition in code as seen here:
(can't think of a better alternative at the moment)
{{ message }}
Refactor should include a complete overhaul, better utilizing cursor and AST.
For example, maybe remove repetition in code as seen here:
(can't think of a better alternative at the moment)