init: Add Token definition and Tokenizer logic.

2026-01-12 16:06:55 +01:00
commit 0599a5fb98
6 changed files with 378 additions and 0 deletions
--- a/.gitignore
+++ b/.gitignore
@@ -0,0 +1,8 @@
+/target
+
+
+# Added by cargo
+#
+# already existing elements were commented out
+
+#/target