History

mirivlad 2f1b822e20 feat: add scripts/build.sh, test.sh, check.sh		2026-06-16 12:11:59 +08:00
..
LICENSE	feat: add scripts/build.sh, test.sh, check.sh	2026-06-16 12:11:59 +08:00
README.md	feat: add scripts/build.sh, test.sh, check.sh	2026-06-16 12:11:59 +08:00
index.d.ts	feat: add scripts/build.sh, test.sh, check.sh	2026-06-16 12:11:59 +08:00
index.js	feat: add scripts/build.sh, test.sh, check.sh	2026-06-16 12:11:59 +08:00
package.json	feat: add scripts/build.sh, test.sh, check.sh	2026-06-16 12:11:59 +08:00

README.md

js-tokens

The tiny, regex powered, lenient, almost spec-compliant JavaScript tokenizer that never fails.

const jsTokens = require("js-tokens");

const jsString = 'JSON.stringify({k:3.14**2}, null /*replacer*/, "\\t")';

Array.from(jsTokens(jsString), (token) => token.value).join("|");
// JSON|.|stringify|(|{|k|:|3.14|**|2|}|,| |null| |/*replacer*/|,| |"\t"|)

➡️ Full readme