How does Claktor measure code, and why?
Claktor removes repeated logic and compresses programs to their crystallized version. Doing that reliably needs a metric. The metric is the token count of the normalized program. It measures program logic, and it is a proxy for the cost of any agent that reads the code later.
To get that count, Claktor parses each file and tokenizes a canonical form with OpenAI's open-source o200k_base vocabulary, so only the information the program carries is charged.
What changes the count?
Why tokens?
A line count says nothing about how long each line is. A token count does. The vocabulary is o200k_base from tiktoken, and tokens are what an agent pays to read the code.
What does the count charge for?
- Every name the code declares becomes a single
$. Long names, short names and aliases cost the same. - Types cost nothing. Removing an annotation or an interface moves the count by zero.
- Whitespace and comments are removed.
- Optional syntax has one form. Redundant parentheses, trailing commas and semicolon style do not count.
- Python and TypeScript are priced alike. Python's newlines, indents and dedents are charged as
;,{and}. - The contents of strings, regexes and docstrings are charged in full.
What is left is logic, so an agent chasing a lower number has to remove logic.
What does a lower count look like in a diff?
The line count can stay the same while each line carries less.
// before
if (user !== null && user !== undefined && user.isActive === true) {
// after
if (user?.isActive === true) {
In steipete/Markdansi PR #23, three flush functions became one. That removed 50 production lines and about 180 tokens.
How do you measure a change?
~/.claktor/bin/claktor length diff --compare <start commit>
Claktor reports the normalized tokens removed since that commit. Add --json for agents. Full workflow: claktor.com/agents.md.