How does Claktor measure code, and why?

Claktor removes repeated logic and compresses programs to their crystallized version. Doing that reliably needs a metric. The metric is the token count of the normalized program. It measures program logic, and it is a proxy for the cost of any agent that reads the code later.

To get that count, Claktor parses each file and tokenizes a canonical form with OpenAI's open-source o200k_base vocabulary, so only the information the program carries is charged.

What changes the count?

Reformatting and renaming leave the count unchanged, removing logic lowers it total=a+b*c total = a + b * c โœ— user_account ua โœ— f(a) g(a) h(a) f(a) โœ“

Why tokens?

A line count says nothing about how long each line is. A token count does. The vocabulary is o200k_base from tiktoken, and tokens are what an agent pays to read the code.

What does the count charge for?

  • Every name the code declares becomes a single $. Long names, short names and aliases cost the same.
  • Types cost nothing. Removing an annotation or an interface moves the count by zero.
  • Whitespace and comments are removed.
  • Optional syntax has one form. Redundant parentheses, trailing commas and semicolon style do not count.
  • Python and TypeScript are priced alike. Python's newlines, indents and dedents are charged as ;, { and }.
  • The contents of strings, regexes and docstrings are charged in full.

What is left is logic, so an agent chasing a lower number has to remove logic.

What does a lower count look like in a diff?

The line count can stay the same while each line carries less.

// before
if (user !== null && user !== undefined && user.isActive === true) {

// after
if (user?.isActive === true) {

In steipete/Markdansi PR #23, three flush functions became one. That removed 50 production lines and about 180 tokens.

How do you measure a change?

~/.claktor/bin/claktor length diff --compare <start commit>

Claktor reports the normalized tokens removed since that commit. Add --json for agents. Full workflow: claktor.com/agents.md.