Token Inspector

Text broken into tokens, so a prompt’s cost stops being a mystery number. Tokenisation is not words — `astralyx` is three tokens — and people shorten prose while a JSON blob below costs ten times more.

src/components/ui/token-inspector.tsx

About

From the component's own source, where it was written.

Text broken into tokens, so a prompt's cost stops being a mystery number.

The thing this makes visible: tokenisation is not words. astralyx is three tokens, a run of emoji can be one each, and indentation in a code block is often billed per level. People shorten prose to save tokens while a JSON blob three lines down costs ten times more.

**It does not tokenise.** Real tokenisation needs the model's own vocabulary, which is a multi-megabyte table this kit is not going to embed — and a component that shipped an approximation would produce numbers people quote in budgets. tokens is a required prop: hand it the output of the tokeniser you already have, and this renders it.

Alternating tints rather than gaps, so the boundaries are visible without the text reflowing into something that no longer reads like the prompt.

Install

npx astralyx-ui add token-inspector

Usage

import { TokenInspector } from '@/components/ui/token-inspector'

<TokenInspector tokens={tokenise(prompt)} pricePerThousand={0.003} showIds />

Composer

Change the props and the source below follows.

Tokens

19 · $0.0001

You·are·supporting·astralyx.↵ ↵ Refunds·above·£200·go·to·a·human.

Props

showIds

limit

tsx

<TokenInspector
  tokens={tokenise(prompt)}
  showIds={false}
  limit={19}
  pricePerThousand={0.003}
/>

Examples

A system prompt, tokenised

tsx

<TokenInspector tokens={tokens} pricePerThousand={0.003} showIds />

API

Generated from the source, so it cannot fall behind the code. One table per exported component, then hooks, functions and types.

`TokenInspector`

Extends Omit<ComponentProps<'div'>, 'children'>.

Prop	Type	Default	Description
tokens	InspectedToken[]	required	Required. This component renders; it does not tokenise — real tokenisation needs the model’s vocabulary, a multi-megabyte table, and an approximation would produce numbers people quote in budgets.
pricePerThousand	number	—	Turns the count into money.
formatCost	(cost: number) => ReactNode	(cost) => `$${cost.toFixed(4)}`	Turns the count into money.
showIds	boolean	false	Show vocabulary ids under each token.
limit	number	—	Tokens past this index are drawn as over the limit — for showing what a context ceiling would truncate.
limitLabel	string	'past the limit'	—
label	string	'Tokens'	—
emptyLabel	string	'Nothing to inspect.'	—

Types

`InspectedToken`

Text broken into tokens, so a prompt's cost stops being a mystery number.

Field	Type	Description
text	string	The token's text as it appears in the source.
id?	number	Vocabulary id, when your tokeniser reports it.

Behaviour

Topic	Detail	Description
whitespace	made visible	Spaces render as · and newlines as ↵. Whitespace is a token too, and an invisible one reads as a gap in the sequence.