Tokenizer·component·@astryxdesign/core

Tokenizer

Tokenizer is a multi-select input that lets users search, select, and manage multiple items displayed as removable chips. Use it when users need to build a set of selections from a searchable data source, like adding team members, applying tags, or choosing filters.

Open in Playground @astryxdesign/core/Tokenizer

Showcases and examples

8 documented examples

Tokenizer

A tokenizer with preset tags and search source.

Open Playground ↗

Preview loads on approachPreview loads on approach

Exact source · tokenizer-showcase

// Copyright (c) Meta Platforms, Inc. and affiliates.

'use client';

import {useState} from 'react';
import {Tokenizer} from '@astryxdesign/core/Tokenizer';
import type {SearchableItem, SearchSource} from '@astryxdesign/core/Typeahead';

const source: SearchSource = {
  search: () => [],
  bootstrap: () => [],
};

export default function TokenizerShowcase() {
  const [value, setValue] = useState<SearchableItem[]>([
    {id: '1', label: 'Design'},
    {id: '2', label: 'Engineering'},
  ]);
  return (
    <Tokenizer
      label="Tags"
      placeholder="Search..."
      searchSource={source}
      value={value}
      onChange={setValue}
      style={{width: 400}}
    />
  );
}

Usage

Write a placeholder that tells users what they can search for, such as "Search people..." or "Add tags...", so the input is not a blank mystery.
Set maxEntries when the number of selections should be bounded, like limiting a review to 5 approvers.
Use hasCreate for free-form tagging where users need to enter values that do not exist in the search source.
Show validation status with the status prop so users know immediately when a selection is missing or invalid.
Don't use Tokenizer for single-item selection; use Typeahead instead. Tokenizer is for building sets of two or more items.
Avoid applying custom colors to individual tokens inside a Tokenizer; use the default token style for visual consistency across the set.
Don't hide the label; every Tokenizer needs a visible label so users understand what they are selecting. Use isLabelHidden only when surrounding context makes the purpose obvious.
Wrap a disabled Tokenizer in Tooltip to explain why it is disabled; disabled controls swallow the hover events the wrapper needs. Use the disabledMessage prop instead.

Typed props

Prop	Type and behavior
`label`	`string` · required Accessible label for the input.
`searchSource`	`SearchSource<T>` · required Data source providing search and bootstrap methods for populating the dropdown.
`value`	`T[]` · required Array of currently selected items.
`onChange`	`(items: T[], change: TokenizerChange<T>) => void` · required Called when selection changes. The change argument includes the affected item and type ('add' \| 'create' \| 'remove' \| 'reorder'). Additions and removals (including Backspace on an empty input) are announced to screen readers via a polite live region.
`placeholder`	`string` Input placeholder text. Only shown when no tokens are selected.
`maxEntries`	`number` Maximum number of selections allowed. Input is hidden when the limit is reached.
`hasClear`	`boolean` · default false Show a clear-all button for bulk removal of all tokens.
`renderToken`	`(item: T, onRemove: () => void) => ReactNode` Custom render function for selected tokens. Default renders Token with label and onRemove.
`renderItem`	`(item: T) => ReactNode` Custom render function for dropdown items. Default renders TypeaheadItem.
`isDisabled`	`boolean` · default false Disables the input and all token interactions.
`htmlName`	`string` The HTML name attribute for form submissions. Renders one hidden input per selected item id.
`disabledMessage`	`string` Explains why the tokenizer is disabled. With isDisabled, shows a tooltip on hover/keyboard focus and keeps the input focusable via aria-disabled (input stays blocked). Use this instead of wrapping a disabled Tokenizer in Tooltip. Disabled controls swallow the hover events an external Tooltip needs.
`status`	`{type: 'warning' \| 'error' \| 'success', message?: string}` Validation status object with type and message for error/warning/success states.
`statusVariant`	`'attached' \| 'detached'` · default 'attached' How the status message is placed relative to the input. attached overlaps directly below the input (bordered treatment); detached floats below as a separate element with spacing.
`isLabelHidden`	`boolean` · default false Visually hides the label while keeping it accessible.
`description`	`string` Helper text displayed below the label.
`isRequired`	`boolean` · default false Marks the field as required.
`isOptional`	`boolean` · default false Shows an optional indicator on the label.
`labelTooltip`	`string` Tooltip text shown on the label.
`hasEntriesOnFocus`	`boolean` · default false Show bootstrap results on focus before typing.
`maxMenuItems`	`number` · default 10 Maximum number of search results to display. The hasCreate entry is offered on top of them, so a menu can show one more than this.
`menuWidth`	`number` Fixed dropdown width in pixels. The menu never shrinks below its anchor width.
`minQueryLength`	`number` · default 1 Minimum query length before the search source is queried. Below it no search runs, and the menu stays closed — unless hasCreate is set, in which case the "Create ..." entry is still offered, being derived from the typed text rather than fetched for it.
`emptySearchResultsText`	`string` · default 'No results found' Text shown when search returns no results.
`hasAutoFocus`	`boolean` · default false Auto-focus the input on mount.
`size`	`'sm' \| 'md' \| 'lg'` · default 'md' Input and token size.
`debounceMs`	`number` · default 150 Debounce delay in ms before triggering search. Set to 0 for synchronous sources.
`hasCreate`	`boolean` · default false Allow users to create new tokens from free-text input. When true, a "Create" option appears in the dropdown for typed text that doesn't match existing results. The onChange change type is 'create' for these items.
`onChangeQuery`	`(query: string) => void` Callback fired when the search query text changes.
`startIcon`	`ReactNode \| IconType` Icon to display at the start of the input, before any tokens. Accepts a semantic icon name, an SVG icon component, or a ReactNode directly. Slot: Icon
`endContent`	`ReactNode` Content to display at the end of the input row. Useful for buttons, result counts, or other controls. Slot: Icon, Badge
`handleRef`	`React.Ref<TokenizerHandle>` Imperative handle exposing focusInput(), focusFirstToken(), focusLastToken(), clearInput(), and selectAll().
`width`	`SizeValue` Width of the field (number = pixels, string used as-is, e.g. "100%"). Sizes the whole field (label, control, and status) so they stay aligned.
`tokenOverflowBehavior`	`'none' \| 'unfocusedInline' \| 'unfocusedLayer'` · default 'none' Controls how tokens overflow when the container is too narrow.
`onFocus`	`(e: FocusEvent<HTMLInputElement>) => void` Fires when focus enters the tokenizer input.
`onBlur`	`(e: FocusEvent<HTMLInputElement>) => void` Fires when focus leaves the tokenizer input.
`xstyle`	`StyleXStyles` StyleX styles for layout customization (margins, positioning, sizing). Must be a stylex.create() value; not an inline style object like style={{}}.

Anatomy

Label · required

The visible text above the input describing what the user is selecting. Also used as the accessible name.

Token chips · optional

Removable chips representing each selected item. Each chip shows a label and a remove button.

Search input · required

The text input where users type to search the data source. Hides when maxEntries is reached.

Dropdown menu · optional

The search results list that appears below the input as the user types.

Spinner · optional

Loading indicator shown at the end of the field while a search is in flight.

End content · optional

A trailing slot after the input for action buttons, counts, or other controls.

Clear button · optional

A button that removes all selected tokens at once. Shown when hasClear is true and tokens are present.

Theming

Targets

astryx-tokenizer

Visual props: size, status

States: disabled