Skip to main content

Class: TokenGraphemeSplitter

libs/cursorless-engine/tokenGraphemeSplitter/tokenGraphemeSplitter.TokenGraphemeSplitter

Constructors​

constructor​

• new TokenGraphemeSplitter()

Defined in​

libs/cursorless-engine/tokenGraphemeSplitter/tokenGraphemeSplitter.ts:81

Properties​

algorithmChangeNotifier​

• Private algorithmChangeNotifier: Notifier<[]>

Defined in​

libs/cursorless-engine/tokenGraphemeSplitter/tokenGraphemeSplitter.ts:78

disposables​

• Private disposables: Disposable[] = []

Defined in​

libs/cursorless-engine/tokenGraphemeSplitter/tokenGraphemeSplitter.ts:77

registerAlgorithmChangeListener​

• registerAlgorithmChangeListener: (listener: Listener<[]>) => Disposable

Type declaration​

▸ (listener): Disposable

Register to be notified when notifyListeners is called

Parameters​
NameTypeDescription
listenerListener<[]>A function to be called when notifyListeners is called
Returns​

Disposable

A function that can be called to unsubscribe from notifications

Defined in​

libs/cursorless-engine/tokenGraphemeSplitter/tokenGraphemeSplitter.ts:187

tokenHatSplittingMode​

• Private tokenHatSplittingMode: TokenHatSplittingMode

Defined in​

libs/cursorless-engine/tokenGraphemeSplitter/tokenGraphemeSplitter.ts:79

Methods​

dispose​

▸ dispose(): void

Returns​

void

Defined in​

libs/cursorless-engine/tokenGraphemeSplitter/tokenGraphemeSplitter.ts:190

getTokenGraphemes​

▸ getTokenGraphemes(token): Grapheme[]

Splits {@link token} into a list of graphemes, normalised as per normalizeGrapheme.

Parameters​

NameTypeDescription
tokenstringThe token to split

Returns​

Grapheme[]

A list of normalised graphemes in {@link token}

Defined in​

libs/cursorless-engine/tokenGraphemeSplitter/tokenGraphemeSplitter.ts:122

normalizeGrapheme​

▸ normalizeGrapheme(rawGraphemeText): string

Normalizes the grapheme {@link rawGraphemeText} based on user configuration. Proceeds as follows:

  1. Runs text through Unicode NFC normalization to ensure that characters that look identical are handled the same (eg whether they use combining mark or single codepoint for diacritics).
  2. If the grapheme is a known grapheme, returns it.
  3. Transforms grapheme to lowercase if TokenHatSplittingMode.preserveCase is false
  4. Returns the (possibly case-normalised) grapheme if it appears in TokenHatSplittingMode.lettersToPreserve
  5. Strips diacritics from the grapheme
  6. If the grapheme doesn't match {@link KNOWN_GRAPHEME_MATCHER}, maps the grapheme to the constant UNKNOWN, so that it can be referred to using "special", "red special", etc.
  7. Returns the grapheme.

Parameters​

NameTypeDescription
rawGraphemeTextstringThe raw grapheme text to normalise

Returns​

string

The normalised grapheme

Defined in​

libs/cursorless-engine/tokenGraphemeSplitter/tokenGraphemeSplitter.ts:150

updateTokenHatSplittingMode​

▸ Private updateTokenHatSplittingMode(): void

Returns​

void

Defined in​

libs/cursorless-engine/tokenGraphemeSplitter/tokenGraphemeSplitter.ts:99