textmate_grammar.engine.tokenize

The vscode-textmate v9.3.2 state model and line tokenization engine.

The main loop follows grammar/tokenizeString.ts. Scope names and state are kept as immutable parent-linked structures, while enter/anchor positions are line-local mutable bookkeeping exactly as in StateStackImpl.

Module Contents

Classes

ScopeStack

The theme.ts linked scope path, in root-first observable order.

AttributedScopeStack

Scope path node plus vscode-textmate’s inherited token/style attributes.

StateStackImpl

Port of grammar.ts StateStackImpl (including line-scoped positions).

IToken

ITokenizeLineResult

BalancedBracketSelectors

Balanced-bracket selector matcher for the optional binary-token API.

LineTokens

Minimal LineTokens implementation for scope-array tokens.

TokenHandler

Injection

Grammar

A loaded grammar and its vscode-textmate line tokenizer.

Functions

attributed_scope_stacks_equal

create_grammar

Construct a grammar; external_grammar is the registry integration seam.

get_scope_names

get_scope_name

Data

API

class textmate_grammar.engine.tokenize.ScopeStack(parent: textmate_grammar.engine.tokenize.ScopeStack | None, scope_name: str)

The theme.ts linked scope path, in root-first observable order.

Initialization

classmethod push_many(path: textmate_grammar.engine.tokenize.ScopeStack | None, scope_names: collections.abc.Sequence[str]) → textmate_grammar.engine.tokenize.ScopeStack | None
classmethod from_segments(*segments: str) → textmate_grammar.engine.tokenize.ScopeStack | None
property scopeName: str
property name: str
push(scope_name: str) → textmate_grammar.engine.tokenize.ScopeStack
get_segments() → list[str]
getSegments() → list[str]
get_extension_if_defined(base: textmate_grammar.engine.tokenize.ScopeStack | None) → list[str] | None
extends(other: textmate_grammar.engine.tokenize.ScopeStack) → bool
class textmate_grammar.engine.tokenize.AttributedScopeStack

Scope path node plus vscode-textmate’s inherited token/style attributes.

parent: textmate_grammar.engine.tokenize.AttributedScopeStack | None = None
scope_path: textmate_grammar.engine.tokenize.ScopeStack = None
token_attributes: int = 0
font_attributes: Any = None
style_attributes: Any = None
property scopePath: textmate_grammar.engine.tokenize.ScopeStack
property tokenAttributes: int
property fontAttributes: Any
property styleAttributes: Any
property scope_name: str
classmethod create_root(scope_name: str) → textmate_grammar.engine.tokenize.AttributedScopeStack
classmethod from_extension(name_scopes_list: textmate_grammar.engine.tokenize.AttributedScopeStack | None, content_name_scopes_list: collections.abc.Sequence[collections.abc.Mapping[str, Any]]) → textmate_grammar.engine.tokenize.AttributedScopeStack | None
classmethod fromExtension(name_scopes_list: textmate_grammar.engine.tokenize.AttributedScopeStack | None, content_name_scopes_list: collections.abc.Sequence[collections.abc.Mapping[str, Any]]) → textmate_grammar.engine.tokenize.AttributedScopeStack | None
push_attributed(scope_names: str | None, grammar: textmate_grammar.engine.tokenize.Grammar) → textmate_grammar.engine.tokenize.AttributedScopeStack
pushAttributed(scope_names: str | None, grammar: textmate_grammar.engine.tokenize.Grammar) → textmate_grammar.engine.tokenize.AttributedScopeStack
get_scope_names() → list[str]
getScopeNames() → list[str]
get_extension_if_defined(base: textmate_grammar.engine.tokenize.AttributedScopeStack | None) → list[dict[str, Any]] | None
getExtensionIfDefined(base: textmate_grammar.engine.tokenize.AttributedScopeStack | None) → list[dict[str, Any]] | None
equals(other: textmate_grammar.engine.tokenize.AttributedScopeStack | None) → bool
textmate_grammar.engine.tokenize.attributed_scope_stacks_equal(left: textmate_grammar.engine.tokenize.AttributedScopeStack | None, right: textmate_grammar.engine.tokenize.AttributedScopeStack | None) → bool
class textmate_grammar.engine.tokenize.StateStackImpl(parent: textmate_grammar.engine.tokenize.StateStackImpl | None, rule_id: int, enter_pos: int, anchor_pos: int, begin_rule_captured_eol: bool, end_rule: str | None, name_scopes_list: textmate_grammar.engine.tokenize.AttributedScopeStack | None, content_name_scopes_list: textmate_grammar.engine.tokenize.AttributedScopeStack | None)

Port of grammar.ts StateStackImpl (including line-scoped positions).

Initialization

NULL: textmate_grammar.engine.tokenize.StateStackImpl = None
property ruleId: int
property begin_rule_captured_eol: bool
property end_rule: str | None
property name_scopes_list: textmate_grammar.engine.tokenize.AttributedScopeStack | None
property content_name_scopes_list: textmate_grammar.engine.tokenize.AttributedScopeStack | None
clone() → textmate_grammar.engine.tokenize.StateStackImpl
structural_equals(other: textmate_grammar.engine.tokenize.StateStackImpl | None) → bool
equals(other: textmate_grammar.engine.tokenize.StateStackImpl | None) → bool
reset() → None
pop() → textmate_grammar.engine.tokenize.StateStackImpl | None
safe_pop() → textmate_grammar.engine.tokenize.StateStackImpl
safePop() → textmate_grammar.engine.tokenize.StateStackImpl
push(rule_id: int, enter_pos: int, anchor_pos: int, begin_rule_captured_eol: bool, end_rule: str | None, name_scopes_list: textmate_grammar.engine.tokenize.AttributedScopeStack | None, content_name_scopes_list: textmate_grammar.engine.tokenize.AttributedScopeStack | None) → textmate_grammar.engine.tokenize.StateStackImpl
get_enter_pos() → int
getEnterPos() → int
get_anchor_pos() → int
getAnchorPos() → int
get_rule(grammar: textmate_grammar.engine.rules.RuleRegistry) → textmate_grammar.engine.rules.Rule
with_content_name_scopes_list(scopes: textmate_grammar.engine.tokenize.AttributedScopeStack) → textmate_grammar.engine.tokenize.StateStackImpl
withContentNameScopesList(scopes: textmate_grammar.engine.tokenize.AttributedScopeStack) → textmate_grammar.engine.tokenize.StateStackImpl
with_end_rule(end_rule: str) → textmate_grammar.engine.tokenize.StateStackImpl
withEndRule(end_rule: str) → textmate_grammar.engine.tokenize.StateStackImpl
has_same_rule_as(other: textmate_grammar.engine.tokenize.StateStackImpl) → bool
hasSameRuleAs(other: textmate_grammar.engine.tokenize.StateStackImpl) → bool
to_state_stack_frame() → dict[str, Any]
toStateStackFrame() → dict[str, Any]
classmethod push_frame(parent: textmate_grammar.engine.tokenize.StateStackImpl | None, frame: collections.abc.Mapping[str, Any]) → textmate_grammar.engine.tokenize.StateStackImpl
classmethod pushFrame(parent: textmate_grammar.engine.tokenize.StateStackImpl | None, frame: collections.abc.Mapping[str, Any]) → textmate_grammar.engine.tokenize.StateStackImpl
to_string() → str
textmate_grammar.engine.tokenize.StateStack = None
textmate_grammar.engine.tokenize.ScopeListElement = None
class textmate_grammar.engine.tokenize.IToken
startIndex: int = None
endIndex: int = None
scopes: list[str] = None
property start_index: int
property end_index: int
class textmate_grammar.engine.tokenize.ITokenizeLineResult
tokens: list[textmate_grammar.engine.tokenize.IToken] = None
ruleStack: textmate_grammar.engine.tokenize.StateStackImpl = None
stoppedEarly: bool = None
property rule_stack: textmate_grammar.engine.tokenize.StateStackImpl
property stopped_early: bool
class textmate_grammar.engine.tokenize.BalancedBracketSelectors(balanced: collections.abc.Sequence[str], unbalanced: collections.abc.Sequence[str])

Balanced-bracket selector matcher for the optional binary-token API.

Initialization

property matchesAlways: bool
property matchesNever: bool
match(scopes: list[str]) → bool
class textmate_grammar.engine.tokenize.LineTokens(line_text: str)

Minimal LineTokens implementation for scope-array tokens.

Initialization

produce_from_scopes(scopes: textmate_grammar.engine.tokenize.AttributedScopeStack | None, end_index: int) → None
get_result(stack: textmate_grammar.engine.tokenize.StateStackImpl, line_length: int) → list[textmate_grammar.engine.tokenize.IToken]
class textmate_grammar.engine.tokenize.TokenHandler(line_text: str)

Initialization

handle(scopes: textmate_grammar.engine.tokenize.AttributedScopeStack | None, end_index: int) → None
get_token_result(stack: textmate_grammar.engine.tokenize.StateStackImpl, line_length: int) → list[textmate_grammar.engine.tokenize.IToken]
class textmate_grammar.engine.tokenize.Injection
debug_selector: str = None
matcher: collections.abc.Callable[[list[str]], bool] = None
priority: int = None
rule_id: int = None
grammar: collections.abc.Mapping[str, Any] = None
class textmate_grammar.engine.tokenize.Grammar(scope_name: str, grammar: collections.abc.Mapping[str, Any], initial_language: int = 1, embedded_languages: collections.abc.Mapping[str, int] | None = None, token_types: collections.abc.Mapping[str, int] | None = None, balanced_bracket_selectors: textmate_grammar.engine.tokenize.BalancedBracketSelectors | None = None, grammar_repository: Any = None, onig_lib: textmate_grammar.engine.oniglib.IOnigLib = DEFAULT_ONIG_LIB, *, external_grammar: collections.abc.Callable[[str], collections.abc.Mapping[str, Any] | None] | None = None)

Bases: textmate_grammar.engine.rules.RuleRegistry, textmate_grammar.engine.rules.IRuleFactoryHelper

A loaded grammar and its vscode-textmate line tokenizer.

Initialization

get_external_grammar(scope_name: str, repository: dict[str, Any] | None) → dict[str, Any] | None
get_injections() → list[textmate_grammar.engine.tokenize.Injection]
tokenize_line(line_text: str, prev_state: textmate_grammar.engine.tokenize.StateStackImpl | None = None, time_limit: int = 0) → textmate_grammar.engine.tokenize.ITokenizeLineResult
tokenizeLine(line_text: str, prev_state: textmate_grammar.engine.tokenize.StateStackImpl | None = None, time_limit: int = 0) → textmate_grammar.engine.tokenize.ITokenizeLineResult
dispose() → None
textmate_grammar.engine.tokenize.create_grammar(scope_name: str, grammar: collections.abc.Mapping[str, Any], initial_language: int = 1, embedded_languages: collections.abc.Mapping[str, int] | None = None, token_types: collections.abc.Mapping[str, int] | None = None, balanced_bracket_selectors: textmate_grammar.engine.tokenize.BalancedBracketSelectors | None = None, grammar_repository: Any = None, onig_lib: textmate_grammar.engine.oniglib.IOnigLib = DEFAULT_ONIG_LIB, *, external_grammar: collections.abc.Callable[[str], collections.abc.Mapping[str, Any] | None] | None = None) → textmate_grammar.engine.tokenize.Grammar

Construct a grammar; external_grammar is the registry integration seam.

textmate_grammar.engine.tokenize.createGrammar = None
textmate_grammar.engine.tokenize.get_scope_names(chain: Any | None) → list[str]
textmate_grammar.engine.tokenize.get_scope_name(chain: Any | None) → str
textmate_grammar.engine.tokenize.getScopeName = None
textmate_grammar.engine.tokenize.getScopeNames = None