textmate_grammar.engine.tokenize¶
The vscode-textmate v9.3.2 state model and line tokenization engine.
The main loop follows grammar/tokenizeString.ts. Scope names and state are
kept as immutable parent-linked structures, while enter/anchor positions are
line-local mutable bookkeeping exactly as in StateStackImpl.
Module Contents¶
Classes¶
The |
|
Scope path node plus vscode-textmate’s inherited token/style attributes. |
|
Port of grammar.ts |
|
Balanced-bracket selector matcher for the optional binary-token API. |
|
Minimal |
|
A loaded grammar and its vscode-textmate line tokenizer. |
Functions¶
Construct a grammar; |
|
Data¶
API¶
- class textmate_grammar.engine.tokenize.ScopeStack(parent: textmate_grammar.engine.tokenize.ScopeStack | None, scope_name: str)¶
The
theme.tslinked scope path, in root-first observable order.Initialization
- classmethod push_many(path: textmate_grammar.engine.tokenize.ScopeStack | None, scope_names: collections.abc.Sequence[str]) textmate_grammar.engine.tokenize.ScopeStack | None¶
- classmethod from_segments(*segments: str) textmate_grammar.engine.tokenize.ScopeStack | None¶
- push(scope_name: str) textmate_grammar.engine.tokenize.ScopeStack¶
- get_extension_if_defined(base: textmate_grammar.engine.tokenize.ScopeStack | None) list[str] | None¶
- extends(other: textmate_grammar.engine.tokenize.ScopeStack) bool¶
- class textmate_grammar.engine.tokenize.AttributedScopeStack¶
Scope path node plus vscode-textmate’s inherited token/style attributes.
- parent: textmate_grammar.engine.tokenize.AttributedScopeStack | None = None¶
- scope_path: textmate_grammar.engine.tokenize.ScopeStack = None¶
- property scopePath: textmate_grammar.engine.tokenize.ScopeStack¶
- classmethod create_root(scope_name: str) textmate_grammar.engine.tokenize.AttributedScopeStack¶
- classmethod from_extension(name_scopes_list: textmate_grammar.engine.tokenize.AttributedScopeStack | None, content_name_scopes_list: collections.abc.Sequence[collections.abc.Mapping[str, Any]]) textmate_grammar.engine.tokenize.AttributedScopeStack | None¶
- classmethod fromExtension(name_scopes_list: textmate_grammar.engine.tokenize.AttributedScopeStack | None, content_name_scopes_list: collections.abc.Sequence[collections.abc.Mapping[str, Any]]) textmate_grammar.engine.tokenize.AttributedScopeStack | None¶
- push_attributed(scope_names: str | None, grammar: textmate_grammar.engine.tokenize.Grammar) textmate_grammar.engine.tokenize.AttributedScopeStack¶
- pushAttributed(scope_names: str | None, grammar: textmate_grammar.engine.tokenize.Grammar) textmate_grammar.engine.tokenize.AttributedScopeStack¶
- get_extension_if_defined(base: textmate_grammar.engine.tokenize.AttributedScopeStack | None) list[dict[str, Any]] | None¶
- getExtensionIfDefined(base: textmate_grammar.engine.tokenize.AttributedScopeStack | None) list[dict[str, Any]] | None¶
- equals(other: textmate_grammar.engine.tokenize.AttributedScopeStack | None) bool¶
- textmate_grammar.engine.tokenize.attributed_scope_stacks_equal(left: textmate_grammar.engine.tokenize.AttributedScopeStack | None, right: textmate_grammar.engine.tokenize.AttributedScopeStack | None) bool¶
- class textmate_grammar.engine.tokenize.StateStackImpl(parent: textmate_grammar.engine.tokenize.StateStackImpl | None, rule_id: int, enter_pos: int, anchor_pos: int, begin_rule_captured_eol: bool, end_rule: str | None, name_scopes_list: textmate_grammar.engine.tokenize.AttributedScopeStack | None, content_name_scopes_list: textmate_grammar.engine.tokenize.AttributedScopeStack | None)¶
Port of grammar.ts
StateStackImpl(including line-scoped positions).Initialization
- NULL: textmate_grammar.engine.tokenize.StateStackImpl = None¶
- property name_scopes_list: textmate_grammar.engine.tokenize.AttributedScopeStack | None¶
- property content_name_scopes_list: textmate_grammar.engine.tokenize.AttributedScopeStack | None¶
- structural_equals(other: textmate_grammar.engine.tokenize.StateStackImpl | None) bool¶
- equals(other: textmate_grammar.engine.tokenize.StateStackImpl | None) bool¶
- push(rule_id: int, enter_pos: int, anchor_pos: int, begin_rule_captured_eol: bool, end_rule: str | None, name_scopes_list: textmate_grammar.engine.tokenize.AttributedScopeStack | None, content_name_scopes_list: textmate_grammar.engine.tokenize.AttributedScopeStack | None) textmate_grammar.engine.tokenize.StateStackImpl¶
- get_rule(grammar: textmate_grammar.engine.rules.RuleRegistry) textmate_grammar.engine.rules.Rule¶
- with_content_name_scopes_list(scopes: textmate_grammar.engine.tokenize.AttributedScopeStack) textmate_grammar.engine.tokenize.StateStackImpl¶
- withContentNameScopesList(scopes: textmate_grammar.engine.tokenize.AttributedScopeStack) textmate_grammar.engine.tokenize.StateStackImpl¶
- with_end_rule(end_rule: str) textmate_grammar.engine.tokenize.StateStackImpl¶
- withEndRule(end_rule: str) textmate_grammar.engine.tokenize.StateStackImpl¶
- has_same_rule_as(other: textmate_grammar.engine.tokenize.StateStackImpl) bool¶
- hasSameRuleAs(other: textmate_grammar.engine.tokenize.StateStackImpl) bool¶
- classmethod push_frame(parent: textmate_grammar.engine.tokenize.StateStackImpl | None, frame: collections.abc.Mapping[str, Any]) textmate_grammar.engine.tokenize.StateStackImpl¶
- classmethod pushFrame(parent: textmate_grammar.engine.tokenize.StateStackImpl | None, frame: collections.abc.Mapping[str, Any]) textmate_grammar.engine.tokenize.StateStackImpl¶
- textmate_grammar.engine.tokenize.StateStack = None¶
- textmate_grammar.engine.tokenize.ScopeListElement = None¶
- class textmate_grammar.engine.tokenize.IToken¶
- class textmate_grammar.engine.tokenize.ITokenizeLineResult¶
- tokens: list[textmate_grammar.engine.tokenize.IToken] = None¶
- ruleStack: textmate_grammar.engine.tokenize.StateStackImpl = None¶
- property rule_stack: textmate_grammar.engine.tokenize.StateStackImpl¶
- class textmate_grammar.engine.tokenize.BalancedBracketSelectors(balanced: collections.abc.Sequence[str], unbalanced: collections.abc.Sequence[str])¶
Balanced-bracket selector matcher for the optional binary-token API.
Initialization
- class textmate_grammar.engine.tokenize.LineTokens(line_text: str)¶
Minimal
LineTokensimplementation for scope-array tokens.Initialization
- produce_from_scopes(scopes: textmate_grammar.engine.tokenize.AttributedScopeStack | None, end_index: int) None¶
- get_result(stack: textmate_grammar.engine.tokenize.StateStackImpl, line_length: int) list[textmate_grammar.engine.tokenize.IToken]¶
- class textmate_grammar.engine.tokenize.TokenHandler(line_text: str)¶
Initialization
- handle(scopes: textmate_grammar.engine.tokenize.AttributedScopeStack | None, end_index: int) None¶
- get_token_result(stack: textmate_grammar.engine.tokenize.StateStackImpl, line_length: int) list[textmate_grammar.engine.tokenize.IToken]¶
- class textmate_grammar.engine.tokenize.Injection¶
-
- matcher: collections.abc.Callable[[list[str]], bool] = None¶
- grammar: collections.abc.Mapping[str, Any] = None¶
- class textmate_grammar.engine.tokenize.Grammar(scope_name: str, grammar: collections.abc.Mapping[str, Any], initial_language: int = 1, embedded_languages: collections.abc.Mapping[str, int] | None = None, token_types: collections.abc.Mapping[str, int] | None = None, balanced_bracket_selectors: textmate_grammar.engine.tokenize.BalancedBracketSelectors | None = None, grammar_repository: Any = None, onig_lib: textmate_grammar.engine.oniglib.IOnigLib = DEFAULT_ONIG_LIB, *, external_grammar: collections.abc.Callable[[str], collections.abc.Mapping[str, Any] | None] | None = None)¶
Bases:
textmate_grammar.engine.rules.RuleRegistry,textmate_grammar.engine.rules.IRuleFactoryHelperA loaded grammar and its vscode-textmate line tokenizer.
Initialization
- get_injections() list[textmate_grammar.engine.tokenize.Injection]¶
- tokenize_line(line_text: str, prev_state: textmate_grammar.engine.tokenize.StateStackImpl | None = None, time_limit: int = 0) textmate_grammar.engine.tokenize.ITokenizeLineResult¶
- tokenizeLine(line_text: str, prev_state: textmate_grammar.engine.tokenize.StateStackImpl | None = None, time_limit: int = 0) textmate_grammar.engine.tokenize.ITokenizeLineResult¶
- textmate_grammar.engine.tokenize.create_grammar(scope_name: str, grammar: collections.abc.Mapping[str, Any], initial_language: int = 1, embedded_languages: collections.abc.Mapping[str, int] | None = None, token_types: collections.abc.Mapping[str, int] | None = None, balanced_bracket_selectors: textmate_grammar.engine.tokenize.BalancedBracketSelectors | None = None, grammar_repository: Any = None, onig_lib: textmate_grammar.engine.oniglib.IOnigLib = DEFAULT_ONIG_LIB, *, external_grammar: collections.abc.Callable[[str], collections.abc.Mapping[str, Any] | None] | None = None) textmate_grammar.engine.tokenize.Grammar¶
Construct a grammar;
external_grammaris the registry integration seam.
- textmate_grammar.engine.tokenize.createGrammar = None¶
- textmate_grammar.engine.tokenize.getScopeName = None¶
- textmate_grammar.engine.tokenize.getScopeNames = None¶