textmate_grammar.engine.rules

Port of the Rule classes, RuleRegistry and the rule factory from vscode-textmate’s src/rule.ts (version 9.3.2, the pinned reference clone).

Ported from rule.ts: endRuleId/whileRuleId (rule.ts:17-20), the Rule base class (rule.ts:43-89), CaptureRule (rule.ts:96-120), MatchRule (rule.ts:122-164), IncludeOnlyRule (rule.ts:166-207), BeginEndRule (rule.ts:209-288), BeginWhileRule (rule.ts:290-379) and the RuleFactory (rule.ts:381-573) — i.e. createCaptureRule, getCompiledRuleId, _compileCaptures and _compilePatterns. The registry lives in Grammar upstream: registerRule/getRule are grammar.ts:245-254; the class RuleRegistry here extracts exactly that behaviour.

parse_include ports parseInclude and IncludeReferenceKind from src/grammar/grammarDependencies.ts (grammarDependencies.ts:247-295) — the minimal piece the rule factory needs for include resolution. init_grammar ports initGrammar (grammar.ts:410-421), which installs the $self/$base repository hooks.

Semantics are 1:1 with the reference; naming is pythonic where it can be (upstream member names are kept when the python port is simply “same words”), and every non-obvious piece carries a comment referencing the ported lines of rule.ts / grammar.ts / grammarDependencies.ts.

Module Contents

Classes

IRuleRegistry

Port of IRuleRegistry (rule.ts:31-34): numeric rule registration and lookup by id.

IGrammarRegistry

Port of IGrammarRegistry (rule.ts:36-38): resolves scope-name includes against grammars looked up by scope name. The concrete implementation arrives with the registry layer (task 4b); here it is only the hook shape.

IRuleFactoryHelper

Port of IRuleFactoryHelper (rule.ts:40-41). Upstream Grammar additionally implements IOnigLib (grammar.ts:98), so the helper doubles as the onig-lib argument of the compile calls; the two createOnig* members come from Grammar.createOnigScanner/createOnigString (grammar.ts:177-181).

RuleRegistry

Port of the rule-id <-> Rule bookkeeping that lives in Grammar (grammar.ts:100, 127-128, 245-254): monotonically increasing numeric ids starting at 1, with the rule objects stored under their id. Serves as the concrete IRuleFactoryHelper for RuleFactory users and, later (task 3), as the registry component of Grammar.

Rule

Port of the abstract Rule (rule.ts:43-89): identity, name/ contentName with capture expansion, and the pattern collection / compile contract the tokenization loop drives.

CompilePatternsResult

Port of ICompilePatternsResult (rule.ts:91-94): the ordered rule ids of a pattern list plus upstream’s flag recording that entries were dropped (include targets that could not be resolved).

CaptureRule

Port of CaptureRule (rule.ts:96-120) — one entry of a captures map.

MatchRule

Port of MatchRule (rule.ts:122-164): a single match pattern with optional captures.

IncludeOnlyRule

Port of IncludeOnlyRule (rule.ts:166-207): a rule made purely of include references whose own (possibly missing) patterns are collected at compile time.

BeginEndRule

Port of BeginEndRule (rule.ts:209-288): begin/end with captures on both, contained patterns and applyEndPatternLast placement.

BeginWhileRule

Port of BeginWhileRule (rule.ts:290-379): begin plus a while condition checked at the start of every line (rule id whileRuleId = -2), with the contained patterns compiled as for BeginEndRule.

IncludeReferenceKind

Port of IncludeReferenceKind (grammarDependencies.ts:247-253).

IncludeReference

Port of the IncludeReference union (grammarDependencies.ts:240-276); fields beyond kind are set per kind (scope_name/rule_name).

RuleFactory

Port of RuleFactory (rule.ts:381-573).

Functions

parse_include

Port of parseInclude (grammarDependencies.ts:278-295): $base / $self, #rule (relative), scope (top level), scope#rule (top-level repository reference). Note: this is the shared include parser; the full grammarDependency machinery (task 4b) should import it from here.

init_grammar

Port of initGrammar (grammar.ts:410-421): deep-copies the raw grammar and installs the $self and $base repository entries the include resolution relies on ($self = the top-level patterns, $base = the caller-provided base grammar’s self entry or the grammar’s own).

create_rule

Mirror of RuleFactory.getCompiledRuleId (task-card spelling); prefer RuleFactory.get_compiled_rule_id().

Data

API

textmate_grammar.engine.rules.T = 'TypeVar(...)'
textmate_grammar.engine.rules.END_RULE_ID: int = None
textmate_grammar.engine.rules.WHILE_RULE_ID = None
textmate_grammar.engine.rules.IRawRuleDict = None
textmate_grammar.engine.rules.IRawRepositoryDict = None
textmate_grammar.engine.rules.IRawGrammarDict = None
textmate_grammar.engine.rules.ILocation = None
class textmate_grammar.engine.rules.IRuleRegistry

Port of IRuleRegistry (rule.ts:31-34): numeric rule registration and lookup by id.

abstractmethod get_rule(rule_id: int) → textmate_grammar.engine.rules.Rule

Port of getRule(ruleId) (rule.ts:32).

abstractmethod register_rule(factory: collections.abc.Callable[[int], textmate_grammar.engine.rules.T]) → textmate_grammar.engine.rules.T

Port of registerRule(factory) (rule.ts:33): the factory is invoked immediately with the freshly allocated id.

class textmate_grammar.engine.rules.IGrammarRegistry

Port of IGrammarRegistry (rule.ts:36-38): resolves scope-name includes against grammars looked up by scope name. The concrete implementation arrives with the registry layer (task 4b); here it is only the hook shape.

abstractmethod get_external_grammar(scope_name: str, repository: textmate_grammar.engine.rules.IRawRepositoryDict | None) → textmate_grammar.engine.rules.IRawGrammarDict | None

Port of getExternalGrammar(scopeName, repository) (rule.ts:37).

class textmate_grammar.engine.rules.IRuleFactoryHelper

Bases: textmate_grammar.engine.rules.IRuleRegistry, textmate_grammar.engine.rules.IGrammarRegistry

Port of IRuleFactoryHelper (rule.ts:40-41). Upstream Grammar additionally implements IOnigLib (grammar.ts:98), so the helper doubles as the onig-lib argument of the compile calls; the two createOnig* members come from Grammar.createOnigScanner/createOnigString (grammar.ts:177-181).

abstractmethod create_onig_scanner(sources: collections.abc.Sequence[str]) → Any

Port of createOnigScanner (grammar.ts:177-179).

abstractmethod create_onig_string(string: str) → Any

Port of createOnigString (grammar.ts:180-181).

class textmate_grammar.engine.rules.RuleRegistry(onig_lib: Any = None)

Bases: textmate_grammar.engine.rules.IRuleFactoryHelper

Port of the rule-id <-> Rule bookkeeping that lives in Grammar (grammar.ts:100, 127-128, 245-254): monotonically increasing numeric ids starting at 1, with the rule objects stored under their id. Serves as the concrete IRuleFactoryHelper for RuleFactory users and, later (task 3), as the registry component of Grammar.

Note: upstream v9.3.2 has no standalone RuleRegistry class and no scope-name interning; the class here is simply the extracted Grammar behaviour.

Initialization

register_rule(factory: collections.abc.Callable[[int], textmate_grammar.engine.rules.T]) → textmate_grammar.engine.rules.T

Port of Grammar.registerRule (grammar.ts:245-250).

get_rule(rule_id: int) → textmate_grammar.engine.rules.Rule

Port of Grammar.getRule (grammar.ts:252-254). May return the (python) None of a not-yet-registered rule while its factory is still running — upstream yields undefined there, and the only reader that can act on it treats non-None like undefined (see RuleFactory.compile_patterns).

get_external_grammar(scope_name: str, repository: textmate_grammar.engine.rules.IRawRepositoryDict | None) → textmate_grammar.engine.rules.IRawGrammarDict | None

Port of Grammar.getExternalGrammar (grammar.ts:256-274): scope-name includes resolve through the grammar repository; without a registry layer behind this helper there is nothing to resolve (task 4b provides it).

create_onig_scanner(sources: collections.abc.Sequence[str]) → Any

Port of Grammar.createOnigScanner (grammar.ts:177-179).

create_onig_string(string: str) → Any

Port of Grammar.createOnigString (grammar.ts:180-181).

class textmate_grammar.engine.rules.Rule(location: textmate_grammar.engine.rules.ILocation, id: int, name: str | None, contentName: str | None)

Port of the abstract Rule (rule.ts:43-89): identity, name/ contentName with capture expansion, and the pattern collection / compile contract the tokenization loop drives.

Initialization

property debugName: str

Port of Rule.debugName (rule.ts:65-68).

getName(lineText: str | None, captureIndices: collections.abc.Sequence[textmate_grammar.engine.oniglib.IOnigCaptureIndex] | None) → str | None

Port of Rule.getName (rule.ts:70-75): expands capture references in the name against the matched text; non-capturing names pass through.

getContentName(lineText: str, captureIndices: collections.abc.Sequence[textmate_grammar.engine.oniglib.IOnigCaptureIndex]) → str | None

Port of Rule.getContentName (rule.ts:77-82).

dispose() → None

Port of Rule.dispose (rule.ts:63); the base rule holds nothing.

Subclasses override to drop their compiled-pattern caches (kept for parity; garbage collection makes it a cache-buster only).

abstractmethod collectPatterns(grammar: textmate_grammar.engine.rules.IRuleRegistry, out: textmate_grammar.engine.regexsource.RegExpSourceList) → None

Port of Rule.collectPatterns (rule.ts:84) — abstract.

abstractmethod compile(grammar: textmate_grammar.engine.rules.IRuleFactoryHelper, endRegexSource: str | None) → textmate_grammar.engine.regexsource.CompiledRule

Port of Rule.compile (rule.ts:86) — abstract.

abstractmethod compileAG(grammar: textmate_grammar.engine.rules.IRuleFactoryHelper, endRegexSource: str | None, allowA: bool, allowG: bool) → textmate_grammar.engine.regexsource.CompiledRule

Port of Rule.compileAG (rule.ts:88) — abstract.

class textmate_grammar.engine.rules.CompilePatternsResult

Port of ICompilePatternsResult (rule.ts:91-94): the ordered rule ids of a pattern list plus upstream’s flag recording that entries were dropped (include targets that could not be resolved).

patterns: tuple[int, ...] = None
hasMissingPatterns: bool = None
class textmate_grammar.engine.rules.CaptureRule(location: textmate_grammar.engine.rules.ILocation, id: int, name: str | None, contentName: str | None, retokenizeCapturedWithRuleId: int)

Bases: textmate_grammar.engine.rules.Rule

Port of CaptureRule (rule.ts:96-120) — one entry of a captures map.

retokenizeCapturedWithRuleId is 0 (no nested rule) unless the capture declares patterns, in which case it is the id of the rule compiled from them (rule.ts:476-480).

Initialization

collectPatterns(grammar: textmate_grammar.engine.rules.IRuleRegistry, out: textmate_grammar.engine.regexsource.RegExpSourceList) → None
compile(grammar: textmate_grammar.engine.rules.IRuleFactoryHelper, endRegexSource: str | None) → textmate_grammar.engine.regexsource.CompiledRule
compileAG(grammar: textmate_grammar.engine.rules.IRuleFactoryHelper, endRegexSource: str | None, allowA: bool, allowG: bool) → textmate_grammar.engine.regexsource.CompiledRule
class textmate_grammar.engine.rules.MatchRule(location: textmate_grammar.engine.rules.ILocation, id: int, name: str | None, match: str, captures: list[textmate_grammar.engine.rules.CaptureRule | None])

Bases: textmate_grammar.engine.rules.Rule

Port of MatchRule (rule.ts:122-164): a single match pattern with optional captures.

Initialization

dispose() → None
property debugMatchRegExp: str

Port of debugMatchRegExp (rule.ts:141-143).

collectPatterns(grammar: textmate_grammar.engine.rules.IRuleRegistry, out: textmate_grammar.engine.regexsource.RegExpSourceList) → None
compile(grammar: textmate_grammar.engine.rules.IRuleFactoryHelper, endRegexSource: str | None) → textmate_grammar.engine.regexsource.CompiledRule
compileAG(grammar: textmate_grammar.engine.rules.IRuleFactoryHelper, endRegexSource: str | None, allowA: bool, allowG: bool) → textmate_grammar.engine.regexsource.CompiledRule
class textmate_grammar.engine.rules.IncludeOnlyRule(location: textmate_grammar.engine.rules.ILocation, id: int, name: str | None, contentName: str | None, patterns: textmate_grammar.engine.rules.CompilePatternsResult)

Bases: textmate_grammar.engine.rules.Rule

Port of IncludeOnlyRule (rule.ts:166-207): a rule made purely of include references whose own (possibly missing) patterns are collected at compile time.

Initialization

dispose() → None
collectPatterns(grammar: textmate_grammar.engine.rules.IRuleRegistry, out: textmate_grammar.engine.regexsource.RegExpSourceList) → None
compile(grammar: textmate_grammar.engine.rules.IRuleFactoryHelper, endRegexSource: str | None) → textmate_grammar.engine.regexsource.CompiledRule
compileAG(grammar: textmate_grammar.engine.rules.IRuleFactoryHelper, endRegexSource: str | None, allowA: bool, allowG: bool) → textmate_grammar.engine.regexsource.CompiledRule
class textmate_grammar.engine.rules.BeginEndRule(location: textmate_grammar.engine.rules.ILocation, id: int, name: str | None, contentName: str | None, begin: str, beginCaptures: list[textmate_grammar.engine.rules.CaptureRule | None], end: str | None, endCaptures: list[textmate_grammar.engine.rules.CaptureRule | None], applyEndPatternLast: bool | None, patterns: textmate_grammar.engine.rules.CompilePatternsResult)

Bases: textmate_grammar.engine.rules.Rule

Port of BeginEndRule (rule.ts:209-288): begin/end with captures on both, contained patterns and applyEndPatternLast placement.

Initialization

dispose() → None
property debugBeginRegExp: str

Port of debugBeginRegExp (rule.ts:240-242).

property debugEndRegExp: str

Port of debugEndRegExp (rule.ts:244-246).

getEndWithResolvedBackReferences(lineText: str, captureIndices: collections.abc.Sequence[textmate_grammar.engine.oniglib.IOnigCaptureIndex]) → str

Port of getEndWithResolvedBackReferences (rule.ts:248-250): the end pattern with backreferences substituted from the begin match.

collectPatterns(grammar: textmate_grammar.engine.rules.IRuleRegistry, out: textmate_grammar.engine.regexsource.RegExpSourceList) → None
compile(grammar: textmate_grammar.engine.rules.IRuleFactoryHelper, endRegexSource: str | None) → textmate_grammar.engine.regexsource.CompiledRule
compileAG(grammar: textmate_grammar.engine.rules.IRuleFactoryHelper, endRegexSource: str | None, allowA: bool, allowG: bool) → textmate_grammar.engine.regexsource.CompiledRule
class textmate_grammar.engine.rules.BeginWhileRule(location: textmate_grammar.engine.rules.ILocation, id: int, name: str | None, contentName: str | None, begin: str, beginCaptures: list[textmate_grammar.engine.rules.CaptureRule | None], while_: str, whileCaptures: list[textmate_grammar.engine.rules.CaptureRule | None], patterns: textmate_grammar.engine.rules.CompilePatternsResult)

Bases: textmate_grammar.engine.rules.Rule

Port of BeginWhileRule (rule.ts:290-379): begin plus a while condition checked at the start of every line (rule id whileRuleId = -2), with the contained patterns compiled as for BeginEndRule.

Initialization

dispose() → None
property debugBeginRegExp: str

Port of debugBeginRegExp (rule.ts:325-327).

property debugWhileRegExp: str

Port of debugWhileRegExp (rule.ts:329-331).

getWhileWithResolvedBackReferences(lineText: str, captureIndices: collections.abc.Sequence[textmate_grammar.engine.oniglib.IOnigCaptureIndex]) → str

Port of getWhileWithResolvedBackReferences (rule.ts:333-335): the while pattern with backreferences substituted from the begin match.

collectPatterns(grammar: textmate_grammar.engine.rules.IRuleRegistry, out: textmate_grammar.engine.regexsource.RegExpSourceList) → None
compile(grammar: textmate_grammar.engine.rules.IRuleFactoryHelper, endRegexSource: str | None) → textmate_grammar.engine.regexsource.CompiledRule
compileAG(grammar: textmate_grammar.engine.rules.IRuleFactoryHelper, endRegexSource: str | None, allowA: bool, allowG: bool) → textmate_grammar.engine.regexsource.CompiledRule
compileWhile(grammar: textmate_grammar.engine.rules.IRuleFactoryHelper, endRegexSource: str | None) → textmate_grammar.engine.regexsource.CompiledRule

Port of compileWhile (rule.ts:361-363).

compileWhileAG(grammar: textmate_grammar.engine.rules.IRuleFactoryHelper, endRegexSource: str | None, allowA: bool, allowG: bool) → textmate_grammar.engine.regexsource.CompiledRule

Port of compileWhileAG (rule.ts:365-367).

class textmate_grammar.engine.rules.IncludeReferenceKind

Bases: enum.IntEnum

Port of IncludeReferenceKind (grammarDependencies.ts:247-253).

Initialization

Initialize self. See help(type(self)) for accurate signature.

Base = 0
Self = 1
RelativeReference = 2
TopLevelReference = 3
TopLevelRepositoryReference = 4
class textmate_grammar.engine.rules.IncludeReference

Port of the IncludeReference union (grammarDependencies.ts:240-276); fields beyond kind are set per kind (scope_name/rule_name).

kind: textmate_grammar.engine.rules.IncludeReferenceKind = None
scopeName: str | None = None
ruleName: str | None = None
textmate_grammar.engine.rules.parse_include(include: str) → textmate_grammar.engine.rules.IncludeReference

Port of parseInclude (grammarDependencies.ts:278-295): $base / $self, #rule (relative), scope (top level), scope#rule (top-level repository reference). Note: this is the shared include parser; the full grammarDependency machinery (task 4b) should import it from here.

textmate_grammar.engine.rules.init_grammar(grammar: textmate_grammar.engine.rules.IRawGrammarDict, base: textmate_grammar.engine.rules.IRawRuleDict | None = None) → textmate_grammar.engine.rules.IRawGrammarDict

Port of initGrammar (grammar.ts:410-421): deep-copies the raw grammar and installs the $self and $base repository entries the include resolution relies on ($self = the top-level patterns, $base = the caller-provided base grammar’s self entry or the grammar’s own).

class textmate_grammar.engine.rules.RuleFactory

Port of RuleFactory (rule.ts:381-573).

static create_capture_rule(helper: textmate_grammar.engine.rules.IRuleFactoryHelper, location: textmate_grammar.engine.rules.ILocation, name: str | None, contentName: str | None, retokenizeCapturedWithRuleId: int) → textmate_grammar.engine.rules.CaptureRule

Port of createCaptureRule (rule.ts:383-387): registers the capture rule under a fresh id.

static get_compiled_rule_id(desc: textmate_grammar.engine.rules.IRawRuleDict, helper: textmate_grammar.engine.rules.IRuleFactoryHelper, repository: textmate_grammar.engine.rules.IRawRepositoryDict) → int

Port of getCompiledRuleId (rule.ts:389-447): the dispatch on the raw rule’s shape, memoized on the rule dict via its internal id field.

Note on recursion: upstream writes desc.id = id inside the factory callback, which is invoked synchronously by registerRule — so the id is stored before any nested getCompiledRuleId runs, exactly as here.

static compile_captures(captures: textmate_grammar.engine.rules.IRawRuleDict | None, helper: textmate_grammar.engine.rules.IRuleFactoryHelper, repository: textmate_grammar.engine.rules.IRawRepositoryDict) → list[textmate_grammar.engine.rules.CaptureRule | None]

Port of _compileCaptures (rule.ts:449-485): builds the (CaptureRule | None)[] indexed by capture group number 1..n.

static compile_patterns(patterns: list[textmate_grammar.engine.rules.IRawRuleDict] | None, helper: textmate_grammar.engine.rules.IRuleFactoryHelper, repository: textmate_grammar.engine.rules.IRawRepositoryDict) → textmate_grammar.engine.rules.CompilePatternsResult

Port of _compilePatterns (rule.ts:487-572): compiles the raw pattern list to rule ids, preserving order, resolving include entries and dropping rules whose include chain resolves to nothing instantiated.

textmate_grammar.engine.rules.create_rule(desc: textmate_grammar.engine.rules.IRawRuleDict, helper: textmate_grammar.engine.rules.IRuleFactoryHelper, repository: textmate_grammar.engine.rules.IRawRepositoryDict) → int

Mirror of RuleFactory.getCompiledRuleId (task-card spelling); prefer RuleFactory.get_compiled_rule_id().