textmate_grammar.engine.oniglib¶
Port of vscode-textmate’s src/onigLib.ts — the oniguruma library adapter.
- vscode-textmate drives tokenization through an injectable
IOnigLibthat provides scanners and strings. Upstream this is backed by
vscode-oniguruma(a wasm build of
oniguruma ~6.9). Here the same surface is provided on top of onigurumacffi, an
ffi binding of oniguruma 6.9 — the same engine family, so the semantics carry over:
Pattern selection (leftmost match wins, ties resolved by the lowest pattern index) is done by oniguruma’s RegSet API (
onig_regset_searchwithONIG_REGSET_POSITION_LEAD), which is exactly the strategy vscode-oniguruma’s C++findNextOnigScannerMatchuses for short strings.The critical semantic (see the parity plan):
\Gmatches exactly at the start offset of the scan, i.e. at thestart_positionpassed to eachOnigScanner.find_next_match_sync()call, not at a stored anchor. oniguruma binds\Gto the search start, and vscode-textmate passes the token offset per call —Pattern.search(content, start=start_position)reproduces this 1:1. A pattern compiled once therefore matches at different offsets for differentstart_positionvalues.
Capture indices are returned in Python code-point offsets (consistent with the rest
of the Python port; upstream returns UTF-16 offsets, having decoded the UTF-8 wasm
buffer). Unmatched capture groups — which oniguruma reports with offsets of -1 —
are mapped to start == end == 0, the same clamp vscode-oniguruma applies when its
convertUtf8OffsetToUtf16 encounters values below zero.
Every non-obvious piece references the ported section of vscode-oniguruma / onigLib.ts.
Module Contents¶
Classes¶
Port of the |
|
Port of |
|
Port of |
|
Port of |
|
Port of |
|
Port of the |
|
The concrete onigurumacffi-backed |
Functions¶
Port of |
Data¶
API¶
- class textmate_grammar.engine.oniglib.FindOption¶
Bases:
enum.IntFlagPort of the
FindOptionconst enum fromonigLib.ts(onigLib.ts:23-41).The bit values are vscode-textmate’s own numbering; they are translated to the oniguruma search options named in the upstream member comments (
_translate_options()).DEBUG_CALLis a debugging marker only and is dropped, like thedebugCallflag is in vscode-oniguruma’s non-debug path.Initialization
Initialize self. See help(type(self)) for accurate signature.
- NONE = 0¶
- NOT_BEGIN_STRING = 1¶
- NOT_END_STRING = 2¶
- NOT_BEGIN_POSITION = 4¶
- DEBUG_CALL = 8¶
- class textmate_grammar.engine.oniglib.IOnigCaptureIndex¶
Port of
IOnigCaptureIndex(onigLib.ts:12-16).
- class textmate_grammar.engine.oniglib.IOnigMatch¶
Port of
IOnigMatch(onigLib.ts:18-21): the index of the matching pattern within the scanner and the captured regions of that match.- capture_indices: tuple[textmate_grammar.engine.oniglib.IOnigCaptureIndex, ...] = None¶
- class textmate_grammar.engine.oniglib.OnigString(content: str)¶
Port of
OnigString(onigLib.ts:48-51).Upstream this wraps the wasm-resident UTF-8 encoding of the string and carries the UTF-8/UTF-16 offset translation tables; Python strings are indexed by code point natively, so this is the light wrapper keeping
contentaround, with a no-opdispose(parity of the optionaldispose?member).Initialization
- textmate_grammar.engine.oniglib.disposeOnigString(string: textmate_grammar.engine.oniglib.OnigString) None¶
Port of
disposeOnigString(onigLib.ts:53-57).
- class textmate_grammar.engine.oniglib.OnigScanner(sources: collections.abc.Sequence[str])¶
Port of
OnigScannerfrom vscode-oniguruma — the scanner behindIOnigLib.createOnigScanner(onigLib.ts:43-46).Patterns are compiled eagerly at construction, like vscode-oniguruma does; an invalid pattern raises
onigurumacffi.OnigErrorwhere upstream throws the oniguruma error message. Matching uses one RegSet search per call, whoseONIG_REGSET_POSITION_LEADstrategy yields the same candidate selection as vscode-oniguruma’s C++findNextOnigScannerMatch: the match beginning at the smallest offset wins and ties break toward the lowest pattern index.Initialization
- find_next_match_sync(string: str | textmate_grammar.engine.oniglib.OnigString, start_position: int = 0, options: textmate_grammar.engine.oniglib.FindOption = FindOption.NONE) textmate_grammar.engine.oniglib.IOnigMatch | None¶
Port of
OnigScanner.findNextMatchSync(onigLib.ts:44).stringmay be a plain string or anOnigString, as upstream.start_positionis the offset the scan starts at, which is where\Gbinds: a pattern containing\Gis compiled once and still matches at whatever initial offset the caller passes.
- find_best_match(position: int, line: str, has_anchor: bool = False, has_back_references: bool = False) textmate_grammar.engine.oniglib.IOnigMatch | None¶
Convenience mirroring the pre-4.0
IOnigScanner.findBestMatch(position, line, hasAnchor, hasBackReferences)surface named in the porting plan.positionis the scan offset, so a\Gbinds there: withhas_anchor=Truethe patterns are scanned as-is, withhas_anchor=FalseFindOption.NOT_BEGIN_POSITIONmakes every\Gfail (the same equivalent option vscode-textmate’sgetFindOptionsbuilds forallowG=Falseingrammar/tokenizeString.ts:549-557).has_back_referencesis bookkeeping parity for the old signature only: in the current vscode-textmate,\nbackreference patterns are resolved before scanning (rule.tsresolveBackReferences, ported intextmate_grammar.engine.regexsource), so the scanner has nothing to do with them — the flag is accepted and unused.
- class textmate_grammar.engine.oniglib.IOnigLib¶
Bases:
typing.ProtocolPort of the
IOnigLibinterface (onigLib.ts:7-10): the library surface vscode-textmate expects the tokenization engine to be wired against.- create_onig_scanner(sources: collections.abc.Sequence[str]) textmate_grammar.engine.oniglib.OnigScanner¶
Port of
createOnigScanner(sources).
- create_onig_string(string: str) textmate_grammar.engine.oniglib.OnigString¶
Port of
createOnigString(str).
- class textmate_grammar.engine.oniglib.OnigLib¶
The concrete onigurumacffi-backed
IOnigLib— the role vscode-oniguruma plays upstream whenmain.tswires up the default library.- create_onig_scanner(sources: collections.abc.Sequence[str]) textmate_grammar.engine.oniglib.OnigScanner¶
- create_onig_string(string: str) textmate_grammar.engine.oniglib.OnigString¶
- textmate_grammar.engine.oniglib.DEFAULT_ONIG_LIB: textmate_grammar.engine.oniglib.IOnigLib = 'OnigLib(...)'¶