textmate_grammar.engine.oniglib

Port of vscode-textmate’s src/onigLib.ts — the oniguruma library adapter.

vscode-textmate drives tokenization through an injectable IOnigLib that provides

scanners and strings. Upstream this is backed by vscode-oniguruma (a wasm build of

oniguruma ~6.9). Here the same surface is provided on top of onigurumacffi, an ffi binding of oniguruma 6.9 — the same engine family, so the semantics carry over:

  • Pattern selection (leftmost match wins, ties resolved by the lowest pattern index) is done by oniguruma’s RegSet API (onig_regset_search with ONIG_REGSET_POSITION_LEAD), which is exactly the strategy vscode-oniguruma’s C++ findNextOnigScannerMatch uses for short strings.

  • The critical semantic (see the parity plan): \G matches exactly at the start offset of the scan, i.e. at the start_position passed to each OnigScanner.find_next_match_sync() call, not at a stored anchor. oniguruma binds \G to the search start, and vscode-textmate passes the token offset per call — Pattern.search(content, start=start_position) reproduces this 1:1. A pattern compiled once therefore matches at different offsets for different start_position values.

Capture indices are returned in Python code-point offsets (consistent with the rest of the Python port; upstream returns UTF-16 offsets, having decoded the UTF-8 wasm buffer). Unmatched capture groups — which oniguruma reports with offsets of -1 — are mapped to start == end == 0, the same clamp vscode-oniguruma applies when its convertUtf8OffsetToUtf16 encounters values below zero.

Every non-obvious piece references the ported section of vscode-oniguruma / onigLib.ts.

Module Contents

Classes

FindOption

Port of the FindOption const enum from onigLib.ts (onigLib.ts:23-41).

IOnigCaptureIndex

Port of IOnigCaptureIndex (onigLib.ts:12-16).

IOnigMatch

Port of IOnigMatch (onigLib.ts:18-21): the index of the matching pattern within the scanner and the captured regions of that match.

OnigString

Port of OnigString (onigLib.ts:48-51).

OnigScanner

Port of OnigScanner from vscode-oniguruma — the scanner behind IOnigLib.createOnigScanner (onigLib.ts:43-46).

IOnigLib

Port of the IOnigLib interface (onigLib.ts:7-10): the library surface vscode-textmate expects the tokenization engine to be wired against.

OnigLib

The concrete onigurumacffi-backed IOnigLib — the role vscode-oniguruma plays upstream when main.ts wires up the default library.

Functions

disposeOnigString

Port of disposeOnigString (onigLib.ts:53-57).

Data

API

class textmate_grammar.engine.oniglib.FindOption

Bases: enum.IntFlag

Port of the FindOption const enum from onigLib.ts (onigLib.ts:23-41).

The bit values are vscode-textmate’s own numbering; they are translated to the oniguruma search options named in the upstream member comments (_translate_options()). DEBUG_CALL is a debugging marker only and is dropped, like the debugCall flag is in vscode-oniguruma’s non-debug path.

Initialization

Initialize self. See help(type(self)) for accurate signature.

NONE = 0
NOT_BEGIN_STRING = 1
NOT_END_STRING = 2
NOT_BEGIN_POSITION = 4
DEBUG_CALL = 8
class textmate_grammar.engine.oniglib.IOnigCaptureIndex

Port of IOnigCaptureIndex (onigLib.ts:12-16).

start: int = None
end: int = None
length: int = None
class textmate_grammar.engine.oniglib.IOnigMatch

Port of IOnigMatch (onigLib.ts:18-21): the index of the matching pattern within the scanner and the captured regions of that match.

index: int = None
capture_indices: tuple[textmate_grammar.engine.oniglib.IOnigCaptureIndex, ...] = None
class textmate_grammar.engine.oniglib.OnigString(content: str)

Port of OnigString (onigLib.ts:48-51).

Upstream this wraps the wasm-resident UTF-8 encoding of the string and carries the UTF-8/UTF-16 offset translation tables; Python strings are indexed by code point natively, so this is the light wrapper keeping content around, with a no-op dispose (parity of the optional dispose? member).

Initialization

dispose() → None

No-op — there is no wasm string buffer to release.

textmate_grammar.engine.oniglib.disposeOnigString(string: textmate_grammar.engine.oniglib.OnigString) → None

Port of disposeOnigString (onigLib.ts:53-57).

class textmate_grammar.engine.oniglib.OnigScanner(sources: collections.abc.Sequence[str])

Port of OnigScanner from vscode-oniguruma — the scanner behind IOnigLib.createOnigScanner (onigLib.ts:43-46).

Patterns are compiled eagerly at construction, like vscode-oniguruma does; an invalid pattern raises onigurumacffi.OnigError where upstream throws the oniguruma error message. Matching uses one RegSet search per call, whose ONIG_REGSET_POSITION_LEAD strategy yields the same candidate selection as vscode-oniguruma’s C++ findNextOnigScannerMatch: the match beginning at the smallest offset wins and ties break toward the lowest pattern index.

Initialization

property sources: tuple[str, ...]
find_next_match_sync(string: str | textmate_grammar.engine.oniglib.OnigString, start_position: int = 0, options: textmate_grammar.engine.oniglib.FindOption = FindOption.NONE) → textmate_grammar.engine.oniglib.IOnigMatch | None

Port of OnigScanner.findNextMatchSync (onigLib.ts:44).

string may be a plain string or an OnigString, as upstream. start_position is the offset the scan starts at, which is where \G binds: a pattern containing \G is compiled once and still matches at whatever initial offset the caller passes.

find_best_match(position: int, line: str, has_anchor: bool = False, has_back_references: bool = False) → textmate_grammar.engine.oniglib.IOnigMatch | None

Convenience mirroring the pre-4.0 IOnigScanner.findBestMatch(position, line, hasAnchor, hasBackReferences) surface named in the porting plan.

position is the scan offset, so a \G binds there: with has_anchor=True the patterns are scanned as-is, with has_anchor=False FindOption.NOT_BEGIN_POSITION makes every \G fail (the same equivalent option vscode-textmate’s getFindOptions builds for allowG=False in grammar/tokenizeString.ts:549-557).

has_back_references is bookkeeping parity for the old signature only: in the current vscode-textmate, \n backreference patterns are resolved before scanning (rule.ts resolveBackReferences, ported in textmate_grammar.engine.regexsource), so the scanner has nothing to do with them — the flag is accepted and unused.

dispose() → None

Port of the optional dispose member: drops the compiled pattern set.

class textmate_grammar.engine.oniglib.IOnigLib

Bases: typing.Protocol

Port of the IOnigLib interface (onigLib.ts:7-10): the library surface vscode-textmate expects the tokenization engine to be wired against.

create_onig_scanner(sources: collections.abc.Sequence[str]) → textmate_grammar.engine.oniglib.OnigScanner

Port of createOnigScanner(sources).

create_onig_string(string: str) → textmate_grammar.engine.oniglib.OnigString

Port of createOnigString(str).

class textmate_grammar.engine.oniglib.OnigLib

The concrete onigurumacffi-backed IOnigLib — the role vscode-oniguruma plays upstream when main.ts wires up the default library.

create_onig_scanner(sources: collections.abc.Sequence[str]) → textmate_grammar.engine.oniglib.OnigScanner
create_onig_string(string: str) → textmate_grammar.engine.oniglib.OnigString
textmate_grammar.engine.oniglib.DEFAULT_ONIG_LIB: textmate_grammar.engine.oniglib.IOnigLib = 'OnigLib(...)'