textmate_grammar.engine.raw_grammar

Faithful port of microsoft/vscode-textmate src/rawGrammar.ts, src/parseRawGrammar.ts and src/json.ts (v9.3.2).

rawGrammar.ts in the pinned upstream release is purely type-level (IRawGrammar, IRawRule, IRawCaptures, IRawRepository, ILocation): there is no runtime RawGrammar/RawCaptures class and no valueAsString coercion in the pinned source (those notions date back to pre-1.0 vscode-textmate; the modern engine simply runs on the any objects produced by JSON.parse/plist.parsePLIST, so upstream types are runtime lies). The upstream interfaces are mirrored here as TypedDict shapes; the scalar coercion helper value_as_string() covers the one coercion downstream rule-building code must not guess at (JS String() semantics for string-typed fields that may technically hold plist scalars).

parseRawGrammar.ts semantics, exactly as upstream:

  • a file path ending with .json (case sensitive suffix test: /\.json$/) → JSON grammar; everything else → plist grammar. There is no content sniffing and no YAML support upstream: unknown extensions are handed to the plist parser, which fails with Near offset ...: expected < for non-XML content (preserved here).

  • the production path is strict JSON: upstream calls JSON.parse and only uses the hand-rolled parseJSON tokenizer from json.ts in debug mode. json.loads mirrors JSON.parse (strict JSON: no comments, no trailing commas).

json.ts is ported in full and exported as parse_json() for the debug path (VSCODE_TEXTMATE_DEBUG environment variable, standing in for upstream DebugFlags.InDebugMode). Its tolerances — verified against the TS source — are not jsonc: it supports neither comments nor trailing commas; its leniency lies in JS parseFloat-style number tokens (1-2 parses as 1) and in accepting trailing garbage after the root construct.

Module Contents

Classes

ILocation

Source location of a grammar construct (ILocation upstream).

JSONStreamState

Tokenizer state (JSONStreamState upstream).

Functions

value_as_string

Coerce a raw grammar scalar to a string with JS String() semantics.

parse_raw_grammar

Parse a grammar from content (parseRawGrammar.ts upstream).

next_json_token

Produce the next token (nextJSONToken upstream).

parse_json

parseJSON upstream — the hand-rolled, JS-quirky JSON tokenizer.

Data

API

textmate_grammar.engine.raw_grammar.LOCATION_KEY = '$vscodeTextmateLocation'
class textmate_grammar.engine.raw_grammar.ILocation

Bases: typing.TypedDict

Source location of a grammar construct (ILocation upstream).

Initialization

Initialize self. See help(type(self)) for accurate signature.

filename: str | None = None
line: int = None
char: int = None
textmate_grammar.engine.raw_grammar.ILocatable = 'TypedDict(...)'
textmate_grammar.engine.raw_grammar.IRawCaptures = None
textmate_grammar.engine.raw_grammar.IRawRepository = None
textmate_grammar.engine.raw_grammar.IRawRule = 'TypedDict(...)'
textmate_grammar.engine.raw_grammar.IRawGrammar = 'TypedDict(...)'
textmate_grammar.engine.raw_grammar.value_as_string(value: Any) → str

Coerce a raw grammar scalar to a string with JS String() semantics.

Practical fallback for rule-building code that must not guess how grammar scalars (which may technically be <integer>/<real>/<true> values in wayward tmLanguage files) render as strings:

  • true/false → "true"/"false" (JS, not Python capitalization)

  • numbers → shortest double representation (1.0 → "1"), NaN/Infinity match JS tokens

  • everything else → str(value)

textmate_grammar.engine.raw_grammar.IN_DEBUG_MODE = 'is_in_debug_mode(...)'
textmate_grammar.engine.raw_grammar.parse_raw_grammar(content: str, file_path: str | None = None) → Any

Parse a grammar from content (parseRawGrammar.ts upstream).

Routing is by file path only: paths ending in .json parse as JSON, everything else (including .jsonc, .yaml and .xml) parses as plist — upstream supports nothing else.

class textmate_grammar.engine.raw_grammar.JSONStreamState(source: str)

Tokenizer state (JSONStreamState upstream).

Initialization

source: str = None
pos: int = None
length: int = None
line: int = None
char: int = None
textmate_grammar.engine.raw_grammar.next_json_token(stream_state: textmate_grammar.engine.raw_grammar.JSONStreamState, out_token: textmate_grammar.engine.raw_grammar._JSONToken) → bool

Produce the next token (nextJSONToken upstream).

Precondition upstream: the source is known to be valid JSON. Note the quirks: on False (EOS or an invalid keyword tail) the stream state is not updated; a failing string/keyword read leaves out_token with the type set but no value/updated state.

textmate_grammar.engine.raw_grammar.parse_json(source: str, filename: str | None, with_metadata: bool) → Any

parseJSON upstream — the hand-rolled, JS-quirky JSON tokenizer.

Only used on the debug path upstream (VSCODE_TEXTMATE_DEBUG); embeds $vscodeTextmateLocation when with_metadata is set.

Like the TS engine, positions are indexed by UTF-16 code units; python values are returned with surrogate pairs recombined into astral characters (see plist._units_of/_text_of).