textmate_grammar.engine.raw_grammar¶
Faithful port of microsoft/vscode-textmate src/rawGrammar.ts,
src/parseRawGrammar.ts and src/json.ts (v9.3.2).
rawGrammar.ts in the pinned upstream release is purely type-level
(IRawGrammar, IRawRule, IRawCaptures, IRawRepository,
ILocation): there is no runtime RawGrammar/RawCaptures class and no
valueAsString coercion in the pinned source (those notions date back to
pre-1.0 vscode-textmate; the modern engine simply runs on the any objects
produced by JSON.parse/plist.parsePLIST, so upstream types are runtime
lies). The upstream interfaces are mirrored here as TypedDict shapes; the
scalar coercion helper value_as_string() covers the one coercion
downstream rule-building code must not guess at (JS String() semantics for
string-typed fields that may technically hold plist scalars).
parseRawGrammar.ts semantics, exactly as upstream:
a file path ending with
.json(case sensitive suffix test:/\.json$/) → JSON grammar; everything else → plist grammar. There is no content sniffing and no YAML support upstream: unknown extensions are handed to the plist parser, which fails withNear offset ...: expected <for non-XML content (preserved here).the production path is strict JSON: upstream calls
JSON.parseand only uses the hand-rolledparseJSONtokenizer fromjson.tsin debug mode.json.loadsmirrorsJSON.parse(strict JSON: no comments, no trailing commas).
json.ts is ported in full and exported as parse_json() for the debug
path (VSCODE_TEXTMATE_DEBUG environment variable, standing in for
upstream DebugFlags.InDebugMode). Its tolerances — verified against the TS
source — are not jsonc: it supports neither comments nor trailing commas; its
leniency lies in JS parseFloat-style number tokens (1-2 parses as
1) and in accepting trailing garbage after the root construct.
Module Contents¶
Classes¶
Source location of a grammar construct ( |
|
Tokenizer state ( |
Functions¶
Coerce a raw grammar scalar to a string with JS |
|
Parse a grammar from |
|
Produce the next token ( |
|
|
Data¶
API¶
- textmate_grammar.engine.raw_grammar.LOCATION_KEY = '$vscodeTextmateLocation'¶
- class textmate_grammar.engine.raw_grammar.ILocation¶
Bases:
typing.TypedDictSource location of a grammar construct (
ILocationupstream).Initialization
Initialize self. See help(type(self)) for accurate signature.
- textmate_grammar.engine.raw_grammar.ILocatable = 'TypedDict(...)'¶
- textmate_grammar.engine.raw_grammar.IRawCaptures = None¶
- textmate_grammar.engine.raw_grammar.IRawRepository = None¶
- textmate_grammar.engine.raw_grammar.IRawRule = 'TypedDict(...)'¶
- textmate_grammar.engine.raw_grammar.IRawGrammar = 'TypedDict(...)'¶
- textmate_grammar.engine.raw_grammar.value_as_string(value: Any) str¶
Coerce a raw grammar scalar to a string with JS
String()semantics.Practical fallback for rule-building code that must not guess how grammar scalars (which may technically be
<integer>/<real>/<true>values in wayward tmLanguage files) render as strings:true/false→"true"/"false"(JS, not Python capitalization)numbers → shortest double representation (
1.0→"1"),NaN/Infinitymatch JS tokenseverything else →
str(value)
- textmate_grammar.engine.raw_grammar.IN_DEBUG_MODE = 'is_in_debug_mode(...)'¶
- textmate_grammar.engine.raw_grammar.parse_raw_grammar(content: str, file_path: str | None = None) Any¶
Parse a grammar from
content(parseRawGrammar.tsupstream).Routing is by file path only: paths ending in
.jsonparse as JSON, everything else (including.jsonc,.yamland.xml) parses as plist — upstream supports nothing else.
- class textmate_grammar.engine.raw_grammar.JSONStreamState(source: str)¶
Tokenizer state (
JSONStreamStateupstream).Initialization
- textmate_grammar.engine.raw_grammar.next_json_token(stream_state: textmate_grammar.engine.raw_grammar.JSONStreamState, out_token: textmate_grammar.engine.raw_grammar._JSONToken) bool¶
Produce the next token (
nextJSONTokenupstream).Precondition upstream: the source is known to be valid JSON. Note the quirks: on
False(EOS or an invalid keyword tail) the stream state is not updated; a failing string/keyword read leavesout_tokenwith the type set but no value/updated state.
- textmate_grammar.engine.raw_grammar.parse_json(source: str, filename: str | None, with_metadata: bool) Any¶
parseJSONupstream — the hand-rolled, JS-quirky JSON tokenizer.Only used on the debug path upstream (
VSCODE_TEXTMATE_DEBUG); embeds$vscodeTextmateLocationwhenwith_metadatais set.Like the TS engine, positions are indexed by UTF-16 code units; python values are returned with surrogate pairs recombined into astral characters (see
plist._units_of/_text_of).