swift-semantic-string

Semantic markup and post-processing for rendered output.

The package ships two products:

OutputTransformer holds no concrete transformer. A transformer's tokens carry the vocabulary of whatever it describes (bitsNeededForTag, ivar offset), which belongs to the library that owns that domain rather than to a general-purpose string package. The concrete modules therefore ship elsewhere and all extend this one namespace, so a consumer importing several of them still writes Transformer.CType unqualified:

• Swift 6.2+
• No external dependencies
• Apple platforms: macOS 10.15+, iOS 13+, macCatalyst 13+, tvOS 13+, watchOS 6+, visionOS 1+
• Also builds on Linux and Windows. The only system dependency is a platform locking primitive (os_unfair_lock, pthread_mutex_t, or SRWLOCK), selected behind #if canImport.

dependencies: [
    .package(url: "https://github.com/MxIris-Reverse-Engineering/swift-semantic-string.git", from: "0.6.0")
]

Then add "Semantic" — and "OutputTransformer" if you are writing a transformer module of your own — as dependencies of your target.

import Semantic

// Use the result builder to compose semantic strings declaratively
@SemanticStringBuilder
var declaration: SemanticString {
    Keyword("public")
    Space()
    Keyword("struct")
    Space()
    TypeName(kind: .struct, "MyType")
}

// Access the plain text
print(declaration.string) // "public struct MyType"

// Enumerate with semantic info for rendering
declaration.enumerate { text, type in
    switch type {
    case .keyword:
        renderAsKeyword(text)
    case .type:
        renderAsTypeName(text)
    default:
        renderAsPlainText(text)
    }
}

SemanticType categorizes each text fragment:

Atomic components are leaf nodes — each holds a single string and its semantic type.

Keyword("func")                        // .keyword
Variable("count")                      // .variable
Numeric("42")                          // .numeric
Argument("name")                       // .argument
Comment("// TODO")                     // .comment
TypeName(kind: .struct, "Int")         // .type(.struct, .name)
TypeDeclaration(kind: .class, "Foo")   // .type(.class, .declaration)
FunctionName("viewDidLoad")            // .function(.name)
FunctionDeclaration("init")            // .function(.declaration)
MemberName("count")                    // .member(.name)
MemberDeclaration("title")             // .member(.declaration)
Standard("(")                          // .standard
Space()                                // " "
BreakLine()                            // "\n"
Indent(level: 2)                       // "        " (4 spaces per level)

Composite components combine multiple children.

Groups components together, with an optional separator:

let modifiers = Group {
    Keyword("public")
    Keyword("static")
    Keyword("func")
}.separator(Space())
// "public static func"

Joins items with a separator, automatically filtering out empty items:

Joined(separator: ", ", prefix: "(", suffix: ")") {
    if hasLabel {
        Standard("label")
    }
    Standard("value")
}
// With hasLabel=true:  "(label, value)"
// With hasLabel=false: "(value)"

Iterate over collections:

ForEach(parameters, separator: ", ") { param in
    Standard(param.name)
    Standard(": ")
    TypeName(kind: .other, param.typeName)
}

ForEachIndexed(items) { item, info in
    if !info.isFirst {
        Standard(", ")
    }
    Standard(item.name)
}

Conditional content based on optionals:

IfLet(superclass) { name in
    Standard(": ")
    TypeName(kind: .class, name)
}

Structured declaration with header, braces, and indented body:

DeclarationBlock(level: 0) {
    Keyword("struct")
    Space()
    TypeName(kind: .struct, "Point")
} body: {
    MemberList(level: 1) {
        SemanticString {
            Keyword("var")
            Space()
            Variable("x")
            Standard(": ")
            TypeName(kind: .struct, "Int")
        }
        SemanticString {
            Keyword("var")
            Space()
            Variable("y")
            Standard(": ")
            TypeName(kind: .struct, "Int")
        }
    }
}
// struct Point {
//     var x: Int
//     var y: Int
// }

@SemanticStringBuilder supports the full range of Swift result builder features:

@SemanticStringBuilder
func describe(_ decl: SomeDecl) -> SemanticString {
    // Conditionals
    if decl.isPublic {
        Keyword("public")
        Space()
    }

    // Switch / if-else
    if decl.isStatic {
        Keyword("static")
        Space()
    }

    Keyword("func")
    Space()
    FunctionName(decl.name)

    // For loops
    Joined(separator: ", ", prefix: "(", suffix: ")") {
        for param in decl.parameters {
            Standard(param)
        }
    }

    // Optionals via IfLet
    IfLet(decl.returnType) { ret in
        Standard(" -> ")
        TypeName(kind: .other, ret)
    }
}

SemanticString provides a rich API for transformation and inspection:

let ss = SemanticString { ... }

// Concatenation
let combined = ss + otherString
ss += moreContent

// Transformation
let uppercased = ss.map { AtomicComponent(string: $0.string.uppercased(), type: $0.type) }
let allStandard = ss.replacing { _ in .standard }

// Wrapping
let wrapped = ss.parenthesized()   // "(content)"
let generic = ss.angleBracketed()  // "<content>"

// Trimming
let trimmed = ss.trimmingWhitespace()

// Conditional operations
let result = ss
    .prefixed(with: "public ", if: isPublic)
    .suffixed(with: "?", if: isOptional)

// Querying
ss.contains(type: .keyword)
ss.hasPrefix("func")
ss.starts(with: .keyword)

// Filtering
let keywords = ss.filter(byType: .keyword)

// Slicing
let first3 = ss.prefix(3)
let rest = ss.dropFirst()

SemanticString conforms to Codable, serializing as an array of AtomicComponent values:

let data = try JSONEncoder().encode(semanticString)
let decoded = try JSONDecoder().decode(SemanticString.self, from: data)

A component whose string is empty contributes nothing to any rendering, and building one through a component (Keyword("")) has always discarded it. Constructing a string from atomic components directly does the same, so both ways of assembling the same content agree:

let components = [
    AtomicComponent(string: "a", type: .standard),
    AtomicComponent(string: "", type: .standard),   // dropped
    AtomicComponent(string: "b", type: .standard),
]
SemanticString(components: components).count   // 2, not 3

This holds on every construction path — hand-built arrays, streamed appends, decoding, and the flattening of custom components whose buildComponents() emits an empty atom. A zero-length entry does not survive into count, components, isEmpty, subscripting, prefix / suffix, equality, hashing, or a Codable round trip (which is therefore always idempotent: encoded output never contains an empty component). The emptiness measures all coincide: isEmpty ⟺ count == 0 ⟺ first == nil ⟺ string.isEmpty, and equal values agree on all of them. Callers that pair a map result positionally against the source array should filter empties themselves first, and callers migrating from the element-tree storage should note that map-ing a component to "" now removes it (shifting indices) instead of keeping an empty placeholder. The same applies across versions: a payload encoded by the element-tree version that contains zero-length components decodes here with those entries dropped — component indices recorded alongside such a payload (a span map, a selection anchor, a decoration table) shift past every dropped entry and must be re-derived from the decoded value.

Empty atoms contribute no text directly, and container emptiness decisions see the filtered form: an item or element consisting only of empty atoms produces no row in MemberList/Rows/BlockList, no separator in Group/Joined/ForEach, and no braces content in DeclarationBlock. This unifies two shapes that historically behaved differently — Keyword("") (whose default buildComponents() returns nothing) was always skipped, while a custom component emitting one empty atom was rendered as an empty row. Both now flatten to the same nothing and get the same treatment.

A component's buildComponents() runs at the moment the component is appended or placed in a container — not when the result is rendered. Its output is copied into storage and the component itself is released. buildComponents() takes no context and is expected to be deterministic, so the flattening can only ever be done once, and doing it eagerly keeps every read path free of side effects.

This matters if a custom component reads mutable state that is resolved after assembly — a deferred-resolution pattern where a printer builds the structure first and fills in names, offsets, or addresses afterwards:

final class LateBoundName { var value = "placeholder" }

struct NameReference: SemanticStringComponent {
    let holder: LateBoundName
    func buildComponents() -> [AtomicComponent] {
        [AtomicComponent(string: holder.value, type: .standard)]
    }
}

let holder = LateBoundName()
let group = Group([SemanticString(NameReference(holder: holder))])
holder.value = "resolved"
SemanticString(group).string   // "placeholder" — captured at append time

The element-tree storage retained the component and called buildComponents() at render time, so the same code produced "resolved". Resolve such values before appending them, or store a SemanticString placeholder and rebuild the container once the value is known.

SemanticString is the builder — composable, streamable, mutable. When a string is fully built and will only ever be read (rendered, encoded, exported, searched), freeze it:

let frozen = declaration.frozen()

print(frozen.string)   // free — the full text is the storage

frozen.enumerateSpans { spanText, type, identifier in
    render(spanText, as: type, linkingTo: identifier)
}

Freezing collapses the per-token component representation into three flat allocations:

Every stored property is let, so FrozenSemanticString is unconditionally Sendable with no locks and no copy-on-write. Freezing is one-way by design: there is no mutation API and no conversion back, which pins the "build, then read-only" lifecycle at the type level.

On a large corpus (3.46M tokens, 27.3 MB of text) this cut resident memory by roughly 78% against the per-token representation, with printing time unchanged.

Notes:

• Use enumerateSpans(_:) on hot paths. frozen.components materializes [AtomicComponent] and recreates exactly the per-token allocation cost freezing removed — it exists for compatibility and tests.
• Every span boundary falls on a Unicode scalar boundary — that is the guarantee to build on. Grapheme cluster (Character) alignment holds only within a token: Span.length is a UInt16, so a token longer than 65535 UTF-8 bytes is split into consecutive spans carrying the same type and identifier, at cluster boundaries (a single cluster that itself exceeds 65535 bytes degrades to scalar boundaries). Boundaries between tokens sit wherever the builder put the tokens, so adjacent tokens can split one Character — e.g. a keyword followed by a combining mark typed as a separate token. Consumers that concatenate text see no difference; styling or selection code that needs cluster-aligned runs must merge adjacent spans, and a token-by-token count comparison against SemanticString.components will see long-token splits. Note that the split also breaks value round trips: rebuilding a SemanticString from frozen.components after a long-token split produces a value that is not == to the source and hashes differently, even though its string matches — across a freeze/thaw boundary, compare rendered text (or re-freeze and compare snapshots), not SemanticString values.
• FrozenSemanticString conforms to Codable with a columnar encoding (one string plus four homogeneous arrays) an order of magnitude smaller on the wire than SemanticString's array-of-objects form. The two formats are intentionally not interchangeable — both ends of any persistence or cross-process channel must agree on which one they use.
• SemanticType values encode to stable UInt8 codes. Codes are assigned append-only, and a code from a newer version decodes as .other rather than failing. Note that resolving to .other is lossy in one direction: a value decoded from a newer encoder keeps the original code in spans, but round-tripping it through components and re-freezing rewrites it to .other's code. Forward and backward compatible readers should walk enumerateSpans or spans rather than components.
• Freeze at a storage boundary — once the string is final. There is deliberately no way to compact a SemanticString in place and keep building with it: the element boundaries a builder exposes are semantic (a container like MemberList treats one element as one row), and collapsing them silently changes layout. frozen() gives you the compact representation with that lifecycle enforced by the type.
• frozen() never inflates the value it is called on, or any value sharing its storage. The components are the storage (there is no component cache to fill), and the text reuses the string cache only when a previous render already paid for it — otherwise it is computed locally and not published.
• Decoding validates the structural invariants a reader would otherwise trap on: column counts must agree, span lengths must cover the text exactly, no span may be zero-length, and every span boundary must fall on a Unicode scalar boundary. Two fields are deliberately kept verbatim rather than rejected — an unknown typeCode resolves to .other and an out-of-range identifier index resolves to nil in every reader — so forward-compatible payloads (and this type's own encoder output) stay decodable. The unchecked init(text:spans:identifierTable:) trusts the caller instead — use it only for parts you produced yourself. Decoding is the path for anything that arrived from another process; note that validation cannot bound what JSONDecoder itself materializes while parsing, so cap the byte size of untrusted payloads before handing them to a decoder.
• Equality and hashing compare rendered content, not storage layout: identifierTable shape (duplicate or unreferenced entries, which a decoded payload keeps as they arrived) does not affect them. Two different SemanticStrings can still freeze to equal snapshots, because a token over 65535 bytes and the same text split across adjacent same-type tokens produce identical spans — compare the source strings when token identity matters.
• frozen() interns identifiers by String equality, which is Unicode-canonical: two identifiers whose bytes differ but compare canonically equal (an NFC and an NFD spelling of the same name) collapse into one table entry, and every span referencing either resolves to whichever spelling was interned first. SemanticString's own Codable path preserves identifier bytes exactly; freezing is the only path that can rewrite them. Mangled names are ASCII and unaffected — if identifiers can carry non-ASCII, keep them in a single normalization form before building.

When what a printer shows depends on display options — offset comments on or off, synthesized members hidden or not — it can print everything once and record, for each piece, under which options that piece is shown. A caller then projects the one printing onto any set of options and gets exactly the text the printer would have produced with them. Flipping an option costs a projection, not a reprint.

@SemanticStringBuilder
var printed: SemanticString {
    Keyword("var")
    Space()
    Variable("count")
    VisibilityRegion(.enabled("showOffsets")) {
        Space()
        Comment("offset: 0x10")
    }
}

let (text, regions) = printed.frozen().separatingVisibilityRegions()

print(regions.projection(of: text) { $0 == "showOffsets" }.text.string) // "var count // offset: 0x10"
print(regions.projection(of: text) { _ in false }.text.string)          // "var count"

• A VisibilityCondition is built from named options: .enabled(_:), .disabled(_:), .all(_:), .any(_:), .always, .never. Nested regions combine with .all.
• The condition travels in each atom's identifier, so the marked string renders byte for byte like printing everything. separatingVisibilityRegions() restores the original identifiers and returns the regions as a table of UTF-8 ranges; do not read a marked string's identifiers yourself.
• Wrap the optional content itself, not the container around it. MemberList, BlockList, Rows, Joined, Group, ForEach, NestedDeclaration and DeclarationBlock carry their items' conditions onto the line breaks, indentation, separators, braces and headers they emit, so a projection lays out like printing only the visible items, including when every item of a group disappears. Two layouts fall outside that: ForEachIndexed, whose content arranges itself from isFirst / isLast, and a DeclarationBlock body whose last visible byte changes with the options, since the block decides its closing line break from the body as printed in full.
• projectedUTF8Offset(ofOriginalUTF8Offset:) maps an offset in the separated text (a search hit, say) into the projection, or returns nil when the projection removed that byte.
• For output written piece by piece rather than composed, markAtoms(from:visibleUnder:) conditions every atom appended after a given index.
• Without regions nothing changes: the output is the same as before, and a container spends one look at each item's first atom.

The design and its validation are in docs/VisibilityRegions.md.

A printer that prints a definition together with the definitions nested in it — a type and its nested types — can wrap each nested one in a DefinitionRegion. A caller that also needs each nested definition on its own then takes it out of the parent's print instead of printing it a second time.

func printStruct(_ name: String, level: Int, nested: [String] = []) -> SemanticString {
    SemanticString {
        DeclarationBlock(level: level) {
            Keyword("struct")
            Space()
            TypeDeclaration(kind: .struct, name)
        } body: {
            for nestedName in nested {
                NestedDeclaration {
                    DefinitionRegion(nestedName, content: printStruct(nestedName, level: level + 1))
                }
            }
        }
    }
}

let printed = printStruct("Outer", level: 1, nested: ["Inner"]).frozen()
let (text, definitions) = printed.separatingDefinitionRegions()
let inner = definitions.regions(withIdentity: "Inner")[0]
let innerOnItsOwn = printed.content(ofDefinitionRegion: inner).removingIndentation(levels: inner.depth + 1)
// innerOnItsOwn == printStruct("Inner", level: 1).frozen()

• The identity travels in each atom's identifier, so the marked string renders byte for byte like the unmarked one. separatingDefinitionRegions() restores the identifiers and returns each region's identity, UTF-8 range, depth and enclosing region. An identity must be non-empty and free of control characters.
• A region covers its content and nothing else: what a container prints around it, such as the line break in front of a NestedDeclaration, belongs to the enclosing text.
• content(ofDefinitionRegion:) slices the marked string to one region, removes the identities of that region and of the regions around it — regions nested inside keep theirs, visibility conditions stay — and rebuilds the identifier table the way freezing does, so the result's separations match those of the definition's own print. Call it on the marked string, or on it after separatingVisibilityRegions(), not on the text separatingDefinitionRegions() returns.
• removingIndentation(levels:) removes four spaces per level from the start of every line, inside multi-line atoms too; lines holding only a line break stay as they are, and it returns nil when another line lacks the indentation.
• Definition and visibility regions combine in either nesting order with identical atoms, and the two separations run in either order.

The design and its validation are in docs/DefinitionRegions.md.

A printer knows the structure of what it prints: that a comment gives a member's address, which part of it is the label and which the value, how the nodes of a field-offset tree nest. It can mark that structure as it prints, so that a reader presents it — draws a comment as a tag, folds a tree to its root — without parsing text that a user may have reworded through a template.

extension AnnotationRegion.Kind {
    static let memberAddress = Self("memberAddress")
}

let printed = SemanticString {
    AnnotationRegion(.memberAddress) {
        AnnotatedAtom(type: .comment) {
            AnnotationSegment(.marker) { "// " }
            AnnotationSegment(.label) { "Address" }
            AnnotationSegment(.separator) { ": " }
            AnnotationSegment(.value) { "0x135DC" }
        }
    }
}.frozen()

let (text, annotations) = printed.separatingAnnotationRegions()
// text == Comment("Address: 0x135DC").asSemanticString().frozen()
// annotations.regions:  memberAddress at UTF-8 0 ..< 19
// annotations.segments: marker 0 ..< 3, label 3 ..< 10, separator 10 ..< 12, value 12 ..< 19

• AnnotationRegion marks content with a kind; AnnotationSegment marks part of a region with a role. Both keep their content's text and element structure. Regions nest — a region inside one of its own kind makes a tree — and two regions or segments of one kind directly next to each other read back as two.
• AnnotatedAtom prints what would be one atom as pieces, so that its parts can sit in segments and regions. The separation merges the pieces back into the spans the atom freezes to, so the separated string equals the one printed with the atom whole, span for span. The pieces take the atom's type and identifier and may not carry their own; marks added around the atom afterwards go on every piece.
• Kinds and roles are strings wrapped in AnnotationRegion.Kind and AnnotationSegment.Role. This library defines the roles comment printers share — marker, label, qualifier, separator, key, value, name, type, treePrefix, text — and the kind note; a printer adds its own kinds in an extension. Neither may be empty or contain control characters.
• A region's range runs from its first atom to its last, including what a container prints between its items, less leading and trailing atoms that hold only whitespace and lie in no segment; a region of nothing else is left out of the table. Segments are never trimmed.
• The marks travel in each atom's identifier, inside a visibility tag and definition identities, so the three kinds of region nest in any order and the three separations run in any order. Separating a projection gives the regions as that projection shows them — the same as printing with those options — and a nested definition taken out with content(ofDefinitionRegion:) and removingIndentation(levels:) separates like its own print.
• One case still reads two neighbours of one kind as one: when neither has an atom that is more than spaces and is visible whenever the rest of the region is — a segment of nothing but spaces, say.

The design and its validation are in docs/AnnotationRegions.md.

MIT