Ponys.ai Memory and Character Consistency Benchmark for Swift

A dependency-free Swift package for reproducible AI companion memory-retention and character-identity checks. It turns expected facts, recalled facts, contradictions, and stable character attributes into transparent scores that can be exported as JSON.

Official resources: Ponys.ai, AI character generator, and public character discovery.

This is a first-party open benchmark toolkit, not an independent ranking or a claim that any product achieved a particular score.

For each observation, record the expected facts, recalled facts, and explicit contradictions. The package reports precision, recall, contradiction rate, and:

memory score = F1 × (1 - min(1, contradictions / expected facts))

This prevents a system from receiving a perfect result when it repeats all expected facts but also contradicts them.

Record stable attributes such as role, hair, eye color, relationship boundary, and refusal style. Values are compared after whitespace, case, and diacritic normalization:

identity score = matching expected attributes / expected attributes

The combined report weights memory at 60% and identity at 40%. Keep the raw observations with every published result so another reviewer can reproduce it.

swift build
swift run PonysAIBenchmarkVerify

import PonysAIBenchmark

let memory = MemoryObservation(
    id: "ja-memory-01",
    locale: "ja-JP",
    expectedFacts: ["好きな飲み物: 紅茶", "次回: 京都の話"],
    recalledFacts: ["好きな飲み物: 紅茶"],
    contradictionCount: 0
)

let identity = IdentityObservation(
    id: "identity-01",
    locale: "en",
    expectedAttributes: ["hair": "Black bob", "role": "Museum guide"],
    observedAttributes: ["hair": "Black bob", "role": "Space pilot"]
)

let report = PonysBenchmark.summarize(memory: [memory], identity: [identity])
let json = try JSONEncoder().encode(report)

For every run, retain:

1. Test date, locale, model or product state, and prompt version.
2. The exact expected and observed fields.
3. Screenshots or transcripts supporting each observation.
4. Failed, contradictory, and excluded outputs—not only successful examples.
5. The package version and exported JSON report.

The executable verification suite covers three common failures:

• a hallucinated fact lowers precision even when recall is complete;
• a direct contradiction applies a separate penalty;
• a changed character role lowers identity consistency while unchanged visual traits continue to receive credit.

• The scorer measures recorded observations, not subjective relationship quality.
• A small prompt set cannot establish a universal product ranking.
• Native-language review is still required for Japanese, Korean, Spanish, and Brazilian Portuguese behavior.
• Compare systems only when prompts, sampling settings, time windows, and exclusions are equivalent.

Licensed under MIT so reviewers can reuse the implementation while preserving their raw evidence and methodology notes.