feat(wiki): sync upstream v2026.10.4 core maintenance
Selectively port source hashing, locked manifest/memory writes, path-aware link lint and graph analysis, and revision-checked staging from upstream 4f457e154653756b04802d5aeb25c20f42b4770c. Record source hashes and exclusions. Preserve local archive/catalog citations, curated manifest metadata, config location, file-index and MCP behavior. Wire maintenance guidance into four skills without migrating vault contents or changing running services. Validation: 609 tests passed, including existing MCP/file-asset regressions and imported upstream maintenance tests. Read-only vault status left shared metadata hashes unchanged. Wiki MCP remains inactive; orders MCP is active.
This commit is contained in:
Codex
@@ -129,6 +129,8 @@ One-paragraph summary of what this project is.
|
||||
|
||||
## Special Files
|
||||
|
||||
For every write or maintenance operation, follow [Core Maintenance](references/CORE-MAINTENANCE.md): shared locks and atomic writes, hash-based source tracking, path-aware lint, and revision-checked staging. Existing index/hot files retain their layout until an explicit migration is authorized. This protocol overrides older manual shared-file rewrite instructions below and in other skills; the local source archive/registry protocol remains required.
|
||||
|
||||
Every wiki has these files at its root:
|
||||
|
||||
### `index.md`
|
||||
|
||||
@@ -0,0 +1,44 @@
|
||||
# Core maintenance (upstream v2026.10.4)
|
||||
|
||||
Use the repository's `obsidian-wiki` CLI (or its `.venv/bin/python -m obsidian_wiki`). Resolve the vault and read its AGENTS.md first. Explicit vault arguments avoid accidentally maintaining another vault. This protocol takes precedence over older instructions to rewrite shared metadata manually.
|
||||
|
||||
## Source tracking
|
||||
|
||||
`obsidian-wiki cache-check "$OBSIDIAN_VAULT_PATH" <source> ... --pretty` detects byte changes. After successful ingestion, `cache-update "$OBSIDIAN_VAULT_PATH" <source> --pages <page.md> ...` records the hash under a manifest lock and atomically replaces the manifest. Omitting `--pages` preserves existing associations; supplying it replaces `pages_produced`, so include every page produced for that source, including earlier still-valid pages. Both list and dictionary manifests are supported; unknown fields are preserved.
|
||||
|
||||
Keep the existing original-path key when enriching an existing entry: use `--key <existing-key>` if the source argument is now an archive copy. Do not create a second entry just to adopt upstream key conventions. Hash checks establish exact identity only, not fuzzy deduplication.
|
||||
|
||||
The helper does **not** replace the local archive/catalog protocol or update `_meta/source-registry.jsonl` and `file-assets.sqlite`. Preserve original paths, archive paths, structured identifiers, ACL fields and curated registry values. Continue the normal file-index rebuild and ingest verification.
|
||||
|
||||
For custom manifest fields, perform the entire read/merge/write under `obsidian_wiki.cache.manifest_lock(vault)` and atomically write using `obsidian_wiki.memory.atomic_write`. Read after acquiring the lock; never write a stale whole-manifest snapshot or call a helper that reacquires the same lock. This lock protects only cooperating writers.
|
||||
|
||||
## Index, log and hot cache
|
||||
|
||||
Inspect first:
|
||||
|
||||
```bash
|
||||
obsidian-wiki memory status --vault "$OBSIDIAN_VAULT_PATH" --json
|
||||
obsidian-wiki memory migrate --vault "$OBSIDIAN_VAULT_PATH" --json
|
||||
```
|
||||
|
||||
`migrate` without `--apply` is a preview. An existing hand-maintained index/hot cache is guarded: upgrading this framework does not authorize regenerating those files. Show the migration preview and obtain authorization for that data migration before running `memory migrate --apply`. Migration backs up the old files under `_archives/`.
|
||||
|
||||
For an adopted vault, finish a write with:
|
||||
|
||||
```bash
|
||||
obsidian-wiki memory sync INGEST source=example.pdf pages=2 --vault "$OBSIDIAN_VAULT_PATH" --takeaways "Concise findings from this operation."
|
||||
```
|
||||
|
||||
Use the actual verb, fields and takeaways. `sync` maintains log, index and hot under one shared lock. If another command already logged the operation, omit the verb to avoid duplicate entries. `memory index --check` previews index drift (exit 2) without writing.
|
||||
|
||||
For a legacy vault, retain its index/hot layout: use `memory log` for append-only logging, and merge necessary index/hot edits with the whole read/merge/atomic-write inside `memory.memory_lock(vault)`. Do not bypass the migration guard with `force=True`, auto-adopt, or replace existing narrative with a generated document. Do not nest locking helpers inside that lock. This compatibility path keeps normal ingest usable without requiring a migration.
|
||||
|
||||
## Lint, links and staging
|
||||
|
||||
Run `obsidian-wiki lint "$OBSIDIAN_VAULT_PATH" --json` for structural checks before semantic review. Link resolution distinguishes same-named pages in different project folders and ignores links inside code examples. Missing trust-ledger information is a legacy warning by default; do not fabricate reviews or change confidence to make checks pass.
|
||||
|
||||
Knowledge walkers exclude `_sources`, `_meta`, `_raw`, `_inbox`, `_bot_exports` and archive/staging directories. `_sources/archive` and `_sources/catalog` links remain valid source references. For nonstandard archive/catalog locations, add exclusions to `.okignore`; inspect any resulting citation warnings instead of moving originals.
|
||||
|
||||
Use `staging list --vault "$OBSIDIAN_VAULT_PATH" --json` and preserve both staged/live revisions shown during review. Promote only with `--expect-staged` plus `--expect-live` or `--expect-new`; exit 9 means the reviewed content changed. Re-read and review the changed content instead of retrying blindly. See `wiki-stage-commit` for the complete workflow.
|
||||
|
||||
These are local operator commands. They do not enable MCP tools, change remote permissions or launch services.
|
||||
@@ -15,6 +15,8 @@ You are ingesting source documents into an Obsidian wiki. Your job is not to sum
|
||||
|
||||
## Before You Start
|
||||
|
||||
Apply `llm-wiki/references/CORE-MAINTENANCE.md` for source hash checks and all shared metadata writes. Use the locked cache helper for standard manifest updates and a locked read/merge/atomic-write for additional local fields. Keep all archive, source-registry, file-index, ACL and deterministic ingest-verification steps below. The helper does not perform those steps for you.
|
||||
|
||||
1. **Resolve config** — follow the Config Resolution Protocol in `llm-wiki/SKILL.md` (walk up CWD for `.env` → `~/.obsidian-wiki/config` → prompt setup). This gives `OBSIDIAN_VAULT_PATH`, `OBSIDIAN_SOURCES_DIR`, `OBSIDIAN_LINK_FORMAT` (default: `wikilink`), `WIKI_STAGED_WRITES`, `WIKI_ARCHIVE_SOURCES`, `OBSIDIAN_SOURCE_ARCHIVE_DIR`, `WIKI_SOURCE_CATALOG`, `OBSIDIAN_SOURCE_CATALOG_DIR`, and optionally `OBSIDIAN_FILE_ASSET_DB`. Only read the specific variables you need — do not log, echo, or reference any other values from these files.
|
||||
2. **Check `WIKI_STAGED_WRITES`** — if set to `true`, all new and updated category pages go to `_staging/<category>/` instead of their final location. Tell the user at the start of the ingest: "Staged writes mode is enabled — pages will land in `_staging/` for your review. Run `/wiki-stage-commit` when ready to promote."
|
||||
3. Read `.manifest.json` at the vault root to check what's already been ingested
|
||||
|
||||
@@ -24,6 +24,8 @@ You are performing a health check on an Obsidian wiki. Your goal is to find and
|
||||
|
||||
## Lint Checks
|
||||
|
||||
First run `obsidian-wiki lint "$OBSIDIAN_VAULT_PATH" --json` using this repository's CLI. Use its findings for structural checks below, then inspect only implicated pages and perform semantic contradiction review. It resolves project-relative and duplicate-name links, ignores code examples, and excludes source archives from knowledge pages. Trust-ledger gaps are warnings in legacy mode; do not invent reviews or change scores to silence them. Follow `llm-wiki/references/CORE-MAINTENANCE.md` for any authorized repairs and tracking-file updates. If the CLI is unavailable, use the manual checks below with the same source-directory exclusions.
|
||||
|
||||
Run these checks in order. Report findings as you go.
|
||||
|
||||
### 1. Orphaned Pages
|
||||
@@ -529,4 +531,4 @@ Record one of:
|
||||
- `QMD refreshed: update only + verified`
|
||||
- `QMD skipped: QMD_WIKI_COLLECTION unset`
|
||||
- `QMD skipped: qmd CLI unavailable`
|
||||
- `QMD failed: <short error summary>`
|
||||
- `QMD failed: <short error summary>`
|
||||
|
||||
@@ -1,11 +1,7 @@
|
||||
---
|
||||
name: wiki-stage-commit
|
||||
description: >
|
||||
Review and promote staged wiki pages to their final locations. Use when WIKI_STAGED_WRITES=true
|
||||
and the user says "/wiki-stage-commit", "review staged pages", "commit staged writes",
|
||||
"promote staged pages", "approve staged changes", or "what's waiting in staging".
|
||||
Shows each staged file, lets the user accept or reject it, and moves accepted files to
|
||||
their final wiki locations. Rejected files are moved back to _raw/ for manual editing.
|
||||
Review staged wiki writes and promote accepted pages to final locations when WIKI_STAGED_WRITES is enabled. Use for staging approval/commit; rejected pages return to _raw/ for editing.
|
||||
---
|
||||
|
||||
# Wiki Stage Commit — Staged Write Promotion
|
||||
@@ -29,9 +25,24 @@ You are reviewing LLM-written pages that are waiting in `_staging/` for human ap
|
||||
|
||||
## Step 1: Inventory Staged Files
|
||||
|
||||
Glob `$OBSIDIAN_VAULT_PATH/_staging/**/*.md` — these are the pending pages.
|
||||
The CLI owns the inventory — don't glob `_staging/` yourself:
|
||||
|
||||
Also glob `$OBSIDIAN_VAULT_PATH/_staging/**/*.patch.md` — these are pending *updates* to existing pages (diff-style files showing proposed additions and deletions).
|
||||
```bash
|
||||
obsidian-wiki staging list --vault "$OBSIDIAN_VAULT_PATH" --json
|
||||
```
|
||||
|
||||
Each entry carries:
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| `staged_path` | vault-relative path of the staged file |
|
||||
| `live_path` | where it will land (a `.patch.md` targets the page it is named after) |
|
||||
| `kind` | `new`, `update`, or `patch` |
|
||||
| `staged_revision` | content hash of the staged file, as you are seeing it now |
|
||||
| `live_revision` | content hash of the live page, or `null` if there is none yet |
|
||||
| `staged_mtime` | when it was staged |
|
||||
|
||||
**Keep `staged_revision` and `live_revision` for every file you show the user.** They are what makes Step 3 refuse to overwrite an agent's concurrent write instead of silently clobbering it.
|
||||
|
||||
Report the inventory:
|
||||
|
||||
@@ -39,17 +50,17 @@ Report the inventory:
|
||||
Staged files: 4 new pages, 2 updates
|
||||
|
||||
New pages:
|
||||
_staging/concepts/attention-mechanism.md (ingested 2 days ago)
|
||||
_staging/entities/andrej-karpathy.md (ingested 2 days ago)
|
||||
_staging/skills/fine-tuning-llms.md (ingested yesterday)
|
||||
_staging/references/attention-is-all-you-need.md (ingested 3 hours ago)
|
||||
_staging/concepts/attention-mechanism.md (staged 2026-09-08)
|
||||
_staging/entities/andrej-karpathy.md (staged 2026-09-08)
|
||||
|
||||
Updates (patch files):
|
||||
_staging/concepts/transformer-architecture.patch.md (target: concepts/transformer-architecture.md)
|
||||
_staging/skills/prompt-engineering.patch.md (target: skills/prompt-engineering.md)
|
||||
Updates:
|
||||
_staging/concepts/transformer-architecture.md (target: concepts/transformer-architecture.md)
|
||||
|
||||
Patches:
|
||||
_staging/skills/prompt-engineering.patch.md (target: skills/prompt-engineering.md)
|
||||
```
|
||||
|
||||
If `_staging/` is empty, report: "Nothing staged. All writes have been committed or no staged writes have been produced yet."
|
||||
If the list is empty, report: "Nothing staged. All writes have been committed or no staged writes have been produced yet."
|
||||
|
||||
## Step 2: Per-File Review (interactive mode)
|
||||
|
||||
@@ -89,7 +100,7 @@ Proposed additions (+):
|
||||
Proposed deletions (-):
|
||||
- The attention mechanism was first described in [Bahdanau 2015]. (to be replaced by updated claim)
|
||||
|
||||
⚠️ Conflict check: target page was modified 3 days after staging. Review carefully.
|
||||
⚠️ Conflict check: live_revision no longer matches what was staged against. Review carefully.
|
||||
|
||||
Accept [a], Reject [r], Skip [s], Preview full diff [p]?
|
||||
```
|
||||
@@ -100,42 +111,56 @@ If `--list` flag is set, stop after printing the inventory (Step 1).
|
||||
|
||||
## Step 3: Apply Decisions
|
||||
|
||||
The moves are mechanical and the CLI does them atomically. Pass the revisions from Step 1 so a decision made on stale information fails loudly.
|
||||
|
||||
### Accepting a new page
|
||||
|
||||
1. Move `_staging/<category>/page.md` → `<category>/page.md` (the final location)
|
||||
2. Update `index.md` with the new page entry
|
||||
3. Remove the staged file
|
||||
```bash
|
||||
obsidian-wiki staging promote <staged_path> --vault "$OBSIDIAN_VAULT_PATH" \
|
||||
--expect-staged <staged_revision> --expect-new
|
||||
```
|
||||
|
||||
### Accepting a patch/update
|
||||
`--expect-new` refuses if a live page has appeared since you listed — someone else got there first.
|
||||
|
||||
1. Read the current page at the target path
|
||||
2. Apply the proposed additions and deletions (merge, don't just overwrite)
|
||||
3. Update the `updated` frontmatter timestamp
|
||||
4. Update `index.md` if the summary changed
|
||||
5. Remove the staged patch file
|
||||
### Accepting an update
|
||||
|
||||
```bash
|
||||
obsidian-wiki staging promote <staged_path> --vault "$OBSIDIAN_VAULT_PATH" \
|
||||
--expect-staged <staged_revision> --expect-live <live_revision>
|
||||
```
|
||||
|
||||
### Rejecting a file
|
||||
|
||||
Move it to `$OBSIDIAN_VAULT_PATH/_raw/` for manual editing:
|
||||
- `_staging/concepts/page.md` → `_raw/rejected-concepts-page.md`
|
||||
- `_staging/concepts/page.patch.md` → `_raw/rejected-patch-concepts-page.md`
|
||||
- Prefix with `rejected-` so the user can identify it
|
||||
```bash
|
||||
obsidian-wiki staging discard <staged_path> --vault "$OBSIDIAN_VAULT_PATH"
|
||||
```
|
||||
|
||||
### Conflict detection on patch accept
|
||||
It lands in `_raw/rejected-<category>-<page>.md` for manual editing. An earlier rejection of the same page is never overwritten — the second becomes `-2`.
|
||||
|
||||
Before applying a patch, check whether the target page's `updated` frontmatter is newer than the patch file's own `updated` field:
|
||||
- If the target was modified AFTER the patch was staged, warn: `⚠️ Conflict: target was updated since this patch was staged. Applying may lose recent changes.`
|
||||
- Give the user a chance to abort: `Apply anyway [y], Skip [s], Reject [r]?`
|
||||
### Accepting a patch
|
||||
|
||||
`promote` refuses `.patch.md` files, because merging a human-readable diff into a page whose surrounding text may have moved is judgment, not a rename. Do it yourself:
|
||||
|
||||
1. Read the target and patch; retain the live revision from the inventory.
|
||||
2. Merge the proposed changes into a complete staged `.md` page, preserving unrelated live content. Bump its `updated` frontmatter. Do not modify the live file yet or overwrite an unrelated queued page.
|
||||
3. Hash the merged staged page and promote it with `--expect-staged <merged-revision>` and the **original** `--expect-live <reviewed-live-revision>` (or `--expect-new`). If either changed, review again.
|
||||
4. After successful promotion, discard the reviewed patch to clear it from the queue. Verify it still matches the reviewed patch before discarding; leave a concurrently changed patch queued.
|
||||
|
||||
### When the CLI reports a conflict
|
||||
|
||||
Exit code **9** with `conflict: ...` on stderr means the staged or live file changed after you listed it. Nothing was moved. Do not retry with fresh revisions blindly — re-run `staging list`, show the user what changed, and ask again. The whole point of the check is that the content they approved is no longer the content that would land.
|
||||
|
||||
Promotion itself does not reconcile `index.md`; do that in Step 4 after pages become live. `log.md` gets one `STAGE_COMMIT` line per invocation, written by the CLI.
|
||||
|
||||
## Step 4: Update Tracking Files
|
||||
|
||||
After processing all staged files:
|
||||
The CLI appends the `STAGE_COMMIT` line to `log.md` itself. Follow `llm-wiki/references/CORE-MAINTENANCE.md`. For an adopted vault, reconcile the index and hot cache after processing all staged files:
|
||||
|
||||
1. **`hot.md`** — update the Recent Activity section: "Committed N staged pages; rejected M."
|
||||
2. **`log.md`** — append:
|
||||
```
|
||||
- [TIMESTAMP] STAGE_COMMIT accepted=N rejected=M skipped=K
|
||||
```
|
||||
```bash
|
||||
obsidian-wiki memory sync --vault "$OBSIDIAN_VAULT_PATH" --takeaways "Committed N staged pages; rejected M."
|
||||
```
|
||||
|
||||
(No `--verb`: the log line was already written by `staging promote`.) The command takes the shared lock. For a legacy vault, preserve the existing index/hot layout with a locked read/merge/atomic-write as documented in Core Maintenance. Do not auto-migrate to complete a promotion.
|
||||
|
||||
## Step 5: Report
|
||||
|
||||
@@ -159,6 +184,7 @@ Staging queue: K files remaining
|
||||
## Notes
|
||||
|
||||
- Staged files use the same page template as live pages — they are ready to land, just awaiting approval
|
||||
- Patch files use a human-readable diff format: lines starting with `+` are additions, lines starting with `-` are deletions
|
||||
- Patch files use a human-readable diff format: lines starting with `+` are additions, lines starting with `-` are deletions. `staging promote` refuses them — merge them yourself (Step 3)
|
||||
- `index.md` and `log.md` are always updated immediately on ingest (they are low-risk tracking files) — only category pages go through staging
|
||||
- The moves live in `obsidian_wiki/staging.py`; this skill supplies the judgment, not the file handling
|
||||
- The `_staging/` directory is not tracked by Obsidian's graph view — pages only appear in the wiki after promotion
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Obsidian Wiki — Agent Context
|
||||
|
||||
A **skill-based framework** for building and maintaining an Obsidian knowledge base. No scripts or dependencies — everything is markdown instructions that you execute directly.
|
||||
A **skill-based framework** for building and maintaining an Obsidian knowledge base. Markdown skills guide the work; standard-library CLI helpers handle shared metadata, links and staging deterministically. MCP support is optional.
|
||||
|
||||
## Configuration
|
||||
|
||||
@@ -137,6 +137,8 @@ See `wiki-query` and `wiki-export` skills for how the filter is applied.
|
||||
|
||||
## Core Principles
|
||||
|
||||
Shared-file maintenance follows `.skills/llm-wiki/references/CORE-MAINTENANCE.md`: use locked cache/memory helpers, path-aware lint and revision-checked staging. Preserve local archive/registry fields. Existing index/hot layouts require explicit migration authorization before generated replacement; legacy edits must still use the shared lock and atomic writes.
|
||||
|
||||
- **Compile, don't retrieve.** The wiki is pre-compiled knowledge. Update existing pages — don't append or duplicate.
|
||||
- **Track everything.** Update `.manifest.json` after ingesting, `index.md`, `log.md`, and `hot.md` after any write operation.
|
||||
- **Connect with `[[wikilinks]]`.** Every page should link to related pages. This is what makes it a knowledge graph, not a folder of files.
|
||||
|
||||
@@ -15,10 +15,12 @@ A knowledge mgmt system inspired by [gist](https://gist.github.com/karpathy/442a
|
||||
|
||||
Instead of asking an LLM the same questions over over (or doing RAG every time), you compile knowledge once into interconnected markdown files and keep them current. In this case Obsidian is the viewer and the LLM is the maintainer.
|
||||
|
||||
We took that and built a framework around it. The whole thing is a set of markdown skill files that any AI coding agent (Claude Code, Cursor, Windsurf, Pi, whatever you use) can read and execute. You point it at your Obsidian vault and tell it what to do.
|
||||
We took that and built a framework around it. Markdown skill files guide AI coding agents (Claude Code, Cursor, Windsurf, Pi, and others), with CLI helpers for deterministic maintenance. You point it at your Obsidian vault and tell it what to do.
|
||||
|
||||
## Quick Start
|
||||
|
||||
This fork selectively includes the core maintenance updates from upstream **v2026.10.4**: hash-based source tracking, locked memory writes, path-aware lint/graph analysis, and revision-checked staging. See [the sync scope and migration notes](docs/upstream-sync.md). Existing source-file lookup and MCP deployment conventions remain local; upgrading the code does not migrate vault contents or enable services.
|
||||
|
||||
### Install via pip (recommended)
|
||||
|
||||
```bash
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
{
|
||||
"upstream": "https://github.com/Ar9av/obsidian-wiki",
|
||||
"release": "v2026.10.4",
|
||||
"commit": "4f457e154653756b04802d5aeb25c20f42b4770c",
|
||||
"scope": "core-maintenance",
|
||||
"source_sha256": {
|
||||
"obsidian_wiki/vault.py": "604e65a01eb44438eea402f00f6c6c534e06856c20fe6c6b5ceb47e69291ee4a",
|
||||
"obsidian_wiki/cache.py": "f410ac542ce47719b5c8aa687bcd8beafd7d70a6f7db3cd2df24c9030628a9f6",
|
||||
"obsidian_wiki/clock.py": "456749a083b29a51820332970d9f07ee49bfaec6314cc51449672c8c96a53dce",
|
||||
"obsidian_wiki/memory.py": "81fc4e28b8cdd761d6be23d0de6e6b35dad49501ebb0ec9e3411f18bf23a3170",
|
||||
"obsidian_wiki/links.py": "79c2ebb69b9ded9823bffebf270b40b520e640e740b57a0f7909deb66cc5e980",
|
||||
"obsidian_wiki/lint.py": "482a69b5a7ef15c45d5054efec31c68f23e7906f62fcf9dd640987deac82c19e",
|
||||
"obsidian_wiki/graph_analysis.py": "a0932567affc219b7a7692036b389ec9674b8b9e1c94f72ce27fcc69078cb25c",
|
||||
"obsidian_wiki/provenance.py": "ce39dca4a9069814ee0b3dd779ebdcf32942d510048b44ed7b71fa9521734b9b",
|
||||
"obsidian_wiki/temporal.py": "9e31cdd08b1dabbc6d94590db7b5b2295923d8f33e299d9179d84820955c3395",
|
||||
"obsidian_wiki/trust.py": "c397465fcdcb2df31c0ea64ba65ef411532161b174f170be2d9662d8e75745cd",
|
||||
"obsidian_wiki/staging.py": "626d6f059ec089ca23981d1e9c3d9ba90266c74292ee313e654ac713eb38dbb7"
|
||||
},
|
||||
"cli_functions": [
|
||||
"SchemaOptions",
|
||||
"_memory_fields",
|
||||
"_memory_takeaways",
|
||||
"_note_source_ambiguity",
|
||||
"_print_lint",
|
||||
"_read_config_file",
|
||||
"_report_empty_local_vault",
|
||||
"_resolve_schema_command_context",
|
||||
"_resolve_source_arg",
|
||||
"_schema_cli_values",
|
||||
"_schema_csv",
|
||||
"_schema_options",
|
||||
"_schema_source_value",
|
||||
"_source_candidates",
|
||||
"_timezone_report",
|
||||
"_verb_and_fields",
|
||||
"cmd_cache_check",
|
||||
"cmd_cache_hash",
|
||||
"cmd_cache_update",
|
||||
"cmd_graph_analyse",
|
||||
"cmd_lint",
|
||||
"cmd_memory",
|
||||
"cmd_projects_check",
|
||||
"cmd_staging"
|
||||
],
|
||||
"commands": [
|
||||
"graph-analyse",
|
||||
"cache-check",
|
||||
"projects-check",
|
||||
"cache-update",
|
||||
"cache-hash",
|
||||
"lint",
|
||||
"memory",
|
||||
"staging"
|
||||
],
|
||||
"excluded": "upstream server, setup/hooks, session brain, retrieval platform and bulk vault migrations",
|
||||
"test_exclusions_outside_scope": {
|
||||
"clock": [
|
||||
"TestDocs",
|
||||
"test_scaffold_writes_the_key_and_stamps_in_the_zone",
|
||||
"test_setup_cli_records_the_zone",
|
||||
"test_setup_cli_refuses_a_bad_zone_before_touching_anything"
|
||||
],
|
||||
"memory": [
|
||||
"test_a_scaffolded_vault_is_born_migrated",
|
||||
"test_a_scaffolded_vault_carries_the_adoption_record"
|
||||
],
|
||||
"links": [
|
||||
"obsidian_wiki.graphrag",
|
||||
"test_graph_query_index_matches_the_graph"
|
||||
],
|
||||
"lint": [
|
||||
"test_invalid_configured_required_trust_field_fails_closed_for_all_cli_paths",
|
||||
"test_empty_configured_schema_values_fail_closed_for_all_cli_paths",
|
||||
"test_blank_config_schema_source_cannot_be_masked_by_valid_cli_source",
|
||||
"test_distributed_schema_config_contract_names_all_four_variables",
|
||||
"test_correction_contract_requires_temporal_authority_and_immutable_hash_check"
|
||||
],
|
||||
"graph_analysis": [
|
||||
"TestGitignoreBackfill"
|
||||
],
|
||||
"trust": [
|
||||
"test_owner_lifecycle_cli_record_then_check",
|
||||
"test_invalid_review_timestamp_is_rejected_by_record_cli",
|
||||
"test_trust_record_cli_requires_explicit_approval",
|
||||
"test_trust_record_and_check_cli_round_trip",
|
||||
"test_owner_schema_cli_excludes_then_removes_no_confidence_ledger_entry",
|
||||
"test_trust_record_all_human_output_lists_no_confidence_without_removal",
|
||||
"test_trust_record_all_human_output_warns_when_rebuild_removes_stale_entry",
|
||||
"test_trust_record_page_human_output_warns_when_stale_entry_is_removed",
|
||||
"test_trust_record_page_human_output_reports_zero_removals_without_warning",
|
||||
"test_explicit_vault_cli_override_reports_cli_schema_source",
|
||||
"test_partial_trust_record_does_not_approve_other_stale_pages",
|
||||
"test_partial_trust_record_reports_malformed_ledger_without_traceback"
|
||||
],
|
||||
"temporal": [
|
||||
"obsidian_wiki.graphrag",
|
||||
"test_expired_page_leaves_retrieval_but_stays_in_the_graph",
|
||||
"test_query_as_of_returns_the_page_that_was_true_then",
|
||||
"test_include_historical_ranks_both",
|
||||
"test_temporal_keys_are_absent_on_pages_that_do_not_opt_in",
|
||||
"test_structural_intents_still_see_historical_pages",
|
||||
"test_graph_query_cli_accepts_as_of",
|
||||
"test_cli_rejects_a_malformed_as_of"
|
||||
]
|
||||
},
|
||||
"local_adaptations": [
|
||||
"Keep existing ~/.obsidian-wiki/config location and adapt the memory CLI fixture accordingly",
|
||||
"Exclude local source, metadata, inbox and bot-export directories from knowledge pages",
|
||||
"Resolve _sources/archive and _sources/catalog citations without changing file extensions",
|
||||
"Preserve pages_produced when cache-update omits --pages",
|
||||
"Keep local CLI, MCP dependency range, service gating and curated source-registry metadata"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
# 上游关键更新同步
|
||||
|
||||
本次选择性同步 [Ar9av/obsidian-wiki v2026.10.4](https://github.com/Ar9av/obsidian-wiki/releases/tag/v2026.10.4)，固定提交 `4f457e154653756b04802d5aeb25c20f42b4770c`。这是核心维护能力的移植，不表示本分支完整等同于该版本。来源哈希、移植的 CLI 函数和未纳入的测试范围记录在 `upstream-sync.json`。
|
||||
|
||||
| 纳入能力 | 实际作用 |
|
||||
| --- | --- |
|
||||
| `cache-check` / `cache-update` / `cache-hash` | 用内容哈希识别变化；manifest 锁和原子写入；兼容列表/字典及旧哈希格式 |
|
||||
| `projects-check` | 按已记录提交检查项目资料是否落后 |
|
||||
| `memory` | 统一维护 index、log、hot；提供只读检查、显式迁移及备份 |
|
||||
| `lint` / `graph-analyse` | 正确区分不同项目中的同名页，解析相对链接，忽略代码示例里的伪链接；检查结构、时效与来源信息 |
|
||||
| `staging` | 暂存晋升前校验已审阅的 staged/live 内容哈希；冲突时退出并保留文件 |
|
||||
|
||||
同步了 `llm-wiki`、`wiki-ingest`、`wiki-lint`、`wiki-stage-commit` 的相关执行指引，详细协议见 `.skills/llm-wiki/references/CORE-MAINTENANCE.md`。图分析是显式调用的维护工具，不接入远程问答或自动调度。
|
||||
|
||||
## 本地兼容
|
||||
|
||||
- 保留 `~/.obsidian-wiki/config`、现有 setup、file-index、bot-view 和 MCP 命令。
|
||||
- 保留 `_sources/archive` / `_sources/catalog`、原文件索引、结构化产品字段及 ACL。知识页遍历排除原始资料、元数据、收件箱和导出副本；归档文件仍可被引用。
|
||||
- `cache-update` 未传 `--pages` 时保留已有页面关联；显式传入时替换 `pages_produced`。自定义元数据需在同一个 manifest 锁内读取、合并并原子写回。registry 和 SQLite 索引仍由本地 ingest 流程维护。
|
||||
- 未改动已有知识页面、manifest、registry、SQLite 或 index/hot；未启动 Wiki MCP，未修改独立订单项目。
|
||||
- 未导入上游 HTTP 服务、MCP 2.x 依赖、会话 hooks、session brain、GraphRAG 查询链路或全量数据迁移。
|
||||
|
||||
## 使用与迁移
|
||||
|
||||
从本项目虚拟环境执行，避免误用系统中另一个版本：
|
||||
|
||||
```bash
|
||||
.venv/bin/python -m obsidian_wiki lint /path/to/vault --json
|
||||
.venv/bin/python -m obsidian_wiki memory status --vault /path/to/vault --json
|
||||
.venv/bin/python -m obsidian_wiki memory migrate --vault /path/to/vault --json
|
||||
.venv/bin/python -m obsidian_wiki staging list --vault /path/to/vault --json
|
||||
```
|
||||
|
||||
这些检查不重写知识库。`memory migrate` 默认只预览；只有明确决定迁移数据格式后才加 `--apply`，届时先备份旧 index/hot。未迁移的库继续保留现有布局，使用共享锁合并必要修改。框架升级本身不触发该迁移。
|
||||
|
||||
lint 的旧库模式把 trust ledger 缺失等问题报告为警告。若以后启用严格审阅，需要独立设计实际审阅流程，不能为了消除警告伪造置信度。自定义归档目录可加入 `.okignore`，其非标准引用需单独核查。
|
||||
|
||||
## 验证与下次同步
|
||||
|
||||
```bash
|
||||
uv sync --extra mcp --group dev
|
||||
PYTHONPATH="$PWD" .venv/bin/python -m pytest tests -q
|
||||
```
|
||||
|
||||
测试包括原有文件检索、ACL、MCP query/ingest 和运维回归，以及本次核心模块的上游测试；另有本地兼容测试验证归档排除、引用解析与 manifest 字段保留。测试使用临时知识库，不启动实际服务。
|
||||
|
||||
本次验证结果：**609 项测试通过**。实际知识库只执行了 `memory status` 只读检查，index/hot/log、manifest、registry、SQLite 的前后 SHA-256 一致。Wiki MCP 保持 inactive，独立订单 MCP 保持 active。
|
||||
|
||||
下次以 `upstream-sync.json` 的提交和源文件哈希作比较基线，逐模块检查新提交，再重放列出的本地适配。不要直接覆盖 `.skills`、运行上游 setup、切换服务依赖或批量改写知识库。上游未纳入模块的测试明确列入排除清单，不能把它们的缺席解释为整版上游已通过验证。
|
||||
@@ -0,0 +1,765 @@
|
||||
"""Content-hash cache for wiki-ingest source tracking.
|
||||
|
||||
Provides a reliable, platform-independent alternative to running `sha256sum`
|
||||
in the skill. The agent calls `obsidian-wiki cache-check` / `cache-update`
|
||||
instead of shelling out to sha256sum and manually parsing .manifest.json.
|
||||
|
||||
Manifest format (.manifest.json in the vault root). Two `sources` shapes are
|
||||
supported, because real vaults contain both:
|
||||
|
||||
1. Dict keyed by path (this module's original format)::
|
||||
|
||||
{"sources": {"<abs-or-rel-path>": {"content_hash": "...", ...}}}
|
||||
|
||||
2. List of entry objects, each carrying its own ``path`` (the shape the
|
||||
wiki-ingest skill writes)::
|
||||
|
||||
{"sources": [{"path": "_raw/foo.md", "content_hash": "sha256:...", ...}]}
|
||||
|
||||
Both shapes are read transparently, and `update_source` edits the manifest
|
||||
*in place* — preserving its shape, any duplicate-path entries, and
|
||||
skill-written fields (``pages_created``, ``size_bytes``, ``source_type``, …).
|
||||
|
||||
Hashes are compared prefix-insensitively: the skill records
|
||||
``"sha256:<hex>"`` while this module computes a bare ``<hex>``, so an optional
|
||||
``algo:`` prefix is stripped from both sides before comparison.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from contextlib import contextmanager
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Iterator, TypedDict
|
||||
|
||||
|
||||
class SourceEntry(TypedDict, total=False):
|
||||
content_hash: str
|
||||
last_ingested: str
|
||||
pages_produced: list[str]
|
||||
|
||||
|
||||
class CheckResult(TypedDict):
|
||||
new: list[str]
|
||||
modified: list[str]
|
||||
unchanged: list[str]
|
||||
missing: list[str] # vault-local source in manifest but no longer on disk
|
||||
unavailable: list[str] # machine-local source absent here (e.g. synced from another host)
|
||||
|
||||
|
||||
def _manifest_path(vault: Path) -> Path:
|
||||
return vault / ".manifest.json"
|
||||
|
||||
|
||||
def _load_raw(vault: Path) -> dict:
|
||||
"""Return the full manifest object (``{}`` if absent/unreadable)."""
|
||||
mp = _manifest_path(vault)
|
||||
if not mp.exists():
|
||||
return {}
|
||||
try:
|
||||
data = json.loads(mp.read_text(encoding="utf-8"))
|
||||
return data if isinstance(data, dict) else {}
|
||||
except (json.JSONDecodeError, OSError):
|
||||
return {}
|
||||
|
||||
|
||||
def _load_manifest(vault: Path):
|
||||
"""Return the raw ``sources`` value — a dict, a list, or ``{}`` if absent.
|
||||
|
||||
Kept for backward compatibility; callers that need shape-agnostic access
|
||||
should use :func:`_iter_entries`.
|
||||
"""
|
||||
return _load_raw(vault).get("sources", {})
|
||||
|
||||
|
||||
def _lock_path(vault: Path) -> Path:
|
||||
return vault / ".manifest.lock"
|
||||
|
||||
|
||||
class AdvisoryLockTimeout(RuntimeError):
|
||||
"""Raised when an advisory lockfile could not be claimed within the timeout."""
|
||||
|
||||
|
||||
class ManifestLockTimeout(AdvisoryLockTimeout):
|
||||
"""Raised when the manifest lock could not be acquired within the timeout."""
|
||||
|
||||
|
||||
@contextmanager
|
||||
def advisory_lock(
|
||||
lock: Path,
|
||||
*,
|
||||
timeout: float = 10.0,
|
||||
stale_after: float = 60.0,
|
||||
error_cls: type = AdvisoryLockTimeout,
|
||||
):
|
||||
"""Claim *lock* as an advisory lockfile for a read-modify-write.
|
||||
|
||||
``O_CREAT | O_EXCL`` is the portable stdlib primitive here — ``fcntl`` is
|
||||
POSIX-only and this ships on Windows. A lockfile older than *stale_after*
|
||||
is assumed to belong to a crashed process and is stolen.
|
||||
|
||||
Shared by the manifest writer and the prose-memory writer
|
||||
(:mod:`obsidian_wiki.memory`) so both serialise the same way.
|
||||
"""
|
||||
deadline = time.monotonic() + timeout
|
||||
fd = None
|
||||
while True:
|
||||
try:
|
||||
fd = os.open(lock, os.O_CREAT | os.O_EXCL | os.O_WRONLY)
|
||||
break
|
||||
except FileExistsError:
|
||||
try:
|
||||
age = time.time() - lock.stat().st_mtime
|
||||
except FileNotFoundError:
|
||||
continue # holder released it between our open and stat
|
||||
if age > stale_after:
|
||||
# ponytail: last-writer-wins steal; a pid check would narrow the
|
||||
# window if two processes ever race to steal the same stale lock
|
||||
try:
|
||||
lock.unlink()
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
continue
|
||||
if time.monotonic() >= deadline:
|
||||
raise error_cls(
|
||||
f"could not acquire {lock} within {timeout}s "
|
||||
f"(held for {age:.1f}s; stale after {stale_after}s)"
|
||||
)
|
||||
time.sleep(0.1)
|
||||
try:
|
||||
os.write(fd, str(os.getpid()).encode())
|
||||
os.close(fd)
|
||||
fd = None
|
||||
yield
|
||||
finally:
|
||||
if fd is not None:
|
||||
os.close(fd)
|
||||
try:
|
||||
lock.unlink()
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
|
||||
@contextmanager
|
||||
def manifest_lock(vault: Path, *, timeout: float = 10.0, stale_after: float = 60.0):
|
||||
"""Advisory lock around a manifest read-modify-write.
|
||||
|
||||
Parallel ingest agents (``batch-plan`` fan-out) and the Docker server can
|
||||
all write one vault's manifest; without this the read-modify-write races
|
||||
and a whole entry is silently lost.
|
||||
"""
|
||||
# ponytail: whole-manifest advisory lock; per-entry locking if batch fan-out ever contends
|
||||
with advisory_lock(
|
||||
_lock_path(vault),
|
||||
timeout=timeout,
|
||||
stale_after=stale_after,
|
||||
error_cls=ManifestLockTimeout,
|
||||
):
|
||||
yield
|
||||
|
||||
|
||||
def _write_manifest(vault: Path, manifest: dict) -> None:
|
||||
"""Atomically replace the manifest file — no torn reads, no partial file."""
|
||||
target = _manifest_path(vault)
|
||||
tmp = target.with_name(f".manifest.json.{os.getpid()}.tmp")
|
||||
try:
|
||||
tmp.write_text(
|
||||
json.dumps(manifest, indent=2, ensure_ascii=False), encoding="utf-8"
|
||||
)
|
||||
os.replace(tmp, target) # atomic on POSIX and Windows
|
||||
except BaseException:
|
||||
try:
|
||||
tmp.unlink()
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
raise
|
||||
|
||||
|
||||
def _save_manifest(vault: Path, sources) -> None:
|
||||
"""Write *sources* back into the manifest, preserving other top-level keys."""
|
||||
with manifest_lock(vault):
|
||||
manifest = _load_raw(vault)
|
||||
manifest["sources"] = sources
|
||||
_write_manifest(vault, manifest)
|
||||
|
||||
|
||||
def _iter_entries(sources) -> Iterator[tuple[str | None, dict]]:
|
||||
"""Yield ``(stored_key, entry)`` pairs for either manifest shape.
|
||||
|
||||
For the dict shape the key is the dict key; for the list shape it is the
|
||||
entry's ``path`` (falling back to ``source_id``).
|
||||
"""
|
||||
if isinstance(sources, dict):
|
||||
for key, entry in sources.items():
|
||||
if isinstance(entry, dict):
|
||||
yield key, entry
|
||||
elif isinstance(sources, list):
|
||||
for entry in sources:
|
||||
if isinstance(entry, dict):
|
||||
yield (entry.get("path") or entry.get("source_id")), entry
|
||||
|
||||
|
||||
# Matches "sha256:", "https://", "slack:#..." — anything with an algo/scheme
|
||||
# prefix that shouldn't be treated as a filesystem path.
|
||||
_SCHEME_RE = re.compile(r"^[A-Za-z][A-Za-z0-9+.\-]*:[^\\/]")
|
||||
|
||||
# A Windows drive-absolute path ("C:\dir\file", "C:/dir/file"). On POSIX this is
|
||||
# not a path at all, so it cannot be resolved and is machine-specific; lint.py
|
||||
# recognises the same shape when reporting stored keys. Kept local rather than
|
||||
# imported so the runtime cache carries no dependency on the lint module.
|
||||
_WINDOWS_ABS_RE = re.compile(r"^[A-Za-z]:[\\/]")
|
||||
|
||||
|
||||
def _strip_algo(value: str | None) -> str:
|
||||
"""Drop an optional ``algo:`` prefix so ``sha256:<hex>`` == ``<hex>``."""
|
||||
value = value or ""
|
||||
return value.split(":", 1)[1] if ":" in value else value
|
||||
|
||||
|
||||
def _format_hash(existing: str | None, new_hex: str) -> str:
|
||||
"""Format *new_hex* keeping any ``algo:`` prefix the existing value used."""
|
||||
if existing and ":" in existing:
|
||||
return f"{existing.split(':', 1)[0]}:{new_hex}"
|
||||
return new_hex
|
||||
|
||||
|
||||
def _is_file_key(key: str | None) -> bool:
|
||||
"""True if *key* looks like a filesystem path rather than a URL/pseudo-key."""
|
||||
return bool(key) and "://" not in key and not _SCHEME_RE.match(key)
|
||||
|
||||
|
||||
def _expand_key(key: str) -> str:
|
||||
"""Expand ``~`` and environment variables, but only when the key uses them.
|
||||
|
||||
A plain vault-relative key like ``Raw/x.pdf`` must survive untouched, so the
|
||||
expansion is gated rather than applied unconditionally.
|
||||
"""
|
||||
if key.startswith("~") or "$" in key:
|
||||
return os.path.expandvars(os.path.expanduser(key))
|
||||
return key
|
||||
|
||||
|
||||
def resolve_key(key: str | None, vault: Path) -> Path | None:
|
||||
"""Normalize a manifest key to an absolute path, or ``None`` for non-file keys.
|
||||
|
||||
Resolution order (the source key contract v2, see ``llm-wiki/SKILL.md``):
|
||||
|
||||
1. pseudo-key (``scheme:`` / ``://``) -> ``None`` — an opaque identifier, not
|
||||
a filesystem location;
|
||||
2. ``~``- or env-var-bearing key -> expand;
|
||||
3. absolute path -> use as-is (legacy compatibility);
|
||||
4. anything else -> resolve against the vault root.
|
||||
"""
|
||||
if not _is_file_key(key):
|
||||
return None
|
||||
path = Path(_expand_key(key))
|
||||
return path if path.is_absolute() else (vault / path)
|
||||
|
||||
|
||||
def stored_key(path: Path, vault: Path) -> str | None:
|
||||
"""Return the portable manifest key for *path*, or ``None`` if not portable.
|
||||
|
||||
In-vault sources become vault-relative (``Raw/x.pdf``); sources under
|
||||
``$HOME`` become home-relative (``~/.claude/...``); anything else has no
|
||||
machine-portable representation and needs an explicit pseudo-key from the
|
||||
caller (``repo:``, ``url:``, ``agent:``).
|
||||
"""
|
||||
p = Path(os.path.abspath(_expand_key(str(path))))
|
||||
vault_root = Path(os.path.abspath(_expand_key(str(vault))))
|
||||
home_root = Path(os.path.expanduser("~"))
|
||||
for root, prefix in ((vault_root, ""), (home_root, "~/")):
|
||||
try:
|
||||
rel = p.relative_to(root)
|
||||
except ValueError:
|
||||
continue
|
||||
if rel == Path("."):
|
||||
return None
|
||||
return prefix + rel.as_posix()
|
||||
return None
|
||||
|
||||
|
||||
def _same_source(stored_key_value: str | None, query: Path, vault: Path) -> bool:
|
||||
"""True if a manifest key refers to the same source as *query*.
|
||||
|
||||
Matches on the raw string first (covers pseudo-keys), then on the resolved
|
||||
absolute form so an absolute query matches a vault-relative or home-relative
|
||||
stored key.
|
||||
"""
|
||||
if not stored_key_value:
|
||||
return False
|
||||
if str(query) == stored_key_value:
|
||||
return True
|
||||
stored_path = resolve_key(stored_key_value, vault)
|
||||
if stored_path is None:
|
||||
return False
|
||||
try:
|
||||
return stored_path.resolve() == query.resolve()
|
||||
except OSError:
|
||||
return False
|
||||
|
||||
|
||||
def _missing_on_disk(key: str | None, vault: Path) -> bool:
|
||||
"""True if a filesystem-style manifest key has no file on disk."""
|
||||
path = resolve_key(key, vault)
|
||||
if path is None:
|
||||
return False
|
||||
return not path.exists()
|
||||
|
||||
|
||||
def _vault_top_names(vault: Path) -> set[str]:
|
||||
"""Names of the vault's top-level entries, or an empty set if unreadable."""
|
||||
try:
|
||||
return {p.name for p in vault.iterdir()}
|
||||
except OSError:
|
||||
return set()
|
||||
|
||||
|
||||
def _is_vault_local(key: str | None, vault: Path, top_names: set[str] | None = None) -> bool:
|
||||
"""True if a file key names a source that travels with the vault.
|
||||
|
||||
Vault-local sources travel with the vault, so their absence is a real loss
|
||||
(``missing``). A key that is machine-specific *and* points outside this vault
|
||||
(``/home/other/wiki/x.md``, ``~/.claude/...``) may simply not exist on the
|
||||
machine reading a synced vault, so it is reported under ``unavailable``. An
|
||||
absolute or ``~``-relative path that resolves *inside* this vault is still
|
||||
vault-local — machine-specific in form, but not in target.
|
||||
|
||||
A *relative* key resolves lexically inside the vault even when it is really
|
||||
relative to some other root — the legacy ingest-root keys such as
|
||||
``-Users-x-github/abc.jsonl``. Path shape cannot tell those apart from vault
|
||||
keys, so the vault's own topology decides: the first segment must name a real
|
||||
top-level entry. That also gives the honest answer for an out-of-vault source
|
||||
parked in a namespace the vault does not contain (``external/.hermes/...``):
|
||||
unavailable, not missing. When the vault cannot be listed the lexical answer
|
||||
stands, so an I/O hiccup never hides a real loss.
|
||||
|
||||
A key written for another OS — a drive-letter path or one using backslashes
|
||||
as separators — cannot be resolved on this host, so it is machine-specific
|
||||
(``unavailable``) rather than a vault-relative key.
|
||||
"""
|
||||
path = resolve_key(key, vault)
|
||||
if path is None:
|
||||
return False
|
||||
raw = str(key)
|
||||
try:
|
||||
rel = Path(os.path.abspath(str(path))).relative_to(Path(os.path.abspath(str(vault))))
|
||||
except ValueError:
|
||||
return False
|
||||
if rel == Path("."):
|
||||
return False
|
||||
if os.name != "nt" and ("\\" in raw or _WINDOWS_ABS_RE.match(raw)):
|
||||
# A key written for another OS: a backslash is not a separator here and a
|
||||
# drive-letter path cannot be resolved, so this is machine-specific — not
|
||||
# a vault source whose file went away.
|
||||
return False
|
||||
if os.path.isabs(raw) or raw.startswith("~") or "$" in raw:
|
||||
return True
|
||||
if "/" not in raw and "\\" not in raw:
|
||||
# A bare filename at the vault root. There is no leading component that
|
||||
# could be a foreign root, so this is a vault source by construction —
|
||||
# and if it is gone, that is a real vault-local loss. The backslash guard
|
||||
# keeps a Windows-form key out of this shortcut on every platform.
|
||||
return True
|
||||
if top_names is None:
|
||||
top_names = _vault_top_names(vault)
|
||||
if not top_names:
|
||||
return True
|
||||
return rel.parts[0] in top_names
|
||||
|
||||
|
||||
def sha256_file(path: Path, chunk: int = 1 << 20) -> str:
|
||||
"""Return the hex SHA-256 digest of *path* without loading it all into RAM."""
|
||||
h = hashlib.sha256()
|
||||
with open(path, "rb") as fh:
|
||||
while True:
|
||||
block = fh.read(chunk)
|
||||
if not block:
|
||||
break
|
||||
h.update(block)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def sha256_dir(path: Path) -> str:
|
||||
"""Stable SHA-256 over all files in a directory tree (sorted by relative path)."""
|
||||
h = hashlib.sha256()
|
||||
for fp in sorted(path.rglob("*"), key=lambda p: p.relative_to(path).as_posix()):
|
||||
if fp.is_file():
|
||||
rel = fp.relative_to(path).as_posix()
|
||||
h.update(rel.encode())
|
||||
h.update(sha256_file(fp).encode())
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def compute_hash(path: Path) -> str:
|
||||
if path.is_dir():
|
||||
return sha256_dir(path)
|
||||
return sha256_file(path)
|
||||
|
||||
|
||||
def _resolved_form(path: Path) -> str | None:
|
||||
"""``realpath`` of *path* as a comparable string, or ``None`` if it cannot be resolved.
|
||||
|
||||
``os.path.normcase`` keeps the comparison equivalent to ``Path`` equality on
|
||||
case-insensitive filesystems.
|
||||
"""
|
||||
try:
|
||||
return os.path.normcase(str(path.resolve()))
|
||||
except OSError:
|
||||
return None
|
||||
|
||||
|
||||
def check_sources(vault: Path, source_paths: list[Path]) -> CheckResult:
|
||||
"""Classify each source as new / modified / unchanged vs. the manifest.
|
||||
|
||||
Also reports manifest entries whose source file no longer exists on disk.
|
||||
Handles both manifest shapes and compares hashes prefix-insensitively.
|
||||
|
||||
Every entry and every query is resolved once and matched through an index
|
||||
of its forms (raw key, resolved path). Resolving is a ``realpath`` walk —
|
||||
one ``lstat`` per path component — so doing it per (entry, query) pair, as
|
||||
the first-match scan did, made a 3,000-entry manifest cost about a second
|
||||
per queried file. The index keeps the first entry for each form, which is
|
||||
what the scan's ``break`` on first hit returned.
|
||||
"""
|
||||
entries = list(_iter_entries(_load_manifest(vault)))
|
||||
result: CheckResult = {
|
||||
"new": [], "modified": [], "unchanged": [], "missing": [], "unavailable": []
|
||||
}
|
||||
|
||||
by_form: dict[str, int] = {}
|
||||
entry_resolved: list[str | None] = []
|
||||
for i, (stored_key, _entry) in enumerate(entries):
|
||||
resolved = None
|
||||
if stored_key:
|
||||
by_form.setdefault(stored_key, i)
|
||||
stored_path = resolve_key(stored_key, vault)
|
||||
if stored_path is not None:
|
||||
resolved = _resolved_form(stored_path)
|
||||
if resolved is not None:
|
||||
by_form.setdefault(resolved, i)
|
||||
entry_resolved.append(resolved)
|
||||
|
||||
matched: set[int] = set()
|
||||
query_forms: set[str] = set()
|
||||
for path in source_paths:
|
||||
key = str(path)
|
||||
forms = [key]
|
||||
resolved = _resolved_form(path)
|
||||
if resolved is not None:
|
||||
forms.append(resolved)
|
||||
query_forms.update(forms)
|
||||
if not path.exists():
|
||||
result["missing"].append(key)
|
||||
continue
|
||||
current_hash = _strip_algo(compute_hash(path))
|
||||
hits = [by_form[f] for f in forms if f in by_form]
|
||||
if not hits:
|
||||
result["new"].append(key)
|
||||
continue
|
||||
i = min(hits) # the earliest entry matching by either form — the old scan's first hit
|
||||
matched.add(i)
|
||||
entry = entries[i][1]
|
||||
if _strip_algo(entry.get("content_hash")) != current_hash:
|
||||
result["modified"].append(key)
|
||||
else:
|
||||
result["unchanged"].append(key)
|
||||
|
||||
# Report manifest entries whose source file no longer exists on disk and
|
||||
# that weren't among the scanned paths (in any path form).
|
||||
top_names = _vault_top_names(vault)
|
||||
for i, (stored_key, _entry) in enumerate(entries):
|
||||
if i in matched:
|
||||
continue
|
||||
if stored_key in query_forms or (entry_resolved[i] is not None and entry_resolved[i] in query_forms):
|
||||
continue
|
||||
if _missing_on_disk(stored_key, vault):
|
||||
bucket = "missing" if _is_vault_local(stored_key, vault, top_names) else "unavailable"
|
||||
result[bucket].append(stored_key)
|
||||
|
||||
return result
|
||||
|
||||
|
||||
def update_source(
|
||||
vault: Path,
|
||||
source_path: Path,
|
||||
*,
|
||||
pages_produced: list[str] | None = None,
|
||||
key: str | None = None,
|
||||
) -> str:
|
||||
"""Record the current hash of *source_path* in the manifest. Returns the hash.
|
||||
|
||||
Edits the manifest in place: matches an existing entry across path forms and
|
||||
updates it, otherwise appends a new one — preserving the manifest's shape
|
||||
(dict or list), duplicate-path entries, and any skill-written fields.
|
||||
|
||||
The stored key is normalized to the portable form (vault-relative for
|
||||
in-vault sources, ``~``-relative under ``$HOME``) so a synced vault never
|
||||
accumulates machine absolute paths. Pass *key* explicitly for sources that
|
||||
have no filesystem representation in either form — a pseudo-key such as
|
||||
``repo:github.com/owner/name``, ``url:https://...``, or ``agent:claude/<id>``.
|
||||
When neither applies, the raw path is kept for backward compatibility and a
|
||||
warning is written to stderr, because that entry is not portable across
|
||||
machines.
|
||||
"""
|
||||
now = datetime.now(timezone.utc).isoformat()
|
||||
explicit_key = key is not None
|
||||
hash_path = source_path
|
||||
if explicit_key:
|
||||
new_key = key
|
||||
else:
|
||||
derived = stored_key(source_path, vault)
|
||||
if derived is None:
|
||||
new_key = str(source_path)
|
||||
print(
|
||||
f"warning: {source_path} is outside the vault and $HOME, so it has no "
|
||||
f"portable key; storing the absolute path. Pass key=<repo:|url:|agent:> "
|
||||
f"to keep the vault portable across machines.",
|
||||
file=sys.stderr,
|
||||
)
|
||||
else:
|
||||
from obsidian_wiki.provenance import prefer_archive_write_path
|
||||
preferred = prefer_archive_write_path(vault, derived)
|
||||
if preferred is not None:
|
||||
new_key = preferred
|
||||
hash_path = vault / preferred
|
||||
else:
|
||||
new_key = derived
|
||||
# Hash outside the lock — hashing a large source tree can take seconds and
|
||||
# nothing else in the manifest depends on it.
|
||||
current_hash = compute_hash(hash_path)
|
||||
|
||||
with manifest_lock(vault):
|
||||
return _update_source_locked(
|
||||
vault, source_path, new_key, explicit_key, current_hash, now, pages_produced,
|
||||
)
|
||||
|
||||
|
||||
def _update_source_locked(
|
||||
vault: Path,
|
||||
source_path: Path,
|
||||
new_key: str,
|
||||
explicit_key: bool,
|
||||
current_hash: str,
|
||||
now: str,
|
||||
pages_produced: list[str] | None,
|
||||
) -> str:
|
||||
"""The manifest read-modify-write half of :func:`update_source`."""
|
||||
manifest = _load_raw(vault)
|
||||
sources = manifest.get("sources")
|
||||
|
||||
if isinstance(sources, list):
|
||||
target: dict | None = None
|
||||
for e in sources:
|
||||
if isinstance(e, dict) and _same_source(
|
||||
e.get("path") or e.get("source_id"), source_path, vault
|
||||
):
|
||||
target = e
|
||||
break
|
||||
if target is None:
|
||||
target = {"path": new_key}
|
||||
sources.append(target)
|
||||
elif target.get("path") != new_key and (
|
||||
explicit_key or _rekey_stale_raw_to_archive(target.get("path"), new_key)
|
||||
):
|
||||
# Explicit --key is authoritative; otherwise only stale
|
||||
# ``_raw/<name>.md`` keys move onto ``_raw/_archived/…``.
|
||||
target["path"] = new_key
|
||||
target["content_hash"] = _format_hash(target.get("content_hash"), current_hash)
|
||||
target["last_ingested"] = now
|
||||
if pages_produced is not None:
|
||||
target["pages_produced"] = pages_produced
|
||||
else:
|
||||
if not isinstance(sources, dict):
|
||||
sources = {}
|
||||
match_key: str | None = None
|
||||
for existing_key in sources:
|
||||
if _same_source(existing_key, source_path, vault):
|
||||
match_key = existing_key
|
||||
break
|
||||
if match_key is not None and match_key != new_key and (
|
||||
explicit_key or _rekey_stale_raw_to_archive(match_key, new_key)
|
||||
):
|
||||
# Explicit key wins; otherwise only stale ``_raw/<name>.md`` keys
|
||||
# move onto ``_raw/_archived/…`` so other portable keys stay put.
|
||||
moved = sources.pop(match_key)
|
||||
existing = sources.get(new_key)
|
||||
if isinstance(existing, dict):
|
||||
moved = {**existing, **moved}
|
||||
sources[new_key] = moved
|
||||
match_key = new_key
|
||||
# A matched legacy entry otherwise keeps its existing key; only new
|
||||
# entries use the portable form, so an in-place update never re-keys a
|
||||
# dict by surprise (migrate handles legacy conversion).
|
||||
manifest_key = match_key if match_key is not None else new_key
|
||||
entry = sources.get(manifest_key) if isinstance(sources.get(manifest_key), dict) else {}
|
||||
entry["content_hash"] = _format_hash(entry.get("content_hash"), current_hash)
|
||||
entry["last_ingested"] = now
|
||||
if pages_produced is not None:
|
||||
entry["pages_produced"] = pages_produced
|
||||
sources[manifest_key] = entry
|
||||
|
||||
manifest["sources"] = sources
|
||||
_write_manifest(vault, manifest)
|
||||
return current_hash
|
||||
|
||||
|
||||
def _rekey_stale_raw_to_archive(old_key: str | None, new_key: str) -> bool:
|
||||
if not old_key:
|
||||
return False
|
||||
old = Path(old_key)
|
||||
new = Path(new_key)
|
||||
return (
|
||||
old.parts[:1] == ("_raw",)
|
||||
and "_archived" not in old.parts
|
||||
and len(new.parts) >= 3
|
||||
and new.parts[0] == "_raw"
|
||||
and new.parts[1] == "_archived"
|
||||
)
|
||||
|
||||
|
||||
def hash_file(path: Path) -> str:
|
||||
"""Just compute and return the hash — no manifest I/O."""
|
||||
return compute_hash(path)
|
||||
|
||||
|
||||
# --- project (git) staleness ------------------------------------------------
|
||||
# `check_sources` answers "did this source file's bytes change?". It cannot
|
||||
# answer the same question for a project, because a git repo is not stored as a
|
||||
# hashed source: wiki-update records it under `projects` keyed by `source_repo`
|
||||
# with a `last_commit_synced` bookmark. Nothing compared that bookmark to HEAD,
|
||||
# so a vault could be arbitrarily far behind a repo with no signal at all.
|
||||
|
||||
_GIT_TIMEOUT = 15
|
||||
|
||||
|
||||
class ProjectStatus(TypedDict, total=False):
|
||||
project: str
|
||||
commits: int
|
||||
last_commit_synced: str
|
||||
path: str
|
||||
reason: str
|
||||
|
||||
|
||||
class ProjectCheckResult(TypedDict):
|
||||
behind: list[ProjectStatus] # checkout has moved past last_commit_synced
|
||||
current: list[str] # last_commit_synced == HEAD
|
||||
unreachable: list[ProjectStatus] # stored sha not usable (rebase/force-push/gc)
|
||||
unsynced: list[str] # entry exists but no last_commit_synced yet
|
||||
unavailable: list[ProjectStatus] # no usable checkout on this machine
|
||||
|
||||
|
||||
def _git_in(path: Path, *args: str) -> subprocess.CompletedProcess | None:
|
||||
"""Run git in *path*; None when git itself is unusable (absent, hung)."""
|
||||
try:
|
||||
return subprocess.run(
|
||||
["git", "-C", str(path), *args],
|
||||
capture_output=True, text=True, timeout=_GIT_TIMEOUT,
|
||||
)
|
||||
except (OSError, subprocess.SubprocessError):
|
||||
return None
|
||||
|
||||
|
||||
def check_projects(vault: Path) -> ProjectCheckResult:
|
||||
"""Classify each manifest ``projects`` entry by how far its checkout has
|
||||
moved past the recorded ``last_commit_synced``.
|
||||
|
||||
Mirrors :func:`check_sources`' vocabulary deliberately: a project whose
|
||||
checkout is not on this machine is ``unavailable``, not stale.
|
||||
``source_cwd_hint`` is a *hint* — the manifest is portable across machines
|
||||
by design, so a missing checkout is an absence of evidence, never a finding.
|
||||
|
||||
Read-only: never runs a git command that can write, and never fails the
|
||||
whole call because one entry is malformed.
|
||||
"""
|
||||
raw = _load_raw(vault).get("projects")
|
||||
projects = raw if isinstance(raw, dict) else {}
|
||||
result: ProjectCheckResult = {
|
||||
"behind": [], "current": [], "unreachable": [], "unsynced": [], "unavailable": []
|
||||
}
|
||||
|
||||
for name in sorted(projects):
|
||||
entry = projects[name]
|
||||
if not isinstance(entry, dict):
|
||||
result["unavailable"].append({"project": name, "reason": "malformed_entry"})
|
||||
continue
|
||||
|
||||
hint = entry.get("source_cwd_hint")
|
||||
if not isinstance(hint, str) or not hint.strip():
|
||||
result["unavailable"].append({"project": name, "reason": "no_source_cwd_hint"})
|
||||
continue
|
||||
path = Path(hint.strip()).expanduser()
|
||||
if not path.is_dir():
|
||||
result["unavailable"].append(
|
||||
{"project": name, "reason": "checkout_not_found", "path": str(path)}
|
||||
)
|
||||
continue
|
||||
|
||||
# `rev-parse --git-dir` rather than a `.git` directory test: a linked
|
||||
# worktree's `.git` is a file, and the hint may point below the root.
|
||||
probe = _git_in(path, "rev-parse", "--git-dir")
|
||||
if probe is None:
|
||||
result["unavailable"].append(
|
||||
{"project": name, "reason": "git_unavailable", "path": str(path)}
|
||||
)
|
||||
continue
|
||||
if probe.returncode != 0:
|
||||
result["unavailable"].append(
|
||||
{"project": name, "reason": "not_a_git_repo", "path": str(path)}
|
||||
)
|
||||
continue
|
||||
|
||||
sha = entry.get("last_commit_synced")
|
||||
if not isinstance(sha, str) or not sha.strip():
|
||||
result["unsynced"].append(name)
|
||||
continue
|
||||
sha = sha.strip()
|
||||
|
||||
ancestor = _git_in(path, "merge-base", "--is-ancestor", sha, "HEAD")
|
||||
if ancestor is None:
|
||||
result["unavailable"].append(
|
||||
{"project": name, "reason": "git_unavailable", "path": str(path)}
|
||||
)
|
||||
continue
|
||||
if ancestor.returncode != 0:
|
||||
# 1 = resolves but is not an ancestor (history was rewritten);
|
||||
# anything else (128) = the object is not in this repo at all.
|
||||
result["unreachable"].append({
|
||||
"project": name,
|
||||
"reason": "not_ancestor" if ancestor.returncode == 1 else "unknown_commit",
|
||||
"last_commit_synced": sha,
|
||||
"path": str(path),
|
||||
})
|
||||
continue
|
||||
|
||||
counted = _git_in(path, "rev-list", "--count", f"{sha}..HEAD")
|
||||
if counted is None or counted.returncode != 0:
|
||||
result["unavailable"].append(
|
||||
{"project": name, "reason": "count_failed", "path": str(path)}
|
||||
)
|
||||
continue
|
||||
try:
|
||||
commits = int(counted.stdout.strip())
|
||||
except ValueError:
|
||||
result["unavailable"].append(
|
||||
{"project": name, "reason": "count_unparsable", "path": str(path)}
|
||||
)
|
||||
continue
|
||||
|
||||
if commits == 0:
|
||||
result["current"].append(name)
|
||||
else:
|
||||
result["behind"].append({
|
||||
"project": name,
|
||||
"commits": commits,
|
||||
"last_commit_synced": sha,
|
||||
"path": str(path),
|
||||
})
|
||||
|
||||
return result
|
||||
+1001
-1
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,215 @@
|
||||
"""The wiki clock: every stamp the memory writer puts in a vault comes from here.
|
||||
|
||||
A vault keeps its timezone in the frontmatter of its ``index.md``::
|
||||
|
||||
---
|
||||
title: Wiki Index
|
||||
timezone: America/Detroit
|
||||
---
|
||||
|
||||
Without the key the writer keeps its original behavior: UTC, in the ``Z``
|
||||
spelling (``2026-10-10T18:40:37Z``), and bare dates are UTC calendar days.
|
||||
With it, every stamp carries the zone's own offset for that instant
|
||||
(``2026-10-10T14:40:37-04:00``), so daylight-saving transitions are right by
|
||||
construction, and a bare date (``created:`` on a profile fact, ``touched`` on a
|
||||
todo) is the calendar day *there* rather than in UTC — the difference that
|
||||
puts an evening's work on tomorrow's date for anyone west of Greenwich.
|
||||
|
||||
The setting lives in the file it governs, like ``index_folder_hubs``: it
|
||||
travels with the vault and applies to every writer — CLI, server, Python API —
|
||||
whatever ``.env`` the caller resolved. ``rebuild_index`` preserves the
|
||||
frontmatter, so the key survives every regeneration.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone, tzinfo
|
||||
from pathlib import Path
|
||||
from typing import Optional, Union
|
||||
from zoneinfo import ZoneInfo, ZoneInfoNotFoundError
|
||||
|
||||
from obsidian_wiki.vault import split_frontmatter
|
||||
|
||||
TIMEZONE_KEY = "timezone"
|
||||
_TIMEZONE_LINE = re.compile(rf"^{TIMEZONE_KEY}:[ \t]*(?P<value>.*)$", re.MULTILINE)
|
||||
|
||||
|
||||
class TimezoneError(ValueError):
|
||||
"""``timezone:`` names a zone this Python cannot resolve."""
|
||||
|
||||
|
||||
def parse_timezone(name: str) -> Optional[tzinfo]:
|
||||
"""IANA name → tzinfo. Empty or ``UTC`` → ``None``, meaning the default.
|
||||
|
||||
``None`` rather than ``timezone.utc`` so callers can tell "nothing
|
||||
configured" from a zone, and keep writing the ``Z`` spelling for it.
|
||||
"""
|
||||
value = (name or "").strip().strip("\"'")
|
||||
if not value or value.upper() in {"UTC", "Z"}:
|
||||
return None
|
||||
try:
|
||||
return ZoneInfo(value)
|
||||
except (ZoneInfoNotFoundError, ValueError, OSError) as exc:
|
||||
raise TimezoneError(
|
||||
f"unknown timezone {value!r} in index.md; use an IANA name such as America/Detroit"
|
||||
) from exc
|
||||
|
||||
|
||||
def timezone_name(vault: Union[str, Path]) -> str:
|
||||
"""The configured zone name, or ``""`` when the vault runs on the default."""
|
||||
index = Path(vault) / "index.md"
|
||||
if not index.is_file():
|
||||
return ""
|
||||
frontmatter, _body = split_frontmatter(index.read_text(encoding="utf-8", errors="replace"))
|
||||
match = _TIMEZONE_LINE.search(frontmatter)
|
||||
return match.group("value").strip().strip("\"'") if match else ""
|
||||
|
||||
|
||||
def wiki_timezone(vault: Union[str, Path]) -> Optional[tzinfo]:
|
||||
"""The vault's zone from ``index.md``, or ``None`` for the UTC default.
|
||||
|
||||
Raises :class:`TimezoneError` when the key names a zone that does not
|
||||
exist — a typo must not silently write UTC under a local label.
|
||||
"""
|
||||
return parse_timezone(timezone_name(vault))
|
||||
|
||||
|
||||
def set_timezone(vault: Union[str, Path], name: str) -> str:
|
||||
"""Write ``timezone: <name>`` into the frontmatter of the vault's ``index.md``.
|
||||
|
||||
Validates the name first, so a typo never lands in the file. Replaces an
|
||||
existing key, otherwise inserts it after ``title:``; an ``index.md`` with
|
||||
no frontmatter gets one. An empty name is a no-op. Returns the stored name.
|
||||
"""
|
||||
value = (name or "").strip().strip("\"'")
|
||||
if not value:
|
||||
return ""
|
||||
parse_timezone(value) # raises TimezoneError on an unknown zone
|
||||
index = Path(vault) / "index.md"
|
||||
if not index.is_file():
|
||||
raise FileNotFoundError(f"no index.md in {vault}; scaffold the vault first")
|
||||
from obsidian_wiki.memory import atomic_write # lazy: memory imports this module
|
||||
|
||||
text = index.read_text(encoding="utf-8")
|
||||
frontmatter, body = split_frontmatter(text)
|
||||
line = f"{TIMEZONE_KEY}: {value}"
|
||||
if _TIMEZONE_LINE.search(frontmatter):
|
||||
frontmatter = _TIMEZONE_LINE.sub(line, frontmatter, count=1)
|
||||
elif frontmatter.strip():
|
||||
lines = frontmatter.rstrip("\n").split("\n")
|
||||
at = next((i + 1 for i, l in enumerate(lines) if l.startswith("title:")), len(lines))
|
||||
lines.insert(at, line)
|
||||
frontmatter = "\n".join(lines)
|
||||
else:
|
||||
frontmatter = f"title: Wiki Index\n{line}"
|
||||
if text.startswith("---"):
|
||||
rest = text.split("---", 2)[2] if text.count("---") >= 2 else body
|
||||
atomic_write(index, f"---\n{frontmatter.strip()}\n---{rest}")
|
||||
else:
|
||||
atomic_write(index, f"---\n{frontmatter.strip()}\n---\n\n{text}")
|
||||
return value
|
||||
|
||||
|
||||
def _moment(now: Optional[datetime]) -> datetime:
|
||||
moment = now or datetime.now(timezone.utc)
|
||||
if moment.tzinfo is None or moment.utcoffset() is None:
|
||||
raise ValueError("clock needs an aware datetime; attach a timezone first")
|
||||
return moment
|
||||
|
||||
|
||||
def stamp(tz: Optional[tzinfo] = None, *, now: Optional[datetime] = None) -> str:
|
||||
"""Full timestamp to the second: ``…Z`` on the default, ``…-04:00`` in a zone."""
|
||||
moment = _moment(now)
|
||||
if tz is None:
|
||||
return moment.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
return moment.astimezone(tz).isoformat(timespec="seconds")
|
||||
|
||||
|
||||
def today(tz: Optional[tzinfo] = None, *, now: Optional[datetime] = None) -> str:
|
||||
"""Bare calendar date in the zone (UTC on the default)."""
|
||||
return _moment(now).astimezone(tz or timezone.utc).date().isoformat()
|
||||
|
||||
|
||||
def compact(tz: Optional[tzinfo] = None, *, now: Optional[datetime] = None) -> str:
|
||||
"""Filesystem-safe form for backup directory names: ``20261010T184037Z``."""
|
||||
moment = _moment(now)
|
||||
if tz is None:
|
||||
return moment.astimezone(timezone.utc).strftime("%Y%m%dT%H%M%SZ")
|
||||
return moment.astimezone(tz).strftime("%Y%m%dT%H%M%S%z")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# converting the stamps already in a vault
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
#: A full ISO-8601 timestamp with a zone: ``2026-10-10T18:40:37Z``,
|
||||
#: ``2026-10-10T14:40:37-04:00``, ``2026-10-10T14:40:37.123-0400``. Bare dates,
|
||||
#: naive stamps (no zone) and the compact backup-directory form
|
||||
#: (``20261010T184037Z``) deliberately do not match: the first two cannot be
|
||||
#: converted without guessing, the last names a directory that exists.
|
||||
STAMP_RE = re.compile(
|
||||
r"(?<![\w/])"
|
||||
r"(?P<date>\d{4}-\d{2}-\d{2})T(?P<time>\d{2}:\d{2}(?::\d{2}(?:\.\d{1,6})?)?)"
|
||||
r"(?P<zone>Z|[+-]\d{2}:?\d{2})"
|
||||
r"(?!\w)"
|
||||
)
|
||||
_NAIVE_RE = re.compile(r"(?<![\w/])\d{4}-\d{2}-\d{2}T\d{2}:\d{2}(?::\d{2})?(?![\w:+-])")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class StampChange:
|
||||
line: int # 1-based
|
||||
old: str
|
||||
new: str
|
||||
|
||||
|
||||
def convert_stamp(text: str, tz: Optional[tzinfo]) -> Optional[str]:
|
||||
"""Re-render one full stamp in *tz* (``Z`` form when *tz* is ``None``).
|
||||
|
||||
Returns ``None`` when the text is not a parseable zoned stamp. The instant
|
||||
is preserved; only the spelling changes, so a stamp already in the zone
|
||||
comes back unchanged and a second run is a no-op.
|
||||
"""
|
||||
match = STAMP_RE.fullmatch(text.strip())
|
||||
if not match:
|
||||
return None
|
||||
time = match.group("time")
|
||||
if time.count(":") == 1:
|
||||
time += ":00"
|
||||
zone = match.group("zone")
|
||||
if zone == "Z":
|
||||
zone = "+00:00"
|
||||
elif ":" not in zone:
|
||||
zone = f"{zone[:3]}:{zone[3:]}"
|
||||
try:
|
||||
moment = datetime.fromisoformat(f"{match.group('date')}T{time}{zone}")
|
||||
except ValueError:
|
||||
return None
|
||||
return stamp(tz, now=moment)
|
||||
|
||||
|
||||
def convert_text(text: str, tz: Optional[tzinfo]) -> "tuple[str, list, int]":
|
||||
"""Every zoned stamp in *text* re-rendered in *tz*.
|
||||
|
||||
Returns ``(new_text, changes, naive)``: the converted text, one
|
||||
:class:`StampChange` per stamp whose spelling changed, and how many naive
|
||||
stamps (no zone, so no conversion possible) were seen and left alone.
|
||||
"""
|
||||
changes: list = []
|
||||
out_lines = []
|
||||
naive = 0
|
||||
for number, line in enumerate(text.split("\n"), start=1):
|
||||
naive += len(_NAIVE_RE.findall(line))
|
||||
|
||||
def _swap(match: "re.Match[str]") -> str:
|
||||
old = match.group(0)
|
||||
new = convert_stamp(old, tz)
|
||||
if new is None or new == old:
|
||||
return old
|
||||
changes.append(StampChange(number, old, new))
|
||||
return new
|
||||
|
||||
out_lines.append(STAMP_RE.sub(_swap, line))
|
||||
return "\n".join(out_lines), changes, naive
|
||||
@@ -0,0 +1,914 @@
|
||||
"""Vault graph analysis: community detection, god nodes, bridges, surprising connections.
|
||||
|
||||
Reads the vault's wikilink graph (from page frontmatter and [[wikilinks]]),
|
||||
then runs the same family of algorithms graphify uses on code graphs — all in
|
||||
pure Python (no networkx / numpy):
|
||||
1. Degree-based god-node ranking (most-linked pages)
|
||||
2. Community detection — Leiden via `obsidian-wiki[graph]` if installed,
|
||||
else greedy label propagation (pure Python, no binary deps)
|
||||
3. Community cohesion — intra-community edge density (graphify
|
||||
`cohesion_score`); < 0.15 flags a fragmented cluster
|
||||
4. Bridge pages — Brandes betweenness centrality; the pages that sit on
|
||||
the most shortest paths between other pages (graphify's bridge nodes)
|
||||
5. Surprising connections — cross-community edges ranked by unexpectedness,
|
||||
de-duplicated one-per-community-pair so a single hub can't dominate
|
||||
6. Suggested questions — derived from bridges, isolates, dead-end hubs and
|
||||
low-cohesion communities (graphify `suggest_questions`)
|
||||
7. Path finding / neighbourhoods — `shortest_path` and BFS `neighborhood`
|
||||
(graphify `path` and blast-radius / `affected`)
|
||||
8. Graph diff — compare against a previous snapshot (graphify `graph_diff`)
|
||||
|
||||
Output JSON:
|
||||
{
|
||||
"god_nodes": [{"page": "...", "degree": N, "in_degree": N, "out_degree": N}, ...],
|
||||
"bridges": [{"page": "...", "betweenness": 0.N, "connects": [comm_id, ...]}, ...],
|
||||
"communities": [{"id": N, "size": N, "pages": [...], "label": "...", "cohesion": 0.N}, ...],
|
||||
"surprising_connections": [{"source": "...", "target": "...", "score": 0.N,
|
||||
"note": "bridges community A -> community B"}, ...],
|
||||
"suggested_questions": [{"type": "...", "question": "...", "why": "..."}, ...],
|
||||
"dead_ends": ["page with 0 outgoing links", ...],
|
||||
"isolated": ["page with 0 links at all", ...],
|
||||
"stats": {"pages": N, "edges": N, "communities": N, "density": 0.N}
|
||||
}
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import random
|
||||
import re
|
||||
import time
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from obsidian_wiki.links import PageIndex, strip_code
|
||||
from obsidian_wiki.provenance import archive_wikilink_relpath
|
||||
from obsidian_wiki.vault import SKIP_DIRS as VAULT_SKIP_DIRS
|
||||
from obsidian_wiki.vault import iter_md, okignore_patterns, okignored, skipped_dir # noqa: F401 (re-exported)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Wikilink / frontmatter parsing
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_WIKILINK_RE = re.compile(r"\[\[([^\]|#]+?)(?:[|#][^\]]*?)?\]\]")
|
||||
# Local `.md` targets only: skip URLs (any `scheme:`) and require `.md` to end
|
||||
# the path, before an optional `#anchor` or `"title"`.
|
||||
_MD_LINK_RE = re.compile(
|
||||
r"\[.*?\]\((?![A-Za-z][A-Za-z0-9+.-]*:)([^)#\s]+\.md)(?:[#\s][^)]*)?\)"
|
||||
)
|
||||
_TAGS_RE = re.compile(r"^tags:\s*\[([^\]]+)\]", re.MULTILINE)
|
||||
_TAGS_LIST_RE = re.compile(r"^tags:\s*\n((?:\s+-\s+\S+\n)+)", re.MULTILINE)
|
||||
|
||||
|
||||
def _slug(page_name: str) -> str:
|
||||
"""Normalise a wikilink target to a page slug."""
|
||||
return page_name.strip().lower().replace(" ", "-")
|
||||
|
||||
|
||||
def _page_slug(path: Path, root: Path) -> str:
|
||||
return _slug(path.stem)
|
||||
|
||||
|
||||
# --- Shared page selection -------------------------------------------------
|
||||
# Every module that builds a graph over the vault MUST use these, so the
|
||||
# wikilink graph is identical no matter which entry point produced it.
|
||||
|
||||
#: Directories that hold staging / archive / config material, not knowledge.
|
||||
#: Dot-prefixed paths (`.venv`, `.git`) are skipped wholesale by `is_wiki_page`;
|
||||
#: this list also covers the non-hidden equivalents (`venv/`, `node_modules/`).
|
||||
SKIP_DIRS = VAULT_SKIP_DIRS | {"_meta", "_readouts"}
|
||||
|
||||
#: Vault bookkeeping files at the vault ROOT. They link to (almost) every page,
|
||||
#: so leaving them in makes every pair of pages look ~2 hops apart and lets a
|
||||
#: table of contents outrank real knowledge in every degree/centrality ranking.
|
||||
#: Only the root-level files are skipped — a legitimate `concepts/index.md`
|
||||
#: is still a normal page.
|
||||
SKIP_ROOT_FILES = frozenset({"index", "log", "hot", "_insights"})
|
||||
|
||||
|
||||
def is_wiki_page(path: Path, vault: Path) -> bool:
|
||||
"""True if `path` is a knowledge page of `vault` (not staging/bookkeeping)."""
|
||||
try:
|
||||
rel = path.relative_to(vault)
|
||||
except ValueError:
|
||||
return False
|
||||
if skipped_dir(rel, SKIP_DIRS):
|
||||
return False
|
||||
if len(rel.parts) == 1 and path.stem in SKIP_ROOT_FILES:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def iter_pages(vault: Path) -> list[Path]:
|
||||
"""All knowledge pages in the vault, in stable sorted order."""
|
||||
return [p for p in iter_md(vault, SKIP_DIRS) if is_wiki_page(p, vault)]
|
||||
|
||||
|
||||
def parse_vault_graph(vault: Path) -> tuple[dict[str, list[str]], dict[str, list[str]]]:
|
||||
"""Parse all .md pages and return (outgoing_edges, page_tags).
|
||||
|
||||
outgoing_edges: {source_slug: [target_slug, ...]}
|
||||
page_tags: {slug: [tag, ...]}
|
||||
"""
|
||||
outgoing: dict[str, list[str]] = defaultdict(list)
|
||||
tags_map: dict[str, list[str]] = {}
|
||||
|
||||
pages: list[Path] = iter_pages(vault)
|
||||
rels = {page: page.relative_to(vault).as_posix() for page in pages}
|
||||
index = PageIndex(rels.values())
|
||||
|
||||
for page in pages:
|
||||
src = index.id_for(rels[page])
|
||||
try:
|
||||
text = page.read_text(encoding="utf-8", errors="replace")
|
||||
except OSError:
|
||||
continue
|
||||
links_text = strip_code(text)
|
||||
|
||||
# Tags
|
||||
m = _TAGS_RE.search(text)
|
||||
if m:
|
||||
tags_map[src] = [t.strip().strip("'\"") for t in m.group(1).split(",")]
|
||||
else:
|
||||
m2 = _TAGS_LIST_RE.search(text)
|
||||
if m2:
|
||||
tags_map[src] = [ln.strip().lstrip("- ") for ln in m2.group(1).splitlines() if ln.strip()]
|
||||
|
||||
# Wikilinks
|
||||
for link in _WIKILINK_RE.findall(links_text):
|
||||
if archive_wikilink_relpath(vault, link) is not None:
|
||||
continue
|
||||
target = index.resolve(link, rels[page])
|
||||
if target and target != src:
|
||||
outgoing[src].append(target)
|
||||
|
||||
# Markdown links (when OBSIDIAN_LINK_FORMAT=markdown)
|
||||
for href in _MD_LINK_RE.findall(links_text):
|
||||
if archive_wikilink_relpath(vault, href) is not None:
|
||||
continue
|
||||
target = index.resolve(href, rels[page])
|
||||
if target and target != src:
|
||||
outgoing[src].append(target)
|
||||
|
||||
if src not in outgoing:
|
||||
outgoing[src] = []
|
||||
|
||||
# Ensure every known page appears as a key
|
||||
for page_id in index.ids():
|
||||
outgoing.setdefault(page_id, [])
|
||||
|
||||
return dict(outgoing), tags_map
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Graph metrics
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _build_degree_tables(
|
||||
outgoing: dict[str, list[str]]
|
||||
) -> tuple[dict[str, int], dict[str, int], dict[str, int]]:
|
||||
out_deg: dict[str, int] = {n: len(v) for n, v in outgoing.items()}
|
||||
in_deg: dict[str, int] = defaultdict(int)
|
||||
for targets in outgoing.values():
|
||||
for t in targets:
|
||||
in_deg[t] += 1
|
||||
degree = {n: out_deg.get(n, 0) + in_deg.get(n, 0) for n in outgoing}
|
||||
for n in in_deg:
|
||||
degree.setdefault(n, in_deg[n])
|
||||
return degree, dict(in_deg), out_deg
|
||||
|
||||
|
||||
def god_nodes(outgoing: dict[str, list[str]], top_n: int = 20) -> list[dict]:
|
||||
degree, in_deg, out_deg = _build_degree_tables(outgoing)
|
||||
ranked = sorted(degree, key=lambda n: -degree[n])[:top_n]
|
||||
return [
|
||||
{"page": n, "degree": degree[n],
|
||||
"in_degree": in_deg.get(n, 0), "out_degree": out_deg.get(n, 0)}
|
||||
for n in ranked
|
||||
]
|
||||
|
||||
|
||||
def dead_ends(outgoing: dict[str, list[str]]) -> list[str]:
|
||||
"""Pages with no outgoing links."""
|
||||
return [n for n, targets in outgoing.items() if not targets]
|
||||
|
||||
|
||||
def isolated(outgoing: dict[str, list[str]]) -> list[str]:
|
||||
"""Pages with zero links in either direction."""
|
||||
_, in_deg, out_deg = _build_degree_tables(outgoing)
|
||||
return [n for n in outgoing if in_deg.get(n, 0) == 0 and out_deg.get(n, 0) == 0]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Community detection — greedy modularity (Newman-Girvan, pure Python)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _modularity(communities: list[set[str]], outgoing: dict[str, list[str]]) -> float:
|
||||
"""Approximate modularity Q for an undirected projection of outgoing edges."""
|
||||
all_edges = [(s, t) for s, targets in outgoing.items() for t in targets]
|
||||
m = len(all_edges)
|
||||
if m == 0:
|
||||
return 0.0
|
||||
degree = defaultdict(int)
|
||||
for s, t in all_edges:
|
||||
degree[s] += 1
|
||||
degree[t] += 1
|
||||
node_comm: dict[str, int] = {}
|
||||
for i, comm in enumerate(communities):
|
||||
for n in comm:
|
||||
node_comm[n] = i
|
||||
Q = 0.0
|
||||
for s, t in all_edges:
|
||||
if node_comm.get(s) == node_comm.get(t):
|
||||
Q += 1 - (degree[s] * degree[t]) / (2 * m)
|
||||
return Q / m
|
||||
|
||||
|
||||
def detect_communities_greedy(outgoing: dict[str, list[str]]) -> list[set[str]]:
|
||||
"""Greedy modularity community detection (label propagation variant).
|
||||
|
||||
Fast O(n·k) approach suitable for vaults up to ~5 000 pages. Each node
|
||||
adopts the most frequent label among its neighbours; iterate until stable.
|
||||
Falls back gracefully to one community per page if the graph is empty.
|
||||
"""
|
||||
nodes = list(outgoing.keys())
|
||||
if not nodes:
|
||||
return []
|
||||
|
||||
# Build undirected adjacency
|
||||
adj: dict[str, list[str]] = defaultdict(list)
|
||||
for src, targets in outgoing.items():
|
||||
for t in targets:
|
||||
adj[src].append(t)
|
||||
adj[t].append(src)
|
||||
|
||||
# Initialise: each node in its own community (label = index)
|
||||
labels: dict[str, int] = {n: i for i, n in enumerate(nodes)}
|
||||
|
||||
# Visit nodes in a shuffled order each round. A fixed sequential sweep
|
||||
# cascades one label along chains and collapses well-separated clusters
|
||||
# into a single community (a barbell graph comes back as one group). The
|
||||
# RNG is seeded from the node set, so the result stays reproducible.
|
||||
rng = random.Random(len(nodes))
|
||||
order = list(nodes)
|
||||
|
||||
for _ in range(20): # max 20 rounds
|
||||
changed = False
|
||||
rng.shuffle(order)
|
||||
for n in order:
|
||||
neighbours = adj[n]
|
||||
if not neighbours:
|
||||
continue
|
||||
freq: dict[int, int] = defaultdict(int)
|
||||
for nb in neighbours:
|
||||
freq[labels[nb]] += 1
|
||||
best = max(freq, key=lambda lbl: (freq[lbl], -lbl))
|
||||
if best != labels[n]:
|
||||
labels[n] = best
|
||||
changed = True
|
||||
if not changed:
|
||||
break
|
||||
|
||||
# Group by label
|
||||
groups: dict[int, set[str]] = defaultdict(set)
|
||||
for n, lbl in labels.items():
|
||||
groups[lbl].add(n)
|
||||
return list(groups.values())
|
||||
|
||||
|
||||
def detect_communities(outgoing: dict[str, list[str]]) -> list[set[str]]:
|
||||
"""Try Leiden (leidenalg + igraph) first; fall back to greedy label propagation."""
|
||||
try:
|
||||
import igraph as ig
|
||||
import leidenalg
|
||||
|
||||
nodes = list(outgoing.keys())
|
||||
node_idx = {n: i for i, n in enumerate(nodes)}
|
||||
edges = [
|
||||
(node_idx[s], node_idx[t])
|
||||
for s, targets in outgoing.items()
|
||||
for t in targets
|
||||
if t in node_idx
|
||||
]
|
||||
g = ig.Graph(n=len(nodes), edges=edges, directed=False)
|
||||
partition = leidenalg.find_partition(g, leidenalg.ModularityVertexPartition)
|
||||
return [
|
||||
{nodes[i] for i in cluster}
|
||||
for cluster in partition
|
||||
]
|
||||
except ImportError:
|
||||
return detect_communities_greedy(outgoing)
|
||||
|
||||
|
||||
def _undirected_adj(outgoing: dict[str, list[str]]) -> dict[str, set[str]]:
|
||||
"""Undirected simple-graph adjacency (duplicate/self edges collapsed)."""
|
||||
adj: dict[str, set[str]] = {n: set() for n in outgoing}
|
||||
for src, targets in outgoing.items():
|
||||
for t in targets:
|
||||
if t == src:
|
||||
continue
|
||||
adj[src].add(t)
|
||||
adj.setdefault(t, set()).add(src)
|
||||
return adj
|
||||
|
||||
|
||||
def _node_community_map(communities: list[set[str]]) -> dict[str, int]:
|
||||
return {n: i for i, comm in enumerate(communities) for n in comm}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Community cohesion (graphify: cluster.cohesion_score)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def cohesion_score(outgoing: dict[str, list[str]], community: set[str]) -> float:
|
||||
"""Ratio of actual intra-community (undirected) edges to the maximum possible.
|
||||
|
||||
1.0 = clique, 0.0 = no internal links. Singletons score 1.0.
|
||||
"""
|
||||
n = len(community)
|
||||
if n <= 1:
|
||||
return 1.0
|
||||
adj = _undirected_adj(outgoing)
|
||||
actual = sum(1 for a in community for b in adj.get(a, ()) if b in community and a < b)
|
||||
possible = n * (n - 1) / 2
|
||||
return round(actual / possible, 4) if possible else 0.0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Betweenness centrality — Brandes (2001), pure Python
|
||||
# (graphify: nx.betweenness_centrality → "bridge nodes")
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def betweenness_centrality(
|
||||
outgoing: dict[str, list[str]],
|
||||
normalized: bool = True,
|
||||
) -> dict[str, float]:
|
||||
"""Exact node betweenness on the undirected projection (Brandes algorithm).
|
||||
|
||||
O(V·E) — fine for vaults up to a few thousand pages. Values are normalised
|
||||
to [0, 1] by 2/((n-1)(n-2)) so scores are comparable across vault sizes.
|
||||
"""
|
||||
adj = _undirected_adj(outgoing)
|
||||
nodes = list(adj)
|
||||
bc: dict[str, float] = {n: 0.0 for n in nodes}
|
||||
for s in nodes:
|
||||
stack: list[str] = []
|
||||
preds: dict[str, list[str]] = {n: [] for n in nodes}
|
||||
sigma: dict[str, float] = {n: 0.0 for n in nodes}
|
||||
dist: dict[str, int] = {n: -1 for n in nodes}
|
||||
sigma[s] = 1.0
|
||||
dist[s] = 0
|
||||
queue = [s]
|
||||
qi = 0
|
||||
while qi < len(queue):
|
||||
v = queue[qi]
|
||||
qi += 1
|
||||
stack.append(v)
|
||||
for w in adj[v]:
|
||||
if dist[w] < 0:
|
||||
dist[w] = dist[v] + 1
|
||||
queue.append(w)
|
||||
if dist[w] == dist[v] + 1:
|
||||
sigma[w] += sigma[v]
|
||||
preds[w].append(v)
|
||||
delta: dict[str, float] = {n: 0.0 for n in nodes}
|
||||
while stack:
|
||||
w = stack.pop()
|
||||
for v in preds[w]:
|
||||
delta[v] += (sigma[v] / sigma[w]) * (1.0 + delta[w])
|
||||
if w != s:
|
||||
bc[w] += delta[w]
|
||||
n = len(nodes)
|
||||
# Undirected: every pair counted twice
|
||||
scale = 0.5
|
||||
if normalized and n > 2:
|
||||
scale = 1.0 / ((n - 1) * (n - 2))
|
||||
return {k: v * scale for k, v in bc.items()}
|
||||
|
||||
|
||||
# --- Betweenness cache -----------------------------------------------------
|
||||
# Betweenness is the ONLY expensive step in this module: O(V*E), ~42s on a
|
||||
# 5 000-page vault, versus 0.13s to parse it and 0.03s for communities. So it
|
||||
# is the only thing worth caching.
|
||||
#
|
||||
# The cache key is a hash of the GRAPH ITSELF, not file mtimes. That makes a
|
||||
# stale hit impossible by construction: if any link changed, the key changes;
|
||||
# if only prose changed, the old (still-correct) result is reused. No
|
||||
# invalidation logic to get wrong.
|
||||
|
||||
CACHE_FILENAME = ".graph-cache.json"
|
||||
_CACHE_MAX_ENTRIES = 3
|
||||
#: Don't write a cache file for graphs that recompute faster than this — small
|
||||
#: vaults should not accumulate cache files for no gain.
|
||||
_CACHE_MIN_SECONDS = 0.5
|
||||
|
||||
|
||||
def graph_fingerprint(outgoing: dict[str, list[str]]) -> str:
|
||||
"""Stable hash of the undirected graph topology (nodes + edges)."""
|
||||
adj = _undirected_adj(outgoing)
|
||||
h = hashlib.sha256()
|
||||
for node in sorted(adj):
|
||||
h.update(node.encode("utf-8"))
|
||||
h.update(b"\x00")
|
||||
for nb in sorted(adj[node]):
|
||||
h.update(nb.encode("utf-8"))
|
||||
h.update(b"\x01")
|
||||
h.update(b"\x02")
|
||||
return h.hexdigest()[:32]
|
||||
|
||||
|
||||
def _cache_path(vault: Path) -> Path:
|
||||
return vault / CACHE_FILENAME
|
||||
|
||||
|
||||
def _read_cache(vault: Path) -> dict:
|
||||
"""Read the cache, failing open (a corrupt or unreadable cache is ignored)."""
|
||||
try:
|
||||
data = json.loads(_cache_path(vault).read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError, ValueError):
|
||||
return {}
|
||||
return data if isinstance(data, dict) and data.get("version") == 1 else {}
|
||||
|
||||
|
||||
def _write_cache(vault: Path, data: dict) -> None:
|
||||
"""Atomically replace the cache file. Never raises — caching is best-effort."""
|
||||
target = _cache_path(vault)
|
||||
tmp = target.with_suffix(f".tmp{os.getpid()}")
|
||||
try:
|
||||
tmp.write_text(json.dumps(data), encoding="utf-8")
|
||||
os.replace(tmp, target) # atomic; concurrent writers can't tear the file
|
||||
except OSError:
|
||||
try:
|
||||
tmp.unlink()
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def betweenness_cached(
|
||||
outgoing: dict[str, list[str]],
|
||||
vault: Path | None = None,
|
||||
normalized: bool = True,
|
||||
) -> dict[str, float]:
|
||||
"""`betweenness_centrality`, memoised on disk under `vault` by graph topology.
|
||||
|
||||
Falls back to a plain in-process computation when `vault` is None or the
|
||||
cache is unusable — the result is always identical either way.
|
||||
"""
|
||||
if vault is None:
|
||||
return betweenness_centrality(outgoing, normalized=normalized)
|
||||
|
||||
key = f"{graph_fingerprint(outgoing)}:{int(normalized)}"
|
||||
cache = _read_cache(vault)
|
||||
hit = cache.get("betweenness", {}).get(key)
|
||||
if isinstance(hit, dict) and hit.keys() >= outgoing.keys():
|
||||
return {n: float(hit[n]) for n in outgoing}
|
||||
|
||||
started = time.monotonic()
|
||||
scores = betweenness_centrality(outgoing, normalized=normalized)
|
||||
elapsed = time.monotonic() - started
|
||||
|
||||
if elapsed >= _CACHE_MIN_SECONDS:
|
||||
entries = cache.get("betweenness", {})
|
||||
if not isinstance(entries, dict):
|
||||
entries = {}
|
||||
entries[key] = {n: round(v, 10) for n, v in scores.items()}
|
||||
# Keep the cache bounded — most recent keys win.
|
||||
for old in list(entries)[:-_CACHE_MAX_ENTRIES]:
|
||||
entries.pop(old, None)
|
||||
_write_cache(vault, {"version": 1, "betweenness": entries})
|
||||
return scores
|
||||
|
||||
|
||||
def bridge_pages(
|
||||
outgoing: dict[str, list[str]],
|
||||
communities: list[set[str]],
|
||||
top_n: int = 10,
|
||||
vault: Path | None = None,
|
||||
) -> list[dict]:
|
||||
"""Pages with the highest betweenness — removing them fragments the graph.
|
||||
|
||||
Each entry lists which communities the page's neighbours belong to, so the
|
||||
caller can say "`P` bridges cluster A ↔ cluster B".
|
||||
"""
|
||||
bc = betweenness_cached(outgoing, vault)
|
||||
if not bc:
|
||||
return []
|
||||
adj = _undirected_adj(outgoing)
|
||||
node_comm = _node_community_map(communities)
|
||||
ranked = sorted((n for n, s in bc.items() if s > 0), key=lambda n: -bc[n])[:top_n]
|
||||
out = []
|
||||
for n in ranked:
|
||||
own = node_comm.get(n)
|
||||
others = sorted({node_comm[nb] for nb in adj[n] if nb in node_comm and node_comm[nb] != own})
|
||||
out.append({
|
||||
"page": n,
|
||||
"betweenness": round(bc[n], 4),
|
||||
"community": own,
|
||||
"connects": others,
|
||||
})
|
||||
return out
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Surprising connections
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def surprising_connections(
|
||||
outgoing: dict[str, list[str]],
|
||||
communities: list[set[str]],
|
||||
top_n: int = 20,
|
||||
) -> list[dict]:
|
||||
"""Edges that cross community boundaries, ranked by unexpectedness.
|
||||
|
||||
Score = 1 / sqrt(cross_degree(source) * cross_degree(target))
|
||||
Low cross-degree nodes connected across communities are the most surprising.
|
||||
|
||||
Results are ordered so that each (community A, community B) boundary is
|
||||
represented once before any boundary repeats — graphify's dedup, which
|
||||
stops one high-betweenness hub from filling every slot.
|
||||
"""
|
||||
node_comm = _node_community_map(communities)
|
||||
|
||||
# Count how many cross-community edges each node already has
|
||||
cross_deg: dict[str, int] = defaultdict(int)
|
||||
for src, targets in outgoing.items():
|
||||
for t in targets:
|
||||
if node_comm.get(src) != node_comm.get(t):
|
||||
cross_deg[src] += 1
|
||||
cross_deg[t] += 1
|
||||
|
||||
results = []
|
||||
seen: set[tuple[str, str]] = set()
|
||||
for src, targets in outgoing.items():
|
||||
for t in targets:
|
||||
pair = tuple(sorted((src, t)))
|
||||
if pair in seen:
|
||||
continue
|
||||
if node_comm.get(src) != node_comm.get(t):
|
||||
cd_s = cross_deg.get(src, 1)
|
||||
cd_t = cross_deg.get(t, 1)
|
||||
score = 1.0 / math.sqrt(cd_s * cd_t)
|
||||
cs, ct = node_comm.get(src), node_comm.get(t)
|
||||
results.append({
|
||||
"source": src, "target": t, "score": round(score, 4),
|
||||
"note": f"bridges community {cs} -> community {ct}",
|
||||
"_pair": (min(cs, ct), max(cs, ct)) if cs is not None and ct is not None else None,
|
||||
})
|
||||
seen.add(pair)
|
||||
|
||||
results.sort(key=lambda x: -x["score"])
|
||||
# One representative per community pair first, then the rest by score.
|
||||
seen_pairs: set = set()
|
||||
first: list[dict] = []
|
||||
rest: list[dict] = []
|
||||
for r in results:
|
||||
pair = r.pop("_pair")
|
||||
if pair not in seen_pairs:
|
||||
seen_pairs.add(pair)
|
||||
first.append(r)
|
||||
else:
|
||||
rest.append(r)
|
||||
return (first + rest)[:top_n]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Path finding & neighbourhoods (graphify: `path`, `affected` blast radius)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _node_for(outgoing: dict[str, list[str]], term: str) -> str:
|
||||
"""Node id for a page reference: an exact id, else the shortest id whose
|
||||
path ends with it. Pages that share a name have path-qualified ids, so a
|
||||
bare name has to be matched by suffix."""
|
||||
term = "/".join(_slug(part) for part in term.strip().removesuffix(".md").split("/") if part)
|
||||
if term in outgoing:
|
||||
return term
|
||||
matches = sorted((n for n in outgoing if n.endswith(f"/{term}")), key=lambda n: (n.count("/"), n))
|
||||
return matches[0] if matches else term
|
||||
|
||||
|
||||
def shortest_path(
|
||||
outgoing: dict[str, list[str]],
|
||||
source: str,
|
||||
target: str,
|
||||
directed: bool = False,
|
||||
) -> list[str] | None:
|
||||
"""BFS shortest path between two pages. Follows links in either direction
|
||||
unless `directed=True`. Returns the node list, or None if unreachable."""
|
||||
source, target = _node_for(outgoing, source), _node_for(outgoing, target)
|
||||
if source not in outgoing or target not in outgoing:
|
||||
return None
|
||||
if source == target:
|
||||
return [source]
|
||||
if directed:
|
||||
adj: dict[str, Any] = {n: set(v) for n, v in outgoing.items()}
|
||||
else:
|
||||
adj = _undirected_adj(outgoing)
|
||||
prev: dict[str, str | None] = {source: None}
|
||||
queue = [source]
|
||||
qi = 0
|
||||
while qi < len(queue):
|
||||
v = queue[qi]
|
||||
qi += 1
|
||||
for w in sorted(adj.get(v, ())):
|
||||
if w in prev:
|
||||
continue
|
||||
prev[w] = v
|
||||
if w == target:
|
||||
path = [w]
|
||||
while prev[path[-1]] is not None:
|
||||
path.append(prev[path[-1]]) # type: ignore[arg-type]
|
||||
return path[::-1]
|
||||
queue.append(w)
|
||||
return None
|
||||
|
||||
|
||||
def neighborhood(
|
||||
outgoing: dict[str, list[str]],
|
||||
seed: str,
|
||||
depth: int = 2,
|
||||
direction: str = "both",
|
||||
) -> list[dict]:
|
||||
"""Breadth-first neighbourhood of a page up to `depth` hops.
|
||||
|
||||
direction: "out" (pages this one links to), "in" (pages linking here —
|
||||
the blast radius if this page is renamed/removed), or "both".
|
||||
Returns [{"page", "depth", "via"}] ordered by depth then name.
|
||||
"""
|
||||
seed = _node_for(outgoing, seed)
|
||||
if seed not in outgoing:
|
||||
return []
|
||||
incoming: dict[str, set[str]] = defaultdict(set)
|
||||
for s, targets in outgoing.items():
|
||||
for t in targets:
|
||||
incoming[t].add(s)
|
||||
|
||||
def nbrs(n: str) -> set[str]:
|
||||
out: set[str] = set()
|
||||
if direction in ("out", "both"):
|
||||
out |= set(outgoing.get(n, ()))
|
||||
if direction in ("in", "both"):
|
||||
out |= incoming.get(n, set())
|
||||
out.discard(n)
|
||||
return out
|
||||
|
||||
seen = {seed: 0}
|
||||
via = {seed: None}
|
||||
frontier = [seed]
|
||||
for d in range(1, depth + 1):
|
||||
nxt = []
|
||||
for v in frontier:
|
||||
for w in sorted(nbrs(v)):
|
||||
if w not in seen:
|
||||
seen[w] = d
|
||||
via[w] = v
|
||||
nxt.append(w)
|
||||
frontier = nxt
|
||||
if not frontier:
|
||||
break
|
||||
return [
|
||||
{"page": n, "depth": d, "via": via[n]}
|
||||
for n, d in sorted(seen.items(), key=lambda kv: (kv[1], kv[0]))
|
||||
if n != seed
|
||||
]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Graph diff (graphify: analyze.graph_diff)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
_SNAPSHOT_RE = re.compile(r"<!--\s*GRAPH_SNAPSHOT:\s*(\{.*?\})\s*-->", re.DOTALL)
|
||||
|
||||
|
||||
def snapshot(outgoing: dict[str, list[str]]) -> dict[str, list]:
|
||||
"""Compact, JSON-serialisable snapshot for embedding in `_insights.md`."""
|
||||
edges = sorted({(s, t) for s, ts in outgoing.items() for t in ts})
|
||||
return {"nodes": sorted(outgoing), "edges": [list(e) for e in edges]}
|
||||
|
||||
|
||||
def load_snapshot(path: Path) -> dict[str, list] | None:
|
||||
"""Read a snapshot from a JSON file or from the `<!-- GRAPH_SNAPSHOT: ... -->`
|
||||
comment inside a previous `_insights.md`. Returns None if not found."""
|
||||
try:
|
||||
text = path.read_text(encoding="utf-8", errors="replace")
|
||||
except OSError:
|
||||
return None
|
||||
m = _SNAPSHOT_RE.search(text)
|
||||
raw = m.group(1) if m else text
|
||||
try:
|
||||
data = json.loads(raw)
|
||||
except json.JSONDecodeError:
|
||||
return None
|
||||
if not isinstance(data, dict) or "nodes" not in data:
|
||||
return None
|
||||
return {"nodes": list(data.get("nodes", [])), "edges": [list(e) for e in data.get("edges", [])]}
|
||||
|
||||
|
||||
def graph_diff(old: dict[str, list], new: dict[str, list]) -> dict[str, Any]:
|
||||
"""What changed between two snapshots (see `snapshot`)."""
|
||||
old_nodes, new_nodes = set(old.get("nodes", [])), set(new.get("nodes", []))
|
||||
old_edges = {tuple(e) for e in old.get("edges", [])}
|
||||
new_edges = {tuple(e) for e in new.get("edges", [])}
|
||||
|
||||
def in_targets(edges: set) -> set[str]:
|
||||
return {t for _, t in edges}
|
||||
|
||||
old_in, new_in = in_targets(old_edges), in_targets(new_edges)
|
||||
added_edges = sorted(new_edges - old_edges)
|
||||
removed_edges = sorted(old_edges - new_edges)
|
||||
return {
|
||||
"added_pages": sorted(new_nodes - old_nodes),
|
||||
"removed_pages": sorted(old_nodes - new_nodes),
|
||||
"added_edges": [list(e) for e in added_edges],
|
||||
"removed_edges": [list(e) for e in removed_edges],
|
||||
"newly_connected": sorted((new_in - old_in) & old_nodes),
|
||||
"lost_incoming": sorted((old_in - new_in) & new_nodes),
|
||||
"summary": {
|
||||
"pages": len(new_nodes) - len(old_nodes),
|
||||
"edges": len(new_edges) - len(old_edges),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Suggested questions (graphify: analyze.suggest_questions)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def suggest_questions(
|
||||
outgoing: dict[str, list[str]],
|
||||
communities: list[set[str]],
|
||||
labels: dict[int, str],
|
||||
bridges: list[dict],
|
||||
top_n: int = 7,
|
||||
) -> list[dict]:
|
||||
"""Questions the graph structure is uniquely positioned to raise.
|
||||
|
||||
Priority: bridge pages → dead-end hubs → isolated pages → low-cohesion
|
||||
communities. Each has `type`, `question`, `why`.
|
||||
"""
|
||||
questions: list[dict] = []
|
||||
degree, in_deg, out_deg = _build_degree_tables(outgoing)
|
||||
|
||||
# 1. Bridge pages -> cross-cutting questions
|
||||
for b in bridges[:3]:
|
||||
if not b["connects"]:
|
||||
continue
|
||||
own = labels.get(b["community"], f"community {b['community']}")
|
||||
others = ", ".join(f"`{labels.get(c, f'community {c}')}`" for c in b["connects"])
|
||||
questions.append({
|
||||
"type": "bridge_node",
|
||||
"question": f"Why does `{b['page']}` connect `{own}` to {others}?",
|
||||
"why": f"High betweenness ({b['betweenness']}) — this page is a cross-community bridge.",
|
||||
})
|
||||
|
||||
# 2. Sink hubs: many incoming, no outgoing
|
||||
sinks = sorted(
|
||||
(n for n in outgoing if out_deg.get(n, 0) == 0 and in_deg.get(n, 0) >= 3),
|
||||
key=lambda n: -in_deg[n],
|
||||
)[:2]
|
||||
for n in sinks:
|
||||
questions.append({
|
||||
"type": "sink_hub",
|
||||
"question": f"What should `{n}` link out to?",
|
||||
"why": f"{in_deg[n]} pages point at it but it links nowhere — a dead end in a busy area.",
|
||||
})
|
||||
|
||||
# 3. Isolated / weakly connected pages
|
||||
weak = sorted(n for n in outgoing if degree.get(n, 0) <= 1)
|
||||
if weak:
|
||||
shown = ", ".join(f"`{n}`" for n in weak[:3])
|
||||
questions.append({
|
||||
"type": "isolated_nodes",
|
||||
"question": f"What connects {shown} to the rest of the vault?",
|
||||
"why": f"{len(weak)} weakly-connected pages — possible gaps or missing links.",
|
||||
})
|
||||
|
||||
# 4. Low-cohesion communities
|
||||
for i, comm in enumerate(communities):
|
||||
if len(comm) >= 5:
|
||||
c = cohesion_score(outgoing, comm)
|
||||
if c < 0.15:
|
||||
questions.append({
|
||||
"type": "low_cohesion",
|
||||
"question": f"Should `{labels.get(i, f'community {i}')}` be split into more focused topics?",
|
||||
"why": f"Cohesion {c} across {len(comm)} pages — they share a cluster but are weakly interlinked.",
|
||||
})
|
||||
|
||||
if not questions:
|
||||
return [{
|
||||
"type": "no_signal",
|
||||
"question": None,
|
||||
"why": "Not enough structure to raise questions — no bridges, sink hubs, isolates or fragmented clusters.",
|
||||
}]
|
||||
return questions[:top_n]
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Community labelling (heuristic: most common tag or longest shared prefix)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def _label_community(
|
||||
pages: list[str],
|
||||
tags_map: dict[str, list[str]],
|
||||
taken: set[str] | None = None,
|
||||
) -> str:
|
||||
"""Most common tag; if that label is already `taken` by a bigger community,
|
||||
qualify it with the next most common tag (`ai-security/roadmap`) or a
|
||||
top page name so every community label is unique."""
|
||||
freq: dict[str, int] = defaultdict(int)
|
||||
for p in pages:
|
||||
for tag in tags_map.get(p, []):
|
||||
if not tag.startswith("visibility/"):
|
||||
freq[tag] += 1
|
||||
if freq:
|
||||
ranked = sorted(freq, key=lambda t: (-freq[t], t))
|
||||
label = ranked[0]
|
||||
if taken is not None and label in taken:
|
||||
for alt in ranked[1:]:
|
||||
cand = f"{label}/{alt}"
|
||||
if cand not in taken:
|
||||
return cand
|
||||
return f"{label}/{pages[0]}"
|
||||
return label
|
||||
# Fallback: shared prefix of page names
|
||||
if len(pages) == 1:
|
||||
return pages[0]
|
||||
prefix = pages[0]
|
||||
for p in pages[1:]:
|
||||
while not p.startswith(prefix):
|
||||
prefix = prefix[:-1]
|
||||
if not prefix:
|
||||
break
|
||||
return prefix.rstrip("-_") or f"cluster-{pages[0][:20]}"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Main entry point
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def analyse_vault(
|
||||
vault: Path,
|
||||
top_n: int = 20,
|
||||
previous_snapshot: dict[str, list] | None = None,
|
||||
include_snapshot: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
outgoing, tags_map = parse_vault_graph(vault)
|
||||
|
||||
if not outgoing:
|
||||
return {
|
||||
"god_nodes": [], "bridges": [], "communities": [],
|
||||
"surprising_connections": [], "suggested_questions": [],
|
||||
"dead_ends": [], "isolated": [],
|
||||
"stats": {"pages": 0, "edges": 0, "communities": 0, "density": 0.0},
|
||||
}
|
||||
|
||||
communities_raw = detect_communities(outgoing)
|
||||
# Sort communities largest-first
|
||||
communities_raw.sort(key=lambda c: -len(c))
|
||||
labels: dict[int, str] = {}
|
||||
taken: set[str] = set()
|
||||
for i, c in enumerate(communities_raw):
|
||||
lbl = _label_community(sorted(c), tags_map, taken)
|
||||
labels[i] = lbl
|
||||
taken.add(lbl)
|
||||
|
||||
total_edges = sum(len(v) for v in outgoing.values())
|
||||
n_pages = len(outgoing)
|
||||
undirected_edges = sum(len(v) for v in _undirected_adj(outgoing).values()) // 2
|
||||
density = undirected_edges / (n_pages * (n_pages - 1) / 2) if n_pages > 1 else 0.0
|
||||
|
||||
bridges = bridge_pages(outgoing, communities_raw, top_n=min(top_n, 10), vault=vault)
|
||||
|
||||
result: dict[str, Any] = {
|
||||
"god_nodes": god_nodes(outgoing, top_n),
|
||||
"bridges": bridges,
|
||||
"communities": [
|
||||
{
|
||||
"id": i,
|
||||
"size": len(comm),
|
||||
"pages": sorted(comm),
|
||||
"label": labels[i],
|
||||
"cohesion": cohesion_score(outgoing, comm),
|
||||
}
|
||||
for i, comm in enumerate(communities_raw)
|
||||
],
|
||||
"surprising_connections": surprising_connections(outgoing, communities_raw, top_n),
|
||||
"suggested_questions": suggest_questions(outgoing, communities_raw, labels, bridges),
|
||||
"dead_ends": dead_ends(outgoing),
|
||||
"isolated": isolated(outgoing),
|
||||
"stats": {
|
||||
"pages": n_pages,
|
||||
"edges": total_edges,
|
||||
"communities": len(communities_raw),
|
||||
"density": round(density, 4),
|
||||
},
|
||||
}
|
||||
if previous_snapshot is not None:
|
||||
result["diff"] = graph_diff(previous_snapshot, snapshot(outgoing))
|
||||
if include_snapshot:
|
||||
result["snapshot"] = snapshot(outgoing)
|
||||
return result
|
||||
@@ -0,0 +1,160 @@
|
||||
"""One link resolver for every module that builds a graph or checks links.
|
||||
|
||||
`graph_analysis`, `graphrag`, and `lint` each identified a page by its slugged
|
||||
filename stem and resolved a link by its last path segment. Two pages with the
|
||||
same name — `projects/a/index.md` and `projects/b/index.md`, or the same
|
||||
concept in two project sub-wikis — therefore became ONE graph node, and a
|
||||
project-relative link such as `[[concepts/plan]]` written inside `projects/a/`
|
||||
was reported missing. This module resolves links the way Obsidian does and
|
||||
gives colliding pages distinct ids, while a vault without collisions keeps
|
||||
exactly the ids it always had (the slugged stem).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import posixpath
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from typing import Iterable, Optional
|
||||
|
||||
#: Obsidian does not treat a `[[link]]` inside inline code or a fenced block as
|
||||
#: a link, so neither does anything that walks the link graph.
|
||||
_FENCE_RE = re.compile(r"^(```|~~~).*?^\1[^\n]*$", re.MULTILINE | re.DOTALL)
|
||||
# Single-line on purpose: a stray backtick must not swallow the rest of a page.
|
||||
_INLINE_CODE_RE = re.compile(r"(`+)(?!`).+?(?<!`)\1(?!`)")
|
||||
|
||||
|
||||
def strip_code(text: str) -> str:
|
||||
"""`text` with fenced code blocks and inline code spans removed."""
|
||||
return _INLINE_CODE_RE.sub("", _FENCE_RE.sub("", text))
|
||||
|
||||
|
||||
def slug(name: str) -> str:
|
||||
"""Normalise one page name / path segment the way page ids are keyed."""
|
||||
return name.strip().lower().replace(" ", "-")
|
||||
|
||||
|
||||
def _key(path: str) -> str:
|
||||
"""Slugged vault-relative path without `.md`: `Concepts/My Page.md` -> `concepts/my-page`."""
|
||||
path = path.strip().rstrip("\\")
|
||||
if path.lower().endswith(".md"):
|
||||
path = path[:-3]
|
||||
return "/".join(slug(part) for part in path.strip("/").split("/") if part and part != ".")
|
||||
|
||||
|
||||
def _shared_dirs(a: str, b: str) -> int:
|
||||
"""How many leading directories two keys have in common."""
|
||||
count = 0
|
||||
for left, right in zip(a.split("/")[:-1], b.split("/")[:-1]):
|
||||
if left != right:
|
||||
break
|
||||
count += 1
|
||||
return count
|
||||
|
||||
|
||||
class PageIndex:
|
||||
"""Page ids and Obsidian-style link resolution over one set of pages.
|
||||
|
||||
A page's id is its slugged stem when no other page shares that stem, and
|
||||
its slugged vault-relative path otherwise, so ids stay stable for any vault
|
||||
without name collisions.
|
||||
"""
|
||||
|
||||
def __init__(self, rel_paths: Iterable[str]) -> None:
|
||||
self._keys: dict[str, str] = {}
|
||||
self._by_stem: dict[str, list[str]] = defaultdict(list)
|
||||
# Files whose paths normalise to the same key (`Foo Bar.md` and
|
||||
# `foo-bar.md`) share one id, as they always have; a link to that key
|
||||
# is ambiguous because nothing distinguishes them.
|
||||
self._key_files: dict[str, int] = defaultdict(int)
|
||||
for rel in rel_paths:
|
||||
key = _key(rel)
|
||||
if not key:
|
||||
continue
|
||||
self._key_files[key] += 1
|
||||
if key not in self._keys:
|
||||
self._keys[key] = rel
|
||||
self._by_stem[key.rsplit("/", 1)[-1]].append(key)
|
||||
self._ids = {
|
||||
key: key.rsplit("/", 1)[-1] if len(self._by_stem[key.rsplit("/", 1)[-1]]) == 1 else key
|
||||
for key in self._keys
|
||||
}
|
||||
|
||||
def ids(self) -> list[str]:
|
||||
return list(self._ids.values())
|
||||
|
||||
def id_for(self, rel_path: str) -> str:
|
||||
return self._ids[_key(rel_path)]
|
||||
|
||||
def path_for(self, page_id: str) -> Optional[str]:
|
||||
"""Vault-relative path of an id (or of any unambiguous key)."""
|
||||
for key, value in self._ids.items():
|
||||
if value == page_id:
|
||||
return self._keys[key]
|
||||
return None
|
||||
|
||||
def _nearest(self, candidates: list[str], source_key: str) -> str:
|
||||
"""The candidate Obsidian would pick from `source_key`: the one sharing
|
||||
the most folders with the linking page, then the shortest path."""
|
||||
return min(
|
||||
candidates,
|
||||
key=lambda key: (-_shared_dirs(key, source_key) if source_key else 0, key.count("/"), key),
|
||||
)
|
||||
|
||||
def _candidates(self, target: str, source: Optional[str]) -> list:
|
||||
"""Keys a link could mean, in resolution order: one key when the link
|
||||
is exact, several when it is a suffix or bare name shared by pages."""
|
||||
raw = target.strip().rstrip("\\")
|
||||
source_key = _key(source) if source else ""
|
||||
source_dir = source_key.rsplit("/", 1)[0] if "/" in source_key else ""
|
||||
if raw.startswith(("./", "../")):
|
||||
joined = posixpath.normpath(posixpath.join(source_dir, raw))
|
||||
key = "" if joined.startswith("..") else _key(joined)
|
||||
return [key] if key in self._keys else []
|
||||
key = _key(raw)
|
||||
if not key:
|
||||
return []
|
||||
if "/" in key:
|
||||
if key in self._keys:
|
||||
return [key]
|
||||
local = f"{source_dir}/{key}" if source_dir else key
|
||||
if local in self._keys:
|
||||
return [local]
|
||||
suffixed = [k for k in self._keys if k.endswith(f"/{key}")]
|
||||
if suffixed:
|
||||
return suffixed
|
||||
return list(self._by_stem.get(key.rsplit("/", 1)[-1], []))
|
||||
|
||||
def resolve(self, target: str, source: Optional[str] = None) -> Optional[str]:
|
||||
"""Id of the page a link `target` points at from page `source`, or None.
|
||||
|
||||
`target` is the bare link target (no `[[`, alias, or heading anchor).
|
||||
Order: `./` / `../` relative to the linking file, an exact vault path,
|
||||
a path relative to the linking file's folder, a path suffix (the
|
||||
nearest when several match), and finally the bare name (the nearest
|
||||
page with that stem) so an over-qualified link still resolves.
|
||||
"""
|
||||
candidates = self._candidates(target, source)
|
||||
if not candidates:
|
||||
return None
|
||||
return self._ids[self._nearest(candidates, _key(source) if source else "")]
|
||||
|
||||
def is_ambiguous(self, target: str, source: Optional[str] = None) -> bool:
|
||||
"""True when several pages match and none is nearer to `source` than
|
||||
the rest, so the link's meaning depends on a tie-break."""
|
||||
candidates = self._candidates(target, source)
|
||||
if len(candidates) == 1:
|
||||
return self._key_files[candidates[0]] > 1
|
||||
if len(candidates) < 2:
|
||||
return False
|
||||
source_key = _key(source) if source else ""
|
||||
scores = sorted((_shared_dirs(key, source_key) for key in candidates), reverse=True)
|
||||
return scores[0] == scores[1]
|
||||
|
||||
def match(self, term: str) -> Optional[str]:
|
||||
"""Id for a free-text page reference (a CLI argument): an id, a path, or
|
||||
a bare name. A bare name shared by several pages picks the shortest path."""
|
||||
key = _key(term)
|
||||
if key in self._ids.values():
|
||||
return key
|
||||
return self.resolve(term)
|
||||
@@ -0,0 +1,720 @@
|
||||
"""Vault lint checks for wiki structure and metadata hygiene."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from collections.abc import Collection
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from obsidian_wiki.cache import _iter_entries, _load_manifest
|
||||
from obsidian_wiki.graph_analysis import _page_slug as graph_page_slug
|
||||
from obsidian_wiki.graph_analysis import iter_pages as iter_graph_pages
|
||||
from obsidian_wiki.links import PageIndex, strip_code
|
||||
from obsidian_wiki.provenance import (
|
||||
archive_wikilink_relpath,
|
||||
clip_url_index,
|
||||
expected_snapshots_for_page,
|
||||
invert_pages,
|
||||
parse_snapshots_field,
|
||||
)
|
||||
from obsidian_wiki.vault import SKIP_DIRS as VAULT_SKIP_DIRS
|
||||
from obsidian_wiki.vault import iter_instruction_files, iter_md, split_frontmatter
|
||||
from obsidian_wiki.temporal import (
|
||||
SUPERSEDED_FIELD,
|
||||
superseded_target,
|
||||
validity_window,
|
||||
)
|
||||
from obsidian_wiki.trust import (
|
||||
ALLOWED_LIFECYCLES,
|
||||
TRUST_LEDGER_RELATIVE_PATH,
|
||||
check_lifecycle_transitions,
|
||||
check_trust_ledger,
|
||||
validate_trust_metadata,
|
||||
)
|
||||
|
||||
SKIP_DIRS = VAULT_SKIP_DIRS | {"_bootstrap"}
|
||||
REQUIRED_FRONTMATTER = (
|
||||
"title",
|
||||
"category",
|
||||
"tags",
|
||||
"sources",
|
||||
"created",
|
||||
"updated",
|
||||
)
|
||||
# Introduced by the trust-ledger rollout (#28, #132). Legacy pages that predate
|
||||
# the schema are missing these by construction; enforcement is staged behind
|
||||
# lint_vault's strict_trust switch so upgrading obsidian-wiki doesn't fail-close
|
||||
# every pre-existing page until a vault owner explicitly opts into strict mode
|
||||
# after a backfill/review pass.
|
||||
TRUST_REQUIRED_FRONTMATTER = (
|
||||
"base_confidence",
|
||||
"lifecycle",
|
||||
)
|
||||
RESERVED_PAGE_STEMS = frozenset({"index", "log", "hot", "_insights"})
|
||||
ALLOWED_RELATIONSHIP_TYPES = frozenset(
|
||||
{"extends", "implements", "contradicts", "derived_from", "uses", "replaces", "related_to"}
|
||||
)
|
||||
|
||||
_FIELD_RE = re.compile(r"^([A-Za-z_][\w-]*):", re.MULTILINE)
|
||||
_WIKILINK_RE = re.compile(r"\[\[([^\]|#]+?)(?:[|#][^\]]*?)?\]\]")
|
||||
# Local `.md` targets only: skip URLs (any `scheme:`) and require `.md` to end
|
||||
# the path, before an optional `#anchor` or `"title"`.
|
||||
_MD_LINK_RE = re.compile(
|
||||
r"\[.*?\]\((?![A-Za-z][A-Za-z0-9+.-]*:)([^)#\s]+\.md)(?:[#\s][^)]*)?\)"
|
||||
)
|
||||
_RELATIONSHIP_LIST_FIELD_RE = re.compile(
|
||||
r"^\s*-\s*(type|target):\s*(.*?)\s*$"
|
||||
)
|
||||
_RELATIONSHIP_ITEM_START_RE = re.compile(r"^\s*-\s*(?:#.*)?$")
|
||||
_RELATIONSHIP_FIELD_RE = re.compile(r"^\s+(type|target):\s*(.*?)\s*$")
|
||||
_WINDOWS_ABS_RE = re.compile(r"^[A-Za-z]:[\\/]")
|
||||
# Freshness: a count or amount with no date beside it reads as true forever
|
||||
# and silently rots ("23 stars", "the pipeline has 13 deals"). Every such fact
|
||||
# should be stamped `(as of YYYY-MM-DD)`, live in a dated note or under a
|
||||
# dated heading, or point at where the truth lives.
|
||||
# ponytail: fixed noun list tuned for precision; widen it if real vaults miss claims.
|
||||
_UNDATED_CLAIM_RE = re.compile(
|
||||
r"\b\d+(?:[.,]\d+)*\s*[km]?\+?\s*(?:★|\s(?:open\s+|active\s+|paying\s+)?"
|
||||
r"(?:users|customers|clients|deals|stars|subscribers|downloads|members|employees|"
|
||||
r"tickets|PRs|pull requests|followers|signups|installs)\b)",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_UNDATED_MONEY_RE = re.compile(
|
||||
r"\b(?:MRR|ARR|revenue|balance|runway)\b[^.\n]{0,20}?\b(?:is|of|at)\s+[$€£]?\s?\d",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
_DATE_HINT_RE = re.compile(r"\b(?:19|20)\d{2}\b|\bas of\b", re.IGNORECASE)
|
||||
_HEADING_RE = re.compile(r"^(#{1,6})\s")
|
||||
_FENCE_LINE_RE = re.compile(r"^\s*(```|~~~)")
|
||||
_INLINE_CODE_SPAN_RE = re.compile(r"`[^`\n]*`")
|
||||
_DATED_STEM_RE = re.compile(r"^(?:19|20)\d{2}-\d{2}")
|
||||
|
||||
|
||||
def _frontmatter_field_block(frontmatter: str, field: str) -> str:
|
||||
"""The raw value of a frontmatter *field*, including a following block list.
|
||||
|
||||
Returns the inline value plus any subsequent ``- item`` lines indented under
|
||||
it; an empty string when the field is absent.
|
||||
"""
|
||||
lines = frontmatter.splitlines()
|
||||
for index, line in enumerate(lines):
|
||||
match = re.match(rf"^{re.escape(field)}\s*:(.*)$", line)
|
||||
if not match:
|
||||
continue
|
||||
block = [match.group(1)]
|
||||
for following in lines[index + 1:]:
|
||||
if re.match(r"^\s+-\s", following) or not following.strip():
|
||||
block.append(following)
|
||||
continue
|
||||
break
|
||||
return "\n".join(block)
|
||||
return ""
|
||||
|
||||
|
||||
def _absolute_source_entries(frontmatter: str) -> list[str]:
|
||||
"""`sources:` entries that are machine absolute paths (contract violation).
|
||||
|
||||
The source key contract bans a stored absolute path in page frontmatter just
|
||||
as it does in the manifest; ``~``-relative and vault-relative entries are
|
||||
fine. Only the ``sources:`` field is inspected — a path in prose may be a
|
||||
legitimate citation.
|
||||
"""
|
||||
raw = _frontmatter_field_block(frontmatter, "sources")
|
||||
if not raw.strip():
|
||||
return []
|
||||
lines = raw.splitlines()
|
||||
inline = lines[0].strip()
|
||||
entries: list[str] = []
|
||||
if inline.startswith("["):
|
||||
entries.extend(inline.strip("[]").split(","))
|
||||
elif inline:
|
||||
# A scalar `sources: <value>` is a single entry — not a block list, and
|
||||
# not a flow list, but still a stored source key.
|
||||
entries.append(inline)
|
||||
entries.extend(
|
||||
line.strip()[1:] for line in lines[1:] if line.strip().startswith("-")
|
||||
)
|
||||
bad: list[str] = []
|
||||
for entry in entries:
|
||||
value = entry.strip().strip("'\"").strip()
|
||||
if not value:
|
||||
continue
|
||||
if value.startswith("/") or _WINDOWS_ABS_RE.match(value):
|
||||
bad.append(value)
|
||||
return bad
|
||||
|
||||
|
||||
def _slug(text: str) -> str:
|
||||
return text.strip().lower().replace(" ", "-")
|
||||
|
||||
|
||||
def _parse_frontmatter_values(frontmatter: str) -> dict[str, str]:
|
||||
values: dict[str, str] = {}
|
||||
lines = frontmatter.splitlines()
|
||||
i = 0
|
||||
while i < len(lines):
|
||||
line = lines[i]
|
||||
if not line or ":" not in line or line.startswith((" ", "\t")):
|
||||
i += 1
|
||||
continue
|
||||
key, raw = line.split(":", 1)
|
||||
key = key.strip()
|
||||
value = raw.strip()
|
||||
if value in {">", ">-", "|", "|-"}:
|
||||
block: list[str] = []
|
||||
i += 1
|
||||
while i < len(lines):
|
||||
child = lines[i]
|
||||
if child.startswith(" ") or child.startswith("\t"):
|
||||
block.append(child.strip())
|
||||
i += 1
|
||||
continue
|
||||
break
|
||||
values[key] = " ".join(part for part in block if part).strip()
|
||||
continue
|
||||
values[key] = value.strip("'\"")
|
||||
i += 1
|
||||
return values
|
||||
|
||||
|
||||
def _relationship_scalar(raw: str) -> str:
|
||||
value = raw.strip()
|
||||
if len(value) >= 2 and value[0] == value[-1] and value[0] in {"'", '"'}:
|
||||
return value[1:-1].strip()
|
||||
return value.split(" #", 1)[0].strip()
|
||||
|
||||
|
||||
def _parse_relationships(frontmatter: str) -> list[dict[str, str]]:
|
||||
relationships: list[dict[str, str]] = []
|
||||
current: dict[str, str] | None = None
|
||||
in_relationships = False
|
||||
for line in frontmatter.splitlines():
|
||||
if line.startswith("relationships:") and not line.startswith((" ", "\t")):
|
||||
in_relationships = True
|
||||
inline = line.split(":", 1)[1].strip()
|
||||
if inline in {"[]", "null", "~"}:
|
||||
return []
|
||||
if inline:
|
||||
relationships.append({"parse_error": "inline_relationships_not_supported"})
|
||||
return relationships
|
||||
continue
|
||||
if in_relationships and line and not line.startswith((" ", "\t")):
|
||||
break
|
||||
if not in_relationships:
|
||||
continue
|
||||
item_match = _RELATIONSHIP_LIST_FIELD_RE.match(line)
|
||||
if item_match:
|
||||
if current is not None:
|
||||
relationships.append(current)
|
||||
current = {item_match.group(1): _relationship_scalar(item_match.group(2))}
|
||||
continue
|
||||
if _RELATIONSHIP_ITEM_START_RE.match(line):
|
||||
if current is not None:
|
||||
relationships.append(current)
|
||||
current = {}
|
||||
continue
|
||||
field_match = _RELATIONSHIP_FIELD_RE.match(line)
|
||||
if field_match and current is not None:
|
||||
key = field_match.group(1)
|
||||
if key in current:
|
||||
current["parse_error"] = f"duplicate_relationship_{key}"
|
||||
else:
|
||||
current[key] = _relationship_scalar(field_match.group(2))
|
||||
continue
|
||||
if line.strip() and not line.lstrip().startswith("#"):
|
||||
if current is None:
|
||||
current = {"parse_error": "malformed_relationship_entry"}
|
||||
else:
|
||||
current.setdefault("parse_error", "malformed_relationship_entry")
|
||||
if current is not None:
|
||||
relationships.append(current)
|
||||
return relationships
|
||||
|
||||
|
||||
def _display_target(raw: str) -> str:
|
||||
"""How a broken link is reported: the slugged last segment, sans `.md`."""
|
||||
name = raw.rstrip("\\").split("/")[-1]
|
||||
return _slug(name[:-3] if name.lower().endswith(".md") else name)
|
||||
|
||||
|
||||
def _relationship_link(raw: str) -> str:
|
||||
"""A `relationships:` target as a bare link target (no brackets, alias, anchor)."""
|
||||
target = raw.strip().strip("\"'").removeprefix("[[").removesuffix("]]")
|
||||
return target.split("|", 1)[0].split("#", 1)[0].strip()
|
||||
|
||||
|
||||
def _normalise_node_id(raw: str) -> str:
|
||||
target = raw.strip().removeprefix("[[").removesuffix("]]")
|
||||
target = target.split("|", 1)[0].split("#", 1)[0].strip()
|
||||
if target.lower().endswith(".md"):
|
||||
target = target[:-3]
|
||||
return "/".join(_slug(part) for part in target.strip("/").split("/") if part)
|
||||
|
||||
|
||||
# Extensions Obsidian embeds as attachments. Anything else after a dot is part
|
||||
# of the page name ("Node.js", "v1.2 release notes"), not a file extension.
|
||||
_ATTACHMENT_SUFFIXES = frozenset({
|
||||
".png", ".jpg", ".jpeg", ".gif", ".bmp", ".svg", ".webp", ".avif",
|
||||
".mp3", ".wav", ".m4a", ".ogg", ".flac", ".mp4", ".webm", ".mov", ".ogv",
|
||||
".pdf", ".canvas", ".base",
|
||||
})
|
||||
|
||||
|
||||
def _wikilink_page_target(raw: str) -> str | None:
|
||||
"""Normalise a `_WIKILINK_RE` capture to a page name, or None for an
|
||||
attachment embed `by_slug` (`.md`-only) can never hold. Also strips an
|
||||
explicit `.md` suffix and a table-escaped pipe's trailing backslash.
|
||||
"""
|
||||
name = raw.rstrip("\\").split("/")[-1]
|
||||
suffix = Path(name).suffix.lower()
|
||||
if suffix == ".md":
|
||||
return name[: -len(suffix)]
|
||||
if suffix in _ATTACHMENT_SUFFIXES:
|
||||
return None
|
||||
return name
|
||||
|
||||
|
||||
def _undated_claims(body: str, first_line: int) -> list[dict[str, Any]]:
|
||||
"""Lines claiming a current count or amount with no date to anchor it."""
|
||||
claims: list[dict[str, Any]] = []
|
||||
in_fence = False
|
||||
dated_level: int | None = None
|
||||
for offset, line in enumerate(body.splitlines()):
|
||||
if _FENCE_LINE_RE.match(line):
|
||||
in_fence = not in_fence
|
||||
continue
|
||||
if in_fence or line.lstrip().startswith(">"):
|
||||
continue
|
||||
heading = _HEADING_RE.match(line)
|
||||
if heading:
|
||||
level = len(heading.group(1))
|
||||
if _DATE_HINT_RE.search(line):
|
||||
dated_level = level if dated_level is None else min(dated_level, level)
|
||||
elif dated_level is not None and level <= dated_level:
|
||||
dated_level = None
|
||||
continue
|
||||
if dated_level is not None:
|
||||
continue
|
||||
prose = _INLINE_CODE_SPAN_RE.sub("", line)
|
||||
if _DATE_HINT_RE.search(prose):
|
||||
continue
|
||||
if _UNDATED_CLAIM_RE.search(prose) or _UNDATED_MONEY_RE.search(prose):
|
||||
claims.append({"line": first_line + offset, "text": line.strip()[:160]})
|
||||
return claims
|
||||
|
||||
|
||||
def _parse_page(path: Path, vault: Path) -> dict[str, Any]:
|
||||
text = path.read_text(encoding="utf-8", errors="replace")
|
||||
frontmatter, body = split_frontmatter(text)
|
||||
relative_parts = path.relative_to(vault).parts
|
||||
dated_page = relative_parts[0] == "journal" or bool(_DATED_STEM_RE.match(path.stem))
|
||||
undated = (
|
||||
[]
|
||||
if dated_page
|
||||
else _undated_claims(body, text[: len(text) - len(body)].count("\n") + 1)
|
||||
)
|
||||
fields = set(_FIELD_RE.findall(frontmatter))
|
||||
values = _parse_frontmatter_values(frontmatter)
|
||||
relative = path.relative_to(vault)
|
||||
|
||||
# Raw targets, resolved per page in `lint_vault` against the whole vault
|
||||
# (`PageIndex`), so a project-relative link resolves like Obsidian does.
|
||||
links: list[str] = []
|
||||
broken_archive_links: list[dict[str, str]] = []
|
||||
text = strip_code(text)
|
||||
for raw in _WIKILINK_RE.findall(text):
|
||||
archive_rel = archive_wikilink_relpath(vault, raw)
|
||||
if archive_rel is not None:
|
||||
if not (vault / archive_rel).is_file():
|
||||
broken_archive_links.append(
|
||||
{"page": relative.as_posix(), "target": archive_rel}
|
||||
)
|
||||
continue
|
||||
if _wikilink_page_target(raw) and raw.strip():
|
||||
links.append(raw.strip().rstrip("\\"))
|
||||
for href in _MD_LINK_RE.findall(text):
|
||||
archive_rel = archive_wikilink_relpath(vault, href)
|
||||
if archive_rel is not None:
|
||||
if not (vault / archive_rel).is_file():
|
||||
broken_archive_links.append(
|
||||
{"page": relative.as_posix(), "target": archive_rel}
|
||||
)
|
||||
continue
|
||||
links.append(href)
|
||||
|
||||
return {
|
||||
"path": relative.as_posix(),
|
||||
"node_id": _normalise_node_id(relative.with_suffix("").as_posix()),
|
||||
"slug": _slug(path.stem),
|
||||
"title": values.get("title", "").strip() or path.stem,
|
||||
"summary": values.get("summary", "").strip(),
|
||||
"fields": fields,
|
||||
"links": links,
|
||||
"relationships": _parse_relationships(frontmatter),
|
||||
"absolute_sources": _absolute_source_entries(frontmatter),
|
||||
"snapshots": parse_snapshots_field(_frontmatter_field_block(frontmatter, "snapshots")),
|
||||
"values": values,
|
||||
"archive_broken": broken_archive_links,
|
||||
"undated_claims": undated,
|
||||
}
|
||||
|
||||
|
||||
def lint_vault(
|
||||
vault: Path,
|
||||
*,
|
||||
require_trust_ledger: bool = True,
|
||||
strict_trust: bool = False,
|
||||
allowed_relationship_types: Collection[str] | None = None,
|
||||
allowed_lifecycles: Collection[str] | None = None,
|
||||
required_trust_fields: Collection[str] | None = None,
|
||||
schema_source: str = "framework-defaults",
|
||||
) -> dict[str, Any]:
|
||||
relationship_types = frozenset(
|
||||
ALLOWED_RELATIONSHIP_TYPES
|
||||
if allowed_relationship_types is None
|
||||
else allowed_relationship_types
|
||||
)
|
||||
lifecycles = frozenset(
|
||||
ALLOWED_LIFECYCLES if allowed_lifecycles is None else allowed_lifecycles
|
||||
)
|
||||
trust_fields = (
|
||||
tuple(required_trust_fields)
|
||||
if required_trust_fields is not None
|
||||
else TRUST_REQUIRED_FRONTMATTER
|
||||
)
|
||||
pages = [_parse_page(path, vault) for path in iter_md(vault, SKIP_DIRS)]
|
||||
page_index = PageIndex(
|
||||
[page["path"] for page in pages]
|
||||
+ [path.relative_to(vault).as_posix() for path in iter_instruction_files(vault, SKIP_DIRS)]
|
||||
)
|
||||
for page in pages:
|
||||
page["id"] = page_index.id_for(page["path"])
|
||||
page["resolved"] = [
|
||||
(raw, page_index.resolve(raw, page["path"])) for raw in page["links"]
|
||||
]
|
||||
incoming: dict[str, int] = defaultdict(int)
|
||||
|
||||
broken_links: list[dict[str, str]] = []
|
||||
for page in pages:
|
||||
for raw, target in page["resolved"]:
|
||||
if target is None:
|
||||
broken_links.append({"page": page["path"], "target": _display_target(raw)})
|
||||
elif target != page["id"]:
|
||||
incoming[target] += 1
|
||||
broken_links.extend(page.get("archive_broken", []))
|
||||
|
||||
missing_frontmatter = []
|
||||
confidence_missing_fields = []
|
||||
trust_metadata_errors = []
|
||||
for page in pages:
|
||||
if page["slug"] in RESERVED_PAGE_STEMS:
|
||||
continue
|
||||
missing = [field for field in REQUIRED_FRONTMATTER if field not in page["fields"]]
|
||||
if missing:
|
||||
missing_frontmatter.append({"page": page["path"], "missing": missing})
|
||||
missing_trust = [field for field in trust_fields if field not in page["fields"]]
|
||||
if missing_trust:
|
||||
confidence_missing_fields.append({"page": page["path"], "missing": missing_trust})
|
||||
try:
|
||||
validate_trust_metadata(
|
||||
vault / page["path"],
|
||||
allowed_lifecycles=lifecycles,
|
||||
required_trust_keys=(),
|
||||
)
|
||||
except ValueError as exc:
|
||||
trust_metadata_errors.append({"page": page["path"], "issue": str(exc)})
|
||||
|
||||
# Two pages with the same slugged stem make a bare `[[stem]]` link mean
|
||||
# whichever is nearer to the linking page (`links.PageIndex`), which is
|
||||
# easy to get wrong. Pages in different folders stay separate graph nodes;
|
||||
# `Vector Search.md` beside `vector-search.md` still share one, since
|
||||
# nothing distinguishes them. Key this check with the graph's own page
|
||||
# selection so it reports exactly the pages the graph sees.
|
||||
stem_index: dict[str, list[str]] = defaultdict(list)
|
||||
for graph_page in iter_graph_pages(vault):
|
||||
stem_index[graph_page_slug(graph_page, vault)].append(
|
||||
graph_page.relative_to(vault).as_posix()
|
||||
)
|
||||
duplicate_stems = [
|
||||
{"stem": stem, "pages": sorted(paths)}
|
||||
for stem, paths in sorted(stem_index.items())
|
||||
if len(paths) > 1
|
||||
]
|
||||
|
||||
title_index: dict[str, list[str]] = defaultdict(list)
|
||||
for page in pages:
|
||||
title_index[page["title"].strip().lower()].append(page["path"])
|
||||
duplicate_titles = [
|
||||
{"title": title, "pages": paths}
|
||||
for title, paths in title_index.items()
|
||||
if title and len(paths) > 1
|
||||
]
|
||||
duplicate_titles.sort(key=lambda item: (item["title"], item["pages"]))
|
||||
|
||||
missing_summaries = [
|
||||
page["path"]
|
||||
for page in pages
|
||||
if page["slug"] not in RESERVED_PAGE_STEMS
|
||||
and ("summary" not in page["fields"] or not page["summary"])
|
||||
]
|
||||
|
||||
# Legacy provenance: a `sources:` entry holding a machine absolute path cannot
|
||||
# resolve on another machine, so a synced vault loses the trail. Read-only —
|
||||
# migration of page frontmatter is a separate, deliberate pass (see
|
||||
# docs/cli.md → Source keys and legacy migration).
|
||||
machine_path_sources = [
|
||||
{"page": page["path"], "sources": page["absolute_sources"]}
|
||||
for page in pages
|
||||
if page["slug"] not in RESERVED_PAGE_STEMS and page["absolute_sources"]
|
||||
]
|
||||
|
||||
undated_claims = [
|
||||
{"page": page["path"], **claim}
|
||||
for page in pages
|
||||
if page["slug"] not in RESERVED_PAGE_STEMS
|
||||
for claim in page["undated_claims"]
|
||||
]
|
||||
|
||||
orphan_pages = []
|
||||
for page in pages:
|
||||
if page["slug"] in RESERVED_PAGE_STEMS:
|
||||
continue
|
||||
outgoing = sum(1 for _, target in page["resolved"] if target and target != page["id"])
|
||||
if outgoing == 0 and incoming.get(page["id"], 0) == 0:
|
||||
orphan_pages.append(page["path"])
|
||||
|
||||
manifest_sources = _load_manifest(vault)
|
||||
inverted = invert_pages(manifest_sources)
|
||||
need_urls = any(
|
||||
(key or "").startswith("url:") for key, _ in _iter_entries(manifest_sources)
|
||||
)
|
||||
url_index = clip_url_index(vault) if need_urls else None
|
||||
snapshot_mismatch = []
|
||||
for page in pages:
|
||||
if page["slug"] in RESERVED_PAGE_STEMS:
|
||||
continue
|
||||
expected = expected_snapshots_for_page(
|
||||
vault,
|
||||
page["path"],
|
||||
manifest_sources,
|
||||
inverted=inverted,
|
||||
url_index=url_index,
|
||||
)
|
||||
if not expected:
|
||||
continue
|
||||
actual = page["snapshots"]
|
||||
if set(expected) != set(actual):
|
||||
snapshot_mismatch.append(
|
||||
{
|
||||
"page": page["path"],
|
||||
"expected": sorted(expected),
|
||||
"actual": sorted(actual),
|
||||
}
|
||||
)
|
||||
snapshot_mismatch.sort(key=lambda item: item["page"])
|
||||
|
||||
typed_relationship_issues: list[dict[str, Any]] = []
|
||||
for page in pages:
|
||||
for index, relationship in enumerate(page["relationships"]):
|
||||
if "parse_error" in relationship:
|
||||
typed_relationship_issues.append(
|
||||
{
|
||||
"page": page["path"],
|
||||
"index": index,
|
||||
"issue": relationship["parse_error"],
|
||||
}
|
||||
)
|
||||
continue
|
||||
relation_type = relationship.get("type", "")
|
||||
target_raw = relationship.get("target", "")
|
||||
if relation_type not in relationship_types:
|
||||
typed_relationship_issues.append(
|
||||
{
|
||||
"page": page["path"],
|
||||
"index": index,
|
||||
"issue": "invalid_type",
|
||||
"type": relation_type,
|
||||
}
|
||||
)
|
||||
continue
|
||||
target = _normalise_node_id(target_raw)
|
||||
link = _relationship_link(target_raw)
|
||||
if page_index.is_ambiguous(link, page["path"]):
|
||||
typed_relationship_issues.append(
|
||||
{
|
||||
"page": page["path"],
|
||||
"index": index,
|
||||
"issue": "ambiguous_target",
|
||||
"target": target,
|
||||
}
|
||||
)
|
||||
continue
|
||||
resolved = page_index.resolve(link, page["path"]) if link else None
|
||||
if resolved is None:
|
||||
typed_relationship_issues.append(
|
||||
{
|
||||
"page": page["path"],
|
||||
"index": index,
|
||||
"issue": "missing_target",
|
||||
"target": target,
|
||||
}
|
||||
)
|
||||
elif resolved == page["id"]:
|
||||
typed_relationship_issues.append(
|
||||
{
|
||||
"page": page["path"],
|
||||
"index": index,
|
||||
"issue": "self_reference",
|
||||
"target": target,
|
||||
}
|
||||
)
|
||||
|
||||
# Event-time validity (obsidian_wiki.temporal). Malformed dates fail: a
|
||||
# page whose window can't be parsed is silently treated as current by
|
||||
# retrieval, so an unreported typo means a stale claim answers as fact.
|
||||
# A dangling `superseded_by` only warns — the pointer is advisory, and the
|
||||
# broken_links check already covers the wikilink if the body carries one.
|
||||
temporal_errors: list[dict[str, str]] = []
|
||||
superseded_dangling: list[dict[str, str]] = []
|
||||
for page in pages:
|
||||
try:
|
||||
validity_window(page["values"])
|
||||
except ValueError as exc:
|
||||
temporal_errors.append({"page": page["path"], "issue": str(exc)})
|
||||
raw_successor = page["values"].get(SUPERSEDED_FIELD, "")
|
||||
if not raw_successor:
|
||||
continue
|
||||
successor = _slug(superseded_target(raw_successor))
|
||||
resolved_successor = page_index.resolve(successor, page["path"]) if successor else None
|
||||
if not successor:
|
||||
temporal_errors.append(
|
||||
{"page": page["path"], "issue": f"empty {SUPERSEDED_FIELD} value"}
|
||||
)
|
||||
elif resolved_successor == page["id"]:
|
||||
superseded_dangling.append(
|
||||
{"page": page["path"], "target": successor, "issue": "self_reference"}
|
||||
)
|
||||
elif resolved_successor is None:
|
||||
superseded_dangling.append(
|
||||
{"page": page["path"], "target": successor, "issue": "missing_target"}
|
||||
)
|
||||
|
||||
ledger_path = vault / TRUST_LEDGER_RELATIVE_PATH
|
||||
trust_report = (
|
||||
check_trust_ledger(
|
||||
vault,
|
||||
ledger_path,
|
||||
allowed_lifecycles=lifecycles,
|
||||
required_trust_keys=trust_fields,
|
||||
schema_source=schema_source,
|
||||
)
|
||||
if ledger_path.is_file() or require_trust_ledger
|
||||
else None
|
||||
)
|
||||
illegal_lifecycle_transitions = (
|
||||
check_lifecycle_transitions(
|
||||
vault,
|
||||
ledger_path,
|
||||
allowed_lifecycles=lifecycles,
|
||||
required_trust_keys=trust_fields,
|
||||
)
|
||||
if ledger_path.is_file()
|
||||
else []
|
||||
)
|
||||
|
||||
findings = {
|
||||
"broken_links": broken_links,
|
||||
"missing_frontmatter": missing_frontmatter,
|
||||
"duplicate_titles": duplicate_titles,
|
||||
"duplicate_stems": duplicate_stems,
|
||||
"missing_summaries": sorted(missing_summaries),
|
||||
"machine_path_sources": machine_path_sources,
|
||||
"snapshot_mismatch": snapshot_mismatch,
|
||||
"orphan_pages": sorted(orphan_pages),
|
||||
"undated_claims": undated_claims,
|
||||
"typed_relationship_issues": typed_relationship_issues,
|
||||
# Sorted by page: these findings get diffed between CI runs.
|
||||
"temporal_errors": sorted(temporal_errors, key=lambda item: item["page"]),
|
||||
"superseded_dangling": sorted(superseded_dangling, key=lambda item: item["page"]),
|
||||
"confidence_missing_fields": confidence_missing_fields,
|
||||
"trust_metadata_errors": trust_metadata_errors,
|
||||
"confidence_review_stale": trust_report["stale"] if trust_report else [],
|
||||
"confidence_unreviewed": trust_report["unreviewed"] if trust_report else [],
|
||||
"confidence_mismatches": trust_report["score_mismatches"] if trust_report else [],
|
||||
"confidence_ledger_errors": trust_report["errors"] if trust_report else [],
|
||||
"illegal_lifecycle_transitions": illegal_lifecycle_transitions,
|
||||
}
|
||||
counts = {name: len(items) for name, items in findings.items()}
|
||||
|
||||
# Staged migration (#28, #146): a missing trust ledger or trust frontmatter
|
||||
# on legacy pages only fails the vault when the owner has explicitly opted
|
||||
# into strict_trust. Ledger presence alone never silently enables strict
|
||||
# enforcement; core structural findings (broken links, missing core
|
||||
# frontmatter) always fail regardless of trust mode.
|
||||
trust_finding_names = (
|
||||
"confidence_missing_fields",
|
||||
"confidence_mismatches",
|
||||
"confidence_ledger_errors",
|
||||
"confidence_review_stale",
|
||||
"confidence_unreviewed",
|
||||
"illegal_lifecycle_transitions",
|
||||
)
|
||||
trust_findings_present = any(counts[name] for name in trust_finding_names)
|
||||
trust_fails = strict_trust and any(
|
||||
counts[name]
|
||||
for name in (
|
||||
"confidence_missing_fields",
|
||||
"confidence_mismatches",
|
||||
"confidence_ledger_errors",
|
||||
"confidence_review_stale",
|
||||
"illegal_lifecycle_transitions",
|
||||
)
|
||||
)
|
||||
|
||||
if (
|
||||
counts["broken_links"]
|
||||
or counts["missing_frontmatter"]
|
||||
or counts["trust_metadata_errors"]
|
||||
or counts["temporal_errors"]
|
||||
or trust_fails
|
||||
):
|
||||
status = "fail"
|
||||
elif (
|
||||
any(
|
||||
counts[name]
|
||||
for name in (
|
||||
"duplicate_titles",
|
||||
"duplicate_stems",
|
||||
"missing_summaries",
|
||||
"machine_path_sources",
|
||||
"snapshot_mismatch",
|
||||
"orphan_pages",
|
||||
"undated_claims",
|
||||
"typed_relationship_issues",
|
||||
"superseded_dangling",
|
||||
)
|
||||
)
|
||||
or trust_findings_present
|
||||
):
|
||||
status = "warn"
|
||||
else:
|
||||
status = "pass"
|
||||
|
||||
return {
|
||||
"status": status,
|
||||
"schema": {
|
||||
"source": schema_source,
|
||||
"allowed_lifecycles": sorted(lifecycles),
|
||||
"allowed_relationship_types": sorted(relationship_types),
|
||||
"required_trust_fields": list(trust_fields),
|
||||
},
|
||||
"stats": {
|
||||
"pages": len(pages),
|
||||
"link_count": sum(len(page["links"]) for page in pages),
|
||||
"findings": counts,
|
||||
"trust": trust_report["counts"] if trust_report else {"ledger": "not_configured"},
|
||||
},
|
||||
"findings": findings,
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,199 @@
|
||||
"""Ledger invert and snapshot-path compare for lint (ADR 0001)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from obsidian_wiki.cache import _is_file_key, _iter_entries
|
||||
from obsidian_wiki.vault import split_frontmatter
|
||||
|
||||
_PAGE_LIST_KEYS = ("pages_produced", "pages_created")
|
||||
_URL_FIELDS = ("url", "source", "source_url")
|
||||
_MARKDOWN_LINK_RE = re.compile(r"^\[([^\]]*)\]\(([^)]+)\)$")
|
||||
|
||||
|
||||
def unwrap_snapshot_value(raw: str) -> str:
|
||||
value = raw.strip().strip("'\"").strip()
|
||||
markdown = _MARKDOWN_LINK_RE.match(value)
|
||||
if markdown:
|
||||
value = markdown.group(2).strip()
|
||||
elif value.startswith("[[") and "]]" in value:
|
||||
inner = value[2 : value.index("]]")]
|
||||
inner = inner.split("|", 1)[0].split("#", 1)[0].strip()
|
||||
value = inner
|
||||
elif "|" in value:
|
||||
# _WIKILINK_RE captures inners as path|display without [[ ]].
|
||||
value = value.split("|", 1)[0].split("#", 1)[0].strip()
|
||||
if value and _is_file_key(value) and Path(value).suffix == "":
|
||||
value = f"{value}.md"
|
||||
return value
|
||||
|
||||
|
||||
def invert_pages(sources: Any) -> dict[str, list[str]]:
|
||||
"""Map wiki page path -> unique ledger keys, first-seen order."""
|
||||
by_page: dict[str, list[str]] = defaultdict(list)
|
||||
seen: dict[str, set[str]] = defaultdict(set)
|
||||
for key, entry in _iter_entries(sources):
|
||||
if not key or not isinstance(entry, dict):
|
||||
continue
|
||||
pages: list[str] = []
|
||||
for field in _PAGE_LIST_KEYS:
|
||||
raw = entry.get(field) or []
|
||||
if isinstance(raw, list):
|
||||
pages.extend(str(p) for p in raw if p)
|
||||
for page in pages:
|
||||
if key in seen[page]:
|
||||
continue
|
||||
seen[page].add(key)
|
||||
by_page[page].append(key)
|
||||
return dict(by_page)
|
||||
|
||||
|
||||
def _archived_dir(vault: Path) -> Path:
|
||||
return vault / "_raw" / "_archived"
|
||||
|
||||
|
||||
def clip_url_index(vault: Path) -> dict[str, list[str]]:
|
||||
"""Map clip YAML url -> archived relative paths (may be 0, 1, or many)."""
|
||||
index: dict[str, list[str]] = defaultdict(list)
|
||||
root = _archived_dir(vault)
|
||||
if not root.is_dir():
|
||||
return {}
|
||||
for path in sorted(root.rglob("*.md")):
|
||||
text = path.read_text(encoding="utf-8", errors="replace")
|
||||
frontmatter = split_frontmatter(text)[0]
|
||||
rel = path.relative_to(vault).as_posix()
|
||||
for line in frontmatter.splitlines():
|
||||
if ":" not in line or line.startswith((" ", "\t")):
|
||||
continue
|
||||
key, raw = line.split(":", 1)
|
||||
if key.strip() not in _URL_FIELDS:
|
||||
continue
|
||||
url = raw.strip().strip("'\"")
|
||||
if url:
|
||||
index[url].append(rel)
|
||||
break # first matching URL field wins per file
|
||||
return dict(index)
|
||||
|
||||
|
||||
def resolve_source_key(
|
||||
vault: Path,
|
||||
key: str,
|
||||
*,
|
||||
url_index: dict[str, list[str]] | None = None,
|
||||
) -> str | None:
|
||||
"""Return a vault-relative archived path, or None if not a unique snapshot file."""
|
||||
if key.startswith("url:"):
|
||||
url = key[4:]
|
||||
index = url_index if url_index is not None else clip_url_index(vault)
|
||||
matches = index.get(url) or []
|
||||
if len(matches) == 1:
|
||||
return matches[0]
|
||||
return None
|
||||
if not _is_file_key(key):
|
||||
return None
|
||||
candidate = Path(key)
|
||||
if candidate.is_absolute() or ".." in candidate.parts:
|
||||
return None
|
||||
if "_archived" in candidate.parts or candidate.parts[:2] == ("_sources", "archive"):
|
||||
return candidate.as_posix() if (vault / candidate).is_file() else None
|
||||
return _stale_raw_archive(vault, candidate)
|
||||
|
||||
|
||||
def _stale_raw_archive(vault: Path, candidate: Path) -> str | None:
|
||||
"""`_raw/<name>` that ingest already moved to `_raw/_archived/<name>`.
|
||||
|
||||
Only `_raw/` drafts qualify, and only once the draft itself is gone: a live
|
||||
file, or a same-named file elsewhere in the vault, is never an archive.
|
||||
"""
|
||||
if candidate.parts[:1] != ("_raw",) or (vault / candidate).is_file():
|
||||
return None
|
||||
archived_rel = Path("_raw") / "_archived" / candidate.name
|
||||
return archived_rel.as_posix() if (vault / archived_rel).is_file() else None
|
||||
|
||||
|
||||
def parse_snapshots_field(raw: str) -> list[str]:
|
||||
if not raw.strip():
|
||||
return []
|
||||
lines = raw.splitlines()
|
||||
inline = lines[0].strip()
|
||||
entries: list[str] = []
|
||||
if inline.startswith("["):
|
||||
entries.extend(inline.strip("[]").split(","))
|
||||
elif inline:
|
||||
entries.append(inline)
|
||||
entries.extend(
|
||||
line.strip()[1:] for line in lines[1:] if line.strip().startswith("-")
|
||||
)
|
||||
result: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for entry in entries:
|
||||
value = unwrap_snapshot_value(entry)
|
||||
if not value or value in seen:
|
||||
continue
|
||||
seen.add(value)
|
||||
result.append(value)
|
||||
return result
|
||||
|
||||
|
||||
def expected_snapshots_for_page(
|
||||
vault: Path,
|
||||
page: str,
|
||||
sources: Any,
|
||||
*,
|
||||
inverted: dict[str, list[str]] | None = None,
|
||||
url_index: dict[str, list[str]] | None = None,
|
||||
) -> list[str]:
|
||||
mapping = inverted if inverted is not None else invert_pages(sources)
|
||||
keys = mapping.get(page) or []
|
||||
need_urls = any(k.startswith("url:") for k in keys)
|
||||
index = url_index
|
||||
if index is None and need_urls:
|
||||
index = clip_url_index(vault)
|
||||
expected: list[str] = []
|
||||
seen: set[str] = set()
|
||||
for key in keys:
|
||||
resolved = resolve_source_key(vault, key, url_index=index)
|
||||
if not resolved or resolved in seen:
|
||||
continue
|
||||
seen.add(resolved)
|
||||
expected.append(resolved)
|
||||
return expected
|
||||
|
||||
|
||||
def prefer_archive_write_path(vault: Path, rel: str) -> str | None:
|
||||
"""Vault-relative archive path to store, or None if nothing exists on disk."""
|
||||
rel = rel.strip().replace("\\", "/")
|
||||
candidate = Path(rel)
|
||||
if candidate.is_absolute() or ".." in candidate.parts:
|
||||
return None
|
||||
if "_archived" in candidate.parts:
|
||||
if (vault / candidate).is_file():
|
||||
return candidate.as_posix()
|
||||
return None
|
||||
if (vault / candidate).is_file():
|
||||
return candidate.as_posix()
|
||||
return _stale_raw_archive(vault, candidate)
|
||||
|
||||
|
||||
def archive_wikilink_relpath(_vault: Path, inner: str) -> str | None:
|
||||
"""Normalise upstream snapshots and local source archive/catalog links.
|
||||
|
||||
Does not require the file to exist. Returns None when the inner is not
|
||||
under a supported source directory. Preserve explicit file extensions.
|
||||
"""
|
||||
value = unwrap_snapshot_value(inner)
|
||||
if not value:
|
||||
return None
|
||||
path = Path(value.replace("\\", "/"))
|
||||
if path.is_absolute() or ".." in path.parts:
|
||||
return None
|
||||
parts = path.parts
|
||||
if len(parts) < 3 or parts[:2] not in {
|
||||
("_raw", "_archived"), ("_sources", "archive"), ("_sources", "catalog"),
|
||||
}:
|
||||
return None
|
||||
return value if path.suffix else f"{value}.md"
|
||||
@@ -0,0 +1,271 @@
|
||||
"""Staged-write promotion — the mechanical half of `wiki-stage-commit`.
|
||||
|
||||
When ``WIKI_STAGED_WRITES=true``, skills write pages into ``_staging/`` instead of
|
||||
the live vault, and a human approves them before they land. This module owns the
|
||||
parts of that workflow that have one correct answer: what is queued, moving a
|
||||
file to its live path, moving a rejection back to ``_raw/``, and refusing to do
|
||||
either when the files changed since they were reviewed.
|
||||
|
||||
Judgment stays with the skill — whether a page *should* land, how to summarise
|
||||
it, and how to merge a ``.patch.md`` whose surrounding text has moved on.
|
||||
Promotion never rewrites a page: it is a rename, so frontmatter, body and links
|
||||
arrive exactly as they were reviewed.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
STAGING_DIR = "_staging"
|
||||
RAW_DIR = "_raw"
|
||||
PATCH_SUFFIX = ".patch.md"
|
||||
|
||||
|
||||
class StagingError(Exception):
|
||||
"""A staged operation could not be performed."""
|
||||
|
||||
|
||||
class StagingConflict(StagingError):
|
||||
"""The staged or live file changed since the caller reviewed it.
|
||||
|
||||
Raised instead of overwriting, so a caller can re-read the pair and show the
|
||||
reviewer what actually changed. Agents write to the vault while a human is
|
||||
still looking at a diff; this is the case that makes promotion unsafe.
|
||||
"""
|
||||
|
||||
|
||||
def resolve_in_vault(vault: Path, rel: str) -> Path:
|
||||
"""Resolve a caller-supplied relative path inside *vault*, or refuse.
|
||||
|
||||
The single path check for staged operations. Symlinks are resolved before
|
||||
the comparison, so a link inside the vault pointing out of it is caught too.
|
||||
"""
|
||||
root = Path(vault).resolve()
|
||||
target = (root / rel).resolve()
|
||||
if target != root and root not in target.parents:
|
||||
raise StagingError(f"path escapes the vault: {rel}")
|
||||
return target
|
||||
|
||||
|
||||
def revision(path: Path) -> str | None:
|
||||
"""Content hash of *path*, or ``None`` when it does not exist.
|
||||
|
||||
``None`` is meaningful: it is the revision of a live page that does not exist
|
||||
yet, so a caller can pin "there was no live page when I reviewed this" and
|
||||
have promotion fail if one appeared in the meantime.
|
||||
"""
|
||||
if not path.is_file():
|
||||
return None
|
||||
return hashlib.sha256(path.read_bytes()).hexdigest()
|
||||
|
||||
|
||||
@dataclass
|
||||
class StagedEntry:
|
||||
"""One file waiting in ``_staging/``."""
|
||||
|
||||
staged_path: str
|
||||
live_path: str
|
||||
kind: str # "new" | "update" | "patch"
|
||||
staged_revision: str | None = None
|
||||
live_revision: str | None = None
|
||||
staged_mtime: str = ""
|
||||
|
||||
def as_dict(self) -> dict[str, Any]:
|
||||
return {
|
||||
"staged_path": self.staged_path,
|
||||
"live_path": self.live_path,
|
||||
"kind": self.kind,
|
||||
"staged_revision": self.staged_revision,
|
||||
"live_revision": self.live_revision,
|
||||
"staged_mtime": self.staged_mtime,
|
||||
}
|
||||
|
||||
|
||||
def _live_relative(staged_rel: Path) -> str:
|
||||
"""Map a path under ``_staging/`` to the live path it targets."""
|
||||
name = staged_rel.name
|
||||
if name.endswith(PATCH_SUFFIX):
|
||||
# `concepts/foo.patch.md` describes an edit to `concepts/foo.md`.
|
||||
name = name[: -len(PATCH_SUFFIX)] + ".md"
|
||||
return str(staged_rel.with_name(name))
|
||||
|
||||
|
||||
def list_staged(vault: Path) -> list[StagedEntry]:
|
||||
"""Inventory ``_staging/``, newest-staged last, with both revisions pinned."""
|
||||
vault = Path(vault)
|
||||
root = vault / STAGING_DIR
|
||||
if not root.is_dir():
|
||||
return []
|
||||
|
||||
entries: list[StagedEntry] = []
|
||||
for staged in sorted(root.rglob("*.md")):
|
||||
staged_rel = staged.relative_to(root)
|
||||
live_rel = _live_relative(staged_rel)
|
||||
live = vault / live_rel
|
||||
if staged.name.endswith(PATCH_SUFFIX):
|
||||
kind = "patch"
|
||||
else:
|
||||
kind = "update" if live.is_file() else "new"
|
||||
stat = staged.stat()
|
||||
entries.append(
|
||||
StagedEntry(
|
||||
staged_path=str(staged.relative_to(vault)),
|
||||
live_path=live_rel,
|
||||
kind=kind,
|
||||
staged_revision=revision(staged),
|
||||
live_revision=revision(live),
|
||||
staged_mtime=date.fromtimestamp(stat.st_mtime).isoformat(),
|
||||
)
|
||||
)
|
||||
return entries
|
||||
|
||||
|
||||
def _staged_file(vault: Path, rel: str) -> Path:
|
||||
"""Resolve *rel* to a file under ``_staging/``, accepting either spelling.
|
||||
|
||||
Callers hold either the vault-relative path from `list_staged`
|
||||
(``_staging/concepts/a.md``) or the staging-relative one (``concepts/a.md``).
|
||||
Both name the same file; accepting one and not the other is a trap.
|
||||
"""
|
||||
rel = rel.replace(os.sep, "/").lstrip("/")
|
||||
if not rel:
|
||||
raise StagingError("no staged path given")
|
||||
candidate = rel if rel.startswith(f"{STAGING_DIR}/") else f"{STAGING_DIR}/{rel}"
|
||||
staged = resolve_in_vault(vault, candidate)
|
||||
# Being inside the vault is not enough: `../concepts/a.md` resolves to a live
|
||||
# page, which would otherwise be promoted or discarded as if it were staged.
|
||||
root = resolve_in_vault(vault, STAGING_DIR)
|
||||
if root not in staged.parents:
|
||||
raise StagingError(f"path is not under {STAGING_DIR}/: {rel}")
|
||||
if not staged.is_file():
|
||||
raise StagingError(f"no staged file: {candidate}")
|
||||
return staged
|
||||
|
||||
|
||||
def _check_revision(path: Path, expected: str | None, label: str) -> None:
|
||||
"""Refuse when *path* no longer matches the revision the caller reviewed.
|
||||
|
||||
``expected`` of ``None`` means the caller did not pin this side and accepts
|
||||
whatever is there — distinct from a pinned ``None``, which asserts absence.
|
||||
"""
|
||||
if expected is None:
|
||||
return
|
||||
actual = revision(path)
|
||||
if actual == expected:
|
||||
return
|
||||
raise StagingConflict(
|
||||
f"{label} changed since it was reviewed "
|
||||
f"(expected {expected or 'no file'}, found {actual or 'no file'})"
|
||||
)
|
||||
|
||||
|
||||
def promote(
|
||||
vault: Path,
|
||||
rel: str,
|
||||
*,
|
||||
expected_staged_revision: str | None = None,
|
||||
expected_live_revision: str | None = None,
|
||||
pin_live_absent: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
"""Move one staged page to its live path.
|
||||
|
||||
The move is a rename, so the promoted page is byte-for-byte what was
|
||||
reviewed — arbitrary frontmatter, body and wikilinks all survive untouched.
|
||||
|
||||
Pass ``expected_staged_revision`` / ``expected_live_revision`` from the
|
||||
listing the reviewer saw to make this fail rather than clobber a concurrent
|
||||
write. ``pin_live_absent=True`` asserts there was no live page at review
|
||||
time, which a bare ``expected_live_revision=None`` cannot express.
|
||||
|
||||
``.patch.md`` files are refused: merging a human-readable diff into a page
|
||||
whose surrounding text may have moved is judgment, and belongs in the skill.
|
||||
"""
|
||||
vault = Path(vault)
|
||||
staged = _staged_file(vault, rel)
|
||||
if staged.name.endswith(PATCH_SUFFIX):
|
||||
raise StagingError(
|
||||
f"{staged.relative_to(vault)} is a patch, not a whole page — "
|
||||
"merge it with the wiki-stage-commit skill, then promote the result"
|
||||
)
|
||||
|
||||
live_rel = _live_relative(staged.relative_to(vault / STAGING_DIR))
|
||||
live = resolve_in_vault(vault, live_rel)
|
||||
|
||||
_check_revision(staged, expected_staged_revision, "staged file")
|
||||
if pin_live_absent and live.is_file():
|
||||
raise StagingConflict(f"live page {live_rel} was created since it was reviewed")
|
||||
_check_revision(live, expected_live_revision, f"live page {live_rel}")
|
||||
|
||||
replaced = live.is_file()
|
||||
live.parent.mkdir(parents=True, exist_ok=True)
|
||||
# os.replace is atomic within a filesystem: no reader ever sees a partial page.
|
||||
os.replace(staged, live)
|
||||
_prune_empty_dirs(vault / STAGING_DIR, staged.parent)
|
||||
|
||||
return {
|
||||
"action": "promote",
|
||||
"staged_path": str(staged.relative_to(vault)),
|
||||
"live_path": live_rel,
|
||||
"kind": "update" if replaced else "new",
|
||||
"revision": revision(live),
|
||||
}
|
||||
|
||||
|
||||
def discard(vault: Path, rel: str) -> dict[str, Any]:
|
||||
"""Move one staged file back to ``_raw/`` for manual editing.
|
||||
|
||||
Named ``rejected-<category>-<page>.md`` so the origin stays readable once the
|
||||
directory structure is flattened away.
|
||||
"""
|
||||
vault = Path(vault)
|
||||
staged = _staged_file(vault, rel)
|
||||
staged_rel = staged.relative_to(vault / STAGING_DIR)
|
||||
|
||||
stem = "-".join(staged_rel.with_suffix("").parts)
|
||||
prefix = "rejected-patch-" if staged.name.endswith(PATCH_SUFFIX) else "rejected-"
|
||||
if stem.endswith(".patch"):
|
||||
stem = stem[: -len(".patch")]
|
||||
raw = resolve_in_vault(vault, f"{RAW_DIR}/{prefix}{stem}.md")
|
||||
raw.parent.mkdir(parents=True, exist_ok=True)
|
||||
# Never silently overwrite an earlier rejection of the same page.
|
||||
counter = 2
|
||||
while raw.exists():
|
||||
raw = raw.with_name(f"{prefix}{stem}-{counter}.md")
|
||||
counter += 1
|
||||
|
||||
os.replace(staged, raw)
|
||||
_prune_empty_dirs(vault / STAGING_DIR, staged.parent)
|
||||
|
||||
return {
|
||||
"action": "discard",
|
||||
"staged_path": str(staged.relative_to(vault)),
|
||||
"raw_path": str(raw.relative_to(vault)),
|
||||
}
|
||||
|
||||
|
||||
def _prune_empty_dirs(root: Path, start: Path) -> None:
|
||||
"""Remove directories left empty under ``_staging/`` after a move."""
|
||||
current = start.resolve()
|
||||
root = root.resolve()
|
||||
while current != root and root in current.parents:
|
||||
try:
|
||||
current.rmdir()
|
||||
except OSError:
|
||||
return
|
||||
current = current.parent
|
||||
|
||||
|
||||
def log_decisions(vault: Path, results: list[dict[str, Any]]) -> None:
|
||||
"""Append one STAGE_COMMIT line to ``log.md``, matching the skill's format."""
|
||||
if not results:
|
||||
return
|
||||
promoted = sum(1 for r in results if r["action"] == "promote")
|
||||
discarded = sum(1 for r in results if r["action"] == "discard")
|
||||
from obsidian_wiki.memory import append_log
|
||||
|
||||
append_log(Path(vault), "STAGE_COMMIT", {"accepted": promoted, "rejected": discarded})
|
||||
@@ -0,0 +1,95 @@
|
||||
"""Bi-temporal validity for vault pages.
|
||||
|
||||
`created`/`updated` are *ingestion* time — when the vault learned something.
|
||||
`valid_from`/`valid_until` are *event* time — when the claim itself was true.
|
||||
Together they make a page's history answerable: a page whose `valid_until` has
|
||||
passed is historical, not wrong, and it stays in the vault (and in the graph)
|
||||
instead of being overwritten. `superseded_by` names what replaced it.
|
||||
|
||||
All three fields are optional. A page carrying none of them is current,
|
||||
always — which is every page in a vault that predates this module.
|
||||
|
||||
Retrieval reads these through `is_current`; `lint` reads them through
|
||||
`validity_window` to report malformed dates.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import date, datetime
|
||||
from typing import Mapping
|
||||
|
||||
TEMPORAL_FIELDS = ("valid_from", "valid_until")
|
||||
SUPERSEDED_FIELD = "superseded_by"
|
||||
|
||||
__all__ = [
|
||||
"SUPERSEDED_FIELD",
|
||||
"TEMPORAL_FIELDS",
|
||||
"is_current",
|
||||
"parse_date",
|
||||
"superseded_target",
|
||||
"validity_window",
|
||||
]
|
||||
|
||||
|
||||
def parse_date(raw: str) -> "date | None":
|
||||
"""Parse `YYYY-MM-DD` or a full ISO 8601 timestamp down to a date.
|
||||
|
||||
Returns None for an empty value. Raises ValueError on a non-empty value
|
||||
that isn't a date, so callers can report it as malformed metadata rather
|
||||
than guessing.
|
||||
"""
|
||||
value = (raw or "").strip().strip("\"'")
|
||||
if not value:
|
||||
return None
|
||||
try:
|
||||
return date.fromisoformat(value)
|
||||
except ValueError:
|
||||
# `2026-04-01T09:30:00+00:00`. The `Z` spelling is rejected by
|
||||
# fromisoformat before 3.11, so normalise it first.
|
||||
return datetime.fromisoformat(value.replace("Z", "+00:00")).date()
|
||||
|
||||
|
||||
def validity_window(values: Mapping[str, str]) -> "tuple[date | None, date | None]":
|
||||
"""`(valid_from, valid_until)` for a frontmatter mapping.
|
||||
|
||||
Raises ValueError if either date is malformed or the window is inverted.
|
||||
"""
|
||||
start = parse_date(values.get("valid_from", ""))
|
||||
end = parse_date(values.get("valid_until", ""))
|
||||
if start and end and end < start:
|
||||
raise ValueError(
|
||||
f"valid_until ({end.isoformat()}) precedes valid_from ({start.isoformat()})"
|
||||
)
|
||||
return start, end
|
||||
|
||||
|
||||
def is_current(values: Mapping[str, str], as_of: "date | None" = None) -> bool:
|
||||
"""Was this page's claim true at `as_of` (default today)?
|
||||
|
||||
`valid_until` is inclusive — the last day the claim held, the way a person
|
||||
writing the date means it. A page with malformed dates counts as current:
|
||||
retrieval must never silently drop a page over a typo, and `lint` is what
|
||||
reports the typo.
|
||||
"""
|
||||
try:
|
||||
start, end = validity_window(values)
|
||||
except ValueError:
|
||||
return True
|
||||
when = as_of or date.today()
|
||||
if start and when < start:
|
||||
return False
|
||||
if end and when > end:
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def superseded_target(raw: str) -> str:
|
||||
"""Bare page name from a `superseded_by` value, `[[wikilink]]` or plain.
|
||||
|
||||
Returns the last path segment with any `.md`, alias, and heading anchor
|
||||
stripped. Callers apply their own slugging.
|
||||
"""
|
||||
target = (raw or "").strip().strip("\"'")
|
||||
target = target.removeprefix("[[").removesuffix("]]")
|
||||
target = target.split("|", 1)[0].split("#", 1)[0].strip()
|
||||
return target.removesuffix(".md").split("/")[-1].strip()
|
||||
@@ -0,0 +1,788 @@
|
||||
"""Manual trust-review ledger for confidence linting.
|
||||
|
||||
Confidence depends on semantic judgments that source-string heuristics cannot make:
|
||||
independent evidence lineages must be collapsed and material claim coverage must be
|
||||
reviewed. This module therefore validates an approved review ledger instead of
|
||||
pretending to recompute confidence from URLs.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import secrets
|
||||
import stat
|
||||
import tempfile
|
||||
from collections.abc import Collection
|
||||
from datetime import datetime
|
||||
from pathlib import Path, PurePosixPath
|
||||
from typing import Any
|
||||
|
||||
from obsidian_wiki.vault import FRONTMATTER_RE as _FRONTMATTER_RE
|
||||
from obsidian_wiki.vault import SKIP_DIRS, iter_md
|
||||
|
||||
TRUST_LEDGER_RELATIVE_PATH = Path("_meta/trust-ledger.json")
|
||||
TRUST_LEDGER_SCHEMA_VERSION = 1
|
||||
TRUST_REVIEW_METHOD = "manual-lineage-and-claim-coverage-v1"
|
||||
TRUST_SKIP_DIRS = SKIP_DIRS | {"_bootstrap"}
|
||||
TRUST_RESERVED_STEMS = frozenset({"index", "log", "hot", "_insights"})
|
||||
ALLOWED_LIFECYCLES = frozenset({"draft", "reviewed", "verified", "disputed", "archived"})
|
||||
# Transitions the llm-wiki state machine forbids outright: only ingest sets
|
||||
# `draft`, so nothing may fall back to it, and `archived` is terminal (a restore
|
||||
# is a deliberate human delete-and-recreate, not a transition).
|
||||
#
|
||||
# Deliberately NOT listed: draft -> verified. Ledger snapshots are sparse, so an
|
||||
# intermediate `reviewed` may well have happened between two reviews; flagging
|
||||
# that pair would fire on legitimate history.
|
||||
ILLEGAL_TRANSITIONS = frozenset(
|
||||
{("reviewed", "draft"), ("verified", "draft"), ("disputed", "draft")}
|
||||
| {("archived", nxt) for nxt in ALLOWED_LIFECYCLES - {"archived"}}
|
||||
)
|
||||
TRUST_REQUIRED_FIELD_ALLOWLIST = frozenset(
|
||||
{"base_confidence", "lifecycle", "lifecycle_changed", "updated"}
|
||||
)
|
||||
_REQUIRED_TRUST_KEYS = ("base_confidence", "lifecycle", "updated")
|
||||
_VOLATILE_CONFIDENCE_KEYS = (
|
||||
"updated",
|
||||
"base_confidence",
|
||||
"lifecycle",
|
||||
"lifecycle_changed",
|
||||
"lifecycle_reason",
|
||||
"superseded_by",
|
||||
)
|
||||
_TOP_LEVEL_FIELD_RE = re.compile(r"^([A-Za-z_][\w-]*):")
|
||||
|
||||
|
||||
def _normalise_text(text: str) -> str:
|
||||
return text.replace("\r\n", "\n").replace("\r", "\n")
|
||||
|
||||
|
||||
def _validate_reviewed_at(value: Any) -> str:
|
||||
if not isinstance(value, str) or not value.strip():
|
||||
raise ValueError("reviewed_at must be an ISO-8601 timestamp with timezone")
|
||||
candidate = value.strip()
|
||||
try:
|
||||
parsed = datetime.fromisoformat(candidate.replace("Z", "+00:00"))
|
||||
except ValueError as exc:
|
||||
raise ValueError("reviewed_at must be an ISO-8601 timestamp with timezone") from exc
|
||||
if parsed.tzinfo is None or parsed.utcoffset() is None:
|
||||
raise ValueError("reviewed_at must be an ISO-8601 timestamp with timezone")
|
||||
return candidate
|
||||
|
||||
|
||||
def _validate_updated(value: str) -> str:
|
||||
candidate = value.strip()
|
||||
try:
|
||||
# Page metadata intentionally permits either an ISO date or an ISO date-time.
|
||||
datetime.fromisoformat(candidate.replace("Z", "+00:00"))
|
||||
except ValueError as exc:
|
||||
raise ValueError("updated is not an ISO-8601 date or timestamp") from exc
|
||||
return candidate
|
||||
|
||||
|
||||
def _reject_json_constant(value: str) -> None:
|
||||
raise ValueError(f"non-standard JSON number: {value}")
|
||||
|
||||
|
||||
def _reject_duplicate_json_keys(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
|
||||
result: dict[str, Any] = {}
|
||||
for key, value in pairs:
|
||||
if key in result:
|
||||
raise ValueError(f"duplicate JSON key: {key}")
|
||||
result[key] = value
|
||||
return result
|
||||
|
||||
|
||||
def _frontmatter(text: str) -> str:
|
||||
match = _FRONTMATTER_RE.match(_normalise_text(text))
|
||||
return match.group(1) if match else ""
|
||||
|
||||
|
||||
def _frontmatter_scalar(raw: str) -> str:
|
||||
value = raw.strip().split(" #", 1)[0].strip()
|
||||
if len(value) >= 2 and value[0] == value[-1] and value[0] in {"'", '"'}:
|
||||
value = value[1:-1].strip()
|
||||
return value
|
||||
|
||||
|
||||
def _effective_schema(
|
||||
*,
|
||||
allowed_lifecycles: Collection[str] | None = None,
|
||||
required_trust_keys: Collection[str] | None = None,
|
||||
) -> tuple[frozenset[str], tuple[str, ...]]:
|
||||
lifecycles = frozenset(
|
||||
ALLOWED_LIFECYCLES if allowed_lifecycles is None else allowed_lifecycles
|
||||
)
|
||||
required = tuple(required_trust_keys) if required_trust_keys is not None else _REQUIRED_TRUST_KEYS
|
||||
unknown = set(required) - TRUST_REQUIRED_FIELD_ALLOWLIST
|
||||
if unknown:
|
||||
raise ValueError(f"unknown required trust field(s): {', '.join(sorted(unknown))}")
|
||||
return lifecycles, required
|
||||
|
||||
|
||||
def _trust_metadata(
|
||||
path: Path,
|
||||
*,
|
||||
allowed_lifecycles: Collection[str] | None = None,
|
||||
required_trust_keys: Collection[str] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Parse and validate trust-sensitive frontmatter without YAML ambiguity."""
|
||||
lifecycles, required = _effective_schema(
|
||||
allowed_lifecycles=allowed_lifecycles,
|
||||
required_trust_keys=required_trust_keys,
|
||||
)
|
||||
try:
|
||||
text = _normalise_text(path.read_text(encoding="utf-8"))
|
||||
except UnicodeError as exc:
|
||||
raise ValueError("page is not valid UTF-8") from exc
|
||||
frontmatter = _frontmatter(text)
|
||||
if not frontmatter:
|
||||
raise ValueError("missing frontmatter")
|
||||
|
||||
records: dict[str, list[tuple[str, list[str]]]] = {}
|
||||
current: tuple[str, list[str]] | None = None
|
||||
for line in frontmatter.splitlines():
|
||||
field = _TOP_LEVEL_FIELD_RE.match(line)
|
||||
if field:
|
||||
key = field.group(1)
|
||||
raw = line.split(":", 1)[1].strip()
|
||||
children: list[str] = []
|
||||
records.setdefault(key, []).append((raw, children))
|
||||
current = (key, children)
|
||||
continue
|
||||
if line.startswith((" ", "\t")) and current is not None:
|
||||
current[1].append(line)
|
||||
|
||||
for key in _VOLATILE_CONFIDENCE_KEYS:
|
||||
entries = records.get(key, [])
|
||||
if len(entries) > 1:
|
||||
raise ValueError(f"duplicate top-level field: {key}")
|
||||
for key in required:
|
||||
if key not in records:
|
||||
raise ValueError(f"missing {key}")
|
||||
|
||||
for key in _VOLATILE_CONFIDENCE_KEYS:
|
||||
entries = records.get(key, [])
|
||||
if not entries:
|
||||
continue
|
||||
raw, children = entries[0]
|
||||
scalar = _frontmatter_scalar(raw)
|
||||
if not scalar or scalar in {">", ">-", "|", "|-"} or children:
|
||||
raise ValueError(f"{key} must be a scalar")
|
||||
|
||||
if "updated" in records:
|
||||
updated = _frontmatter_scalar(records["updated"][0][0])
|
||||
_validate_updated(updated)
|
||||
|
||||
confidence: float | None = None
|
||||
if "base_confidence" in records:
|
||||
confidence_raw = _frontmatter_scalar(records["base_confidence"][0][0])
|
||||
try:
|
||||
confidence = float(confidence_raw)
|
||||
except ValueError as exc:
|
||||
raise ValueError("base_confidence is not numeric") from exc
|
||||
if not math.isfinite(confidence):
|
||||
raise ValueError("base_confidence is not finite")
|
||||
if not 0.0 <= confidence <= 1.0:
|
||||
raise ValueError("base_confidence is outside [0.0, 1.0]")
|
||||
|
||||
lifecycle: str | None = None
|
||||
if "lifecycle" in records:
|
||||
lifecycle = _frontmatter_scalar(records["lifecycle"][0][0])
|
||||
if lifecycle not in lifecycles:
|
||||
raise ValueError(f"invalid lifecycle: {lifecycle}")
|
||||
return {
|
||||
"text": text,
|
||||
"frontmatter": frontmatter,
|
||||
"confidence": confidence,
|
||||
"lifecycle": lifecycle,
|
||||
"review_status": "reviewable" if confidence is not None else "not_applicable",
|
||||
}
|
||||
|
||||
|
||||
def _strip_volatile_confidence_fields(frontmatter: str) -> str:
|
||||
"""Remove confidence/lifecycle bookkeeping while retaining material metadata."""
|
||||
kept: list[str] = []
|
||||
skipping = False
|
||||
for line in frontmatter.splitlines():
|
||||
field = _TOP_LEVEL_FIELD_RE.match(line)
|
||||
if field:
|
||||
skipping = field.group(1) in _VOLATILE_CONFIDENCE_KEYS
|
||||
if not skipping:
|
||||
kept.append(line.rstrip())
|
||||
return "\n".join(kept).strip()
|
||||
|
||||
|
||||
def page_fingerprint(
|
||||
path: Path,
|
||||
*,
|
||||
allowed_lifecycles: Collection[str] | None = None,
|
||||
required_trust_keys: Collection[str] | None = None,
|
||||
) -> str:
|
||||
"""Hash material claims and evidence, excluding validated volatile bookkeeping."""
|
||||
metadata = _trust_metadata(
|
||||
path,
|
||||
allowed_lifecycles=allowed_lifecycles,
|
||||
required_trust_keys=required_trust_keys,
|
||||
)
|
||||
text = metadata["text"]
|
||||
match = _FRONTMATTER_RE.match(text)
|
||||
assert match is not None
|
||||
stable_frontmatter = _strip_volatile_confidence_fields(metadata["frontmatter"])
|
||||
body = text[match.end() :].strip()
|
||||
material = f"---\n{stable_frontmatter}\n---\n{body}\n"
|
||||
return "sha256:" + hashlib.sha256(material.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def validate_trust_metadata(
|
||||
path: Path,
|
||||
*,
|
||||
allowed_lifecycles: Collection[str] | None = None,
|
||||
required_trust_keys: Collection[str] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Validate present trust fields even when no review ledger is configured."""
|
||||
return _trust_metadata(
|
||||
path,
|
||||
allowed_lifecycles=allowed_lifecycles,
|
||||
required_trust_keys=required_trust_keys,
|
||||
)
|
||||
|
||||
|
||||
def _parse_confidence(
|
||||
path: Path,
|
||||
*,
|
||||
allowed_lifecycles: Collection[str] | None = None,
|
||||
required_trust_keys: Collection[str] | None = None,
|
||||
) -> float:
|
||||
value = _trust_metadata(
|
||||
path,
|
||||
allowed_lifecycles=allowed_lifecycles,
|
||||
required_trust_keys=required_trust_keys,
|
||||
)["confidence"]
|
||||
if value is None:
|
||||
raise ValueError("missing base_confidence for trust review")
|
||||
return float(value)
|
||||
|
||||
|
||||
def iter_trust_pages(vault: Path) -> list[Path]:
|
||||
"""Return every non-reserved content page that must participate in trust review.
|
||||
|
||||
Honors the vault-root `.okignore`, so a vault can keep quarantined content
|
||||
(`_excluded/`, PII inboxes) out of the ledger without editing TRUST_SKIP_DIRS.
|
||||
"""
|
||||
pages = [p for p in iter_md(vault, TRUST_SKIP_DIRS) if p.stem not in TRUST_RESERVED_STEMS]
|
||||
# Keyed on the posix string, not Path order: the ledger is diffed across runs.
|
||||
return sorted(pages, key=lambda item: item.relative_to(vault).as_posix())
|
||||
|
||||
|
||||
def build_trust_ledger(
|
||||
vault: Path,
|
||||
*,
|
||||
reviewed_at: str,
|
||||
allowed_lifecycles: Collection[str] | None = None,
|
||||
required_trust_keys: Collection[str] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Capture explicitly approved confidence values and material fingerprints."""
|
||||
reviewed_at = _validate_reviewed_at(reviewed_at)
|
||||
pages: dict[str, dict[str, Any]] = {}
|
||||
not_applicable: list[str] = []
|
||||
for path in iter_trust_pages(vault):
|
||||
rel = path.relative_to(vault).as_posix()
|
||||
metadata = _trust_metadata(
|
||||
path,
|
||||
allowed_lifecycles=allowed_lifecycles,
|
||||
required_trust_keys=required_trust_keys,
|
||||
)
|
||||
if metadata["review_status"] == "not_applicable":
|
||||
not_applicable.append(rel)
|
||||
continue
|
||||
pages[rel] = _review_entry(
|
||||
path,
|
||||
reviewed_at,
|
||||
allowed_lifecycles=allowed_lifecycles,
|
||||
required_trust_keys=required_trust_keys,
|
||||
)
|
||||
return {
|
||||
"schema_version": TRUST_LEDGER_SCHEMA_VERSION,
|
||||
"method": TRUST_REVIEW_METHOD,
|
||||
"reviewed_at": reviewed_at,
|
||||
"pages": pages,
|
||||
"not_applicable": not_applicable,
|
||||
}
|
||||
|
||||
|
||||
def _review_entry(
|
||||
path: Path,
|
||||
reviewed_at: str,
|
||||
*,
|
||||
allowed_lifecycles: Collection[str] | None = None,
|
||||
required_trust_keys: Collection[str] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
metadata = _trust_metadata(
|
||||
path,
|
||||
allowed_lifecycles=allowed_lifecycles,
|
||||
required_trust_keys=required_trust_keys,
|
||||
)
|
||||
if metadata["review_status"] == "not_applicable":
|
||||
raise ValueError("trust review is not applicable: base_confidence is absent")
|
||||
return {
|
||||
"reviewed_confidence": _parse_confidence(
|
||||
path,
|
||||
allowed_lifecycles=allowed_lifecycles,
|
||||
required_trust_keys=required_trust_keys,
|
||||
),
|
||||
"material_fingerprint": page_fingerprint(
|
||||
path,
|
||||
allowed_lifecycles=allowed_lifecycles,
|
||||
required_trust_keys=required_trust_keys,
|
||||
),
|
||||
"reviewed_at": reviewed_at,
|
||||
# Recorded so lint can check the next transition against
|
||||
# ILLEGAL_TRANSITIONS. Ledgers written before this field simply skip
|
||||
# the check.
|
||||
"lifecycle": metadata["lifecycle"],
|
||||
}
|
||||
|
||||
|
||||
def _validate_ledger_entry(entry: Any) -> tuple[str, float, str]:
|
||||
if not isinstance(entry, dict):
|
||||
raise ValueError("invalid_ledger_entry")
|
||||
fingerprint = entry.get("material_fingerprint")
|
||||
reviewed_confidence = entry.get("reviewed_confidence")
|
||||
if (
|
||||
not isinstance(fingerprint, str)
|
||||
or re.fullmatch(r"sha256:[0-9a-f]{64}", fingerprint) is None
|
||||
):
|
||||
raise ValueError("invalid_ledger_entry")
|
||||
if not isinstance(reviewed_confidence, (int, float)) or isinstance(reviewed_confidence, bool):
|
||||
raise ValueError("invalid_ledger_entry")
|
||||
reviewed_value = float(reviewed_confidence)
|
||||
if not math.isfinite(reviewed_value) or not 0.0 <= reviewed_value <= 1.0:
|
||||
raise ValueError("invalid_ledger_entry")
|
||||
try:
|
||||
reviewed_at = _validate_reviewed_at(entry.get("reviewed_at"))
|
||||
except ValueError as exc:
|
||||
raise ValueError("invalid_ledger_entry") from exc
|
||||
return fingerprint, reviewed_value, reviewed_at
|
||||
|
||||
|
||||
def _validate_ledger_page_path(raw: Any) -> str:
|
||||
if not isinstance(raw, str) or not raw or "\\" in raw:
|
||||
raise ValueError("invalid_ledger_page_path")
|
||||
path = PurePosixPath(raw)
|
||||
if path.is_absolute() or ".." in path.parts or path.as_posix() != raw:
|
||||
raise ValueError("invalid_ledger_page_path")
|
||||
return raw
|
||||
|
||||
|
||||
def update_trust_ledger(
|
||||
vault: Path,
|
||||
ledger_path: Path,
|
||||
*,
|
||||
reviewed_at: str,
|
||||
page_paths: list[str],
|
||||
allowed_lifecycles: Collection[str] | None = None,
|
||||
required_trust_keys: Collection[str] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
"""Update reviewed pages and remove entries whose confidence is not applicable."""
|
||||
reviewed_at = _validate_reviewed_at(reviewed_at)
|
||||
if ledger_path.is_file():
|
||||
try:
|
||||
ledger = json.loads(
|
||||
ledger_path.read_text(encoding="utf-8"),
|
||||
parse_constant=_reject_json_constant,
|
||||
object_pairs_hook=_reject_duplicate_json_keys,
|
||||
)
|
||||
except (OSError, UnicodeError, ValueError) as exc:
|
||||
raise RuntimeError(f"cannot update unreadable trust ledger: {exc}") from exc
|
||||
if not isinstance(ledger, dict):
|
||||
raise RuntimeError("cannot update trust ledger: top level must be an object")
|
||||
if type(ledger.get("schema_version")) is not int or ledger.get("schema_version") != TRUST_LEDGER_SCHEMA_VERSION:
|
||||
raise RuntimeError("cannot update unsupported trust ledger schema")
|
||||
if ledger.get("method") != TRUST_REVIEW_METHOD or not isinstance(ledger.get("pages"), dict):
|
||||
raise RuntimeError("cannot update malformed trust ledger")
|
||||
try:
|
||||
_validate_reviewed_at(ledger.get("reviewed_at"))
|
||||
for rel, entry in ledger["pages"].items():
|
||||
_validate_ledger_page_path(rel)
|
||||
_validate_ledger_entry(entry)
|
||||
except ValueError as exc:
|
||||
raise RuntimeError("cannot update malformed trust ledger") from exc
|
||||
else:
|
||||
ledger = {
|
||||
"schema_version": TRUST_LEDGER_SCHEMA_VERSION,
|
||||
"method": TRUST_REVIEW_METHOD,
|
||||
"reviewed_at": reviewed_at,
|
||||
"pages": {},
|
||||
}
|
||||
|
||||
current = {path.relative_to(vault).as_posix(): path for path in iter_trust_pages(vault)}
|
||||
not_applicable: list[str] = []
|
||||
for rel, page in current.items():
|
||||
metadata = _trust_metadata(
|
||||
page,
|
||||
allowed_lifecycles=allowed_lifecycles,
|
||||
required_trust_keys=required_trust_keys,
|
||||
)
|
||||
if metadata["review_status"] == "not_applicable":
|
||||
not_applicable.append(rel)
|
||||
removed_not_applicable = sorted(set(ledger["pages"]) & set(not_applicable))
|
||||
for rel in removed_not_applicable:
|
||||
del ledger["pages"][rel]
|
||||
|
||||
selected: list[str] = []
|
||||
for raw in page_paths:
|
||||
candidate = Path(raw)
|
||||
if candidate.is_absolute() or ".." in candidate.parts:
|
||||
raise RuntimeError(f"trust page must be a safe vault-relative path: {raw}")
|
||||
rel = candidate.as_posix().removeprefix("./")
|
||||
page = current.get(rel)
|
||||
if page is None:
|
||||
raise RuntimeError(f"trust page is missing or lacks the trust schema: {raw}")
|
||||
if rel not in selected:
|
||||
selected.append(rel)
|
||||
if rel in not_applicable:
|
||||
continue
|
||||
ledger["pages"][rel] = _review_entry(
|
||||
page,
|
||||
reviewed_at,
|
||||
allowed_lifecycles=allowed_lifecycles,
|
||||
required_trust_keys=required_trust_keys,
|
||||
)
|
||||
ledger["reviewed_at"] = reviewed_at
|
||||
ledger["not_applicable"] = sorted(not_applicable)
|
||||
ledger["removed_not_applicable"] = removed_not_applicable
|
||||
return ledger
|
||||
|
||||
|
||||
def write_trust_ledger(
|
||||
path: Path,
|
||||
ledger: dict[str, Any],
|
||||
*,
|
||||
vault: Path | None = None,
|
||||
) -> None:
|
||||
"""Write a ledger atomically without following vault-internal symlinks."""
|
||||
vault_root = (vault or path.parent.parent).expanduser().resolve(strict=True)
|
||||
expected = vault_root / TRUST_LEDGER_RELATIVE_PATH
|
||||
requested = path.expanduser().absolute()
|
||||
if requested.resolve(strict=False) != expected:
|
||||
raise RuntimeError("trust ledger destination resolves outside the resolved vault")
|
||||
|
||||
parent = expected.parent
|
||||
if parent.exists() and parent.is_symlink():
|
||||
raise RuntimeError("trust ledger parent must not be a symlink")
|
||||
parent.mkdir(parents=True, exist_ok=True)
|
||||
if parent.resolve(strict=True) != vault_root / TRUST_LEDGER_RELATIVE_PATH.parent:
|
||||
raise RuntimeError("trust ledger parent resolves outside the resolved vault")
|
||||
if expected.is_symlink():
|
||||
raise RuntimeError("trust ledger destination must not be a symlink")
|
||||
|
||||
try:
|
||||
payload = json.dumps(ledger, indent=2, sort_keys=True, allow_nan=False) + "\n"
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise RuntimeError(f"trust ledger is not valid JSON data: {exc}") from exc
|
||||
|
||||
if os.name == "posix":
|
||||
directory_flags = os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) | getattr(os, "O_NOFOLLOW", 0)
|
||||
try:
|
||||
directory_fd = os.open(parent, directory_flags)
|
||||
except OSError as exc:
|
||||
raise RuntimeError(f"cannot securely open trust ledger directory: {exc}") from exc
|
||||
temporary_name = f".trust-ledger-{secrets.token_hex(16)}.tmp"
|
||||
temporary_created = False
|
||||
try:
|
||||
try:
|
||||
destination_stat = os.stat(
|
||||
expected.name,
|
||||
dir_fd=directory_fd,
|
||||
follow_symlinks=False,
|
||||
)
|
||||
except FileNotFoundError:
|
||||
destination_stat = None
|
||||
if destination_stat is not None and stat.S_ISLNK(destination_stat.st_mode):
|
||||
raise RuntimeError("trust ledger destination must not be a symlink")
|
||||
|
||||
descriptor = os.open(
|
||||
temporary_name,
|
||||
os.O_WRONLY
|
||||
| os.O_CREAT
|
||||
| os.O_EXCL
|
||||
| getattr(os, "O_NOFOLLOW", 0),
|
||||
0o600,
|
||||
dir_fd=directory_fd,
|
||||
)
|
||||
temporary_created = True
|
||||
with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
|
||||
handle.write(payload)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
os.replace(
|
||||
temporary_name,
|
||||
expected.name,
|
||||
src_dir_fd=directory_fd,
|
||||
dst_dir_fd=directory_fd,
|
||||
)
|
||||
temporary_created = False
|
||||
os.fsync(directory_fd)
|
||||
except OSError as exc:
|
||||
raise RuntimeError(f"cannot securely write trust ledger: {exc}") from exc
|
||||
finally:
|
||||
if temporary_created:
|
||||
try:
|
||||
os.unlink(temporary_name, dir_fd=directory_fd)
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
os.close(directory_fd)
|
||||
return
|
||||
|
||||
try:
|
||||
descriptor, temporary_name = tempfile.mkstemp(
|
||||
dir=parent,
|
||||
prefix=".trust-ledger-",
|
||||
suffix=".tmp",
|
||||
text=True,
|
||||
)
|
||||
except OSError as exc:
|
||||
raise RuntimeError(f"cannot create trust ledger temporary file: {exc}") from exc
|
||||
temporary = Path(temporary_name)
|
||||
try:
|
||||
with os.fdopen(descriptor, "w", encoding="utf-8") as handle:
|
||||
handle.write(payload)
|
||||
handle.flush()
|
||||
os.fsync(handle.fileno())
|
||||
temporary.replace(expected)
|
||||
except OSError as exc:
|
||||
temporary.unlink(missing_ok=True)
|
||||
raise RuntimeError(f"cannot write trust ledger: {exc}") from exc
|
||||
|
||||
|
||||
def _empty_report(
|
||||
ledger_path: Path,
|
||||
*,
|
||||
allowed_lifecycles: Collection[str],
|
||||
required_trust_keys: Collection[str],
|
||||
schema_source: str,
|
||||
) -> dict[str, Any]:
|
||||
return {
|
||||
"status": "pass",
|
||||
"ledger_path": str(ledger_path),
|
||||
"schema": {
|
||||
"source": schema_source,
|
||||
"allowed_lifecycles": sorted(allowed_lifecycles),
|
||||
"required_trust_fields": list(required_trust_keys),
|
||||
},
|
||||
"reviewed": [],
|
||||
"not_applicable": [],
|
||||
"stale": [],
|
||||
"unreviewed": [],
|
||||
"score_mismatches": [],
|
||||
"missing_pages": [],
|
||||
"errors": [],
|
||||
"counts": {},
|
||||
}
|
||||
|
||||
|
||||
def check_trust_ledger(
|
||||
vault: Path,
|
||||
ledger_path: Path | None = None,
|
||||
*,
|
||||
allowed_lifecycles: Collection[str] | None = None,
|
||||
required_trust_keys: Collection[str] | None = None,
|
||||
schema_source: str = "framework-defaults",
|
||||
) -> dict[str, Any]:
|
||||
"""Validate current pages against an approved manual review ledger."""
|
||||
lifecycles, required = _effective_schema(
|
||||
allowed_lifecycles=allowed_lifecycles,
|
||||
required_trust_keys=required_trust_keys,
|
||||
)
|
||||
path = ledger_path or vault / TRUST_LEDGER_RELATIVE_PATH
|
||||
report = _empty_report(
|
||||
path,
|
||||
allowed_lifecycles=lifecycles,
|
||||
required_trust_keys=required,
|
||||
schema_source=schema_source,
|
||||
)
|
||||
try:
|
||||
ledger = json.loads(
|
||||
path.read_text(encoding="utf-8"),
|
||||
parse_constant=_reject_json_constant,
|
||||
object_pairs_hook=_reject_duplicate_json_keys,
|
||||
)
|
||||
except FileNotFoundError:
|
||||
report["errors"].append({"issue": "ledger_missing", "path": str(path)})
|
||||
return _finalise_report(report)
|
||||
except (OSError, UnicodeError, ValueError) as exc:
|
||||
report["errors"].append({"issue": "ledger_unreadable", "detail": str(exc)})
|
||||
return _finalise_report(report)
|
||||
|
||||
if not isinstance(ledger, dict):
|
||||
report["errors"].append({"issue": "ledger_must_be_an_object"})
|
||||
return _finalise_report(report)
|
||||
if type(ledger.get("schema_version")) is not int or ledger.get("schema_version") != TRUST_LEDGER_SCHEMA_VERSION:
|
||||
report["errors"].append(
|
||||
{
|
||||
"issue": "unsupported_schema_version",
|
||||
"found": ledger.get("schema_version"),
|
||||
"expected": TRUST_LEDGER_SCHEMA_VERSION,
|
||||
}
|
||||
)
|
||||
return _finalise_report(report)
|
||||
if ledger.get("method") != TRUST_REVIEW_METHOD:
|
||||
report["errors"].append(
|
||||
{"issue": "unsupported_review_method", "found": ledger.get("method")}
|
||||
)
|
||||
return _finalise_report(report)
|
||||
try:
|
||||
_validate_reviewed_at(ledger.get("reviewed_at"))
|
||||
except ValueError:
|
||||
report["errors"].append({"issue": "invalid_reviewed_at"})
|
||||
return _finalise_report(report)
|
||||
entries = ledger.get("pages")
|
||||
if not isinstance(entries, dict):
|
||||
report["errors"].append({"issue": "pages_must_be_an_object"})
|
||||
return _finalise_report(report)
|
||||
|
||||
validated_entries: dict[str, tuple[str, float, str]] = {}
|
||||
for raw_rel, entry in entries.items():
|
||||
try:
|
||||
rel = _validate_ledger_page_path(raw_rel)
|
||||
validated_entries[rel] = _validate_ledger_entry(entry)
|
||||
except ValueError as exc:
|
||||
report["errors"].append(
|
||||
{"page": raw_rel, "issue": str(exc)}
|
||||
)
|
||||
if report["errors"]:
|
||||
return _finalise_report(report)
|
||||
|
||||
current: dict[str, Path] = {
|
||||
page.relative_to(vault).as_posix(): page for page in iter_trust_pages(vault)
|
||||
}
|
||||
for rel, page in current.items():
|
||||
try:
|
||||
page_metadata = _trust_metadata(
|
||||
page,
|
||||
allowed_lifecycles=lifecycles,
|
||||
required_trust_keys=required,
|
||||
)
|
||||
except ValueError as exc:
|
||||
report["errors"].append({"page": rel, "issue": str(exc)})
|
||||
continue
|
||||
if page_metadata["review_status"] == "not_applicable":
|
||||
report["not_applicable"].append(
|
||||
{"page": rel, "reason": "base_confidence_absent_by_owner_schema"}
|
||||
)
|
||||
if rel in validated_entries:
|
||||
report["stale"].append(
|
||||
{
|
||||
"page": rel,
|
||||
"reason": "confidence_not_applicable_but_ledger_entry_exists",
|
||||
}
|
||||
)
|
||||
continue
|
||||
stored_confidence = float(page_metadata["confidence"])
|
||||
if rel not in validated_entries:
|
||||
report["unreviewed"].append({"page": rel, "reason": "not_in_manual_ledger"})
|
||||
continue
|
||||
fingerprint, reviewed_value, reviewed_at = validated_entries[rel]
|
||||
if page_fingerprint(
|
||||
page,
|
||||
allowed_lifecycles=lifecycles,
|
||||
required_trust_keys=required,
|
||||
) != fingerprint:
|
||||
report["stale"].append({"page": rel, "reason": "material_fingerprint_changed"})
|
||||
continue
|
||||
if abs(stored_confidence - reviewed_value) > 1e-9:
|
||||
report["score_mismatches"].append(
|
||||
{
|
||||
"page": rel,
|
||||
"stored": stored_confidence,
|
||||
"reviewed": reviewed_value,
|
||||
}
|
||||
)
|
||||
continue
|
||||
report["reviewed"].append(
|
||||
{
|
||||
"page": rel,
|
||||
"reviewed_confidence": stored_confidence,
|
||||
"reviewed_at": reviewed_at,
|
||||
}
|
||||
)
|
||||
|
||||
for rel in sorted(set(entries) - set(current)):
|
||||
report["missing_pages"].append({"page": rel, "reason": "reviewed_page_missing"})
|
||||
return _finalise_report(report)
|
||||
|
||||
|
||||
def _finalise_report(report: dict[str, Any]) -> dict[str, Any]:
|
||||
keys = (
|
||||
"reviewed",
|
||||
"not_applicable",
|
||||
"stale",
|
||||
"unreviewed",
|
||||
"score_mismatches",
|
||||
"missing_pages",
|
||||
"errors",
|
||||
)
|
||||
report["counts"] = {key: len(report[key]) for key in keys}
|
||||
if report["errors"] or report["score_mismatches"]:
|
||||
report["status"] = "fail"
|
||||
elif report["stale"] or report["unreviewed"] or report["missing_pages"]:
|
||||
report["status"] = "warn"
|
||||
else:
|
||||
report["status"] = "pass"
|
||||
return report
|
||||
|
||||
|
||||
def check_lifecycle_transitions(
|
||||
vault: Path,
|
||||
ledger_path: Path | None = None,
|
||||
*,
|
||||
allowed_lifecycles: Collection[str] | None = None,
|
||||
required_trust_keys: Collection[str] | None = None,
|
||||
) -> list[dict[str, str]]:
|
||||
"""Report pages whose lifecycle moved along a forbidden edge.
|
||||
|
||||
Compares each page's current ``lifecycle`` against the value recorded in the
|
||||
approved ledger at its last review. Ledgers written before the ``lifecycle``
|
||||
field existed carry no baseline, so those pages are skipped — the check is
|
||||
additive and silent on legacy vaults.
|
||||
"""
|
||||
lifecycles, required = _effective_schema(
|
||||
allowed_lifecycles=allowed_lifecycles,
|
||||
required_trust_keys=required_trust_keys,
|
||||
)
|
||||
path = ledger_path or vault / TRUST_LEDGER_RELATIVE_PATH
|
||||
try:
|
||||
ledger = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, UnicodeError, ValueError):
|
||||
return [] # check_trust_ledger already reports unreadable ledgers
|
||||
entries = ledger.get("pages") if isinstance(ledger, dict) else None
|
||||
if not isinstance(entries, dict):
|
||||
return []
|
||||
|
||||
findings: list[dict[str, str]] = []
|
||||
for page in iter_trust_pages(vault):
|
||||
rel = page.relative_to(vault).as_posix()
|
||||
entry = entries.get(rel)
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
previous = entry.get("lifecycle")
|
||||
if not isinstance(previous, str):
|
||||
continue # pre-lifecycle ledger entry: no baseline to compare
|
||||
try:
|
||||
metadata = _trust_metadata(
|
||||
page,
|
||||
allowed_lifecycles=lifecycles,
|
||||
required_trust_keys=required,
|
||||
)
|
||||
except ValueError:
|
||||
continue # malformed frontmatter is another check's finding
|
||||
current = metadata["lifecycle"]
|
||||
if not isinstance(current, str):
|
||||
continue
|
||||
if (previous, current) in ILLEGAL_TRANSITIONS:
|
||||
findings.append({"page": rel, "from": previous, "to": current})
|
||||
return sorted(findings, key=lambda f: f["page"])
|
||||
@@ -0,0 +1,127 @@
|
||||
"""Shared vault primitives: frontmatter splitting, skip dirs, and the page walker.
|
||||
|
||||
Every module that reads pages out of a vault goes through these, so a vault
|
||||
parses and walks the same way no matter which command touches it (#238).
|
||||
Modules layer their own extra skip dirs / reserved files on top.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fnmatch
|
||||
import os
|
||||
import re
|
||||
from pathlib import Path
|
||||
from typing import Iterable
|
||||
|
||||
#: Staging, archive, and tool-owned directories that never hold knowledge pages.
|
||||
#: Dot-prefixed paths (`.venv`, `.git`, `.obsidian`) are skipped wholesale by
|
||||
#: `iter_md`; this list adds the non-hidden equivalents.
|
||||
SKIP_DIRS = frozenset({
|
||||
"_raw", "_archived", "_staging", "_archives",
|
||||
"_sources", "_meta", "_bot_exports", "_inbox",
|
||||
".obsidian", ".git", "venv", "node_modules", "__pycache__",
|
||||
})
|
||||
|
||||
#: CRLF-tolerant so a vault edited on Windows parses the same as one from Unix.
|
||||
FRONTMATTER_RE = re.compile(r"^---\r?\n(.*?)\r?\n---(?:\r?\n|$)", re.DOTALL)
|
||||
# A bare `>`, `>-`, `>+`, `|`, `|-`, `|+` (optionally followed by an indent
|
||||
# indicator digit) marks a YAML block scalar — the real value lives on the
|
||||
# following indented lines, not on this line.
|
||||
BLOCK_SCALAR_RE = re.compile(r"^[>|][+-]?\d*$")
|
||||
|
||||
|
||||
def split_frontmatter(text: str) -> tuple[str, str]:
|
||||
"""(frontmatter, body); frontmatter is "" when the page has none."""
|
||||
match = FRONTMATTER_RE.match(text)
|
||||
return (match.group(1), text[match.end():]) if match else ("", text)
|
||||
|
||||
|
||||
def okignore_patterns(vault: Path) -> list[str]:
|
||||
"""Patterns from the vault-root `.okignore` (gitignore syntax, subset)."""
|
||||
try:
|
||||
lines = (vault / ".okignore").read_text(encoding="utf-8").splitlines()
|
||||
except OSError:
|
||||
return []
|
||||
# ponytail: no `!` negation or `**`-specific semantics; add if a vault needs them.
|
||||
return [
|
||||
line.strip().rstrip("/") for line in lines
|
||||
if line.strip() and not line.strip().startswith(("#", "!"))
|
||||
]
|
||||
|
||||
|
||||
def okignored(rel: Path, patterns: list[str]) -> bool:
|
||||
"""True if vault-relative `rel` is excluded by any `.okignore` pattern.
|
||||
|
||||
A pattern without a slash matches any path component (`_inbox`, `*.draft.md`);
|
||||
one with a slash is anchored to the vault root and matches that path or
|
||||
anything beneath it (`/drafts`, `notes/old`).
|
||||
"""
|
||||
parts = rel.parts
|
||||
prefixes = ["/".join(parts[: i + 1]) for i in range(len(parts))]
|
||||
for pattern in patterns:
|
||||
if "/" in pattern:
|
||||
anchored = pattern.lstrip("/")
|
||||
if any(fnmatch.fnmatchcase(prefix, anchored) for prefix in prefixes):
|
||||
return True
|
||||
elif any(fnmatch.fnmatchcase(part, pattern) for part in parts):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
#: Instructions for coding agents, not knowledge, wherever they sit: a vault
|
||||
#: built from per-project sub-wikis carries one per project. A `README.md` is
|
||||
#: only excluded at the vault root, where it describes the vault itself.
|
||||
AGENT_INSTRUCTION_FILES = frozenset({"AGENTS.md", "CLAUDE.md", "GEMINI.md"})
|
||||
|
||||
|
||||
def is_instruction_file(rel: Path) -> bool:
|
||||
"""True if vault-relative `rel` is an agent-instruction file, not a page."""
|
||||
return rel.name in AGENT_INSTRUCTION_FILES or (len(rel.parts) == 1 and rel.name == "README.md")
|
||||
|
||||
|
||||
def skipped_dir(rel: Path, skip_dirs: Iterable[str] = SKIP_DIRS) -> bool:
|
||||
"""True if any component of vault-relative `rel` is hidden or in `skip_dirs`."""
|
||||
return any(part in skip_dirs or part.startswith(".") for part in rel.parts)
|
||||
|
||||
|
||||
def _walk_md(vault: Path) -> Iterable[Path]:
|
||||
"""Every `.md` under `vault`, descending into symlinked directories.
|
||||
|
||||
Obsidian follows a symlinked folder as part of the vault, but `Path.rglob`
|
||||
does not descend into one before Python 3.13, so a vault that mounts its
|
||||
project folders by symlink looked empty and `memory index` dropped their
|
||||
entries. A directory whose real path was already visited is pruned, which
|
||||
breaks symlink loops.
|
||||
"""
|
||||
seen: set[str] = set()
|
||||
for root, dirs, files in os.walk(vault, followlinks=True):
|
||||
real = os.path.realpath(root)
|
||||
if real in seen:
|
||||
dirs[:] = []
|
||||
continue
|
||||
seen.add(real)
|
||||
for name in files:
|
||||
if name.endswith(".md"):
|
||||
yield Path(root) / name
|
||||
|
||||
|
||||
def iter_instruction_files(vault: Path, skip_dirs: Iterable[str] = SKIP_DIRS) -> list[Path]:
|
||||
"""The agent-instruction files `iter_md` leaves out. Not pages, but real
|
||||
files a `[[CLAUDE]]` link opens, so link checks must still find them."""
|
||||
return sorted(
|
||||
path for path in _walk_md(vault)
|
||||
if is_instruction_file(path.relative_to(vault))
|
||||
and not skipped_dir(path.relative_to(vault), skip_dirs)
|
||||
)
|
||||
|
||||
|
||||
def iter_md(vault: Path, skip_dirs: Iterable[str] = SKIP_DIRS) -> list[Path]:
|
||||
"""Every `.md` page under `vault`, sorted, minus hidden/skipped dirs,
|
||||
agent-instruction files, and `.okignore`."""
|
||||
patterns = okignore_patterns(vault)
|
||||
return sorted(
|
||||
path for path in _walk_md(vault)
|
||||
if not skipped_dir(path.relative_to(vault), skip_dirs)
|
||||
and not is_instruction_file(path.relative_to(vault))
|
||||
and not okignored(path.relative_to(vault), patterns)
|
||||
)
|
||||
@@ -48,6 +48,9 @@ Issues = "https://github.com/Ar9av/obsidian-wiki/issues"
|
||||
obsidian-wiki = "obsidian_wiki.cli:main"
|
||||
obsidian-wiki-mcp = "obsidian_wiki.mcp_server:main"
|
||||
|
||||
[dependency-groups]
|
||||
dev = ["pytest>=8,<9"]
|
||||
|
||||
# Version is derived from the git tag (CalVer, e.g. v2026.05.2 -> 2026.5.2),
|
||||
# matching the repo's release tags. Tagging a release is the version bump.
|
||||
[tool.hatch.version]
|
||||
|
||||
@@ -29,6 +29,9 @@ Usage:
|
||||
scripts/obsidian-wiki-ops.py setup [args...]
|
||||
scripts/obsidian-wiki-ops.py info
|
||||
scripts/obsidian-wiki-ops.py file-index rebuild [args...]
|
||||
scripts/obsidian-wiki-ops.py lint [vault] --json
|
||||
scripts/obsidian-wiki-ops.py memory status --vault /path/to/vault
|
||||
scripts/obsidian-wiki-ops.py staging list --vault /path/to/vault --json
|
||||
|
||||
|
||||
scripts/obsidian-wiki-ops.py mcp start [args...]
|
||||
@@ -51,6 +54,8 @@ CLI_COMMANDS = {
|
||||
"file-index",
|
||||
"mcp-server",
|
||||
"bot-view",
|
||||
"graph-analyse", "cache-check", "projects-check", "cache-update",
|
||||
"cache-hash", "staging", "lint", "memory",
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,536 @@
|
||||
"""Tests for the content-hash cache module."""
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from obsidian_wiki.cache import (
|
||||
ManifestLockTimeout,
|
||||
check_sources,
|
||||
compute_hash,
|
||||
hash_file,
|
||||
manifest_lock,
|
||||
sha256_file,
|
||||
sha256_dir,
|
||||
update_source,
|
||||
_load_manifest,
|
||||
_lock_path,
|
||||
_manifest_path,
|
||||
_write_manifest,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def vault(tmp_path):
|
||||
v = tmp_path / "vault"
|
||||
v.mkdir()
|
||||
return v
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def src_file(tmp_path):
|
||||
f = tmp_path / "doc.md"
|
||||
f.write_text("# Hello\nSome content.", encoding="utf-8")
|
||||
return f
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def src_dir(tmp_path):
|
||||
d = tmp_path / "repo"
|
||||
d.mkdir()
|
||||
(d / "a.py").write_text("x = 1")
|
||||
(d / "b.py").write_text("y = 2")
|
||||
return d
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Hash functions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestHashing:
|
||||
def test_sha256_file_deterministic(self, src_file):
|
||||
assert sha256_file(src_file) == sha256_file(src_file)
|
||||
|
||||
def test_sha256_file_changes_on_edit(self, src_file):
|
||||
h1 = sha256_file(src_file)
|
||||
src_file.write_text("# Different content")
|
||||
h2 = sha256_file(src_file)
|
||||
assert h1 != h2
|
||||
|
||||
def test_sha256_dir_deterministic(self, src_dir):
|
||||
assert sha256_dir(src_dir) == sha256_dir(src_dir)
|
||||
|
||||
def test_sha256_dir_changes_on_edit(self, src_dir):
|
||||
h1 = sha256_dir(src_dir)
|
||||
(src_dir / "a.py").write_text("x = 999")
|
||||
h2 = sha256_dir(src_dir)
|
||||
assert h1 != h2
|
||||
|
||||
def test_compute_hash_dispatches(self, src_file, src_dir):
|
||||
assert len(compute_hash(src_file)) == 64 # hex SHA-256
|
||||
assert len(compute_hash(src_dir)) == 64
|
||||
|
||||
def test_sha256_dir_independent_of_path_separator(self, src_dir):
|
||||
"""Hash must match hashlib computed with POSIX separators regardless of platform (#178)."""
|
||||
import hashlib
|
||||
|
||||
(src_dir / "sub").mkdir()
|
||||
(src_dir / "sub" / "c.py").write_text("z = 3")
|
||||
|
||||
h = hashlib.sha256()
|
||||
for fp in sorted(src_dir.rglob("*"), key=lambda p: p.relative_to(src_dir).as_posix()):
|
||||
if fp.is_file():
|
||||
h.update(fp.relative_to(src_dir).as_posix().encode())
|
||||
h.update(sha256_file(fp).encode())
|
||||
assert sha256_dir(src_dir) == h.hexdigest()
|
||||
|
||||
def test_hash_file_alias(self, src_file):
|
||||
assert hash_file(src_file) == sha256_file(src_file)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# check_sources
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestCheckSources:
|
||||
def test_new_source(self, vault, src_file):
|
||||
result = check_sources(vault, [src_file])
|
||||
assert str(src_file) in result["new"]
|
||||
assert result["modified"] == []
|
||||
assert result["unchanged"] == []
|
||||
|
||||
def test_unchanged_after_update(self, vault, src_file):
|
||||
update_source(vault, src_file)
|
||||
result = check_sources(vault, [src_file])
|
||||
assert str(src_file) in result["unchanged"]
|
||||
assert result["new"] == []
|
||||
assert result["modified"] == []
|
||||
|
||||
def test_modified_after_content_change(self, vault, src_file):
|
||||
update_source(vault, src_file)
|
||||
src_file.write_text("# Changed content")
|
||||
result = check_sources(vault, [src_file])
|
||||
assert str(src_file) in result["modified"]
|
||||
|
||||
def test_missing_path(self, vault, tmp_path):
|
||||
ghost = tmp_path / "ghost.md"
|
||||
result = check_sources(vault, [ghost])
|
||||
assert str(ghost) in result["missing"]
|
||||
|
||||
def test_empty_source_list(self, vault):
|
||||
result = check_sources(vault, [])
|
||||
assert result == {
|
||||
"new": [], "modified": [], "unchanged": [], "missing": [], "unavailable": []
|
||||
}
|
||||
|
||||
def test_multiple_sources(self, vault, src_file, src_dir):
|
||||
update_source(vault, src_file)
|
||||
result = check_sources(vault, [src_file, src_dir])
|
||||
assert str(src_file) in result["unchanged"]
|
||||
assert str(src_dir) in result["new"]
|
||||
|
||||
def test_timestamp_irrelevant(self, vault, src_file):
|
||||
# Touch the file (change mtime) without changing content — still unchanged
|
||||
update_source(vault, src_file)
|
||||
src_file.touch()
|
||||
result = check_sources(vault, [src_file])
|
||||
assert str(src_file) in result["unchanged"]
|
||||
|
||||
def _write_relative_manifest(self, vault, rel_key, content_hash):
|
||||
"""Write a manifest whose source key is stored vault-relative."""
|
||||
_manifest_path(vault).write_text(
|
||||
json.dumps(
|
||||
{"sources": {rel_key: {"content_hash": content_hash, "last_ingested": "2026-07-14"}}}
|
||||
),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
def test_relative_manifest_key_unchanged_for_abs_path(self, vault):
|
||||
# Manifest stores a vault-relative key; caller passes the absolute path.
|
||||
src = vault / "_raw" / "articles" / "foo.md"
|
||||
src.parent.mkdir(parents=True)
|
||||
src.write_text("body", encoding="utf-8")
|
||||
self._write_relative_manifest(vault, "_raw/articles/foo.md", sha256_file(src))
|
||||
result = check_sources(vault, [src])
|
||||
assert str(src) in result["unchanged"]
|
||||
assert result["new"] == []
|
||||
assert result["missing"] == []
|
||||
|
||||
def test_relative_manifest_key_not_falsely_missing(self, vault):
|
||||
# A relative key whose file exists under the vault must not be flagged missing,
|
||||
# even when CWD != vault root.
|
||||
src = vault / "_raw" / "articles" / "foo.md"
|
||||
src.parent.mkdir(parents=True)
|
||||
src.write_text("body", encoding="utf-8")
|
||||
self._write_relative_manifest(vault, "_raw/articles/foo.md", sha256_file(src))
|
||||
result = check_sources(vault, [])
|
||||
assert "_raw/articles/foo.md" not in result["missing"]
|
||||
|
||||
def test_relative_manifest_key_modified(self, vault):
|
||||
src = vault / "_raw" / "articles" / "foo.md"
|
||||
src.parent.mkdir(parents=True)
|
||||
src.write_text("body", encoding="utf-8")
|
||||
self._write_relative_manifest(vault, "_raw/articles/foo.md", "stale-hash")
|
||||
result = check_sources(vault, [src])
|
||||
assert str(src) in result["modified"]
|
||||
|
||||
def test_relative_manifest_key_genuinely_missing(self, vault):
|
||||
# A relative key whose top-level directory really is part of this vault
|
||||
# is a vault-local loss when the file is gone.
|
||||
(vault / "_raw" / "articles").mkdir(parents=True)
|
||||
self._write_relative_manifest(vault, "_raw/articles/gone.md", "abc")
|
||||
result = check_sources(vault, [])
|
||||
assert "_raw/articles/gone.md" in result["missing"]
|
||||
|
||||
def test_relative_key_under_an_unknown_top_dir_is_unavailable(self, vault):
|
||||
# A relative key whose first segment is not a vault entry is relative to
|
||||
# some other root (a legacy ingest root, or an out-of-vault namespace), so
|
||||
# it is unavailable here rather than a missing vault source.
|
||||
self._write_relative_manifest(vault, "-Users-x/abc.jsonl", "abc")
|
||||
result = check_sources(vault, [])
|
||||
assert result["missing"] == []
|
||||
assert "-Users-x/abc.jsonl" in result["unavailable"]
|
||||
|
||||
def test_check_sources_resolves_each_path_once(self, vault, tmp_path, monkeypatch):
|
||||
"""Matching resolves each entry and each query once (an index of forms), not once
|
||||
per (entry, query) pair: with relative keys the string fast path never fires, and a
|
||||
3,000-entry manifest made every queried file cost a second of ``realpath`` walks."""
|
||||
src = tmp_path / "src"
|
||||
src.mkdir()
|
||||
files = []
|
||||
for i in range(300):
|
||||
f = src / f"f{i:03d}.md"
|
||||
f.write_text(f"doc {i}", encoding="utf-8")
|
||||
files.append(f)
|
||||
sources = {}
|
||||
for i, f in enumerate(files):
|
||||
# "../src/f000.md" — the relative form the CLI writes for out-of-vault sources
|
||||
sources[os.path.relpath(f, vault)] = {"content_hash": sha256_file(f) if i % 3 else "0" * 64}
|
||||
for i in range(2000): # unrelated entries, like a real manifest
|
||||
sources[f"../elsewhere/e{i:04d}.md"] = {"content_hash": "1" * 64}
|
||||
_write_manifest(vault, {"version": 1, "sources": sources})
|
||||
|
||||
calls = {"n": 0}
|
||||
real_resolve = Path.resolve
|
||||
|
||||
def counting_resolve(self, *args, **kwargs):
|
||||
calls["n"] += 1
|
||||
return real_resolve(self, *args, **kwargs)
|
||||
|
||||
monkeypatch.setattr(Path, "resolve", counting_resolve)
|
||||
result = check_sources(vault, files)
|
||||
|
||||
assert len(result["unchanged"]) == 200
|
||||
assert len(result["modified"]) == 100
|
||||
assert result["new"] == []
|
||||
assert len(result["missing"]) + len(result["unavailable"]) == 2000
|
||||
# one resolve per entry and per query (plus a handful inside helpers), never entries × queries
|
||||
assert calls["n"] <= 3 * (2300 + 300), calls["n"]
|
||||
|
||||
def test_check_sources_first_entry_wins_for_duplicate_forms(self, vault, tmp_path):
|
||||
"""Two entries naming the same file in different forms: the earlier one is the match,
|
||||
the later one is not reported missing (it is a duplicate of a scanned path)."""
|
||||
f = tmp_path / "dup.md"
|
||||
f.write_text("same", encoding="utf-8")
|
||||
sources = {
|
||||
os.path.relpath(f, vault): {"content_hash": "0" * 64}, # earlier: stale hash → modified
|
||||
str(f): {"content_hash": sha256_file(f)}, # later: same file, absolute form
|
||||
}
|
||||
_write_manifest(vault, {"version": 1, "sources": sources})
|
||||
result = check_sources(vault, [f])
|
||||
assert result["modified"] == [str(f)] and result["unchanged"] == []
|
||||
assert result["missing"] == [] and result["unavailable"] == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# update_source / manifest
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestUpdateSource:
|
||||
def test_writes_manifest(self, vault, src_file):
|
||||
update_source(vault, src_file)
|
||||
assert _manifest_path(vault).exists()
|
||||
|
||||
def test_records_correct_hash(self, vault, src_file):
|
||||
h = update_source(vault, src_file)
|
||||
assert h == sha256_file(src_file)
|
||||
sources = _load_manifest(vault)
|
||||
assert sources[str(src_file)]["content_hash"] == h
|
||||
|
||||
def test_records_pages_produced(self, vault, src_file):
|
||||
update_source(vault, src_file, pages_produced=["concepts/foo.md", "entities/bar.md"])
|
||||
sources = _load_manifest(vault)
|
||||
assert sources[str(src_file)]["pages_produced"] == ["concepts/foo.md", "entities/bar.md"]
|
||||
|
||||
def test_records_last_ingested_timestamp(self, vault, src_file):
|
||||
update_source(vault, src_file)
|
||||
sources = _load_manifest(vault)
|
||||
assert "last_ingested" in sources[str(src_file)]
|
||||
|
||||
def test_update_overwrites_old_hash(self, vault, src_file):
|
||||
update_source(vault, src_file)
|
||||
src_file.write_text("new content")
|
||||
h2 = update_source(vault, src_file)
|
||||
sources = _load_manifest(vault)
|
||||
assert sources[str(src_file)]["content_hash"] == h2
|
||||
|
||||
def test_preserves_other_manifest_entries(self, vault, src_file, src_dir):
|
||||
update_source(vault, src_file)
|
||||
update_source(vault, src_dir)
|
||||
sources = _load_manifest(vault)
|
||||
assert str(src_file) in sources
|
||||
assert str(src_dir) in sources
|
||||
|
||||
def test_update_source_live_draft_beats_same_named_archive(self, vault):
|
||||
staging = vault / "_raw" / "notes.md"
|
||||
archived = vault / "_raw" / "_archived" / "notes.md"
|
||||
archived.parent.mkdir(parents=True, exist_ok=True)
|
||||
staging.parent.mkdir(parents=True, exist_ok=True)
|
||||
staging.write_text("staging-bytes\n", encoding="utf-8")
|
||||
archived.write_text("archived-bytes\n", encoding="utf-8")
|
||||
h = update_source(vault, staging)
|
||||
assert h == compute_hash(staging)
|
||||
sources = _load_manifest(vault)
|
||||
assert "_raw/notes.md" in sources
|
||||
assert "_raw/_archived/notes.md" not in sources
|
||||
|
||||
def test_update_source_hashes_archive_when_staging_gone(self, vault):
|
||||
staging = vault / "_raw" / "notes.md"
|
||||
archived = vault / "_raw" / "_archived" / "notes.md"
|
||||
archived.parent.mkdir(parents=True, exist_ok=True)
|
||||
staging.parent.mkdir(parents=True, exist_ok=True)
|
||||
staging.write_text("staging-bytes\n", encoding="utf-8")
|
||||
archived.write_text("archived-bytes\n", encoding="utf-8")
|
||||
update_source(vault, staging)
|
||||
staging.unlink()
|
||||
h = update_source(vault, staging)
|
||||
assert h == compute_hash(archived)
|
||||
assert "_raw/_archived/notes.md" in _load_manifest(vault)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestCacheCLI:
|
||||
def _run(self, *args, cwd=None):
|
||||
return subprocess.run(
|
||||
[sys.executable, "-m", "obsidian_wiki.cli", *args],
|
||||
capture_output=True, text=True, cwd=cwd,
|
||||
)
|
||||
|
||||
def test_cache_hash_file(self, src_file):
|
||||
proc = self._run("cache-hash", str(src_file))
|
||||
assert proc.returncode == 0
|
||||
data = json.loads(proc.stdout)
|
||||
assert data["sha256"] == sha256_file(src_file)
|
||||
|
||||
def test_cache_hash_missing_exits_nonzero(self, tmp_path):
|
||||
proc = self._run("cache-hash", str(tmp_path / "nope.md"))
|
||||
assert proc.returncode != 0
|
||||
|
||||
def test_cache_check_new(self, vault, src_file):
|
||||
proc = self._run("cache-check", str(vault), str(src_file))
|
||||
assert proc.returncode == 0
|
||||
data = json.loads(proc.stdout)
|
||||
assert str(src_file) in data["new"]
|
||||
|
||||
def test_cache_check_pretty(self, vault, src_file):
|
||||
proc = self._run("cache-check", "--pretty", str(vault), str(src_file))
|
||||
assert proc.returncode == 0
|
||||
assert "\n " in proc.stdout
|
||||
|
||||
def test_cache_update_then_check_unchanged(self, vault, src_file):
|
||||
self._run("cache-update", str(vault), str(src_file))
|
||||
proc = self._run("cache-check", str(vault), str(src_file))
|
||||
data = json.loads(proc.stdout)
|
||||
assert str(src_file) in data["unchanged"]
|
||||
|
||||
def test_cache_update_with_pages(self, vault, src_file):
|
||||
proc = self._run("cache-update", str(vault), str(src_file),
|
||||
"--pages", "concepts/foo.md", "entities/bar.md")
|
||||
assert proc.returncode == 0
|
||||
sources = _load_manifest(vault)
|
||||
assert sources[str(src_file)]["pages_produced"] == ["concepts/foo.md", "entities/bar.md"]
|
||||
|
||||
def test_cache_update_non_portable_source_warns_on_stderr(self, vault, src_file):
|
||||
# src_file lives outside the vault and $HOME, so the fallback absolute
|
||||
# key is stored — but the CLI must say so rather than stay silent.
|
||||
proc = self._run("cache-update", str(vault), str(src_file))
|
||||
assert proc.returncode == 0
|
||||
assert "no portable key" in proc.stderr
|
||||
|
||||
def test_cache_update_explicit_key_warns_nothing(self, vault, src_file):
|
||||
proc = self._run("cache-update", str(vault), str(src_file), "--key", "repo:o/n")
|
||||
assert proc.returncode == 0
|
||||
assert "no portable key" not in proc.stderr
|
||||
assert "repo:o/n" in _load_manifest(vault)
|
||||
|
||||
def test_cache_update_cli_uses_archive_if_staging_gone(self, vault):
|
||||
staging = vault / "_raw" / "notes.md"
|
||||
archived = vault / "_raw" / "_archived" / "notes.md"
|
||||
archived.parent.mkdir(parents=True, exist_ok=True)
|
||||
staging.parent.mkdir(parents=True, exist_ok=True)
|
||||
staging.write_text("staging-bytes\n", encoding="utf-8")
|
||||
archived.write_text("archived-bytes\n", encoding="utf-8")
|
||||
staging.unlink()
|
||||
proc = self._run("cache-update", str(vault), str(staging))
|
||||
assert proc.returncode == 0
|
||||
data = json.loads(proc.stdout)
|
||||
assert data["key"] == "_raw/_archived/notes.md"
|
||||
assert data["content_hash"] == compute_hash(archived)
|
||||
|
||||
|
||||
class TestCacheCLIVaultRelativeSources:
|
||||
"""Relative source arguments must also resolve against the vault root.
|
||||
|
||||
The manifest stores vault-relative keys, and ``cache.resolve_key`` reads
|
||||
them that way — so ``cache-update <vault> Raw/x.pdf`` has to work from any
|
||||
CWD, not just when the shell happens to sit at the vault root.
|
||||
"""
|
||||
|
||||
@pytest.fixture
|
||||
def vault(self, tmp_path):
|
||||
v = tmp_path / "vault"
|
||||
(v / "Raw" / "database").mkdir(parents=True)
|
||||
(v / "Raw" / "database" / "x.pdf").write_bytes(b"pdf-bytes")
|
||||
return v
|
||||
|
||||
def _run(self, *args, cwd=None):
|
||||
return subprocess.run(
|
||||
[sys.executable, "-m", "obsidian_wiki.cli", *args],
|
||||
capture_output=True, text=True, cwd=cwd,
|
||||
)
|
||||
|
||||
def test_cache_update_vault_relative_from_other_cwd(self, vault, tmp_path):
|
||||
elsewhere = tmp_path / "elsewhere"
|
||||
elsewhere.mkdir()
|
||||
proc = self._run("cache-update", str(vault), "Raw/database/x.pdf",
|
||||
cwd=elsewhere)
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
data = json.loads(proc.stdout)
|
||||
assert data["key"] == "Raw/database/x.pdf"
|
||||
assert "Raw/database/x.pdf" in json.dumps(_load_manifest(vault))
|
||||
|
||||
def test_cache_check_vault_relative_from_other_cwd(self, vault, tmp_path):
|
||||
update_source(vault, vault / "Raw" / "database" / "x.pdf")
|
||||
elsewhere = tmp_path / "elsewhere"
|
||||
elsewhere.mkdir()
|
||||
proc = self._run("cache-check", str(vault), "Raw/database/x.pdf",
|
||||
cwd=elsewhere)
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
data = json.loads(proc.stdout)
|
||||
assert str(vault / "Raw" / "database" / "x.pdf") in data["unchanged"]
|
||||
assert data["missing"] == []
|
||||
|
||||
def test_cache_update_cwd_relative_still_wins(self, vault, tmp_path):
|
||||
# Backward compat: an out-of-vault relative path resolves against the
|
||||
# CWD as before, even when the vault contains a same-named file.
|
||||
workdir = tmp_path / "workdir"
|
||||
(workdir / "Raw" / "database").mkdir(parents=True)
|
||||
(workdir / "Raw" / "database" / "x.pdf").write_bytes(b"different-bytes")
|
||||
proc = self._run("cache-update", str(vault), "Raw/database/x.pdf",
|
||||
cwd=workdir)
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
data = json.loads(proc.stdout)
|
||||
assert data["path"] == str(workdir / "Raw" / "database" / "x.pdf")
|
||||
# The typed string is byte-for-byte a manifest key, yet the vault file
|
||||
# it names is not the one hashed — say so rather than choosing silently.
|
||||
assert (f"note: Raw/database/x.pdf matched both "
|
||||
f"{workdir / 'Raw' / 'database' / 'x.pdf'} and "
|
||||
f"{vault / 'Raw' / 'database' / 'x.pdf'}") in proc.stderr
|
||||
|
||||
def test_cache_update_missing_source_clean_error(self, vault, tmp_path):
|
||||
elsewhere = tmp_path / "elsewhere"
|
||||
elsewhere.mkdir()
|
||||
proc = self._run("cache-update", str(vault), "Raw/absent.md", cwd=elsewhere)
|
||||
assert proc.returncode == 1
|
||||
assert "Traceback" not in proc.stderr
|
||||
assert str(elsewhere / "Raw" / "absent.md") in proc.stderr
|
||||
assert str(vault / "Raw" / "absent.md") in proc.stderr
|
||||
|
||||
def test_cache_update_absolute_missing_clean_error(self, vault):
|
||||
proc = self._run("cache-update", str(vault), str(vault / "Raw" / "gone.pdf"))
|
||||
assert proc.returncode == 1
|
||||
assert "Traceback" not in proc.stderr
|
||||
assert str(vault / "Raw" / "gone.pdf") in proc.stderr
|
||||
|
||||
|
||||
class TestManifestLock:
|
||||
"""Concurrent writers must serialize instead of clobbering the manifest."""
|
||||
|
||||
def test_lock_is_released_after_use(self, vault):
|
||||
with manifest_lock(vault):
|
||||
assert _lock_path(vault).exists()
|
||||
assert not _lock_path(vault).exists()
|
||||
|
||||
def test_second_holder_times_out(self, vault):
|
||||
with manifest_lock(vault):
|
||||
with pytest.raises(ManifestLockTimeout):
|
||||
with manifest_lock(vault, timeout=0.3):
|
||||
pass
|
||||
|
||||
def test_stale_lock_is_stolen(self, vault):
|
||||
lock = _lock_path(vault)
|
||||
lock.write_text("999999")
|
||||
old = time.time() - 120
|
||||
os.utime(lock, (old, old))
|
||||
with manifest_lock(vault, timeout=0.5, stale_after=60.0):
|
||||
pass
|
||||
assert not lock.exists()
|
||||
|
||||
def test_lock_released_when_body_raises(self, vault):
|
||||
with pytest.raises(ValueError):
|
||||
with manifest_lock(vault):
|
||||
raise ValueError("boom")
|
||||
assert not _lock_path(vault).exists()
|
||||
|
||||
def test_update_source_leaves_no_lock_behind(self, vault, src_file):
|
||||
update_source(vault, src_file)
|
||||
assert not _lock_path(vault).exists()
|
||||
|
||||
def test_concurrent_updates_both_survive(self, vault, tmp_path):
|
||||
"""The race the lock exists for: two processes, neither entry lost."""
|
||||
a = tmp_path / "a.md"
|
||||
a.write_text("alpha", encoding="utf-8")
|
||||
b = tmp_path / "b.md"
|
||||
b.write_text("beta", encoding="utf-8")
|
||||
code = (
|
||||
"import sys;from pathlib import Path;"
|
||||
"from obsidian_wiki.cache import update_source;"
|
||||
"update_source(Path(sys.argv[1]), Path(sys.argv[2]))"
|
||||
)
|
||||
env = {**os.environ, "PYTHONPATH": str(Path(__file__).resolve().parent.parent)}
|
||||
procs = [
|
||||
subprocess.Popen([sys.executable, "-c", code, str(vault), str(src)], env=env)
|
||||
for src in (a, b)
|
||||
]
|
||||
for p in procs:
|
||||
assert p.wait(timeout=30) == 0
|
||||
sources = _load_manifest(vault)
|
||||
assert str(a) in sources and str(b) in sources
|
||||
|
||||
|
||||
class TestAtomicWrite:
|
||||
def test_failed_write_leaves_original_intact(self, vault, monkeypatch):
|
||||
_write_manifest(vault, {"sources": {"kept": {"content_hash": "abc"}}})
|
||||
|
||||
def boom(*args, **kwargs):
|
||||
raise OSError("disk full")
|
||||
|
||||
monkeypatch.setattr("obsidian_wiki.cache.os.replace", boom)
|
||||
with pytest.raises(OSError):
|
||||
_write_manifest(vault, {"sources": {"lost": {}}})
|
||||
|
||||
assert _load_manifest(vault) == {"kept": {"content_hash": "abc"}}
|
||||
leftovers = list(vault.glob(".manifest.json.*.tmp"))
|
||||
assert leftovers == []
|
||||
|
||||
def test_manifest_is_never_partially_written(self, vault, src_file):
|
||||
update_source(vault, src_file)
|
||||
# A complete, parseable JSON document — never a truncated prefix.
|
||||
json.loads(_manifest_path(vault).read_text(encoding="utf-8"))
|
||||
@@ -0,0 +1,172 @@
|
||||
"""Regression tests for issue #134.
|
||||
|
||||
cache-check / cache-update were written for a dict-keyed manifest with bare hex
|
||||
hashes, but the wiki-ingest skill writes ``sources`` as a list of objects with
|
||||
``sha256:``-prefixed hashes and vault-relative paths. That combination caused a
|
||||
crash, then false "modified"/"new" once the crash was patched. These tests pin
|
||||
all four failure modes the issue enumerated.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from obsidian_wiki.cache import (
|
||||
check_sources,
|
||||
compute_hash,
|
||||
update_source,
|
||||
_load_raw,
|
||||
_manifest_path,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def vault(tmp_path):
|
||||
v = tmp_path / "vault"
|
||||
v.mkdir()
|
||||
return v
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def raw_file(vault):
|
||||
f = vault / "_raw" / "foo.md"
|
||||
f.parent.mkdir(parents=True)
|
||||
f.write_text("# Foo\nbody", encoding="utf-8")
|
||||
return f
|
||||
|
||||
|
||||
def _write_list_manifest(vault: Path, entries: list[dict]) -> None:
|
||||
_manifest_path(vault).write_text(
|
||||
json.dumps({"version": 1, "sources": entries}, indent=2), encoding="utf-8"
|
||||
)
|
||||
|
||||
|
||||
# 1. Crash on list-shaped sources ------------------------------------------
|
||||
|
||||
def test_list_shaped_manifest_does_not_crash(vault, raw_file):
|
||||
_write_list_manifest(vault, [{"path": "_raw/foo.md", "content_hash": "sha256:x"}])
|
||||
result = check_sources(vault, [raw_file]) # previously: AttributeError
|
||||
assert isinstance(result, dict)
|
||||
|
||||
|
||||
# 2. Hash prefix mismatch ---------------------------------------------------
|
||||
|
||||
def test_sha256_prefixed_hash_reads_as_unchanged(vault, raw_file):
|
||||
# Skill records "sha256:<hex>"; compute_hash returns bare hex.
|
||||
_write_list_manifest(
|
||||
vault, [{"path": "_raw/foo.md", "content_hash": f"sha256:{compute_hash(raw_file)}"}]
|
||||
)
|
||||
result = check_sources(vault, [raw_file])
|
||||
assert str(raw_file) in result["unchanged"]
|
||||
assert result["modified"] == []
|
||||
assert result["new"] == []
|
||||
|
||||
|
||||
def test_prefixed_hash_still_detects_real_change(vault, raw_file):
|
||||
_write_list_manifest(
|
||||
vault, [{"path": "_raw/foo.md", "content_hash": f"sha256:{compute_hash(raw_file)}"}]
|
||||
)
|
||||
raw_file.write_text("# Foo\nchanged", encoding="utf-8")
|
||||
result = check_sources(vault, [raw_file])
|
||||
assert str(raw_file) in result["modified"]
|
||||
|
||||
|
||||
# 3. Vault-relative path stored, absolute path queried ----------------------
|
||||
|
||||
def test_relative_list_path_matches_absolute_query(vault, raw_file):
|
||||
_write_list_manifest(
|
||||
vault, [{"path": "_raw/foo.md", "content_hash": f"sha256:{compute_hash(raw_file)}"}]
|
||||
)
|
||||
result = check_sources(vault, [raw_file.resolve()])
|
||||
assert str(raw_file.resolve()) in result["unchanged"]
|
||||
|
||||
|
||||
# 4. update_source preserves shape, duplicates, and extra fields ------------
|
||||
|
||||
def test_update_preserves_list_shape(vault, raw_file):
|
||||
_write_list_manifest(
|
||||
vault,
|
||||
[{"path": "_raw/foo.md", "content_hash": "sha256:old", "source_type": "document",
|
||||
"pages_produced": ["concepts/foo.md"]}],
|
||||
)
|
||||
update_source(vault, raw_file, pages_produced=["concepts/foo.md", "concepts/bar.md"])
|
||||
sources = _load_raw(vault)["sources"]
|
||||
assert isinstance(sources, list), "must stay a list, not collapse to a dict"
|
||||
assert len(sources) == 1
|
||||
entry = sources[0]
|
||||
assert entry["source_type"] == "document" # skill field preserved
|
||||
assert entry["content_hash"] == f"sha256:{compute_hash(raw_file)}" # prefix kept
|
||||
assert entry["pages_produced"] == ["concepts/foo.md", "concepts/bar.md"]
|
||||
|
||||
|
||||
def test_update_preserves_duplicate_path_entries(vault, raw_file):
|
||||
# Same path ingested on two occasions — a dict round-trip would collapse these.
|
||||
_write_list_manifest(
|
||||
vault,
|
||||
[
|
||||
{"path": "_raw/foo.md", "content_hash": "sha256:v1", "ingested_at": "2026-01-01"},
|
||||
{"path": "_raw/foo.md", "content_hash": "sha256:v2", "ingested_at": "2026-02-01"},
|
||||
],
|
||||
)
|
||||
update_source(vault, raw_file)
|
||||
sources = _load_raw(vault)["sources"]
|
||||
assert len(sources) == 2, "duplicate-path entries must survive"
|
||||
# First match updated; the second occasion is left intact.
|
||||
assert sources[0]["content_hash"] == f"sha256:{compute_hash(raw_file)}"
|
||||
assert sources[1]["content_hash"] == "sha256:v2"
|
||||
|
||||
|
||||
def test_update_appends_new_list_entry(vault, raw_file):
|
||||
_write_list_manifest(vault, [{"path": "_raw/other.md", "content_hash": "sha256:z"}])
|
||||
update_source(vault, raw_file)
|
||||
sources = _load_raw(vault)["sources"]
|
||||
assert isinstance(sources, list)
|
||||
# A new in-vault source is keyed vault-relative (source key contract v2),
|
||||
# not by the machine absolute path, so the vault stays portable.
|
||||
assert {e["path"] for e in sources} == {"_raw/other.md", "_raw/foo.md"}
|
||||
|
||||
|
||||
def test_top_level_keys_preserved(vault, raw_file):
|
||||
_manifest_path(vault).write_text(
|
||||
json.dumps({"version": 1, "stats": {"total_sources_ingested": 5}, "sources": []}),
|
||||
encoding="utf-8",
|
||||
)
|
||||
update_source(vault, raw_file)
|
||||
manifest = _load_raw(vault)
|
||||
assert manifest["version"] == 1
|
||||
assert manifest["stats"]["total_sources_ingested"] == 5
|
||||
|
||||
|
||||
# Non-file keys (URLs / pseudo-paths) must not be flagged missing -----------
|
||||
|
||||
def test_list_manifest_rekeys_stale_raw_to_archived(vault):
|
||||
staging = vault / "_raw" / "foo.md"
|
||||
archived = vault / "_raw" / "_archived" / "foo.md"
|
||||
archived.parent.mkdir(parents=True, exist_ok=True)
|
||||
staging.parent.mkdir(parents=True, exist_ok=True)
|
||||
staging.write_text("staging-bytes\n", encoding="utf-8")
|
||||
archived.write_text("archived-bytes\n", encoding="utf-8")
|
||||
_write_list_manifest(
|
||||
vault,
|
||||
[{"path": "_raw/foo.md", "content_hash": "sha256:old", "source_type": "document"}],
|
||||
)
|
||||
staging.unlink() # ingest moved the draft into _archived/
|
||||
h = update_source(vault, staging)
|
||||
assert h == compute_hash(archived)
|
||||
entries = _load_raw(vault)["sources"]
|
||||
assert isinstance(entries, list)
|
||||
paths = [e.get("path") for e in entries]
|
||||
assert paths == ["_raw/_archived/foo.md"]
|
||||
assert entries[0]["source_type"] == "document"
|
||||
|
||||
|
||||
def test_url_source_key_not_flagged_missing(vault):
|
||||
_write_list_manifest(
|
||||
vault,
|
||||
[{"path": "https://example.com/x", "content_hash": "sha256:x", "source_type": "repository"}],
|
||||
)
|
||||
result = check_sources(vault, [])
|
||||
assert result["missing"] == []
|
||||
@@ -0,0 +1,293 @@
|
||||
"""Tests for the wiki clock (`obsidian_wiki.clock`) and the writers that use it.
|
||||
|
||||
The guarantees: a vault without ``timezone:`` gets byte-identical stamps to
|
||||
before (UTC, ``Z``); a vault with one gets that zone's own offset on every
|
||||
stamp, correct across a daylight-saving transition; bare dates are calendar
|
||||
days in the zone; a misspelled zone fails the write instead of silently
|
||||
writing UTC; and the docs name the knob.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from zoneinfo import ZoneInfo
|
||||
|
||||
import pytest
|
||||
|
||||
from obsidian_wiki import clock
|
||||
from obsidian_wiki import memory as mem
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
DETROIT = ZoneInfo("America/Detroit")
|
||||
Z_STAMP = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$")
|
||||
OFFSET_STAMP = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}[+-]\d{2}:\d{2}$")
|
||||
DETROIT_STAMP = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}-0[45]:00$")
|
||||
BARE_DATE = re.compile(r"^\d{4}-\d{2}-\d{2}$")
|
||||
|
||||
|
||||
def _index(vault: Path, *extra: str) -> Path:
|
||||
lines = ["---", "title: Wiki Index", *extra, "generated_by: obsidian-wiki memory index", "---", "", "# Wiki Index", ""]
|
||||
path = vault / "index.md"
|
||||
path.write_text("\n".join(lines), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def vault(tmp_path: Path) -> Path:
|
||||
for name in ("concepts", "_meta"):
|
||||
(tmp_path / name).mkdir()
|
||||
(tmp_path / "concepts" / "attention.md").write_text(
|
||||
"---\ntitle: Attention\nsummary: Core block\ntags: [ml]\nupdated: 2026-09-01\n---\n\n# Attention\n\nBody.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
_index(tmp_path)
|
||||
return tmp_path
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def detroit_vault(vault: Path) -> Path:
|
||||
_index(vault, "timezone: America/Detroit")
|
||||
return vault
|
||||
|
||||
|
||||
def _cli(*args: str) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
[sys.executable, "-m", "obsidian_wiki.cli", "memory", *args],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# pure functions
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestParse:
|
||||
def test_empty_and_utc_mean_the_default(self) -> None:
|
||||
for name in ("", " ", "UTC", "utc", "Z", '"UTC"'):
|
||||
assert clock.parse_timezone(name) is None, name
|
||||
|
||||
def test_an_iana_name_resolves(self) -> None:
|
||||
assert clock.parse_timezone("America/Detroit") == DETROIT
|
||||
assert clock.parse_timezone("'Europe/Berlin'") == ZoneInfo("Europe/Berlin")
|
||||
|
||||
def test_an_unknown_name_fails_loudly(self) -> None:
|
||||
with pytest.raises(clock.TimezoneError, match="Mars/Olympus"):
|
||||
clock.parse_timezone("Mars/Olympus")
|
||||
assert issubclass(clock.TimezoneError, ValueError)
|
||||
|
||||
|
||||
class TestStamp:
|
||||
SUMMER = datetime(2026, 10, 10, 18, 40, 37, tzinfo=timezone.utc) # EDT
|
||||
WINTER = datetime(2026, 11, 2, 18, 40, 37, tzinfo=timezone.utc) # EST
|
||||
|
||||
def test_default_is_the_original_z_form(self) -> None:
|
||||
assert clock.stamp(None, now=self.SUMMER) == "2026-10-10T18:40:37Z"
|
||||
assert clock.compact(None, now=self.SUMMER) == "20261010T184037Z"
|
||||
assert clock.today(None, now=self.SUMMER) == "2026-10-10"
|
||||
|
||||
def test_a_zone_carries_its_own_offset_for_that_instant(self) -> None:
|
||||
assert clock.stamp(DETROIT, now=self.SUMMER) == "2026-10-10T14:40:37-04:00"
|
||||
assert clock.stamp(DETROIT, now=self.WINTER) == "2026-11-02T13:40:37-05:00"
|
||||
assert clock.compact(DETROIT, now=self.SUMMER) == "20261010T144037-0400"
|
||||
|
||||
def test_a_zone_input_is_fine_too(self) -> None:
|
||||
local = self.SUMMER.astimezone(ZoneInfo("Asia/Tokyo"))
|
||||
assert clock.stamp(DETROIT, now=local) == "2026-10-10T14:40:37-04:00"
|
||||
assert clock.stamp(None, now=local) == "2026-10-10T18:40:37Z"
|
||||
|
||||
def test_a_bare_date_is_the_calendar_day_in_the_zone(self) -> None:
|
||||
evening = datetime(2026, 10, 11, 0, 30, 0, tzinfo=timezone.utc) # 20:30 the 10th in Detroit
|
||||
assert clock.today(None, now=evening) == "2026-10-11"
|
||||
assert clock.today(DETROIT, now=evening) == "2026-10-10"
|
||||
|
||||
def test_live_stamps_have_the_expected_shape(self) -> None:
|
||||
assert Z_STAMP.match(clock.stamp(None))
|
||||
assert DETROIT_STAMP.match(clock.stamp(DETROIT))
|
||||
assert BARE_DATE.match(clock.today(DETROIT))
|
||||
|
||||
def test_a_naive_now_is_refused(self) -> None:
|
||||
with pytest.raises(ValueError, match="aware"):
|
||||
clock.stamp(DETROIT, now=datetime(2026, 10, 10, 12, 0, 0))
|
||||
|
||||
|
||||
class TestVaultLookup:
|
||||
def test_no_index_or_no_key_means_default(self, tmp_path: Path) -> None:
|
||||
assert clock.wiki_timezone(tmp_path) is None
|
||||
assert clock.timezone_name(tmp_path) == ""
|
||||
_index(tmp_path)
|
||||
assert clock.wiki_timezone(tmp_path) is None
|
||||
assert clock.timezone_name(tmp_path) == ""
|
||||
|
||||
def test_the_key_is_read_from_index_frontmatter(self, detroit_vault: Path) -> None:
|
||||
assert clock.wiki_timezone(detroit_vault) == DETROIT
|
||||
assert clock.timezone_name(detroit_vault) == "America/Detroit"
|
||||
|
||||
def test_quotes_are_tolerated(self, vault: Path) -> None:
|
||||
_index(vault, 'timezone: "Europe/Berlin"')
|
||||
assert clock.wiki_timezone(vault) == ZoneInfo("Europe/Berlin")
|
||||
|
||||
def test_the_key_in_the_body_does_not_count(self, vault: Path) -> None:
|
||||
(vault / "index.md").write_text(
|
||||
"---\ntitle: Wiki Index\n---\n\n# Wiki Index\n\ntimezone: America/Detroit\n", encoding="utf-8"
|
||||
)
|
||||
assert clock.wiki_timezone(vault) is None
|
||||
|
||||
def test_a_typo_raises(self, vault: Path) -> None:
|
||||
_index(vault, "timezone: America/Detroi")
|
||||
with pytest.raises(clock.TimezoneError):
|
||||
clock.wiki_timezone(vault)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# the writers
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestWritersWithoutAZone:
|
||||
"""The regression guard: nothing changes for a vault that never set the key."""
|
||||
|
||||
def test_log_hot_adopted_and_tables_stay_utc_z(self, vault: Path) -> None:
|
||||
line = mem.append_log(vault, "INGEST", {"source": "x"})
|
||||
assert Z_STAMP.match(mem.parse_log_line(line).timestamp)
|
||||
hot = mem.rebuild_hot(vault, force=True)
|
||||
assert Z_STAMP.match(mem.parse_frontmatter(mem.split_frontmatter(hot.text)[0])["updated"])
|
||||
mem.mark_adopted(vault)
|
||||
adopted = (vault / mem.ADOPTED_REL).read_text(encoding="utf-8")
|
||||
assert re.search(r"^adopted: \d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$", adopted, re.MULTILINE)
|
||||
todo = mem.add_todo(vault, "Ship the parser")
|
||||
assert BARE_DATE.match(todo.created) and todo.created == clock.today(None)
|
||||
fact = mem.set_fact(vault, "stack", "Python")
|
||||
assert fact.updated == clock.today(None)
|
||||
assert mem.memory_status(vault)["timezone"] == "UTC"
|
||||
|
||||
|
||||
class TestWritersWithAZone:
|
||||
def test_every_stamp_carries_the_zone_offset(self, detroit_vault: Path) -> None:
|
||||
vault = detroit_vault
|
||||
line = mem.append_log(vault, "INGEST", {"source": "x"})
|
||||
assert DETROIT_STAMP.match(mem.parse_log_line(line).timestamp), line
|
||||
hot = mem.rebuild_hot(vault, force=True)
|
||||
updated = mem.parse_frontmatter(mem.split_frontmatter(hot.text)[0])["updated"]
|
||||
assert DETROIT_STAMP.match(updated), updated
|
||||
mem.mark_adopted(vault)
|
||||
adopted = (vault / mem.ADOPTED_REL).read_text(encoding="utf-8")
|
||||
assert re.search(r"^adopted: \d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}-0[45]:00$", adopted, re.MULTILINE)
|
||||
|
||||
def test_bare_dates_are_the_zone_calendar_day(self, detroit_vault: Path) -> None:
|
||||
vault = detroit_vault
|
||||
todo = mem.add_todo(vault, "Ship the parser", origin="projects/x.md")
|
||||
assert todo.created == clock.today(DETROIT) == todo.touched
|
||||
closed = mem.set_todo_status(vault, todo.id, "done")
|
||||
assert closed.touched == clock.today(DETROIT)
|
||||
fact = mem.set_fact(vault, "stack", "Python")
|
||||
assert fact.updated == clock.today(DETROIT)
|
||||
table = (vault / "_meta" / "todos.md").read_text(encoding="utf-8")
|
||||
assert f"created: {clock.today(DETROIT)}" in table
|
||||
assert mem.memory_status(vault)["timezone"] == "America/Detroit"
|
||||
|
||||
def test_an_explicit_log_timestamp_still_wins(self, detroit_vault: Path) -> None:
|
||||
line = mem.append_log(detroit_vault, "INGEST", {}, timestamp="2026-01-01T00:00:00Z")
|
||||
assert line.startswith("- [2026-01-01T00:00:00Z] INGEST")
|
||||
|
||||
def test_rebuilding_the_index_keeps_the_key(self, detroit_vault: Path) -> None:
|
||||
mem.rebuild_index(detroit_vault, force=True)
|
||||
assert clock.timezone_name(detroit_vault) == "America/Detroit"
|
||||
text = (detroit_vault / "index.md").read_text(encoding="utf-8")
|
||||
assert text.count("timezone:") == 1
|
||||
|
||||
def test_the_migration_backup_uses_the_zone(self, detroit_vault: Path) -> None:
|
||||
(detroit_vault / "hot.md").write_text("# Hot\n\n## Key Takeaways\n\nhand-written\n", encoding="utf-8")
|
||||
result = mem.migrate(detroit_vault)
|
||||
assert re.search(r"pre-memory-migration-\d{8}T\d{6}-0[45]00$", result["backup_dir"]), result["backup_dir"]
|
||||
|
||||
def test_a_typo_fails_the_write(self, vault: Path) -> None:
|
||||
_index(vault, "timezone: America/Detroi")
|
||||
with pytest.raises(clock.TimezoneError):
|
||||
mem.append_log(vault, "INGEST", {})
|
||||
assert not (vault / "log.md").exists()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# the CLI
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestCli:
|
||||
def test_now_prints_the_stamp_to_write(self, detroit_vault: Path) -> None:
|
||||
result = _cli("now", "--vault", str(detroit_vault))
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert DETROIT_STAMP.match(result.stdout.strip()), result.stdout
|
||||
|
||||
def test_now_without_a_zone_is_utc_z(self, vault: Path) -> None:
|
||||
result = _cli("now", "--vault", str(vault))
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert Z_STAMP.match(result.stdout.strip()), result.stdout
|
||||
|
||||
def test_now_date_and_json(self, detroit_vault: Path) -> None:
|
||||
date = _cli("now", "--date", "--vault", str(detroit_vault))
|
||||
assert date.returncode == 0 and date.stdout.strip() == clock.today(DETROIT)
|
||||
as_json = _cli("now", "--json", "--vault", str(detroit_vault))
|
||||
assert as_json.returncode == 0, as_json.stderr
|
||||
import json
|
||||
payload = json.loads(as_json.stdout)
|
||||
assert set(payload) == {"timestamp", "date", "timezone"}
|
||||
assert payload["timezone"] == "America/Detroit"
|
||||
assert DETROIT_STAMP.match(payload["timestamp"]) and BARE_DATE.match(payload["date"])
|
||||
|
||||
def test_status_names_the_zone(self, detroit_vault: Path, vault: Path) -> None:
|
||||
result = _cli("status", "--vault", str(detroit_vault))
|
||||
assert result.returncode == 0, result.stderr
|
||||
assert "timezone: America/Detroit" in result.stdout
|
||||
|
||||
def test_a_typo_is_an_error_not_a_utc_stamp(self, vault: Path) -> None:
|
||||
_index(vault, "timezone: Americas/Detroit")
|
||||
result = _cli("now", "--vault", str(vault))
|
||||
assert result.returncode == 1
|
||||
assert "Americas/Detroit" in result.stderr
|
||||
assert result.stdout.strip() == ""
|
||||
|
||||
def test_sync_stamps_the_log_in_the_zone(self, detroit_vault: Path) -> None:
|
||||
mem.mark_adopted(detroit_vault)
|
||||
result = _cli("sync", "INGEST", "source=x", "--vault", str(detroit_vault))
|
||||
assert result.returncode == 0, result.stderr
|
||||
log = (detroit_vault / "log.md").read_text(encoding="utf-8")
|
||||
assert re.search(r"^- \[\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}-0[45]:00\] INGEST", log, re.MULTILINE), log
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# docs
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# setup
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
class TestSetup:
|
||||
def test_set_timezone_inserts_replaces_and_validates(self, vault: Path) -> None:
|
||||
assert clock.set_timezone(vault, "America/Detroit") == "America/Detroit"
|
||||
text = (vault / "index.md").read_text(encoding="utf-8")
|
||||
assert text.startswith("---\ntitle: Wiki Index\ntimezone: America/Detroit\n"), text
|
||||
assert "# Wiki Index" in text and text.count("timezone:") == 1
|
||||
clock.set_timezone(vault, "Europe/Berlin")
|
||||
text = (vault / "index.md").read_text(encoding="utf-8")
|
||||
assert text.count("timezone:") == 1 and "timezone: Europe/Berlin" in text
|
||||
assert clock.set_timezone(vault, "") == "" and "Europe/Berlin" in (vault / "index.md").read_text()
|
||||
with pytest.raises(clock.TimezoneError):
|
||||
clock.set_timezone(vault, "America/Detroi")
|
||||
assert "Europe/Berlin" in (vault / "index.md").read_text(encoding="utf-8")
|
||||
|
||||
def test_set_timezone_on_an_index_without_frontmatter(self, tmp_path: Path) -> None:
|
||||
(tmp_path / "index.md").write_text("# Wiki Index\n\n## Concepts\n", encoding="utf-8")
|
||||
clock.set_timezone(tmp_path, "America/Detroit")
|
||||
text = (tmp_path / "index.md").read_text(encoding="utf-8")
|
||||
assert text.startswith("---\ntitle: Wiki Index\ntimezone: America/Detroit\n---\n\n# Wiki Index\n"), text
|
||||
assert clock.wiki_timezone(tmp_path) == DETROIT
|
||||
@@ -0,0 +1,716 @@
|
||||
"""Tests for vault graph analysis: community detection, god nodes, surprising connections."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import subprocess
|
||||
import sys
|
||||
import textwrap
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from obsidian_wiki.graph_analysis import (
|
||||
analyse_vault,
|
||||
dead_ends,
|
||||
detect_communities_greedy,
|
||||
god_nodes,
|
||||
isolated,
|
||||
parse_vault_graph,
|
||||
surprising_connections,
|
||||
)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Fixtures — synthetic vault
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@pytest.fixture
|
||||
def vault(tmp_path):
|
||||
v = tmp_path / "vault"
|
||||
v.mkdir()
|
||||
return v
|
||||
|
||||
|
||||
def _page(vault: Path, name: str, links: list[str], tags: list[str] | None = None) -> Path:
|
||||
"""Write a minimal wiki page with wikilinks."""
|
||||
lines = ["---", f"title: {name}"]
|
||||
if tags:
|
||||
lines.append(f"tags: [{', '.join(tags)}]")
|
||||
lines += ["---", f"# {name}", ""]
|
||||
lines += [f"[[{lnk}]]" for lnk in links]
|
||||
p = vault / f"{name}.md"
|
||||
p.write_text("\n".join(lines) + "\n")
|
||||
return p
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def simple_vault(vault):
|
||||
"""
|
||||
A → B → C
|
||||
A → C
|
||||
D (isolated)
|
||||
E → F (dead-end cluster)
|
||||
"""
|
||||
_page(vault, "a", ["b", "c"], tags=["concepts"])
|
||||
_page(vault, "b", ["c"], tags=["concepts"])
|
||||
_page(vault, "c", [], tags=["references"])
|
||||
_page(vault, "d", [], tags=["entities"])
|
||||
_page(vault, "e", ["f"])
|
||||
_page(vault, "f", [])
|
||||
return vault
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# parse_vault_graph
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestParseVaultGraph:
|
||||
def test_reads_wikilinks(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
assert "b" in outgoing["a"]
|
||||
assert "c" in outgoing["a"]
|
||||
|
||||
def test_all_pages_present_as_keys(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
assert set(outgoing.keys()) == {"a", "b", "c", "d", "e", "f"}
|
||||
|
||||
def test_reads_tags(self, simple_vault):
|
||||
_, tags = parse_vault_graph(simple_vault)
|
||||
assert "concepts" in tags.get("a", [])
|
||||
|
||||
def test_empty_vault(self, vault):
|
||||
outgoing, tags = parse_vault_graph(vault)
|
||||
assert outgoing == {}
|
||||
|
||||
def test_self_links_ignored(self, vault):
|
||||
_page(vault, "selfref", ["selfref"])
|
||||
outgoing, _ = parse_vault_graph(vault)
|
||||
assert "selfref" not in outgoing.get("selfref", [])
|
||||
|
||||
def test_links_to_nonexistent_pages_excluded(self, vault):
|
||||
_page(vault, "orphan", ["doesnotexist"])
|
||||
outgoing, _ = parse_vault_graph(vault)
|
||||
assert outgoing["orphan"] == []
|
||||
|
||||
def test_archive_wikilink_is_not_an_edge_to_same_stem(self, vault):
|
||||
_page(vault, "attention", ["other"])
|
||||
_page(vault, "other", ["attention"])
|
||||
arch = vault / "_raw" / "_archived" / "attention.md"
|
||||
arch.parent.mkdir(parents=True)
|
||||
arch.write_text("clip\n", encoding="utf-8")
|
||||
cited = vault / "cited.md"
|
||||
cited.write_text(
|
||||
"---\ntitle: cited\nsnapshots:\n - [[_raw/_archived/attention]]\n---\n# cited\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
outgoing, _ = parse_vault_graph(vault)
|
||||
assert "attention" not in outgoing.get("cited", [])
|
||||
|
||||
def test_archive_markdown_snapshot_is_not_an_edge_to_same_stem(self, vault):
|
||||
_page(vault, "attention", ["other"])
|
||||
_page(vault, "other", ["attention"])
|
||||
arch = vault / "_raw" / "_archived" / "attention.md"
|
||||
arch.parent.mkdir(parents=True)
|
||||
arch.write_text("clip\n", encoding="utf-8")
|
||||
cited = vault / "cited.md"
|
||||
cited.write_text(
|
||||
"---\ntitle: cited\nsnapshots:\n"
|
||||
' - "[attention](_raw/_archived/attention.md)"\n---\n# cited\n',
|
||||
encoding="utf-8",
|
||||
)
|
||||
outgoing, _ = parse_vault_graph(vault)
|
||||
assert "attention" not in outgoing.get("cited", [])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# god_nodes
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestGodNodes:
|
||||
def test_c_is_top_hub(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
result = god_nodes(outgoing, top_n=3)
|
||||
# c has 2 in-links (from a and b), so should be in top 3
|
||||
top_pages = {r["page"] for r in result}
|
||||
assert "c" in top_pages
|
||||
|
||||
def test_degree_sum(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
result = god_nodes(outgoing)
|
||||
for node in result:
|
||||
assert node["degree"] == node["in_degree"] + node["out_degree"]
|
||||
|
||||
def test_respects_top_n(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
result = god_nodes(outgoing, top_n=2)
|
||||
assert len(result) <= 2
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# dead_ends / isolated
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestDeadEndsIsolated:
|
||||
def test_dead_ends(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
de = dead_ends(outgoing)
|
||||
assert "c" in de
|
||||
assert "f" in de
|
||||
assert "a" not in de
|
||||
|
||||
def test_isolated(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
iso = isolated(outgoing)
|
||||
assert "d" in iso
|
||||
assert "a" not in iso
|
||||
assert "c" not in iso # c has incoming links so not isolated
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Community detection
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestCommunityDetection:
|
||||
def test_returns_list_of_sets(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
comms = detect_communities_greedy(outgoing)
|
||||
assert isinstance(comms, list)
|
||||
for c in comms:
|
||||
assert isinstance(c, set)
|
||||
|
||||
def test_all_nodes_assigned(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
comms = detect_communities_greedy(outgoing)
|
||||
all_nodes = set(outgoing.keys())
|
||||
assigned = set()
|
||||
for c in comms:
|
||||
assigned |= c
|
||||
assert assigned == all_nodes
|
||||
|
||||
def test_no_overlap(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
comms = detect_communities_greedy(outgoing)
|
||||
seen = set()
|
||||
for c in comms:
|
||||
assert c.isdisjoint(seen), "Node appears in two communities"
|
||||
seen |= c
|
||||
|
||||
def test_connected_cluster_grouped(self, vault):
|
||||
# a-b-c tightly connected, x-y-z separate — should land in different communities
|
||||
_page(vault, "a", ["b", "c"])
|
||||
_page(vault, "b", ["a", "c"])
|
||||
_page(vault, "c", ["a", "b"])
|
||||
_page(vault, "x", ["y", "z"])
|
||||
_page(vault, "y", ["x", "z"])
|
||||
_page(vault, "z", ["x", "y"])
|
||||
outgoing, _ = parse_vault_graph(vault)
|
||||
comms = detect_communities_greedy(outgoing)
|
||||
# At least 2 communities
|
||||
assert len(comms) >= 2
|
||||
|
||||
def test_empty_graph(self, vault):
|
||||
outgoing, _ = parse_vault_graph(vault)
|
||||
comms = detect_communities_greedy(outgoing)
|
||||
assert comms == []
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Surprising connections
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestSurprisingConnections:
|
||||
def test_cross_community_edge_found(self, vault):
|
||||
# Build two tight clusters with one bridge
|
||||
_page(vault, "a", ["b", "c", "x"])
|
||||
_page(vault, "b", ["a", "c"])
|
||||
_page(vault, "c", ["a", "b"])
|
||||
_page(vault, "x", ["y", "z"])
|
||||
_page(vault, "y", ["x", "z"])
|
||||
_page(vault, "z", ["x", "y"])
|
||||
outgoing, _ = parse_vault_graph(vault)
|
||||
comms = detect_communities_greedy(outgoing)
|
||||
sc = surprising_connections(outgoing, comms)
|
||||
sources = {s["source"] for s in sc}
|
||||
targets = {s["target"] for s in sc}
|
||||
assert sources | targets # at least one cross-community edge found
|
||||
|
||||
def test_no_intra_community_edges(self, vault):
|
||||
_page(vault, "a", ["b"])
|
||||
_page(vault, "b", ["a"])
|
||||
outgoing, _ = parse_vault_graph(vault)
|
||||
comms = [{"a", "b"}] # one community
|
||||
sc = surprising_connections(outgoing, comms)
|
||||
assert sc == []
|
||||
|
||||
def test_scores_positive(self, vault):
|
||||
_page(vault, "a", ["b", "x"])
|
||||
_page(vault, "b", ["a"])
|
||||
_page(vault, "x", ["y"])
|
||||
_page(vault, "y", ["x"])
|
||||
outgoing, _ = parse_vault_graph(vault)
|
||||
comms = detect_communities_greedy(outgoing)
|
||||
sc = surprising_connections(outgoing, comms)
|
||||
for item in sc:
|
||||
assert item["score"] > 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# analyse_vault (integration)
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestAnalyseVault:
|
||||
def test_returns_all_keys(self, simple_vault):
|
||||
result = analyse_vault(simple_vault)
|
||||
assert set(result.keys()) == {
|
||||
"god_nodes", "bridges", "communities", "surprising_connections",
|
||||
"suggested_questions", "dead_ends", "isolated", "stats",
|
||||
}
|
||||
|
||||
def test_stats_correct(self, simple_vault):
|
||||
result = analyse_vault(simple_vault)
|
||||
assert result["stats"]["pages"] == 6
|
||||
assert result["stats"]["edges"] > 0
|
||||
|
||||
def test_empty_vault(self, vault):
|
||||
result = analyse_vault(vault)
|
||||
assert result["stats"]["pages"] == 0
|
||||
|
||||
def test_json_serialisable(self, simple_vault):
|
||||
result = analyse_vault(simple_vault)
|
||||
json.dumps(result) # must not raise
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# CLI
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class TestGraphAnalyseCLI:
|
||||
def test_outputs_json(self, simple_vault):
|
||||
proc = subprocess.run(
|
||||
[sys.executable, "-m", "obsidian_wiki.cli", "graph-analyse", str(simple_vault)],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
assert proc.returncode == 0
|
||||
data = json.loads(proc.stdout)
|
||||
assert "god_nodes" in data
|
||||
|
||||
def test_pretty_flag(self, simple_vault):
|
||||
proc = subprocess.run(
|
||||
[sys.executable, "-m", "obsidian_wiki.cli", "graph-analyse", str(simple_vault), "--pretty"],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
assert proc.returncode == 0
|
||||
assert "\n " in proc.stdout
|
||||
|
||||
def test_top_flag(self, simple_vault):
|
||||
proc = subprocess.run(
|
||||
[sys.executable, "-m", "obsidian_wiki.cli", "graph-analyse", str(simple_vault), "--top", "3"],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
assert proc.returncode == 0
|
||||
data = json.loads(proc.stdout)
|
||||
assert len(data["god_nodes"]) <= 3
|
||||
|
||||
def test_missing_vault_exits_nonzero(self, tmp_path):
|
||||
proc = subprocess.run(
|
||||
[sys.executable, "-m", "obsidian_wiki.cli", "graph-analyse", str(tmp_path / "nope")],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
assert proc.returncode != 0
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# graphify-derived algorithms: betweenness, cohesion, path, neighbourhood,
|
||||
# diff, suggested questions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
from obsidian_wiki.graph_analysis import ( # noqa: E402
|
||||
betweenness_centrality,
|
||||
bridge_pages,
|
||||
cohesion_score,
|
||||
graph_diff,
|
||||
load_snapshot,
|
||||
neighborhood,
|
||||
shortest_path,
|
||||
snapshot,
|
||||
suggest_questions,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def barbell_vault(vault):
|
||||
"""Two triangles joined through a single bridge page `m`.
|
||||
a-b-c clique — m — x-y-z clique
|
||||
"""
|
||||
_page(vault, "a", ["b", "c", "m"], tags=["left"])
|
||||
_page(vault, "b", ["a", "c"], tags=["left"])
|
||||
_page(vault, "c", ["a", "b"], tags=["left"])
|
||||
_page(vault, "m", ["a", "x"], tags=["mid"])
|
||||
_page(vault, "x", ["m", "y", "z"], tags=["right"])
|
||||
_page(vault, "y", ["x", "z"], tags=["right"])
|
||||
_page(vault, "z", ["x", "y"], tags=["right"])
|
||||
return vault
|
||||
|
||||
|
||||
class TestBetweenness:
|
||||
def test_bridge_has_highest_betweenness(self, barbell_vault):
|
||||
outgoing, _ = parse_vault_graph(barbell_vault)
|
||||
bc = betweenness_centrality(outgoing)
|
||||
assert max(bc, key=bc.get) == "m"
|
||||
# leaf-of-clique nodes lie on no shortest paths
|
||||
assert bc["b"] == 0.0 and bc["y"] == 0.0
|
||||
|
||||
def test_normalised_range(self, barbell_vault):
|
||||
outgoing, _ = parse_vault_graph(barbell_vault)
|
||||
for v in betweenness_centrality(outgoing).values():
|
||||
assert 0.0 <= v <= 1.0
|
||||
|
||||
def test_matches_star_exact_value(self, vault):
|
||||
# Star: hub h with 4 leaves — hub betweenness = 1.0 normalised
|
||||
_page(vault, "h", ["p", "q", "r", "s"])
|
||||
for leaf in "pqrs":
|
||||
_page(vault, leaf, [])
|
||||
outgoing, _ = parse_vault_graph(vault)
|
||||
bc = betweenness_centrality(outgoing)
|
||||
assert bc["h"] == pytest.approx(1.0)
|
||||
|
||||
def test_bridge_pages_reports_connected_communities(self, barbell_vault):
|
||||
outgoing, _ = parse_vault_graph(barbell_vault)
|
||||
comms = detect_communities_greedy(outgoing)
|
||||
bridges = bridge_pages(outgoing, comms, top_n=3)
|
||||
assert bridges[0]["page"] == "m"
|
||||
assert bridges[0]["betweenness"] > 0
|
||||
assert "connects" in bridges[0]
|
||||
|
||||
def test_empty(self):
|
||||
assert betweenness_centrality({}) == {}
|
||||
assert bridge_pages({}, [], 5) == []
|
||||
|
||||
|
||||
class TestCohesion:
|
||||
def test_clique_is_one(self, barbell_vault):
|
||||
outgoing, _ = parse_vault_graph(barbell_vault)
|
||||
assert cohesion_score(outgoing, {"a", "b", "c"}) == 1.0
|
||||
|
||||
def test_no_internal_links_is_zero(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
assert cohesion_score(outgoing, {"d", "f", "c"}) == 0.0
|
||||
|
||||
def test_singleton_is_one(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
assert cohesion_score(outgoing, {"d"}) == 1.0
|
||||
|
||||
def test_analyse_includes_cohesion(self, barbell_vault):
|
||||
result = analyse_vault(barbell_vault)
|
||||
for c in result["communities"]:
|
||||
assert 0.0 <= c["cohesion"] <= 1.0
|
||||
|
||||
|
||||
class TestShortestPath:
|
||||
def test_finds_path_across_bridge(self, barbell_vault):
|
||||
outgoing, _ = parse_vault_graph(barbell_vault)
|
||||
path = shortest_path(outgoing, "b", "y")
|
||||
assert path[0] == "b" and path[-1] == "y"
|
||||
assert "m" in path
|
||||
assert len(path) == 5 # b-a-m-x-y
|
||||
|
||||
def test_directed_respects_link_direction(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
assert shortest_path(outgoing, "a", "c", directed=True) == ["a", "c"]
|
||||
assert shortest_path(outgoing, "c", "a", directed=True) is None
|
||||
assert shortest_path(outgoing, "c", "a") == ["c", "a"]
|
||||
|
||||
def test_unreachable_and_unknown(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
assert shortest_path(outgoing, "a", "d") is None
|
||||
assert shortest_path(outgoing, "a", "nope") is None
|
||||
assert shortest_path(outgoing, "a", "a") == ["a"]
|
||||
|
||||
def test_slug_normalisation(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
assert shortest_path(outgoing, "A", "B") == ["a", "b"]
|
||||
|
||||
|
||||
class TestNeighborhood:
|
||||
def test_depth_layers(self, barbell_vault):
|
||||
outgoing, _ = parse_vault_graph(barbell_vault)
|
||||
hits = neighborhood(outgoing, "m", depth=1)
|
||||
assert {h["page"] for h in hits} == {"a", "x"}
|
||||
hits2 = neighborhood(outgoing, "m", depth=2)
|
||||
assert {h["page"] for h in hits2} == {"a", "x", "b", "c", "y", "z"}
|
||||
assert all(h["depth"] in (1, 2) for h in hits2)
|
||||
|
||||
def test_incoming_blast_radius(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
# who would break if `c` were renamed? a and b link to it
|
||||
hits = neighborhood(outgoing, "c", depth=1, direction="in")
|
||||
assert {h["page"] for h in hits} == {"a", "b"}
|
||||
assert neighborhood(outgoing, "c", depth=1, direction="out") == []
|
||||
|
||||
def test_unknown_seed(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
assert neighborhood(outgoing, "nope") == []
|
||||
|
||||
|
||||
class TestGraphDiff:
|
||||
def test_added_removed(self):
|
||||
old = {"nodes": ["a", "b", "c"], "edges": [["a", "b"]]}
|
||||
new = {"nodes": ["a", "b", "d"], "edges": [["a", "b"], ["a", "d"], ["d", "b"]]}
|
||||
d = graph_diff(old, new)
|
||||
assert d["added_pages"] == ["d"]
|
||||
assert d["removed_pages"] == ["c"]
|
||||
assert ["a", "d"] in d["added_edges"]
|
||||
assert d["removed_edges"] == []
|
||||
assert d["summary"] == {"pages": 0, "edges": 2}
|
||||
|
||||
def test_newly_connected_and_lost_incoming(self):
|
||||
old = {"nodes": ["a", "b", "c"], "edges": [["a", "b"]]}
|
||||
new = {"nodes": ["a", "b", "c"], "edges": [["a", "c"]]}
|
||||
d = graph_diff(old, new)
|
||||
assert d["newly_connected"] == ["c"]
|
||||
assert d["lost_incoming"] == ["b"]
|
||||
|
||||
def test_snapshot_roundtrip_via_insights_comment(self, simple_vault, tmp_path):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
snap = snapshot(outgoing)
|
||||
insights = tmp_path / "_insights.md"
|
||||
insights.write_text("# Insights\n\nstuff\n\n<!-- GRAPH_SNAPSHOT: " + json.dumps(snap) + " -->\n")
|
||||
loaded = load_snapshot(insights)
|
||||
assert loaded == snap
|
||||
assert graph_diff(loaded, snap)["summary"] == {"pages": 0, "edges": 0}
|
||||
|
||||
def test_load_snapshot_missing(self, tmp_path):
|
||||
p = tmp_path / "x.md"
|
||||
p.write_text("no snapshot here")
|
||||
assert load_snapshot(p) is None
|
||||
assert load_snapshot(tmp_path / "absent.md") is None
|
||||
|
||||
def test_analyse_with_previous(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
prev = snapshot(outgoing)
|
||||
_page(simple_vault, "g", ["a"])
|
||||
result = analyse_vault(simple_vault, previous_snapshot=prev, include_snapshot=True)
|
||||
assert result["diff"]["added_pages"] == ["g"]
|
||||
assert "snapshot" in result
|
||||
|
||||
|
||||
class TestSuggestQuestions:
|
||||
def test_bridge_question_first(self, barbell_vault):
|
||||
outgoing, _ = parse_vault_graph(barbell_vault)
|
||||
comms = detect_communities_greedy(outgoing)
|
||||
labels = {i: f"c{i}" for i in range(len(comms))}
|
||||
bridges = bridge_pages(outgoing, comms)
|
||||
qs = suggest_questions(outgoing, comms, labels, bridges)
|
||||
assert qs
|
||||
assert qs[0]["type"] == "bridge_node" and "`m`" in qs[0]["question"]
|
||||
|
||||
def test_isolated_question(self, simple_vault):
|
||||
outgoing, _ = parse_vault_graph(simple_vault)
|
||||
qs = suggest_questions(outgoing, [], {}, [])
|
||||
assert any(q["type"] == "isolated_nodes" for q in qs)
|
||||
|
||||
def test_no_signal(self):
|
||||
qs = suggest_questions({"a": ["b"], "b": ["a"]}, [{"a", "b"}], {0: "x"}, [])
|
||||
assert qs == [] or qs[0]["type"] in ("no_signal", "isolated_nodes")
|
||||
|
||||
def test_included_in_analyse(self, barbell_vault):
|
||||
result = analyse_vault(barbell_vault)
|
||||
assert "suggested_questions" in result and "bridges" in result
|
||||
assert result["stats"]["density"] > 0
|
||||
|
||||
|
||||
class TestSurprisingDedup:
|
||||
def test_note_and_pair_dedup(self, vault):
|
||||
# Hub h in community 0 links to 3 different pages of community 1;
|
||||
# community 1 also has one link to community 2. Pair (1,2) must not
|
||||
# be pushed out by the three (0,1) edges.
|
||||
_page(vault, "h", ["p", "q", "r"])
|
||||
_page(vault, "p", ["q"]); _page(vault, "q", ["p"]); _page(vault, "r", ["p", "s"])
|
||||
_page(vault, "s", ["t"]); _page(vault, "t", ["s"])
|
||||
outgoing, _ = parse_vault_graph(vault)
|
||||
comms = [{"h"}, {"p", "q", "r"}, {"s", "t"}]
|
||||
sc = surprising_connections(outgoing, comms, top_n=2)
|
||||
assert all("note" in x for x in sc)
|
||||
pairs = {tuple(sorted(x["note"].replace("bridges community ", "").split(" -> community "))) for x in sc}
|
||||
assert pairs == {("0", "1"), ("1", "2")}
|
||||
|
||||
|
||||
class TestGraphAnalyseCLIQueryModes:
|
||||
def _run(self, *argv):
|
||||
return subprocess.run(
|
||||
[sys.executable, "-m", "obsidian_wiki.cli", "graph-analyse", *argv],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
|
||||
def test_path(self, barbell_vault):
|
||||
proc = self._run(str(barbell_vault), "--path", "b", "y")
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
data = json.loads(proc.stdout)
|
||||
assert data["hops"] == 4 and "m" in data["path"]
|
||||
|
||||
def test_around(self, barbell_vault):
|
||||
proc = self._run(str(barbell_vault), "--around", "m", "--depth", "1")
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
data = json.loads(proc.stdout)
|
||||
assert {p["page"] for p in data["pages"]} == {"a", "x"}
|
||||
|
||||
def test_diff_against_and_snapshot(self, simple_vault, tmp_path):
|
||||
proc = self._run(str(simple_vault), "--snapshot")
|
||||
snap = json.loads(proc.stdout)["snapshot"]
|
||||
prev = tmp_path / "_insights.md"
|
||||
prev.write_text("<!-- GRAPH_SNAPSHOT: " + json.dumps(snap) + " -->")
|
||||
_page(simple_vault, "new", ["a"])
|
||||
proc = self._run(str(simple_vault), "--diff-against", str(prev))
|
||||
assert proc.returncode == 0, proc.stderr
|
||||
assert json.loads(proc.stdout)["diff"]["added_pages"] == ["new"]
|
||||
|
||||
|
||||
class TestUniqueCommunityLabels:
|
||||
def test_labels_disambiguated(self, vault):
|
||||
# two disconnected clusters both dominated by tag `x`
|
||||
_page(vault, "a", ["b"], tags=["x", "left"]); _page(vault, "b", ["a"], tags=["x"])
|
||||
_page(vault, "c", ["d"], tags=["x", "right"]); _page(vault, "d", ["c"], tags=["x"])
|
||||
result = analyse_vault(vault)
|
||||
labels = [c["label"] for c in result["communities"]]
|
||||
assert len(labels) == len(set(labels))
|
||||
assert "x" in labels and any(l.startswith("x/") for l in labels)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Shared page selection + betweenness cache
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
from obsidian_wiki.graph_analysis import ( # noqa: E402
|
||||
CACHE_FILENAME,
|
||||
SKIP_ROOT_FILES,
|
||||
betweenness_cached,
|
||||
graph_fingerprint,
|
||||
is_wiki_page,
|
||||
iter_pages,
|
||||
)
|
||||
|
||||
|
||||
class TestPageSelection:
|
||||
def test_root_bookkeeping_excluded(self, vault):
|
||||
_page(vault, "a", ["b"]); _page(vault, "b", [])
|
||||
for name in SKIP_ROOT_FILES:
|
||||
(vault / f"{name}.md").write_text("---\ntitle: x\n---\n[[a]]\n")
|
||||
out, _ = parse_vault_graph(vault)
|
||||
assert set(out) == {"a", "b"}
|
||||
|
||||
def test_nested_index_is_a_real_page(self, vault):
|
||||
"""Only ROOT index/log/hot are bookkeeping — concepts/index.md is content."""
|
||||
(vault / "concepts").mkdir()
|
||||
(vault / "concepts" / "index.md").write_text("---\ntitle: Concept Index\n---\n[[a]]\n")
|
||||
_page(vault, "a", [])
|
||||
out, _ = parse_vault_graph(vault)
|
||||
assert "index" in out
|
||||
assert is_wiki_page(vault / "concepts" / "index.md", vault)
|
||||
assert not is_wiki_page(vault / "index.md", vault)
|
||||
|
||||
def test_skip_dirs_excluded(self, vault):
|
||||
for d in ("_raw", "_meta", "_readouts", "_staging"):
|
||||
(vault / d).mkdir()
|
||||
(vault / d / "note.md").write_text("---\ntitle: x\n---\n")
|
||||
_page(vault, "a", [])
|
||||
assert [p.stem for p in iter_pages(vault)] == ["a"]
|
||||
|
||||
def test_tool_owned_directories_excluded(self, vault):
|
||||
"""Dot-dirs are skipped outright (`.venv`, `.git`, `.trash`, …); the
|
||||
non-hidden dependency trees batch.py skips are skipped here too."""
|
||||
for d in (".venv/lib/site-packages/pkg", ".git/hooks", ".trash",
|
||||
"node_modules/pkg", "venv", "__pycache__"):
|
||||
(vault / d).mkdir(parents=True, exist_ok=True)
|
||||
(vault / d / "note.md").write_text("---\ntitle: x\n---\n")
|
||||
_page(vault, "a", [])
|
||||
assert [p.stem for p in iter_pages(vault)] == ["a"]
|
||||
assert not is_wiki_page(vault / ".venv" / "lib" / "site-packages" / "pkg" / "note.md", vault)
|
||||
assert not is_wiki_page(vault / "node_modules" / "pkg" / "note.md", vault)
|
||||
|
||||
def test_iter_pages_is_sorted(self, vault):
|
||||
for n in ("zebra", "alpha", "middle"):
|
||||
_page(vault, n, [])
|
||||
assert [p.stem for p in iter_pages(vault)] == ["alpha", "middle", "zebra"]
|
||||
|
||||
def test_path_outside_vault_is_not_a_page(self, vault, tmp_path):
|
||||
assert not is_wiki_page(tmp_path / "elsewhere.md", vault)
|
||||
|
||||
|
||||
class TestGraphFingerprint:
|
||||
def test_stable_across_calls(self, simple_vault):
|
||||
out, _ = parse_vault_graph(simple_vault)
|
||||
assert graph_fingerprint(out) == graph_fingerprint(out)
|
||||
|
||||
def test_changes_when_an_edge_changes(self, simple_vault):
|
||||
out, _ = parse_vault_graph(simple_vault)
|
||||
before = graph_fingerprint(out)
|
||||
_page(simple_vault, "e", ["f", "a"]) # add an edge
|
||||
out2, _ = parse_vault_graph(simple_vault)
|
||||
assert graph_fingerprint(out2) != before
|
||||
|
||||
def test_insensitive_to_key_order(self, simple_vault):
|
||||
out, _ = parse_vault_graph(simple_vault)
|
||||
shuffled = {k: out[k] for k in reversed(list(out))}
|
||||
assert graph_fingerprint(shuffled) == graph_fingerprint(out)
|
||||
|
||||
|
||||
class TestBetweennessCache:
|
||||
def test_matches_uncached(self, barbell_vault):
|
||||
out, _ = parse_vault_graph(barbell_vault)
|
||||
exact = betweenness_centrality(out)
|
||||
cached = betweenness_cached(out, barbell_vault)
|
||||
assert all(abs(exact[k] - cached[k]) < 1e-12 for k in exact)
|
||||
|
||||
def test_no_cache_file_for_small_vault(self, barbell_vault):
|
||||
"""Tiny graphs recompute in microseconds — don't litter the vault."""
|
||||
out, _ = parse_vault_graph(barbell_vault)
|
||||
betweenness_cached(out, barbell_vault)
|
||||
assert not (barbell_vault / CACHE_FILENAME).exists()
|
||||
|
||||
def test_none_vault_still_works(self, barbell_vault):
|
||||
out, _ = parse_vault_graph(barbell_vault)
|
||||
assert betweenness_cached(out, None) == betweenness_centrality(out)
|
||||
|
||||
def test_corrupt_cache_fails_open(self, barbell_vault):
|
||||
(barbell_vault / CACHE_FILENAME).write_text("{ not json")
|
||||
out, _ = parse_vault_graph(barbell_vault)
|
||||
assert betweenness_cached(out, barbell_vault) == betweenness_centrality(out)
|
||||
|
||||
def test_hit_is_used_when_present(self, barbell_vault):
|
||||
"""A planted cache entry is returned verbatim — proves the key matches."""
|
||||
out, _ = parse_vault_graph(barbell_vault)
|
||||
key = f"{graph_fingerprint(out)}:1"
|
||||
planted = {n: 0.5 for n in out}
|
||||
(barbell_vault / CACHE_FILENAME).write_text(
|
||||
json.dumps({"version": 1, "betweenness": {key: planted}})
|
||||
)
|
||||
assert betweenness_cached(out, barbell_vault) == planted
|
||||
|
||||
def test_stale_key_is_ignored(self, barbell_vault):
|
||||
out, _ = parse_vault_graph(barbell_vault)
|
||||
(barbell_vault / CACHE_FILENAME).write_text(
|
||||
json.dumps({"version": 1, "betweenness": {"someotherkey:1": {n: 9.0 for n in out}}})
|
||||
)
|
||||
assert betweenness_cached(out, barbell_vault) == betweenness_centrality(out)
|
||||
|
||||
def test_unwritable_vault_does_not_raise(self, barbell_vault, tmp_path):
|
||||
out, _ = parse_vault_graph(barbell_vault)
|
||||
assert betweenness_cached(out, tmp_path / "does" / "not" / "exist")
|
||||
|
||||
|
||||
class TestCommunityQuality:
|
||||
def test_barbell_splits_into_two(self, barbell_vault):
|
||||
"""A sequential label-propagation sweep collapses this into one group."""
|
||||
out, _ = parse_vault_graph(barbell_vault)
|
||||
comms = detect_communities_greedy(out)
|
||||
assert len(comms) == 2
|
||||
assert {"x", "y", "z"} in [set(c) for c in comms]
|
||||
|
||||
def test_deterministic_across_runs(self, barbell_vault):
|
||||
out, _ = parse_vault_graph(barbell_vault)
|
||||
sigs = {
|
||||
json.dumps(sorted(sorted(c) for c in detect_communities_greedy(out)))
|
||||
for _ in range(10)
|
||||
}
|
||||
assert len(sigs) == 1
|
||||
@@ -0,0 +1,120 @@
|
||||
"""Links resolve the way Obsidian resolves them, across every graph consumer.
|
||||
|
||||
A vault of per-project sub-wikis has pages that share a name
|
||||
(`projects/*/index.md`, the same concept in two projects) and links written
|
||||
relative to their project. Before `obsidian_wiki/links.py`, graph-analyse,
|
||||
graph-query, and lint keyed pages by bare filename, so those pages merged into
|
||||
one node and project-relative links were reported missing.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from obsidian_wiki.graph_analysis import neighborhood, parse_vault_graph, shortest_path
|
||||
from obsidian_wiki.links import PageIndex, strip_code
|
||||
from obsidian_wiki.lint import lint_vault
|
||||
|
||||
|
||||
def _write(vault: Path, rel: str, body: str = "", frontmatter: str = "") -> None:
|
||||
path = vault / rel
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
head = (
|
||||
f"---\ntitle: {path.stem}\ncategory: concepts\ntags: [a]\nsources: []\n"
|
||||
f"created: 2026-01-01\nupdated: 2026-01-01\nsummary: s\n{frontmatter}---\n\n"
|
||||
)
|
||||
path.write_text(head + body + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def _two_projects(vault: Path) -> None:
|
||||
for name in ("alpha", "beta"):
|
||||
_write(vault, f"projects/{name}/index.md", f"[[concepts/plan]] [[decisions/{name}-choice]]")
|
||||
_write(vault, f"projects/{name}/concepts/plan.md", "See [[index]].")
|
||||
_write(vault, f"projects/{name}/decisions/{name}-choice.md", "[[../concepts/plan]]")
|
||||
_write(vault, "concepts/shared.md", "No collision here: [[projects/alpha/concepts/plan]]")
|
||||
|
||||
|
||||
# -- PageIndex --------------------------------------------------------------
|
||||
|
||||
|
||||
def test_unique_names_keep_their_bare_ids() -> None:
|
||||
index = PageIndex(["concepts/Vector Search.md", "entities/karpathy.md"])
|
||||
assert sorted(index.ids()) == ["karpathy", "vector-search"]
|
||||
assert index.resolve("Vector Search", "synthesis/x.md") == "vector-search"
|
||||
assert index.resolve("concepts/vector-search.md", None) == "vector-search"
|
||||
|
||||
|
||||
def test_shared_names_get_path_ids_and_resolve_to_the_nearest_page() -> None:
|
||||
index = PageIndex(["projects/a/concepts/plan.md", "projects/b/concepts/plan.md"])
|
||||
assert sorted(index.ids()) == ["projects/a/concepts/plan", "projects/b/concepts/plan"]
|
||||
assert index.resolve("plan", "projects/b/index.md") == "projects/b/concepts/plan"
|
||||
assert index.resolve("concepts/plan", "projects/a/index.md") == "projects/a/concepts/plan"
|
||||
assert index.resolve("../concepts/plan", "projects/b/decisions/d.md") == "projects/b/concepts/plan"
|
||||
assert not index.is_ambiguous("plan", "projects/a/index.md")
|
||||
assert index.is_ambiguous("plan", "synthesis/elsewhere.md") # nothing breaks the tie
|
||||
|
||||
|
||||
def test_relative_link_cannot_escape_the_vault() -> None:
|
||||
assert PageIndex(["a.md"]).resolve("../../a", "x/y.md") is None
|
||||
|
||||
|
||||
def test_strip_code_drops_fences_and_inline_spans_only() -> None:
|
||||
text = "Real [[a]]. Example `[[b]]`.\n```\n[[c]]\n```\nAfter [[d]]."
|
||||
kept = strip_code(text)
|
||||
assert "[[a]]" in kept and "[[d]]" in kept
|
||||
assert "[[b]]" not in kept and "[[c]]" not in kept
|
||||
|
||||
|
||||
# -- every consumer ---------------------------------------------------------
|
||||
|
||||
|
||||
def test_graph_keeps_same_named_pages_apart(tmp_path: Path) -> None:
|
||||
_two_projects(tmp_path)
|
||||
outgoing, _ = parse_vault_graph(tmp_path)
|
||||
assert "projects/alpha/index" in outgoing and "projects/beta/index" in outgoing
|
||||
assert "index" not in outgoing
|
||||
assert sorted(outgoing["projects/beta/index"]) == ["beta-choice", "projects/beta/concepts/plan"]
|
||||
assert outgoing["projects/alpha/concepts/plan"] == ["projects/alpha/index"]
|
||||
assert outgoing["beta-choice"] == ["projects/beta/concepts/plan"]
|
||||
assert outgoing["shared"] == ["projects/alpha/concepts/plan"]
|
||||
|
||||
|
||||
def test_graph_queries_accept_a_bare_name_for_a_shared_page(tmp_path: Path) -> None:
|
||||
_two_projects(tmp_path)
|
||||
outgoing, _ = parse_vault_graph(tmp_path)
|
||||
assert shortest_path(outgoing, "beta-choice", "projects/beta/index") is not None
|
||||
assert neighborhood(outgoing, "plan", depth=1) # a bare shared name still finds a node
|
||||
|
||||
|
||||
|
||||
|
||||
def test_lint_resolves_project_relative_links_and_relationships(tmp_path: Path) -> None:
|
||||
_two_projects(tmp_path)
|
||||
_write(
|
||||
tmp_path,
|
||||
"projects/alpha/concepts/router.md",
|
||||
"Uses the plan.",
|
||||
frontmatter='relationships:\n - target: "[[concepts/plan]]"\n type: uses\n',
|
||||
)
|
||||
findings = lint_vault(tmp_path, require_trust_ledger=False)["findings"]
|
||||
assert findings["broken_links"] == []
|
||||
assert findings["typed_relationship_issues"] == []
|
||||
|
||||
|
||||
def test_lint_ignores_links_in_code_and_still_reports_real_breaks(tmp_path: Path) -> None:
|
||||
_write(tmp_path, "concepts/a.md", "Write `[[wikilinks]]` like this.\n\n```\n[[example]]\n```\n[[missing]]")
|
||||
broken = lint_vault(tmp_path, require_trust_ledger=False)["findings"]["broken_links"]
|
||||
assert broken == [{"page": "concepts/a.md", "target": "missing"}]
|
||||
|
||||
|
||||
def test_agent_instruction_files_are_not_pages(tmp_path: Path) -> None:
|
||||
_write(tmp_path, "concepts/a.md", "Conventions live in [[CLAUDE]].") # still a real link target
|
||||
for rel in ("AGENTS.md", "README.md", "projects/alpha/CLAUDE.md", "projects/alpha/AGENTS.md"):
|
||||
(tmp_path / rel).parent.mkdir(parents=True, exist_ok=True)
|
||||
(tmp_path / rel).write_text("# Instructions\n", encoding="utf-8")
|
||||
_write(tmp_path, "projects/alpha/README.md") # below the root a README can be a page
|
||||
|
||||
report = lint_vault(tmp_path, require_trust_ledger=False)
|
||||
assert report["stats"]["pages"] == 2
|
||||
assert "README" not in str(report["findings"]["missing_frontmatter"])
|
||||
assert report["findings"]["broken_links"] == []
|
||||
+1067
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,943 @@
|
||||
"""Tests for the shared memory writer (`obsidian_wiki.memory`).
|
||||
|
||||
These cover the guarantees the prose instructions could not make: that a
|
||||
parallel writer cannot drop an update, that the hot cache honours its word cap,
|
||||
that reconciling the index does not eat a human's own section, and that the
|
||||
profile and todo tables round-trip content containing table syntax.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import concurrent.futures
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from obsidian_wiki import memory as mem
|
||||
|
||||
|
||||
def _page(vault: Path, rel: str, **fields: str) -> Path:
|
||||
path = vault / rel
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
header = "\n".join(f"{key}: {value}" for key, value in fields.items())
|
||||
path.write_text(f"---\n{header}\n---\n\n# {fields.get('title', rel)}\n\nBody.\n", encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def vault(tmp_path: Path) -> Path:
|
||||
for name in ("concepts", "entities", "projects", "_meta"):
|
||||
(tmp_path / name).mkdir(parents=True)
|
||||
_page(tmp_path, "concepts/attention.md", title="Attention", summary="Core transformer block", tags="[ml, nlp]", updated="2026-09-01")
|
||||
_page(tmp_path, "entities/karpathy.md", title="Andrej Karpathy", summary="AI researcher", tags="[person]", updated="2026-09-10")
|
||||
return tmp_path
|
||||
|
||||
|
||||
def _cli(*args: str) -> subprocess.CompletedProcess:
|
||||
return subprocess.run(
|
||||
[sys.executable, "-m", "obsidian_wiki.cli", "memory", *args],
|
||||
capture_output=True, text=True,
|
||||
)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# frontmatter
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
@pytest.mark.parametrize("indicator", [">", ">-", ">+", "|", "|-", "|+", ">2", ">-2", ">+2", "|2", "|-2", "|+2"])
|
||||
def test_frontmatter_folds_block_scalars(indicator: str) -> None:
|
||||
values = mem.parse_frontmatter(
|
||||
f"title: {indicator}\n A sample\n page title\n"
|
||||
f"summary: {indicator}\n\n First line: details\n \n second line\n\n"
|
||||
"tags: [example, test]\nupdated: 2026-09-26\n"
|
||||
)
|
||||
assert values == {
|
||||
"title": "A sample page title",
|
||||
"summary": "First line: details second line",
|
||||
"tags": ["example", "test"],
|
||||
"updated": "2026-09-26",
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("indent", [" ", " ", " ", "\t"])
|
||||
def test_frontmatter_block_scalar_at_end_of_input(indent: str) -> None:
|
||||
assert mem.parse_frontmatter(f"summary: >-\n{indent}Last field") == {"summary": "Last field"}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("ending", ["", "\n", "\n \n\n", "\ntags: [test]\n"])
|
||||
def test_frontmatter_empty_block_scalar(ending: str) -> None:
|
||||
values = mem.parse_frontmatter("summary: >-" + ending)
|
||||
assert values["summary"] == ""
|
||||
if "tags:" in ending:
|
||||
assert values["tags"] == ["test"]
|
||||
|
||||
|
||||
def test_frontmatter_preserves_inline_scalars_and_lists() -> None:
|
||||
assert mem.parse_frontmatter(
|
||||
'title: ">-"\nsummary: > not a block scalar\n'
|
||||
'tags: [one, "two"]\nsources:\n - first.md\n - second.md\n'
|
||||
'relationships:\n - target: "[[concepts/example]]"\n type: contradicts\n'
|
||||
) == {
|
||||
"title": ">-",
|
||||
"summary": "> not a block scalar",
|
||||
"tags": ["one", "two"],
|
||||
"sources": ["first.md", "second.md"],
|
||||
"relationships": ['target: "[[concepts/example]]"', "type: contradicts"],
|
||||
}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# log.md
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_log_line_matches_the_documented_format(vault: Path) -> None:
|
||||
line = mem.append_log(vault, "INGEST", {"source": "papers/attention.pdf", "pages_created": 3})
|
||||
assert line.startswith("- [")
|
||||
entry = mem.parse_log_line(line)
|
||||
assert entry is not None
|
||||
assert entry.verb == "INGEST"
|
||||
assert entry.fields == {"source": "papers/attention.pdf", "pages_created": "3"}
|
||||
|
||||
|
||||
def test_log_quotes_values_containing_spaces(vault: Path) -> None:
|
||||
line = mem.append_log(vault, "QUERY", {"query": "how do transformers work", "result_pages": 4})
|
||||
assert 'query="how do transformers work"' in line
|
||||
assert mem.parse_log_line(line).fields["query"] == "how do transformers work"
|
||||
|
||||
|
||||
def test_log_rejects_a_lowercase_verb(vault: Path) -> None:
|
||||
with pytest.raises(mem.MemoryError_) as excinfo:
|
||||
mem.append_log(vault, "ingest something")
|
||||
assert excinfo.value.code == "bad_verb"
|
||||
|
||||
|
||||
def test_log_creates_the_file_and_never_rewrites_history(vault: Path) -> None:
|
||||
first = mem.append_log(vault, "INIT", {"pages": 2})
|
||||
mem.append_log(vault, "LINT", {"issues_found": 0})
|
||||
text = (vault / "log.md").read_text(encoding="utf-8")
|
||||
assert text.startswith("---\ntitle: Wiki Log\n---")
|
||||
assert first in text
|
||||
assert len(mem.read_log(vault)) == 2
|
||||
|
||||
|
||||
def test_log_filters_by_verb_and_limit(vault: Path) -> None:
|
||||
for index in range(5):
|
||||
mem.append_log(vault, "INGEST", {"n": index})
|
||||
mem.append_log(vault, "LINT", {"issues_found": 1})
|
||||
assert len(mem.read_log(vault, verbs=["INGEST"])) == 5
|
||||
assert len(mem.read_log(vault, limit=2)) == 2
|
||||
|
||||
|
||||
def test_parallel_writers_do_not_drop_a_log_line(vault: Path) -> None:
|
||||
"""The regression the lock exists for: wholesale rewrites lost updates."""
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=8) as pool:
|
||||
list(pool.map(lambda n: mem.append_log(vault, "CAPTURE", {"n": n}), range(60)))
|
||||
entries = mem.read_log(vault, verbs=["CAPTURE"])
|
||||
assert len(entries) == 60
|
||||
assert {entry.fields["n"] for entry in entries} == {str(n) for n in range(60)}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# index.md
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_index_lists_every_page_under_its_category(vault: Path) -> None:
|
||||
result = mem.rebuild_index(vault)
|
||||
text = (vault / "index.md").read_text(encoding="utf-8")
|
||||
assert result.total == 2
|
||||
assert "## Concepts" in text and "## Entities" in text
|
||||
assert "[[concepts/attention]]" in text
|
||||
assert set(result.added) == {"concepts/attention", "entities/karpathy"}
|
||||
|
||||
|
||||
def test_index_keeps_the_space_after_the_opening_paren(vault: Path) -> None:
|
||||
"""The documented format rule — `(#tag)` breaks Obsidian tag parsing."""
|
||||
mem.rebuild_index(vault)
|
||||
text = (vault / "index.md").read_text(encoding="utf-8")
|
||||
assert "( #ml #nlp)" in text
|
||||
assert "(#ml" not in text
|
||||
|
||||
|
||||
def test_index_honours_the_markdown_link_format(vault: Path) -> None:
|
||||
mem.rebuild_index(vault, link_format="markdown")
|
||||
text = (vault / "index.md").read_text(encoding="utf-8")
|
||||
assert "[Attention](concepts/attention.md)" in text
|
||||
assert "[[" not in text
|
||||
|
||||
|
||||
def test_index_round_trips_names_a_wikilink_cannot_carry(vault: Path) -> None:
|
||||
"""`]`, `#` and `|` cannot appear in a wikilink target, so such entries render
|
||||
as markdown links and the parser has to read them back as themselves (#294)."""
|
||||
_page(vault, "concepts/2026-06[final].md", title="2026-06[final]")
|
||||
_page(vault, "concepts/notes#1.md", title="notes#1")
|
||||
mem.rebuild_index(vault)
|
||||
|
||||
text = (vault / "index.md").read_text(encoding="utf-8")
|
||||
assert r"[2026-06[final\]](concepts/2026-06[final].md)" in text
|
||||
assert "[notes#1](concepts/notes#1.md)" in text
|
||||
assert "[[concepts/notes#1]]" not in text
|
||||
|
||||
result = mem.rebuild_index(vault, write=False)
|
||||
|
||||
assert result.added == ()
|
||||
assert result.removed == ()
|
||||
assert result.changed is False
|
||||
|
||||
|
||||
def test_index_drops_a_deleted_page_but_keeps_a_human_section(vault: Path) -> None:
|
||||
mem.rebuild_index(vault)
|
||||
index = vault / "index.md"
|
||||
index.write_text(index.read_text(encoding="utf-8") + "\n## Reading Queue\n\n- A book I mean to read\n", encoding="utf-8")
|
||||
(vault / "concepts/attention.md").unlink()
|
||||
|
||||
result = mem.rebuild_index(vault)
|
||||
text = index.read_text(encoding="utf-8")
|
||||
assert result.removed == ("concepts/attention",)
|
||||
assert "concepts/attention" not in text
|
||||
assert "A book I mean to read" in text
|
||||
|
||||
|
||||
def test_index_drops_a_stale_section_when_its_category_empties(vault: Path) -> None:
|
||||
"""Deriving categories from pages alone stranded the emptied section."""
|
||||
mem.rebuild_index(vault)
|
||||
(vault / "concepts/attention.md").unlink()
|
||||
mem.rebuild_index(vault)
|
||||
text = (vault / "index.md").read_text(encoding="utf-8")
|
||||
assert "## Concepts" not in text
|
||||
assert "## Entities" in text
|
||||
|
||||
|
||||
def test_index_drops_a_stale_uncategorized_section_and_reports_the_removal(vault: Path) -> None:
|
||||
"""`_heading_for(None)` renders "Uncategorized", but that heading was never
|
||||
in the generated-heading set, so a root-level page's stale entry survived
|
||||
as if hand-written and the removal never showed up in `removed`."""
|
||||
_page(vault, "loose-note.md", title="Loose Note")
|
||||
mem.rebuild_index(vault)
|
||||
(vault / "loose-note.md").unlink()
|
||||
|
||||
result = mem.rebuild_index(vault)
|
||||
text = (vault / "index.md").read_text(encoding="utf-8")
|
||||
assert "## Uncategorized" not in text
|
||||
assert "loose-note" not in text
|
||||
assert result.removed == ("loose-note",)
|
||||
|
||||
|
||||
def test_index_check_mode_reports_drift_without_writing(vault: Path) -> None:
|
||||
mem.rebuild_index(vault)
|
||||
before = (vault / "index.md").read_text(encoding="utf-8")
|
||||
_page(vault, "concepts/rag.md", title="RAG", summary="Retrieval augmented generation")
|
||||
result = mem.rebuild_index(vault, write=False)
|
||||
assert result.changed is True
|
||||
assert result.added == ("concepts/rag",)
|
||||
assert (vault / "index.md").read_text(encoding="utf-8") == before
|
||||
|
||||
|
||||
def test_index_ignores_staging_and_memory_files(vault: Path) -> None:
|
||||
_page(vault, "_raw/2026-09-21-scratch.md", title="Scratch")
|
||||
_page(vault, "_staging/concepts/pending.md", title="Pending")
|
||||
_page(vault, "_meta/taxonomy.md", title="Taxonomy")
|
||||
result = mem.rebuild_index(vault)
|
||||
text = (vault / "index.md").read_text(encoding="utf-8")
|
||||
assert result.total == 2
|
||||
for excluded in ("scratch", "pending", "taxonomy"):
|
||||
assert excluded not in text
|
||||
|
||||
|
||||
def test_index_drift_ignores_links_inside_an_entry_summary(vault: Path) -> None:
|
||||
"""Only an entry's own link says what the index lists. A wikilink in the
|
||||
page's summary text was counted too, so a summary pointing at a short-form
|
||||
or not-yet-written page reported a phantom removal on every rebuild."""
|
||||
_page(vault, "concepts/rag.md", title="RAG", summary='"See [[overview|the overview]] first"')
|
||||
mem.rebuild_index(vault)
|
||||
result = mem.rebuild_index(vault)
|
||||
assert result.removed == ()
|
||||
assert result.added == ()
|
||||
assert result.changed is False
|
||||
|
||||
|
||||
def test_index_is_idempotent(vault: Path) -> None:
|
||||
mem.rebuild_index(vault)
|
||||
first = (vault / "index.md").read_text(encoding="utf-8")
|
||||
second = mem.rebuild_index(vault)
|
||||
assert second.changed is False
|
||||
assert second.text == first
|
||||
|
||||
|
||||
def _sub_wiki(vault: Path) -> None:
|
||||
"""projects/alpha is a sub-wiki: its own index.md plus pages under it."""
|
||||
hub = vault / "projects/alpha/index.md"
|
||||
hub.parent.mkdir(parents=True, exist_ok=True)
|
||||
hub.write_text("# Alpha Project\n\nEverything about alpha.\n", encoding="utf-8")
|
||||
_page(vault, "projects/alpha/concepts/plan.md", title="Plan", summary="The plan")
|
||||
_page(vault, "projects/alpha/entities/bob.md", title="Bob", summary="A person")
|
||||
_page(vault, "projects/alpha/deep/index.md", title="Deep", summary="A nested hub")
|
||||
|
||||
|
||||
def _enable_hubs(vault: Path) -> None:
|
||||
(vault / "index.md").write_text(
|
||||
f"---\ntitle: Wiki Index\n{mem.FOLDER_HUBS_KEY}: true\n"
|
||||
f"generated_by: {mem.GENERATED_MARKER} index\n---\n\n# Wiki Index\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
def test_index_lists_sub_wiki_pages_individually_by_default(vault: Path) -> None:
|
||||
_sub_wiki(vault)
|
||||
text = mem.rebuild_index(vault).text
|
||||
assert "[[projects/alpha/concepts/plan]]" in text
|
||||
assert "(3 pages)" not in text
|
||||
|
||||
|
||||
def test_folder_hubs_link_a_sub_wiki_once_instead_of_inlining_it(vault: Path) -> None:
|
||||
"""A vault of sub-wikis would otherwise inline every page of every one of
|
||||
them into the root index, which every skill reads at startup."""
|
||||
_sub_wiki(vault)
|
||||
_enable_hubs(vault)
|
||||
|
||||
result = mem.rebuild_index(vault)
|
||||
text = (vault / "index.md").read_text(encoding="utf-8")
|
||||
assert "- [[projects/alpha/index|Alpha Project]] — Everything about alpha. (3 pages)" in text
|
||||
for inlined in ("projects/alpha/concepts/plan", "projects/alpha/entities/bob", "projects/alpha/deep"):
|
||||
assert inlined not in text # including the nested hub: the outermost wins
|
||||
assert "[[concepts/attention]]" in text # pages outside a hub are unaffected
|
||||
assert result.total == 6
|
||||
assert "projects/alpha/index" in result.added
|
||||
assert f"{mem.FOLDER_HUBS_KEY}: true" in text # the switch survives the rebuild
|
||||
assert mem.rebuild_index(vault).changed is False
|
||||
|
||||
|
||||
def test_folder_hub_title_falls_back_to_the_folder_name(vault: Path) -> None:
|
||||
hub = vault / "projects/beta-tool/index.md"
|
||||
hub.parent.mkdir(parents=True)
|
||||
hub.write_text("Just a list of links.\n", encoding="utf-8")
|
||||
_enable_hubs(vault)
|
||||
assert "[[projects/beta-tool/index|Beta Tool]]" in mem.rebuild_index(vault).text
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# hot.md
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_hot_has_the_four_documented_sections(vault: Path) -> None:
|
||||
mem.rebuild_hot(vault)
|
||||
text = (vault / "hot.md").read_text(encoding="utf-8")
|
||||
for heading in ("Recent Activity", "Active Threads", "Key Takeaways", "Flagged Contradictions"):
|
||||
assert f"## {heading}" in text
|
||||
assert "generated_by: obsidian-wiki memory hot" in text
|
||||
|
||||
|
||||
def test_hot_preserves_key_takeaways_across_a_rebuild(vault: Path) -> None:
|
||||
"""The one LLM-owned slot: mechanical rebuilds must not erase it."""
|
||||
mem.rebuild_hot(vault, takeaways="Retrieval is lexical only; precision is the weak metric.")
|
||||
mem.append_log(vault, "INGEST", {"source": "x"})
|
||||
mem.rebuild_hot(vault)
|
||||
assert "Retrieval is lexical only" in (vault / "hot.md").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_hot_replaces_takeaways_when_given_new_text(vault: Path) -> None:
|
||||
mem.rebuild_hot(vault, takeaways="Old conclusion.")
|
||||
mem.rebuild_hot(vault, takeaways="New conclusion.")
|
||||
text = (vault / "hot.md").read_text(encoding="utf-8")
|
||||
assert "New conclusion." in text and "Old conclusion." not in text
|
||||
|
||||
|
||||
def test_hot_enforces_the_word_cap(vault: Path) -> None:
|
||||
for index in range(50):
|
||||
mem.append_log(vault, "INGEST", {"source": f"file-{index}.pdf", "note": "a fairly long note value"})
|
||||
for index in range(20):
|
||||
mem.add_todo(vault, f"Thread number {index} with a reasonably long description")
|
||||
result = mem.rebuild_hot(vault, takeaways=" ".join(["takeaway"] * 400), max_words=200, activity_limit=40)
|
||||
assert result.trimmed is True
|
||||
assert result.words <= 200
|
||||
assert mem.content_words((vault / "hot.md").read_text(encoding="utf-8")) <= 200
|
||||
|
||||
|
||||
def test_hot_word_cap_excludes_frontmatter_and_boilerplate(vault: Path) -> None:
|
||||
"""Counting the header against the cap put the floor within a few words of it."""
|
||||
text = (vault / "hot.md")
|
||||
result = mem.rebuild_hot(vault)
|
||||
raw = text.read_text(encoding="utf-8")
|
||||
assert result.words == mem.content_words(raw)
|
||||
assert result.words < mem.word_count(raw)
|
||||
|
||||
|
||||
def test_hot_respects_the_env_override(vault: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("OBSIDIAN_HOT_MAX_WORDS", "80")
|
||||
assert mem.hot_max_words() == 80
|
||||
for index in range(30):
|
||||
mem.append_log(vault, "INGEST", {"source": f"file-{index}.pdf"})
|
||||
assert mem.rebuild_hot(vault, activity_limit=30).words <= 80
|
||||
|
||||
|
||||
def test_hot_falls_back_when_the_env_override_is_junk(vault: Path, monkeypatch: pytest.MonkeyPatch) -> None:
|
||||
monkeypatch.setenv("OBSIDIAN_HOT_MAX_WORDS", "not-a-number")
|
||||
assert mem.hot_max_words() == mem.DEFAULT_HOT_MAX_WORDS
|
||||
|
||||
|
||||
def test_hot_lists_open_threads_and_omits_closed_ones(vault: Path) -> None:
|
||||
open_todo = mem.add_todo(vault, "Wire the systemd timer")
|
||||
closed = mem.add_todo(vault, "Already finished")
|
||||
mem.set_todo_status(vault, closed.id, "done")
|
||||
mem.rebuild_hot(vault)
|
||||
text = (vault / "hot.md").read_text(encoding="utf-8")
|
||||
assert open_todo.text in text
|
||||
assert "Already finished" not in text
|
||||
|
||||
|
||||
def test_hot_flags_disputed_pages_and_contradiction_edges(vault: Path) -> None:
|
||||
_page(vault, "concepts/scaling.md", title="Scaling", lifecycle="disputed", updated="2026-09-11")
|
||||
(vault / "concepts/rebuttal.md").write_text(
|
||||
"---\ntitle: Rebuttal\nupdated: 2026-09-12\nrelationships:\n"
|
||||
' - target: "[[concepts/scaling]]"\n type: contradicts\n---\n\nBody.\n',
|
||||
encoding="utf-8",
|
||||
)
|
||||
mem.rebuild_hot(vault)
|
||||
text = (vault / "hot.md").read_text(encoding="utf-8")
|
||||
assert "marked disputed" in text
|
||||
assert "contradicts [[concepts/scaling]]" in text
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# profile
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_profile_round_trips_a_fact(vault: Path) -> None:
|
||||
mem.set_fact(vault, "stack", "Python, FastAPI", confidence=0.85, source="session:abc")
|
||||
facts = mem.load_profile(vault)
|
||||
assert len(facts) == 1
|
||||
assert facts[0].key == "stack"
|
||||
assert facts[0].value == "Python, FastAPI"
|
||||
assert facts[0].confidence == pytest.approx(0.85)
|
||||
assert facts[0].source == "session:abc"
|
||||
|
||||
|
||||
def test_profile_round_trips_a_value_containing_a_pipe(vault: Path) -> None:
|
||||
"""A raw `|` would tear the markdown row into extra cells."""
|
||||
mem.set_fact(vault, "shell", "bash | zsh")
|
||||
assert mem.load_profile(vault)[0].value == "bash | zsh"
|
||||
|
||||
|
||||
def test_profile_set_replaces_rather_than_duplicates(vault: Path) -> None:
|
||||
mem.set_fact(vault, "editor", "vim", confidence=0.5)
|
||||
mem.set_fact(vault, "EDITOR", "neovim", confidence=0.9)
|
||||
facts = mem.load_profile(vault)
|
||||
assert len(facts) == 1
|
||||
assert facts[0].value == "neovim"
|
||||
|
||||
|
||||
def test_profile_rejects_confidence_outside_the_unit_interval(vault: Path) -> None:
|
||||
for bad in (-0.1, 1.5):
|
||||
with pytest.raises(mem.MemoryError_) as excinfo:
|
||||
mem.set_fact(vault, "k", "v", confidence=bad)
|
||||
assert excinfo.value.code == "bad_confidence"
|
||||
|
||||
|
||||
def test_profile_forget_reports_whether_it_removed_anything(vault: Path) -> None:
|
||||
mem.set_fact(vault, "timezone", "Asia/Kolkata")
|
||||
assert mem.forget_fact(vault, "timezone") is True
|
||||
assert mem.forget_fact(vault, "timezone") is False
|
||||
assert mem.load_profile(vault) == []
|
||||
|
||||
|
||||
def test_profile_survives_a_hand_edit_in_obsidian(vault: Path) -> None:
|
||||
mem.set_fact(vault, "stack", "Python")
|
||||
path = vault / mem.PROFILE_REL
|
||||
path.write_text(
|
||||
path.read_text(encoding="utf-8") + "| timezone | Asia/Kolkata | 0.90 | manual | 2026-09-21 |\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
assert {fact.key for fact in mem.load_profile(vault)} == {"stack", "timezone"}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# todos
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_todo_add_assigns_sequential_ids(vault: Path) -> None:
|
||||
assert mem.add_todo(vault, "First").id == "t1"
|
||||
assert mem.add_todo(vault, "Second").id == "t2"
|
||||
|
||||
|
||||
def test_todo_add_is_idempotent_for_an_open_thread(vault: Path) -> None:
|
||||
first = mem.add_todo(vault, "Wire the systemd timer")
|
||||
again = mem.add_todo(vault, "wire the SYSTEMD timer")
|
||||
assert first.id == again.id
|
||||
assert len(mem.load_todos(vault)) == 1
|
||||
|
||||
|
||||
def test_todo_status_transitions_and_prune(vault: Path) -> None:
|
||||
keep = mem.add_todo(vault, "Still open")
|
||||
done = mem.add_todo(vault, "Finished")
|
||||
dropped = mem.add_todo(vault, "Abandoned")
|
||||
mem.set_todo_status(vault, done.id, "done")
|
||||
mem.set_todo_status(vault, dropped.id, "dropped")
|
||||
assert mem.prune_todos(vault) == 2
|
||||
remaining = mem.load_todos(vault)
|
||||
assert [todo.id for todo in remaining] == [keep.id]
|
||||
|
||||
|
||||
def test_todo_rejects_an_unknown_id_and_status(vault: Path) -> None:
|
||||
todo = mem.add_todo(vault, "Something")
|
||||
with pytest.raises(mem.MemoryError_) as unknown:
|
||||
mem.set_todo_status(vault, "t99", "done")
|
||||
assert unknown.value.code == "no_such_todo"
|
||||
with pytest.raises(mem.MemoryError_) as bad:
|
||||
mem.set_todo_status(vault, todo.id, "maybe")
|
||||
assert bad.value.code == "bad_status"
|
||||
|
||||
|
||||
def test_todo_staleness_is_reported_not_enforced(vault: Path) -> None:
|
||||
mem.add_todo(vault, "Old thread")
|
||||
path = vault / mem.TODOS_REL
|
||||
old = (datetime.now(timezone.utc) - timedelta(days=45)).date().isoformat()
|
||||
path.write_text(path.read_text(encoding="utf-8").replace(mem.today(), old), encoding="utf-8")
|
||||
todo = mem.load_todos(vault)[0]
|
||||
assert todo.is_stale() is True
|
||||
assert todo.status == "open" # still open; staleness is a signal, not a close
|
||||
|
||||
|
||||
def test_closed_todos_are_never_stale(vault: Path) -> None:
|
||||
todo = mem.add_todo(vault, "Done long ago")
|
||||
mem.set_todo_status(vault, todo.id, "done")
|
||||
assert mem.load_todos(vault)[0].is_stale(days=0) is False
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# recap + status
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_recap_gathers_profile_threads_and_activity(vault: Path) -> None:
|
||||
mem.set_fact(vault, "stack", "Python")
|
||||
mem.add_todo(vault, "Persist the retrieval index")
|
||||
mem.append_log(vault, "INGEST", {"source": "papers/rag.pdf"})
|
||||
recap = mem.build_recap(vault)
|
||||
assert "**stack**: Python" in recap
|
||||
assert "Persist the retrieval index" in recap
|
||||
assert "INGEST" in recap
|
||||
|
||||
|
||||
def test_recap_drops_a_time_bound_fact_once_it_is_over(vault: Path) -> None:
|
||||
""""Exams this weekend" must not still be injected next month."""
|
||||
mem.set_fact(vault, "exams", "this weekend", until="2000-01-02")
|
||||
mem.set_fact(vault, "trip", "in Lisbon", until="2999-01-01")
|
||||
facts = {f.key: f for f in mem.load_profile(vault)}
|
||||
assert facts["exams"].until == "2000-01-02" and facts["exams"].is_expired()
|
||||
recap = mem.build_recap(vault)
|
||||
assert "exams" not in recap
|
||||
assert "**trip**: in Lisbon" in recap
|
||||
with pytest.raises(mem.MemoryError_) as excinfo:
|
||||
mem.set_fact(vault, "k", "v", until="next friday")
|
||||
assert excinfo.value.code == "bad_until"
|
||||
|
||||
|
||||
def test_recap_is_honest_about_an_empty_vault(vault: Path) -> None:
|
||||
assert "No vault memory recorded yet" in mem.build_recap(vault)
|
||||
|
||||
|
||||
def test_recap_suggests_a_first_step_until_the_vault_is_used(vault: Path) -> None:
|
||||
mem.append_log(vault, "INIT", {"vault_path": str(vault)})
|
||||
recap = mem.build_recap(vault)
|
||||
assert "## Getting started" in recap and "/wiki-capture" in recap
|
||||
mem.append_log(vault, "INGEST", {"source": "notes.md"})
|
||||
assert "Getting started" not in mem.build_recap(vault)
|
||||
assert "Getting started" not in mem.build_recap(vault, project="nomatch")
|
||||
|
||||
|
||||
def test_recap_filters_low_confidence_facts(vault: Path) -> None:
|
||||
mem.set_fact(vault, "guess", "maybe Postgres", confidence=0.2)
|
||||
mem.set_fact(vault, "certain", "Python", confidence=0.95)
|
||||
recap = mem.build_recap(vault, min_confidence=0.5)
|
||||
assert "certain" in recap and "guess" not in recap
|
||||
|
||||
|
||||
def test_recap_respects_its_word_budget(vault: Path) -> None:
|
||||
for index in range(60):
|
||||
mem.set_fact(vault, f"fact-{index}", f"a reasonably long value number {index}")
|
||||
assert mem.word_count(mem.build_recap(vault, max_words=100)) <= 101 # +1 for the ellipsis
|
||||
|
||||
|
||||
def test_status_reports_index_drift_and_hot_budget(vault: Path) -> None:
|
||||
status = mem.memory_status(vault)
|
||||
assert status["pages"] == 2
|
||||
assert status["index_drift"]["stale"] is True
|
||||
assert status["hot"]["generated"] is False
|
||||
|
||||
mem.rebuild_index(vault)
|
||||
mem.rebuild_hot(vault)
|
||||
status = mem.memory_status(vault)
|
||||
assert status["index_drift"]["stale"] is False
|
||||
assert status["hot"]["generated"] is True
|
||||
assert status["hot"]["over_budget"] is False
|
||||
|
||||
|
||||
def test_operations_refuse_a_missing_vault(tmp_path: Path) -> None:
|
||||
missing = tmp_path / "nope"
|
||||
for call in (
|
||||
lambda: mem.append_log(missing, "INGEST"),
|
||||
lambda: mem.rebuild_index(missing),
|
||||
lambda: mem.rebuild_hot(missing),
|
||||
lambda: mem.build_recap(missing),
|
||||
):
|
||||
with pytest.raises(mem.MemoryError_) as excinfo:
|
||||
call()
|
||||
assert excinfo.value.code == "vault_not_found"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# CLI surface
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_cli_status_and_sync_round_trip(vault: Path) -> None:
|
||||
stale = _cli("status", "--vault", str(vault), "--json")
|
||||
assert stale.returncode == 0
|
||||
assert '"stale": true' in stale.stdout
|
||||
|
||||
synced = _cli("sync", "--verb", "INGEST", "--field", "source=papers/rag.pdf", "--vault", str(vault))
|
||||
assert synced.returncode == 0
|
||||
assert "INGEST" in synced.stdout
|
||||
|
||||
after = _cli("status", "--vault", str(vault), "--json")
|
||||
assert '"stale": false' in after.stdout
|
||||
assert (vault / "index.md").is_file()
|
||||
assert (vault / "hot.md").is_file()
|
||||
|
||||
|
||||
def test_cli_index_check_exits_two_on_drift(vault: Path) -> None:
|
||||
assert _cli("index", "--vault", str(vault)).returncode == 0
|
||||
assert _cli("index", "--check", "--vault", str(vault)).returncode == 0
|
||||
_page(vault, "concepts/rag.md", title="RAG", summary="Retrieval augmented generation")
|
||||
drifted = _cli("index", "--check", "--vault", str(vault))
|
||||
assert drifted.returncode == 2
|
||||
assert "concepts/rag" in drifted.stdout
|
||||
|
||||
|
||||
@pytest.mark.parametrize("command", ["index", "sync"])
|
||||
@pytest.mark.parametrize("link_format", ["markdown", "wikilink"])
|
||||
def test_cli_indexes_block_scalar_titles_and_summaries(
|
||||
vault: Path, command: str, link_format: str, monkeypatch: pytest.MonkeyPatch,
|
||||
) -> None:
|
||||
config_dir = vault / ".obsidian-wiki"
|
||||
config_dir.mkdir(parents=True)
|
||||
(config_dir / "config").write_text(f"OBSIDIAN_LINK_FORMAT={link_format}\n", encoding="utf-8")
|
||||
monkeypatch.setenv("HOME", str(vault))
|
||||
monkeypatch.setenv("XDG_CONFIG_HOME", str(config_dir.parent))
|
||||
_page(
|
||||
vault, "concepts/block-scalars.md",
|
||||
title=">-\n A sample\n page title",
|
||||
summary="|-\n A summary\n spanning two lines.",
|
||||
tags="[example]",
|
||||
)
|
||||
result = _cli(command, "--vault", str(vault))
|
||||
assert result.returncode == 0, result.stderr
|
||||
text = (vault / "index.md").read_text(encoding="utf-8")
|
||||
link = "[A sample page title](concepts/block-scalars.md)" if link_format == "markdown" else "[[concepts/block-scalars]]"
|
||||
assert f"- {link} — A summary spanning two lines. ( #example)" in text
|
||||
assert ">-" not in text and "|-" not in text
|
||||
assert _cli("index", "--check", "--vault", str(vault)).returncode == 0
|
||||
|
||||
|
||||
def test_cli_rejects_a_malformed_field(vault: Path) -> None:
|
||||
result = _cli("log", "INGEST", "--field", "no-equals-sign", "--vault", str(vault))
|
||||
assert result.returncode == 1
|
||||
assert "key=value" in result.stderr
|
||||
|
||||
|
||||
def test_cli_profile_and_todo_lifecycle(vault: Path) -> None:
|
||||
assert _cli("profile", "set", "stack", "Python, FastAPI", "--confidence", "0.8", "--vault", str(vault)).returncode == 0
|
||||
listed = _cli("profile", "list", "--vault", str(vault))
|
||||
assert "stack: Python, FastAPI" in listed.stdout
|
||||
|
||||
added = _cli("todo", "add", "Persist the retrieval index", "--vault", str(vault))
|
||||
assert added.returncode == 0
|
||||
assert _cli("todo", "done", "t1", "--vault", str(vault)).returncode == 0
|
||||
assert "no todos recorded" in _cli("todo", "list", "--vault", str(vault)).stdout
|
||||
assert "done" in _cli("todo", "list", "--all", "--vault", str(vault)).stdout
|
||||
|
||||
|
||||
def test_cli_forget_exits_one_when_nothing_was_removed(vault: Path) -> None:
|
||||
assert _cli("profile", "forget", "absent", "--vault", str(vault)).returncode == 1
|
||||
|
||||
|
||||
def test_cli_takeaways_can_be_piped_in(vault: Path) -> None:
|
||||
result = subprocess.run(
|
||||
[sys.executable, "-m", "obsidian_wiki.cli", "memory", "hot", "--takeaways", "-", "--vault", str(vault)],
|
||||
input="Retrieval is lexical only.", capture_output=True, text=True,
|
||||
)
|
||||
assert result.returncode == 0
|
||||
assert "Retrieval is lexical only." in (vault / "hot.md").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_cli_recap_prints_the_injectable_block(vault: Path) -> None:
|
||||
_cli("profile", "set", "stack", "Python", "--vault", str(vault))
|
||||
recap = _cli("recap", "--vault", str(vault))
|
||||
assert recap.returncode == 0
|
||||
assert recap.stdout.startswith("# Vault memory")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# migration — adopting a vault that predates the writer
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def _legacy(vault: Path) -> None:
|
||||
"""A hand-curated vault: custom index sections, narrative hot cache."""
|
||||
(vault / "index.md").write_text(
|
||||
"# Project Brain\n\nCentral index.\n\n"
|
||||
"## Project Index\n\n| Project | Repo |\n|---|---|\n| [[alpha]] | r |\n\n"
|
||||
"## Quick Reference\n\n```bash\ngh repo list\n```\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
(vault / "hot.md").write_text(
|
||||
"---\nupdated: 2026-06-04T00:00:00Z\n---\n\n# Hot\n\n"
|
||||
"## Recent Activity (last 3 ops)\n\n1. **2026-06-04** — researched karpathy\n\n"
|
||||
"## Active Threads\n\n"
|
||||
"- **karpathy-llm-resources** — Full coverage of the educational ecosystem.\n"
|
||||
"- **tractorex** — domain-configurable document intelligence.\n\n"
|
||||
"## Key Takeaways\n\n- Build the mechanism before the abstraction.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
|
||||
|
||||
|
||||
def test_the_guard_refuses_to_clobber_a_hand_curated_vault(vault: Path) -> None:
|
||||
_legacy(vault)
|
||||
for call in (lambda: mem.rebuild_index(vault), lambda: mem.rebuild_hot(vault)):
|
||||
with pytest.raises(mem.MemoryError_) as excinfo:
|
||||
call()
|
||||
assert excinfo.value.code == "unmigrated"
|
||||
assert "Project Brain" in (vault / "index.md").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_dry_runs_are_allowed_on_an_unmigrated_vault(vault: Path) -> None:
|
||||
_legacy(vault)
|
||||
assert mem.rebuild_index(vault, write=False, force=True).total == 2
|
||||
assert mem.migration_status(vault)["migrated"] is False
|
||||
|
||||
|
||||
def test_migration_preview_reports_what_would_change(vault: Path) -> None:
|
||||
_legacy(vault)
|
||||
status = mem.migration_status(vault)
|
||||
assert status["index"]["sections_preserved"] == ["Project Index", "Quick Reference"]
|
||||
assert status["hot"]["takeaways_carried"] is True
|
||||
assert set(status["hot"]["sections_regenerated"]) == {"Active Threads", "Recent Activity"}
|
||||
assert len(status["hot"]["threads_to_seed"]) == 2
|
||||
|
||||
|
||||
def test_migration_backs_up_before_writing(vault: Path) -> None:
|
||||
_legacy(vault)
|
||||
original = (vault / "index.md").read_text(encoding="utf-8")
|
||||
result = mem.migrate(vault)
|
||||
assert len(result["backup"]) == 2
|
||||
restored = Path(result["backup_dir"]) / "index.md"
|
||||
assert restored.read_text(encoding="utf-8") == original
|
||||
|
||||
|
||||
def test_migration_preserves_custom_sections_above_the_catalog(vault: Path) -> None:
|
||||
_legacy(vault)
|
||||
mem.migrate(vault)
|
||||
text = (vault / "index.md").read_text(encoding="utf-8")
|
||||
headings = re.findall(r"^##\s+(.+)$", text, re.MULTILINE)
|
||||
assert headings[:2] == ["Project Index", "Quick Reference"]
|
||||
assert "Concepts" in headings # catalog appended after
|
||||
assert "gh repo list" in text # the author's code block survives
|
||||
assert "# Project Brain" in text # and their preamble
|
||||
|
||||
|
||||
def test_migration_converts_narrative_threads_into_todos(vault: Path) -> None:
|
||||
"""The one thing the generator cannot reconstruct, so it is carried over."""
|
||||
_legacy(vault)
|
||||
result = mem.migrate(vault)
|
||||
assert len(result["threads_seeded"]) == 2
|
||||
texts = " ".join(todo.text for todo in mem.load_todos(vault))
|
||||
assert "karpathy" in texts and "tractorex" in texts
|
||||
assert "karpathy" in (vault / "hot.md").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_migration_never_overwrites_existing_todos(vault: Path) -> None:
|
||||
_legacy(vault)
|
||||
mem.add_todo(vault, "A thread I already had")
|
||||
result = mem.migrate(vault)
|
||||
assert result["threads_seeded"] == []
|
||||
assert [todo.text for todo in mem.load_todos(vault)] == ["A thread I already had"]
|
||||
|
||||
|
||||
def test_migration_marks_a_custom_preamble_so_it_is_not_asked_again(vault: Path) -> None:
|
||||
"""The marker has to land inside the author's own preamble, or the guard
|
||||
refuses forever and sync silently skips index and hot on every run."""
|
||||
_legacy(vault)
|
||||
mem.migrate(vault)
|
||||
assert mem.is_generated(vault / "index.md")
|
||||
mem.rebuild_index(vault) # must not raise now
|
||||
assert mem.rebuild_index(vault, write=False).changed is False
|
||||
|
||||
|
||||
def test_migration_is_idempotent(vault: Path) -> None:
|
||||
_legacy(vault)
|
||||
mem.migrate(vault)
|
||||
first = (vault / "index.md").read_text(encoding="utf-8")
|
||||
second = mem.migrate(vault)
|
||||
assert second["was_migrated"] is True
|
||||
assert (vault / "index.md").read_text(encoding="utf-8") == first
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# hot cache: log lines must not blow the budget
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_a_huge_log_line_is_summarised_not_pasted(vault: Path) -> None:
|
||||
"""A research ingest records every page it made in one field."""
|
||||
mem.append_log(vault, "WIKI_RESEARCH", {
|
||||
"source": "karpathy",
|
||||
"pages_created": 18,
|
||||
"note": " ".join(f"references/page-{n}" for n in range(40)),
|
||||
"extra": "dropped",
|
||||
})
|
||||
mem.rebuild_hot(vault)
|
||||
activity = mem._section_body((vault / "hot.md").read_text(encoding="utf-8"), "Recent Activity")
|
||||
assert len(activity.split()) < 25
|
||||
assert "page-39" not in activity
|
||||
assert "WIKI_RESEARCH" in activity
|
||||
|
||||
|
||||
def test_summarised_entries_keep_the_date_and_verb(vault: Path) -> None:
|
||||
entry = mem.parse_log_line(mem.format_log_line("INGEST", {"source": "a.pdf"}, timestamp="2026-09-21T10:00:00Z"))
|
||||
assert mem.summarize_log_entry(entry) == "- [2026-09-21] INGEST source=a.pdf"
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# recap scoping
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_recap_scopes_threads_to_a_project(vault: Path) -> None:
|
||||
mem.add_todo(vault, "Ship the tractorex pipeline")
|
||||
mem.add_todo(vault, "Read the karpathy series")
|
||||
scoped = mem.build_recap(vault, project="tractorex")
|
||||
assert "tractorex" in scoped
|
||||
assert "karpathy" not in scoped
|
||||
|
||||
|
||||
def test_recap_matches_a_project_named_in_the_origin(vault: Path) -> None:
|
||||
mem.add_todo(vault, "Something unrelated in wording", origin="projects/tractorex.md")
|
||||
assert "Something unrelated" in mem.build_recap(vault, project="tractorex")
|
||||
|
||||
|
||||
def test_recap_falls_back_rather_than_claiming_no_history(vault: Path) -> None:
|
||||
"""An unmatched filter must not look like an empty vault."""
|
||||
mem.add_todo(vault, "Only thread")
|
||||
mem.append_log(vault, "INGEST", {"source": "a.pdf"})
|
||||
recap = mem.build_recap(vault, project="nothing-matches-this")
|
||||
assert "Only thread" in recap
|
||||
assert "INGEST" in recap
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# adoption record — survives a legacy skill rewriting hot.md
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_a_legacy_rewrite_of_hot_md_does_not_unadopt_the_vault(vault: Path) -> None:
|
||||
"""Sixteen skills still rewrite hot.md by hand. If that dropped the
|
||||
per-file marker and nothing else recorded adoption, the next sync would
|
||||
refuse as though the vault had never been migrated."""
|
||||
_legacy(vault)
|
||||
mem.migrate(vault)
|
||||
(vault / "hot.md").write_text(
|
||||
"---\nupdated: now\n---\n# Hot\n\n## Key Takeaways\n\nWritten by an old skill.\n",
|
||||
encoding="utf-8",
|
||||
)
|
||||
assert not mem.is_generated(vault / "hot.md")
|
||||
assert mem.is_adopted(vault)
|
||||
result = mem.rebuild_hot(vault) # must not raise
|
||||
assert result.words > 0
|
||||
assert "Written by an old skill" in (vault / "hot.md").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_deleting_the_adoption_record_re_arms_the_guard(vault: Path) -> None:
|
||||
"""The documented way back: restore a curated file from the backup and
|
||||
delete the record. Both are needed — a regenerated file is legitimately
|
||||
ours and carries its own marker."""
|
||||
_legacy(vault)
|
||||
result = mem.migrate(vault)
|
||||
backup = Path(result["backup_dir"])
|
||||
(vault / "index.md").write_text((backup / "index.md").read_text(encoding="utf-8"), encoding="utf-8")
|
||||
(vault / mem.ADOPTED_REL).unlink()
|
||||
with pytest.raises(mem.MemoryError_) as excinfo:
|
||||
mem.rebuild_index(vault)
|
||||
assert excinfo.value.code == "unmigrated"
|
||||
|
||||
|
||||
|
||||
|
||||
def test_migrate_records_adoption(vault: Path) -> None:
|
||||
_legacy(vault)
|
||||
assert not mem.is_adopted(vault)
|
||||
mem.migrate(vault)
|
||||
assert mem.is_adopted(vault)
|
||||
assert mem.memory_status(vault)["migrated"] is True
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------
|
||||
# the short form — a verb and bare key=value pairs
|
||||
# --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def test_sync_takes_a_positional_verb_and_fields(vault: Path) -> None:
|
||||
"""The flag form ran to seven lines in the ingest skills."""
|
||||
result = _cli("sync", "INGEST", "source=papers/attention.pdf", "pages_created=3",
|
||||
"--vault", str(vault))
|
||||
assert result.returncode == 0, result.stderr
|
||||
entry = mem.read_log(vault, verbs=["INGEST"])[-1]
|
||||
assert entry.fields == {"source": "papers/attention.pdf", "pages_created": "3"}
|
||||
|
||||
|
||||
def test_log_takes_a_positional_verb_and_fields(vault: Path) -> None:
|
||||
assert _cli("log", "LINT", "issues_found=2", "orphans=1", "--vault", str(vault)).returncode == 0
|
||||
assert mem.read_log(vault, verbs=["LINT"])[-1].fields == {"issues_found": "2", "orphans": "1"}
|
||||
|
||||
|
||||
def test_the_old_flag_form_still_works(vault: Path) -> None:
|
||||
"""Back-compat: --verb/--field are documented as the older spelling."""
|
||||
assert _cli("log", "--verb", "DEDUP", "--field", "merged=2", "--vault", str(vault)).returncode == 0
|
||||
assert mem.read_log(vault, verbs=["DEDUP"])[-1].fields == {"merged": "2"}
|
||||
|
||||
|
||||
def test_both_forms_can_be_mixed(vault: Path) -> None:
|
||||
_cli("log", "QUERY", "query=how do transformers work", "--field", "result_pages=4",
|
||||
"--vault", str(vault))
|
||||
entry = mem.read_log(vault, verbs=["QUERY"])[-1]
|
||||
assert entry.fields == {"query": "how do transformers work", "result_pages": "4"}
|
||||
|
||||
|
||||
def test_a_value_may_contain_an_equals_sign(vault: Path) -> None:
|
||||
_cli("log", "INGEST", "source=https://x.test/a?b=c", "--vault", str(vault))
|
||||
assert mem.read_log(vault, verbs=["INGEST"])[-1].fields["source"] == "https://x.test/a?b=c"
|
||||
|
||||
|
||||
def test_sync_without_a_verb_still_reconciles(vault: Path) -> None:
|
||||
"""wiki-stage-commit's case: the log line was already written by the CLI."""
|
||||
result = _cli("sync", "--takeaways", "Committed 3 staged pages.", "--vault", str(vault))
|
||||
assert result.returncode == 0
|
||||
assert mem.read_log(vault) == []
|
||||
assert "Committed 3 staged pages." in (vault / "hot.md").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_a_bare_word_after_the_verb_is_a_clear_error(vault: Path) -> None:
|
||||
result = _cli("sync", "INGEST", "oops", "--vault", str(vault))
|
||||
assert result.returncode == 1
|
||||
assert "expected key=value" in result.stderr
|
||||
assert "INGEST" in result.stderr # says which verb it already has
|
||||
@@ -0,0 +1,220 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from obsidian_wiki.provenance import (
|
||||
archive_wikilink_relpath,
|
||||
expected_snapshots_for_page,
|
||||
invert_pages,
|
||||
parse_snapshots_field,
|
||||
prefer_archive_write_path,
|
||||
resolve_source_key,
|
||||
unwrap_snapshot_value,
|
||||
)
|
||||
|
||||
|
||||
def test_invert_unions_pages_produced_and_pages_created() -> None:
|
||||
sources = {
|
||||
"_raw/_archived/a.md": {
|
||||
"pages_produced": ["concepts/foo.md"],
|
||||
"pages_created": ["concepts/bar.md"],
|
||||
},
|
||||
"_raw/_archived/b.md": {"pages_produced": ["concepts/foo.md"]},
|
||||
}
|
||||
idx = invert_pages(sources)
|
||||
assert set(idx["concepts/foo.md"]) == {
|
||||
"_raw/_archived/a.md",
|
||||
"_raw/_archived/b.md",
|
||||
}
|
||||
assert idx["concepts/bar.md"] == ["_raw/_archived/a.md"]
|
||||
|
||||
|
||||
def test_invert_list_shaped_manifest() -> None:
|
||||
sources = [
|
||||
{
|
||||
"path": "_raw/_archived/a.md",
|
||||
"pages_produced": ["concepts/foo.md"],
|
||||
}
|
||||
]
|
||||
idx = invert_pages(sources)
|
||||
assert idx["concepts/foo.md"] == ["_raw/_archived/a.md"]
|
||||
|
||||
|
||||
def test_unwrap_wikilink_and_quotes() -> None:
|
||||
assert unwrap_snapshot_value("[[_raw/_archived/Old English]]") == (
|
||||
"_raw/_archived/Old English.md"
|
||||
)
|
||||
assert unwrap_snapshot_value('"_raw/_archived/a.md"') == "_raw/_archived/a.md"
|
||||
assert unwrap_snapshot_value(
|
||||
'"[a](_raw/_archived/a.md)"'
|
||||
) == "_raw/_archived/a.md"
|
||||
assert unwrap_snapshot_value(
|
||||
"[clip](_raw/_archived/topic/clip.md)"
|
||||
) == "_raw/_archived/topic/clip.md"
|
||||
assert unwrap_snapshot_value(
|
||||
"[[_raw/_archived/foo|foo]]"
|
||||
) == "_raw/_archived/foo.md"
|
||||
assert unwrap_snapshot_value("_raw/_archived/foo|foo") == (
|
||||
"_raw/_archived/foo.md"
|
||||
)
|
||||
|
||||
|
||||
def test_resolve_stale_raw_to_archived(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
archived = vault / "_raw" / "_archived" / "notes.md"
|
||||
archived.parent.mkdir(parents=True)
|
||||
archived.write_text("# notes\n", encoding="utf-8")
|
||||
assert resolve_source_key(vault, "_raw/notes.md") == "_raw/_archived/notes.md"
|
||||
|
||||
|
||||
def test_resolve_url_unique_clip(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
clip = vault / "_raw" / "_archived" / "clip.md"
|
||||
clip.parent.mkdir(parents=True)
|
||||
clip.write_text("---\nurl: https://example.com/x\n---\n", encoding="utf-8")
|
||||
assert resolve_source_key(vault, "url:https://example.com/x") == (
|
||||
"_raw/_archived/clip.md"
|
||||
)
|
||||
|
||||
|
||||
def test_resolve_url_ambiguous_returns_none(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
arch = vault / "_raw" / "_archived"
|
||||
arch.mkdir(parents=True)
|
||||
(arch / "a.md").write_text("---\nurl: https://dup.example/\n---\n", encoding="utf-8")
|
||||
(arch / "b.md").write_text("---\nurl: https://dup.example/\n---\n", encoding="utf-8")
|
||||
assert resolve_source_key(vault, "url:https://dup.example/") is None
|
||||
|
||||
|
||||
def test_resolve_agent_key_is_none(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
vault.mkdir()
|
||||
assert resolve_source_key(vault, "agent:claude/abc") is None
|
||||
|
||||
|
||||
def test_expected_snapshots_skips_unresolvable_keys(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
(vault / "_raw" / "_archived").mkdir(parents=True)
|
||||
(vault / "_raw" / "_archived" / "a.md").write_text("x\n", encoding="utf-8")
|
||||
sources = {
|
||||
"_raw/_archived/a.md": {"pages_produced": ["concepts/foo.md"]},
|
||||
"url:https://no-clip.example/": {"pages_produced": ["concepts/foo.md"]},
|
||||
"agent:claude/abc": {"pages_produced": ["concepts/foo.md"]},
|
||||
}
|
||||
assert expected_snapshots_for_page(vault, "concepts/foo.md", sources) == [
|
||||
"_raw/_archived/a.md"
|
||||
]
|
||||
|
||||
|
||||
def test_parse_snapshots_block_list() -> None:
|
||||
raw = (
|
||||
"\n - [[_raw/_archived/a.md]]\n"
|
||||
" - _raw/_archived/b.md\n"
|
||||
' - "[c](_raw/_archived/c.md)"\n'
|
||||
)
|
||||
assert parse_snapshots_field(raw) == [
|
||||
"_raw/_archived/a.md",
|
||||
"_raw/_archived/b.md",
|
||||
"_raw/_archived/c.md",
|
||||
]
|
||||
|
||||
|
||||
def test_invert_ten_thousand_keys() -> None:
|
||||
sources = {
|
||||
f"_raw/_archived/doc-{i}.md": {
|
||||
"pages_produced": [f"concepts/p{i % 50}.md"]
|
||||
}
|
||||
for i in range(10_000)
|
||||
}
|
||||
idx = invert_pages(sources)
|
||||
assert len(idx) == 50
|
||||
assert len(idx["concepts/p0.md"]) == 200
|
||||
|
||||
|
||||
def test_prefer_keeps_nested_archived_when_flat_namesake_exists(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
nested = vault / "_raw" / "_archived" / "topic" / "clip.md"
|
||||
flat = vault / "_raw" / "_archived" / "clip.md"
|
||||
nested.parent.mkdir(parents=True)
|
||||
nested.write_text("nested\n", encoding="utf-8")
|
||||
flat.write_text("flat\n", encoding="utf-8")
|
||||
assert prefer_archive_write_path(vault, "_raw/_archived/topic/clip.md") == (
|
||||
"_raw/_archived/topic/clip.md"
|
||||
)
|
||||
|
||||
|
||||
def test_prefer_stale_raw_uses_flat_archived_basename(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
archived = vault / "_raw" / "_archived" / "notes.md"
|
||||
archived.parent.mkdir(parents=True)
|
||||
archived.write_text("x\n", encoding="utf-8")
|
||||
assert prefer_archive_write_path(vault, "_raw/notes.md") == "_raw/_archived/notes.md"
|
||||
|
||||
|
||||
def test_prefer_live_draft_over_same_named_archive(tmp_path: Path) -> None:
|
||||
# A new _raw/notes.md next to an old archived notes.md is a new draft.
|
||||
vault = tmp_path / "vault"
|
||||
archived = vault / "_raw" / "_archived" / "notes.md"
|
||||
archived.parent.mkdir(parents=True)
|
||||
archived.write_text("x\n", encoding="utf-8")
|
||||
(vault / "_raw" / "notes.md").write_text("staging\n", encoding="utf-8")
|
||||
assert prefer_archive_write_path(vault, "_raw/notes.md") == "_raw/notes.md"
|
||||
|
||||
|
||||
def test_archive_basename_only_matches_raw_keys(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
archived = vault / "_raw" / "_archived" / "README.md"
|
||||
archived.parent.mkdir(parents=True)
|
||||
archived.write_text("x\n", encoding="utf-8")
|
||||
(vault / "docs").mkdir()
|
||||
(vault / "docs" / "README.md").write_text("doc\n", encoding="utf-8")
|
||||
assert prefer_archive_write_path(vault, "docs/README.md") == "docs/README.md"
|
||||
assert resolve_source_key(vault, "docs/README.md") is None
|
||||
assert resolve_source_key(vault, "projects/x/README.md") is None
|
||||
|
||||
|
||||
def test_prefer_missing_returns_none(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
vault.mkdir()
|
||||
assert prefer_archive_write_path(vault, "_raw/_archived/ghost.md") is None
|
||||
|
||||
|
||||
def test_archive_wikilink_relpath_existing_and_missing(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
clip = vault / "_raw" / "_archived" / "foo.md"
|
||||
clip.parent.mkdir(parents=True)
|
||||
clip.write_text("x\n", encoding="utf-8")
|
||||
assert archive_wikilink_relpath(vault, "_raw/_archived/foo") == (
|
||||
"_raw/_archived/foo.md"
|
||||
)
|
||||
assert archive_wikilink_relpath(vault, "_raw/_archived/foo.md") == (
|
||||
"_raw/_archived/foo.md"
|
||||
)
|
||||
assert archive_wikilink_relpath(vault, "concepts/foo") is None
|
||||
missing = archive_wikilink_relpath(vault, "_raw/_archived/missing")
|
||||
assert missing == "_raw/_archived/missing.md"
|
||||
|
||||
|
||||
def test_prefer_live_staging_without_archive_returns_staging(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
staging = vault / "_raw" / "notes.md"
|
||||
staging.parent.mkdir(parents=True)
|
||||
staging.write_text("staging\n", encoding="utf-8")
|
||||
assert prefer_archive_write_path(vault, "_raw/notes.md") == "_raw/notes.md"
|
||||
|
||||
|
||||
def test_prefer_archive_write_path_rejects_traversal_and_absolute(
|
||||
tmp_path: Path,
|
||||
) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
vault.mkdir()
|
||||
outside = tmp_path / "outside.md"
|
||||
outside.write_text("secret\n", encoding="utf-8")
|
||||
assert prefer_archive_write_path(vault, "../outside.md") is None
|
||||
assert prefer_archive_write_path(vault, outside.as_posix()) is None
|
||||
|
||||
|
||||
def test_archive_wikilink_relpath_rejects_parent_traversal(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
vault.mkdir()
|
||||
assert archive_wikilink_relpath(vault, "_raw/_archived/../attention") is None
|
||||
@@ -0,0 +1,202 @@
|
||||
"""Staged-write promotion: inventory, promote, discard, and the conflict path."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from obsidian_wiki.staging import (
|
||||
StagingConflict,
|
||||
StagingError,
|
||||
discard,
|
||||
list_staged,
|
||||
log_decisions,
|
||||
promote,
|
||||
revision,
|
||||
)
|
||||
|
||||
|
||||
def write(vault, rel, body):
|
||||
path = vault / rel
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(body, encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def vault(tmp_path):
|
||||
write(tmp_path, "concepts/live.md", "---\ntitle: Live\n---\n\noriginal\n")
|
||||
write(tmp_path, "_staging/concepts/live.md", "---\ntitle: Live\n---\n\nrevised\n")
|
||||
write(tmp_path, "_staging/concepts/fresh.md", "---\ntitle: Fresh\n---\n\nbrand new\n")
|
||||
write(tmp_path, "_staging/concepts/live.patch.md", "+ added\n- removed\n")
|
||||
return tmp_path
|
||||
|
||||
|
||||
def test_listing_classifies_new_update_and_patch(vault):
|
||||
by_staged = {e.staged_path: e for e in list_staged(vault)}
|
||||
assert by_staged["_staging/concepts/fresh.md"].kind == "new"
|
||||
assert by_staged["_staging/concepts/live.md"].kind == "update"
|
||||
|
||||
patch = by_staged["_staging/concepts/live.patch.md"]
|
||||
assert patch.kind == "patch"
|
||||
# A patch targets the page it is named after, with the .patch infix dropped.
|
||||
assert patch.live_path == "concepts/live.md"
|
||||
|
||||
|
||||
def test_listing_pins_both_revisions(vault):
|
||||
entries = {e.staged_path: e for e in list_staged(vault)}
|
||||
update = entries["_staging/concepts/live.md"]
|
||||
assert update.staged_revision == revision(vault / "_staging/concepts/live.md")
|
||||
assert update.live_revision == revision(vault / "concepts/live.md")
|
||||
# There is no live page for a new one, and that absence is itself the revision.
|
||||
assert entries["_staging/concepts/fresh.md"].live_revision is None
|
||||
|
||||
|
||||
def test_empty_staging_lists_nothing(tmp_path):
|
||||
assert list_staged(tmp_path) == []
|
||||
|
||||
|
||||
def test_promoting_a_new_page_moves_it_live(vault):
|
||||
result = promote(vault, "concepts/fresh.md")
|
||||
assert result["kind"] == "new"
|
||||
assert (vault / "concepts/fresh.md").read_text(encoding="utf-8") == (
|
||||
"---\ntitle: Fresh\n---\n\nbrand new\n"
|
||||
)
|
||||
assert not (vault / "_staging/concepts/fresh.md").exists()
|
||||
|
||||
|
||||
def test_promoting_an_update_replaces_the_live_page(vault):
|
||||
assert promote(vault, "concepts/live.md")["kind"] == "update"
|
||||
assert "revised" in (vault / "concepts/live.md").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_promotion_preserves_arbitrary_frontmatter_byte_for_byte(vault):
|
||||
staged = write(
|
||||
vault,
|
||||
"_staging/references/odd.md",
|
||||
"---\ntitle: Odd\nhouse_style: {a: 1}\ntags: [x, y]\n---\n\nbody with spacing\n",
|
||||
)
|
||||
original = staged.read_bytes()
|
||||
promote(vault, "references/odd.md")
|
||||
# A rename, not a rewrite: nothing normalises frontmatter, prose or links.
|
||||
assert (vault / "references/odd.md").read_bytes() == original
|
||||
|
||||
|
||||
def test_promotion_accepts_either_path_spelling(vault):
|
||||
promote(vault, "_staging/concepts/fresh.md")
|
||||
assert (vault / "concepts/fresh.md").is_file()
|
||||
|
||||
|
||||
def test_promotion_refuses_a_patch(vault):
|
||||
with pytest.raises(StagingError, match="is a patch"):
|
||||
promote(vault, "concepts/live.patch.md")
|
||||
assert (vault / "_staging/concepts/live.patch.md").is_file()
|
||||
|
||||
|
||||
def test_promotion_refuses_when_the_live_page_moved_on(vault):
|
||||
reviewed = revision(vault / "concepts/live.md")
|
||||
# An agent rewrites the live page while a human is still reading the diff.
|
||||
write(vault, "concepts/live.md", "---\ntitle: Live\n---\n\nagent wrote this\n")
|
||||
|
||||
with pytest.raises(StagingConflict, match="changed since it was reviewed"):
|
||||
promote(vault, "concepts/live.md", expected_live_revision=reviewed)
|
||||
|
||||
assert "agent wrote this" in (vault / "concepts/live.md").read_text(encoding="utf-8")
|
||||
assert (vault / "_staging/concepts/live.md").is_file()
|
||||
|
||||
|
||||
def test_promotion_refuses_when_the_staged_file_moved_on(vault):
|
||||
reviewed = revision(vault / "_staging/concepts/fresh.md")
|
||||
write(vault, "_staging/concepts/fresh.md", "---\ntitle: Fresh\n---\n\nrestaged\n")
|
||||
|
||||
with pytest.raises(StagingConflict):
|
||||
promote(vault, "concepts/fresh.md", expected_staged_revision=reviewed)
|
||||
assert not (vault / "concepts/fresh.md").exists()
|
||||
|
||||
|
||||
def test_pin_live_absent_catches_a_page_that_appeared(vault):
|
||||
"""A bare expected_live_revision=None cannot say "there was no page"."""
|
||||
write(vault, "concepts/fresh.md", "---\ntitle: Fresh\n---\n\nsomeone got there first\n")
|
||||
|
||||
with pytest.raises(StagingConflict, match="was created since"):
|
||||
promote(vault, "concepts/fresh.md", pin_live_absent=True)
|
||||
assert "got there first" in (vault / "concepts/fresh.md").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_matching_revisions_promote_normally(vault):
|
||||
# Keyed by staged path: a page and its patch share one live_path.
|
||||
entry = {e.staged_path: e for e in list_staged(vault)}["_staging/concepts/live.md"]
|
||||
promote(
|
||||
vault,
|
||||
entry.staged_path,
|
||||
expected_staged_revision=entry.staged_revision,
|
||||
expected_live_revision=entry.live_revision,
|
||||
)
|
||||
assert "revised" in (vault / "concepts/live.md").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_discard_moves_the_file_to_raw(vault):
|
||||
result = discard(vault, "concepts/fresh.md")
|
||||
assert result["raw_path"] == "_raw/rejected-concepts-fresh.md"
|
||||
assert "brand new" in (vault / result["raw_path"]).read_text(encoding="utf-8")
|
||||
assert not (vault / "_staging/concepts/fresh.md").exists()
|
||||
|
||||
|
||||
def test_discarding_a_patch_is_labelled_as_one(vault):
|
||||
assert discard(vault, "concepts/live.patch.md")["raw_path"] == (
|
||||
"_raw/rejected-patch-concepts-live.md"
|
||||
)
|
||||
|
||||
|
||||
def test_a_second_rejection_does_not_clobber_the_first(vault):
|
||||
discard(vault, "concepts/fresh.md")
|
||||
write(vault, "_staging/concepts/fresh.md", "---\ntitle: Fresh\n---\n\nsecond attempt\n")
|
||||
|
||||
assert discard(vault, "concepts/fresh.md")["raw_path"] == "_raw/rejected-concepts-fresh-2.md"
|
||||
assert "brand new" in (vault / "_raw/rejected-concepts-fresh.md").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_emptied_staging_directories_are_pruned(vault):
|
||||
for rel in ("concepts/live.md", "concepts/fresh.md"):
|
||||
promote(vault, rel)
|
||||
discard(vault, "concepts/live.patch.md")
|
||||
assert not (vault / "_staging/concepts").exists()
|
||||
|
||||
|
||||
def test_a_traversing_path_is_refused_outright(vault):
|
||||
with pytest.raises(StagingError, match="escapes the vault"):
|
||||
promote(vault, "../../etc/passwd")
|
||||
|
||||
|
||||
@pytest.mark.parametrize("rel", ["/etc/passwd", "concepts/absent.md"])
|
||||
def test_odd_but_contained_paths_fail_as_missing(vault, rel):
|
||||
"""Anchoring under _staging/ and stripping a leading slash contains these."""
|
||||
with pytest.raises(StagingError, match="no staged file"):
|
||||
promote(vault, rel)
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"rel",
|
||||
["../concepts/live.md", "_staging/../concepts/live.md", "concepts/../../escape.md"],
|
||||
)
|
||||
def test_a_live_page_cannot_be_promoted_as_if_it_were_staged(vault, rel):
|
||||
"""Inside the vault is not enough — `..` reaches live pages from _staging/."""
|
||||
with pytest.raises(StagingError, match="not under _staging/"):
|
||||
promote(vault, rel)
|
||||
with pytest.raises(StagingError, match="not under _staging/"):
|
||||
discard(vault, rel)
|
||||
assert (vault / "concepts/live.md").read_text(encoding="utf-8").endswith("original\n")
|
||||
|
||||
|
||||
def test_missing_staged_file_is_reported(vault):
|
||||
with pytest.raises(StagingError, match="no staged file"):
|
||||
promote(vault, "concepts/absent.md")
|
||||
|
||||
|
||||
def test_log_records_one_line_per_batch(vault):
|
||||
log_decisions(vault, [promote(vault, "concepts/fresh.md"), discard(vault, "concepts/live.md")])
|
||||
assert "STAGE_COMMIT accepted=1 rejected=1" in (vault / "log.md").read_text(encoding="utf-8")
|
||||
|
||||
|
||||
def test_log_is_silent_when_nothing_happened(vault):
|
||||
log_decisions(vault, [])
|
||||
assert not (vault / "log.md").exists()
|
||||
@@ -0,0 +1,172 @@
|
||||
"""Tests for bi-temporal page validity."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from datetime import date
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from obsidian_wiki.lint import lint_vault
|
||||
from obsidian_wiki.temporal import (
|
||||
is_current,
|
||||
parse_date,
|
||||
superseded_target,
|
||||
validity_window,
|
||||
)
|
||||
|
||||
|
||||
def _page(vault: Path, relpath: str, *, extra: str = "", summary: str = "A page.") -> Path:
|
||||
path = vault / relpath
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
lines = [
|
||||
"---",
|
||||
f"title: {path.stem}",
|
||||
"category: references",
|
||||
"tags: [test]",
|
||||
"sources: [manual]",
|
||||
"created: 2026-01-01",
|
||||
"updated: 2026-01-01",
|
||||
f"summary: {summary}",
|
||||
]
|
||||
if extra:
|
||||
lines.append(extra)
|
||||
lines += ["---", f"# {path.stem}"]
|
||||
path.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
# --- parsing ---------------------------------------------------------------
|
||||
|
||||
def test_parse_date_accepts_dates_timestamps_and_empty() -> None:
|
||||
assert parse_date("2026-04-01") == date(2026, 4, 1)
|
||||
assert parse_date("2026-04-01T09:30:00+00:00") == date(2026, 4, 1)
|
||||
assert parse_date("2026-04-01T09:30:00Z") == date(2026, 4, 1)
|
||||
assert parse_date(' "2026-04-01" ') == date(2026, 4, 1)
|
||||
assert parse_date("") is None
|
||||
assert parse_date(None) is None
|
||||
|
||||
|
||||
def test_parse_date_rejects_nonsense() -> None:
|
||||
with pytest.raises(ValueError):
|
||||
parse_date("last tuesday")
|
||||
|
||||
|
||||
def test_validity_window_rejects_inverted_window() -> None:
|
||||
with pytest.raises(ValueError, match="precedes valid_from"):
|
||||
validity_window({"valid_from": "2026-05-01", "valid_until": "2026-04-01"})
|
||||
|
||||
|
||||
def test_superseded_target_unwraps_wikilinks_aliases_and_paths() -> None:
|
||||
assert superseded_target("[[gateway-envoy]]") == "gateway-envoy"
|
||||
assert superseded_target('"[[references/gateway-envoy.md|Envoy]]"') == "gateway-envoy"
|
||||
assert superseded_target("[[gateway-envoy#Routes]]") == "gateway-envoy"
|
||||
assert superseded_target("") == ""
|
||||
|
||||
|
||||
# --- currency --------------------------------------------------------------
|
||||
|
||||
def test_page_without_temporal_fields_is_always_current() -> None:
|
||||
assert is_current({}, date(1999, 1, 1)) is True
|
||||
|
||||
|
||||
def test_valid_until_is_inclusive_and_valid_from_is_a_floor() -> None:
|
||||
window = {"valid_from": "2026-04-01", "valid_until": "2026-06-30"}
|
||||
assert is_current(window, date(2026, 3, 31)) is False # not yet true
|
||||
assert is_current(window, date(2026, 4, 1)) is True
|
||||
assert is_current(window, date(2026, 6, 30)) is True # last day counts
|
||||
assert is_current(window, date(2026, 7, 1)) is False # no longer true
|
||||
|
||||
|
||||
def test_malformed_dates_count_as_current_so_retrieval_never_silently_drops() -> None:
|
||||
assert is_current({"valid_until": "whenever"}, date(2026, 6, 1)) is True
|
||||
|
||||
|
||||
# --- retrieval -------------------------------------------------------------
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
# --- lint ------------------------------------------------------------------
|
||||
|
||||
def test_lint_fails_on_malformed_temporal_metadata(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "references/inverted.md",
|
||||
extra="valid_from: 2026-05-01\nvalid_until: 2026-04-01")
|
||||
_page(vault, "references/garbage.md", extra="valid_until: soon")
|
||||
|
||||
report = lint_vault(vault, require_trust_ledger=False)
|
||||
|
||||
issues = {item["page"] for item in report["findings"]["temporal_errors"]}
|
||||
assert issues == {"references/inverted.md", "references/garbage.md"}
|
||||
assert report["status"] == "fail"
|
||||
|
||||
|
||||
def test_lint_reports_dangling_superseded_by(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "references/old.md", extra="superseded_by: \"[[does-not-exist]]\"")
|
||||
_page(vault, "references/self.md", extra="superseded_by: \"[[self]]\"")
|
||||
|
||||
report = lint_vault(vault, require_trust_ledger=False)
|
||||
|
||||
assert report["findings"]["temporal_errors"] == []
|
||||
assert report["findings"]["superseded_dangling"] == [
|
||||
{"page": "references/old.md", "target": "does-not-exist", "issue": "missing_target"},
|
||||
{"page": "references/self.md", "target": "self", "issue": "self_reference"},
|
||||
]
|
||||
# A bracketed value is a wikilink like any other, so the pre-existing
|
||||
# broken_links check fires too and the vault fails — same as a typed
|
||||
# `relationships:` target pointing at nothing.
|
||||
assert report["findings"]["broken_links"] == [
|
||||
{"page": "references/old.md", "target": "does-not-exist"}
|
||||
]
|
||||
assert report["status"] == "fail"
|
||||
|
||||
|
||||
def test_lint_warns_on_a_dangling_unbracketed_superseded_by(tmp_path: Path) -> None:
|
||||
"""Written as a plain name it isn't a wikilink, so only the specific
|
||||
finding fires and the vault warns rather than fails."""
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "references/old.md", extra="superseded_by: does-not-exist")
|
||||
|
||||
report = lint_vault(vault, require_trust_ledger=False)
|
||||
|
||||
assert report["findings"]["broken_links"] == []
|
||||
assert report["findings"]["superseded_dangling"] == [
|
||||
{"page": "references/old.md", "target": "does-not-exist", "issue": "missing_target"}
|
||||
]
|
||||
assert report["status"] == "warn"
|
||||
|
||||
|
||||
def test_lint_passes_a_valid_supersession_chain(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "references/old.md",
|
||||
extra="valid_until: 2026-03-31\nsuperseded_by: \"[[new]]\"")
|
||||
_page(vault, "references/new.md", extra="valid_from: 2026-04-01")
|
||||
|
||||
report = lint_vault(vault, require_trust_ledger=False)
|
||||
|
||||
assert report["findings"]["temporal_errors"] == []
|
||||
assert report["findings"]["superseded_dangling"] == []
|
||||
|
||||
|
||||
# --- CLI -------------------------------------------------------------------
|
||||
|
||||
def _run(*args: str) -> subprocess.CompletedProcess[str]:
|
||||
env = os.environ.copy()
|
||||
env["PYTHONPATH"] = str(Path(__file__).parents[1])
|
||||
return subprocess.run(
|
||||
[sys.executable, "-m", "obsidian_wiki.cli", *args],
|
||||
capture_output=True, check=False, text=True, env=env,
|
||||
)
|
||||
@@ -0,0 +1,883 @@
|
||||
"""Regression tests for manually reviewed confidence and relationship linting."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
from obsidian_wiki.lint import lint_vault
|
||||
from obsidian_wiki.trust import (
|
||||
build_trust_ledger,
|
||||
check_trust_ledger,
|
||||
iter_trust_pages,
|
||||
page_fingerprint,
|
||||
update_trust_ledger,
|
||||
write_trust_ledger,
|
||||
)
|
||||
|
||||
|
||||
def _page(
|
||||
vault: Path,
|
||||
relpath: str,
|
||||
*,
|
||||
confidence: float = 0.80,
|
||||
updated: str = "2026-07-12T17:38:39+07:00",
|
||||
body: str = "Reviewed material claim.",
|
||||
relationships: list[tuple[str, str]] | None = None,
|
||||
) -> Path:
|
||||
path = vault / relpath
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
lines = [
|
||||
"---",
|
||||
f"title: {path.stem}",
|
||||
f"category: {path.parts[-2] if len(path.parts) > 1 else 'concepts'}",
|
||||
"tags: [test]",
|
||||
"sources:",
|
||||
" - github.com/example/repo@0123456789abcdef0123456789abcdef01234567",
|
||||
"created: 2026-07-01",
|
||||
f"updated: {updated}",
|
||||
f"base_confidence: {confidence:.2f}",
|
||||
"lifecycle: reviewed",
|
||||
]
|
||||
if relationships:
|
||||
lines.append("relationships:")
|
||||
for relation_type, target in relationships:
|
||||
lines.extend([f" - type: {relation_type}", f' target: "[[{target}]]"'])
|
||||
lines.extend(["---", f"# {path.stem}", "", body, ""])
|
||||
path.write_text("\n".join(lines), encoding="utf-8")
|
||||
return path
|
||||
|
||||
|
||||
def _write_ledger(vault: Path) -> Path:
|
||||
ledger = build_trust_ledger(vault, reviewed_at="2026-07-12T17:38:39+07:00")
|
||||
path = vault / "_meta" / "trust-ledger.json"
|
||||
write_trust_ledger(path, ledger)
|
||||
return path
|
||||
|
||||
|
||||
def _run_cli(home: Path, *args: str) -> subprocess.CompletedProcess[str]:
|
||||
env = os.environ.copy()
|
||||
env["HOME"] = str(home)
|
||||
return subprocess.run(
|
||||
[sys.executable, "-m", "obsidian_wiki.cli", *args],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env=env,
|
||||
)
|
||||
|
||||
|
||||
def test_reviewed_ledger_is_authoritative_instead_of_reclassifying_sources(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md", confidence=0.53)
|
||||
_page(vault, "skills/beta.md", confidence=0.80)
|
||||
ledger_path = _write_ledger(vault)
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "pass"
|
||||
assert report["counts"] == {
|
||||
"reviewed": 2,
|
||||
"not_applicable": 0,
|
||||
"stale": 0,
|
||||
"unreviewed": 0,
|
||||
"score_mismatches": 0,
|
||||
"missing_pages": 0,
|
||||
"errors": 0,
|
||||
}
|
||||
assert {item["page"] for item in report["reviewed"]} == {
|
||||
"concepts/alpha.md",
|
||||
"skills/beta.md",
|
||||
}
|
||||
|
||||
|
||||
def test_claim_change_invalidates_review_but_updated_timestamp_does_not(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md", confidence=0.53)
|
||||
ledger_path = _write_ledger(vault)
|
||||
|
||||
page.write_text(page.read_text().replace("updated: 2026-07-12T17:38:39+07:00", "updated: 2026-07-13T09:00:00+07:00"))
|
||||
assert check_trust_ledger(vault, ledger_path)["counts"]["reviewed"] == 1
|
||||
|
||||
page.write_text(page.read_text().replace("Reviewed material claim.", "Changed material claim."))
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
assert report["status"] == "warn"
|
||||
assert report["stale"] == [{"page": "concepts/alpha.md", "reason": "material_fingerprint_changed"}]
|
||||
|
||||
|
||||
def test_material_change_marks_review_stale(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md", confidence=0.53)
|
||||
ledger_path = _write_ledger(vault)
|
||||
page.write_text(page.read_text().replace("Reviewed material claim.", "Changed material claim."))
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "warn"
|
||||
assert report["stale"] == [
|
||||
{"page": "concepts/alpha.md", "reason": "material_fingerprint_changed"}
|
||||
]
|
||||
|
||||
|
||||
def test_duplicate_confidence_field_fails_instead_of_bypassing_fingerprint(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md", confidence=0.53)
|
||||
ledger_path = _write_ledger(vault)
|
||||
page.write_text(page.read_text().replace("lifecycle: reviewed", "base_confidence: 0.99\nlifecycle: reviewed"))
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["errors"] == [
|
||||
{"page": "concepts/alpha.md", "issue": "duplicate top-level field: base_confidence"}
|
||||
]
|
||||
|
||||
|
||||
def test_volatile_field_with_nested_content_fails_closed(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
page.write_text(page.read_text().replace("updated: 2026-07-12T17:38:39+07:00", "updated: 2026-07-12T17:38:39+07:00\n hidden: material"))
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["errors"] == [
|
||||
{"page": "concepts/alpha.md", "issue": "updated must be a scalar"}
|
||||
]
|
||||
|
||||
|
||||
def test_invalid_confidence_is_checked_before_stale_short_circuit(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
changed = page.read_text().replace("Reviewed material claim.", "Changed material claim.")
|
||||
page.write_text(changed.replace("base_confidence: 0.80", "base_confidence: nan"))
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["stale"] == []
|
||||
assert report["errors"] == [
|
||||
{"page": "concepts/alpha.md", "issue": "base_confidence is not finite"}
|
||||
]
|
||||
|
||||
|
||||
def test_invalid_lifecycle_fails_closed(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
page.write_text(page.read_text().replace("lifecycle: reviewed", "lifecycle: totally-invalid"))
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["errors"] == [
|
||||
{"page": "concepts/alpha.md", "issue": "invalid lifecycle: totally-invalid"}
|
||||
]
|
||||
|
||||
|
||||
def test_owner_trust_schema_accepts_lifecycle_and_optional_fields(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
page.write_text(page.read_text().replace("base_confidence: 0.80\n", "").replace("lifecycle: reviewed", "lifecycle: active"))
|
||||
|
||||
report = check_trust_ledger(
|
||||
vault,
|
||||
ledger_path,
|
||||
allowed_lifecycles={"active", "confirmed", "stub"},
|
||||
required_trust_keys=("updated",),
|
||||
schema_source="wiki/AGENTS.md",
|
||||
)
|
||||
|
||||
assert report["status"] == "warn"
|
||||
assert report["errors"] == []
|
||||
assert report["not_applicable"] == [
|
||||
{
|
||||
"page": "concepts/alpha.md",
|
||||
"reason": "base_confidence_absent_by_owner_schema",
|
||||
}
|
||||
]
|
||||
assert report["stale"] == [
|
||||
{
|
||||
"page": "concepts/alpha.md",
|
||||
"reason": "confidence_not_applicable_but_ledger_entry_exists",
|
||||
}
|
||||
]
|
||||
assert report["schema"] == {
|
||||
"source": "wiki/AGENTS.md",
|
||||
"allowed_lifecycles": ["active", "confirmed", "stub"],
|
||||
"required_trust_fields": ["updated"],
|
||||
}
|
||||
|
||||
|
||||
def test_owner_schema_build_excludes_no_confidence_page_as_not_applicable(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
page.write_text(page.read_text().replace("base_confidence: 0.80\n", ""))
|
||||
|
||||
ledger = build_trust_ledger(
|
||||
vault,
|
||||
reviewed_at="2026-08-05T12:00:00+09:00",
|
||||
required_trust_keys=("updated",),
|
||||
)
|
||||
|
||||
assert ledger["pages"] == {}
|
||||
assert ledger["not_applicable"] == ["concepts/alpha.md"]
|
||||
assert page_fingerprint(page, required_trust_keys=("updated",)).startswith("sha256:")
|
||||
|
||||
|
||||
def test_owner_schema_update_removes_stale_no_confidence_entry(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
page.write_text(page.read_text().replace("base_confidence: 0.80\n", ""))
|
||||
|
||||
before = check_trust_ledger(
|
||||
vault,
|
||||
ledger_path,
|
||||
required_trust_keys=("updated",),
|
||||
)
|
||||
assert before["status"] == "warn"
|
||||
assert before["stale"] == [
|
||||
{
|
||||
"page": "concepts/alpha.md",
|
||||
"reason": "confidence_not_applicable_but_ledger_entry_exists",
|
||||
}
|
||||
]
|
||||
|
||||
ledger = update_trust_ledger(
|
||||
vault,
|
||||
ledger_path,
|
||||
reviewed_at="2026-08-05T13:00:00+09:00",
|
||||
page_paths=["concepts/alpha.md"],
|
||||
required_trust_keys=("updated",),
|
||||
)
|
||||
assert ledger["pages"] == {}
|
||||
assert ledger["not_applicable"] == ["concepts/alpha.md"]
|
||||
assert ledger["removed_not_applicable"] == ["concepts/alpha.md"]
|
||||
write_trust_ledger(ledger_path, ledger, vault=vault)
|
||||
|
||||
after = check_trust_ledger(
|
||||
vault,
|
||||
ledger_path,
|
||||
required_trust_keys=("updated",),
|
||||
)
|
||||
assert after["status"] == "pass"
|
||||
assert after["counts"]["not_applicable"] == 1
|
||||
assert after["stale"] == []
|
||||
|
||||
|
||||
def test_owner_trust_schema_rejects_unknown_lifecycle_typo(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
page.write_text(page.read_text().replace("lifecycle: reviewed", "lifecycle: activ"))
|
||||
|
||||
report = check_trust_ledger(
|
||||
vault,
|
||||
ledger_path,
|
||||
allowed_lifecycles={"active", "confirmed", "stub"},
|
||||
required_trust_keys=("updated",),
|
||||
schema_source="wiki/AGENTS.md",
|
||||
)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["errors"] == [
|
||||
{"page": "concepts/alpha.md", "issue": "invalid lifecycle: activ"}
|
||||
]
|
||||
|
||||
|
||||
def test_owner_lifecycle_propagates_through_record_update_and_check(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
page.write_text(page.read_text().replace("lifecycle: reviewed", "lifecycle: active"))
|
||||
schema = {
|
||||
"allowed_lifecycles": {"active", "confirmed", "stub"},
|
||||
"required_trust_keys": ("base_confidence", "lifecycle", "updated"),
|
||||
}
|
||||
|
||||
ledger = build_trust_ledger(
|
||||
vault,
|
||||
reviewed_at="2026-08-05T12:00:00+09:00",
|
||||
**schema,
|
||||
)
|
||||
ledger_path = vault / "_meta" / "trust-ledger.json"
|
||||
write_trust_ledger(ledger_path, ledger, vault=vault)
|
||||
assert check_trust_ledger(vault, ledger_path, schema_source="wiki/AGENTS.md", **schema)["status"] == "pass"
|
||||
|
||||
page.write_text(page.read_text().replace("Reviewed material claim.", "Reviewed updated claim."))
|
||||
updated = update_trust_ledger(
|
||||
vault,
|
||||
ledger_path,
|
||||
reviewed_at="2026-08-05T13:00:00+09:00",
|
||||
page_paths=["concepts/alpha.md"],
|
||||
**schema,
|
||||
)
|
||||
write_trust_ledger(ledger_path, updated, vault=vault)
|
||||
assert check_trust_ledger(vault, ledger_path, schema_source="wiki/AGENTS.md", **schema)["status"] == "pass"
|
||||
|
||||
|
||||
|
||||
|
||||
def test_mismatched_scalar_quotes_fail_closed(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
page.write_text(page.read_text().replace("lifecycle: reviewed", 'lifecycle: "reviewed\''))
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["errors"] == [
|
||||
{"page": "concepts/alpha.md", "issue": 'invalid lifecycle: "reviewed\''}
|
||||
]
|
||||
|
||||
|
||||
def test_invalid_updated_timestamp_fails_before_fingerprint_match(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
page.write_text(
|
||||
page.read_text().replace(
|
||||
"updated: 2026-07-12T17:38:39+07:00",
|
||||
"updated: definitely-not-a-timestamp",
|
||||
)
|
||||
)
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["errors"] == [
|
||||
{"page": "concepts/alpha.md", "issue": "updated is not an ISO-8601 date or timestamp"}
|
||||
]
|
||||
|
||||
|
||||
def test_inline_comments_on_scalar_trust_fields_remain_valid(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
page.write_text(
|
||||
page.read_text()
|
||||
.replace("base_confidence: 0.80", "base_confidence: 0.80 # reviewed score")
|
||||
.replace("lifecycle: reviewed", "lifecycle: reviewed # human-reviewed")
|
||||
)
|
||||
|
||||
ledger_path = _write_ledger(vault)
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "pass"
|
||||
assert report["counts"]["reviewed"] == 1
|
||||
|
||||
|
||||
def test_score_only_change_is_a_failing_mismatch_not_stale(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md", confidence=0.53)
|
||||
ledger_path = _write_ledger(vault)
|
||||
page.write_text(page.read_text().replace("base_confidence: 0.53", "base_confidence: 0.88"))
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["stale"] == []
|
||||
assert report["score_mismatches"] == [
|
||||
{"page": "concepts/alpha.md", "stored": 0.88, "reviewed": 0.53}
|
||||
]
|
||||
|
||||
|
||||
def test_new_page_requires_manual_review_instead_of_formula_guess(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
_page(vault, "concepts/new.md", confidence=0.95)
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "warn"
|
||||
assert report["unreviewed"] == [{"page": "concepts/new.md", "reason": "not_in_manual_ledger"}]
|
||||
|
||||
|
||||
def test_standalone_trust_check_fails_for_page_missing_trust_fields(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
page.write_text(
|
||||
"\n".join(
|
||||
line
|
||||
for line in page.read_text().splitlines()
|
||||
if not line.startswith(("base_confidence:", "lifecycle:"))
|
||||
)
|
||||
+ "\n"
|
||||
)
|
||||
ledger_path = vault / "_meta" / "trust-ledger.json"
|
||||
write_trust_ledger(
|
||||
ledger_path,
|
||||
{
|
||||
"schema_version": 1,
|
||||
"method": "manual-lineage-and-claim-coverage-v1",
|
||||
"reviewed_at": "2026-07-12T17:38:39+07:00",
|
||||
"pages": {},
|
||||
},
|
||||
vault=vault,
|
||||
)
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["errors"] == [
|
||||
{"page": "concepts/alpha.md", "issue": "missing base_confidence"}
|
||||
]
|
||||
|
||||
|
||||
def test_lint_vault_requires_trust_ledger_by_default(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md")
|
||||
|
||||
report = lint_vault(vault)
|
||||
|
||||
# Staged migration (#28, #146): a missing ledger only warns until the
|
||||
# vault owner opts into strict_trust.
|
||||
assert report["status"] == "warn"
|
||||
assert report["findings"]["confidence_ledger_errors"] == [
|
||||
{"issue": "ledger_missing", "path": str(vault / "_meta" / "trust-ledger.json")}
|
||||
]
|
||||
|
||||
strict_report = lint_vault(vault, strict_trust=True)
|
||||
assert strict_report["status"] == "fail"
|
||||
|
||||
|
||||
def test_non_object_ledger_fails_cleanly(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md")
|
||||
ledger_path = vault / "_meta" / "trust-ledger.json"
|
||||
ledger_path.parent.mkdir(parents=True)
|
||||
ledger_path.write_text("[]\n")
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["errors"] == [{"issue": "ledger_must_be_an_object"}]
|
||||
|
||||
|
||||
def test_duplicate_json_keys_are_rejected(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
text = ledger_path.read_text()
|
||||
ledger_path.write_text(text.replace('"schema_version": 1', '"schema_version": 1, "schema_version": 1'))
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["errors"][0]["issue"] == "ledger_unreadable"
|
||||
assert "duplicate JSON key" in report["errors"][0]["detail"]
|
||||
|
||||
|
||||
def test_boolean_schema_version_fails_closed(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
ledger = json.loads(ledger_path.read_text())
|
||||
ledger["schema_version"] = True
|
||||
ledger_path.write_text(json.dumps(ledger))
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["errors"][0]["issue"] == "unsupported_schema_version"
|
||||
|
||||
|
||||
def test_invalid_utf8_ledger_returns_structured_failure(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md")
|
||||
ledger_path = vault / "_meta" / "trust-ledger.json"
|
||||
ledger_path.parent.mkdir(parents=True)
|
||||
ledger_path.write_bytes(b"\xff\xfe")
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["errors"][0]["issue"] == "ledger_unreadable"
|
||||
|
||||
|
||||
|
||||
|
||||
def test_invalid_ledger_review_timestamp_fails_closed(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
ledger = json.loads(ledger_path.read_text())
|
||||
ledger["reviewed_at"] = "yesterday"
|
||||
ledger_path.write_text(json.dumps(ledger))
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["errors"] == [{"issue": "invalid_reviewed_at"}]
|
||||
|
||||
|
||||
def test_trust_writer_rejects_symlinked_meta_directory(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
outside = tmp_path / "outside"
|
||||
vault.mkdir()
|
||||
outside.mkdir()
|
||||
(vault / "_meta").symlink_to(outside, target_is_directory=True)
|
||||
ledger = {"schema_version": 1, "method": "manual-lineage-and-claim-coverage-v1", "pages": {}}
|
||||
|
||||
try:
|
||||
write_trust_ledger(vault / "_meta" / "trust-ledger.json", ledger, vault=vault)
|
||||
except RuntimeError as exc:
|
||||
assert "outside the resolved vault" in str(exc) or "symlink" in str(exc)
|
||||
else:
|
||||
raise AssertionError("symlinked _meta directory was accepted")
|
||||
|
||||
assert not (outside / "trust-ledger.json").exists()
|
||||
|
||||
|
||||
def test_trust_writer_never_follows_predictable_temp_symlink(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
meta = vault / "_meta"
|
||||
meta.mkdir(parents=True)
|
||||
outside = tmp_path / "outside.txt"
|
||||
outside.write_text("sentinel\n")
|
||||
(meta / "trust-ledger.json.tmp").symlink_to(outside)
|
||||
ledger = {"schema_version": 1, "method": "manual-lineage-and-claim-coverage-v1", "pages": {}}
|
||||
|
||||
write_trust_ledger(meta / "trust-ledger.json", ledger, vault=vault)
|
||||
|
||||
assert outside.read_text() == "sentinel\n"
|
||||
assert json.loads((meta / "trust-ledger.json").read_text())["schema_version"] == 1
|
||||
|
||||
|
||||
def test_trust_writer_rejects_symlinked_ledger_destination(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
meta = vault / "_meta"
|
||||
meta.mkdir(parents=True)
|
||||
outside = tmp_path / "outside.txt"
|
||||
outside.write_text("sentinel\n")
|
||||
destination = meta / "trust-ledger.json"
|
||||
destination.symlink_to(outside)
|
||||
ledger = {"schema_version": 1, "method": "manual-lineage-and-claim-coverage-v1", "pages": {}}
|
||||
|
||||
try:
|
||||
write_trust_ledger(destination, ledger, vault=vault)
|
||||
except RuntimeError as exc:
|
||||
assert "outside the resolved vault" in str(exc) or "symlink" in str(exc)
|
||||
else:
|
||||
raise AssertionError("symlinked ledger destination was accepted")
|
||||
|
||||
assert outside.read_text() == "sentinel\n"
|
||||
|
||||
|
||||
def test_out_of_range_reviewed_confidence_is_invalid_ledger_data(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
ledger = json.loads(ledger_path.read_text())
|
||||
ledger["pages"]["concepts/alpha.md"]["reviewed_confidence"] = 2.0
|
||||
ledger_path.write_text(json.dumps(ledger))
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["errors"] == [{"page": "concepts/alpha.md", "issue": "invalid_ledger_entry"}]
|
||||
|
||||
|
||||
def test_boolean_reviewed_confidence_is_invalid_ledger_data(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
ledger = json.loads(ledger_path.read_text())
|
||||
ledger["pages"]["concepts/alpha.md"]["reviewed_confidence"] = True
|
||||
ledger_path.write_text(json.dumps(ledger))
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["errors"] == [{"page": "concepts/alpha.md", "issue": "invalid_ledger_entry"}]
|
||||
|
||||
|
||||
def test_non_standard_json_number_is_unreadable_not_approved(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
ledger = json.loads(ledger_path.read_text())
|
||||
ledger["pages"]["concepts/alpha.md"]["reviewed_confidence"] = float("nan")
|
||||
ledger_path.write_text(json.dumps(ledger))
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["errors"][0]["issue"] == "ledger_unreadable"
|
||||
|
||||
|
||||
def test_malformed_fingerprint_and_entry_timestamp_fail_closed(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
ledger = json.loads(ledger_path.read_text())
|
||||
entry = ledger["pages"]["concepts/alpha.md"]
|
||||
entry["material_fingerprint"] = "garbage"
|
||||
entry["reviewed_at"] = "yesterday"
|
||||
ledger_path.write_text(json.dumps(ledger))
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["errors"] == [{"page": "concepts/alpha.md", "issue": "invalid_ledger_entry"}]
|
||||
|
||||
|
||||
def test_non_object_page_entry_is_invalid_not_unreviewed(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md")
|
||||
ledger_path = _write_ledger(vault)
|
||||
ledger = json.loads(ledger_path.read_text())
|
||||
ledger["pages"]["concepts/alpha.md"] = []
|
||||
ledger_path.write_text(json.dumps(ledger))
|
||||
|
||||
report = check_trust_ledger(vault, ledger_path)
|
||||
|
||||
assert report["status"] == "fail"
|
||||
assert report["unreviewed"] == []
|
||||
assert report["errors"] == [{"page": "concepts/alpha.md", "issue": "invalid_ledger_entry"}]
|
||||
|
||||
|
||||
def test_lint_vault_surfaces_manual_trust_status_without_formula_drift(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md", confidence=0.53, relationships=[("related_to", "skills/beta")])
|
||||
_page(vault, "skills/beta.md", confidence=0.80, relationships=[("related_to", "concepts/alpha")])
|
||||
_write_ledger(vault)
|
||||
|
||||
report = lint_vault(vault)
|
||||
|
||||
assert report["findings"]["confidence_review_stale"] == []
|
||||
assert report["findings"]["confidence_unreviewed"] == []
|
||||
assert report["findings"]["confidence_mismatches"] == []
|
||||
assert report["findings"]["typed_relationship_issues"] == []
|
||||
assert report["stats"]["trust"]["reviewed"] == 2
|
||||
|
||||
|
||||
def test_typed_relationship_resolver_uses_exact_vault_paths(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "projects/demo/alpha.md", relationships=[("implements", "projects/demo/beta")])
|
||||
_page(vault, "projects/demo/beta.md", relationships=[("related_to", "projects/demo/alpha")])
|
||||
_page(vault, "concepts/broken.md", relationships=[("related_to", "skills/missing")])
|
||||
|
||||
report = lint_vault(vault, require_trust_ledger=False)
|
||||
|
||||
assert report["findings"]["typed_relationship_issues"] == [
|
||||
{
|
||||
"page": "concepts/broken.md",
|
||||
"index": 0,
|
||||
"issue": "missing_target",
|
||||
"target": "skills/missing",
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def test_target_first_relationship_order_is_parsed_and_validated(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
_page(vault, "skills/beta.md")
|
||||
page.write_text(
|
||||
page.read_text().replace(
|
||||
"lifecycle: reviewed",
|
||||
'lifecycle: reviewed\nrelationships:\n - target: "[[skills/missing]]"\n type: related_to',
|
||||
)
|
||||
)
|
||||
|
||||
report = lint_vault(vault, require_trust_ledger=False)
|
||||
|
||||
assert report["findings"]["typed_relationship_issues"] == [
|
||||
{
|
||||
"page": "concepts/alpha.md",
|
||||
"index": 0,
|
||||
"issue": "missing_target",
|
||||
"target": "skills/missing",
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def test_dash_only_relationship_item_is_parsed_and_validated(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
page.write_text(
|
||||
page.read_text().replace(
|
||||
"lifecycle: reviewed",
|
||||
'lifecycle: reviewed\nrelationships:\n -\n target: "[[skills/missing]]"\n type: related_to',
|
||||
)
|
||||
)
|
||||
|
||||
report = lint_vault(vault, require_trust_ledger=False)
|
||||
|
||||
assert report["findings"]["typed_relationship_issues"] == [
|
||||
{
|
||||
"page": "concepts/alpha.md",
|
||||
"index": 0,
|
||||
"issue": "missing_target",
|
||||
"target": "skills/missing",
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def test_unknown_relationship_field_fails_closed(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
_page(vault, "skills/beta.md")
|
||||
page.write_text(
|
||||
page.read_text().replace(
|
||||
"lifecycle: reviewed",
|
||||
'lifecycle: reviewed\nrelationships:\n - target: "[[skills/beta]]"\n type: related_to\n weight: 5',
|
||||
)
|
||||
)
|
||||
|
||||
report = lint_vault(vault, require_trust_ledger=False)
|
||||
|
||||
assert report["findings"]["typed_relationship_issues"] == [
|
||||
{
|
||||
"page": "concepts/alpha.md",
|
||||
"index": 0,
|
||||
"issue": "malformed_relationship_entry",
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def test_basename_only_relationship_is_rejected_when_ambiguous(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md", relationships=[("related_to", "beta")])
|
||||
_page(vault, "skills/beta.md")
|
||||
_page(vault, "projects/demo/beta.md")
|
||||
|
||||
report = lint_vault(vault, require_trust_ledger=False)
|
||||
|
||||
assert report["findings"]["typed_relationship_issues"] == [
|
||||
{
|
||||
"page": "concepts/alpha.md",
|
||||
"index": 0,
|
||||
"issue": "ambiguous_target",
|
||||
"target": "beta",
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def test_qualified_relationship_is_ambiguous_when_normalized_paths_collide(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md", relationships=[("related_to", "skills/foo-bar")])
|
||||
_page(vault, "skills/Foo Bar.md")
|
||||
_page(vault, "skills/foo-bar.md")
|
||||
|
||||
report = lint_vault(vault, require_trust_ledger=False)
|
||||
|
||||
assert report["findings"]["typed_relationship_issues"] == [
|
||||
{
|
||||
"page": "concepts/alpha.md",
|
||||
"index": 0,
|
||||
"issue": "ambiguous_target",
|
||||
"target": "skills/foo-bar",
|
||||
}
|
||||
]
|
||||
|
||||
|
||||
def test_empty_inline_relationship_list_is_valid(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
page.write_text(page.read_text().replace("lifecycle: reviewed", "lifecycle: reviewed\nrelationships: []"))
|
||||
|
||||
report = lint_vault(vault, require_trust_ledger=False)
|
||||
|
||||
assert report["findings"]["typed_relationship_issues"] == []
|
||||
|
||||
|
||||
def test_lint_excludes_bootstrap_scaffolding_from_page_health(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md", relationships=[("related_to", "concepts/beta")])
|
||||
_page(vault, "concepts/beta.md", relationships=[("related_to", "concepts/alpha")])
|
||||
bootstrap = vault / "_bootstrap" / "README.md"
|
||||
bootstrap.parent.mkdir(parents=True)
|
||||
bootstrap.write_text("# Setup instructions\n")
|
||||
|
||||
report = lint_vault(vault)
|
||||
|
||||
serialized = json.dumps(report)
|
||||
assert "_bootstrap/README.md" not in serialized
|
||||
|
||||
|
||||
def test_iter_trust_pages_skips_tool_owned_directories(tmp_path: Path) -> None:
|
||||
"""A `.venv` inside a project-vault must not enter the ledger's page set —
|
||||
the ledger would otherwise record reviews for dependency READMEs, and each
|
||||
dependency upgrade would churn its entries."""
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md")
|
||||
junk = (
|
||||
".venv/lib/python3.12/site-packages/somepkg/README.md",
|
||||
".trash/draft.md",
|
||||
"node_modules/pkg/README.md",
|
||||
"venv/readme.md",
|
||||
"__pycache__/x.md",
|
||||
)
|
||||
for relative in junk:
|
||||
path = vault / relative
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text("# dependency readme\nno frontmatter\n", encoding="utf-8")
|
||||
|
||||
assert [page.relative_to(vault).as_posix() for page in iter_trust_pages(vault)] == [
|
||||
"concepts/alpha.md"
|
||||
]
|
||||
|
||||
|
||||
def test_iter_trust_pages_honors_okignore(tmp_path: Path) -> None:
|
||||
"""A vault-configured quarantine dir (`_excluded/`) stays out of the ledger."""
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md")
|
||||
(vault / ".okignore").write_text("_excluded/\n", encoding="utf-8")
|
||||
excluded = vault / "_excluded" / "pii.md"
|
||||
excluded.parent.mkdir(parents=True, exist_ok=True)
|
||||
excluded.write_text("plain text, no frontmatter\n", encoding="utf-8")
|
||||
|
||||
assert [page.relative_to(vault).as_posix() for page in iter_trust_pages(vault)] == [
|
||||
"concepts/alpha.md"
|
||||
]
|
||||
|
||||
|
||||
def test_lint_flags_content_page_missing_trust_schema(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
page = _page(vault, "concepts/alpha.md")
|
||||
page.write_text(
|
||||
"\n".join(
|
||||
line
|
||||
for line in page.read_text().splitlines()
|
||||
if not line.startswith(("base_confidence:", "lifecycle:"))
|
||||
)
|
||||
+ "\n"
|
||||
)
|
||||
|
||||
report = lint_vault(vault)
|
||||
|
||||
# Staged migration (#28, #146): legacy pages predating the trust schema
|
||||
# warn instead of failing lint until the owner opts into strict_trust.
|
||||
assert report["status"] == "warn"
|
||||
assert report["findings"]["missing_frontmatter"] == []
|
||||
assert report["findings"]["confidence_missing_fields"] == [
|
||||
{"page": "concepts/alpha.md", "missing": ["base_confidence", "lifecycle"]}
|
||||
]
|
||||
|
||||
strict_report = lint_vault(vault, strict_trust=True)
|
||||
assert strict_report["status"] == "fail"
|
||||
|
||||
|
||||
def test_required_trust_ledger_fails_closed_when_missing(tmp_path: Path) -> None:
|
||||
vault = tmp_path / "vault"
|
||||
_page(vault, "concepts/alpha.md")
|
||||
|
||||
report = lint_vault(vault, require_trust_ledger=True)
|
||||
|
||||
# require_trust_ledger controls whether the ledger check runs at all (and
|
||||
# is thus reported); strict_trust controls whether its findings fail lint.
|
||||
assert report["status"] == "warn"
|
||||
assert report["findings"]["confidence_ledger_errors"] == [
|
||||
{"issue": "ledger_missing", "path": str(vault / "_meta" / "trust-ledger.json")}
|
||||
]
|
||||
|
||||
strict_report = lint_vault(vault, require_trust_ledger=True, strict_trust=True)
|
||||
assert strict_report["status"] == "fail"
|
||||
@@ -0,0 +1,85 @@
|
||||
"""Local data contracts that the selected upstream port must preserve."""
|
||||
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from obsidian_wiki.cli import main
|
||||
from obsidian_wiki.graph_analysis import parse_vault_graph
|
||||
from obsidian_wiki.lint import lint_vault
|
||||
from obsidian_wiki.provenance import archive_wikilink_relpath, resolve_source_key
|
||||
from obsidian_wiki.vault import iter_md
|
||||
|
||||
|
||||
@pytest.mark.parametrize("shape", ["dict", "list"])
|
||||
def test_cache_cli_preserves_curated_metadata_and_page_associations(tmp_path, shape):
|
||||
source = tmp_path / "source.pdf"
|
||||
source.write_bytes(b"new revision")
|
||||
entry = {
|
||||
"path": "source.pdf", "pages_produced": ["concepts/a.md"],
|
||||
"archive_path": "_sources/archive/a.pdf", "content_hash": "old",
|
||||
"bjd_part_numbers": ["BJD-123"], "competitor_part_numbers": ["OTHER-456"],
|
||||
"series": ["WTB"], "pin_count": [6], "pitch": [2.0],
|
||||
"catalog_paths": ["_sources/catalog/a.md"], "visibility": "internal",
|
||||
}
|
||||
sources = {"source.pdf": entry} if shape == "dict" else [entry]
|
||||
manifest = tmp_path / ".manifest.json"
|
||||
manifest.write_text(json.dumps({"sources": sources, "local_revision": 7}))
|
||||
assert main(["cache-update", str(tmp_path), str(source)]) == 0
|
||||
updated = json.loads(manifest.read_text())
|
||||
actual = updated["sources"]["source.pdf"] if shape == "dict" else updated["sources"][0]
|
||||
assert updated["local_revision"] == 7
|
||||
for key, value in entry.items():
|
||||
if key != "content_hash":
|
||||
assert actual[key] == value
|
||||
assert actual["content_hash"] != "old"
|
||||
|
||||
|
||||
def test_local_sources_are_citable_but_not_knowledge_pages(tmp_path):
|
||||
refs = ["_sources/archive/doc.pdf", "_sources/catalog/products.md"]
|
||||
for rel in [*refs, "_sources/archive/raw.md", "_meta/taxonomy.md", "_inbox/draft.md", "_bot_exports/copy.md"]:
|
||||
p = tmp_path / rel
|
||||
p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_text("Raw source, not a knowledge page")
|
||||
page = tmp_path / "concepts" / "a.md"
|
||||
page.parent.mkdir()
|
||||
page.write_text(
|
||||
"---\ntitle: A\ncategory: concepts\ntags: [a]\nsources: []\n"
|
||||
"created: 2026-10-01\nupdated: 2026-10-01\nsummary: Example\n---\n"
|
||||
+ " ".join(f"[[{rel}]]" for rel in refs)
|
||||
)
|
||||
assert iter_md(tmp_path) == [page]
|
||||
graph, _ = parse_vault_graph(tmp_path)
|
||||
assert set(graph) == {"a"}
|
||||
assert graph["a"] == []
|
||||
findings = lint_vault(tmp_path, require_trust_ledger=False)["findings"]
|
||||
assert findings["broken_links"] == []
|
||||
assert resolve_source_key(tmp_path, refs[0]) == refs[0]
|
||||
assert archive_wikilink_relpath(tmp_path, refs[0] + "|PDF") == refs[0]
|
||||
assert archive_wikilink_relpath(tmp_path, "_sources/catalog/products") == refs[1]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("target", ["../_sources/archive/a.pdf", "/tmp/_sources/archive/a.pdf"])
|
||||
def test_local_source_resolution_refuses_paths_outside_vault(tmp_path, target):
|
||||
assert resolve_source_key(tmp_path, target) is None
|
||||
assert archive_wikilink_relpath(tmp_path, target) is None
|
||||
|
||||
|
||||
def test_staging_cli_conflict_keeps_reviewed_and_live_files(tmp_path, capsys):
|
||||
staged = tmp_path / "_staging/concepts/a.md"
|
||||
staged.parent.mkdir(parents=True)
|
||||
staged.write_text("Reviewed proposal")
|
||||
assert main(["staging", "list", "--vault", str(tmp_path), "--json"]) == 0
|
||||
inventory = json.loads(capsys.readouterr().out)
|
||||
# The runtime schema is asserted here so the skill can pin listed revisions.
|
||||
entry = inventory[0]
|
||||
live = tmp_path / "concepts/a.md"
|
||||
live.parent.mkdir()
|
||||
live.write_text("Concurrent live update")
|
||||
assert main([
|
||||
"staging", "promote", entry["staged_path"], "--vault", str(tmp_path),
|
||||
"--expect-staged", entry["staged_revision"], "--expect-new",
|
||||
]) == 9
|
||||
assert live.read_text() == "Concurrent live update"
|
||||
assert staged.read_text() == "Reviewed proposal"
|
||||
Reference in New Issue
Block a user