diff --git a/.skills/llm-wiki/SKILL.md b/.skills/llm-wiki/SKILL.md index f69e8ef..944d799 100644 --- a/.skills/llm-wiki/SKILL.md +++ b/.skills/llm-wiki/SKILL.md @@ -129,6 +129,8 @@ One-paragraph summary of what this project is. ## Special Files +For every write or maintenance operation, follow [Core Maintenance](references/CORE-MAINTENANCE.md): shared locks and atomic writes, hash-based source tracking, path-aware lint, and revision-checked staging. Existing index/hot files retain their layout until an explicit migration is authorized. This protocol overrides older manual shared-file rewrite instructions below and in other skills; the local source archive/registry protocol remains required. + Every wiki has these files at its root: ### `index.md` diff --git a/.skills/llm-wiki/references/CORE-MAINTENANCE.md b/.skills/llm-wiki/references/CORE-MAINTENANCE.md new file mode 100644 index 0000000..292807d --- /dev/null +++ b/.skills/llm-wiki/references/CORE-MAINTENANCE.md @@ -0,0 +1,44 @@ +# Core maintenance (upstream v2026.10.4) + +Use the repository's `obsidian-wiki` CLI (or its `.venv/bin/python -m obsidian_wiki`). Resolve the vault and read its AGENTS.md first. Explicit vault arguments avoid accidentally maintaining another vault. This protocol takes precedence over older instructions to rewrite shared metadata manually. + +## Source tracking + +`obsidian-wiki cache-check "$OBSIDIAN_VAULT_PATH" ... --pretty` detects byte changes. After successful ingestion, `cache-update "$OBSIDIAN_VAULT_PATH" --pages ...` records the hash under a manifest lock and atomically replaces the manifest. Omitting `--pages` preserves existing associations; supplying it replaces `pages_produced`, so include every page produced for that source, including earlier still-valid pages. Both list and dictionary manifests are supported; unknown fields are preserved. + +Keep the existing original-path key when enriching an existing entry: use `--key ` if the source argument is now an archive copy. Do not create a second entry just to adopt upstream key conventions. Hash checks establish exact identity only, not fuzzy deduplication. + +The helper does **not** replace the local archive/catalog protocol or update `_meta/source-registry.jsonl` and `file-assets.sqlite`. Preserve original paths, archive paths, structured identifiers, ACL fields and curated registry values. Continue the normal file-index rebuild and ingest verification. + +For custom manifest fields, perform the entire read/merge/write under `obsidian_wiki.cache.manifest_lock(vault)` and atomically write using `obsidian_wiki.memory.atomic_write`. Read after acquiring the lock; never write a stale whole-manifest snapshot or call a helper that reacquires the same lock. This lock protects only cooperating writers. + +## Index, log and hot cache + +Inspect first: + +```bash +obsidian-wiki memory status --vault "$OBSIDIAN_VAULT_PATH" --json +obsidian-wiki memory migrate --vault "$OBSIDIAN_VAULT_PATH" --json +``` + +`migrate` without `--apply` is a preview. An existing hand-maintained index/hot cache is guarded: upgrading this framework does not authorize regenerating those files. Show the migration preview and obtain authorization for that data migration before running `memory migrate --apply`. Migration backs up the old files under `_archives/`. + +For an adopted vault, finish a write with: + +```bash +obsidian-wiki memory sync INGEST source=example.pdf pages=2 --vault "$OBSIDIAN_VAULT_PATH" --takeaways "Concise findings from this operation." +``` + +Use the actual verb, fields and takeaways. `sync` maintains log, index and hot under one shared lock. If another command already logged the operation, omit the verb to avoid duplicate entries. `memory index --check` previews index drift (exit 2) without writing. + +For a legacy vault, retain its index/hot layout: use `memory log` for append-only logging, and merge necessary index/hot edits with the whole read/merge/atomic-write inside `memory.memory_lock(vault)`. Do not bypass the migration guard with `force=True`, auto-adopt, or replace existing narrative with a generated document. Do not nest locking helpers inside that lock. This compatibility path keeps normal ingest usable without requiring a migration. + +## Lint, links and staging + +Run `obsidian-wiki lint "$OBSIDIAN_VAULT_PATH" --json` for structural checks before semantic review. Link resolution distinguishes same-named pages in different project folders and ignores links inside code examples. Missing trust-ledger information is a legacy warning by default; do not fabricate reviews or change confidence to make checks pass. + +Knowledge walkers exclude `_sources`, `_meta`, `_raw`, `_inbox`, `_bot_exports` and archive/staging directories. `_sources/archive` and `_sources/catalog` links remain valid source references. For nonstandard archive/catalog locations, add exclusions to `.okignore`; inspect any resulting citation warnings instead of moving originals. + +Use `staging list --vault "$OBSIDIAN_VAULT_PATH" --json` and preserve both staged/live revisions shown during review. Promote only with `--expect-staged` plus `--expect-live` or `--expect-new`; exit 9 means the reviewed content changed. Re-read and review the changed content instead of retrying blindly. See `wiki-stage-commit` for the complete workflow. + +These are local operator commands. They do not enable MCP tools, change remote permissions or launch services. diff --git a/.skills/wiki-ingest/SKILL.md b/.skills/wiki-ingest/SKILL.md index ec7b7c4..2bde7cd 100644 --- a/.skills/wiki-ingest/SKILL.md +++ b/.skills/wiki-ingest/SKILL.md @@ -15,6 +15,8 @@ You are ingesting source documents into an Obsidian wiki. Your job is not to sum ## Before You Start +Apply `llm-wiki/references/CORE-MAINTENANCE.md` for source hash checks and all shared metadata writes. Use the locked cache helper for standard manifest updates and a locked read/merge/atomic-write for additional local fields. Keep all archive, source-registry, file-index, ACL and deterministic ingest-verification steps below. The helper does not perform those steps for you. + 1. **Resolve config** — follow the Config Resolution Protocol in `llm-wiki/SKILL.md` (walk up CWD for `.env` → `~/.obsidian-wiki/config` → prompt setup). This gives `OBSIDIAN_VAULT_PATH`, `OBSIDIAN_SOURCES_DIR`, `OBSIDIAN_LINK_FORMAT` (default: `wikilink`), `WIKI_STAGED_WRITES`, `WIKI_ARCHIVE_SOURCES`, `OBSIDIAN_SOURCE_ARCHIVE_DIR`, `WIKI_SOURCE_CATALOG`, `OBSIDIAN_SOURCE_CATALOG_DIR`, and optionally `OBSIDIAN_FILE_ASSET_DB`. Only read the specific variables you need — do not log, echo, or reference any other values from these files. 2. **Check `WIKI_STAGED_WRITES`** — if set to `true`, all new and updated category pages go to `_staging//` instead of their final location. Tell the user at the start of the ingest: "Staged writes mode is enabled — pages will land in `_staging/` for your review. Run `/wiki-stage-commit` when ready to promote." 3. Read `.manifest.json` at the vault root to check what's already been ingested diff --git a/.skills/wiki-lint/SKILL.md b/.skills/wiki-lint/SKILL.md index 1be6f13..6f9985d 100644 --- a/.skills/wiki-lint/SKILL.md +++ b/.skills/wiki-lint/SKILL.md @@ -24,6 +24,8 @@ You are performing a health check on an Obsidian wiki. Your goal is to find and ## Lint Checks +First run `obsidian-wiki lint "$OBSIDIAN_VAULT_PATH" --json` using this repository's CLI. Use its findings for structural checks below, then inspect only implicated pages and perform semantic contradiction review. It resolves project-relative and duplicate-name links, ignores code examples, and excludes source archives from knowledge pages. Trust-ledger gaps are warnings in legacy mode; do not invent reviews or change scores to silence them. Follow `llm-wiki/references/CORE-MAINTENANCE.md` for any authorized repairs and tracking-file updates. If the CLI is unavailable, use the manual checks below with the same source-directory exclusions. + Run these checks in order. Report findings as you go. ### 1. Orphaned Pages @@ -529,4 +531,4 @@ Record one of: - `QMD refreshed: update only + verified` - `QMD skipped: QMD_WIKI_COLLECTION unset` - `QMD skipped: qmd CLI unavailable` -- `QMD failed: ` \ No newline at end of file +- `QMD failed: ` diff --git a/.skills/wiki-stage-commit/SKILL.md b/.skills/wiki-stage-commit/SKILL.md index cd2ae4f..df02e15 100644 --- a/.skills/wiki-stage-commit/SKILL.md +++ b/.skills/wiki-stage-commit/SKILL.md @@ -1,11 +1,7 @@ --- name: wiki-stage-commit description: > - Review and promote staged wiki pages to their final locations. Use when WIKI_STAGED_WRITES=true - and the user says "/wiki-stage-commit", "review staged pages", "commit staged writes", - "promote staged pages", "approve staged changes", or "what's waiting in staging". - Shows each staged file, lets the user accept or reject it, and moves accepted files to - their final wiki locations. Rejected files are moved back to _raw/ for manual editing. + Review staged wiki writes and promote accepted pages to final locations when WIKI_STAGED_WRITES is enabled. Use for staging approval/commit; rejected pages return to _raw/ for editing. --- # Wiki Stage Commit — Staged Write Promotion @@ -29,9 +25,24 @@ You are reviewing LLM-written pages that are waiting in `_staging/` for human ap ## Step 1: Inventory Staged Files -Glob `$OBSIDIAN_VAULT_PATH/_staging/**/*.md` — these are the pending pages. +The CLI owns the inventory — don't glob `_staging/` yourself: -Also glob `$OBSIDIAN_VAULT_PATH/_staging/**/*.patch.md` — these are pending *updates* to existing pages (diff-style files showing proposed additions and deletions). +```bash +obsidian-wiki staging list --vault "$OBSIDIAN_VAULT_PATH" --json +``` + +Each entry carries: + +| Field | Meaning | +|---|---| +| `staged_path` | vault-relative path of the staged file | +| `live_path` | where it will land (a `.patch.md` targets the page it is named after) | +| `kind` | `new`, `update`, or `patch` | +| `staged_revision` | content hash of the staged file, as you are seeing it now | +| `live_revision` | content hash of the live page, or `null` if there is none yet | +| `staged_mtime` | when it was staged | + +**Keep `staged_revision` and `live_revision` for every file you show the user.** They are what makes Step 3 refuse to overwrite an agent's concurrent write instead of silently clobbering it. Report the inventory: @@ -39,17 +50,17 @@ Report the inventory: Staged files: 4 new pages, 2 updates New pages: - _staging/concepts/attention-mechanism.md (ingested 2 days ago) - _staging/entities/andrej-karpathy.md (ingested 2 days ago) - _staging/skills/fine-tuning-llms.md (ingested yesterday) - _staging/references/attention-is-all-you-need.md (ingested 3 hours ago) + _staging/concepts/attention-mechanism.md (staged 2026-09-08) + _staging/entities/andrej-karpathy.md (staged 2026-09-08) -Updates (patch files): - _staging/concepts/transformer-architecture.patch.md (target: concepts/transformer-architecture.md) - _staging/skills/prompt-engineering.patch.md (target: skills/prompt-engineering.md) +Updates: + _staging/concepts/transformer-architecture.md (target: concepts/transformer-architecture.md) + +Patches: + _staging/skills/prompt-engineering.patch.md (target: skills/prompt-engineering.md) ``` -If `_staging/` is empty, report: "Nothing staged. All writes have been committed or no staged writes have been produced yet." +If the list is empty, report: "Nothing staged. All writes have been committed or no staged writes have been produced yet." ## Step 2: Per-File Review (interactive mode) @@ -89,7 +100,7 @@ Proposed additions (+): Proposed deletions (-): - The attention mechanism was first described in [Bahdanau 2015]. (to be replaced by updated claim) -⚠️ Conflict check: target page was modified 3 days after staging. Review carefully. +⚠️ Conflict check: live_revision no longer matches what was staged against. Review carefully. Accept [a], Reject [r], Skip [s], Preview full diff [p]? ``` @@ -100,42 +111,56 @@ If `--list` flag is set, stop after printing the inventory (Step 1). ## Step 3: Apply Decisions +The moves are mechanical and the CLI does them atomically. Pass the revisions from Step 1 so a decision made on stale information fails loudly. + ### Accepting a new page -1. Move `_staging//page.md` → `/page.md` (the final location) -2. Update `index.md` with the new page entry -3. Remove the staged file +```bash +obsidian-wiki staging promote --vault "$OBSIDIAN_VAULT_PATH" \ + --expect-staged --expect-new +``` -### Accepting a patch/update +`--expect-new` refuses if a live page has appeared since you listed — someone else got there first. -1. Read the current page at the target path -2. Apply the proposed additions and deletions (merge, don't just overwrite) -3. Update the `updated` frontmatter timestamp -4. Update `index.md` if the summary changed -5. Remove the staged patch file +### Accepting an update + +```bash +obsidian-wiki staging promote --vault "$OBSIDIAN_VAULT_PATH" \ + --expect-staged --expect-live +``` ### Rejecting a file -Move it to `$OBSIDIAN_VAULT_PATH/_raw/` for manual editing: -- `_staging/concepts/page.md` → `_raw/rejected-concepts-page.md` -- `_staging/concepts/page.patch.md` → `_raw/rejected-patch-concepts-page.md` -- Prefix with `rejected-` so the user can identify it +```bash +obsidian-wiki staging discard --vault "$OBSIDIAN_VAULT_PATH" +``` -### Conflict detection on patch accept +It lands in `_raw/rejected--.md` for manual editing. An earlier rejection of the same page is never overwritten — the second becomes `-2`. -Before applying a patch, check whether the target page's `updated` frontmatter is newer than the patch file's own `updated` field: -- If the target was modified AFTER the patch was staged, warn: `⚠️ Conflict: target was updated since this patch was staged. Applying may lose recent changes.` -- Give the user a chance to abort: `Apply anyway [y], Skip [s], Reject [r]?` +### Accepting a patch + +`promote` refuses `.patch.md` files, because merging a human-readable diff into a page whose surrounding text may have moved is judgment, not a rename. Do it yourself: + +1. Read the target and patch; retain the live revision from the inventory. +2. Merge the proposed changes into a complete staged `.md` page, preserving unrelated live content. Bump its `updated` frontmatter. Do not modify the live file yet or overwrite an unrelated queued page. +3. Hash the merged staged page and promote it with `--expect-staged ` and the **original** `--expect-live ` (or `--expect-new`). If either changed, review again. +4. After successful promotion, discard the reviewed patch to clear it from the queue. Verify it still matches the reviewed patch before discarding; leave a concurrently changed patch queued. + +### When the CLI reports a conflict + +Exit code **9** with `conflict: ...` on stderr means the staged or live file changed after you listed it. Nothing was moved. Do not retry with fresh revisions blindly — re-run `staging list`, show the user what changed, and ask again. The whole point of the check is that the content they approved is no longer the content that would land. + +Promotion itself does not reconcile `index.md`; do that in Step 4 after pages become live. `log.md` gets one `STAGE_COMMIT` line per invocation, written by the CLI. ## Step 4: Update Tracking Files -After processing all staged files: +The CLI appends the `STAGE_COMMIT` line to `log.md` itself. Follow `llm-wiki/references/CORE-MAINTENANCE.md`. For an adopted vault, reconcile the index and hot cache after processing all staged files: -1. **`hot.md`** — update the Recent Activity section: "Committed N staged pages; rejected M." -2. **`log.md`** — append: - ``` - - [TIMESTAMP] STAGE_COMMIT accepted=N rejected=M skipped=K - ``` +```bash +obsidian-wiki memory sync --vault "$OBSIDIAN_VAULT_PATH" --takeaways "Committed N staged pages; rejected M." +``` + +(No `--verb`: the log line was already written by `staging promote`.) The command takes the shared lock. For a legacy vault, preserve the existing index/hot layout with a locked read/merge/atomic-write as documented in Core Maintenance. Do not auto-migrate to complete a promotion. ## Step 5: Report @@ -159,6 +184,7 @@ Staging queue: K files remaining ## Notes - Staged files use the same page template as live pages — they are ready to land, just awaiting approval -- Patch files use a human-readable diff format: lines starting with `+` are additions, lines starting with `-` are deletions +- Patch files use a human-readable diff format: lines starting with `+` are additions, lines starting with `-` are deletions. `staging promote` refuses them — merge them yourself (Step 3) - `index.md` and `log.md` are always updated immediately on ingest (they are low-risk tracking files) — only category pages go through staging +- The moves live in `obsidian_wiki/staging.py`; this skill supplies the judgment, not the file handling - The `_staging/` directory is not tracked by Obsidian's graph view — pages only appear in the wiki after promotion diff --git a/AGENTS.md b/AGENTS.md index 32f6f37..b523636 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,6 +1,6 @@ # Obsidian Wiki — Agent Context -A **skill-based framework** for building and maintaining an Obsidian knowledge base. No scripts or dependencies — everything is markdown instructions that you execute directly. +A **skill-based framework** for building and maintaining an Obsidian knowledge base. Markdown skills guide the work; standard-library CLI helpers handle shared metadata, links and staging deterministically. MCP support is optional. ## Configuration @@ -137,6 +137,8 @@ See `wiki-query` and `wiki-export` skills for how the filter is applied. ## Core Principles +Shared-file maintenance follows `.skills/llm-wiki/references/CORE-MAINTENANCE.md`: use locked cache/memory helpers, path-aware lint and revision-checked staging. Preserve local archive/registry fields. Existing index/hot layouts require explicit migration authorization before generated replacement; legacy edits must still use the shared lock and atomic writes. + - **Compile, don't retrieve.** The wiki is pre-compiled knowledge. Update existing pages — don't append or duplicate. - **Track everything.** Update `.manifest.json` after ingesting, `index.md`, `log.md`, and `hot.md` after any write operation. - **Connect with `[[wikilinks]]`.** Every page should link to related pages. This is what makes it a knowledge graph, not a folder of files. diff --git a/README.md b/README.md index a50e0af..0f68203 100644 --- a/README.md +++ b/README.md @@ -15,10 +15,12 @@ A knowledge mgmt system inspired by [gist](https://gist.github.com/karpathy/442a Instead of asking an LLM the same questions over over (or doing RAG every time), you compile knowledge once into interconnected markdown files and keep them current. In this case Obsidian is the viewer and the LLM is the maintainer. -We took that and built a framework around it. The whole thing is a set of markdown skill files that any AI coding agent (Claude Code, Cursor, Windsurf, Pi, whatever you use) can read and execute. You point it at your Obsidian vault and tell it what to do. +We took that and built a framework around it. Markdown skill files guide AI coding agents (Claude Code, Cursor, Windsurf, Pi, and others), with CLI helpers for deterministic maintenance. You point it at your Obsidian vault and tell it what to do. ## Quick Start +This fork selectively includes the core maintenance updates from upstream **v2026.10.4**: hash-based source tracking, locked memory writes, path-aware lint/graph analysis, and revision-checked staging. See [the sync scope and migration notes](docs/upstream-sync.md). Existing source-file lookup and MCP deployment conventions remain local; upgrading the code does not migrate vault contents or enable services. + ### Install via pip (recommended) ```bash diff --git a/docs/upstream-sync.json b/docs/upstream-sync.json new file mode 100644 index 0000000..4ad66b2 --- /dev/null +++ b/docs/upstream-sync.json @@ -0,0 +1,113 @@ +{ + "upstream": "https://github.com/Ar9av/obsidian-wiki", + "release": "v2026.10.4", + "commit": "4f457e154653756b04802d5aeb25c20f42b4770c", + "scope": "core-maintenance", + "source_sha256": { + "obsidian_wiki/vault.py": "604e65a01eb44438eea402f00f6c6c534e06856c20fe6c6b5ceb47e69291ee4a", + "obsidian_wiki/cache.py": "f410ac542ce47719b5c8aa687bcd8beafd7d70a6f7db3cd2df24c9030628a9f6", + "obsidian_wiki/clock.py": "456749a083b29a51820332970d9f07ee49bfaec6314cc51449672c8c96a53dce", + "obsidian_wiki/memory.py": "81fc4e28b8cdd761d6be23d0de6e6b35dad49501ebb0ec9e3411f18bf23a3170", + "obsidian_wiki/links.py": "79c2ebb69b9ded9823bffebf270b40b520e640e740b57a0f7909deb66cc5e980", + "obsidian_wiki/lint.py": "482a69b5a7ef15c45d5054efec31c68f23e7906f62fcf9dd640987deac82c19e", + "obsidian_wiki/graph_analysis.py": "a0932567affc219b7a7692036b389ec9674b8b9e1c94f72ce27fcc69078cb25c", + "obsidian_wiki/provenance.py": "ce39dca4a9069814ee0b3dd779ebdcf32942d510048b44ed7b71fa9521734b9b", + "obsidian_wiki/temporal.py": "9e31cdd08b1dabbc6d94590db7b5b2295923d8f33e299d9179d84820955c3395", + "obsidian_wiki/trust.py": "c397465fcdcb2df31c0ea64ba65ef411532161b174f170be2d9662d8e75745cd", + "obsidian_wiki/staging.py": "626d6f059ec089ca23981d1e9c3d9ba90266c74292ee313e654ac713eb38dbb7" + }, + "cli_functions": [ + "SchemaOptions", + "_memory_fields", + "_memory_takeaways", + "_note_source_ambiguity", + "_print_lint", + "_read_config_file", + "_report_empty_local_vault", + "_resolve_schema_command_context", + "_resolve_source_arg", + "_schema_cli_values", + "_schema_csv", + "_schema_options", + "_schema_source_value", + "_source_candidates", + "_timezone_report", + "_verb_and_fields", + "cmd_cache_check", + "cmd_cache_hash", + "cmd_cache_update", + "cmd_graph_analyse", + "cmd_lint", + "cmd_memory", + "cmd_projects_check", + "cmd_staging" + ], + "commands": [ + "graph-analyse", + "cache-check", + "projects-check", + "cache-update", + "cache-hash", + "lint", + "memory", + "staging" + ], + "excluded": "upstream server, setup/hooks, session brain, retrieval platform and bulk vault migrations", + "test_exclusions_outside_scope": { + "clock": [ + "TestDocs", + "test_scaffold_writes_the_key_and_stamps_in_the_zone", + "test_setup_cli_records_the_zone", + "test_setup_cli_refuses_a_bad_zone_before_touching_anything" + ], + "memory": [ + "test_a_scaffolded_vault_is_born_migrated", + "test_a_scaffolded_vault_carries_the_adoption_record" + ], + "links": [ + "obsidian_wiki.graphrag", + "test_graph_query_index_matches_the_graph" + ], + "lint": [ + "test_invalid_configured_required_trust_field_fails_closed_for_all_cli_paths", + "test_empty_configured_schema_values_fail_closed_for_all_cli_paths", + "test_blank_config_schema_source_cannot_be_masked_by_valid_cli_source", + "test_distributed_schema_config_contract_names_all_four_variables", + "test_correction_contract_requires_temporal_authority_and_immutable_hash_check" + ], + "graph_analysis": [ + "TestGitignoreBackfill" + ], + "trust": [ + "test_owner_lifecycle_cli_record_then_check", + "test_invalid_review_timestamp_is_rejected_by_record_cli", + "test_trust_record_cli_requires_explicit_approval", + "test_trust_record_and_check_cli_round_trip", + "test_owner_schema_cli_excludes_then_removes_no_confidence_ledger_entry", + "test_trust_record_all_human_output_lists_no_confidence_without_removal", + "test_trust_record_all_human_output_warns_when_rebuild_removes_stale_entry", + "test_trust_record_page_human_output_warns_when_stale_entry_is_removed", + "test_trust_record_page_human_output_reports_zero_removals_without_warning", + "test_explicit_vault_cli_override_reports_cli_schema_source", + "test_partial_trust_record_does_not_approve_other_stale_pages", + "test_partial_trust_record_reports_malformed_ledger_without_traceback" + ], + "temporal": [ + "obsidian_wiki.graphrag", + "test_expired_page_leaves_retrieval_but_stays_in_the_graph", + "test_query_as_of_returns_the_page_that_was_true_then", + "test_include_historical_ranks_both", + "test_temporal_keys_are_absent_on_pages_that_do_not_opt_in", + "test_structural_intents_still_see_historical_pages", + "test_graph_query_cli_accepts_as_of", + "test_cli_rejects_a_malformed_as_of" + ] + }, + "local_adaptations": [ + "Keep existing ~/.obsidian-wiki/config location and adapt the memory CLI fixture accordingly", + "Exclude local source, metadata, inbox and bot-export directories from knowledge pages", + "Resolve _sources/archive and _sources/catalog citations without changing file extensions", + "Preserve pages_produced when cache-update omits --pages", + "Keep local CLI, MCP dependency range, service gating and curated source-registry metadata" + ] +} diff --git a/docs/upstream-sync.md b/docs/upstream-sync.md new file mode 100644 index 0000000..975b708 --- /dev/null +++ b/docs/upstream-sync.md @@ -0,0 +1,49 @@ +# 上游关键更新同步 + +本次选择性同步 [Ar9av/obsidian-wiki v2026.10.4](https://github.com/Ar9av/obsidian-wiki/releases/tag/v2026.10.4)，固定提交 `4f457e154653756b04802d5aeb25c20f42b4770c`。这是核心维护能力的移植，不表示本分支完整等同于该版本。来源哈希、移植的 CLI 函数和未纳入的测试范围记录在 `upstream-sync.json`。 + +| 纳入能力 | 实际作用 | +| --- | --- | +| `cache-check` / `cache-update` / `cache-hash` | 用内容哈希识别变化；manifest 锁和原子写入；兼容列表/字典及旧哈希格式 | +| `projects-check` | 按已记录提交检查项目资料是否落后 | +| `memory` | 统一维护 index、log、hot；提供只读检查、显式迁移及备份 | +| `lint` / `graph-analyse` | 正确区分不同项目中的同名页，解析相对链接，忽略代码示例里的伪链接；检查结构、时效与来源信息 | +| `staging` | 暂存晋升前校验已审阅的 staged/live 内容哈希；冲突时退出并保留文件 | + +同步了 `llm-wiki`、`wiki-ingest`、`wiki-lint`、`wiki-stage-commit` 的相关执行指引，详细协议见 `.skills/llm-wiki/references/CORE-MAINTENANCE.md`。图分析是显式调用的维护工具，不接入远程问答或自动调度。 + +## 本地兼容 + +- 保留 `~/.obsidian-wiki/config`、现有 setup、file-index、bot-view 和 MCP 命令。 +- 保留 `_sources/archive` / `_sources/catalog`、原文件索引、结构化产品字段及 ACL。知识页遍历排除原始资料、元数据、收件箱和导出副本；归档文件仍可被引用。 +- `cache-update` 未传 `--pages` 时保留已有页面关联；显式传入时替换 `pages_produced`。自定义元数据需在同一个 manifest 锁内读取、合并并原子写回。registry 和 SQLite 索引仍由本地 ingest 流程维护。 +- 未改动已有知识页面、manifest、registry、SQLite 或 index/hot；未启动 Wiki MCP，未修改独立订单项目。 +- 未导入上游 HTTP 服务、MCP 2.x 依赖、会话 hooks、session brain、GraphRAG 查询链路或全量数据迁移。 + +## 使用与迁移 + +从本项目虚拟环境执行，避免误用系统中另一个版本： + +```bash +.venv/bin/python -m obsidian_wiki lint /path/to/vault --json +.venv/bin/python -m obsidian_wiki memory status --vault /path/to/vault --json +.venv/bin/python -m obsidian_wiki memory migrate --vault /path/to/vault --json +.venv/bin/python -m obsidian_wiki staging list --vault /path/to/vault --json +``` + +这些检查不重写知识库。`memory migrate` 默认只预览；只有明确决定迁移数据格式后才加 `--apply`，届时先备份旧 index/hot。未迁移的库继续保留现有布局，使用共享锁合并必要修改。框架升级本身不触发该迁移。 + +lint 的旧库模式把 trust ledger 缺失等问题报告为警告。若以后启用严格审阅，需要独立设计实际审阅流程，不能为了消除警告伪造置信度。自定义归档目录可加入 `.okignore`，其非标准引用需单独核查。 + +## 验证与下次同步 + +```bash +uv sync --extra mcp --group dev +PYTHONPATH="$PWD" .venv/bin/python -m pytest tests -q +``` + +测试包括原有文件检索、ACL、MCP query/ingest 和运维回归，以及本次核心模块的上游测试；另有本地兼容测试验证归档排除、引用解析与 manifest 字段保留。测试使用临时知识库，不启动实际服务。 + +本次验证结果：**609 项测试通过**。实际知识库只执行了 `memory status` 只读检查，index/hot/log、manifest、registry、SQLite 的前后 SHA-256 一致。Wiki MCP 保持 inactive，独立订单 MCP 保持 active。 + +下次以 `upstream-sync.json` 的提交和源文件哈希作比较基线，逐模块检查新提交，再重放列出的本地适配。不要直接覆盖 `.skills`、运行上游 setup、切换服务依赖或批量改写知识库。上游未纳入模块的测试明确列入排除清单，不能把它们的缺席解释为整版上游已通过验证。 diff --git a/obsidian_wiki/cache.py b/obsidian_wiki/cache.py new file mode 100644 index 0000000..c2f9b73 --- /dev/null +++ b/obsidian_wiki/cache.py @@ -0,0 +1,765 @@ +"""Content-hash cache for wiki-ingest source tracking. + +Provides a reliable, platform-independent alternative to running `sha256sum` +in the skill. The agent calls `obsidian-wiki cache-check` / `cache-update` +instead of shelling out to sha256sum and manually parsing .manifest.json. + +Manifest format (.manifest.json in the vault root). Two `sources` shapes are +supported, because real vaults contain both: + +1. Dict keyed by path (this module's original format):: + + {"sources": {"": {"content_hash": "...", ...}}} + +2. List of entry objects, each carrying its own ``path`` (the shape the + wiki-ingest skill writes):: + + {"sources": [{"path": "_raw/foo.md", "content_hash": "sha256:...", ...}]} + +Both shapes are read transparently, and `update_source` edits the manifest +*in place* — preserving its shape, any duplicate-path entries, and +skill-written fields (``pages_created``, ``size_bytes``, ``source_type``, …). + +Hashes are compared prefix-insensitively: the skill records +``"sha256:"`` while this module computes a bare ````, so an optional +``algo:`` prefix is stripped from both sides before comparison. +""" + +from __future__ import annotations + +import hashlib +import json +import os +import re +import subprocess +import sys +import time +from contextlib import contextmanager +from datetime import datetime, timezone +from pathlib import Path +from typing import Iterator, TypedDict + + +class SourceEntry(TypedDict, total=False): + content_hash: str + last_ingested: str + pages_produced: list[str] + + +class CheckResult(TypedDict): + new: list[str] + modified: list[str] + unchanged: list[str] + missing: list[str] # vault-local source in manifest but no longer on disk + unavailable: list[str] # machine-local source absent here (e.g. synced from another host) + + +def _manifest_path(vault: Path) -> Path: + return vault / ".manifest.json" + + +def _load_raw(vault: Path) -> dict: + """Return the full manifest object (``{}`` if absent/unreadable).""" + mp = _manifest_path(vault) + if not mp.exists(): + return {} + try: + data = json.loads(mp.read_text(encoding="utf-8")) + return data if isinstance(data, dict) else {} + except (json.JSONDecodeError, OSError): + return {} + + +def _load_manifest(vault: Path): + """Return the raw ``sources`` value — a dict, a list, or ``{}`` if absent. + + Kept for backward compatibility; callers that need shape-agnostic access + should use :func:`_iter_entries`. + """ + return _load_raw(vault).get("sources", {}) + + +def _lock_path(vault: Path) -> Path: + return vault / ".manifest.lock" + + +class AdvisoryLockTimeout(RuntimeError): + """Raised when an advisory lockfile could not be claimed within the timeout.""" + + +class ManifestLockTimeout(AdvisoryLockTimeout): + """Raised when the manifest lock could not be acquired within the timeout.""" + + +@contextmanager +def advisory_lock( + lock: Path, + *, + timeout: float = 10.0, + stale_after: float = 60.0, + error_cls: type = AdvisoryLockTimeout, +): + """Claim *lock* as an advisory lockfile for a read-modify-write. + + ``O_CREAT | O_EXCL`` is the portable stdlib primitive here — ``fcntl`` is + POSIX-only and this ships on Windows. A lockfile older than *stale_after* + is assumed to belong to a crashed process and is stolen. + + Shared by the manifest writer and the prose-memory writer + (:mod:`obsidian_wiki.memory`) so both serialise the same way. + """ + deadline = time.monotonic() + timeout + fd = None + while True: + try: + fd = os.open(lock, os.O_CREAT | os.O_EXCL | os.O_WRONLY) + break + except FileExistsError: + try: + age = time.time() - lock.stat().st_mtime + except FileNotFoundError: + continue # holder released it between our open and stat + if age > stale_after: + # ponytail: last-writer-wins steal; a pid check would narrow the + # window if two processes ever race to steal the same stale lock + try: + lock.unlink() + except FileNotFoundError: + pass + continue + if time.monotonic() >= deadline: + raise error_cls( + f"could not acquire {lock} within {timeout}s " + f"(held for {age:.1f}s; stale after {stale_after}s)" + ) + time.sleep(0.1) + try: + os.write(fd, str(os.getpid()).encode()) + os.close(fd) + fd = None + yield + finally: + if fd is not None: + os.close(fd) + try: + lock.unlink() + except FileNotFoundError: + pass + + +@contextmanager +def manifest_lock(vault: Path, *, timeout: float = 10.0, stale_after: float = 60.0): + """Advisory lock around a manifest read-modify-write. + + Parallel ingest agents (``batch-plan`` fan-out) and the Docker server can + all write one vault's manifest; without this the read-modify-write races + and a whole entry is silently lost. + """ + # ponytail: whole-manifest advisory lock; per-entry locking if batch fan-out ever contends + with advisory_lock( + _lock_path(vault), + timeout=timeout, + stale_after=stale_after, + error_cls=ManifestLockTimeout, + ): + yield + + +def _write_manifest(vault: Path, manifest: dict) -> None: + """Atomically replace the manifest file — no torn reads, no partial file.""" + target = _manifest_path(vault) + tmp = target.with_name(f".manifest.json.{os.getpid()}.tmp") + try: + tmp.write_text( + json.dumps(manifest, indent=2, ensure_ascii=False), encoding="utf-8" + ) + os.replace(tmp, target) # atomic on POSIX and Windows + except BaseException: + try: + tmp.unlink() + except FileNotFoundError: + pass + raise + + +def _save_manifest(vault: Path, sources) -> None: + """Write *sources* back into the manifest, preserving other top-level keys.""" + with manifest_lock(vault): + manifest = _load_raw(vault) + manifest["sources"] = sources + _write_manifest(vault, manifest) + + +def _iter_entries(sources) -> Iterator[tuple[str | None, dict]]: + """Yield ``(stored_key, entry)`` pairs for either manifest shape. + + For the dict shape the key is the dict key; for the list shape it is the + entry's ``path`` (falling back to ``source_id``). + """ + if isinstance(sources, dict): + for key, entry in sources.items(): + if isinstance(entry, dict): + yield key, entry + elif isinstance(sources, list): + for entry in sources: + if isinstance(entry, dict): + yield (entry.get("path") or entry.get("source_id")), entry + + +# Matches "sha256:", "https://", "slack:#..." — anything with an algo/scheme +# prefix that shouldn't be treated as a filesystem path. +_SCHEME_RE = re.compile(r"^[A-Za-z][A-Za-z0-9+.\-]*:[^\\/]") + +# A Windows drive-absolute path ("C:\dir\file", "C:/dir/file"). On POSIX this is +# not a path at all, so it cannot be resolved and is machine-specific; lint.py +# recognises the same shape when reporting stored keys. Kept local rather than +# imported so the runtime cache carries no dependency on the lint module. +_WINDOWS_ABS_RE = re.compile(r"^[A-Za-z]:[\\/]") + + +def _strip_algo(value: str | None) -> str: + """Drop an optional ``algo:`` prefix so ``sha256:`` == ````.""" + value = value or "" + return value.split(":", 1)[1] if ":" in value else value + + +def _format_hash(existing: str | None, new_hex: str) -> str: + """Format *new_hex* keeping any ``algo:`` prefix the existing value used.""" + if existing and ":" in existing: + return f"{existing.split(':', 1)[0]}:{new_hex}" + return new_hex + + +def _is_file_key(key: str | None) -> bool: + """True if *key* looks like a filesystem path rather than a URL/pseudo-key.""" + return bool(key) and "://" not in key and not _SCHEME_RE.match(key) + + +def _expand_key(key: str) -> str: + """Expand ``~`` and environment variables, but only when the key uses them. + + A plain vault-relative key like ``Raw/x.pdf`` must survive untouched, so the + expansion is gated rather than applied unconditionally. + """ + if key.startswith("~") or "$" in key: + return os.path.expandvars(os.path.expanduser(key)) + return key + + +def resolve_key(key: str | None, vault: Path) -> Path | None: + """Normalize a manifest key to an absolute path, or ``None`` for non-file keys. + + Resolution order (the source key contract v2, see ``llm-wiki/SKILL.md``): + + 1. pseudo-key (``scheme:`` / ``://``) -> ``None`` — an opaque identifier, not + a filesystem location; + 2. ``~``- or env-var-bearing key -> expand; + 3. absolute path -> use as-is (legacy compatibility); + 4. anything else -> resolve against the vault root. + """ + if not _is_file_key(key): + return None + path = Path(_expand_key(key)) + return path if path.is_absolute() else (vault / path) + + +def stored_key(path: Path, vault: Path) -> str | None: + """Return the portable manifest key for *path*, or ``None`` if not portable. + + In-vault sources become vault-relative (``Raw/x.pdf``); sources under + ``$HOME`` become home-relative (``~/.claude/...``); anything else has no + machine-portable representation and needs an explicit pseudo-key from the + caller (``repo:``, ``url:``, ``agent:``). + """ + p = Path(os.path.abspath(_expand_key(str(path)))) + vault_root = Path(os.path.abspath(_expand_key(str(vault)))) + home_root = Path(os.path.expanduser("~")) + for root, prefix in ((vault_root, ""), (home_root, "~/")): + try: + rel = p.relative_to(root) + except ValueError: + continue + if rel == Path("."): + return None + return prefix + rel.as_posix() + return None + + +def _same_source(stored_key_value: str | None, query: Path, vault: Path) -> bool: + """True if a manifest key refers to the same source as *query*. + + Matches on the raw string first (covers pseudo-keys), then on the resolved + absolute form so an absolute query matches a vault-relative or home-relative + stored key. + """ + if not stored_key_value: + return False + if str(query) == stored_key_value: + return True + stored_path = resolve_key(stored_key_value, vault) + if stored_path is None: + return False + try: + return stored_path.resolve() == query.resolve() + except OSError: + return False + + +def _missing_on_disk(key: str | None, vault: Path) -> bool: + """True if a filesystem-style manifest key has no file on disk.""" + path = resolve_key(key, vault) + if path is None: + return False + return not path.exists() + + +def _vault_top_names(vault: Path) -> set[str]: + """Names of the vault's top-level entries, or an empty set if unreadable.""" + try: + return {p.name for p in vault.iterdir()} + except OSError: + return set() + + +def _is_vault_local(key: str | None, vault: Path, top_names: set[str] | None = None) -> bool: + """True if a file key names a source that travels with the vault. + + Vault-local sources travel with the vault, so their absence is a real loss + (``missing``). A key that is machine-specific *and* points outside this vault + (``/home/other/wiki/x.md``, ``~/.claude/...``) may simply not exist on the + machine reading a synced vault, so it is reported under ``unavailable``. An + absolute or ``~``-relative path that resolves *inside* this vault is still + vault-local — machine-specific in form, but not in target. + + A *relative* key resolves lexically inside the vault even when it is really + relative to some other root — the legacy ingest-root keys such as + ``-Users-x-github/abc.jsonl``. Path shape cannot tell those apart from vault + keys, so the vault's own topology decides: the first segment must name a real + top-level entry. That also gives the honest answer for an out-of-vault source + parked in a namespace the vault does not contain (``external/.hermes/...``): + unavailable, not missing. When the vault cannot be listed the lexical answer + stands, so an I/O hiccup never hides a real loss. + + A key written for another OS — a drive-letter path or one using backslashes + as separators — cannot be resolved on this host, so it is machine-specific + (``unavailable``) rather than a vault-relative key. + """ + path = resolve_key(key, vault) + if path is None: + return False + raw = str(key) + try: + rel = Path(os.path.abspath(str(path))).relative_to(Path(os.path.abspath(str(vault)))) + except ValueError: + return False + if rel == Path("."): + return False + if os.name != "nt" and ("\\" in raw or _WINDOWS_ABS_RE.match(raw)): + # A key written for another OS: a backslash is not a separator here and a + # drive-letter path cannot be resolved, so this is machine-specific — not + # a vault source whose file went away. + return False + if os.path.isabs(raw) or raw.startswith("~") or "$" in raw: + return True + if "/" not in raw and "\\" not in raw: + # A bare filename at the vault root. There is no leading component that + # could be a foreign root, so this is a vault source by construction — + # and if it is gone, that is a real vault-local loss. The backslash guard + # keeps a Windows-form key out of this shortcut on every platform. + return True + if top_names is None: + top_names = _vault_top_names(vault) + if not top_names: + return True + return rel.parts[0] in top_names + + +def sha256_file(path: Path, chunk: int = 1 << 20) -> str: + """Return the hex SHA-256 digest of *path* without loading it all into RAM.""" + h = hashlib.sha256() + with open(path, "rb") as fh: + while True: + block = fh.read(chunk) + if not block: + break + h.update(block) + return h.hexdigest() + + +def sha256_dir(path: Path) -> str: + """Stable SHA-256 over all files in a directory tree (sorted by relative path).""" + h = hashlib.sha256() + for fp in sorted(path.rglob("*"), key=lambda p: p.relative_to(path).as_posix()): + if fp.is_file(): + rel = fp.relative_to(path).as_posix() + h.update(rel.encode()) + h.update(sha256_file(fp).encode()) + return h.hexdigest() + + +def compute_hash(path: Path) -> str: + if path.is_dir(): + return sha256_dir(path) + return sha256_file(path) + + +def _resolved_form(path: Path) -> str | None: + """``realpath`` of *path* as a comparable string, or ``None`` if it cannot be resolved. + + ``os.path.normcase`` keeps the comparison equivalent to ``Path`` equality on + case-insensitive filesystems. + """ + try: + return os.path.normcase(str(path.resolve())) + except OSError: + return None + + +def check_sources(vault: Path, source_paths: list[Path]) -> CheckResult: + """Classify each source as new / modified / unchanged vs. the manifest. + + Also reports manifest entries whose source file no longer exists on disk. + Handles both manifest shapes and compares hashes prefix-insensitively. + + Every entry and every query is resolved once and matched through an index + of its forms (raw key, resolved path). Resolving is a ``realpath`` walk — + one ``lstat`` per path component — so doing it per (entry, query) pair, as + the first-match scan did, made a 3,000-entry manifest cost about a second + per queried file. The index keeps the first entry for each form, which is + what the scan's ``break`` on first hit returned. + """ + entries = list(_iter_entries(_load_manifest(vault))) + result: CheckResult = { + "new": [], "modified": [], "unchanged": [], "missing": [], "unavailable": [] + } + + by_form: dict[str, int] = {} + entry_resolved: list[str | None] = [] + for i, (stored_key, _entry) in enumerate(entries): + resolved = None + if stored_key: + by_form.setdefault(stored_key, i) + stored_path = resolve_key(stored_key, vault) + if stored_path is not None: + resolved = _resolved_form(stored_path) + if resolved is not None: + by_form.setdefault(resolved, i) + entry_resolved.append(resolved) + + matched: set[int] = set() + query_forms: set[str] = set() + for path in source_paths: + key = str(path) + forms = [key] + resolved = _resolved_form(path) + if resolved is not None: + forms.append(resolved) + query_forms.update(forms) + if not path.exists(): + result["missing"].append(key) + continue + current_hash = _strip_algo(compute_hash(path)) + hits = [by_form[f] for f in forms if f in by_form] + if not hits: + result["new"].append(key) + continue + i = min(hits) # the earliest entry matching by either form — the old scan's first hit + matched.add(i) + entry = entries[i][1] + if _strip_algo(entry.get("content_hash")) != current_hash: + result["modified"].append(key) + else: + result["unchanged"].append(key) + + # Report manifest entries whose source file no longer exists on disk and + # that weren't among the scanned paths (in any path form). + top_names = _vault_top_names(vault) + for i, (stored_key, _entry) in enumerate(entries): + if i in matched: + continue + if stored_key in query_forms or (entry_resolved[i] is not None and entry_resolved[i] in query_forms): + continue + if _missing_on_disk(stored_key, vault): + bucket = "missing" if _is_vault_local(stored_key, vault, top_names) else "unavailable" + result[bucket].append(stored_key) + + return result + + +def update_source( + vault: Path, + source_path: Path, + *, + pages_produced: list[str] | None = None, + key: str | None = None, +) -> str: + """Record the current hash of *source_path* in the manifest. Returns the hash. + + Edits the manifest in place: matches an existing entry across path forms and + updates it, otherwise appends a new one — preserving the manifest's shape + (dict or list), duplicate-path entries, and any skill-written fields. + + The stored key is normalized to the portable form (vault-relative for + in-vault sources, ``~``-relative under ``$HOME``) so a synced vault never + accumulates machine absolute paths. Pass *key* explicitly for sources that + have no filesystem representation in either form — a pseudo-key such as + ``repo:github.com/owner/name``, ``url:https://...``, or ``agent:claude/``. + When neither applies, the raw path is kept for backward compatibility and a + warning is written to stderr, because that entry is not portable across + machines. + """ + now = datetime.now(timezone.utc).isoformat() + explicit_key = key is not None + hash_path = source_path + if explicit_key: + new_key = key + else: + derived = stored_key(source_path, vault) + if derived is None: + new_key = str(source_path) + print( + f"warning: {source_path} is outside the vault and $HOME, so it has no " + f"portable key; storing the absolute path. Pass key= " + f"to keep the vault portable across machines.", + file=sys.stderr, + ) + else: + from obsidian_wiki.provenance import prefer_archive_write_path + preferred = prefer_archive_write_path(vault, derived) + if preferred is not None: + new_key = preferred + hash_path = vault / preferred + else: + new_key = derived + # Hash outside the lock — hashing a large source tree can take seconds and + # nothing else in the manifest depends on it. + current_hash = compute_hash(hash_path) + + with manifest_lock(vault): + return _update_source_locked( + vault, source_path, new_key, explicit_key, current_hash, now, pages_produced, + ) + + +def _update_source_locked( + vault: Path, + source_path: Path, + new_key: str, + explicit_key: bool, + current_hash: str, + now: str, + pages_produced: list[str] | None, +) -> str: + """The manifest read-modify-write half of :func:`update_source`.""" + manifest = _load_raw(vault) + sources = manifest.get("sources") + + if isinstance(sources, list): + target: dict | None = None + for e in sources: + if isinstance(e, dict) and _same_source( + e.get("path") or e.get("source_id"), source_path, vault + ): + target = e + break + if target is None: + target = {"path": new_key} + sources.append(target) + elif target.get("path") != new_key and ( + explicit_key or _rekey_stale_raw_to_archive(target.get("path"), new_key) + ): + # Explicit --key is authoritative; otherwise only stale + # ``_raw/.md`` keys move onto ``_raw/_archived/…``. + target["path"] = new_key + target["content_hash"] = _format_hash(target.get("content_hash"), current_hash) + target["last_ingested"] = now + if pages_produced is not None: + target["pages_produced"] = pages_produced + else: + if not isinstance(sources, dict): + sources = {} + match_key: str | None = None + for existing_key in sources: + if _same_source(existing_key, source_path, vault): + match_key = existing_key + break + if match_key is not None and match_key != new_key and ( + explicit_key or _rekey_stale_raw_to_archive(match_key, new_key) + ): + # Explicit key wins; otherwise only stale ``_raw/.md`` keys + # move onto ``_raw/_archived/…`` so other portable keys stay put. + moved = sources.pop(match_key) + existing = sources.get(new_key) + if isinstance(existing, dict): + moved = {**existing, **moved} + sources[new_key] = moved + match_key = new_key + # A matched legacy entry otherwise keeps its existing key; only new + # entries use the portable form, so an in-place update never re-keys a + # dict by surprise (migrate handles legacy conversion). + manifest_key = match_key if match_key is not None else new_key + entry = sources.get(manifest_key) if isinstance(sources.get(manifest_key), dict) else {} + entry["content_hash"] = _format_hash(entry.get("content_hash"), current_hash) + entry["last_ingested"] = now + if pages_produced is not None: + entry["pages_produced"] = pages_produced + sources[manifest_key] = entry + + manifest["sources"] = sources + _write_manifest(vault, manifest) + return current_hash + + +def _rekey_stale_raw_to_archive(old_key: str | None, new_key: str) -> bool: + if not old_key: + return False + old = Path(old_key) + new = Path(new_key) + return ( + old.parts[:1] == ("_raw",) + and "_archived" not in old.parts + and len(new.parts) >= 3 + and new.parts[0] == "_raw" + and new.parts[1] == "_archived" + ) + + +def hash_file(path: Path) -> str: + """Just compute and return the hash — no manifest I/O.""" + return compute_hash(path) + + +# --- project (git) staleness ------------------------------------------------ +# `check_sources` answers "did this source file's bytes change?". It cannot +# answer the same question for a project, because a git repo is not stored as a +# hashed source: wiki-update records it under `projects` keyed by `source_repo` +# with a `last_commit_synced` bookmark. Nothing compared that bookmark to HEAD, +# so a vault could be arbitrarily far behind a repo with no signal at all. + +_GIT_TIMEOUT = 15 + + +class ProjectStatus(TypedDict, total=False): + project: str + commits: int + last_commit_synced: str + path: str + reason: str + + +class ProjectCheckResult(TypedDict): + behind: list[ProjectStatus] # checkout has moved past last_commit_synced + current: list[str] # last_commit_synced == HEAD + unreachable: list[ProjectStatus] # stored sha not usable (rebase/force-push/gc) + unsynced: list[str] # entry exists but no last_commit_synced yet + unavailable: list[ProjectStatus] # no usable checkout on this machine + + +def _git_in(path: Path, *args: str) -> subprocess.CompletedProcess | None: + """Run git in *path*; None when git itself is unusable (absent, hung).""" + try: + return subprocess.run( + ["git", "-C", str(path), *args], + capture_output=True, text=True, timeout=_GIT_TIMEOUT, + ) + except (OSError, subprocess.SubprocessError): + return None + + +def check_projects(vault: Path) -> ProjectCheckResult: + """Classify each manifest ``projects`` entry by how far its checkout has + moved past the recorded ``last_commit_synced``. + + Mirrors :func:`check_sources`' vocabulary deliberately: a project whose + checkout is not on this machine is ``unavailable``, not stale. + ``source_cwd_hint`` is a *hint* — the manifest is portable across machines + by design, so a missing checkout is an absence of evidence, never a finding. + + Read-only: never runs a git command that can write, and never fails the + whole call because one entry is malformed. + """ + raw = _load_raw(vault).get("projects") + projects = raw if isinstance(raw, dict) else {} + result: ProjectCheckResult = { + "behind": [], "current": [], "unreachable": [], "unsynced": [], "unavailable": [] + } + + for name in sorted(projects): + entry = projects[name] + if not isinstance(entry, dict): + result["unavailable"].append({"project": name, "reason": "malformed_entry"}) + continue + + hint = entry.get("source_cwd_hint") + if not isinstance(hint, str) or not hint.strip(): + result["unavailable"].append({"project": name, "reason": "no_source_cwd_hint"}) + continue + path = Path(hint.strip()).expanduser() + if not path.is_dir(): + result["unavailable"].append( + {"project": name, "reason": "checkout_not_found", "path": str(path)} + ) + continue + + # `rev-parse --git-dir` rather than a `.git` directory test: a linked + # worktree's `.git` is a file, and the hint may point below the root. + probe = _git_in(path, "rev-parse", "--git-dir") + if probe is None: + result["unavailable"].append( + {"project": name, "reason": "git_unavailable", "path": str(path)} + ) + continue + if probe.returncode != 0: + result["unavailable"].append( + {"project": name, "reason": "not_a_git_repo", "path": str(path)} + ) + continue + + sha = entry.get("last_commit_synced") + if not isinstance(sha, str) or not sha.strip(): + result["unsynced"].append(name) + continue + sha = sha.strip() + + ancestor = _git_in(path, "merge-base", "--is-ancestor", sha, "HEAD") + if ancestor is None: + result["unavailable"].append( + {"project": name, "reason": "git_unavailable", "path": str(path)} + ) + continue + if ancestor.returncode != 0: + # 1 = resolves but is not an ancestor (history was rewritten); + # anything else (128) = the object is not in this repo at all. + result["unreachable"].append({ + "project": name, + "reason": "not_ancestor" if ancestor.returncode == 1 else "unknown_commit", + "last_commit_synced": sha, + "path": str(path), + }) + continue + + counted = _git_in(path, "rev-list", "--count", f"{sha}..HEAD") + if counted is None or counted.returncode != 0: + result["unavailable"].append( + {"project": name, "reason": "count_failed", "path": str(path)} + ) + continue + try: + commits = int(counted.stdout.strip()) + except ValueError: + result["unavailable"].append( + {"project": name, "reason": "count_unparsable", "path": str(path)} + ) + continue + + if commits == 0: + result["current"].append(name) + else: + result["behind"].append({ + "project": name, + "commits": commits, + "last_commit_synced": sha, + "path": str(path), + }) + + return result diff --git a/obsidian_wiki/cli.py b/obsidian_wiki/cli.py index 8d4b1df..d4fd708 100644 --- a/obsidian_wiki/cli.py +++ b/obsidian_wiki/cli.py @@ -12,9 +12,11 @@ from __future__ import annotations import argparse import json import os +import re import shutil import sys from pathlib import Path +from typing import TypedDict from obsidian_wiki import __version__ from obsidian_wiki.access_control import audit_profile, export_profile, list_profiles @@ -551,6 +553,819 @@ def cmd_bot_view_export(args: argparse.Namespace) -> int: # ── Argument parsing ───────────────────────────────────────────────────────── +# Core maintenance commands ported from upstream v2026.10.4. + +class SchemaOptions(TypedDict): + allowed_lifecycles: frozenset[str] + allowed_relationship_types: frozenset[str] + required_trust_fields: tuple[str, ...] + schema_source: str + + +def cmd_graph_analyse(args: argparse.Namespace) -> int: + from obsidian_wiki import graph_analysis as ga + vault = Path(args.vault).expanduser().resolve() + if not vault.is_dir(): + print(f"error: vault not found: {vault}", file=sys.stderr) + return 1 + + if args.path or args.around: + # Query modes: no full analysis, just the graph walk. + outgoing, _ = ga.parse_vault_graph(vault) + if args.path: + src, tgt = args.path + path = ga.shortest_path(outgoing, src, tgt, directed=args.direction == "out") + result: dict = {"source": ga._slug(src), "target": ga._slug(tgt), "path": path, + "hops": (len(path) - 1) if path else None} + else: + hits = ga.neighborhood(outgoing, args.around, depth=args.depth, direction=args.direction) + result = {"seed": ga._slug(args.around), "depth": args.depth, + "direction": args.direction, "pages": hits, "count": len(hits), + "note": "pages and count exclude the seed itself"} + else: + previous = None + if args.diff_against: + previous = ga.load_snapshot(Path(args.diff_against).expanduser()) + if previous is None: + print(f"warning: no GRAPH_SNAPSHOT found in {args.diff_against}; skipping diff", + file=sys.stderr) + result = ga.analyse_vault(vault, top_n=args.top, previous_snapshot=previous, + include_snapshot=args.snapshot) + if args.pretty: + print(json.dumps(result, indent=2)) + else: + print(json.dumps(result)) + return 0 + + +def _source_candidates(vault: Path, raw: str) -> list[Path]: + """Candidate absolute paths for a source argument, in resolution order. + + Relative arguments resolve against the CWD first (the long-standing + behavior out-of-vault sources rely on), then against the vault root — + relative manifest keys are read as vault-relative by + ``cache.resolve_key``, so a vault-relative argument must resolve the same + way on the way in. Absolute arguments keep only themselves. + """ + p = Path(raw).expanduser() + if p.is_absolute(): + return [p.resolve()] + candidates = [] + for base in (Path.cwd(), vault): + resolved = (base / p).resolve() + if resolved not in candidates: + candidates.append(resolved) + return candidates + + +def _resolve_source_arg(vault: Path, raw: str) -> Path: + """First *source_candidates* form that exists on disk. + + When no candidate exists, the CWD-relative form is returned so read-only + callers (cache-check) can classify it as missing; writers must treat a + missing result as an error rather than hashing a wrong file. + """ + candidates = _source_candidates(vault, raw) + for candidate in candidates: + if candidate.exists(): + return candidate + return candidates[0] + + +def _note_source_ambiguity(vault: Path, raw: str, chosen: Path) -> None: + """Warn when a relative argument exists under both the CWD and the vault. + + The typed string is then byte-for-byte a manifest key, but the vault file + it names is not the one being hashed. Staying quiet would let the vault + entry keep a stale hash while a second entry appears under the CWD key. + """ + candidates = _source_candidates(vault, raw) + if len(candidates) != 2 or not all(c.exists() for c in candidates): + return + cwd_path, vault_path = candidates + print(f"note: {raw} matched both {cwd_path} and {vault_path}; " + f"using {chosen}", file=sys.stderr) + + +def cmd_cache_check(args: argparse.Namespace) -> int: + from obsidian_wiki.cache import check_sources + vault = Path(args.vault).expanduser().resolve() + sources = [_resolve_source_arg(vault, p) for p in args.sources] + result = check_sources(vault, sources) + if args.pretty: + print(json.dumps(result, indent=2)) + else: + print(json.dumps(result)) + return 0 + + +def cmd_projects_check(args: argparse.Namespace) -> int: + from obsidian_wiki.cache import check_projects + vault = Path(args.vault).expanduser().resolve() + result = check_projects(vault) + print(json.dumps(result, indent=2) if args.pretty else json.dumps(result)) + # Exit 2 on a real staleness finding so a scheduled run can branch on it; + # unavailable/unsynced are absences of evidence, not findings. + return 2 if (result["behind"] or result["unreachable"]) else 0 + + +def cmd_cache_update(args: argparse.Namespace) -> int: + from obsidian_wiki.cache import stored_key, update_source + from obsidian_wiki.provenance import prefer_archive_write_path + vault = Path(args.vault).expanduser().resolve() + source = _resolve_source_arg(vault, args.source) + rel = stored_key(source, vault) or args.source + preferred = None if args.key else prefer_archive_write_path(vault, rel) + if not source.exists() and preferred is None: + tried = " and ".join(str(c) for c in _source_candidates(vault, args.source)) + print(f"error: source {args.source} does not exist (tried {tried})", + file=sys.stderr) + return 1 + _note_source_ambiguity(vault, args.source, source) + pages = args.pages # Omission must preserve an existing page association. + h = update_source(vault, source, pages_produced=pages, key=args.key) + stored = args.key or preferred or stored_key(source, vault) or str(source) + print(json.dumps({ + "path": str(source if source.exists() else vault / (preferred or rel)), + "key": stored, + "content_hash": h, + })) + return 0 + + +def cmd_cache_hash(args: argparse.Namespace) -> int: + from obsidian_wiki.cache import hash_file + path = Path(args.path).expanduser().resolve() + if not path.exists(): + print(f"error: {path} does not exist", file=sys.stderr) + return 1 + print(json.dumps({"path": str(path), "sha256": hash_file(path)})) + return 0 + + +def _print_lint(report: dict[str, object]) -> None: + print(f"obsidian-wiki lint: {report['status']}") + stats = report["stats"] + print(f"pages: {stats['pages']} links: {stats['link_count']}") + for name, count in stats["findings"].items(): + print(f"{name}: {count}") + + +def _schema_csv(config: dict[str, str], key: str) -> list[str]: + if key not in config: + return [] + values = [item.strip() for item in config[key].split(",")] + if any(not item for item in values): + raise ValueError(f"invalid {key} value: entries must not be empty") + return values + + +def _schema_cli_values(values: list[str] | None, flag: str) -> list[str]: + normalised = [item.strip() for item in values or []] + if any(not item for item in normalised): + raise ValueError(f"invalid {flag} value: must not be empty") + return normalised + + +def _schema_source_value( + args: argparse.Namespace, + config: dict[str, str], +) -> str | None: + configured_value: str | None = None + if "OBSIDIAN_SCHEMA_SOURCE" in config: + configured_value = config["OBSIDIAN_SCHEMA_SOURCE"].strip() + if not configured_value: + raise ValueError("invalid OBSIDIAN_SCHEMA_SOURCE value: must not be empty") + + cli_value = getattr(args, "schema_source", None) + if cli_value is not None: + value = cli_value.strip() + if not value: + raise ValueError("invalid --schema-source value: must not be empty") + return value + return configured_value + + +def _read_config_file(path: Path) -> dict[str, str]: + if not path.is_file(): + return {} + values: dict[str, str] = {} + for raw in path.read_text(encoding="utf-8").splitlines(): + line = raw.strip() + if not line or line.startswith("#") or "=" not in line: + continue + key, value = line.split("=", 1) + values[key.strip()] = value.strip().strip("'\"") + return values + + +def _report_empty_local_vault(env_file: Path | str) -> None: + # An empty OBSIDIAN_VAULT_PATH= in a project .env deliberately blocks the + # global vault, but a copied .env.example does the same by accident, so + # name the file that stopped the walk. + print( + f"error: vault not configured; {env_file} sets OBSIDIAN_VAULT_PATH to empty, " + "which blocks the global config. Set it, delete the line, or pass a path", + file=sys.stderr, + ) + + +def _resolve_schema_command_context( + vault_arg: str | None, +) -> tuple[Path, dict[str, str], str] | None: + config: dict[str, str] + config_source: str + if vault_arg and vault_arg.startswith("@"): + name = vault_arg[1:] + if not name or re.fullmatch(r"[A-Za-z0-9_-]+", name) is None: + print("error: named vault must use @ followed by letters, digits, _ or -", file=sys.stderr) + return None + path = GLOBAL_CONFIG_DIR / f"config.{name}" + config = _read_config_file(path) + config_source = str(path) + resolved = config.get("OBSIDIAN_VAULT_PATH", "") + elif vault_arg is not None: + config = {} + config_source = "explicit-vault" + resolved = vault_arg + else: + current = Path.cwd().resolve() + config = {} + config_source = str(GLOBAL_CONFIG) + while True: + candidate = current / ".env" + local = _read_config_file(candidate) + if "OBSIDIAN_VAULT_PATH" in local: + config = local + config_source = str(candidate) + break + if current == HOME or current.parent == current: + break + current = current.parent + if not config: + config = _read_config_file(GLOBAL_CONFIG) + resolved = config.get("OBSIDIAN_VAULT_PATH", "") + if not resolved: + if config_source.endswith(".env"): + _report_empty_local_vault(config_source) + else: + print("error: vault not configured; pass a path, @name, or run obsidian-wiki setup", file=sys.stderr) + return None + vault = Path(resolved).expanduser().resolve() + if not vault.is_dir(): + print(f"error: vault not found: {vault}", file=sys.stderr) + return None + return vault, config, config_source + + +def _schema_options( + args: argparse.Namespace, + config: dict[str, str], + config_source: str, + *, + default_required_trust_fields: tuple[str, ...] | None = None, +) -> SchemaOptions: + from obsidian_wiki.lint import ( + ALLOWED_RELATIONSHIP_TYPES, + TRUST_REQUIRED_FRONTMATTER, + ) + from obsidian_wiki.trust import ( + ALLOWED_LIFECYCLES, + TRUST_REQUIRED_FIELD_ALLOWLIST, + ) + + cli_lifecycles = _schema_cli_values( + getattr(args, "allow_lifecycle", None), "--allow-lifecycle" + ) + cli_relationships = _schema_cli_values( + getattr(args, "allow_relationship_type", None), "--allow-relationship-type" + ) + raw_cli_required = getattr(args, "required_trust_field", None) + cli_required = ( + _schema_cli_values(raw_cli_required, "--required-trust-field") + if raw_cli_required is not None + else None + ) + configured_lifecycles = _schema_csv(config, "OBSIDIAN_ALLOWED_LIFECYCLES") + configured_relationships = _schema_csv(config, "OBSIDIAN_ALLOWED_RELATIONSHIP_TYPES") + configured_required = _schema_csv(config, "OBSIDIAN_REQUIRED_TRUST_FIELDS") + unknown_required = sorted( + set(configured_required).union(cli_required or ()) - TRUST_REQUIRED_FIELD_ALLOWLIST + ) + if unknown_required: + allowed = ", ".join(sorted(TRUST_REQUIRED_FIELD_ALLOWLIST)) + unknown = ", ".join(unknown_required) + raise ValueError( + "invalid OBSIDIAN_REQUIRED_TRUST_FIELDS value(s): " + f"{unknown}; allowed values: {allowed}" + ) + required = tuple( + cli_required + if cli_required is not None + else configured_required + or list(default_required_trust_fields or TRUST_REQUIRED_FRONTMATTER) + ) + cli_overrides = bool( + cli_lifecycles + or cli_relationships + or cli_required is not None + ) + configured_overrides = bool( + configured_lifecycles + or configured_relationships + or configured_required + ) + source = _schema_source_value(args, config) + if not source: + if cli_overrides and configured_overrides: + source = f"cli+config:{config_source}" + elif cli_overrides: + source = f"cli:{config_source}" + elif configured_overrides: + source = f"config:{config_source}" + else: + source = "framework-defaults" + return { + "allowed_lifecycles": ALLOWED_LIFECYCLES.union(configured_lifecycles, cli_lifecycles), + "allowed_relationship_types": ALLOWED_RELATIONSHIP_TYPES.union( + configured_relationships, cli_relationships + ), + "required_trust_fields": required, + "schema_source": source, + } + + +def cmd_staging(args: argparse.Namespace) -> int: + """List, promote, or discard staged pages. + + Mirrors `/wiki-stage-commit`, which now calls this instead of restating the + steps. Revision pins are opt-in: pass them for a reviewed decision, omit them + for an unattended `--all`. + """ + from obsidian_wiki.staging import ( + StagingConflict, + StagingError, + discard, + list_staged, + log_decisions, + promote, + ) + + context = _resolve_schema_command_context(args.vault) + if context is None: + return 1 + vault = context[0] + + if args.staging_action == "list": + entries = [entry.as_dict() for entry in list_staged(vault)] + if args.json: + print(json.dumps(entries, indent=2) if args.pretty else json.dumps(entries)) + elif not entries: + print("Nothing staged.") + else: + for entry in entries: + print(f"{entry['kind']:>6} {entry['live_path']} ({entry['staged_path']})") + print(f"\n{len(entries)} staged file(s)") + return 0 + + if not args.path: + print(f"error: `staging {args.staging_action}` needs a path", file=sys.stderr) + return 1 + + try: + if args.staging_action == "promote": + result = promote( + vault, + args.path, + expected_staged_revision=args.expect_staged, + expected_live_revision=args.expect_live, + pin_live_absent=args.expect_new, + ) + else: + result = discard(vault, args.path) + except StagingConflict as exc: + # Distinct exit code: a conflict is "re-read and retry", not "bad input". + print(f"conflict: {exc}", file=sys.stderr) + return 9 + except StagingError as exc: + print(f"error: {exc}", file=sys.stderr) + return 1 + + log_decisions(vault, [result]) + if args.json: + print(json.dumps(result, indent=2) if args.pretty else json.dumps(result)) + elif result["action"] == "promote": + print(f"promoted {result['staged_path']} -> {result['live_path']} ({result['kind']})") + else: + print(f"discarded {result['staged_path']} -> {result['raw_path']}") + return 0 + + +def cmd_lint(args: argparse.Namespace) -> int: + from obsidian_wiki.lint import lint_vault + + context = _resolve_schema_command_context(args.vault) + if context is None: + return 1 + vault, config, config_source = context + + strict_trust = args.strict_trust or config.get("OBSIDIAN_TRUST_STRICT", "").strip().lower() in ( + "1", + "true", + "yes", + ) + try: + schema = _schema_options(args, config, config_source) + report = lint_vault( + vault, + require_trust_ledger=True, + strict_trust=strict_trust, + **schema, + ) + except ValueError as exc: + print(f"error: {exc}", file=sys.stderr) + return 1 + if args.json: + if args.pretty: + print(json.dumps(report, indent=2)) + else: + print(json.dumps(report)) + else: + _print_lint(report) + findings = report["findings"] + strict_relevant = any( + items + for name, items in findings.items() + if name != "snapshot_mismatch" and items + ) + if report["status"] == "fail" or (args.strict and strict_relevant): + return 1 + return 0 + + +def _memory_fields(pairs: list) -> dict: + fields: dict[str, str] = {} + for pair in pairs or []: + if "=" not in pair: + raise ValueError(f"--field expects key=value, got {pair!r}") + key, value = pair.split("=", 1) + fields[key.strip()] = value + return fields + + +def _verb_and_fields(rest: list, verb_flag: str | None, field_flags: list) -> tuple: + """Read `VERB key=value ...` positionally, falling back to the flag form. + + `--verb X --field a=1 --field b=2` is precise and unreadable; the six-field + calls the ingest skills make ran to seven lines. A bare verb and bare + key=value pairs say the same thing on one line. Both forms still work. + """ + verb, fields = verb_flag, {} + for token in rest or []: + if "=" in token: + key, value = token.split("=", 1) + fields[key.strip()] = value + elif verb is None: + verb = token + else: + raise ValueError( + f"unexpected argument {token!r} — expected key=value " + f"(the verb {verb!r} is already set)" + ) + fields.update(_memory_fields(field_flags)) + return verb, fields + + +def _memory_takeaways(raw: str | None) -> str | None: + """`-` reads the takeaway prose from stdin, so a skill can pipe it in.""" + if raw is None: + return None + return sys.stdin.read().strip() if raw == "-" else raw + + +def _timezone_report(plan: dict, *, verbose: bool, applied: bool) -> list: + """Human form of a timezone migration plan or result.""" + name = plan["timezone"] + files, count, stamps = plan["files"], plan["file_count"], plan["stamp_count"] + would = "" if applied else "would " + lines = [] + if applied: + key = " (recorded in index.md)" if plan.get("key_set") else "" + lines.append(f"timezone: {name}{key}") + if plan.get("backup_dir"): + lines.append(f"backed up {len(plan['backup'])} file(s) to {plan['backup_dir']}") + else: + current = plan.get("key_current", "") + key = "" if current == name else (" (not yet in index.md; --apply records it)" if not current else f" (index.md says {current}; --apply changes it)") + lines.append(f"timezone: {name}{key}") + lines.append( + f"{count} file(s), {stamps} stamp(s) {would}change; " + f"{plan['naive_skipped']} naive stamp(s) and {plan['bare_dates']} bare date(s) left alone" + ) + shown = files if verbose else files[:40] + for entry in shown: + lines.append(f" {entry['path']:<60} {len(entry['stamps'])}") + samples = entry["stamps"] if verbose else entry["stamps"][:2] + for change in samples: + lines.append(f" {change['line']:>5}: {change['old']} -> {change['new']}") + if not verbose and len(entry["stamps"]) > 2: + lines.append(f" … {len(entry['stamps']) - 2} more in this file") + if len(files) > len(shown): + lines.append(f" … {len(files) - len(shown)} more file(s); --verbose lists every file and stamp, --json has them all") + if applied: + lines.append(plan["log_line"]) + elif count: + lines += [ + "", + "Every file touched is copied to _archives/pre-timezone-migration-/ first; --skip GLOB excludes a file.", + f"Apply with: obsidian-wiki memory timezone{' ' + name if plan.get('key_current') != name else ''} --apply", + ] + return lines + + +def cmd_memory(args: argparse.Namespace) -> int: + """Maintain the vault memory surface: log, index, hot cache, profile, todos. + + One code path for files that fifteen skills used to each rewrite from prose, + all of it under a single advisory lock so a parallel writer cannot drop an + update. `--check` reports drift and exits 2 without writing, for CI. + """ + from obsidian_wiki import clock + from obsidian_wiki import memory as mem + + context = _resolve_schema_command_context(args.vault) + if context is None: + return 1 + vault, config, _source = context + link_format = (config.get("OBSIDIAN_LINK_FORMAT") or _read_config_value("OBSIDIAN_LINK_FORMAT") or "wikilink").strip() + rest = list(args.rest or []) + + def emit(payload: dict, lines: list) -> int: + if args.json: + print(json.dumps(payload, indent=2 if args.pretty else None)) + else: + for line in lines: + print(line) + return 0 + + try: + action = args.memory_action + + if action == "status": + status = mem.memory_status(vault) + drift = status["index_drift"] + return emit(status, [ + f"vault: {status['vault']}", + f"timezone: {status['timezone']}", + f"pages: {status['pages']}", + f"index: {'stale' if drift['stale'] else 'current'}" + + (f" (+{len(drift['added'])} / -{len(drift['removed'])})" if drift["stale"] else ""), + f"log: {status['log_entries']} entries", + ("migrated: yes" if status["migrated"] else + "migrated: NO — run `obsidian-wiki memory migrate` before index/hot writes"), + f"hot: {status['hot']['words']}/{status['hot']['max_words']} words" + + (" OVER BUDGET" if status["hot"]["over_budget"] else "") + + ("" if status["hot"]["generated"] else " (hand-written, not yet generated)"), + f"profile: {status['profile_facts']} fact(s)", + f"todos: {status['todos']['open']} open, {status['todos']['stale']} stale," + f" {status['todos']['closed']} closed", + ]) + + if action == "now": + # The stamp a skill should write into a page: the vault's own zone, + # so an agent never has to know which .env it resolved. + tz = clock.wiki_timezone(vault) + payload = { + "timestamp": clock.stamp(tz), + "date": clock.today(tz), + "timezone": clock.timezone_name(vault) or "UTC", + } + return emit(payload, [payload["date"] if args.date else payload["timestamp"]]) + + if action == "timezone": + zone = rest[0] if rest else "" + skip = list(args.skip or []) + if args.check or not args.apply: + plan = mem.timezone_migration_plan(vault, zone, skip=skip) + lines = _timezone_report(plan, verbose=args.verbose, applied=False) + emit(plan, lines) + return 2 if plan["file_count"] else 0 + result = mem.migrate_timezone(vault, zone, skip=skip) + return emit(result, _timezone_report(result, verbose=args.verbose, applied=True)) + + if action == "migrate": + if args.check or not args.apply: + preview = mem.migration_status(vault) + if preview["migrated"]: + return emit(preview, ["already migrated — index.md and hot.md are generated"]) + idx, hot = preview["index"], preview["hot"] + lines = [ + "This vault predates the memory writer. Migrating would:", + "", + f" index.md {idx['lines_before']} -> {idx['lines_after']} lines," + f" {idx['entries_added']} catalog entries appended", + ] + lines += [f" keep, unchanged, above the generated catalog: {name}" + for name in idx["sections_preserved"]] + lines.append(f" hot.md regenerates: {', '.join(hot['sections_regenerated']) or 'nothing'}") + lines.append(f" Key Takeaways carried over: {'yes' if hot['takeaways_carried'] else 'no'}") + if hot["threads_to_seed"]: + lines.append(f" {len(hot['threads_to_seed'])} thread(s) converted to todos:") + lines += [f" - {t[:80]}" for t in hot["threads_to_seed"]] + lines += ["", "A timestamped backup is written to _archives/ first.", + "Apply with: obsidian-wiki memory migrate --apply"] + emit(preview, lines) + return 0 + result = mem.migrate(vault, link_format=link_format) + lines = [f"backed up to {result['backup_dir']}"] if result["backup"] else [] + lines.append(f"index: {result['index']['total']} page(s), +{len(result['index']['added'])}") + lines.append(f"hot: {result['hot']['words']} words") + if result["threads_seeded"]: + lines.append(f"todos: seeded {len(result['threads_seeded'])} thread(s) from hot.md") + return emit(result, lines) + + if action == "log": + verb, fields = _verb_and_fields(rest, args.verb, args.field) + if not verb: + print("error: memory log needs a VERB, e.g. `memory log INGEST source=x`", file=sys.stderr) + return 1 + line = mem.append_log(vault, verb, fields) + return emit({"line": line}, [line]) + + if action == "index": + result = mem.rebuild_index(vault, link_format=link_format, write=not args.check) + payload = { + "total": result.total, + "added": list(result.added), + "removed": list(result.removed), + "changed": result.changed, + "written": result.changed and not args.check, + } + lines = [ + f"{result.total} page(s); " + + (f"+{len(result.added)} / -{len(result.removed)}" if result.changed else "no drift") + ] + lines += [f" + {path}" for path in result.added] + lines += [f" - {path}" for path in result.removed] + emit(payload, lines) + return 2 if (args.check and result.changed) else 0 + + if action == "hot": + result = mem.rebuild_hot( + vault, + write=not args.check, + takeaways=_memory_takeaways(args.takeaways), + link_format=link_format, + max_words=args.max_words, + ) + payload = { + "words": result.words, + "max_words": args.max_words or mem.hot_max_words(), + "trimmed": result.trimmed, + "activity": result.activity, + "threads": result.threads, + "contradictions": result.contradictions, + "written": not args.check, + } + emit(payload, [ + f"hot.md {'would be ' if args.check else ''}rebuilt: {result.words} words" + + (" (trimmed to fit)" if result.trimmed else ""), + f" {result.activity} activity, {result.threads} thread(s)," + f" {result.contradictions} contradiction(s)", + ]) + return 0 + + if action == "recap": + print( + mem.build_recap( + vault, + max_words=args.max_words or 400, + min_confidence=args.min_confidence, + project=args.project, + ), + end="", + ) + return 0 + + if action == "sync": + # The post-write call: one lock held across log, index, and hot, so + # another writer cannot interleave between the three. + verb, fields = _verb_and_fields(rest, args.verb, args.field) + index = hot = None + skipped = "" + with mem.memory_lock(vault): + line = mem.append_log(vault, verb, fields, lock=False) if verb else "" + try: + index = mem.rebuild_index(vault, link_format=link_format, lock=False) + hot = mem.rebuild_hot( + vault, + lock=False, + takeaways=_memory_takeaways(args.takeaways), + link_format=link_format, + max_words=args.max_words, + ) + except mem.MemoryError_ as exc: + # An unmigrated vault must not fail the whole ingest: the + # log line already landed and is append-only. Say so loudly + # instead, so it is not silently skipped forever. + if exc.code != "unmigrated": + raise + skipped = str(exc) + if skipped: + print(f"warning: index.md and hot.md not updated — {skipped}", file=sys.stderr) + payload = { + "log_line": line, + "skipped": skipped, + "index": None if index is None else { + "total": index.total, "added": list(index.added), "removed": list(index.removed), + }, + "hot": None if hot is None else {"words": hot.words, "trimmed": hot.trimmed}, + } + lines = [line] if line else [] + if index is not None and hot is not None: + lines.append(f"index: {index.total} page(s), +{len(index.added)} / -{len(index.removed)}") + lines.append(f"hot: {hot.words} words" + (" (trimmed)" if hot.trimmed else "")) + else: + lines.append("index/hot: skipped (run `obsidian-wiki memory migrate`)") + return emit(payload, lines) + + if action == "profile": + sub_action = rest[0] if rest else "list" + if sub_action == "list": + facts = mem.load_profile(vault) + return emit( + {"facts": [f.__dict__ for f in facts]}, + [ + f"{f.key}: {f.value} ({f.confidence:.2f}, {f.source}, {f.updated}" + + (f", until {f.until}{' — expired' if f.is_expired() else ''}" if f.until else "") + + ")" + for f in facts + ] + or ["no profile facts recorded"], + ) + if sub_action == "set": + if len(rest) < 3: + print("error: memory profile set KEY VALUE", file=sys.stderr) + return 1 + fact = mem.set_fact( + vault, rest[1], " ".join(rest[2:]), + confidence=args.confidence, source=args.source, until=args.until or "", + ) + return emit({"fact": fact.__dict__}, [f"set {fact.key}: {fact.value} ({fact.confidence:.2f})"]) + if sub_action == "forget": + if len(rest) < 2: + print("error: memory profile forget KEY", file=sys.stderr) + return 1 + removed = mem.forget_fact(vault, rest[1]) + emit({"removed": removed, "key": rest[1]}, + [f"forgot {rest[1]}" if removed else f"no such fact: {rest[1]}"]) + return 0 if removed else 1 + print(f"error: memory profile takes list|set|forget, got {sub_action!r}", file=sys.stderr) + return 1 + + if action == "todo": + sub_action = rest[0] if rest else "list" + if sub_action == "list": + todos = mem.load_todos(vault) + shown = todos if args.all else [t for t in todos if t.status == "open"] + return emit( + {"todos": [dict(t.__dict__, stale=t.is_stale(days=args.stale_days)) for t in shown]}, + [ + f"[{t.status:7}] {t.id:4} {t.text}" + + (f" ({t.origin})" if t.origin else "") + + (" STALE" if t.is_stale(days=args.stale_days) else "") + for t in shown + ] or ["no todos recorded"], + ) + if sub_action == "add": + if len(rest) < 2: + print("error: memory todo add TEXT", file=sys.stderr) + return 1 + todo = mem.add_todo(vault, " ".join(rest[1:]), origin=args.origin or "") + return emit({"todo": todo.__dict__}, [f"{todo.id}: {todo.text}"]) + if sub_action in ("done", "drop"): + if len(rest) < 2: + print(f"error: memory todo {sub_action} ID", file=sys.stderr) + return 1 + status = "done" if sub_action == "done" else "dropped" + todo = mem.set_todo_status(vault, rest[1], status) + return emit({"todo": todo.__dict__}, [f"{todo.id} -> {todo.status}"]) + if sub_action == "prune": + removed = mem.prune_todos(vault) + return emit({"pruned": removed}, [f"pruned {removed} closed item(s)"]) + print(f"error: memory todo takes list|add|done|drop|prune, got {sub_action!r}", file=sys.stderr) + return 1 + + print(f"error: unknown memory action {action!r}", file=sys.stderr) + return 1 + except mem.MemoryError_ as exc: + print(f"error: {exc}", file=sys.stderr) + return 1 + except ValueError as exc: + print(f"error: {exc}", file=sys.stderr) + return 1 + + def build_parser() -> argparse.ArgumentParser: p = argparse.ArgumentParser( prog="obsidian-wiki", @@ -636,6 +1451,187 @@ def build_parser() -> argparse.ArgumentParser: bep.add_argument("--include-files", action="store_true", help="also copy allowed archived source files cited by exported pages") bep.set_defaults(func=cmd_bot_view_export) + + ga = sub.add_parser( + "graph-analyse", + help="analyse the vault's wikilink graph: god nodes, bridges, communities, " + "surprising connections, suggested questions; or walk paths/neighbourhoods", + ) + ga.add_argument("vault", help="path to the Obsidian vault") + ga.add_argument("--top", type=int, default=20, help="number of top results to return (default: 20)") + ga.add_argument("--path", nargs=2, metavar=("FROM", "TO"), + help="shortest link path between two pages (query mode)") + ga.add_argument("--around", metavar="PAGE", + help="pages within --depth hops of PAGE (query mode; blast radius with --direction in)") + ga.add_argument("--depth", type=int, default=2, help="hops for --around (default: 2)") + ga.add_argument("--direction", choices=["both", "in", "out"], default="both", + help="link direction for --around / --path (default: both)") + ga.add_argument("--diff-against", metavar="FILE", + help="previous _insights.md (GRAPH_SNAPSHOT comment) or snapshot JSON to diff against") + ga.add_argument("--snapshot", action="store_true", + help="include a compact graph snapshot in the output for future --diff-against") + ga.add_argument("--pretty", action="store_true", help="pretty-print JSON output") + ga.set_defaults(func=cmd_graph_analyse) + cc = sub.add_parser( + "cache-check", + help="check which sources are new/modified/unchanged vs. .manifest.json", + ) + cc.add_argument("vault", help="path to the Obsidian vault") + cc.add_argument("sources", nargs="+", help="source file or directory paths to check") + cc.add_argument("--pretty", action="store_true", help="pretty-print JSON output") + cc.set_defaults(func=cmd_cache_check) + pc = sub.add_parser( + "projects-check", + help="check how far each project's checkout has moved past last_commit_synced", + ) + pc.add_argument("vault", help="path to the Obsidian vault") + pc.add_argument("--pretty", action="store_true", help="pretty-print JSON output") + pc.set_defaults(func=cmd_projects_check) + cu = sub.add_parser( + "cache-update", + help="record a source's current SHA-256 hash in .manifest.json after ingestion", + ) + cu.add_argument("vault", help="path to the Obsidian vault") + cu.add_argument("source", help="source file or directory that was just ingested") + cu.add_argument("--pages", nargs="*", metavar="PAGE", help="vault-relative paths of pages produced") + cu.add_argument( + "--key", + default=None, + help="explicit portable key (repo:/url:/agent:) for sources outside the vault and $HOME", + ) + cu.set_defaults(func=cmd_cache_update) + ch = sub.add_parser( + "cache-hash", + help="compute the SHA-256 hash of a file or directory (no manifest I/O)", + ) + ch.add_argument("path", help="file or directory to hash") + ch.set_defaults(func=cmd_cache_hash) + st = sub.add_parser( + "staging", + help="list, promote, or discard pages waiting in _staging/ (WIKI_STAGED_WRITES)", + ) + st.add_argument("staging_action", choices=["list", "promote", "discard"], help="what to do") + st.add_argument("path", nargs="?", help="staged path, as printed by `staging list`") + st.add_argument("--vault", dest="vault", help="vault path or @name (defaults via CWD .env, then global config)") + st.add_argument("--expect-staged", metavar="SHA", help="refuse if the staged file no longer matches this revision") + st.add_argument("--expect-live", metavar="SHA", help="refuse if the live page no longer matches this revision") + st.add_argument( + "--expect-new", + action="store_true", + help="refuse if a live page has appeared since review (pairs with kind=new)", + ) + st.add_argument("--json", action="store_true", help="emit machine-readable JSON") + st.add_argument("--pretty", action="store_true", help="pretty-print JSON output") + st.set_defaults(func=cmd_staging) + lt = sub.add_parser( + "lint", + help="lint a vault for missing frontmatter, broken links, duplicates, and orphans", + ) + lt.add_argument("vault", nargs="?", help="vault path or @name (defaults via CWD .env, then global config)") + lt.add_argument("--json", action="store_true", help="emit machine-readable JSON") + lt.add_argument("--pretty", action="store_true", help="pretty-print JSON output") + lt.add_argument("--strict", action="store_true", help="exit non-zero on warnings as well as failures") + lt.add_argument( + "--strict-trust", + action="store_true", + help=( + "fail lint on missing trust fields, ledger errors, stale reviews, and " + "score mismatches (default: legacy mode, these are warnings only). " + "Also settable per-vault via OBSIDIAN_TRUST_STRICT=1 in the config." + ), + ) + lt.add_argument( + "--allow-lifecycle", + action="append", + metavar="VALUE", + help="extend the framework lifecycle allowlist (repeatable)", + ) + lt.add_argument( + "--allow-relationship-type", + action="append", + metavar="VALUE", + help="extend the framework relationship-type allowlist (repeatable)", + ) + lt.add_argument( + "--required-trust-field", + action="append", + choices=("base_confidence", "lifecycle", "lifecycle_changed", "updated"), + help="replace default trust-field requiredness (repeatable)", + ) + lt.add_argument( + "--schema-source", + help="authority locator recorded in the lint report (for example, vault/AGENTS.md)", + ) + lt.set_defaults(func=cmd_lint) + mm = sub.add_parser( + "memory", + help="maintain the memory surface: log, index, hot cache, owner profile, todos", + usage="obsidian-wiki memory [ARGS ...] [options]", + description="Maintain the vault's memory: index.md, log.md, hot.md, and the _meta/ tables.", + epilog="""examples: + memory status is the surface current? + memory now the timestamp to write into a page + memory timezone America/Detroit --apply convert every existing stamp to the zone + memory sync INGEST source=x.pdf pages=3 log + index + hot, one lock + memory sync reconcile after writing pages + memory recap --project myapp what a new session should know + memory profile set stack "Python, FastAPI" a durable fact about the owner + memory todo add "Ship the parser" a thread for the next session + memory todo done t1 close it + memory migrate adopt a vault with curated files + memory index --check CI gate; exit 2 on drift + +`sync` and `log` take a VERB then bare key=value pairs. Values with spaces +need quoting: query="how do transformers work".""", + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + mm.add_argument( + "memory_action", + choices=["status", "now", "log", "index", "hot", "recap", "sync", "migrate", "timezone", "profile", "todo"], + help="what to do", + ) + mm.add_argument( + "rest", + nargs="*", + metavar="ARGS", + help="sync/log: VERB then key=value pairs. profile: list|set KEY VALUE|forget KEY. " + "todo: list|add TEXT|done ID|drop ID|prune. timezone: [ZONE]", + ) + mm.add_argument("--vault", help="vault path or @name (defaults via CWD .env, then global config)") + mm.add_argument("--json", action="store_true", help="emit machine-readable JSON") + mm.add_argument("--pretty", action="store_true", help="pretty-print JSON output") + mm_write = mm.add_argument_group("sync / hot") + mm_write.add_argument( + "--takeaways", + help="replace the Key Takeaways section; `-` reads it from stdin", + ) + mm_write.add_argument("--max-words", type=int, help="word cap for hot/recap (default: OBSIDIAN_HOT_MAX_WORDS or 500)") + mm_write.add_argument("--check", action="store_true", help="report drift and exit 2 without writing (CI gate)") + mm_recap = mm.add_argument_group("recap") + mm_recap.add_argument("--project", help="scope to one project's threads and activity") + mm_recap.add_argument("--min-confidence", type=float, default=0.0, help="drop profile facts below this") + mm_profile = mm.add_argument_group("profile set") + mm_profile.add_argument("--confidence", type=float, default=0.6, help="how sure you are, 0..1 (default: 0.6)") + mm_profile.add_argument("--source", default="conversation", help="where the fact came from") + mm_profile.add_argument("--until", metavar="YYYY-MM-DD", help="time-bound fact: drop it from the recap after this date") + mm_todo = mm.add_argument_group("todo") + mm_todo.add_argument("--origin", help="originating page for `todo add`") + mm_todo.add_argument("--stale-days", type=int, default=30, help="staleness threshold (default: 30)") + mm_todo.add_argument("--all", action="store_true", help="include closed items in `todo list`") + mm_now = mm.add_argument_group("now") + mm_now.add_argument("--date", action="store_true", help="print the bare date (YYYY-MM-DD) instead of the timestamp") + mm_tz = mm.add_argument_group("timezone") + mm_tz.add_argument("--skip", action="append", metavar="GLOB", help="leave files matching this vault-relative glob alone; repeatable") + mm_tz.add_argument("--verbose", action="store_true", help="list every file and every stamp in the preview") + mm_other = mm.add_argument_group("migrate / compatibility") + mm_other.add_argument("--apply", action="store_true", help="perform the migration / timezone conversion (default is a preview)") + mm_other.add_argument("--verb", help="older form of the positional VERB") + mm_other.add_argument( + "--field", action="append", metavar="KEY=VALUE", + help="older form of a positional key=value; repeatable", + ) + mm.set_defaults(func=cmd_memory) + return p @@ -683,7 +1679,11 @@ def main(argv: list[str] | None = None) -> int: return 0 # Warn about stale installs on every command except `setup` (which fixes it) # and `info` (which calls _check_stale itself with richer output). - if getattr(args, "command", None) not in ("setup", "info", "file-index", "bot-view", None): + if getattr(args, "command", None) not in ( + "setup", "info", "file-index", "bot-view", "graph-analyse", + "cache-check", "projects-check", "cache-update", "cache-hash", + "lint", "memory", "staging", None, + ): _check_stale() try: return args.func(args) diff --git a/obsidian_wiki/clock.py b/obsidian_wiki/clock.py new file mode 100644 index 0000000..1b914a8 --- /dev/null +++ b/obsidian_wiki/clock.py @@ -0,0 +1,215 @@ +"""The wiki clock: every stamp the memory writer puts in a vault comes from here. + +A vault keeps its timezone in the frontmatter of its ``index.md``:: + + --- + title: Wiki Index + timezone: America/Detroit + --- + +Without the key the writer keeps its original behavior: UTC, in the ``Z`` +spelling (``2026-10-10T18:40:37Z``), and bare dates are UTC calendar days. +With it, every stamp carries the zone's own offset for that instant +(``2026-10-10T14:40:37-04:00``), so daylight-saving transitions are right by +construction, and a bare date (``created:`` on a profile fact, ``touched`` on a +todo) is the calendar day *there* rather than in UTC — the difference that +puts an evening's work on tomorrow's date for anyone west of Greenwich. + +The setting lives in the file it governs, like ``index_folder_hubs``: it +travels with the vault and applies to every writer — CLI, server, Python API — +whatever ``.env`` the caller resolved. ``rebuild_index`` preserves the +frontmatter, so the key survives every regeneration. +""" + +from __future__ import annotations + +import re +from dataclasses import dataclass +from datetime import datetime, timezone, tzinfo +from pathlib import Path +from typing import Optional, Union +from zoneinfo import ZoneInfo, ZoneInfoNotFoundError + +from obsidian_wiki.vault import split_frontmatter + +TIMEZONE_KEY = "timezone" +_TIMEZONE_LINE = re.compile(rf"^{TIMEZONE_KEY}:[ \t]*(?P.*)$", re.MULTILINE) + + +class TimezoneError(ValueError): + """``timezone:`` names a zone this Python cannot resolve.""" + + +def parse_timezone(name: str) -> Optional[tzinfo]: + """IANA name → tzinfo. Empty or ``UTC`` → ``None``, meaning the default. + + ``None`` rather than ``timezone.utc`` so callers can tell "nothing + configured" from a zone, and keep writing the ``Z`` spelling for it. + """ + value = (name or "").strip().strip("\"'") + if not value or value.upper() in {"UTC", "Z"}: + return None + try: + return ZoneInfo(value) + except (ZoneInfoNotFoundError, ValueError, OSError) as exc: + raise TimezoneError( + f"unknown timezone {value!r} in index.md; use an IANA name such as America/Detroit" + ) from exc + + +def timezone_name(vault: Union[str, Path]) -> str: + """The configured zone name, or ``""`` when the vault runs on the default.""" + index = Path(vault) / "index.md" + if not index.is_file(): + return "" + frontmatter, _body = split_frontmatter(index.read_text(encoding="utf-8", errors="replace")) + match = _TIMEZONE_LINE.search(frontmatter) + return match.group("value").strip().strip("\"'") if match else "" + + +def wiki_timezone(vault: Union[str, Path]) -> Optional[tzinfo]: + """The vault's zone from ``index.md``, or ``None`` for the UTC default. + + Raises :class:`TimezoneError` when the key names a zone that does not + exist — a typo must not silently write UTC under a local label. + """ + return parse_timezone(timezone_name(vault)) + + +def set_timezone(vault: Union[str, Path], name: str) -> str: + """Write ``timezone: `` into the frontmatter of the vault's ``index.md``. + + Validates the name first, so a typo never lands in the file. Replaces an + existing key, otherwise inserts it after ``title:``; an ``index.md`` with + no frontmatter gets one. An empty name is a no-op. Returns the stored name. + """ + value = (name or "").strip().strip("\"'") + if not value: + return "" + parse_timezone(value) # raises TimezoneError on an unknown zone + index = Path(vault) / "index.md" + if not index.is_file(): + raise FileNotFoundError(f"no index.md in {vault}; scaffold the vault first") + from obsidian_wiki.memory import atomic_write # lazy: memory imports this module + + text = index.read_text(encoding="utf-8") + frontmatter, body = split_frontmatter(text) + line = f"{TIMEZONE_KEY}: {value}" + if _TIMEZONE_LINE.search(frontmatter): + frontmatter = _TIMEZONE_LINE.sub(line, frontmatter, count=1) + elif frontmatter.strip(): + lines = frontmatter.rstrip("\n").split("\n") + at = next((i + 1 for i, l in enumerate(lines) if l.startswith("title:")), len(lines)) + lines.insert(at, line) + frontmatter = "\n".join(lines) + else: + frontmatter = f"title: Wiki Index\n{line}" + if text.startswith("---"): + rest = text.split("---", 2)[2] if text.count("---") >= 2 else body + atomic_write(index, f"---\n{frontmatter.strip()}\n---{rest}") + else: + atomic_write(index, f"---\n{frontmatter.strip()}\n---\n\n{text}") + return value + + +def _moment(now: Optional[datetime]) -> datetime: + moment = now or datetime.now(timezone.utc) + if moment.tzinfo is None or moment.utcoffset() is None: + raise ValueError("clock needs an aware datetime; attach a timezone first") + return moment + + +def stamp(tz: Optional[tzinfo] = None, *, now: Optional[datetime] = None) -> str: + """Full timestamp to the second: ``…Z`` on the default, ``…-04:00`` in a zone.""" + moment = _moment(now) + if tz is None: + return moment.astimezone(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ") + return moment.astimezone(tz).isoformat(timespec="seconds") + + +def today(tz: Optional[tzinfo] = None, *, now: Optional[datetime] = None) -> str: + """Bare calendar date in the zone (UTC on the default).""" + return _moment(now).astimezone(tz or timezone.utc).date().isoformat() + + +def compact(tz: Optional[tzinfo] = None, *, now: Optional[datetime] = None) -> str: + """Filesystem-safe form for backup directory names: ``20261010T184037Z``.""" + moment = _moment(now) + if tz is None: + return moment.astimezone(timezone.utc).strftime("%Y%m%dT%H%M%SZ") + return moment.astimezone(tz).strftime("%Y%m%dT%H%M%S%z") + + +# -------------------------------------------------------------------------- +# converting the stamps already in a vault +# -------------------------------------------------------------------------- + +#: A full ISO-8601 timestamp with a zone: ``2026-10-10T18:40:37Z``, +#: ``2026-10-10T14:40:37-04:00``, ``2026-10-10T14:40:37.123-0400``. Bare dates, +#: naive stamps (no zone) and the compact backup-directory form +#: (``20261010T184037Z``) deliberately do not match: the first two cannot be +#: converted without guessing, the last names a directory that exists. +STAMP_RE = re.compile( + r"(?\d{4}-\d{2}-\d{2})T(?P