gitea-admin/obsidian-wiki
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(mcp): harden remote assets and job execution

Browse Source 
Gate source-file downloads and verify registry paths and content hashes. Stream multipart uploads with bounded limits, supervise OpenCode jobs with explicit result markers and timeouts, and preserve managed endpoint security settings. Document the trusted-client boundary and add focused regression coverage.

Validation: PYTHONPATH=<staged-tree> python -m unittest discover -s tests (182 tests)
This commit is contained in:
Codex
2026-07-09 21:44:48 +00:00
parent d47bb2c549
commit 7eb9b987ab
16 changed files with 3709 additions and 500 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+10 -8
View File
@@ -391,7 +391,7 @@ Configure your agent to run `obsidian-wiki-mcp` as a stdio MCP server. The serve
- `get_mcp_capabilities` — tells remote agents which tool groups are enabled and which tool to use first.
- `asset://source/sha256/{digest}` — binary resource for an allowed archived file.
By default the MCP layer is stricter than local CLI lookup: it only serves engineering files such as drawings, 3D/STEP models, specifications, product parameters, and test reports. Customer-shareable engineering deliverables, such as PDF customer drawings, are allowed for source-file lookup when the active profile already allows technical engineering data. It blocks raw orders, prices, email correspondence, complaint evidence, claims, personal data, and files missing from `_meta/source-registry.jsonl`. Use `OBSIDIAN_MCP_BOT_PROFILE`, `OBSIDIAN_MCP_MAX_FILE_BYTES`, and `OBSIDIAN_MCP_EXPOSE_PATHS` to tune the runtime behavior.
By default the MCP layer is stricter than local CLI lookup: it only returns metadata for engineering files such as drawings, 3D/STEP models, specifications, product parameters, and test reports. Customer-shareable engineering deliverables, such as PDF customer drawings, are eligible for lookup when the active profile already allows technical engineering data. It blocks raw orders, prices, email correspondence, complaint evidence, claims, personal data, and files missing from `_meta/source-registry.jsonl`. Downloading the underlying `asset://source/...` bytes is a separate opt-in and defaults off for new managed installs; set `OBSIDIAN_MCP_ALLOW_SOURCE_FILES=true` (or pass `--allow-source-files`) only for trusted clients. Use `OBSIDIAN_MCP_BOT_PROFILE`, `OBSIDIAN_MCP_MAX_FILE_BYTES`, and `OBSIDIAN_MCP_EXPOSE_PATHS` to tune the remaining runtime behavior.
Optional: run the same MCP server over Streamable HTTP for a remote MCP client:
@@ -428,7 +428,7 @@ For long-running access from another machine, use the managed user service:
scripts/obsidian-wiki-ops.py mcp start
```
`scripts/obsidian-wiki-ops.py mcp start` creates `.runtime/obsidian-wiki-mcp.env` with `0600` permissions if it does not exist, writes `~/.config/systemd/user/obsidian-wiki-mcp.service`, starts the service, and prints the remote URL, `Authorization` header, enabled tool groups, config file path, and client JSON to copy into another agent. The user service restarts on failure and sets `MemoryMax=2G`. When `ingest` is enabled, it also prints the `/mcp-ingest/uploads` file upload endpoint. It serves the configured `OBSIDIAN_VAULT_PATH` by default; pass `--vault /path/to/vault` to override. The service defaults to `0.0.0.0:8765/mcp` because it is intended for remote clients, but it still requires the bearer token. Prefer SSH tunnel, Tailscale, WireGuard, or another trusted network boundary instead of directly exposing the port to the public internet.
`scripts/obsidian-wiki-ops.py mcp start` creates `.runtime/obsidian-wiki-mcp.env` with `0600` permissions if it does not exist, writes `~/.config/systemd/user/obsidian-wiki-mcp.service`, starts the service, and prints the remote URL, `Authorization` header, enabled tool groups, config file path, and client JSON to copy into another agent. The managed endpoint values (bind host/port/path, public URLs, and transport allowlists) live in that env file and are preserved across later starts and token rotations. The user service restarts on failure and sets `MemoryMax=2G`. When `ingest` is enabled, it also prints the `/mcp-ingest/uploads` file upload endpoint. It serves the configured `OBSIDIAN_VAULT_PATH` by default; pass `--vault /path/to/vault` to override. The service defaults to `0.0.0.0:8765/mcp` because it is intended for remote clients, but it still requires the bearer token. Managed installs derive `OBSIDIAN_MCP_ALLOWED_HOSTS` and `OBSIDIAN_MCP_ALLOWED_ORIGINS` from their bind/public URLs; review them when using custom DNS, a tunnel, or a reverse proxy. Prefer SSH tunnel, Tailscale, WireGuard, or another trusted network boundary instead of directly exposing the port to the public internet.
Customize the first startup if needed:
@@ -471,7 +471,7 @@ This adds write-capable MCP tools to the same `wiki-file-assets` server and bear
- `start_ingest_job` — queues a local `opencode run --auto --format json` job with a fixed `/wiki-ingest` prompt for the completed uploads.
- `get_ingest_job_status`, `get_ingest_job_result`, `cancel_ingest_job` — inspect or cancel the local job.
The managed service config sets the OpenCode model server-side. By default ingest uses `OBSIDIAN_MCP_INGEST_OPENCODE_MODEL=openai/gpt-5.5` and `OBSIDIAN_MCP_INGEST_OPENCODE_VARIANT=high` for stronger file processing. It also writes `OBSIDIAN_MCP_PUBLIC_BASE_URL` from the detected remote address so ticket responses can include a complete upload URL; set it manually when the service is behind a tunnel, reverse proxy, or custom DNS name. Change these values in `.runtime/obsidian-wiki-mcp.env` on the server if needed; remote MCP clients do not choose the model per call.
The managed service config sets the OpenCode model server-side. By default ingest uses `OBSIDIAN_MCP_INGEST_OPENCODE_MODEL=openai/gpt-5.5`, `OBSIDIAN_MCP_INGEST_OPENCODE_VARIANT=high`, and an `OBSIDIAN_MCP_INGEST_JOB_TIMEOUT_SECONDS=1800` deadline. It also writes `OBSIDIAN_MCP_PUBLIC_BASE_URL` from the detected remote address so ticket responses can include a complete upload URL; set it manually when the service is behind a tunnel, reverse proxy, or custom DNS name. If the upload endpoint is exposed through a different public URL than `/mcp`, set `OBSIDIAN_MCP_INGEST_PUBLIC_UPLOAD_URL` explicitly and it will override the derived upload URL in capabilities, upload instructions, tickets, and `show-config`. Change these values in `.runtime/obsidian-wiki-mcp.env` on the server if needed; remote MCP clients do not choose the model per call.
For short text thoughts, notes, findings, or pasted context, call `ingest_text_note(text, title?, project?, doc_type_hint?, note?, start_job?)`. It uses `doc_type_hint="thought"` by default and returns the created upload plus the queued job when `start_job=true`.
@@ -486,11 +486,13 @@ curl -H "X-Obsidian-Wiki-Upload-Token: <upload-token>" \
<upload_url>
```
The endpoint accepts repeated `files` fields plus optional `project`, `doc_type_hint`, `note`, and `start_job` fields. It still accepts the same bearer token as `/mcp` for local/admin compatibility, but remote agents should prefer single-use upload tickets. Upload tickets are consumed only by authorized `POST` uploads, not by accidental `GET`/wrong-method requests. The upload route enforces configured `--allowed-host` and `--allowed-origin` checks, and rejects oversized request bodies with HTTP `413` before parsing multipart content. Base64 file upload is intentionally not exposed as an MCP tool because large JSON arguments are fragile in many agent tool-call implementations. Duplicate detection is exact sha256 matching only; it does not replace human review for near-duplicate or renamed source files.
The endpoint accepts repeated `files` fields plus optional `project`, `doc_type_hint`, `note`, and `start_job` fields. Multipart files are parsed as streams into spooled storage instead of buffering the whole request in memory, and `OBSIDIAN_MCP_INGEST_MAX_FILE_BYTES` is enforced while each file part is being parsed. `OBSIDIAN_MCP_INGEST_MAX_FILES_PER_JOB` limits the count. With `OBSIDIAN_MCP_INGEST_MAX_REQUEST_BYTES=0`, the route derives its total request-body cap from per-file size times maximum file count plus multipart overhead; a positive value adds a stricter total cap. The stricter upload-ticket limits still apply. Ticket-bound `project`, `doc_type_hint`, and `note` values cannot be replaced by form data, and a ticket created with `start_job=false` cannot be escalated to start a job by sending `start_job=true` in the form. Oversized requests return HTTP `413`. The endpoint still accepts the same bearer token as `/mcp` for local/admin compatibility, but remote agents should prefer single-use upload tickets. Upload tickets are consumed by authorized `POST` attempts, including malformed multipart or oversized uploads, but not by accidental `GET`/wrong-method requests. When an upload-ticket POST fails after the ticket is consumed, the JSON error includes `ticket_consumed: true` and `retry_action: "create_new_upload_ticket"`, so the remote agent should request a fresh ticket before retrying. The upload route enforces configured `--allowed-host` and `--allowed-origin` checks. Base64 file upload is intentionally not exposed as an MCP tool because large JSON arguments are fragile in many agent tool-call implementations. Duplicate detection is exact sha256 matching only; it does not replace human review for near-duplicate or renamed source files.
Run `check_ingest_opencode_readiness` before the first remote upload. Use `run_smoke_test: true` when you want OpenCode itself to prove it can load the project `wiki-ingest` skill; otherwise the static preflight is also run automatically before every `start_ingest_job`.
Uploads and text notes land under `.runtime/mcp-ingest/uploads/`, outside the Obsidian vault, and are treated only as temporary source staging. Job state and logs live under `.runtime/mcp-ingest/jobs/`. Remote upload and job responses expose upload/job ids, hashes, states, and log tails, but omit local runtime file paths; the internal job record still keeps paths so the fixed local OpenCode prompt can ingest only the uploaded files. `get_mcp_capabilities` includes `server_version`, `capabilities_schema_version`, `tools_version`, file-index freshness, ingest queue state, and known limitations so remote agents can detect stale client instructions. The MCP server refuses to enable ingest if `OBSIDIAN_MCP_INGEST_RUNTIME_ROOT` points inside the vault; let `wiki-ingest` decide how to archive sources into `_sources/archive/`, update registries, and rebuild the file index. The ingest prompt is generated by the MCP service, requires OpenCode to load `wiki-ingest`, and restricts OpenCode to the uploaded file list; clients cannot pass arbitrary shell commands through this interface. Because enabling ingest gives the remote token write access to the vault through local OpenCode, keep it on a trusted network and rotate the token if it has been shared too broadly.
Uploads and text notes land under `.runtime/mcp-ingest/uploads/`, outside the Obsidian vault, and are treated only as temporary source staging. Job state and logs live under `.runtime/mcp-ingest/jobs/`. Remote upload and job responses expose upload/job ids, hashes, states, and bounded result text, but omit local runtime file paths; the internal job record still keeps paths so the fixed local OpenCode prompt can ingest only the uploaded files. Completion is accepted only from the final JSON text event containing `MCP_INGEST_OK`; `MCP_INGEST_FAILED`, a missing marker, a nonzero exit, timeout, or cancellation marks the job failed/cancelled. `get_mcp_capabilities` includes `server_version`, `schema_version`/`capabilities_schema_version`, `tools_version`, file-index freshness, ingest queue state, and known limitations so remote agents can detect stale client instructions. Its `recommended_first_steps` only names currently enabled tool groups. The MCP server refuses to enable ingest if `OBSIDIAN_MCP_INGEST_RUNTIME_ROOT` points inside the vault; let `wiki-ingest` decide how to archive sources into `_sources/archive/`, update registries, and rebuild the file index.
Ingest and query launch OpenCode with a fixed prompt and `--auto`, but the MCP service does not enforce a shell sandbox and prompt restrictions are not a security boundary. There is no direct arbitrary-prompt or arbitrary-shell MCP tool, yet a client able to influence vault/upload content must still be trusted. Keep these tool groups behind a trusted network boundary and rotate the token if it has been shared too broadly.
To let a remote MCP client ask synthesized questions against the compiled wiki, enable the `query` tool group:
@@ -504,12 +506,12 @@ This adds read-oriented OpenCode-backed query tools to the same `wiki-file-asset
- `check_query_opencode_readiness` — verifies the local OpenCode binary, required run flags, project `wiki-query`/`llm-wiki` skills, and optionally runs a no-write smoke test.
- `ask_wiki` — preferred one-call query path for remote agents. It starts the local job, polls briefly, and returns either a structured answer or a `job_id` to continue polling.
- `start_wiki_query` — queues a local `opencode run --auto --format default` job with a fixed `/wiki-query` prompt. Parameters: `query`, `mode` (`normal` or `index_only`), `public_only` (default `false`), and optional `note`.
- `start_wiki_query` — queues a local `opencode run --auto --format json` job with a fixed `/wiki-query` prompt. Parameters: `query`, `mode` (`normal` or `index_only`), `public_only` (default `false`), and optional `note`.
- `get_wiki_query_status`, `get_wiki_query_result`, `cancel_wiki_query` — inspect, retrieve, or cancel the local query job.
The managed service config sets query to `OBSIDIAN_MCP_QUERY_OPENCODE_MODEL=openai/gpt-5.5` and `OBSIDIAN_MCP_QUERY_OPENCODE_VARIANT=minimal` by default, prioritizing response speed over maximum reasoning depth. Tune the server env file if a different balance is needed.
The managed service config sets query to `OBSIDIAN_MCP_QUERY_OPENCODE_MODEL=openai/gpt-5.5`, `OBSIDIAN_MCP_QUERY_OPENCODE_VARIANT=minimal`, and `OBSIDIAN_MCP_QUERY_JOB_TIMEOUT_SECONDS=300` by default, prioritizing response speed over maximum reasoning depth. Tune the server env file if a different balance is needed.
`public_only` defaults to `false`, so the answer may use the same full vault visibility that a local operator sees. Set `public_only=true` when the remote answer must exclude `visibility/internal` and `visibility/pii` pages. Job state and logs live under `.runtime/mcp-query/jobs/`, outside the vault; remote query status/result responses include log tails but omit local log file paths. Query results keep the legacy `answer` field and also include `answer_markdown`, `citations`, `used_wiki`, `used_business_facts`, `data_freshness`, `warnings`, and `confidence` so agents can parse the response without scraping the log tail. A normal `wiki-query` log entry is allowed. The generated prompt forbids ingest/update/rebuild workflows, writes, sync/config/timer operations, and arbitrary shell execution; it only allows the `obsidian-wiki business schema/template/sql` read-only commands needed by the wiki-query business facts route. Hybrid business + wiki questions may run a wiki context track and a business facts track, using parallel subagents only when the host supports them and otherwise running sequentially. When `public_only=true`, business facts remain disabled.
`public_only` defaults to `false`, so the answer may use the same full vault visibility that a local operator sees. Set `public_only=true` when the remote answer must exclude `visibility/internal` and `visibility/pii` pages. Job state and logs live under `.runtime/mcp-query/jobs/`, outside the vault; remote query status/result responses expose bounded, path-redacted text and omit local log paths. The final JSON text event must contain `MCP_QUERY_OK`; `MCP_QUERY_FAILED`, a missing marker, a nonzero exit, timeout, or cancellation cannot be reported as a successful answer. Query results keep the legacy `answer` field and also include `answer_markdown`, `citations`, `used_wiki`, `used_business_facts`, `data_freshness`, `warnings`, and `confidence` so agents can parse the response without scraping raw logs. A normal `wiki-query` log entry is allowed. The generated prompt instructs the agent to avoid ingest/update/rebuild workflows, writes, sync/config/timer operations, and arbitrary shell execution; it only permits the `obsidian-wiki business schema/template/sql` read-only commands needed by the wiki-query business facts route, but these prompt restrictions do not replace the trusted-client boundary described above. Hybrid business + wiki questions may run a wiki context track and a business facts track, using parallel subagents only when the host supports them and otherwise running sequentially. When `public_only=true`, business facts remain disabled.
Clean old MCP runtime staging data with the default 30-day retention window: