fix(mcp): harden remote service boundaries
Harden the HTTP ingest upload path by validating host/origin and method before ticket consumption, enforcing request body limits, and returning 413 for oversized uploads. Hide local runtime/log paths from default remote responses while preserving internal paths for local OpenCode prompts. Add conservative runtime cleanup for tickets, uploads, jobs, and logs, plus MemoryMax on the managed systemd unit. Validation: python3 -m unittest tests/test_mcp_server_remote.py tests/test_mcp_ingest.py tests/test_mcp_query.py tests/test_mcp_service.py -v; python3 -m unittest discover -v; python3 -m compileall obsidian_wiki tests scripts; git diff --check. Restarted obsidian-wiki-mcp.service and confirmed ActiveState=active, SubState=running, MemoryMax=2147483648, Restart=on-failure.
This commit is contained in:
@@ -130,8 +130,8 @@ See `wiki-query` and `wiki-export` skills for how the filter is applied.
|
||||
- **Keep product lookup fields explicit.** Carry forward `bjd_part_numbers`, `competitor_part_numbers`, `series`, `pitch`, `pin_count`, `product_lines`, `customers`, `competitors`, `standards`, `business_categories`, `related_pages`, and `catalog_paths` whenever they are known.
|
||||
- **Merge, do not overwrite.** Automatic identifier extraction may add terms, but it must not discard curated registry metadata. Registry values are the durable source for future index rebuilds and remote MCP lookup.
|
||||
- **Prefer precise identifiers in search ranking.** Exact or prefix matches in structured part-number and series fields should rank above incidental filename substrings such as timestamps or archive hashes.
|
||||
- **Keep remote ingest gated.** MCP ingest tools are write-capable and must remain disabled unless the service config explicitly enables the `ingest` tool group via `OBSIDIAN_MCP_ENABLED_TOOLS=file-assets,ingest`, `mcp_service.sh tools enable ingest`, or `mcp_service.sh start --enable-ingest`. Uploaded files and direct text notes must stage outside the vault under the MCP runtime root; `wiki-ingest` is responsible for accepting, archiving, and indexing sources inside the vault. Use `ingest_text_note` only for short remote-agent thoughts/notes; it must wrap the text as a temporary markdown upload and then use the fixed `/wiki-ingest` job path. Use the HTTP multipart endpoint `/mcp-ingest/uploads` for file transfer; if the remote agent cannot read the main MCP bearer token, it must call `create_ingest_upload_ticket` and send the returned single-use `X-Obsidian-Wiki-Upload-Token` header. Do not expose base64 or chunked upload tools in the MCP tool surface. They may launch local OpenCode only through the fixed `/wiki-ingest` prompt; never expose arbitrary prompt or shell execution through MCP. Remote ingest must keep the OpenCode readiness preflight in front of job launch, and `check_ingest_opencode_readiness(run_smoke_test=true)` should stay available for operators to verify that OpenCode can load the project `wiki-ingest` skill before accepting uploads.
|
||||
- **Keep remote query gated.** MCP query tools must remain disabled unless the service config explicitly enables the `query` tool group via `OBSIDIAN_MCP_ENABLED_TOOLS=file-assets,query`, `mcp_service.sh tools enable query`, or `mcp_service.sh start --enable-query`. They may launch local OpenCode only through the fixed `/wiki-query` prompt, with no arbitrary prompt or shell execution. `public_only` defaults to false, meaning remote answers may use the full local vault visibility; enable the tool group only for trusted clients and set `public_only=true` per call for filtered user-facing answers.
|
||||
- **Keep remote ingest gated.** MCP ingest tools are write-capable and must remain disabled unless the service config explicitly enables the `ingest` tool group via `OBSIDIAN_MCP_ENABLED_TOOLS=file-assets,ingest`, `mcp_service.sh tools enable ingest`, or `mcp_service.sh start --enable-ingest`. Uploaded files and direct text notes must stage outside the vault under the MCP runtime root; `wiki-ingest` is responsible for accepting, archiving, and indexing sources inside the vault. Use `ingest_text_note` only for short remote-agent thoughts/notes; it must wrap the text as a temporary markdown upload and then use the fixed `/wiki-ingest` job path. Use the HTTP multipart endpoint `/mcp-ingest/uploads` for file transfer; if the remote agent cannot read the main MCP bearer token, it must call `create_ingest_upload_ticket` and send the returned single-use `X-Obsidian-Wiki-Upload-Token` header. Upload tickets are consumed only by authorized `POST` uploads, the upload endpoint must enforce configured host/origin restrictions, and oversized request bodies must fail with HTTP `413`. Do not expose base64 or chunked upload tools in the MCP tool surface. They may launch local OpenCode only through the fixed `/wiki-ingest` prompt; never expose arbitrary prompt or shell execution through MCP. Remote ingest responses must not expose local runtime paths; internal job records may keep paths for the fixed local prompt. Remote ingest must keep the OpenCode readiness preflight in front of job launch, and `check_ingest_opencode_readiness(run_smoke_test=true)` should stay available for operators to verify that OpenCode can load the project `wiki-ingest` skill before accepting uploads.
|
||||
- **Keep remote query gated.** MCP query tools must remain disabled unless the service config explicitly enables the `query` tool group via `OBSIDIAN_MCP_ENABLED_TOOLS=file-assets,query`, `mcp_service.sh tools enable query`, or `mcp_service.sh start --enable-query`. They may launch local OpenCode only through the fixed `/wiki-query` prompt, with no arbitrary prompt or shell execution. `public_only` defaults to false, meaning remote answers may use the full local vault visibility; enable the tool group only for trusted clients and set `public_only=true` per call for filtered user-facing answers. Remote query responses may expose log tails but must not expose local log file paths.
|
||||
- **Keep OpenCode model choice server-side.** MCP clients must not pass arbitrary model names through ingest/query tools. Configure local defaults in `.runtime/obsidian-wiki-mcp.env`: ingest uses `OBSIDIAN_MCP_INGEST_OPENCODE_MODEL=openai/gpt-5.5` with `OBSIDIAN_MCP_INGEST_OPENCODE_VARIANT=high`; query uses `OBSIDIAN_MCP_QUERY_OPENCODE_MODEL=openai/gpt-5.5` with `OBSIDIAN_MCP_QUERY_OPENCODE_VARIANT=minimal`.
|
||||
|
||||
## Core Principles
|
||||
|
||||
@@ -427,7 +427,7 @@ For long-running access from another machine, use the managed user service:
|
||||
scripts/obsidian-wiki-ops.py mcp start
|
||||
```
|
||||
|
||||
`scripts/obsidian-wiki-ops.py mcp start` creates `.runtime/obsidian-wiki-mcp.env` with `0600` permissions if it does not exist, writes `~/.config/systemd/user/obsidian-wiki-mcp.service`, starts the service, and prints the remote URL, `Authorization` header, enabled tool groups, config file path, and client JSON to copy into another agent. When `ingest` is enabled, it also prints the `/mcp-ingest/uploads` file upload endpoint. It serves the configured `OBSIDIAN_VAULT_PATH` by default; pass `--vault /path/to/vault` to override. The service defaults to `0.0.0.0:8765/mcp` because it is intended for remote clients, but it still requires the bearer token. Prefer SSH tunnel, Tailscale, WireGuard, or another trusted network boundary instead of directly exposing the port to the public internet.
|
||||
`scripts/obsidian-wiki-ops.py mcp start` creates `.runtime/obsidian-wiki-mcp.env` with `0600` permissions if it does not exist, writes `~/.config/systemd/user/obsidian-wiki-mcp.service`, starts the service, and prints the remote URL, `Authorization` header, enabled tool groups, config file path, and client JSON to copy into another agent. The user service restarts on failure and sets `MemoryMax=2G`. When `ingest` is enabled, it also prints the `/mcp-ingest/uploads` file upload endpoint. It serves the configured `OBSIDIAN_VAULT_PATH` by default; pass `--vault /path/to/vault` to override. The service defaults to `0.0.0.0:8765/mcp` because it is intended for remote clients, but it still requires the bearer token. Prefer SSH tunnel, Tailscale, WireGuard, or another trusted network boundary instead of directly exposing the port to the public internet.
|
||||
|
||||
Customize the first startup if needed:
|
||||
|
||||
@@ -483,11 +483,11 @@ curl -H "X-Obsidian-Wiki-Upload-Token: <upload-token>" \
|
||||
http://127.0.0.1:8765/mcp-ingest/uploads
|
||||
```
|
||||
|
||||
The endpoint accepts repeated `files` fields plus optional `project`, `doc_type_hint`, `note`, and `start_job` fields. It accepts either the same bearer token as `/mcp` or the single-use upload ticket header. Base64 file upload is intentionally not exposed as an MCP tool because large JSON arguments are fragile in many agent tool-call implementations.
|
||||
The endpoint accepts repeated `files` fields plus optional `project`, `doc_type_hint`, `note`, and `start_job` fields. It accepts either the same bearer token as `/mcp` or the single-use upload ticket header. Upload tickets are consumed only by authorized `POST` uploads, not by accidental `GET`/wrong-method requests. The upload route enforces configured `--allowed-host` and `--allowed-origin` checks, and rejects oversized request bodies with HTTP `413` before parsing multipart content. Base64 file upload is intentionally not exposed as an MCP tool because large JSON arguments are fragile in many agent tool-call implementations.
|
||||
|
||||
Run `check_ingest_opencode_readiness` before the first remote upload. Use `run_smoke_test: true` when you want OpenCode itself to prove it can load the project `wiki-ingest` skill; otherwise the static preflight is also run automatically before every `start_ingest_job`.
|
||||
|
||||
Uploads and text notes land under `.runtime/mcp-ingest/uploads/`, outside the Obsidian vault, and are treated only as temporary source staging. Job state and logs live under `.runtime/mcp-ingest/jobs/`. The MCP server refuses to enable ingest if `OBSIDIAN_MCP_INGEST_RUNTIME_ROOT` points inside the vault; let `wiki-ingest` decide how to archive sources into `_sources/archive/`, update registries, and rebuild the file index. The ingest prompt is generated by the MCP service, requires OpenCode to load `wiki-ingest`, and restricts OpenCode to the uploaded file list; clients cannot pass arbitrary shell commands through this interface. Because enabling ingest gives the remote token write access to the vault through local OpenCode, keep it on a trusted network and rotate the token if it has been shared too broadly.
|
||||
Uploads and text notes land under `.runtime/mcp-ingest/uploads/`, outside the Obsidian vault, and are treated only as temporary source staging. Job state and logs live under `.runtime/mcp-ingest/jobs/`. Remote upload and job responses expose upload/job ids, hashes, states, and log tails, but omit local runtime file paths; the internal job record still keeps paths so the fixed local OpenCode prompt can ingest only the uploaded files. The MCP server refuses to enable ingest if `OBSIDIAN_MCP_INGEST_RUNTIME_ROOT` points inside the vault; let `wiki-ingest` decide how to archive sources into `_sources/archive/`, update registries, and rebuild the file index. The ingest prompt is generated by the MCP service, requires OpenCode to load `wiki-ingest`, and restricts OpenCode to the uploaded file list; clients cannot pass arbitrary shell commands through this interface. Because enabling ingest gives the remote token write access to the vault through local OpenCode, keep it on a trusted network and rotate the token if it has been shared too broadly.
|
||||
|
||||
To let a remote MCP client ask synthesized questions against the compiled wiki, enable the `query` tool group:
|
||||
|
||||
@@ -505,7 +505,16 @@ This adds read-oriented OpenCode-backed query tools to the same `wiki-file-asset
|
||||
|
||||
The managed service config sets query to `OBSIDIAN_MCP_QUERY_OPENCODE_MODEL=openai/gpt-5.5` and `OBSIDIAN_MCP_QUERY_OPENCODE_VARIANT=minimal` by default, prioritizing response speed over maximum reasoning depth. Tune the server env file if a different balance is needed.
|
||||
|
||||
`public_only` defaults to `false`, so the answer may use the same full vault visibility that a local operator sees. Set `public_only=true` when the remote answer must exclude `visibility/internal` and `visibility/pii` pages. Job state and logs live under `.runtime/mcp-query/jobs/`, outside the vault. A normal `wiki-query` log entry is allowed. The generated prompt forbids ingest/update/rebuild workflows, writes, sync/config/timer operations, and arbitrary shell execution; it only allows the `obsidian-wiki business schema/template/sql` read-only commands needed by the wiki-query business facts route. Hybrid business + wiki questions may run a wiki context track and a business facts track, using parallel subagents only when the host supports them and otherwise running sequentially. When `public_only=true`, business facts remain disabled.
|
||||
`public_only` defaults to `false`, so the answer may use the same full vault visibility that a local operator sees. Set `public_only=true` when the remote answer must exclude `visibility/internal` and `visibility/pii` pages. Job state and logs live under `.runtime/mcp-query/jobs/`, outside the vault; remote query status/result responses include log tails but omit local log file paths. A normal `wiki-query` log entry is allowed. The generated prompt forbids ingest/update/rebuild workflows, writes, sync/config/timer operations, and arbitrary shell execution; it only allows the `obsidian-wiki business schema/template/sql` read-only commands needed by the wiki-query business facts route. Hybrid business + wiki questions may run a wiki context track and a business facts track, using parallel subagents only when the host supports them and otherwise running sequentially. When `public_only=true`, business facts remain disabled.
|
||||
|
||||
Clean old MCP runtime staging data with the default 30-day retention window:
|
||||
|
||||
```bash
|
||||
scripts/obsidian-wiki-ops.py mcp cleanup
|
||||
scripts/obsidian-wiki-ops.py mcp cleanup --max-age-days 14
|
||||
```
|
||||
|
||||
Cleanup removes expired upload tickets and old completed/cancelled/failed jobs, logs, and upload staging directories that are not referenced by active jobs.
|
||||
|
||||
Rotate the token later with:
|
||||
|
||||
@@ -520,6 +529,7 @@ scripts/obsidian-wiki-ops.py mcp show-config
|
||||
scripts/obsidian-wiki-ops.py mcp tools list
|
||||
scripts/obsidian-wiki-ops.py mcp tools enable ingest
|
||||
scripts/obsidian-wiki-ops.py mcp tools disable ingest
|
||||
scripts/obsidian-wiki-ops.py mcp cleanup
|
||||
scripts/obsidian-wiki-ops.py mcp stop
|
||||
scripts/obsidian-wiki-ops.py mcp uninstall
|
||||
scripts/obsidian-wiki-ops.py mcp status
|
||||
|
||||
@@ -9,6 +9,7 @@ import json
|
||||
import os
|
||||
import re
|
||||
import secrets
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
import uuid
|
||||
@@ -33,6 +34,7 @@ SHA256_RE = re.compile(r"^(?:sha256:)?([0-9a-fA-F]{64})$")
|
||||
OPENCODE_REQUIRED_RUN_FLAGS = ("--format", "--auto", "--dir", "--title")
|
||||
OPENCODE_SMOKE_SUCCESS = "WIKI_INGEST_SKILL_LOADED"
|
||||
OPENCODE_SMOKE_FAILURE = "SKILL_LOAD_FAILED"
|
||||
TERMINAL_JOB_STATES = {"succeeded", "failed", "cancelled"}
|
||||
|
||||
|
||||
class IngestError(RuntimeError):
|
||||
@@ -269,7 +271,7 @@ class IngestManager:
|
||||
"job_id": job_id,
|
||||
"state": "queued",
|
||||
"upload_ids": upload_ids,
|
||||
"files": [self._public_upload(upload) for upload in uploads],
|
||||
"files": [self._public_upload(upload, expose_path=True) for upload in uploads],
|
||||
"note": self._safe_optional_text(note),
|
||||
"created_at": utc_now(),
|
||||
"updated_at": utc_now(),
|
||||
@@ -366,6 +368,65 @@ class IngestManager:
|
||||
self._processes.pop(job_id, None)
|
||||
self._start_next_job_locked()
|
||||
|
||||
def cleanup_runtime(self, *, max_age_days: int = 30) -> dict[str, Any]:
|
||||
if max_age_days <= 0:
|
||||
raise IngestError("max_age_days must be positive")
|
||||
cutoff = time.time() - (max_age_days * 24 * 60 * 60)
|
||||
result = {
|
||||
"runtime_root": str(self.runtime_root),
|
||||
"max_age_days": max_age_days,
|
||||
"expired_tickets_deleted": 0,
|
||||
"old_uploads_deleted": 0,
|
||||
"old_jobs_deleted": 0,
|
||||
}
|
||||
with self._lock:
|
||||
active_upload_ids = {
|
||||
str(upload_id)
|
||||
for job in self._iter_jobs()
|
||||
if job.get("state") not in TERMINAL_JOB_STATES
|
||||
for upload_id in job.get("upload_ids", [])
|
||||
}
|
||||
now = time.time()
|
||||
for ticket_path in self.upload_ticket_root.glob("*.json"):
|
||||
try:
|
||||
meta = self._read_json(ticket_path)
|
||||
expired = now > float(meta.get("expires_epoch", 0))
|
||||
except (OSError, ValueError, json.JSONDecodeError):
|
||||
expired = ticket_path.stat().st_mtime < cutoff
|
||||
if expired:
|
||||
ticket_path.unlink(missing_ok=True)
|
||||
result["expired_tickets_deleted"] += 1
|
||||
|
||||
seen_upload_dirs: set[Path] = set()
|
||||
for meta_path in self.upload_root.glob("*/*/upload.json"):
|
||||
upload_dir = meta_path.parent
|
||||
if upload_dir in seen_upload_dirs:
|
||||
continue
|
||||
seen_upload_dirs.add(upload_dir)
|
||||
try:
|
||||
meta = self._read_json(meta_path)
|
||||
except (OSError, json.JSONDecodeError):
|
||||
continue
|
||||
if str(meta.get("upload_id", "")) in active_upload_ids:
|
||||
continue
|
||||
if self._tree_mtime(upload_dir) >= cutoff:
|
||||
continue
|
||||
shutil.rmtree(upload_dir, ignore_errors=True)
|
||||
result["old_uploads_deleted"] += 1
|
||||
|
||||
for job in self._iter_jobs():
|
||||
if job.get("state") not in TERMINAL_JOB_STATES:
|
||||
continue
|
||||
job_id = str(job.get("job_id", ""))
|
||||
if not re.fullmatch(r"[0-9a-f]{32}", job_id):
|
||||
continue
|
||||
job_dir = self.jobs_root / job_id
|
||||
if self._tree_mtime(job_dir) >= cutoff:
|
||||
continue
|
||||
shutil.rmtree(job_dir, ignore_errors=True)
|
||||
result["old_jobs_deleted"] += 1
|
||||
return result
|
||||
|
||||
def _start_next_job_locked(self) -> None:
|
||||
if any(job.get("state") == "running" for job in self._iter_jobs()):
|
||||
return
|
||||
@@ -505,20 +566,25 @@ class IngestManager:
|
||||
job["updated_at"] = utc_now()
|
||||
self._write_job(job)
|
||||
|
||||
def _public_upload(self, meta: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
def _public_upload(self, meta: dict[str, Any], *, expose_path: bool = False) -> dict[str, Any]:
|
||||
digest = str(meta["sha256"])
|
||||
if not digest.startswith("sha256:"):
|
||||
digest = "sha256:" + digest
|
||||
upload = {
|
||||
"upload_id": meta["upload_id"],
|
||||
"state": meta["state"],
|
||||
"file_name": meta["file_name"],
|
||||
"size_bytes": meta["size_bytes"],
|
||||
"received_bytes": meta["received_bytes"],
|
||||
"sha256": "sha256:" + meta["sha256"],
|
||||
"sha256": digest,
|
||||
"project": meta.get("project"),
|
||||
"doc_type_hint": meta.get("doc_type_hint"),
|
||||
"path": meta["path"],
|
||||
"created_at": meta["created_at"],
|
||||
"updated_at": meta["updated_at"],
|
||||
}
|
||||
if expose_path:
|
||||
upload["path"] = meta["path"]
|
||||
return upload
|
||||
|
||||
def _public_upload_ticket(self, meta: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
@@ -535,7 +601,7 @@ class IngestManager:
|
||||
return {
|
||||
"job_id": job["job_id"],
|
||||
"state": job["state"],
|
||||
"files": job["files"],
|
||||
"files": [self._public_upload(file_meta) for file_meta in job["files"]],
|
||||
"created_at": job["created_at"],
|
||||
"started_at": job.get("started_at"),
|
||||
"finished_at": job.get("finished_at"),
|
||||
@@ -546,6 +612,17 @@ class IngestManager:
|
||||
"log_tail": self._read_tail(Path(job["log_path"])),
|
||||
}
|
||||
|
||||
def _tree_mtime(self, path: Path) -> float:
|
||||
if not path.exists():
|
||||
return time.time()
|
||||
latest = path.stat().st_mtime
|
||||
for child in path.rglob("*"):
|
||||
try:
|
||||
latest = max(latest, child.stat().st_mtime)
|
||||
except OSError:
|
||||
continue
|
||||
return latest
|
||||
|
||||
def _summary_for_status(self, status: dict[str, Any]) -> str:
|
||||
state = status["state"]
|
||||
if state == "succeeded":
|
||||
|
||||
@@ -5,6 +5,7 @@ from __future__ import annotations
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
import uuid
|
||||
@@ -25,6 +26,7 @@ QUERY_MODES = {"normal", "index_only"}
|
||||
OPENCODE_REQUIRED_RUN_FLAGS = ("--format", "--auto", "--dir", "--title")
|
||||
OPENCODE_SMOKE_SUCCESS = "WIKI_QUERY_SKILL_LOADED"
|
||||
OPENCODE_SMOKE_FAILURE = "SKILL_LOAD_FAILED"
|
||||
TERMINAL_JOB_STATES = {"succeeded", "failed", "cancelled"}
|
||||
|
||||
|
||||
class QueryError(RuntimeError):
|
||||
@@ -188,6 +190,30 @@ class QueryManager:
|
||||
self._processes.pop(job_id, None)
|
||||
self._start_next_job_locked()
|
||||
|
||||
def cleanup_runtime(self, *, max_age_days: int = 30) -> dict[str, Any]:
|
||||
if max_age_days <= 0:
|
||||
raise QueryError("max_age_days must be positive")
|
||||
cutoff = time.time() - (max_age_days * 24 * 60 * 60)
|
||||
result = {
|
||||
"runtime_root": str(self.runtime_root),
|
||||
"max_age_days": max_age_days,
|
||||
"old_jobs_deleted": 0,
|
||||
}
|
||||
with self._lock:
|
||||
self.refresh_jobs()
|
||||
for job in self._iter_jobs():
|
||||
if job.get("state") not in TERMINAL_JOB_STATES:
|
||||
continue
|
||||
job_id = str(job.get("job_id", ""))
|
||||
if not re.fullmatch(r"[0-9a-f]{32}", job_id):
|
||||
continue
|
||||
job_dir = self.jobs_root / job_id
|
||||
if self._tree_mtime(job_dir) >= cutoff:
|
||||
continue
|
||||
shutil.rmtree(job_dir, ignore_errors=True)
|
||||
result["old_jobs_deleted"] += 1
|
||||
return result
|
||||
|
||||
def _start_next_job_locked(self) -> None:
|
||||
if any(job.get("state") == "running" for job in self._iter_jobs()):
|
||||
return
|
||||
@@ -319,10 +345,20 @@ class QueryManager:
|
||||
"pid": job.get("pid"),
|
||||
"error": job.get("error", ""),
|
||||
"preflight": job.get("preflight"),
|
||||
"log_path": job.get("log_path"),
|
||||
"log_tail": self._read_tail(Path(job["log_path"])),
|
||||
}
|
||||
|
||||
def _tree_mtime(self, path: Path) -> float:
|
||||
if not path.exists():
|
||||
return time.time()
|
||||
latest = path.stat().st_mtime
|
||||
for child in path.rglob("*"):
|
||||
try:
|
||||
latest = max(latest, child.stat().st_mtime)
|
||||
except OSError:
|
||||
continue
|
||||
return latest
|
||||
|
||||
def _check_opencode_version(self, result: dict[str, Any]) -> None:
|
||||
try:
|
||||
completed = subprocess.run(
|
||||
|
||||
+94
-16
@@ -9,6 +9,7 @@ import os
|
||||
import sys
|
||||
from email import policy
|
||||
from email.parser import BytesParser
|
||||
from fnmatch import fnmatch
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
@@ -172,6 +173,10 @@ class BearerTokenAuthMiddleware:
|
||||
await self.app(scope, receive, send)
|
||||
|
||||
|
||||
class RequestBodyTooLarge(IngestError):
|
||||
"""Raised when an HTTP ingest upload body exceeds the configured request limit."""
|
||||
|
||||
|
||||
def create_server(
|
||||
*,
|
||||
vault_path: str | Path,
|
||||
@@ -244,11 +249,8 @@ def create_server(
|
||||
def file_index_status() -> dict[str, Any]:
|
||||
"""Show the configured vault, file index, registry, and MCP limits."""
|
||||
registry = registry_path_for_vault(vault)
|
||||
return {
|
||||
"vault_path": str(vault),
|
||||
"db_path": str(db),
|
||||
status = {
|
||||
"db_exists": db.is_file(),
|
||||
"registry_path": str(registry),
|
||||
"registry_exists": registry.is_file(),
|
||||
"source_registry_records": len(load_registry_by_file_id(vault)),
|
||||
"profile_name": profile_name,
|
||||
@@ -266,6 +268,15 @@ def create_server(
|
||||
"query_opencode_model": query_opencode_model or "",
|
||||
"query_opencode_variant": query_opencode_variant or "",
|
||||
}
|
||||
if expose_paths:
|
||||
status.update(
|
||||
{
|
||||
"vault_path": str(vault),
|
||||
"db_path": str(db),
|
||||
"registry_path": str(registry),
|
||||
}
|
||||
)
|
||||
return status
|
||||
|
||||
if enable_ingest:
|
||||
if ingest_manager is None:
|
||||
@@ -324,8 +335,6 @@ def create_server(
|
||||
"max_files_per_job": ingest_manager.max_files_per_job,
|
||||
},
|
||||
"staging": {
|
||||
"runtime_upload_root": str(ingest_manager.upload_root),
|
||||
"vault_path": str(ingest_manager.vault),
|
||||
"note": "uploads are temporary staging files outside the vault; wiki-ingest decides final archive/index placement",
|
||||
},
|
||||
}
|
||||
@@ -582,34 +591,65 @@ class IngestUploadRouter:
|
||||
ingest_manager: IngestManager | None = None,
|
||||
upload_path: str = DEFAULT_INGEST_UPLOAD_PATH,
|
||||
bearer_auth_token: str = "",
|
||||
allowed_hosts: list[str] | None = None,
|
||||
allowed_origins: list[str] | None = None,
|
||||
) -> None:
|
||||
self.app = app
|
||||
self.ingest_manager = ingest_manager
|
||||
self.upload_path = upload_path
|
||||
self.bearer_auth_token = bearer_auth_token
|
||||
self.allowed_hosts = allowed_hosts or []
|
||||
self.allowed_origins = allowed_origins or []
|
||||
|
||||
async def __call__(self, scope: dict[str, Any], receive: Any, send: Any) -> None:
|
||||
if scope.get("type") != "http" or scope.get("path") != self.upload_path:
|
||||
await self.app(scope, receive, send)
|
||||
return
|
||||
if not self._transport_headers_allowed(scope):
|
||||
await _send_json(send, 403, {"error": "forbidden"})
|
||||
return
|
||||
if scope.get("method", "").upper() != "POST":
|
||||
if not self._has_upload_auth(scope):
|
||||
await _send_json(send, 401, {"error": "unauthorized"})
|
||||
return
|
||||
await _send_json(send, 405, {"error": "method_not_allowed"})
|
||||
return
|
||||
try:
|
||||
upload_ticket = self._authorize_upload(scope)
|
||||
except IngestError:
|
||||
await _send_json(send, 401, {"error": "unauthorized"})
|
||||
return
|
||||
if scope.get("method", "").upper() != "POST":
|
||||
await _send_json(send, 405, {"error": "method_not_allowed"})
|
||||
return
|
||||
if self.ingest_manager is None:
|
||||
await _send_json(send, 404, {"error": "ingest_not_enabled"})
|
||||
return
|
||||
try:
|
||||
data = await self._handle_upload(scope, receive, upload_ticket=upload_ticket)
|
||||
except RequestBodyTooLarge as exc:
|
||||
await _send_json(send, 413, {"error": str(exc)})
|
||||
return
|
||||
except IngestError as exc:
|
||||
await _send_json(send, 400, {"error": str(exc)})
|
||||
return
|
||||
await _send_json(send, 200, data)
|
||||
|
||||
def _transport_headers_allowed(self, scope: dict[str, Any]) -> bool:
|
||||
host = _header_value(scope, b"host")
|
||||
if self.allowed_hosts and (not host or not _matches_host(host, self.allowed_hosts)):
|
||||
return False
|
||||
origin = _header_value(scope, b"origin")
|
||||
if origin and self.allowed_origins and not _matches_any(origin, self.allowed_origins):
|
||||
return False
|
||||
return True
|
||||
|
||||
def _has_upload_auth(self, scope: dict[str, Any]) -> bool:
|
||||
authorization = _header_value(scope, b"authorization")
|
||||
token = bearer_token(authorization)
|
||||
if self.bearer_auth_token and hmac.compare_digest(token, self.bearer_auth_token):
|
||||
return True
|
||||
if _header_value(scope, b"x-obsidian-wiki-upload-token"):
|
||||
return True
|
||||
return bool(_query_param(scope, "upload_token"))
|
||||
|
||||
def _authorize_upload(self, scope: dict[str, Any]) -> dict[str, Any] | None:
|
||||
authorization = _header_value(scope, b"authorization")
|
||||
token = bearer_token(authorization)
|
||||
@@ -631,14 +671,14 @@ class IngestUploadRouter:
|
||||
*,
|
||||
upload_ticket: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
body = await _read_http_body(receive)
|
||||
manager = self.ingest_manager
|
||||
if manager is None:
|
||||
raise IngestError("ingest is not enabled")
|
||||
body = await _read_http_body(receive, max_bytes=_max_upload_body_bytes(manager, upload_ticket))
|
||||
content_type = _header_value(scope, b"content-type")
|
||||
fields, files = _parse_multipart_form(content_type, body)
|
||||
if not files:
|
||||
raise IngestError("multipart field 'files' must include at least one file")
|
||||
manager = self.ingest_manager
|
||||
if manager is None:
|
||||
raise IngestError("ingest is not enabled")
|
||||
max_files = int(upload_ticket.get("max_files", manager.max_files_per_job)) if upload_ticket else manager.max_files_per_job
|
||||
if len(files) > max_files:
|
||||
raise IngestError(f"too many files for one ingest job: {len(files)} > {max_files}")
|
||||
@@ -666,6 +706,8 @@ def create_http_app(
|
||||
*,
|
||||
ingest_manager: IngestManager | None = None,
|
||||
upload_path: str = DEFAULT_INGEST_UPLOAD_PATH,
|
||||
allowed_hosts: list[str] | None = None,
|
||||
allowed_origins: list[str] | None = None,
|
||||
) -> Any:
|
||||
app = server.streamable_http_app() if hasattr(server, "streamable_http_app") else server
|
||||
app = IngestUploadRouter(
|
||||
@@ -673,24 +715,39 @@ def create_http_app(
|
||||
ingest_manager=ingest_manager,
|
||||
upload_path=upload_path,
|
||||
bearer_auth_token=token,
|
||||
allowed_hosts=allowed_hosts,
|
||||
allowed_origins=allowed_origins,
|
||||
)
|
||||
return BearerTokenAuthMiddleware(app, token, bypass_paths={upload_path})
|
||||
|
||||
|
||||
async def _read_http_body(receive: Any) -> bytes:
|
||||
async def _read_http_body(receive: Any, *, max_bytes: int | None = None) -> bytes:
|
||||
chunks: list[bytes] = []
|
||||
received = 0
|
||||
while True:
|
||||
message = await receive()
|
||||
if message.get("type") == "http.disconnect":
|
||||
break
|
||||
if message.get("type") != "http.request":
|
||||
continue
|
||||
chunks.append(message.get("body", b""))
|
||||
chunk = message.get("body", b"")
|
||||
received += len(chunk)
|
||||
if max_bytes is not None and received > max_bytes:
|
||||
raise RequestBodyTooLarge(f"request body exceeds max upload size: {received} > {max_bytes}")
|
||||
chunks.append(chunk)
|
||||
if not message.get("more_body", False):
|
||||
break
|
||||
return b"".join(chunks)
|
||||
|
||||
|
||||
def _max_upload_body_bytes(manager: IngestManager, upload_ticket: dict[str, Any] | None) -> int | None:
|
||||
if manager.max_file_bytes <= 0:
|
||||
return None
|
||||
max_files = int(upload_ticket.get("max_files", manager.max_files_per_job)) if upload_ticket else manager.max_files_per_job
|
||||
overhead = max(16 * 1024, max_files * 4096)
|
||||
return manager.max_file_bytes * max_files + overhead
|
||||
|
||||
|
||||
def _header_value(scope: dict[str, Any], name: bytes) -> str:
|
||||
headers = {key.lower(): value for key, value in scope.get("headers", [])}
|
||||
return headers.get(name.lower(), b"").decode("latin1")
|
||||
@@ -705,6 +762,21 @@ def _query_param(scope: dict[str, Any], name: str) -> str:
|
||||
return values[0] if values else ""
|
||||
|
||||
|
||||
def _matches_any(value: str, patterns: list[str]) -> bool:
|
||||
return any(fnmatch(value, pattern) for pattern in patterns)
|
||||
|
||||
|
||||
def _matches_host(value: str, patterns: list[str]) -> bool:
|
||||
if _matches_any(value, patterns):
|
||||
return True
|
||||
host_only = value
|
||||
if value.startswith("[") and "]" in value:
|
||||
host_only = value[1 : value.index("]")]
|
||||
elif ":" in value:
|
||||
host_only = value.split(":", 1)[0]
|
||||
return _matches_any(host_only, patterns)
|
||||
|
||||
|
||||
def _parse_multipart_form(content_type: str, body: bytes) -> tuple[dict[str, list[str]], list[dict[str, Any]]]:
|
||||
if not content_type.lower().startswith("multipart/form-data"):
|
||||
raise IngestError("content-type must be multipart/form-data")
|
||||
@@ -862,7 +934,13 @@ def run_http_server(
|
||||
ingest_manager=ingest_manager,
|
||||
query_manager=query_manager,
|
||||
)
|
||||
app = create_http_app(server, token, ingest_manager=ingest_manager)
|
||||
app = create_http_app(
|
||||
server,
|
||||
token,
|
||||
ingest_manager=ingest_manager,
|
||||
allowed_hosts=allowed_hosts,
|
||||
allowed_origins=allowed_origins,
|
||||
)
|
||||
uvicorn.run(app, host=host, port=port, log_level="info")
|
||||
|
||||
|
||||
|
||||
@@ -4,6 +4,7 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import secrets
|
||||
import shlex
|
||||
@@ -11,6 +12,7 @@ import shutil
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Sequence
|
||||
@@ -36,11 +38,14 @@ DEFAULT_INGEST_OPENCODE_MODEL = "openai/gpt-5.5"
|
||||
DEFAULT_INGEST_OPENCODE_VARIANT = "high"
|
||||
DEFAULT_QUERY_OPENCODE_MODEL = "openai/gpt-5.5"
|
||||
DEFAULT_QUERY_OPENCODE_VARIANT = "minimal"
|
||||
DEFAULT_SERVICE_MEMORY_MAX = "2G"
|
||||
DEFAULT_RUNTIME_CLEANUP_DAYS = 30
|
||||
TOOL_FILE_ASSETS = "file-assets"
|
||||
TOOL_INGEST = "ingest"
|
||||
TOOL_QUERY = "query"
|
||||
AVAILABLE_TOOL_GROUPS = (TOOL_FILE_ASSETS, TOOL_INGEST, TOOL_QUERY)
|
||||
DEFAULT_ENABLED_TOOLS = (TOOL_FILE_ASSETS,)
|
||||
TERMINAL_JOB_STATES = {"succeeded", "failed", "cancelled"}
|
||||
PROXY_ENV_NAMES = ("http_proxy", "https_proxy", "all_proxy", "no_proxy", "HTTP_PROXY", "HTTPS_PROXY", "ALL_PROXY", "NO_PROXY")
|
||||
|
||||
|
||||
@@ -181,6 +186,7 @@ def render_service_unit(
|
||||
path: str = DEFAULT_PATH,
|
||||
uv_cache_dir: str = DEFAULT_UV_CACHE_DIR,
|
||||
uv_path: Path | None = None,
|
||||
memory_max: str = DEFAULT_SERVICE_MEMORY_MAX,
|
||||
) -> str:
|
||||
resolved_uv_path = uv_path or find_uv_path()
|
||||
command = " ".join(
|
||||
@@ -218,6 +224,7 @@ def render_service_unit(
|
||||
f"ExecStart={command}\n"
|
||||
"Restart=on-failure\n"
|
||||
"RestartSec=5\n"
|
||||
f"MemoryMax={memory_max}\n"
|
||||
"\n"
|
||||
"[Install]\n"
|
||||
"WantedBy=default.target\n"
|
||||
@@ -677,6 +684,132 @@ def service_status() -> None:
|
||||
run_systemctl(["status", SERVICE_NAME], check=False)
|
||||
|
||||
|
||||
def cleanup_runtime(project_root: Path, *, max_age_days: int = DEFAULT_RUNTIME_CLEANUP_DAYS) -> str:
|
||||
if max_age_days <= 0:
|
||||
raise RuntimeError("max_age_days must be positive")
|
||||
root = project_root.expanduser().resolve()
|
||||
cutoff = time.time() - (max_age_days * 24 * 60 * 60)
|
||||
ingest_root = root / ".runtime" / "mcp-ingest"
|
||||
query_root = root / ".runtime" / "mcp-query"
|
||||
active_upload_ids = _active_ingest_upload_ids(ingest_root / "jobs")
|
||||
result = {
|
||||
"expired_upload_tickets_deleted": _delete_expired_upload_tickets(ingest_root / "upload-tickets"),
|
||||
"old_ingest_uploads_deleted": _delete_old_ingest_uploads(
|
||||
ingest_root / "uploads",
|
||||
cutoff=cutoff,
|
||||
active_upload_ids=active_upload_ids,
|
||||
),
|
||||
"old_ingest_jobs_deleted": _delete_old_terminal_jobs(ingest_root / "jobs", cutoff=cutoff),
|
||||
"old_query_jobs_deleted": _delete_old_terminal_jobs(query_root / "jobs", cutoff=cutoff),
|
||||
}
|
||||
return "\n".join(
|
||||
[
|
||||
"MCP runtime cleanup.",
|
||||
"",
|
||||
f"Project root: {root}",
|
||||
f"Max age: {max_age_days} days",
|
||||
f"Expired upload tickets deleted: {result['expired_upload_tickets_deleted']}",
|
||||
f"Old ingest uploads deleted: {result['old_ingest_uploads_deleted']}",
|
||||
f"Old ingest jobs/logs deleted: {result['old_ingest_jobs_deleted']}",
|
||||
f"Old query jobs/logs deleted: {result['old_query_jobs_deleted']}",
|
||||
]
|
||||
) + "\n"
|
||||
|
||||
|
||||
def _active_ingest_upload_ids(jobs_root: Path) -> set[str]:
|
||||
upload_ids: set[str] = set()
|
||||
for job in _iter_job_records(jobs_root):
|
||||
if job.get("state") in TERMINAL_JOB_STATES:
|
||||
continue
|
||||
upload_ids.update(str(upload_id) for upload_id in job.get("upload_ids", []))
|
||||
return upload_ids
|
||||
|
||||
|
||||
def _delete_expired_upload_tickets(ticket_root: Path) -> int:
|
||||
if not ticket_root.exists():
|
||||
return 0
|
||||
deleted = 0
|
||||
now = time.time()
|
||||
for ticket_path in ticket_root.glob("*.json"):
|
||||
meta = _read_json(ticket_path)
|
||||
if meta is None:
|
||||
continue
|
||||
try:
|
||||
expired = now > float(meta.get("expires_epoch", 0))
|
||||
except (TypeError, ValueError):
|
||||
expired = False
|
||||
if not expired:
|
||||
continue
|
||||
ticket_path.unlink(missing_ok=True)
|
||||
deleted += 1
|
||||
return deleted
|
||||
|
||||
|
||||
def _delete_old_ingest_uploads(upload_root: Path, *, cutoff: float, active_upload_ids: set[str]) -> int:
|
||||
if not upload_root.exists():
|
||||
return 0
|
||||
deleted = 0
|
||||
for meta_path in upload_root.glob("*/*/upload.json"):
|
||||
upload_dir = meta_path.parent
|
||||
meta = _read_json(meta_path)
|
||||
if meta is None:
|
||||
continue
|
||||
if str(meta.get("upload_id", "")) in active_upload_ids:
|
||||
continue
|
||||
if _tree_mtime(upload_dir) >= cutoff:
|
||||
continue
|
||||
shutil.rmtree(upload_dir, ignore_errors=True)
|
||||
deleted += 1
|
||||
return deleted
|
||||
|
||||
|
||||
def _delete_old_terminal_jobs(jobs_root: Path, *, cutoff: float) -> int:
|
||||
if not jobs_root.exists():
|
||||
return 0
|
||||
deleted = 0
|
||||
for job in _iter_job_records(jobs_root):
|
||||
if job.get("state") not in TERMINAL_JOB_STATES:
|
||||
continue
|
||||
job_id = str(job.get("job_id", ""))
|
||||
job_dir = jobs_root / job_id
|
||||
if not job_dir.exists() or _tree_mtime(job_dir) >= cutoff:
|
||||
continue
|
||||
shutil.rmtree(job_dir, ignore_errors=True)
|
||||
deleted += 1
|
||||
return deleted
|
||||
|
||||
|
||||
def _iter_job_records(jobs_root: Path) -> list[dict[str, object]]:
|
||||
jobs: list[dict[str, object]] = []
|
||||
if not jobs_root.exists():
|
||||
return jobs
|
||||
for job_path in jobs_root.glob("*/job.json"):
|
||||
record = _read_json(job_path)
|
||||
if record is not None:
|
||||
jobs.append(record)
|
||||
return jobs
|
||||
|
||||
|
||||
def _read_json(path: Path) -> dict[str, object] | None:
|
||||
try:
|
||||
data = json.loads(path.read_text(encoding="utf-8"))
|
||||
except (OSError, json.JSONDecodeError):
|
||||
return None
|
||||
return data if isinstance(data, dict) else None
|
||||
|
||||
|
||||
def _tree_mtime(path: Path) -> float:
|
||||
if not path.exists():
|
||||
return time.time()
|
||||
latest = path.stat().st_mtime
|
||||
for child in path.rglob("*"):
|
||||
try:
|
||||
latest = max(latest, child.stat().st_mtime)
|
||||
except OSError:
|
||||
continue
|
||||
return latest
|
||||
|
||||
|
||||
def main(argv: Sequence[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(description="Manage the readonly HTTP obsidian-wiki MCP user service.")
|
||||
subparsers = parser.add_subparsers(dest="command", required=True)
|
||||
@@ -734,6 +867,10 @@ def main(argv: Sequence[str] | None = None) -> int:
|
||||
|
||||
subparsers.add_parser("status", help="Show systemd status for the managed service.")
|
||||
|
||||
cleanup_parser = subparsers.add_parser("cleanup", help="Prune expired tickets and old MCP runtime uploads/jobs/logs.")
|
||||
cleanup_parser.add_argument("--project-root", default=Path(__file__).resolve().parent.parent, type=Path)
|
||||
cleanup_parser.add_argument("--max-age-days", default=DEFAULT_RUNTIME_CLEANUP_DAYS, type=int)
|
||||
|
||||
tools_parser = subparsers.add_parser("tools", help="List or update enabled MCP tool groups in the service config.")
|
||||
tools_subparsers = tools_parser.add_subparsers(dest="tools_command", required=True)
|
||||
|
||||
@@ -791,6 +928,9 @@ def main(argv: Sequence[str] | None = None) -> int:
|
||||
if args.command == "status":
|
||||
service_status()
|
||||
return 0
|
||||
if args.command == "cleanup":
|
||||
print(cleanup_runtime(args.project_root, max_age_days=args.max_age_days), end="")
|
||||
return 0
|
||||
if args.command == "tools":
|
||||
if args.tools_command == "list":
|
||||
print(list_tools(args.project_root), end="")
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
import base64
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
@@ -46,8 +47,9 @@ class McpIngestTests(unittest.TestCase):
|
||||
|
||||
def test_uploads_are_staged_in_runtime_not_vault(self):
|
||||
upload = self.manager.create_upload(file_name="drawing.pdf", size_bytes=0, sha256=self._sha256(b""))
|
||||
upload_path = Path(upload["path"]).resolve()
|
||||
upload_path = self._upload_payload_path(upload["upload_id"]).resolve()
|
||||
|
||||
self.assertNotIn("path", upload)
|
||||
self.assertTrue(upload_path.is_relative_to((self.runtime / "uploads").resolve()))
|
||||
self.assertFalse(upload_path.is_relative_to(self.vault.resolve()))
|
||||
|
||||
@@ -169,7 +171,8 @@ class McpIngestTests(unittest.TestCase):
|
||||
complete = self.manager.complete_upload(upload["upload_id"])
|
||||
|
||||
self.assertEqual(complete["state"], "complete")
|
||||
self.assertEqual(Path(complete["path"]).read_bytes(), payload)
|
||||
self.assertNotIn("path", complete)
|
||||
self.assertEqual(self._upload_payload_path(complete["upload_id"]).read_bytes(), payload)
|
||||
|
||||
def test_create_completed_upload_accepts_bytes_without_agent_chunking(self):
|
||||
payload = b"direct upload payload"
|
||||
@@ -187,7 +190,8 @@ class McpIngestTests(unittest.TestCase):
|
||||
self.assertEqual(upload["sha256"], "sha256:" + self._sha256(payload))
|
||||
self.assertEqual(upload["project"], "bjd")
|
||||
self.assertEqual(upload["doc_type_hint"], "drawing")
|
||||
self.assertEqual(Path(upload["path"]).read_bytes(), payload)
|
||||
self.assertNotIn("path", upload)
|
||||
self.assertEqual(self._upload_payload_path(upload["upload_id"]).read_bytes(), payload)
|
||||
|
||||
def test_upload_ticket_is_single_use_and_preserves_defaults(self):
|
||||
ticket = self.manager.create_upload_ticket(
|
||||
@@ -255,17 +259,19 @@ class McpIngestTests(unittest.TestCase):
|
||||
self.assertIn(f"MCP ingest {job['job_id']}", args)
|
||||
prompt = args[-1]
|
||||
self.assertIn("/wiki-ingest", prompt)
|
||||
self.assertIn(str(Path(upload["path"])), prompt)
|
||||
self.assertIn(str(self._upload_payload_path(upload["upload_id"])), prompt)
|
||||
self.assertIn("temporary runtime staging files", prompt)
|
||||
self.assertIn("Do not ingest any file outside this list.", prompt)
|
||||
self.assertIn("obsidian-wiki file-index rebuild", prompt)
|
||||
self.assertNotIn("../secret", prompt)
|
||||
self.assertNotIn("path", job["files"][0])
|
||||
|
||||
popen.returncode = 0
|
||||
self.manager.refresh_jobs()
|
||||
status = self.manager.get_job_status(job["job_id"])
|
||||
self.assertEqual(status["state"], "succeeded")
|
||||
self.assertEqual(status["exit_code"], 0)
|
||||
self.assertNotIn("path", status["files"][0])
|
||||
|
||||
def test_jobs_queue_one_at_a_time_and_cancel_queued_job(self):
|
||||
first = self._complete_upload("first.pdf", b"first")
|
||||
@@ -286,6 +292,39 @@ class McpIngestTests(unittest.TestCase):
|
||||
|
||||
self.assertEqual(cancelled["state"], "cancelled")
|
||||
|
||||
def test_cleanup_runtime_prunes_expired_tickets_and_old_completed_artifacts(self):
|
||||
upload = self._complete_upload("old.pdf", b"old payload")
|
||||
ticket = self.manager.create_upload_ticket(expires_seconds=60, max_files=1)
|
||||
ticket_path = next(self.manager.upload_ticket_root.glob("*.json"))
|
||||
ticket_meta = json.loads(ticket_path.read_text(encoding="utf-8"))
|
||||
ticket_meta["expires_epoch"] = time.time() - 10
|
||||
ticket_path.write_text(json.dumps(ticket_meta), encoding="utf-8")
|
||||
|
||||
popen = FakePopen(returncode=0)
|
||||
with (
|
||||
patch("obsidian_wiki.mcp_ingest.subprocess.run", side_effect=self._fake_static_readiness_run),
|
||||
patch("obsidian_wiki.mcp_ingest.subprocess.Popen", return_value=popen),
|
||||
):
|
||||
job = self.manager.start_ingest_job([upload["upload_id"]])
|
||||
self.manager.refresh_jobs()
|
||||
|
||||
old_epoch = time.time() - 40 * 24 * 60 * 60
|
||||
upload_path = self._upload_payload_path(upload["upload_id"])
|
||||
upload_dir = upload_path.parent
|
||||
job_dir = self.manager.jobs_root / job["job_id"]
|
||||
self._touch_tree(upload_dir, old_epoch)
|
||||
self._touch_tree(job_dir, old_epoch)
|
||||
os.utime(ticket_path, (old_epoch, old_epoch))
|
||||
|
||||
result = self.manager.cleanup_runtime(max_age_days=30)
|
||||
|
||||
self.assertGreaterEqual(result["expired_tickets_deleted"], 1)
|
||||
self.assertEqual(result["old_uploads_deleted"], 1)
|
||||
self.assertEqual(result["old_jobs_deleted"], 1)
|
||||
self.assertFalse(upload_dir.exists())
|
||||
self.assertFalse(job_dir.exists())
|
||||
self.assertFalse(ticket_path.exists())
|
||||
|
||||
def _complete_upload(self, name: str, payload: bytes) -> dict:
|
||||
upload = self.manager.create_upload(name, len(payload), self._sha256(payload))
|
||||
offset = 0
|
||||
@@ -299,6 +338,14 @@ class McpIngestTests(unittest.TestCase):
|
||||
offset += len(chunk)
|
||||
return self.manager.complete_upload(upload["upload_id"])
|
||||
|
||||
def _upload_payload_path(self, upload_id: str) -> Path:
|
||||
return Path(self.manager._load_upload(upload_id)["path"])
|
||||
|
||||
def _touch_tree(self, path: Path, epoch: float) -> None:
|
||||
for child in path.rglob("*"):
|
||||
os.utime(child, (epoch, epoch))
|
||||
os.utime(path, (epoch, epoch))
|
||||
|
||||
def _sha256(self, payload: bytes) -> str:
|
||||
import hashlib
|
||||
|
||||
|
||||
+26
-1
@@ -1,4 +1,5 @@
|
||||
import subprocess
|
||||
import os
|
||||
import tempfile
|
||||
import time
|
||||
import unittest
|
||||
@@ -140,7 +141,9 @@ class McpQueryTests(unittest.TestCase):
|
||||
):
|
||||
job = self.manager.start_query_job("what do I know?")
|
||||
|
||||
log_path = Path(self.manager.get_query_job_status(job["job_id"])["log_path"])
|
||||
status = self.manager.get_query_job_status(job["job_id"])
|
||||
self.assertNotIn("log_path", status)
|
||||
log_path = Path(self.manager._load_job(job["job_id"])["log_path"])
|
||||
log_path.write_text("Based on the wiki:\nAnswer body\n", encoding="utf-8")
|
||||
popen.returncode = 0
|
||||
result = self.manager.get_query_job_result(job["job_id"])
|
||||
@@ -166,6 +169,28 @@ class McpQueryTests(unittest.TestCase):
|
||||
|
||||
self.assertEqual(cancelled["state"], "cancelled")
|
||||
|
||||
def test_cleanup_runtime_prunes_old_terminal_query_jobs(self):
|
||||
popen = FakePopen(returncode=0)
|
||||
with (
|
||||
patch("obsidian_wiki.mcp_query.subprocess.run", side_effect=self._fake_static_readiness_run),
|
||||
patch("obsidian_wiki.mcp_query.subprocess.Popen", return_value=popen),
|
||||
):
|
||||
job = self.manager.start_query_job("old query")
|
||||
self.manager.refresh_jobs()
|
||||
job_dir = self.manager.jobs_root / job["job_id"]
|
||||
old_epoch = time.time() - 40 * 24 * 60 * 60
|
||||
self._touch_tree(job_dir, old_epoch)
|
||||
|
||||
result = self.manager.cleanup_runtime(max_age_days=30)
|
||||
|
||||
self.assertEqual(result["old_jobs_deleted"], 1)
|
||||
self.assertFalse(job_dir.exists())
|
||||
|
||||
def _touch_tree(self, path: Path, epoch: float) -> None:
|
||||
for child in path.rglob("*"):
|
||||
os.utime(child, (epoch, epoch))
|
||||
os.utime(path, (epoch, epoch))
|
||||
|
||||
def _write_opencode_skill_fixture(self, root: Path) -> None:
|
||||
(root / "AGENTS.md").write_text("# Agent rules\n", encoding="utf-8")
|
||||
wiki_query = root / ".agents" / "skills" / "wiki-query"
|
||||
|
||||
@@ -35,15 +35,16 @@ def run_asgi_app(app, *, headers=None, scope_type="http"):
|
||||
return messages
|
||||
|
||||
|
||||
def run_asgi_http_request(app, *, method="GET", path="/", headers=None, body=b""):
|
||||
def run_asgi_http_request(app, *, method="GET", path="/", headers=None, body=b"", query_string=b""):
|
||||
messages = []
|
||||
sent = False
|
||||
query = query_string.encode("utf-8") if isinstance(query_string, str) else query_string
|
||||
scope = {
|
||||
"type": "http",
|
||||
"method": method,
|
||||
"path": path,
|
||||
"raw_path": path.encode("ascii"),
|
||||
"query_string": b"",
|
||||
"query_string": query,
|
||||
"headers": headers or [],
|
||||
}
|
||||
|
||||
@@ -197,6 +198,7 @@ class RemoteMcpServerTests(unittest.TestCase):
|
||||
self.assertEqual(data["uploads"][0]["state"], "complete")
|
||||
self.assertEqual(data["uploads"][0]["file_name"], "drawing.pdf")
|
||||
self.assertEqual(data["uploads"][0]["sha256"], "sha256:" + hashlib.sha256(payload).hexdigest())
|
||||
self.assertNotIn("path", data["uploads"][0])
|
||||
|
||||
def test_http_upload_endpoint_accepts_upload_ticket_without_main_bearer_token(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
@@ -249,8 +251,118 @@ class RemoteMcpServerTests(unittest.TestCase):
|
||||
self.assertEqual(data["uploads"][0]["project"], "bjd")
|
||||
self.assertEqual(data["uploads"][0]["doc_type_hint"], "drawing")
|
||||
self.assertEqual(data["uploads"][0]["sha256"], "sha256:" + hashlib.sha256(payload).hexdigest())
|
||||
self.assertNotIn("path", data["uploads"][0])
|
||||
self.assertEqual(reused[0]["status"], 401)
|
||||
|
||||
def test_http_upload_ticket_is_not_consumed_by_non_post_request(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
manager = IngestManager(
|
||||
vault_path=root / "vault",
|
||||
project_root=root / "project",
|
||||
runtime_root=root / "runtime",
|
||||
)
|
||||
ticket = manager.create_upload_ticket(expires_seconds=60, max_files=1, start_job=False)
|
||||
app = mcp_server.create_http_app(FakeApp(), "secret-token", ingest_manager=manager)
|
||||
payload = b"ticket should survive get"
|
||||
body, boundary = multipart_body({}, [("files", "survived.txt", "text/plain", payload)])
|
||||
|
||||
get_messages = run_asgi_http_request(
|
||||
app,
|
||||
method="GET",
|
||||
path="/mcp-ingest/uploads",
|
||||
headers=[(b"x-obsidian-wiki-upload-token", ticket["upload_token"].encode("ascii"))],
|
||||
)
|
||||
post_messages = run_asgi_http_request(
|
||||
app,
|
||||
method="POST",
|
||||
path="/mcp-ingest/uploads",
|
||||
headers=[
|
||||
(b"x-obsidian-wiki-upload-token", ticket["upload_token"].encode("ascii")),
|
||||
(b"content-type", f"multipart/form-data; boundary={boundary}".encode("ascii")),
|
||||
],
|
||||
body=body,
|
||||
)
|
||||
|
||||
self.assertEqual(get_messages[0]["status"], 405)
|
||||
self.assertEqual(post_messages[0]["status"], 200)
|
||||
|
||||
def test_http_upload_endpoint_rejects_oversized_body_with_413(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
manager = IngestManager(
|
||||
vault_path=root / "vault",
|
||||
project_root=root / "project",
|
||||
runtime_root=root / "runtime",
|
||||
max_file_bytes=1,
|
||||
max_files_per_job=1,
|
||||
)
|
||||
app = mcp_server.create_http_app(FakeApp(), "secret-token", ingest_manager=manager)
|
||||
body, boundary = multipart_body({}, [("files", "too-large.bin", "application/octet-stream", b"x" * 25000)])
|
||||
|
||||
messages = run_asgi_http_request(
|
||||
app,
|
||||
method="POST",
|
||||
path="/mcp-ingest/uploads",
|
||||
headers=[
|
||||
(b"authorization", b"Bearer secret-token"),
|
||||
(b"content-type", f"multipart/form-data; boundary={boundary}".encode("ascii")),
|
||||
],
|
||||
body=body,
|
||||
)
|
||||
|
||||
self.assertEqual(messages[0]["status"], 413)
|
||||
|
||||
def test_http_upload_endpoint_enforces_allowed_host_and_origin(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
manager = IngestManager(
|
||||
vault_path=root / "vault",
|
||||
project_root=root / "project",
|
||||
runtime_root=root / "runtime",
|
||||
)
|
||||
app = mcp_server.create_http_app(
|
||||
FakeApp(),
|
||||
"secret-token",
|
||||
ingest_manager=manager,
|
||||
allowed_hosts=["good.example"],
|
||||
allowed_origins=["https://app.example"],
|
||||
)
|
||||
body, boundary = multipart_body(
|
||||
{"start_job": "false"},
|
||||
[("files", "allowed.txt", "text/plain", b"ok")],
|
||||
)
|
||||
common_headers = [
|
||||
(b"authorization", b"Bearer secret-token"),
|
||||
(b"content-type", f"multipart/form-data; boundary={boundary}".encode("ascii")),
|
||||
]
|
||||
|
||||
bad_host = run_asgi_http_request(
|
||||
app,
|
||||
method="POST",
|
||||
path="/mcp-ingest/uploads",
|
||||
headers=[*common_headers, (b"host", b"bad.example:8765"), (b"origin", b"https://app.example")],
|
||||
body=body,
|
||||
)
|
||||
bad_origin = run_asgi_http_request(
|
||||
app,
|
||||
method="POST",
|
||||
path="/mcp-ingest/uploads",
|
||||
headers=[*common_headers, (b"host", b"good.example:8765"), (b"origin", b"https://evil.example")],
|
||||
body=body,
|
||||
)
|
||||
allowed = run_asgi_http_request(
|
||||
app,
|
||||
method="POST",
|
||||
path="/mcp-ingest/uploads",
|
||||
headers=[*common_headers, (b"host", b"good.example:8765"), (b"origin", b"https://app.example")],
|
||||
body=body,
|
||||
)
|
||||
|
||||
self.assertEqual(bad_host[0]["status"], 403)
|
||||
self.assertEqual(bad_origin[0]["status"], 403)
|
||||
self.assertEqual(allowed[0]["status"], 200)
|
||||
|
||||
def test_http_upload_endpoint_requires_ingest_and_bearer_token(self):
|
||||
no_ingest_app = mcp_server.create_http_app(FakeApp(), "secret-token")
|
||||
missing_token = run_asgi_http_request(no_ingest_app, method="POST", path="/mcp-ingest/uploads")
|
||||
@@ -492,6 +604,24 @@ class RemoteMcpServerTests(unittest.TestCase):
|
||||
self.assertEqual(status["ingest_opencode_variant"], "high")
|
||||
self.assertEqual(status["query_opencode_model"], "openai/gpt-5.5")
|
||||
self.assertEqual(status["query_opencode_variant"], "minimal")
|
||||
self.assertNotIn("vault_path", status)
|
||||
self.assertNotIn("db_path", status)
|
||||
self.assertNotIn("registry_path", status)
|
||||
|
||||
with (
|
||||
tempfile.TemporaryDirectory() as tmp,
|
||||
patch.dict("sys.modules", {"mcp.server.fastmcp": type("Mod", (), {"FastMCP": FakeMCP})()}),
|
||||
):
|
||||
root = Path(tmp)
|
||||
mcp_server.create_server(
|
||||
vault_path=root / "vault",
|
||||
expose_paths=True,
|
||||
)
|
||||
path_status = FakeMCP.tools["file_index_status"]()
|
||||
|
||||
self.assertIn("vault_path", path_status)
|
||||
self.assertIn("db_path", path_status)
|
||||
self.assertIn("registry_path", path_status)
|
||||
|
||||
def test_create_ingest_upload_ticket_tool_returns_http_only_token(self):
|
||||
class FakeMCP:
|
||||
@@ -599,7 +729,8 @@ class RemoteMcpServerTests(unittest.TestCase):
|
||||
note="captured from remote agent",
|
||||
start_job=False,
|
||||
)
|
||||
payload = Path(result["uploads"][0]["path"]).read_text(encoding="utf-8")
|
||||
payload_path = Path(manager._load_upload(result["uploads"][0]["upload_id"])["path"])
|
||||
payload = payload_path.read_text(encoding="utf-8")
|
||||
|
||||
self.assertEqual(result["job"], None)
|
||||
self.assertEqual(len(result["uploads"]), 1)
|
||||
@@ -607,6 +738,7 @@ class RemoteMcpServerTests(unittest.TestCase):
|
||||
self.assertEqual(upload["state"], "complete")
|
||||
self.assertEqual(upload["project"], "bjd")
|
||||
self.assertEqual(upload["doc_type_hint"], "thought")
|
||||
self.assertNotIn("path", upload)
|
||||
self.assertTrue(upload["file_name"].endswith(".md"))
|
||||
self.assertNotIn("/", upload["file_name"])
|
||||
self.assertIn("# Customer thought: ACME / risk", payload)
|
||||
|
||||
@@ -48,6 +48,8 @@ class McpServiceConfigTests(unittest.TestCase):
|
||||
self.assertIn("--host 0.0.0.0", unit)
|
||||
self.assertIn("--port 8765", unit)
|
||||
self.assertIn("PassEnvironment=http_proxy https_proxy all_proxy no_proxy HTTP_PROXY HTTPS_PROXY ALL_PROXY NO_PROXY", unit)
|
||||
self.assertIn("MemoryMax=2G", unit)
|
||||
self.assertIn("Restart=on-failure", unit)
|
||||
self.assertNotIn("OBSIDIAN_MCP_TOKEN=", unit)
|
||||
|
||||
def test_write_token_env_file_is_private(self):
|
||||
@@ -485,6 +487,17 @@ class McpServiceConfigTests(unittest.TestCase):
|
||||
self.assertNotIn("enable_ingest", new_token.call_args.kwargs)
|
||||
print_fn.assert_called_once_with("config\n", end="")
|
||||
|
||||
def test_main_cleanup_delegates_to_runtime_cleanup(self):
|
||||
with (
|
||||
patch.object(manage_mcp_service, "cleanup_runtime", return_value="cleanup\n") as cleanup_runtime,
|
||||
patch("builtins.print") as print_fn,
|
||||
):
|
||||
status = manage_mcp_service.main(["cleanup", "--project-root", "/tmp/project", "--max-age-days", "14"])
|
||||
|
||||
self.assertEqual(status, 0)
|
||||
cleanup_runtime.assert_called_once_with(Path("/tmp/project"), max_age_days=14)
|
||||
print_fn.assert_called_once_with("cleanup\n", end="")
|
||||
|
||||
def test_uninstall_service_removes_only_managed_files(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
project_root = Path(tmp) / "project"
|
||||
|
||||
Reference in New Issue
Block a user