feat(mcp): issue upload tickets for ingest
Add short-lived single-use HTTP upload tickets that authenticated MCP clients can request before uploading files to /mcp-ingest/uploads. This lets remote agents use curl without needing to read or expose the main MCP bearer token, avoiding oversized base64 tool arguments for larger files. The upload endpoint now accepts either the normal bearer token or X-Obsidian-Wiki-Upload-Token. Ticket defaults can carry project, doc_type_hint, note, start_job, and file-count limits while uploads still stage outside the vault. Validation: python3 -m unittest discover -v; python3 -m py_compile obsidian_wiki/mcp_ingest.py obsidian_wiki/mcp_server.py tests/test_mcp_ingest.py tests/test_mcp_server_remote.py; git diff --check
This commit is contained in:
+4
-2
@@ -57,8 +57,10 @@ OBSIDIAN_MCP_ENABLED_TOOLS=file-assets
|
||||
# the first upload; set run_smoke_test=true to verify OpenCode can load the
|
||||
# project wiki-ingest skill without editing files.
|
||||
# Preferred remote file upload path: POST multipart/form-data to
|
||||
# /mcp-ingest/uploads with the same Authorization bearer token as /mcp. Agents
|
||||
# can also call `get_ingest_upload_instructions` for exact fields/examples.
|
||||
# /mcp-ingest/uploads. Agents that cannot read the main MCP bearer token should
|
||||
# call `create_ingest_upload_ticket` first and send the returned
|
||||
# X-Obsidian-Wiki-Upload-Token header. `get_ingest_upload_instructions` returns
|
||||
# exact fields/examples.
|
||||
# Keep this runtime root outside OBSIDIAN_VAULT_PATH; uploads are staging inputs,
|
||||
# and wiki-ingest decides how sources are archived into the vault.
|
||||
# Prefer OBSIDIAN_MCP_ENABLED_TOOLS=file-assets,ingest for new configs. This
|
||||
|
||||
@@ -130,7 +130,7 @@ See `wiki-query` and `wiki-export` skills for how the filter is applied.
|
||||
- **Keep product lookup fields explicit.** Carry forward `bjd_part_numbers`, `competitor_part_numbers`, `series`, `pitch`, `pin_count`, `product_lines`, `customers`, `competitors`, `standards`, `business_categories`, `related_pages`, and `catalog_paths` whenever they are known.
|
||||
- **Merge, do not overwrite.** Automatic identifier extraction may add terms, but it must not discard curated registry metadata. Registry values are the durable source for future index rebuilds and remote MCP lookup.
|
||||
- **Prefer precise identifiers in search ranking.** Exact or prefix matches in structured part-number and series fields should rank above incidental filename substrings such as timestamps or archive hashes.
|
||||
- **Keep remote ingest gated.** MCP ingest tools are write-capable and must remain disabled unless the service config explicitly enables the `ingest` tool group via `OBSIDIAN_MCP_ENABLED_TOOLS=file-assets,ingest`, `mcp_service.sh tools enable ingest`, or `mcp_service.sh start --enable-ingest`. Uploaded files must stage outside the vault under the MCP runtime root; `wiki-ingest` is responsible for accepting, archiving, and indexing sources inside the vault. Use the HTTP multipart endpoint `/mcp-ingest/uploads` for remote file transfer, or `ingest_file_base64` for small files when a client cannot make multipart HTTP requests. Do not expose chunked upload tools in the MCP tool surface. They may launch local OpenCode only through the fixed `/wiki-ingest` prompt; never expose arbitrary prompt or shell execution through MCP. Remote ingest must keep the OpenCode readiness preflight in front of job launch, and `check_ingest_opencode_readiness(run_smoke_test=true)` should stay available for operators to verify that OpenCode can load the project `wiki-ingest` skill before accepting uploads.
|
||||
- **Keep remote ingest gated.** MCP ingest tools are write-capable and must remain disabled unless the service config explicitly enables the `ingest` tool group via `OBSIDIAN_MCP_ENABLED_TOOLS=file-assets,ingest`, `mcp_service.sh tools enable ingest`, or `mcp_service.sh start --enable-ingest`. Uploaded files must stage outside the vault under the MCP runtime root; `wiki-ingest` is responsible for accepting, archiving, and indexing sources inside the vault. Use the HTTP multipart endpoint `/mcp-ingest/uploads` for remote file transfer; if the remote agent cannot read the main MCP bearer token, it must call `create_ingest_upload_ticket` and send the returned single-use `X-Obsidian-Wiki-Upload-Token` header. Use `ingest_file_base64` only for small files when a client cannot make multipart HTTP requests. Do not expose chunked upload tools in the MCP tool surface. They may launch local OpenCode only through the fixed `/wiki-ingest` prompt; never expose arbitrary prompt or shell execution through MCP. Remote ingest must keep the OpenCode readiness preflight in front of job launch, and `check_ingest_opencode_readiness(run_smoke_test=true)` should stay available for operators to verify that OpenCode can load the project `wiki-ingest` skill before accepting uploads.
|
||||
|
||||
## Core Principles
|
||||
|
||||
|
||||
@@ -401,15 +401,16 @@ After `ingest` is enabled once, later `bash scripts/mcp_service.sh start` preser
|
||||
This adds write-capable MCP tools to the same `wiki-file-assets` server and bearer token. The server is tool-first: some MCP clients may show empty `resources` or `prompts` lists until a concrete `asset://source/sha256/...` URI is returned by `get_file_asset_metadata`, so use the tool list as the source of truth.
|
||||
|
||||
- `get_ingest_upload_instructions` — returns the preferred upload methods, HTTP endpoint path, fields, limits, and curl example for remote agents.
|
||||
- `create_ingest_upload_ticket` — returns a short-lived, single-use upload token for the HTTP endpoint so remote agents can upload with curl without reading the main MCP bearer token.
|
||||
- `ingest_file_base64` — uploads one complete base64 file and optionally starts the local OpenCode ingest job; useful for small files or MCP clients that cannot make multipart HTTP requests.
|
||||
- `check_ingest_opencode_readiness` — verifies the local OpenCode binary, required run flags, project ingest skills, and optionally runs a no-write smoke test that confirms `wiki-ingest` can be loaded.
|
||||
- `start_ingest_job` — queues a local `opencode run --auto --format json` job with a fixed `/wiki-ingest` prompt for the completed uploads.
|
||||
- `get_ingest_job_status`, `get_ingest_job_result`, `cancel_ingest_job` — inspect or cancel the local job.
|
||||
|
||||
For binary files, use the HTTP upload endpoint:
|
||||
For binary files, use the HTTP upload endpoint. If the remote agent cannot read the main MCP bearer token, first call `create_ingest_upload_ticket`, then upload with the returned token:
|
||||
|
||||
```bash
|
||||
curl -H "Authorization: Bearer <long-random-token>" \
|
||||
curl -H "X-Obsidian-Wiki-Upload-Token: <upload-token>" \
|
||||
-F "files=@drawing.pdf" \
|
||||
-F "project=bjd" \
|
||||
-F "doc_type_hint=drawing" \
|
||||
@@ -417,7 +418,7 @@ curl -H "Authorization: Bearer <long-random-token>" \
|
||||
http://127.0.0.1:8765/mcp-ingest/uploads
|
||||
```
|
||||
|
||||
The endpoint accepts repeated `files` fields plus optional `project`, `doc_type_hint`, `note`, and `start_job` fields. It uses the same bearer token as `/mcp`; there is no second token to manage. For MCP clients that cannot make HTTP multipart requests, use `ingest_file_base64` for small files.
|
||||
The endpoint accepts repeated `files` fields plus optional `project`, `doc_type_hint`, `note`, and `start_job` fields. It accepts either the same bearer token as `/mcp` or the single-use upload ticket header. For MCP clients that cannot make HTTP multipart requests, `ingest_file_base64` remains available for small files, but large base64 arguments are fragile in many agent tool-call implementations.
|
||||
|
||||
Run `check_ingest_opencode_readiness` before the first remote upload. Use `run_smoke_test: true` when you want OpenCode itself to prove it can load the project `wiki-ingest` skill; otherwise the static preflight is also run automatically before every `start_ingest_job`.
|
||||
|
||||
|
||||
@@ -8,6 +8,7 @@ import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import secrets
|
||||
import subprocess
|
||||
import time
|
||||
import uuid
|
||||
@@ -21,6 +22,8 @@ from obsidian_wiki.file_assets import clean_config_value
|
||||
DEFAULT_INGEST_MAX_FILE_BYTES = 100 * 1024 * 1024
|
||||
DEFAULT_INGEST_MAX_CHUNK_BYTES = 4 * 1024 * 1024
|
||||
DEFAULT_INGEST_MAX_FILES_PER_JOB = 10
|
||||
DEFAULT_UPLOAD_TICKET_SECONDS = 30 * 60
|
||||
MAX_UPLOAD_TICKET_SECONDS = 24 * 60 * 60
|
||||
DEFAULT_OPENCODE_PATH = "opencode"
|
||||
DEFAULT_OPENCODE_SMOKE_TIMEOUT_SECONDS = 120
|
||||
|
||||
@@ -78,10 +81,12 @@ class IngestManager:
|
||||
self.max_files_per_job = max_files_per_job
|
||||
self.upload_root = self.runtime_root / "uploads"
|
||||
self.jobs_root = self.runtime_root / "jobs"
|
||||
self.upload_ticket_root = self.runtime_root / "upload-tickets"
|
||||
self._lock = RLock()
|
||||
self._processes: dict[str, subprocess.Popen[Any]] = {}
|
||||
self.upload_root.mkdir(parents=True, exist_ok=True)
|
||||
self.jobs_root.mkdir(parents=True, exist_ok=True)
|
||||
self.upload_ticket_root.mkdir(parents=True, exist_ok=True)
|
||||
self._mark_interrupted_jobs()
|
||||
|
||||
def create_upload(
|
||||
@@ -147,6 +152,56 @@ class IngestManager:
|
||||
self._write_json(self._upload_meta_path(path.parent), meta)
|
||||
return self.complete_upload(upload["upload_id"])
|
||||
|
||||
def create_upload_ticket(
|
||||
self,
|
||||
*,
|
||||
expires_seconds: int = DEFAULT_UPLOAD_TICKET_SECONDS,
|
||||
max_files: int | None = None,
|
||||
project: str | None = None,
|
||||
doc_type_hint: str | None = None,
|
||||
note: str | None = None,
|
||||
start_job: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
if expires_seconds <= 0 or expires_seconds > MAX_UPLOAD_TICKET_SECONDS:
|
||||
raise IngestError(
|
||||
f"expires_seconds must be between 1 and {MAX_UPLOAD_TICKET_SECONDS}"
|
||||
)
|
||||
file_limit = self.max_files_per_job if max_files is None else int(max_files)
|
||||
if file_limit <= 0 or file_limit > self.max_files_per_job:
|
||||
raise IngestError(f"max_files must be between 1 and {self.max_files_per_job}")
|
||||
token = secrets.token_urlsafe(32)
|
||||
token_hash = self._upload_ticket_hash(token)
|
||||
now = time.time()
|
||||
meta = {
|
||||
"token_hash": token_hash,
|
||||
"created_at": utc_now(),
|
||||
"expires_at": self._utc_from_epoch(now + expires_seconds),
|
||||
"expires_epoch": now + expires_seconds,
|
||||
"used_at": None,
|
||||
"max_files": file_limit,
|
||||
"project": self._safe_optional_text(project),
|
||||
"doc_type_hint": self._safe_optional_text(doc_type_hint),
|
||||
"note": self._safe_optional_text(note),
|
||||
"start_job": bool(start_job),
|
||||
}
|
||||
self._write_json(self._upload_ticket_path(token_hash), meta)
|
||||
return {**self._public_upload_ticket(meta), "upload_token": token}
|
||||
|
||||
def consume_upload_ticket(self, upload_token: str) -> dict[str, Any]:
|
||||
with self._lock:
|
||||
token_hash = self._upload_ticket_hash(upload_token)
|
||||
path = self._upload_ticket_path(token_hash)
|
||||
if not path.is_file():
|
||||
raise IngestError("upload ticket is invalid")
|
||||
meta = self._read_json(path)
|
||||
if meta.get("used_at"):
|
||||
raise IngestError("upload ticket has already been used")
|
||||
if time.time() > float(meta.get("expires_epoch", 0)):
|
||||
raise IngestError("upload ticket has expired")
|
||||
meta["used_at"] = utc_now()
|
||||
self._write_json(path, meta)
|
||||
return self._public_upload_ticket(meta)
|
||||
|
||||
def append_upload_chunk(self, upload_id: str, *, offset: int, data_base64: str) -> dict[str, Any]:
|
||||
with self._lock:
|
||||
meta = self._load_upload(upload_id)
|
||||
@@ -456,6 +511,17 @@ class IngestManager:
|
||||
"updated_at": meta["updated_at"],
|
||||
}
|
||||
|
||||
def _public_upload_ticket(self, meta: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"expires_at": meta["expires_at"],
|
||||
"used_at": meta.get("used_at"),
|
||||
"max_files": meta["max_files"],
|
||||
"project": meta.get("project"),
|
||||
"doc_type_hint": meta.get("doc_type_hint"),
|
||||
"note": meta.get("note"),
|
||||
"start_job": bool(meta.get("start_job", True)),
|
||||
}
|
||||
|
||||
def _public_job(self, job: dict[str, Any]) -> dict[str, Any]:
|
||||
return {
|
||||
"job_id": job["job_id"],
|
||||
@@ -662,6 +728,23 @@ class IngestManager:
|
||||
def _upload_meta_path(self, upload_dir: Path) -> Path:
|
||||
return upload_dir / "upload.json"
|
||||
|
||||
def _upload_ticket_path(self, token_hash: str) -> Path:
|
||||
return self.upload_ticket_root / f"{token_hash}.json"
|
||||
|
||||
def _upload_ticket_hash(self, upload_token: str) -> str:
|
||||
token = str(upload_token).strip()
|
||||
if not token:
|
||||
raise IngestError("upload ticket is missing")
|
||||
return hashlib.sha256(token.encode("utf-8")).hexdigest()
|
||||
|
||||
def _utc_from_epoch(self, epoch_seconds: float) -> str:
|
||||
return (
|
||||
datetime.fromtimestamp(epoch_seconds, tz=timezone.utc)
|
||||
.replace(microsecond=0)
|
||||
.isoformat()
|
||||
.replace("+00:00", "Z")
|
||||
)
|
||||
|
||||
def _read_json(self, path: Path) -> dict[str, Any]:
|
||||
return json.loads(path.read_text(encoding="utf-8"))
|
||||
|
||||
|
||||
+103
-12
@@ -20,7 +20,9 @@ from obsidian_wiki.mcp_ingest import (
|
||||
DEFAULT_INGEST_MAX_CHUNK_BYTES,
|
||||
DEFAULT_INGEST_MAX_FILE_BYTES,
|
||||
DEFAULT_INGEST_MAX_FILES_PER_JOB,
|
||||
DEFAULT_UPLOAD_TICKET_SECONDS,
|
||||
DEFAULT_OPENCODE_PATH,
|
||||
MAX_UPLOAD_TICKET_SECONDS,
|
||||
IngestError,
|
||||
IngestManager,
|
||||
parse_bool,
|
||||
@@ -94,16 +96,20 @@ def bearer_token(authorization: str) -> str:
|
||||
|
||||
|
||||
class BearerTokenAuthMiddleware:
|
||||
def __init__(self, app: Any, token: str) -> None:
|
||||
def __init__(self, app: Any, token: str, *, bypass_paths: set[str] | None = None) -> None:
|
||||
if not token:
|
||||
raise ValueError("HTTP MCP token must not be empty")
|
||||
self.app = app
|
||||
self.token = token
|
||||
self.bypass_paths = bypass_paths or set()
|
||||
|
||||
async def __call__(self, scope: dict[str, Any], receive: Any, send: Any) -> None:
|
||||
if scope.get("type") != "http":
|
||||
await self.app(scope, receive, send)
|
||||
return
|
||||
if scope.get("path") in self.bypass_paths:
|
||||
await self.app(scope, receive, send)
|
||||
return
|
||||
headers = {key.lower(): value for key, value in scope.get("headers", [])}
|
||||
authorization = headers.get(b"authorization", b"").decode("latin1")
|
||||
token = bearer_token(authorization)
|
||||
@@ -220,7 +226,10 @@ def create_server(
|
||||
"method": "POST",
|
||||
"path": DEFAULT_INGEST_UPLOAD_PATH,
|
||||
"file_field": "files",
|
||||
"auth": "same Authorization: Bearer <token> header as the MCP endpoint",
|
||||
"auth": (
|
||||
"use the same Authorization: Bearer <token> header as the MCP endpoint, "
|
||||
"or call create_ingest_upload_ticket and send X-Obsidian-Wiki-Upload-Token"
|
||||
),
|
||||
"optional_fields": ["project", "doc_type_hint", "note", "start_job"],
|
||||
"start_job_default": True,
|
||||
"curl_example": (
|
||||
@@ -230,6 +239,15 @@ def create_server(
|
||||
"http://<host>:<port>/mcp-ingest/uploads"
|
||||
),
|
||||
},
|
||||
"upload_ticket_tool": {
|
||||
"name": "create_ingest_upload_ticket",
|
||||
"token_header": "X-Obsidian-Wiki-Upload-Token",
|
||||
"best_for": "remote agents that can run curl but cannot read the main MCP bearer token",
|
||||
"curl_example": (
|
||||
"curl -H 'X-Obsidian-Wiki-Upload-Token: <upload-token>' "
|
||||
"-F 'files=@drawing.pdf' http://<host>:<port>/mcp-ingest/uploads"
|
||||
),
|
||||
},
|
||||
"mcp_base64_tool": {
|
||||
"name": "ingest_file_base64",
|
||||
"best_for": "small files or MCP clients that cannot make HTTP multipart requests",
|
||||
@@ -270,6 +288,36 @@ def create_server(
|
||||
job = ingest_manager.start_ingest_job([upload["upload_id"]], note=note) if start_job else None
|
||||
return {"upload": upload, "job": job}
|
||||
|
||||
@server.tool()
|
||||
def create_ingest_upload_ticket(
|
||||
expires_seconds: int = DEFAULT_UPLOAD_TICKET_SECONDS,
|
||||
max_files: int | None = None,
|
||||
project: str | None = None,
|
||||
doc_type_hint: str | None = None,
|
||||
note: str | None = None,
|
||||
start_job: bool = True,
|
||||
) -> dict[str, Any]:
|
||||
"""Create a short-lived HTTP upload ticket so remote agents can upload files without the main MCP token."""
|
||||
ticket = ingest_manager.create_upload_ticket(
|
||||
expires_seconds=expires_seconds,
|
||||
max_files=max_files,
|
||||
project=project,
|
||||
doc_type_hint=doc_type_hint,
|
||||
note=note,
|
||||
start_job=start_job,
|
||||
)
|
||||
return {
|
||||
**ticket,
|
||||
"upload_path": DEFAULT_INGEST_UPLOAD_PATH,
|
||||
"token_header": "X-Obsidian-Wiki-Upload-Token",
|
||||
"expires_seconds_max": MAX_UPLOAD_TICKET_SECONDS,
|
||||
"curl_example": (
|
||||
"curl -H 'X-Obsidian-Wiki-Upload-Token: "
|
||||
+ ticket["upload_token"]
|
||||
+ "' -F 'files=@/path/to/file.pdf' http://<host>:<port>/mcp-ingest/uploads"
|
||||
),
|
||||
}
|
||||
|
||||
@server.tool()
|
||||
def start_ingest_job(upload_ids: list[str], note: str | None = None) -> dict[str, Any]:
|
||||
"""Start or queue a local OpenCode wiki-ingest job for completed uploads."""
|
||||
@@ -424,15 +472,22 @@ class IngestUploadRouter:
|
||||
*,
|
||||
ingest_manager: IngestManager | None = None,
|
||||
upload_path: str = DEFAULT_INGEST_UPLOAD_PATH,
|
||||
bearer_auth_token: str = "",
|
||||
) -> None:
|
||||
self.app = app
|
||||
self.ingest_manager = ingest_manager
|
||||
self.upload_path = upload_path
|
||||
self.bearer_auth_token = bearer_auth_token
|
||||
|
||||
async def __call__(self, scope: dict[str, Any], receive: Any, send: Any) -> None:
|
||||
if scope.get("type") != "http" or scope.get("path") != self.upload_path:
|
||||
await self.app(scope, receive, send)
|
||||
return
|
||||
try:
|
||||
upload_ticket = self._authorize_upload(scope)
|
||||
except IngestError:
|
||||
await _send_json(send, 401, {"error": "unauthorized"})
|
||||
return
|
||||
if scope.get("method", "").upper() != "POST":
|
||||
await _send_json(send, 405, {"error": "method_not_allowed"})
|
||||
return
|
||||
@@ -440,13 +495,33 @@ class IngestUploadRouter:
|
||||
await _send_json(send, 404, {"error": "ingest_not_enabled"})
|
||||
return
|
||||
try:
|
||||
data = await self._handle_upload(scope, receive)
|
||||
data = await self._handle_upload(scope, receive, upload_ticket=upload_ticket)
|
||||
except IngestError as exc:
|
||||
await _send_json(send, 400, {"error": str(exc)})
|
||||
return
|
||||
await _send_json(send, 200, data)
|
||||
|
||||
async def _handle_upload(self, scope: dict[str, Any], receive: Any) -> dict[str, Any]:
|
||||
def _authorize_upload(self, scope: dict[str, Any]) -> dict[str, Any] | None:
|
||||
authorization = _header_value(scope, b"authorization")
|
||||
token = bearer_token(authorization)
|
||||
if self.bearer_auth_token and hmac.compare_digest(token, self.bearer_auth_token):
|
||||
return None
|
||||
if self.ingest_manager is None:
|
||||
raise IngestError("unauthorized")
|
||||
upload_token = _header_value(scope, b"x-obsidian-wiki-upload-token")
|
||||
if not upload_token:
|
||||
upload_token = _query_param(scope, "upload_token")
|
||||
if not upload_token:
|
||||
raise IngestError("unauthorized")
|
||||
return self.ingest_manager.consume_upload_ticket(upload_token)
|
||||
|
||||
async def _handle_upload(
|
||||
self,
|
||||
scope: dict[str, Any],
|
||||
receive: Any,
|
||||
*,
|
||||
upload_ticket: dict[str, Any] | None = None,
|
||||
) -> dict[str, Any]:
|
||||
body = await _read_http_body(receive)
|
||||
content_type = _header_value(scope, b"content-type")
|
||||
fields, files = _parse_multipart_form(content_type, body)
|
||||
@@ -455,12 +530,14 @@ class IngestUploadRouter:
|
||||
manager = self.ingest_manager
|
||||
if manager is None:
|
||||
raise IngestError("ingest is not enabled")
|
||||
if len(files) > manager.max_files_per_job:
|
||||
raise IngestError(f"too many files for one ingest job: {len(files)} > {manager.max_files_per_job}")
|
||||
project = _first_field(fields, "project")
|
||||
doc_type_hint = _first_field(fields, "doc_type_hint")
|
||||
note = _first_field(fields, "note")
|
||||
start_job = parse_bool(_first_field(fields, "start_job"), default=True)
|
||||
max_files = int(upload_ticket.get("max_files", manager.max_files_per_job)) if upload_ticket else manager.max_files_per_job
|
||||
if len(files) > max_files:
|
||||
raise IngestError(f"too many files for one ingest job: {len(files)} > {max_files}")
|
||||
project = _first_field(fields, "project") or (upload_ticket or {}).get("project")
|
||||
doc_type_hint = _first_field(fields, "doc_type_hint") or (upload_ticket or {}).get("doc_type_hint")
|
||||
note = _first_field(fields, "note") or (upload_ticket or {}).get("note")
|
||||
start_default = bool((upload_ticket or {}).get("start_job", True))
|
||||
start_job = parse_bool(_first_field(fields, "start_job"), default=start_default)
|
||||
uploads = [
|
||||
manager.create_completed_upload(
|
||||
file_name=file["file_name"],
|
||||
@@ -482,8 +559,13 @@ def create_http_app(
|
||||
upload_path: str = DEFAULT_INGEST_UPLOAD_PATH,
|
||||
) -> Any:
|
||||
app = server.streamable_http_app() if hasattr(server, "streamable_http_app") else server
|
||||
app = IngestUploadRouter(app, ingest_manager=ingest_manager, upload_path=upload_path)
|
||||
return BearerTokenAuthMiddleware(app, token)
|
||||
app = IngestUploadRouter(
|
||||
app,
|
||||
ingest_manager=ingest_manager,
|
||||
upload_path=upload_path,
|
||||
bearer_auth_token=token,
|
||||
)
|
||||
return BearerTokenAuthMiddleware(app, token, bypass_paths={upload_path})
|
||||
|
||||
|
||||
async def _read_http_body(receive: Any) -> bytes:
|
||||
@@ -505,6 +587,15 @@ def _header_value(scope: dict[str, Any], name: bytes) -> str:
|
||||
return headers.get(name.lower(), b"").decode("latin1")
|
||||
|
||||
|
||||
def _query_param(scope: dict[str, Any], name: str) -> str:
|
||||
from urllib.parse import parse_qs
|
||||
|
||||
raw = scope.get("query_string", b"")
|
||||
query = raw.decode("utf-8", errors="replace") if isinstance(raw, bytes) else str(raw)
|
||||
values = parse_qs(query).get(name, [])
|
||||
return values[0] if values else ""
|
||||
|
||||
|
||||
def _parse_multipart_form(content_type: str, body: bytes) -> tuple[dict[str, list[str]], list[dict[str, Any]]]:
|
||||
if not content_type.lower().startswith("multipart/form-data"):
|
||||
raise IngestError("content-type must be multipart/form-data")
|
||||
|
||||
@@ -185,6 +185,31 @@ class McpIngestTests(unittest.TestCase):
|
||||
self.assertEqual(upload["doc_type_hint"], "drawing")
|
||||
self.assertEqual(Path(upload["path"]).read_bytes(), payload)
|
||||
|
||||
def test_upload_ticket_is_single_use_and_preserves_defaults(self):
|
||||
ticket = self.manager.create_upload_ticket(
|
||||
expires_seconds=60,
|
||||
max_files=1,
|
||||
project="bjd",
|
||||
doc_type_hint="drawing",
|
||||
note="customer uploaded drawing",
|
||||
start_job=False,
|
||||
)
|
||||
|
||||
self.assertIn("upload_token", ticket)
|
||||
self.assertEqual(ticket["max_files"], 1)
|
||||
self.assertEqual(ticket["project"], "bjd")
|
||||
self.assertEqual(ticket["doc_type_hint"], "drawing")
|
||||
self.assertEqual(ticket["note"], "customer uploaded drawing")
|
||||
self.assertFalse(ticket["start_job"])
|
||||
self.assertEqual(ticket["used_at"], None)
|
||||
|
||||
used = self.manager.consume_upload_ticket(ticket["upload_token"])
|
||||
|
||||
self.assertEqual(used["project"], "bjd")
|
||||
self.assertIsNotNone(used["used_at"])
|
||||
with self.assertRaises(IngestError):
|
||||
self.manager.consume_upload_ticket(ticket["upload_token"])
|
||||
|
||||
def test_upload_rejects_unsafe_names_oversized_files_and_large_chunks(self):
|
||||
with self.assertRaises(IngestError):
|
||||
self.manager.create_upload(file_name="../secret.pdf", size_bytes=10, sha256="a" * 64)
|
||||
|
||||
@@ -196,6 +196,59 @@ class RemoteMcpServerTests(unittest.TestCase):
|
||||
self.assertEqual(data["uploads"][0]["file_name"], "drawing.pdf")
|
||||
self.assertEqual(data["uploads"][0]["sha256"], "sha256:" + hashlib.sha256(payload).hexdigest())
|
||||
|
||||
def test_http_upload_endpoint_accepts_upload_ticket_without_main_bearer_token(self):
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
root = Path(tmp)
|
||||
manager = IngestManager(
|
||||
vault_path=root / "vault",
|
||||
project_root=root / "project",
|
||||
runtime_root=root / "runtime",
|
||||
)
|
||||
ticket = manager.create_upload_ticket(
|
||||
expires_seconds=60,
|
||||
max_files=1,
|
||||
project="bjd",
|
||||
doc_type_hint="drawing",
|
||||
start_job=False,
|
||||
)
|
||||
app = mcp_server.create_http_app(FakeApp(), "secret-token", ingest_manager=manager)
|
||||
payload = b"ticket uploaded drawing"
|
||||
body, boundary = multipart_body(
|
||||
{},
|
||||
[("files", "ticket-drawing.pdf", "application/pdf", payload)],
|
||||
)
|
||||
|
||||
messages = run_asgi_http_request(
|
||||
app,
|
||||
method="POST",
|
||||
path="/mcp-ingest/uploads",
|
||||
headers=[
|
||||
(b"x-obsidian-wiki-upload-token", ticket["upload_token"].encode("ascii")),
|
||||
(b"content-type", f"multipart/form-data; boundary={boundary}".encode("ascii")),
|
||||
],
|
||||
body=body,
|
||||
)
|
||||
|
||||
reused = run_asgi_http_request(
|
||||
app,
|
||||
method="POST",
|
||||
path="/mcp-ingest/uploads",
|
||||
headers=[
|
||||
(b"x-obsidian-wiki-upload-token", ticket["upload_token"].encode("ascii")),
|
||||
(b"content-type", f"multipart/form-data; boundary={boundary}".encode("ascii")),
|
||||
],
|
||||
body=body,
|
||||
)
|
||||
|
||||
self.assertEqual(messages[0]["status"], 200)
|
||||
response = b"".join(message.get("body", b"") for message in messages if message["type"] == "http.response.body")
|
||||
data = json.loads(response.decode("utf-8"))
|
||||
self.assertEqual(data["job"], None)
|
||||
self.assertEqual(data["uploads"][0]["project"], "bjd")
|
||||
self.assertEqual(data["uploads"][0]["doc_type_hint"], "drawing")
|
||||
self.assertEqual(data["uploads"][0]["sha256"], "sha256:" + hashlib.sha256(payload).hexdigest())
|
||||
self.assertEqual(reused[0]["status"], 401)
|
||||
|
||||
def test_http_upload_endpoint_requires_ingest_and_bearer_token(self):
|
||||
no_ingest_app = mcp_server.create_http_app(FakeApp(), "secret-token")
|
||||
missing_token = run_asgi_http_request(no_ingest_app, method="POST", path="/mcp-ingest/uploads")
|
||||
@@ -300,8 +353,60 @@ class RemoteMcpServerTests(unittest.TestCase):
|
||||
self.assertIn("cancel_ingest_job", FakeMCP.tool_names)
|
||||
self.assertIn("check_ingest_opencode_readiness", FakeMCP.tool_names)
|
||||
self.assertIn("get_ingest_upload_instructions", FakeMCP.tool_names)
|
||||
self.assertIn("create_ingest_upload_ticket", FakeMCP.tool_names)
|
||||
self.assertIn("ingest_file_base64", FakeMCP.tool_names)
|
||||
|
||||
def test_create_ingest_upload_ticket_tool_returns_http_only_token(self):
|
||||
class FakeMCP:
|
||||
tools = {}
|
||||
|
||||
def __init__(self, _name, **_kwargs):
|
||||
pass
|
||||
|
||||
def tool(self):
|
||||
def decorator(func):
|
||||
self.tools[func.__name__] = func
|
||||
return func
|
||||
|
||||
return decorator
|
||||
|
||||
def resource(self, *_args, **_kwargs):
|
||||
def decorator(func):
|
||||
return func
|
||||
|
||||
return decorator
|
||||
|
||||
def prompt(self, *_args, **_kwargs):
|
||||
def decorator(func):
|
||||
return func
|
||||
|
||||
return decorator
|
||||
|
||||
with (
|
||||
tempfile.TemporaryDirectory() as tmp,
|
||||
patch.dict("sys.modules", {"mcp.server.fastmcp": type("Mod", (), {"FastMCP": FakeMCP})()}),
|
||||
):
|
||||
root = Path(tmp)
|
||||
mcp_server.create_server(
|
||||
vault_path=root / "vault",
|
||||
enable_ingest=True,
|
||||
project_root=root / "project",
|
||||
ingest_runtime_root=root / "runtime",
|
||||
)
|
||||
ticket = FakeMCP.tools["create_ingest_upload_ticket"](
|
||||
expires_seconds=60,
|
||||
max_files=1,
|
||||
project="bjd",
|
||||
doc_type_hint="drawing",
|
||||
start_job=False,
|
||||
)
|
||||
|
||||
self.assertIn("upload_token", ticket)
|
||||
self.assertEqual(ticket["token_header"], "X-Obsidian-Wiki-Upload-Token")
|
||||
self.assertEqual(ticket["upload_path"], "/mcp-ingest/uploads")
|
||||
self.assertEqual(ticket["max_files"], 1)
|
||||
self.assertFalse(ticket["start_job"])
|
||||
|
||||
def test_ingest_file_base64_tool_uploads_without_chunking(self):
|
||||
class FakeMCP:
|
||||
tools = {}
|
||||
|
||||
Reference in New Issue
Block a user