diff --git a/docs/superpowers/plans/2026-07-12-project-write-and-upload.md b/docs/superpowers/plans/2026-07-12-project-write-and-upload.md new file mode 100644 index 0000000..d1551f1 --- /dev/null +++ b/docs/superpowers/plans/2026-07-12-project-write-and-upload.md @@ -0,0 +1,347 @@ +# 项目写入与附件上传 Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (\`- [ ]\`) syntax for tracking. + +**Goal:** Add fixed-schema project creation and update MCP tools plus an authenticated multipart upload endpoint that stores files in NocoBase and incrementally associates them with projects. + +**Architecture:** Preserve all existing read-only behavior by default. A focused \`ProjectService\` validates and maps public project inputs, \`NbClient\` writes scalar project data through the existing CLI, and \`NocoBaseHttpClient\` performs the JWT-only multipart upload and association actions. The existing bearer middleware wraps both \`/mcp\` and a new \`/uploads\` route. + +**Tech Stack:** Python 3.12, FastMCP, Starlette, httpx, python-multipart, NocoBase CLI/HTTP API, unittest. + +## Global Constraints + +- Scope all writes to \`t_projects\`; never accept a caller-selected collection or field. +- Support exactly the confirmed fields, create/update, and incremental attachment add/remove. Do not implement deletes, replacement attachment lists, or orphan cleanup. +- Creation requires nonblank \`name\`; dates are \`YYYY-MM-DD\`; valid progress labels are \`询价阶段\`、\`已报价\`、\`送样\`、\`已定点\`、\`SOP\`、\`终止\`. +- \`POST /uploads\` accepts exactly one \`multipart/form-data\` \`file\` part, with a 25 MiB default limit. +- Use \`NOCOBASE_MCP_TOKEN\` for the public service and keep NocoBase upload credentials in \`NOCOBASE_MCP_UPLOAD_API_BASE_URL\` and \`NOCOBASE_MCP_UPLOAD_JWT\`. +- Do not log files, authorization headers, request bodies, or JWTs. Always delete temporary files in \`finally\`. +- Keep stdio and \`create_server(..., readonly=True)\` read-only; enable write tools and \`/uploads\` only for the HTTP runtime's explicit write mode. +- Test only against fake CLI runners and mocked HTTP transports. + +--- + +## File structure + +| File | Responsibility | +| --- | --- | +| \`pyproject.toml\` | Declare direct HTTP and multipart dependencies. | +| \`nocobase_crm_mcp/config.py\` | Parse fixed \`projects\` mapping. | +| \`nocobase_crm_mcp/nb_client.py\` | Construct safe scalar create/update CLI calls. | +| \`nocobase_crm_mcp/nocobase_http.py\` | Upload an attachment and invoke NocoBase association add/remove actions. | +| \`nocobase_crm_mcp/projects.py\` | Validate/mapping/orchestration for the two public project tools. | +| \`nocobase_crm_mcp/uploads.py\` | One-file multipart endpoint and temporary-file lifecycle. | +| \`nocobase_crm_mcp/server.py\` | Write-mode tool and route composition. | +| \`scripts/manage_mcp_service.py\` | Generate only the non-secret upload size setting in the private env file. | +| \`config.example.json\`, \`README.md\` | Fixed mapping, ACL, JWT, endpoint, and client-flow documentation. | +| \`tests/test_*.py\` | Unit and ASGI coverage without real NocoBase writes. | + +### Task 1: Fixed project configuration and NocoBase HTTP client + +**Files:** +- Modify: \`pyproject.toml:6-9\` +- Modify: \`nocobase_crm_mcp/config.py:1-173\` +- Create: \`nocobase_crm_mcp/nocobase_http.py\` +- Create: \`tests/test_nocobase_http.py\` + +**Interfaces:** +- Produces \`ProjectConfig\` with resource/field mapping and \`progress_values: dict[str, str]\`. +- Produces \`NocoBaseHttpClient(api_base_url, jwt, timeout=30.0, transport=None)\`. +- Produces \`upload_attachment(path, filename, mimetype) -> dict[str, Any]\`, \`add_attachment_ids(project_id, attachment_ids) -> None\`, and \`remove_attachment_ids(project_id, attachment_ids) -> None\`. + +- [ ] **Step 1: Write failing configuration and HTTP client tests** + +~~~python +def test_load_config_parses_project_defaults(tmp_path: Path) -> None: + path = tmp_path / "config.json" + path.write_text(json.dumps({"projects": {"resource": "t_projects"}}), encoding="utf-8") + config = load_config(path) + assert config.project.name_field == "f_project_name" + assert config.project.progress_values["SOP"] == "a9t9awhtt3r" + +def test_upload_posts_file_with_attachment_field(tmp_path: Path) -> None: + seen = {} + def handler(request: httpx.Request) -> httpx.Response: + seen["url"] = str(request.url) + seen["auth"] = request.headers["authorization"] + return httpx.Response(200, json={"data": {"id": 42}}) + path = tmp_path / "quote.pdf" + path.write_bytes(b"pdf") + client = NocoBaseHttpClient("https://nb.example/api", "jwt", transport=httpx.MockTransport(handler)) + assert client.upload_attachment(path, "quote.pdf", "application/pdf")["id"] == 42 + assert seen["url"].endswith("/attachments:create?attachmentField=t_projects.f_projects_attachments") + assert seen["auth"] == "Bearer jwt" +~~~ + +- [ ] **Step 2: Run the test to verify it fails** + +Run: \`uv run python -m unittest tests.test_nocobase_http -v\` + +Expected: FAIL because \`ProjectConfig\` and \`NocoBaseHttpClient\` do not exist. + +- [ ] **Step 3: Implement the configuration and client** + +Add \`httpx>=0.27,<1\` and \`python-multipart>=0.0.9,<1\` to \`pyproject.toml\`. Add frozen \`ProjectConfig\` to \`config.py\` and add \`project: ProjectConfig = ProjectConfig()\` to \`CrmConfig\`. Parse only the \`projects\` config block; do not accept MCP input for its values. + +Use these exact progress values: + +~~~python +PROJECT_PROGRESS_VALUES = { + "询价阶段": "fk9xj8akn46", "已报价": "k4w2vcz819w", "送样": "g95vqeetu83", + "已定点": "fe0peg505d7", "SOP": "a9t9awhtt3r", "终止": "a9t9awhtsfasg3r", +} +~~~ + +Implement \`NocoBaseHttpClient\` with a synchronous \`httpx.Client\`. It strips a trailing slash from the API URL, rejects blank URL/JWT, posts multipart \`files={"file": (filename, handle, mimetype or "application/octet-stream")}\` to \`/attachments:create\` with \`params={"attachmentField": "t_projects.f_projects_attachments"}\`, and posts JSON \`{"tk": attachment_ids}\` to \`/t_projects/{project_id}/f_projects_attachments:add\` or \`:remove\`. Raise \`NocoBaseHttpError(status_code, message)\` for non-2xx responses after redacting the JWT. + +- [ ] **Step 4: Run tests and commit** + +Run: \`uv run python -m unittest tests.test_nocobase_http -v\` + +Expected: PASS. + +~~~bash +git add pyproject.toml nocobase_crm_mcp/config.py nocobase_crm_mcp/nocobase_http.py tests/test_nocobase_http.py +git commit -m "feat: add project config and NocoBase HTTP client" +~~~ + +### Task 2: Scalar project writes through the existing CLI client + +**Files:** +- Modify: \`nocobase_crm_mcp/nb_client.py:50-154\` +- Modify: \`tests/test_nb_client.py:1-160\` + +**Interfaces:** +- Produces \`NbClient.create_record(resource, values, whitelist) -> dict[str, Any]\`. +- Produces \`NbClient.update_record(resource, record_id, values, whitelist) -> dict[str, Any]\`. + +- [ ] **Step 1: Write failing CLI command tests** + +~~~python +def test_create_record_uses_values_and_whitelist(self) -> None: + client = NbClient(env_name="nb-mcp", runner=runner) + client.create_record("t_projects", {"f_project_name": "EV inverter"}, ["f_project_name"]) + assert commands[0][:4] == ["nb", "api", "resource", "create"] + assert json.loads(commands[0][commands[0].index("--values") + 1]) == {"f_project_name": "EV inverter"} + assert commands[0][commands[0].index("--whitelist") + 1] == "f_project_name" + +def test_update_record_uses_filter_by_tk(self) -> None: + client = NbClient(env_name="nb-mcp", runner=runner) + client.update_record("t_projects", 7, {"client_ID": 3}, ["client_ID"]) + assert commands[0][:4] == ["nb", "api", "resource", "update"] + assert commands[0][commands[0].index("--filter-by-tk") + 1] == "7" +~~~ + +- [ ] **Step 2: Run the test to verify it fails** + +Run: \`uv run python -m unittest tests.test_nb_client -v\` + +Expected: FAIL because the two methods are absent. + +- [ ] **Step 3: Implement safe write methods** + +Use \`_resource_command("create", resource)\` or \`_resource_command("update", resource)\`, then \`_append_json(command, "--values", values)\` and \`_append_repeated(command, "--whitelist", whitelist)\`. For update, add \`--filter-by-tk\` via \`_append_json_or_scalar\`. Reject an empty values mapping before launching a subprocess. + +- [ ] **Step 4: Run tests and commit** + +Run: \`uv run python -m unittest tests.test_nb_client -v\` + +Expected: PASS. + +~~~bash +git add nocobase_crm_mcp/nb_client.py tests/test_nb_client.py +git commit -m "feat: add NocoBase project record writes" +~~~ + +### Task 3: Fixed-schema project service + +**Files:** +- Create: \`nocobase_crm_mcp/projects.py\` +- Create: \`tests/test_projects.py\` + +**Interfaces:** +- Consumes \`ProjectConfig\`, the two \`NbClient\` write methods, and \`NocoBaseHttpClient\` association methods. +- Produces \`ProjectService.create_project(...) -> dict[str, Any]\` and \`ProjectService.update_project(...) -> dict[str, Any]\`. +- Produces structured error codes \`invalid_project_input\`, \`invalid_progress\`, \`attachment_not_found\`, \`nb_upload_unauthorized\`, and \`nb_upload_failed\`. + +- [ ] **Step 1: Write failing service tests** + +~~~python +def test_create_maps_fixed_fields_then_adds_attachment_ids() -> None: + result = service.create_project(name=" EV inverter ", customer_id=3, progress="SOP", attachment_ids=[8, 9]) + assert nb.create_calls == [("t_projects", {"f_project_name": "EV inverter", "client_ID": 3, "f_projects_progress": "a9t9awhtt3r"}, PROJECT_WHITELIST)] + assert http.add_calls == [(41, [8, 9])] + assert result["project_id"] == 41 + +def test_update_rejects_overlapping_attachment_lists_before_writing() -> None: + result = service.update_project(41, add_attachment_ids=[8], remove_attachment_ids=[8]) + assert result["error"]["code"] == "invalid_project_input" + assert nb.update_calls == [] + +def test_invalid_date_and_progress_have_stable_codes() -> None: + assert service.create_project(name="x", inquiry_date="2026/07/12")["error"]["code"] == "invalid_project_input" + assert service.create_project(name="x", progress="草稿")["error"]["code"] == "invalid_progress" +~~~ + +- [ ] **Step 2: Run the test to verify it fails** + +Run: \`uv run python -m unittest tests.test_projects -v\` + +Expected: FAIL with \`ModuleNotFoundError: nocobase_crm_mcp.projects\`. + +- [ ] **Step 3: Implement the service** + +Define \`PROJECT_WHITELIST\` from the eight configured scalar fields. \`_project_values()\` trims name, maps public names to configured field names, parses dates with \`datetime.date.fromisoformat\`, and maps progress labels from \`config.progress_values\`. It must not produce any attachment field value. + +Creation rejects blank names, creates scalar values, extracts the configured ID, then adds deduplicated attachment IDs. Update rejects an empty request and overlapping add/remove sets before scalar writes; it updates scalars, then adds IDs, then removes IDs. If association work fails after a scalar write, return the project ID plus a structured attachment failure; never roll back the project. + +- [ ] **Step 4: Run tests and commit** + +Run: \`uv run python -m unittest tests.test_projects -v\` + +Expected: PASS. + +~~~bash +git add nocobase_crm_mcp/projects.py tests/test_projects.py +git commit -m "feat: add fixed project write service" +~~~ + +### Task 4: One-file multipart upload endpoint + +**Files:** +- Create: \`nocobase_crm_mcp/uploads.py\` +- Create: \`tests/test_uploads.py\` + +**Interfaces:** +- Produces \`UploadEndpoint(http_client, max_bytes, temp_dir, allowed_mime_types=None)\`. +- Produces \`await UploadEndpoint(request) -> JSONResponse\`. +- Success JSON is exactly \`{"upload_id": int, "filename": str, "size": int, "mimetype": str | null}\`. + +- [ ] **Step 1: Write failing ASGI tests** + +~~~python +async def test_upload_returns_nocobase_attachment_id() -> None: + response = await client.post("/uploads", headers={"Authorization": "Bearer secret"}, + files={"file": ("quote.pdf", b"pdf", "application/pdf")}) + assert response.status_code == 200 + assert response.json()["upload_id"] == 42 + +async def test_upload_rejects_missing_file_and_over_limit() -> None: + assert (await client.post("/uploads", headers=AUTH, files={})).status_code == 400 + response = await client.post("/uploads", headers=AUTH, + files={"file": ("big.bin", b"x" * 6, "application/octet-stream")}) + assert response.status_code == 413 +~~~ + +Inject a fake HTTP client and a temporary directory; assert it is empty after successful upload and upstream failure. + +- [ ] **Step 2: Run the test to verify it fails** + +Run: \`uv run python -m unittest tests.test_uploads -v\` + +Expected: FAIL because \`UploadEndpoint\` is absent. + +- [ ] **Step 3: Implement the endpoint** + +Require POST and multipart form data. Require exactly one Starlette \`UploadFile\` named \`file\`. Copy it in chunks to \`tempfile.NamedTemporaryFile(delete=False, dir=temp_dir)\`, count bytes while copying, and return 413 \`upload_too_large\` when over limit. If configured, compare MIME values using \`fnmatch\` and return 415 \`unsupported_media_type\`. Call \`http_client.upload_attachment\`; map 401/403 to 502 \`nb_upload_unauthorized\` and other upstream failures to 502 \`nb_upload_failed\`. Unlink the path in \`finally\` on every path. + +- [ ] **Step 4: Run tests and commit** + +Run: \`uv run python -m unittest tests.test_uploads -v\` + +Expected: PASS. + +~~~bash +git add nocobase_crm_mcp/uploads.py tests/test_uploads.py +git commit -m "feat: add authenticated attachment uploads" +~~~ + +### Task 5: Server wiring, private service configuration, and documentation + +**Files:** +- Modify: \`nocobase_crm_mcp/server.py:91-367\` +- Modify: \`scripts/manage_mcp_service.py:1-180\` +- Modify: \`config.example.json:1-126\` +- Modify: \`README.md:1-151\` +- Modify: \`tests/test_server.py:1-180\` +- Modify: \`tests/test_mcp_service.py:1-180\` + +**Interfaces:** +- Produces \`create_server(..., readonly=False, project_service=...)\` with MCP tools \`create_project\` and \`update_project\`. +- Produces \`create_http_app(server, token, upload_endpoint=None)\` where bearer authentication wraps \`/uploads\` and \`/mcp\`. + +- [ ] **Step 1: Write failing integration tests** + +~~~python +def test_write_server_exposes_fixed_project_tools() -> None: + server = create_server(crm_service=FakeCrmService(), project_service=FakeProjectService(), readonly=False) + names = {tool.name for tool in asyncio.run(server.list_tools())} + assert {"create_project", "update_project"}.issubset(names) + assert "destroy_project" not in names + +def test_upload_route_requires_the_same_bearer_token() -> None: + app = create_http_app(server, "secret", upload_endpoint=FakeUploadEndpoint()) + assert run_asgi_request(app, path="/uploads")[0]["status"] == 401 + +def test_generated_env_has_size_but_no_upload_jwt(tmp_path: Path) -> None: + env_path = tmp_path / "mcp.env" + manage_mcp_service.write_token_env_file(env_path, "token") + content = env_path.read_text(encoding="utf-8") + assert "NOCOBASE_MCP_UPLOAD_MAX_BYTES=26214400" in content + assert "NOCOBASE_MCP_UPLOAD_JWT" not in content +~~~ + +- [ ] **Step 2: Run the test to verify it fails** + +Run: \`uv run python -m unittest tests.test_server tests.test_mcp_service -v\` + +Expected: FAIL because write tools and upload route are absent. + +- [ ] **Step 3: Implement composition without changing read-only defaults** + +When \`readonly=True\`, retain the exact existing read tool set. When false, require a project service and register only \`create_project\` and \`update_project\` using existing \`_safe_tool_result\`. + +Build a Starlette app with \`Route("/uploads", upload_endpoint, methods=["POST"])\` before \`Mount("/", app=server.streamable_http_app())\`, then pass the combined app through \`BearerTokenAuthMiddleware\`. In \`run_http_server\`, create \`NocoBaseHttpClient\` and \`UploadEndpoint\` from the required upload environment variables and call write mode. Leave stdio read-only. + +Have \`write_token_env_file\` add only \`NOCOBASE_MCP_UPLOAD_MAX_BYTES=26214400\`; operators add API URL and JWT themselves to the already-0600 env file. + +- [ ] **Step 4: Update configuration and README** + +Add the approved fixed \`projects\` block to \`config.example.json\`. Document the two tools, \`POST /uploads\`, 25 MiB default, JWT environment variables, required ACLs (\`t_projects\` create/update, \`attachments\` upload/create, and attachment add/remove), retained orphan attachments, and absence of delete operations. + +Include this operational sequence: + +~~~bash +NOCOBASE_MCP_UPLOAD_API_BASE_URL='https://nocobase.example/api' +NOCOBASE_MCP_UPLOAD_JWT='' +bash scripts/mcp_service.sh start +curl -X POST -H "Authorization: Bearer " -F "file=@quote.pdf" http://:10087/uploads +~~~ + +- [ ] **Step 5: Run focused and full verification** + +Run: \`uv run python -m unittest tests.test_nb_client tests.test_nocobase_http tests.test_projects tests.test_uploads tests.test_server tests.test_mcp_service -v\` + +Expected: PASS. + +Run: \`uv run python -m unittest discover -s tests -v\` + +Expected: PASS with no real NocoBase request, project write, or attachment upload. + +- [ ] **Step 6: Inspect and commit** + +Run: \`git diff --check\` + +Expected: no output. + +~~~bash +git add nocobase_crm_mcp/server.py scripts/manage_mcp_service.py config.example.json README.md tests/test_server.py tests/test_mcp_service.py +git commit -m "feat: expose project writes and upload endpoint" +~~~ + +## Plan self-review + +- **Spec coverage:** Tasks 1–3 cover mapping, field validation, label mapping, scalar creates/updates, incremental attachments, and structured failures. Task 4 covers the one-file authenticated upload, 25 MiB default, redaction, and temporary cleanup. Task 5 covers bearer-route composition, read-only compatibility, deployment configuration, ACL documentation, and full verification. +- **Placeholder scan:** The plan names all files, interfaces, error paths, checks, and commands; it contains no unresolved requirements. +- **Type consistency:** Project inputs enter only through \`ProjectService\`; scalar values flow to \`NbClient\`; uploads and association IDs flow to \`NocoBaseHttpClient\`; \`UploadEndpoint\` and \`server.py\` only compose those components. +