diff --git a/AGENTS.md b/AGENTS.md index dfa97da..4dd779b 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -124,6 +124,13 @@ See `wiki-query` and `wiki-export` skills for how the filter is applied. - **Write complete commit messages.** Use Conventional Commits (`type(scope): summary`) and include a body when useful that explains the reason for the change, user-visible impact, validation performed, and any known follow-up. - **Commit only intended files.** Before committing, inspect `git status` and stage only files that belong to the current feature or maintenance task. +## File Asset Indexing + +- **Index source files as richly as possible.** When writing `_meta/source-registry.jsonl` or rebuilding `_meta/file-assets.sqlite`, preserve structured identifiers instead of relying only on filenames or summaries. +- **Keep product lookup fields explicit.** Carry forward `bjd_part_numbers`, `competitor_part_numbers`, `series`, `pitch`, `pin_count`, `product_lines`, `customers`, `competitors`, `standards`, `business_categories`, `related_pages`, and `catalog_paths` whenever they are known. +- **Merge, do not overwrite.** Automatic identifier extraction may add terms, but it must not discard curated registry metadata. Registry values are the durable source for future index rebuilds and remote MCP lookup. +- **Prefer precise identifiers in search ranking.** Exact or prefix matches in structured part-number and series fields should rank above incidental filename substrings such as timestamps or archive hashes. + ## Core Principles - **Compile, don't retrieve.** The wiki is pre-compiled knowledge. Update existing pages — don't append or duplicate. diff --git a/README.md b/README.md index 6e4d664..570204b 100644 --- a/README.md +++ b/README.md @@ -251,7 +251,7 @@ Modes: `by-tag` (default — top 10 tags), `by-category` (the seven vault folder - **Source archive and catalog.** When `WIKI_ARCHIVE_SOURCES=true`, file-based ingest copies original sources into `_sources/archive/` inside the vault and records them in `_meta/source-registry.jsonl`. When `WIKI_SOURCE_CATALOG=true`, it also maintains `_sources/catalog/` pages organized by document type, product line, customer, competitor, standard, and market. Page citations can then point at vault-relative archived files instead of machine-local paths, while catalog pages provide clean business navigation without duplicating files. -- **SQLite file asset lookup.** `_meta/file-assets.sqlite` indexes archived and staged source files for low-latency file search. Use `obsidian-wiki file-index search "part/report/customer"` when the answer should be an original drawing, 3D/CAD file, spreadsheet, PDF, or test report path instead of a synthesized wiki answer. The database lives inside the vault and can be rebuilt from `_meta/source-registry.jsonl`. +- **SQLite file asset lookup.** `_meta/file-assets.sqlite` indexes archived and staged source files for low-latency file search. Use `obsidian-wiki file-index search "part/report/customer"` when the answer should be an original drawing, 3D/CAD file, spreadsheet, PDF, or test report path instead of a synthesized wiki answer. The database lives inside the vault and can be rebuilt from `_meta/source-registry.jsonl`; rebuilds preserve structured registry fields such as part numbers, series, pitch, pin count, product lines, customers, standards, related pages, and catalog paths. - **Multimodal sources.** Screenshots, whiteboard photos, slide captures, and diagrams ingest the same way as text — the agent transcribes any visible text verbatim and tags interpreted content as inferred. Requires a vision-capable model. @@ -305,7 +305,7 @@ obsidian-wiki file-index search "Mini FAKRA 高频测试" --doc-type test_report obsidian-wiki file-index search "X4104 STEP" --source-type cad -n 10 ``` -The database is stored at `/_meta/file-assets.sqlite`. It is rebuilt from `/_meta/source-registry.jsonl`, so deleting the database is safe as long as the registry and `_sources/archive/` remain. To queue a new file for later ingest, copy it into the vault inbox and add a staged index row: +The database is stored at `/_meta/file-assets.sqlite`. It is rebuilt from `/_meta/source-registry.jsonl`, so deleting the database is safe as long as the registry and `_sources/archive/` remain. Registry metadata is treated as durable index input: `bjd_part_numbers`, `competitor_part_numbers`, `series`, `pitch`, `pin_count`, `product_lines`, `customers`, `competitors`, `standards`, `business_categories`, `related_pages`, and `catalog_paths` are preserved and merged with automatic extraction on every rebuild. To queue a new file for later ingest, copy it into the vault inbox and add a staged index row: ```bash obsidian-wiki file-index stage ./drawing.step --project bjd --doc-type drawing @@ -326,11 +326,12 @@ obsidian-wiki-mcp Configure your agent to run `obsidian-wiki-mcp` as a stdio MCP server. The server exposes: - `search_file_assets` — finds safe source files by product code, series, customer, doc type, or file type. +- `list_file_assets` — browses safe source files with optional filters. - `get_file_asset_metadata` — returns one file's safe metadata and `asset://source/sha256/...` URI. - `file_index_status` — shows vault/index/profile status. - `asset://source/sha256/{digest}` — binary resource for an allowed archived file. -By default the MCP layer is stricter than local CLI lookup: it only serves engineering files such as drawings, 3D/STEP models, specifications, product parameters, and test reports. It blocks raw orders, prices, email correspondence, complaint evidence, claims, personal data, and files missing from `_meta/source-registry.jsonl`. Use `OBSIDIAN_MCP_BOT_PROFILE`, `OBSIDIAN_MCP_MAX_FILE_BYTES`, and `OBSIDIAN_MCP_EXPOSE_PATHS` to tune the runtime behavior. +By default the MCP layer is stricter than local CLI lookup: it only serves engineering files such as drawings, 3D/STEP models, specifications, product parameters, and test reports. Customer-shareable engineering deliverables, such as PDF customer drawings, are allowed for source-file lookup when the active profile already allows technical engineering data. It blocks raw orders, prices, email correspondence, complaint evidence, claims, personal data, and files missing from `_meta/source-registry.jsonl`. Use `OBSIDIAN_MCP_BOT_PROFILE`, `OBSIDIAN_MCP_MAX_FILE_BYTES`, and `OBSIDIAN_MCP_EXPOSE_PATHS` to tune the runtime behavior. Optional: run the same MCP server over Streamable HTTP for a remote MCP client: diff --git a/obsidian_wiki/file_assets.py b/obsidian_wiki/file_assets.py index cfeaf4d..6d1d495 100644 --- a/obsidian_wiki/file_assets.py +++ b/obsidian_wiki/file_assets.py @@ -32,6 +32,14 @@ PITCH_RE = re.compile( re.IGNORECASE, ) PIN_RE = re.compile(r"(? str: @@ -205,9 +213,21 @@ def asset_from_registry_record(record: dict[str, Any]) -> dict[str, Any]: for key in ("original_name", "title", "summary", "archive_path", "original_path") ) text_blob += " " + " ".join(_flatten_to_strings(record.get(key)) for key in _CLASSIFICATION_KEYS) + text_blob += " " + " ".join(_flatten_to_strings(record.get(key)) for key in STRUCTURED_IDENTIFIER_FIELDS) + text_blob = ARCHIVE_HASH_PREFIX_RE.sub("", text_blob) derived = derive_identifiers(text_blob) file_ext = Path(original_name).suffix.lower() confidence = _float_or_none(record.get("classification_confidence")) + bjd_part_numbers = _merge_index_terms(record.get("bjd_part_numbers"), derived["bjd_part_numbers"], uppercase=True) + competitor_part_numbers = _merge_index_terms( + record.get("competitor_part_numbers"), + derived["competitor_part_numbers"], + uppercase=True, + limit=40, + ) + series = _merge_index_terms(record.get("series"), derived["series"], uppercase=True, limit=40) + pitch = _merge_index_terms(record.get("pitch"), derived["pitch"], limit=20) + pin_count = _merge_index_terms(record.get("pin_count"), derived["pin_count"], limit=20) return { "file_id": source_id or f"sha256:{sha256}", @@ -229,11 +249,11 @@ def asset_from_registry_record(record: dict[str, Any]) -> dict[str, Any]: "competitors": _json_list(record.get("competitors")), "standards": _json_list(record.get("standards")), "market_tags": _json_list(record.get("market_tags")), - "bjd_part_numbers": _json_list(derived["bjd_part_numbers"]), - "competitor_part_numbers": _json_list(derived["competitor_part_numbers"]), - "series": _json_list(derived["series"]), - "pitch": _json_list(derived["pitch"]), - "pin_count": _json_list(derived["pin_count"]), + "bjd_part_numbers": _json_list(bjd_part_numbers), + "competitor_part_numbers": _json_list(competitor_part_numbers), + "series": _json_list(series), + "pitch": _json_list(pitch), + "pin_count": _json_list(pin_count), "related_pages": _json_list(record.get("related_pages")), "catalog_paths": _json_list(record.get("catalog_paths")), "extracted_text": None, @@ -533,6 +553,11 @@ def score_asset(record: dict[str, Any], query: str) -> int: query = query.strip() if not query: return 1 + structured_terms = [ + term.lower() + for field in STRUCTURED_IDENTIFIER_FIELDS + for term in _iter_string_values(record.get(field)) + ] haystack = _searchable( [ record.get("original_name"), @@ -563,6 +588,8 @@ def score_asset(record: dict[str, Any], query: str) -> int: for token in tokens: if token in haystack: score += 7 if len(token) >= 4 else 3 + if _structured_term_matches(token, structured_terms): + score += 30 if len(token) >= 4 else 8 if tokens and all(token in haystack for token in tokens): score += 12 if score > 0 and record.get("doc_type") in {"drawing", "test_report", "product_parameter", "cross_reference"}: @@ -639,6 +666,58 @@ def _json_list(value: Any) -> str: return json.dumps([item for item in data if item not in (None, "")], ensure_ascii=False) +def _merge_index_terms(*values: Any, uppercase: bool = False, limit: int | None = None) -> list[str]: + terms: list[str] = [] + seen: set[str] = set() + for value in values: + for item in _iter_string_values(value): + term = item.strip() + if not term: + continue + if uppercase: + term = term.upper() + key = term.lower() + if key in seen: + continue + seen.add(key) + terms.append(term) + if limit is not None and len(terms) >= limit: + return terms + return terms + + +def _iter_string_values(value: Any) -> list[str]: + if value is None: + return [] + if isinstance(value, str): + stripped = value.strip() + if not stripped: + return [] + try: + parsed = json.loads(stripped) + except json.JSONDecodeError: + return [stripped] + if parsed == value: + return [stripped] + return _iter_string_values(parsed) + if isinstance(value, (list, tuple, set)): + items: list[str] = [] + for item in value: + items.extend(_iter_string_values(item)) + return items + if isinstance(value, dict): + items: list[str] = [] + for item in value.values(): + items.extend(_iter_string_values(item)) + return items + return [str(value)] + + +def _structured_term_matches(token: str, terms: list[str]) -> bool: + token = token.lower() + return any(token == term or term.startswith(token) or token in term for term in terms) + + def _flatten_to_strings(value: Any) -> str: if value is None: return "" diff --git a/obsidian_wiki/mcp_assets.py b/obsidian_wiki/mcp_assets.py index 8ab8284..d8b86e8 100644 --- a/obsidian_wiki/mcp_assets.py +++ b/obsidian_wiki/mcp_assets.py @@ -81,6 +81,8 @@ SAFE_RESULT_FIELDS = ( "competitors", "standards", "business_categories", + "bjd_part_numbers", + "competitor_part_numbers", "series", "pitch", "pin_count", @@ -282,7 +284,7 @@ def record_allowed_for_mcp( if profile is None: return True, "allowed" - decision = source_decision(profile_name, {**profile, "include_source_files": True}, record) + decision = source_decision(profile_name, _source_file_profile(profile, record), record) if not decision.include: return False, decision.reason return True, "allowed" @@ -338,6 +340,15 @@ def _load_profile(vault: Path, profile_name: str) -> dict[str, Any] | None: return None +def _source_file_profile(profile: dict[str, Any], record: dict[str, Any]) -> dict[str, Any]: + runtime_profile = {**profile, "include_source_files": True} + data_classification = str(record.get("data_classification") or "") + allowed = as_list(runtime_profile.get("allowed_data_classification")) + if data_classification == "customer-shareable" and "technical-confidential" in allowed: + runtime_profile["allowed_data_classification"] = [*allowed, "customer-shareable"] + return runtime_profile + + def _vault_path(vault_path: str | Path) -> Path: return Path(clean_config_value(str(vault_path))).expanduser() diff --git a/tests/test_file_assets.py b/tests/test_file_assets.py new file mode 100644 index 0000000..3ee4e57 --- /dev/null +++ b/tests/test_file_assets.py @@ -0,0 +1,80 @@ +import json +import tempfile +import unittest +from pathlib import Path + +from obsidian_wiki.file_assets import rebuild_from_registry, search_assets + + +class FileAssetIndexTests(unittest.TestCase): + def setUp(self): + self.tempdir = tempfile.TemporaryDirectory() + self.vault = Path(self.tempdir.name) + (self.vault / "_meta").mkdir() + self.registry_path = self.vault / "_meta" / "source-registry.jsonl" + self.db_path = self.vault / "_meta" / "file-assets.sqlite" + + def tearDown(self): + self.tempdir.cleanup() + + def test_rebuild_preserves_registry_identifiers_for_part_number_search(self): + drawing_id = "sha256:" + "a" * 64 + image_id = "sha256:" + "b" * 64 + self._write_registry_record( + source_id=drawing_id, + archive_path="_sources/archive/bjd/pdf/a.pdf", + original_name="北极大-0.5-18客户图.pdf", + title="北极大 FPC 0.5-18 客户图纸", + summary="FPC 0.5-18 客户图 PDF 归档。", + doc_type="drawing", + source_type="pdf", + bjd_part_numbers=["101049-X0XXX0", "101049-20XXX0"], + series=["0.5-18"], + pitch=["0.5"], + pin_count=["18"], + ) + self._write_registry_record( + source_id=image_id, + archive_path="_sources/archive/bjd/quality/photo.jpg", + original_name="微信图片_20250714101049.jpg", + title="微信图片_20250714101049", + summary="客诉图片。", + doc_type="evidence_media", + source_type="image", + business_categories=["quality/customer-complaints"], + ) + + rebuild_from_registry(self.vault, db_path=self.db_path) + + rows = search_assets(self.vault, "101049 图纸", db_path=self.db_path, limit=5, read_only=True) + + self.assertGreaterEqual(len(rows), 2) + self.assertEqual(rows[0]["file_id"], drawing_id) + self.assertEqual(json.loads(rows[0]["bjd_part_numbers"]), ["101049-X0XXX0", "101049-20XXX0"]) + self.assertEqual(json.loads(rows[0]["series"]), ["0.5-18"]) + self.assertEqual(json.loads(rows[0]["pitch"]), ["0.5"]) + self.assertEqual(json.loads(rows[0]["pin_count"]), ["18"]) + + def _write_registry_record(self, **fields): + record = { + "record_type": "source_archive", + "project": "bjd", + "visibility": "internal", + "business_categories": ["products/fpc"], + "product_lines": ["FPC", "FFC"], + "customers": [], + "competitors": [], + "standards": [], + "market_tags": [], + "related_pages": ["projects/bjd/references/example.md"], + "catalog_paths": ["_sources/catalog/bjd/products/fpc.md"], + "classification_confidence": 0.9, + "size_bytes": 1024, + **fields, + } + with self.registry_path.open("a", encoding="utf-8") as fh: + fh.write(json.dumps(record, ensure_ascii=False) + "\n") + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_mcp_assets.py b/tests/test_mcp_assets.py index 00cbb03..6ea2088 100644 --- a/tests/test_mcp_assets.py +++ b/tests/test_mcp_assets.py @@ -42,6 +42,10 @@ class McpAssetTests(unittest.TestCase): data_classification="technical-confidential", visibility="internal", summary="B3501 customer drawing.", + bjd_part_numbers=["101049-X0XXX0"], + series=["0.5-18"], + pitch=["0.5"], + pin_count=["18"], ) self._insert_asset( self.order_id, @@ -106,11 +110,11 @@ class McpAssetTests(unittest.TestCase): "competitors": "[]", "standards": "[]", "market_tags": "[]", - "bjd_part_numbers": "[]", - "competitor_part_numbers": "[]", - "series": "[]", - "pitch": "[]", - "pin_count": "[]", + "bjd_part_numbers": json.dumps(fields.get("bjd_part_numbers", []), ensure_ascii=False), + "competitor_part_numbers": json.dumps(fields.get("competitor_part_numbers", []), ensure_ascii=False), + "series": json.dumps(fields.get("series", []), ensure_ascii=False), + "pitch": json.dumps(fields.get("pitch", []), ensure_ascii=False), + "pin_count": json.dumps(fields.get("pin_count", []), ensure_ascii=False), "related_pages": json.dumps(record["related_pages"], ensure_ascii=False), "catalog_paths": "[]", "extracted_text": None, @@ -124,6 +128,25 @@ class McpAssetTests(unittest.TestCase): with connect(self.db_path) as conn: upsert_asset(conn, asset, status="active") + def _write_profile_file(self): + data = { + "version": 2, + "profiles": { + "bjd-general-bot": { + "allowed_visibility": ["public", "internal"], + "allowed_data_classification": [ + "public", + "internal", + "technical-confidential", + ], + "allowed_business_domains": ["products", "engineering"], + "blocked_doc_types": ["order_data", "evidence_media"], + "include_source_files": False, + } + }, + } + (self.vault / "_meta" / "bot-profiles.json").write_text(json.dumps(data), encoding="utf-8") + def test_search_returns_allowed_engineering_assets_only(self): rows = search_file_assets(self.vault, "B3501 drawing", db_path=self.db_path) @@ -131,6 +154,38 @@ class McpAssetTests(unittest.TestCase): self.assertEqual(rows[0]["resource_uri"], asset_uri_for_file_id(self.drawing_id)) self.assertNotIn("absolute_path", rows[0]) + def test_search_exposes_safe_identifier_fields(self): + rows = search_file_assets(self.vault, "101049 图纸", db_path=self.db_path) + + self.assertEqual([row["file_id"] for row in rows], [self.drawing_id]) + self.assertEqual(rows[0]["bjd_part_numbers"], ["101049-X0XXX0"]) + self.assertEqual(rows[0]["series"], ["0.5-18"]) + self.assertEqual(rows[0]["pitch"], ["0.5"]) + self.assertEqual(rows[0]["pin_count"], ["18"]) + + def test_customer_shareable_engineering_files_work_with_source_file_profile(self): + self._write_profile_file() + shareable_id = "sha256:" + "d" * 64 + self._write_asset_file("drawings/customer-shareable.pdf", b"%PDF customer drawing") + self._insert_asset( + shareable_id, + archive_path="_sources/archive/drawings/customer-shareable.pdf", + original_name="北极大-0.5-18客户图.pdf", + doc_type="drawing", + source_type="pdf", + product_lines=["FPC"], + business_categories=["products/fpc", "drawings/customer-drawings"], + data_classification="customer-shareable", + visibility="public", + summary="FPC 0.5-18 customer-shareable drawing.", + bjd_part_numbers=["101049-X0XXX0"], + series=["0.5-18"], + ) + + rows = search_file_assets(self.vault, "101049 图纸", db_path=self.db_path, profile_name="bjd-general-bot") + + self.assertEqual({row["file_id"] for row in rows}, {shareable_id, self.drawing_id}) + def test_search_blocks_sensitive_order_and_complaint_sources(self): self.assertEqual(search_file_assets(self.vault, "订单", db_path=self.db_path), []) self.assertEqual(search_file_assets(self.vault, "来料标签错误", db_path=self.db_path), [])