chore(repo): establish project baseline

Track the source-of-truth code, skills, bootstrap rules, docs, and packaging files before adding new features. Exclude local vault config, generated skill mirrors, temporary exports, virtual environments, and business data files from version control. Validation: inspected staged file list and git ignored status before commit.
2026-07-05 11:03:40 +00:00
parent 2a3b69eaa7
commit 777607dc29
111 changed files with 26461 additions and 0 deletions
@@ -0,0 +1,245 @@
+---
+name: codex-history-ingest
+description: >
+  Ingest Codex CLI conversation history into the Obsidian wiki. Use this skill when the user wants to mine
+  their past Codex sessions for knowledge, import their ~/.codex folder, extract insights from previous coding
+  sessions, or says things like "process my Codex history", "add my Codex conversations to the wiki", or
+  "what have I discussed in Codex before". Also triggers when the user mentions .codex sessions, rollout files,
+  session_index.jsonl, or Codex transcript logs.
+---
+
+# Codex History Ingest — Conversation Mining
+
+You are extracting knowledge from the user's past Codex sessions and distilling it into the Obsidian wiki. Session logs are rich but noisy: focus on durable knowledge, not operational telemetry.
+
+This skill can be invoked directly or via the `wiki-history-ingest` router (`/wiki-history-ingest codex`).
+
+## Before You Start
+
+1. **Resolve config** — follow the Config Resolution Protocol in `llm-wiki/SKILL.md` (walk up CWD for `.env` → `~/.obsidian-wiki/config` → prompt setup). This gives `OBSIDIAN_VAULT_PATH` and `CODEX_HISTORY_PATH` (defaults to `~/.codex`)
+2. Read `.manifest.json` at the vault root to check what has already been ingested
+3. Read `index.md` at the vault root to understand what the wiki already contains
+
+## Ingest Modes
+
+### Append Mode (default)
+
+Check `.manifest.json` for each source file. Only process:
+
+- Files not in the manifest (new session rollouts, new index files)
+- Files whose modification time is newer than `ingested_at` in the manifest
+
+Use this mode for regular syncs.
+
+### Full Mode
+
+Process everything regardless of manifest. Use after `wiki-rebuild` or if the user explicitly asks for a full re-ingest.
+
+## Codex Data Layout
+
+Codex stores local artifacts under `~/.codex/`.
+
+```
+~/.codex/
+├── sessions/                          # Session rollout logs by date
+│   └── YYYY/MM/DD/
+│       └── rollout-<timestamp>-<id>.jsonl
+├── archived_sessions/                 # Archived rollout logs
+├── session_index.jsonl                # Lightweight index of thread id/name/updated_at
+├── history.jsonl                      # Local transcript history (if persistence enabled)
+├── config.toml                        # User config (contains history settings)
+└── state_*.sqlite / logs_*.sqlite     # Runtime DBs (usually skip)
+```
+
+### Key data sources ranked by value
+
+1. `session_index.jsonl` — best inventory source for IDs, titles, and freshness
+2. `sessions/**/rollout-*.jsonl` — rich structured transcript events
+3. `history.jsonl` — useful fallback/timeline aid if enabled
+
+Avoid ingesting SQLite internals unless the user explicitly asks.
+
+## Step 1: Survey and Compute Delta
+
+Scan `CODEX_HISTORY_PATH` and compare against `.manifest.json`:
+
+- `~/.codex/session_index.jsonl`
+- `~/.codex/sessions/**/rollout-*.jsonl`
+- `~/.codex/archived_sessions/**` (optional; only if user asks for archived history)
+- `~/.codex/history.jsonl` (optional fallback)
+
+Classify each file:
+
+- **New** — not in manifest
+- **Modified** — in manifest but file is newer than `ingested_at`
+- **Unchanged** — already ingested and unchanged
+
+Report a concise delta summary before deep parsing.
+
+## Step 2: Parse Session Index First
+
+`session_index.jsonl` typically has entries like:
+
+```json
+{"id":"...","thread_name":"...","updated_at":"..."}
+```
+
+Use it to:
+
+- Build a canonical session inventory
+- Prioritize recent/high-signal sessions
+- Map rollout IDs to human-readable thread names
+
+## Step 3: Parse Rollout JSONL Safely
+
+Each `rollout-*.jsonl` line is an event envelope with:
+
+```json
+{
+  "timestamp": "...",
+  "type": "session_meta|turn_context|event_msg|response_item",
+  "payload": { ... }
+}
+```
+
+### Extraction rules
+
+- Prioritize user intent and assistant-visible outputs
+- Favor `response_item` records with user/assistant message content
+- Use `event_msg` selectively for meaningful milestones; ignore pure telemetry
+- Treat `session_meta` as metadata (cwd, model, ids), not user knowledge
+
+### Skip/noise filters
+
+- Token accounting events
+- Tool plumbing with no semantic content
+- Raw command output unless it contains reusable decisions/patterns
+- Repeated plan snapshots unless they add novel decisions
+
+### Critical privacy filter
+
+Rollout logs can include injected instructions, tool payloads, and sensitive text. Do not ingest verbatim system/developer prompts or secrets.
+
+- Remove API keys, tokens, passwords, credentials
+- Redact private identifiers unless relevant and approved
+- Summarize instead of quoting raw transcripts
+
+## Step 4: Cluster by Topic
+
+Do not create one wiki page per session.
+
+- Group by stable topics across many sessions
+- Split mixed sessions into separate themes
+- Merge recurring concepts across dates/projects
+- Use `cwd` from metadata to infer project scope
+
+## Step 5: Distill into Wiki Pages
+
+Route extracted knowledge using existing wiki conventions:
+
+- Project-specific architecture/process -> `projects/<name>/...`
+- General concepts -> `concepts/`
+- Recurring techniques/debug playbooks -> `skills/`
+- Tools/services -> `entities/`
+- Cross-session patterns -> `synthesis/`
+
+For each impacted project, create/update `projects/<name>/<name>.md` (project name as filename, never `_project.md`).
+
+### Writing rules
+
+- Distill knowledge, not chronology
+- Avoid "on date X we discussed..." unless date context is essential
+- Add `summary:` frontmatter on each new/updated page (1-2 sentences, <= 200 chars)
+- Add confidence and lifecycle fields to every new page:
+  ```yaml
+  base_confidence: 0.42
+  lifecycle: draft
+  lifecycle_changed: <ISO date today>
+  ```
+  Leave `lifecycle` unchanged on update.
+- Add provenance markers:
+  - `^[extracted]` when directly grounded in explicit session content
+  - `^[inferred]` when synthesizing patterns across events/sessions
+  - `^[ambiguous]` when sessions conflict
+- Add/update `provenance:` frontmatter mix for each changed page
+
+## Step 6: Update Manifest, Log, and Index
+
+### Update `.manifest.json`
+
+For each processed source file:
+
+- `ingested_at`, `size_bytes`, `modified_at`
+- `source_type`: `codex_rollout` | `codex_index` | `codex_history`
+- `project`: inferred project name (when applicable)
+- `pages_created`, `pages_updated`
+
+Add/update a top-level project/session summary block:
+
+```json
+{
+  "project-name": {
+    "source_path": "~/.codex/sessions/...",
+    "last_ingested": "TIMESTAMP",
+    "sessions_ingested": 12,
+    "sessions_total": 40,
+    "index_updated_at": "TIMESTAMP"
+  }
+}
+```
+
+### Update special files
+
+Update `index.md` and `log.md`:
+
+```
+- [TIMESTAMP] CODEX_HISTORY_INGEST sessions=N pages_updated=X pages_created=Y mode=append|full
+```
+
+**`hot.md`** — Read `$OBSIDIAN_VAULT_PATH/hot.md` (create from the template in `wiki-ingest` if missing). Update **Recent Activity** with a one-line summary — e.g. "Ingested 12 Codex sessions; surfaced recurring patterns in CLI tooling and shell scripting." Keep the last 3 operations. Update `updated` timestamp.
+
+## Privacy and Compliance
+
+- Distill and synthesize; avoid raw transcript dumps
+- Default to redaction for anything that looks sensitive
+- Ask the user before storing personal/sensitive details
+- Keep references to other people minimal and purpose-bound
+
+## Reference
+
+See `references/codex-data-format.md` for field-level parsing notes and extraction guidance.
+
+## QMD Refresh After Vault Writes
+
+QMD is a search index, not the source of truth. If `$QMD_WIKI_COLLECTION` is empty or unset, skip this step. Run it only after this skill has written or rewritten vault markdown. If QMD refresh fails, do not roll back the vault changes; report the QMD status separately.
+
+Use `$QMD_CLI` if set; otherwise use `qmd`.
+
+```bash
+${QMD_CLI:-qmd} update
+```
+
+If the output says vectors are needed or embeddings may be stale, run:
+
+```bash
+${QMD_CLI:-qmd} embed
+```
+
+Verify the collection with either:
+
+```bash
+${QMD_CLI:-qmd} ls "$QMD_WIKI_COLLECTION"
+```
+
+or, when a specific page path is known:
+
+```bash
+${QMD_CLI:-qmd} get "qmd://$QMD_WIKI_COLLECTION/<page>.md" -l 5
+```
+
+Record one of:
+- `QMD refreshed: update + embed + verified`
+- `QMD refreshed: update only + verified`
+- `QMD skipped: QMD_WIKI_COLLECTION unset`
+- `QMD skipped: qmd CLI unavailable`
+- `QMD failed: <short error summary>`
@@ -0,0 +1,82 @@
+# Codex Data Format — Detailed Reference
+
+This reference describes practical, observed structures for Codex local history ingestion.
+
+## Root Layout
+
+`~/.codex/` usually contains:
+
+- `sessions/YYYY/MM/DD/rollout-*.jsonl` — primary structured session logs
+- `archived_sessions/` — archived rollouts
+- `session_index.jsonl` — session id/name/update index
+- `history.jsonl` — transcript history (depends on config)
+- `config.toml` — history persistence controls
+
+## Session Index
+
+`~/.codex/session_index.jsonl` entries are one JSON object per line, commonly:
+
+```json
+{"id":"<thread-id>","thread_name":"<title>","updated_at":"<timestamp>"}
+```
+
+Use this as the inventory backbone for append/full mode deltas.
+
+## Rollout JSONL
+
+`rollout-*.jsonl` files are event streams with envelope fields:
+
+```json
+{
+  "timestamp": "2026-04-12T09:40:02.337Z",
+  "type": "session_meta|turn_context|event_msg|response_item",
+  "payload": { "...": "..." }
+}
+```
+
+Common `type` values:
+
+- `session_meta` — run metadata (id, cwd, model/provider, etc.)
+- `turn_context` — turn-scoped context envelope
+- `event_msg` — runtime events (task lifecycle, token/accounting, tool-call markers)
+- `response_item` — model response items (messages, tool calls, reasoning blocks)
+
+### Typical payload subtypes
+
+Observed examples include:
+
+- `event_msg.payload.type`: `task_started`, `user_message`, `agent_message`, `mcp_tool_call_end`, `exec_command_end`, `token_count`
+- `response_item.payload.type`: `message`, `function_call`, `function_call_output`, `reasoning`
+
+## Extraction Strategy
+
+### Keep
+
+- User intent from user-message records
+- Assistant conclusions/decisions from assistant message records
+- High-signal tool outputs that encode reusable knowledge
+
+### Skip
+
+- Pure telemetry (`token_count`, low-level plumbing events)
+- Internal reasoning traces unless user explicitly asks to retain them
+- Verbose execution dumps with no durable insight
+
+## Privacy Notes
+
+Rollouts can contain sensitive data:
+
+- Injected instruction layers
+- Tool inputs/outputs
+- Potential secrets in command output
+
+Always redact secrets and summarize instead of copying raw transcript content.
+
+## Config Interaction
+
+`~/.codex/config.toml` keys that affect ingestion completeness:
+
+- `history.persistence = "save-all" | "none"`
+- `history.max_bytes = <int>` (truncation/compaction cap)
+
+`codex exec --ephemeral` runs may not persist rollout files.