# Persistent Search Index And Runtime Provider Hosting Design

## Context

Phase 4 requires `verstak.search` to move beyond live recursive scans:

- keep search as an official plugin, not a core feature;
- persist a workspace-scoped search index;
- host `searchProviders` contributed by other plugins at runtime;
- preserve the local-first, readable vault model;
- avoid copying code or architecture from the old Verstak repository.

Current implementation status:

- `verstak.search` is a workspace item and contributes `searchProviders`;
- it searches while typing by walking files through `api.files.list`;
- it reads text-like files through `api.files.readText`;
- it opens file results through `api.workbench.openResource`;
- the contribution registry already exposes `searchProviders`;
- `api.commands.executeFor(pluginId, commandId, args)` already hosts frontend
  provider handlers for other contribution types;
- desktop backend already has plugin-scoped JSON data methods
  `ReadPluginDataJSON` and `WritePluginDataJSON`, but the frontend plugin API
  currently exposes only `settings`.

UX direction:

- the primary search entry point belongs in the workspace header next to the
  workspace title;
- `verstak.search` remains the owner of indexing and provider hosting;
- a standalone Search workspace item may stay as an expanded results surface,
  but it is not the primary entry point.

No visual companion is needed for this design because the decision is about
runtime contracts and data flow, not layout.

## Assumptions

- The next implementation should be a small reversible platform step.
- Search remains replaceable plugin functionality; core should expose generic
  storage and contribution execution only.
- `searchProviders[].handler` is treated as a command id. A provider plugin
  must declare that command in `contributes.commands` and register its frontend
  handler with `api.commands.register`.
- The initial persistent index is JSON-backed plugin data, not SQLite FTS and
  not a background sidecar.
- The index is an optimization and discovery layer. User files remain the
  source of truth.

## Alternatives Considered

### Recommended: plugin-owned index plus command-backed provider hosting

`verstak.search` owns its local index in plugin storage, uses public Files API
and events to keep it fresh, and fans out to contributed providers with
`api.commands.executeFor`.

Trade-offs:

- smallest core/runtime change;
- matches existing Files/Notes contribution execution pattern;
- keeps user-facing search outside core;
- JSON index is simpler than full text search but sufficient for the current
  roadmap item.

### Core search service

Desktop core would own indexing and execute provider searches.

Trade-offs:

- easier to centralize later ranking and indexing;
- violates the current direction that search is a plugin-level user feature;
- makes core understand official search semantics too early.

### Sidecar or SQLite FTS indexer now

Introduce a dedicated indexer process or SQLite FTS schema immediately.

Trade-offs:

- better long-term scalability;
- too large for the current milestone;
- adds migration, lifecycle, and sync/cache policy decisions before the basic
  provider runtime contract is proven.

## Chosen Design

Use the recommended approach.

`verstak.search` becomes both:

- the workspace search runtime and expanded results UI;
- the runtime host for all enabled `searchProviders`.

The shell may render the compact input in the workspace header, but it should
call into the Search plugin/runtime contract rather than implement search
semantics in core.

The desktop and SDK expose a generic frontend storage surface:

```ts
api.storage.data.read(name: string): Promise<Record<string, unknown>>
api.storage.data.write(name: string, data: Record<string, unknown>): Promise<void>
```

This maps to existing plugin-scoped backend methods. It requires
`storage.namespace`, follows current plugin ownership rules, and does not let a
plugin read another plugin's namespace.

The Search plugin stores its persistent index as plugin data named
`search-index`. Settings remain for user preferences only.

## Provider Runtime Contract

`searchProviders` keep their current manifest shape:

```json
{
  "id": "verstak.search.vault-text",
  "label": "Vault Text Search",
  "handler": "verstak.search.searchVaultText"
}
```

The `handler` value must name a command declared by the same plugin:

```json
{
  "contributes": {
    "commands": [
      {
        "id": "verstak.search.searchVaultText",
        "title": "Search Vault Text",
        "handler": "searchVaultText"
      }
    ],
    "searchProviders": [
      {
        "id": "verstak.search.vault-text",
        "label": "Vault Text Search",
        "handler": "verstak.search.searchVaultText"
      }
    ]
  }
}
```

The provider registers the command handler at mount time:

```js
api.commands.register('verstak.search.searchVaultText', searchVaultText)
```

Runtime availability:

- this milestone does not auto-start unloaded frontend bundles or sidecars;
- a provider is executable only after its command handler is registered in the
  current frontend runtime;
- declared but unregistered providers are skipped and reported as unavailable;
- later sidecar/background activation can extend this without changing the
  provider manifest shape.

Provider input:

```ts
{
  source: 'search',
  providerId: string,
  query: string,
  workspaceRootPath: string,
  limit: number
}
```

Provider output:

```ts
{
  results: SearchResult[]
}
```

`SearchResult`:

```ts
{
  id?: string,
  path?: string,
  title?: string,
  snippet?: string,
  matchType?: string,
  providerId?: string,
  providerLabel?: string,
  type?: 'file' | 'folder' | 'activity' | 'worklog' | string,
  openable?: boolean,
  line?: number,
  score?: number,
  resource?: {
    kind: 'vault-file',
    path: string,
    mode?: 'view' | 'edit'
  }
}
```

The Search host normalizes missing optional fields. Invalid provider responses
are ignored with a visible status warning; they do not fail the whole search.

## Persistent Index Shape

The first index version is intentionally small:

```json
{
  "version": 1,
  "workspaceRootPath": "Project",
  "builtAt": "2026-06-29T00:00:00Z",
  "entries": [
    {
      "path": "Project/Docs/case.md",
      "name": "case.md",
      "type": "file",
      "extension": "md",
      "size": 1234,
      "modifiedAt": "2026-06-29T00:00:00Z",
      "text": "short normalized searchable text or snippet"
    }
  ]
}
```

Rules:

- index only the current workspace root;
- store vault-relative slash paths;
- index folders and regular files;
- read content only for text-like files already handled by current Search;
- rely on the host `readText` limit for large text files;
- store short normalized text, not full arbitrary binary data;
- rebuild when version or workspace root differs.

## Index Lifecycle

On mount:

1. Read `api.storage.data.read('search-index')`.
2. If version/root matches, use it immediately.
3. If missing or stale, build an index from `api.files.list` and text reads.
4. Write the built index with `api.storage.data.write`.
5. Subscribe to `file.changed`.

On `file.changed`:

- if event path is outside the workspace, ignore it;
- for create/update, refresh that path's metadata/content if readable;
- for delete/trash, remove the path and descendants;
- for move, remove `fromPath` when present and refresh the new path;
- write the updated index after the change is applied.

If an incremental update fails, mark the index stale in UI and continue serving
the last usable index until rebuild succeeds.

## Search Flow

For a query shorter than two characters, return no results.

For a valid query:

1. Query local persistent index for path, name, folder, and text matches.
2. List enabled `searchProviders` through `api.contributions.list`.
3. Call each provider except duplicates that would recurse into the same
   in-flight command.
4. Merge and normalize results.
5. Sort by score, then provider order, then path/title.
6. Render provider label, match type, path/title, snippet, and open action when
   the result has a supported resource.

Provider failures are isolated. The status line reports how many providers
failed without hiding successful results.

## Error Handling

- Missing `storage` API: fall back to live scanning and show a degraded status.
- Missing `storage.namespace` permission: Search runs without persistence and
  reports degraded persistence.
- Corrupt index JSON: ignore it, rebuild, and overwrite only after a successful
  rebuild.
- Provider command not declared or not registered: skip that provider and show a
  warning count.
- File read errors: skip that file, matching the current Search behavior.
- Event subscription errors: keep search usable, mark index refresh as manual.

## Testing And Verification

Expected test coverage:

- SDK types include `api.storage.data.read/write` and search result/provider
  contracts.
- SDK mock API stores plugin data separately from settings.
- Desktop frontend bridge exposes `api.storage.data.read/write`.
- Desktop bridge smoke test verifies plugin data round-trip through Wails mock.
- Search plugin smoke test verifies:
  - persisted index is read before scanning;
  - missing/stale index triggers build and write;
  - query uses persisted entries;
  - `file.changed` updates or removes entries;
  - provider fan-out uses `contributions.list('searchProviders')` and
    `commands.executeFor`;
  - provider failure does not hide local results.
- Roadmap and official plugin docs are updated after implementation.

Manual smoke after implementation:

- start desktop frontend/dev flow used by the repository;
- open Search workspace item;
- search a known text file;
- reload/remount Search and confirm results come from stored index;
- enable a test provider and confirm its results appear with provider label.

## Out Of Scope

- SQLite FTS;
- typo/layout tolerant search;
- binary OCR or PDF extraction;
- background sidecar indexing;
- cross-workspace global search;
- sync policy for index cache;
- journal/worklog/activity reconstruction implementation.

Those remain later roadmap items.