diff --git a/README.md b/README.md index 7f85a72..7766c07 100644 --- a/README.md +++ b/README.md @@ -224,6 +224,24 @@ For synchronization between devices, deploy the optional self-hosted Each vault is connected separately. The local vault remains the primary copy of your data. +Desktop core maintains an atomic local sync snapshot under `.verstak/sync/`. +It scans when a vault opens, before a manual sync, and after debounced external +file-watcher events, so files changed in an editor or while Desktop was closed +are reconciled too. The watcher only speeds up discovery. A first connection +pulls before it publishes a local bootstrap: an empty local vault does not send +deletes, and incompatible files become visible conflicts rather than silent +overwrites. Pairing can optionally use an existing remote vault ID to restore +that scope on a new device. + +The current operation transport supports ordinary files and folders plus +workspace (Deal) create/rename/trash/restore with a durable workspace UUID. +`.verstak`, trash, temporary files, and symlinks are excluded from ordinary +file sync. Text remains bounded to 2 MB and binary payloads to 8 MB; a larger +or unsupported file is left as a visible unresolved sync warning instead of +being treated as synchronized. Blob transfer, quotas, pagination, retention, +and Secrets, plugin settings, Todo, Journal, Activity, and Browser Inbox sync +are future milestones. + ## Build from source ### Requirements diff --git a/docs/PLUGIN_RUNTIME.md b/docs/PLUGIN_RUNTIME.md index 616b11b..2f96314 100644 --- a/docs/PLUGIN_RUNTIME.md +++ b/docs/PLUGIN_RUNTIME.md @@ -477,8 +477,8 @@ contributions summary. - All paths are canonical vault-relative slash paths. Backslashes, POSIX absolute paths, Windows drive paths, UNC/network paths, `..`, null bytes, symlink traversal, and public access to `.verstak/` are rejected. -- `.verstak` is reserved case-insensitively: `.verstak`, `.Verstak`, and any - first path segment with that spelling are internal-only. +- `.verstak` is reserved case-insensitively in every path segment: its marker, + sync snapshot, trash, and workspace identity are internal-only. - `files.metadata` may report a final symlink as `type: "symlink"`, but `files.list` through a symlink directory and all read/write/move/trash operations through symlinks are forbidden in Milestone 6a. @@ -486,20 +486,37 @@ contributions summary. `writeBytes` are bounded byte contracts up to 8 MB; chunked streaming is deferred. - Live watcher refresh is active while Verstak is running and a vault is open. - It performs an initial no-event snapshot, then publishes `file.changed` for - external creates, updates, and deletes outside `.verstak/`. It does not keep a - persistent snapshot or report what changed while Verstak was closed. + It publishes `file.changed` as a UI hint and debounces a full core scanner; + the persistent `.verstak/sync/snapshot.json` scanner, not the watcher, is the + source of truth. It runs on open, before manual sync, after watcher events, + and therefore detects changes made while Verstak was closed. Internal paths, + trash, temporary files, and symlinks are excluded. `sync` -- `sync.now()` pushes local operations, pulls remote operations, and returns - `{ pushed, pulled, serverSequence, conflicts?, applyErrors? }`. +- `sync.now()` scans local files, pulls/reconciles in `server_sequence` order, + then pushes local operations and returns `{ pushed, pulled, serverSequence, + conflicts? }`. A remote apply failure stops the batch immediately, keeps that + operation and later sequences unacknowledged, and retries it on the next run. - `conflicts` is an array of server-reported sync conflicts. Conflict objects may include `op_id`, `entity_type`, `entity_id`, `reason`, and additional server fields. The Sync plugin must show conflict details instead of only a count, and it must not silently resolve or overwrite local data. -- `applyErrors` lists local apply failures for pulled operations. These are - user-visible warnings and do not imply that sync was fully successful. +- `sync.status()` includes `vaultId` (the paired remote scope) and + `lastWarning` for unresolved scanner input. A first connection pulls before + bootstrap: an empty local snapshot never publishes deletes, and incompatible + local/remote content is an explicit conflict rather than an overwrite. +- File and folder operations come from the scanner. An external rename is + intentionally represented as delete + create in this milestone. Workspace + lifecycle is a separate core `workspace` entity (`create`, `rename`, + `trash`, `restore`) carrying the durable `workspaceId`; plugins never obtain + access to the nested workspace marker. +- File transport remains bounded: UTF-8 text uses the 2 MB Files API limit and + binary/other regular files use the existing base64 path up to 8 MB. Larger or + unsupported files remain visible unresolved warnings and are not added to a + successful snapshot. Blob transport, quotas, pagination, retention, and + synchronization of Secrets, plugin settings, Todo, Journal, Activity, and + Browser Inbox are future work. - Transport push/pull uses bounded retry/backoff for transient HTTP/network failures. Client/auth errors are not retried. @@ -585,8 +602,8 @@ bundled runtime. Это реальный runtime contract для cooperative bun | `api.files.showInFolder(relativePath)` | ✅ Работает | Показывает vault file/folder в системном файловом менеджере, требует `files.openExternal` | | `api.workbench.openResource(request)` | ✅ Работает | Routes vault resources to `openProviders` | | `api.workbench.editResource(request)` | ✅ Работает | Same routing, forcing `mode: "edit"` | -| `api.sync.now()` | ✅ Работает | Push/pull с bounded retry/backoff для transient HTTP/network failures | -| `api.sync.status()` | ✅ Работает | Возвращает configured/connected/error/revoked state, lastError, unpushed count | +| `api.sync.now()` | ✅ Работает | Snapshot scan, строгий ordered pull/retry и bounded retry/backoff для transient HTTP/network failures | +| `api.sync.status()` | ✅ Работает | Возвращает configured/connected/error/revoked, remote `vaultId`, lastError/lastWarning и unpushed count | | `api.dispose()` | ✅ Работает | Очищает command handlers и event subscriptions текущего API instance | Ограничения: diff --git a/frontend/e2e/command-palette.spec.js b/frontend/e2e/command-palette.spec.js index e5f76f7..ec99fb2 100644 --- a/frontend/e2e/command-palette.spec.js +++ b/frontend/e2e/command-palette.spec.js @@ -99,7 +99,7 @@ test.describe('Command Palette', () => { test('runs sync workflow commands', async ({ page }) => { await page.evaluate(async () => { - const err = await window.go.api.App.PluginSyncConfigure('verstak.sync', 'https://sync.example.test', 'alice', 'secret'); + const err = await window.go.api.App.PluginSyncConfigure('verstak.sync', 'https://sync.example.test', 'alice', 'secret', ''); if (err) throw new Error(err); }); diff --git a/frontend/e2e/plugin-api-bridge.spec.js b/frontend/e2e/plugin-api-bridge.spec.js index cd14676..8a8bdf2 100644 --- a/frontend/e2e/plugin-api-bridge.spec.js +++ b/frontend/e2e/plugin-api-bridge.spec.js @@ -102,7 +102,7 @@ test.describe('D: Plugin API bridge', () => { const api = window.createPluginAPI('verstak.sync'); const initial = await api.sync.status(); await api.sync.testConnection('https://sync.example.test', 'alice', 'secret'); - await api.sync.configure('https://sync.example.test', 'alice', 'secret'); + await api.sync.configure('https://sync.example.test', 'alice', 'secret', 'existing-remote-vault'); await api.sync.setInterval(15); const configured = await api.sync.status(); const syncNow = await api.sync.now(); @@ -117,6 +117,8 @@ test.describe('D: Plugin API bridge', () => { expect(result.initial.statusLabel).toBe('disabled'); expect(result.configured.configured).toBe(true); expect(result.configured.serverUrl).toBe('https://sync.example.test'); + expect(result.configured.vaultId).toBe('existing-remote-vault'); + expect(result.configured.lastWarning).toBe(''); expect(result.configured.syncInterval).toBe(15); expect(result.syncNow).toEqual({ pushed: 0, pulled: 0, serverSequence: 0 }); expect(result.reset.configured).toBe(false); diff --git a/frontend/src/lib/plugin-host/VerstakPluginAPI.js b/frontend/src/lib/plugin-host/VerstakPluginAPI.js index 9c255bc..361d8d9 100644 --- a/frontend/src/lib/plugin-host/VerstakPluginAPI.js +++ b/frontend/src/lib/plugin-host/VerstakPluginAPI.js @@ -468,10 +468,10 @@ export function createPluginAPI(pluginId) { return App.PluginSyncStatus(pluginId); }); }, - configure: function(serverURL, username, password) { + configure: function(serverURL, username, password, vaultId) { assertActive('sync.configure'); return callBackendErrorString(pluginId, 'sync.configure', function() { - return App.PluginSyncConfigure(pluginId, serverURL || '', username || '', password || ''); + return App.PluginSyncConfigure(pluginId, serverURL || '', username || '', password || '', vaultId || ''); }); }, disconnect: function() { diff --git a/frontend/src/lib/test/wails-mock.js b/frontend/src/lib/test/wails-mock.js index dfa4f4d..c4c5a3e 100644 --- a/frontend/src/lib/test/wails-mock.js +++ b/frontend/src/lib/test/wails-mock.js @@ -626,6 +626,7 @@ import journalSource from '../../../../../verstak-official-plugins/plugins/journ return { configured: false, serverUrl: '', + vaultId: '', deviceId: 'mock-device', deviceName: '', connected: false, @@ -635,6 +636,7 @@ import journalSource from '../../../../../verstak-official-plugins/plugins/journ lastSyncAt: '', syncInterval: 0, lastError: '', + lastWarning: '', statusLabel: 'disabled', serverSequence: 0 }; @@ -649,7 +651,7 @@ import journalSource from '../../../../../verstak-official-plugins/plugins/journ if (p.charAt(0) === '/' || /^[A-Za-z]:/.test(p)) return { error: 'invalid-path: absolute path rejected' }; var parts = p.split('/').filter(Boolean); if (parts.indexOf('..') !== -1) return { error: 'invalid-path: path-traversal' }; - if (parts[0] && parts[0].toLowerCase() === '.verstak') return { error: 'reserved-path: .verstak is internal' }; + if (parts.some(function(part) { return part.toLowerCase() === '.verstak'; })) return { error: 'reserved-path: .verstak is internal' }; return { path: parts.join('/') }; } @@ -751,6 +753,7 @@ import journalSource from '../../../../../verstak-official-plugins/plugins/journ return { configured: syncState.configured, serverUrl: syncState.serverUrl, + vaultId: syncState.vaultId, deviceId: syncState.deviceId, deviceName: syncState.deviceName, connected: syncState.connected, @@ -760,6 +763,7 @@ import journalSource from '../../../../../verstak-official-plugins/plugins/journ lastSyncAt: syncState.lastSyncAt, syncInterval: syncState.syncInterval, lastError: syncState.lastError, + lastWarning: syncState.lastWarning, statusLabel: syncState.statusLabel }; } @@ -3477,11 +3481,12 @@ import journalSource from '../../../../../verstak-official-plugins/plugins/journ if (err) return Promise.resolve([{}, err]); return Promise.resolve([syncStatusDTO(), '']); }, - PluginSyncConfigure: function (pluginId, serverUrl) { + PluginSyncConfigure: function (pluginId, serverUrl, username, password, vaultId) { var err = requirePluginSyncPermission(pluginId, true); if (err) return Promise.resolve(err); syncState.configured = true; syncState.serverUrl = serverUrl || ''; + syncState.vaultId = vaultId || 'test-vault-001'; syncState.deviceId = 'mock-device'; syncState.deviceName = 'mock-device'; syncState.connected = true; @@ -3491,6 +3496,7 @@ import journalSource from '../../../../../verstak-official-plugins/plugins/journ syncState.statusLabel = 'connected'; pluginSettings[pluginId] = Object.assign({}, pluginSettings[pluginId] || {}, { serverUrl: syncState.serverUrl, + vaultId: syncState.vaultId, syncStatus: syncState.statusLabel }); return Promise.resolve(''); diff --git a/frontend/wailsjs/go/api/App.d.ts b/frontend/wailsjs/go/api/App.d.ts index f254684..cae403d 100755 --- a/frontend/wailsjs/go/api/App.d.ts +++ b/frontend/wailsjs/go/api/App.d.ts @@ -114,7 +114,7 @@ export function PluginSecretsUnlock(arg1:string,arg2:string):Promise; export function PluginSecretsWrite(arg1:string,arg2:Record):Promise|string>; -export function PluginSyncConfigure(arg1:string,arg2:string,arg3:string,arg4:string):Promise; +export function PluginSyncConfigure(arg1:string,arg2:string,arg3:string,arg4:string,arg5:string):Promise; export function PluginSyncDisconnect(arg1:string):Promise; diff --git a/frontend/wailsjs/go/api/App.js b/frontend/wailsjs/go/api/App.js index 4848e10..315fc09 100755 --- a/frontend/wailsjs/go/api/App.js +++ b/frontend/wailsjs/go/api/App.js @@ -210,8 +210,8 @@ export function PluginSecretsWrite(arg1, arg2) { return window['go']['api']['App']['PluginSecretsWrite'](arg1, arg2); } -export function PluginSyncConfigure(arg1, arg2, arg3, arg4) { - return window['go']['api']['App']['PluginSyncConfigure'](arg1, arg2, arg3, arg4); +export function PluginSyncConfigure(arg1, arg2, arg3, arg4, arg5) { + return window['go']['api']['App']['PluginSyncConfigure'](arg1, arg2, arg3, arg4, arg5); } export function PluginSyncDisconnect(arg1) { diff --git a/frontend/wailsjs/go/models.ts b/frontend/wailsjs/go/models.ts index 09d819e..48ca0c9 100755 --- a/frontend/wailsjs/go/models.ts +++ b/frontend/wailsjs/go/models.ts @@ -310,6 +310,7 @@ export namespace api { export class SyncStatusDTO { configured: boolean; serverUrl: string; + vaultId: string; deviceId: string; deviceName: string; connected: boolean; @@ -319,6 +320,7 @@ export namespace api { lastSyncAt: string; syncInterval: number; lastError: string; + lastWarning: string; statusLabel: string; static createFrom(source: any = {}) { @@ -329,6 +331,7 @@ export namespace api { if ('string' === typeof source) source = JSON.parse(source); this.configured = source["configured"]; this.serverUrl = source["serverUrl"]; + this.vaultId = source["vaultId"]; this.deviceId = source["deviceId"]; this.deviceName = source["deviceName"]; this.connected = source["connected"]; @@ -338,6 +341,7 @@ export namespace api { this.lastSyncAt = source["lastSyncAt"]; this.syncInterval = source["syncInterval"]; this.lastError = source["lastError"]; + this.lastWarning = source["lastWarning"]; this.statusLabel = source["statusLabel"]; } } diff --git a/internal/api/app.go b/internal/api/app.go index 44b8fa5..b5de9e5 100644 --- a/internal/api/app.go +++ b/internal/api/app.go @@ -3,8 +3,11 @@ package api import ( "context" + "crypto/sha256" + "encoding/base64" "encoding/json" "fmt" + "io" "log" "net/url" "os" @@ -95,6 +98,9 @@ type App struct { browserReceiver *browserreceiver.Receiver secretsSession *coresecrets.VaultSession fileWatcher *filewatcher.Service + syncRunMu sync.Mutex + syncTimerMu sync.Mutex + syncScanTimer *time.Timer notifications notificationService debug bool activityEvents map[string]bool @@ -1228,6 +1234,7 @@ func (a *App) CloseVault() error { if a.fileWatcher != nil { a.fileWatcher.Stop() } + a.stopScheduledSnapshotScan() a.vault.CloseVault() a.syncSvc = nil a.secretsSession = nil @@ -1738,10 +1745,30 @@ func (a *App) externalOpenService() externalOpenService { } func (a *App) recordFileSyncOp(entityType, entityID, opType string, payload interface{}) error { - if a.syncSvc == nil { + _, err := a.scanLocalChanges() + return err +} + +func (a *App) recordWorkspaceSyncOp(opType, workspaceID, path, previousPath, name string) error { + if a.syncSvc == nil || a.workspace == nil { return nil } - return a.syncSvc.RecordOp(entityType, entityID, opType, payload) + meta, err := a.workspace.GetWorkspaceMetadata(path) + if err != nil && opType != syncsvc.OpTrash { + return err + } + payload := syncWorkspacePayload{ + WorkspaceID: workspaceID, + Path: path, + PreviousPath: previousPath, + Name: name, + Metadata: meta, + } + if err := a.syncSvc.RecordOp(syncsvc.EntityWorkspace, workspaceID, opType, payload); err != nil { + return err + } + _, err = a.syncSvc.RebaseSnapshot() + return err } func (a *App) publishFileActivity(eventName, pluginID, relativePath string, extra map[string]interface{}) { @@ -2314,9 +2341,69 @@ func (a *App) startFileWatcherForOpenVault() { if a.fileWatcher == nil { a.fileWatcher = filewatcher.NewService(a.eventBus, 0) } + a.fileWatcher.SetOnChange(a.scheduleSnapshotScan) if err := a.fileWatcher.Start(a.vault.GetVaultPath()); err != nil { log.Printf("[api] file watcher start failed: %v", err) } + if _, err := a.scanLocalChanges(); err != nil { + log.Printf("[api] initial sync snapshot scan failed: %v", err) + } +} + +const snapshotScanDebounce = 300 * time.Millisecond + +func (a *App) scheduleSnapshotScan() { + if a == nil { + return + } + a.syncTimerMu.Lock() + if a.syncScanTimer != nil { + a.syncScanTimer.Stop() + } + a.syncScanTimer = time.AfterFunc(snapshotScanDebounce, func() { + if _, err := a.scanLocalChanges(); err != nil { + log.Printf("[api] watcher sync snapshot scan failed: %v", err) + } + }) + a.syncTimerMu.Unlock() +} + +func (a *App) stopScheduledSnapshotScan() { + if a == nil { + return + } + a.syncTimerMu.Lock() + if a.syncScanTimer != nil { + a.syncScanTimer.Stop() + a.syncScanTimer = nil + } + a.syncTimerMu.Unlock() +} + +func (a *App) scanLocalChanges() ([]string, error) { + if a == nil { + return nil, nil + } + a.syncRunMu.Lock() + defer a.syncRunMu.Unlock() + return a.scanLocalChangesLocked() +} + +func (a *App) scanLocalChangesLocked() ([]string, error) { + if a.syncSvc == nil { + return nil, nil + } + warnings, err := a.syncSvc.ScanAndRecord() + if err != nil { + if a.appSettings != nil { + _ = a.updateSyncError("snapshot scan: " + err.Error()) + } + return nil, err + } + if err := a.syncSvc.SetLastWarning(strings.Join(warnings, "\n")); err != nil { + return nil, err + } + return warnings, nil } // ─── Workspace API ───────────────────────────────────────── @@ -2373,6 +2460,9 @@ func (a *App) CreateWorkspace(name, templateID string) (workspace.Workspace, str if err != nil { return workspace.Workspace{}, err.Error() } + if err := a.recordWorkspaceSyncOp(syncsvc.OpCreate, ws.ID, ws.RootPath, "", ws.Name); err != nil { + return workspace.Workspace{}, err.Error() + } a.publishWorkspaceLifecycleEvent(workspaceCreatedEventName, map[string]interface{}{ "operation": "create", "workspaceId": ws.ID, @@ -2395,6 +2485,9 @@ func (a *App) RenameWorkspace(oldName, newName string) string { if err != nil { return err.Error() } + if err := a.recordWorkspaceSyncOp(syncsvc.OpRename, identity.WorkspaceID, newName, oldName, newName); err != nil { + return err.Error() + } a.publishWorkspaceLifecycleEvent(workspaceRenamedEventName, map[string]interface{}{ "operation": "rename", "workspaceId": identity.WorkspaceID, @@ -2415,6 +2508,9 @@ func (a *App) TrashWorkspace(name string) (workspace.TrashResult, string) { if err != nil { return workspace.TrashResult{}, err.Error() } + if err := a.recordWorkspaceSyncOp(syncsvc.OpTrash, result.WorkspaceID, name, "", name); err != nil { + return workspace.TrashResult{}, err.Error() + } a.publishWorkspaceLifecycleEvent(workspaceTrashedEventName, map[string]interface{}{ "operation": "trash", "workspaceId": result.WorkspaceID, @@ -2436,6 +2532,9 @@ func (a *App) RestoreWorkspaceTrash(trashID, targetName string) (workspace.Works if err != nil { return workspace.Workspace{}, err.Error() } + if err := a.recordWorkspaceSyncOp(syncsvc.OpRestore, restored.ID, restored.RootPath, "", restored.Name); err != nil { + return workspace.Workspace{}, err.Error() + } a.publishWorkspaceLifecycleEvent(workspaceRestoredEventName, map[string]interface{}{ "operation": "restore", "workspaceId": restored.ID, @@ -2943,6 +3042,7 @@ func (a *App) vaultPath() string { type SyncStatusDTO struct { Configured bool `json:"configured"` ServerURL string `json:"serverUrl"` + VaultID string `json:"vaultId"` DeviceID string `json:"deviceId"` DeviceName string `json:"deviceName"` Connected bool `json:"connected"` @@ -2952,6 +3052,7 @@ type SyncStatusDTO struct { LastSyncAt string `json:"lastSyncAt"` SyncInterval int `json:"syncInterval"` LastError string `json:"lastError"` + LastWarning string `json:"lastWarning"` StatusLabel string `json:"statusLabel"` } @@ -2972,15 +3073,19 @@ func (a *App) syncStatus() (*SyncStatusDTO, error) { cfg := a.appSettings.Get() deviceToken := syncsvc.LoadDeviceToken(vaultPath) + remoteVaultID, _ := a.syncSvc.RemoteVaultID() + lastWarning, _ := a.syncSvc.LastWarning() dto := &SyncStatusDTO{ Configured: serverURL != "" && (apiKey != "" || deviceToken != ""), ServerURL: serverURL, + VaultID: remoteVaultID, LastSyncAt: lastSyncAt, UnpushedOps: 0, TokenStored: deviceToken != "", SyncInterval: cfg.Sync.SyncInterval, LastError: cfg.Sync.LastError, + LastWarning: lastWarning, } if deviceID := a.syncSvc.GetDeviceID(); deviceID != "" { @@ -3048,7 +3153,7 @@ func (a *App) PluginSyncStatus(pluginID string) (*SyncStatusDTO, string) { return dto, "" } -func (a *App) syncConfigure(serverURL, username, password string) error { +func (a *App) syncConfigure(serverURL, username, password, remoteVaultID string) error { if err := a.requireVault(); err != nil { return err } @@ -3061,8 +3166,28 @@ func (a *App) syncConfigure(serverURL, username, password string) error { if hostname == "" { hostname = "unknown" } + targetVaultID := strings.TrimSpace(remoteVaultID) + if targetVaultID == "" { + targetVaultID = meta.VaultID + } + if a.syncSvc != nil { + previousServerURL, _, _, _, stateErr := a.syncSvc.GetState() + previousVaultID, _ := a.syncSvc.RemoteVaultID() + if previousVaultID == "" { + previousVaultID = meta.VaultID + } + if stateErr == nil && previousServerURL != "" && previousVaultID != targetVaultID { + pending, err := a.syncSvc.GetUnpushedOps() + if err != nil { + return fmt.Errorf("read pending operations before changing remote vault: %w", err) + } + if len(pending) > 0 { + return fmt.Errorf("cannot change remote vault with %d unpushed local operation(s); synchronize or resolve them first", len(pending)) + } + } + } client := newSyncClient(serverURL, "", "", vaultPath) - deviceID, deviceToken, err := client.PairDevice(serverURL, username, password, hostname, "verstak-desktop/v2", meta.VaultID) + deviceID, deviceToken, err := client.PairDevice(serverURL, username, password, hostname, "verstak-desktop/v2", targetVaultID) if err != nil { return fmt.Errorf("pair: %w", err) } @@ -3073,6 +3198,18 @@ func (a *App) syncConfigure(serverURL, username, password string) error { if err := a.syncSvc.SetState(serverURL, ""); err != nil { return err } + if err := a.syncSvc.SetRemoteVaultID(targetVaultID); err != nil { + return err + } + if err := a.syncSvc.SetLastPullSeq(0); err != nil { + return err + } + if err := a.syncSvc.SetBootstrapComplete(false); err != nil { + return err + } + if err := a.syncSvc.SetLastWarning(""); err != nil { + return err + } cfg := a.appSettings.Get() cfg.Sync.Enabled = true @@ -3087,11 +3224,11 @@ func (a *App) syncConfigure(serverURL, username, password string) error { } // PluginSyncConfigure pairs the current vault with a sync server for a plugin. -func (a *App) PluginSyncConfigure(pluginID, serverURL, username, password string) string { +func (a *App) PluginSyncConfigure(pluginID, serverURL, username, password, remoteVaultID string) string { if err := a.requirePluginSyncAccess(pluginID, true); err != nil { return err.Error() } - if err := a.syncConfigure(serverURL, username, password); err != nil { + if err := a.syncConfigure(serverURL, username, password, remoteVaultID); err != nil { return err.Error() } return "" @@ -3130,7 +3267,13 @@ func (a *App) syncDisconnect() error { if a.syncSvc == nil { return nil } - return a.syncSvc.SetState("", "") + if err := a.syncSvc.SetState("", ""); err != nil { + return err + } + if err := a.syncSvc.SetRemoteVaultID(""); err != nil { + return err + } + return a.syncSvc.SetLastWarning("") } // PluginSyncDisconnect disconnects sync for a plugin with sync permission. @@ -3230,6 +3373,8 @@ func (a *App) PluginSyncResetKey(pluginID string) string { } func (a *App) syncNow() (map[string]interface{}, error) { + a.syncRunMu.Lock() + defer a.syncRunMu.Unlock() if err := a.requireVault(); err != nil { return nil, err } @@ -3240,6 +3385,9 @@ func (a *App) syncNow() (map[string]interface{}, error) { if a.syncSvc == nil { return nil, fmt.Errorf("sync service not initialized") } + if _, err := a.scanLocalChangesLocked(); err != nil { + return nil, fmt.Errorf("snapshot scan: %w", err) + } serverURL, apiKey, lastPullSeq, _, err := a.syncSvc.GetState() deviceToken := syncsvc.LoadDeviceToken(vaultPath) @@ -3270,12 +3418,44 @@ func (a *App) syncNow() (map[string]interface{}, error) { client.DeviceID = deviceID } + bootstrapComplete, err := a.syncSvc.BootstrapComplete() + if err != nil { + return nil, fmt.Errorf("read bootstrap state: %w", err) + } + initialSnapshot, err := a.syncSvc.LoadSnapshot() + if err != nil { + return nil, fmt.Errorf("load initial snapshot: %w", err) + } + + // Pull before publishing. This makes the first reconciliation safe: remote + // state is applied or reported as a conflict before a pre-existing local + // vault can enqueue its bootstrap creates. + pulled, cursor, serverSequence, err := a.pullRemoteOps(client, lastPullSeq, !bootstrapComplete) + if err != nil { + return nil, err + } + if pulled > 0 { + if warnings, err := a.syncSvc.RebaseSnapshot(); err != nil { + return nil, fmt.Errorf("rebase remote snapshot: %w", err) + } else if err := a.syncSvc.SetLastWarning(strings.Join(warnings, "\n")); err != nil { + return nil, fmt.Errorf("save sync warning: %w", err) + } + } + if !bootstrapComplete { + if err := a.syncSvc.RecordBootstrapOps(initialSnapshot); err != nil { + return nil, fmt.Errorf("record initial local snapshot: %w", err) + } + if err := a.syncSvc.SetBootstrapComplete(true); err != nil { + return nil, fmt.Errorf("save bootstrap state: %w", err) + } + } + unpushed, err := a.syncSvc.GetUnpushedOps() if err != nil { return nil, fmt.Errorf("get ops: %w", err) } for i := range unpushed { - unpushed[i].LastSeenServerSeq = lastPullSeq + unpushed[i].LastSeenServerSeq = cursor } pushResult := &syncsvc.PushResponse{} if len(unpushed) > 0 { @@ -3289,50 +3469,35 @@ func (a *App) syncNow() (map[string]interface{}, error) { } } - pullResult, err := client.Pull(lastPullSeq) + // Pull once more so this device durably acknowledges its own accepted + // operations and any concurrent remote operations without reapplying its own. + pulledAfterPush, _, finalServerSequence, err := a.pullRemoteOps(client, cursor, false) if err != nil { - _ = a.updateSyncError(fmt.Sprintf("pull: %v", err)) - return nil, fmt.Errorf("pull: %w", err) + return nil, err } - - var applyErrors []string - for _, op := range pullResult.Ops { - if err := a.applyRemoteOp(op); err != nil { - applyErrors = append(applyErrors, fmt.Sprintf("%s/%s: %v", op.EntityType, op.OpID, err)) + if pulledAfterPush > 0 { + if warnings, err := a.syncSvc.RebaseSnapshot(); err != nil { + return nil, fmt.Errorf("rebase final remote snapshot: %w", err) + } else if err := a.syncSvc.SetLastWarning(strings.Join(warnings, "\n")); err != nil { + return nil, fmt.Errorf("save sync warning: %w", err) } - _ = a.syncSvc.RecordRemoteOp(op) } - if len(pullResult.Ops) > 0 { - opIDs := make([]string, len(pullResult.Ops)) - for i, op := range pullResult.Ops { - opIDs[i] = op.OpID - } - _ = a.syncSvc.MarkApplied(opIDs) + if finalServerSequence > serverSequence { + serverSequence = finalServerSequence } - if len(pushResult.Conflicts) > 0 { log.Printf("[sync] %d conflict(s) detected on push", len(pushResult.Conflicts)) - for _, c := range pushResult.Conflicts { - log.Printf("[sync] conflict: op=%v entity=%v/%v", - c["op_id"], c["entity_type"], c["entity_id"]) - } } - - if pullResult.ServerSequence > lastPullSeq { - _ = a.syncSvc.SetLastPullSeq(pullResult.ServerSequence) - } - _ = a.syncSvc.SetLastSyncAt(time.Now().UTC().Format(time.RFC3339)) - now := time.Now().UTC().Format(time.RFC3339) - a.updateSyncSuccess(now) + if err := a.syncSvc.SetLastSyncAt(now); err != nil { + return nil, fmt.Errorf("save sync time: %w", err) + } + _ = a.updateSyncSuccess(now) result := map[string]interface{}{ "pushed": len(pushResult.Accepted), - "pulled": len(pullResult.Ops), - "serverSequence": pullResult.ServerSequence, - } - if len(applyErrors) > 0 { - result["applyErrors"] = applyErrors + "pulled": pulled + pulledAfterPush, + "serverSequence": serverSequence, } if len(pushResult.Conflicts) > 0 { result["conflicts"] = pushResult.Conflicts @@ -3340,6 +3505,74 @@ func (a *App) syncNow() (map[string]interface{}, error) { return result, nil } +func (a *App) pullRemoteOps(client *syncsvc.Client, cursor int, initialReconciliation bool) (pulled, nextCursor, serverSequence int, err error) { + pullResult, err := client.Pull(cursor) + if err != nil { + _ = a.updateSyncError(fmt.Sprintf("pull: %v", err)) + return 0, cursor, cursor, fmt.Errorf("pull: %w", err) + } + nextCursor = cursor + lastSequenceInBatch := cursor + for _, op := range pullResult.Ops { + if op.ServerSequence <= cursor { + continue + } + if op.ServerSequence <= lastSequenceInBatch { + errMsg := fmt.Sprintf("pull response is not strictly ordered at sequence %d (%s)", op.ServerSequence, op.OpID) + _ = a.updateSyncError(errMsg) + return pulled, nextCursor, pullResult.ServerSequence, fmt.Errorf("%s", errMsg) + } + lastSequenceInBatch = op.ServerSequence + if err := a.applyRemoteOpForReconciliation(op, initialReconciliation); err != nil { + path := syncOperationPath(op) + errMsg := fmt.Sprintf("pull apply failed at sequence %d for %s %s (%s): %v", op.ServerSequence, op.EntityType, path, op.OpID, err) + _ = a.updateSyncError(errMsg) + return pulled, nextCursor, pullResult.ServerSequence, fmt.Errorf("%s", errMsg) + } + if err := a.syncSvc.RecordRemoteOp(op); err != nil { + errMsg := fmt.Sprintf("record applied remote operation at sequence %d (%s): %v", op.ServerSequence, op.OpID, err) + _ = a.updateSyncError(errMsg) + return pulled, nextCursor, pullResult.ServerSequence, fmt.Errorf("%s", errMsg) + } + if err := a.syncSvc.SetLastPullSeq(op.ServerSequence); err != nil { + errMsg := fmt.Sprintf("save pull cursor at sequence %d (%s): %v", op.ServerSequence, op.OpID, err) + _ = a.updateSyncError(errMsg) + return pulled, nextCursor, pullResult.ServerSequence, fmt.Errorf("%s", errMsg) + } + if warnings, err := a.syncSvc.RebaseSnapshot(); err != nil { + errMsg := fmt.Sprintf("rebase snapshot after sequence %d (%s): %v", op.ServerSequence, op.OpID, err) + _ = a.updateSyncError(errMsg) + return pulled, nextCursor, pullResult.ServerSequence, fmt.Errorf("%s", errMsg) + } else if err := a.syncSvc.SetLastWarning(strings.Join(warnings, "\n")); err != nil { + return pulled, nextCursor, pullResult.ServerSequence, fmt.Errorf("save sync warning: %w", err) + } + nextCursor = op.ServerSequence + pulled++ + } + if pullResult.ServerSequence > nextCursor { + if err := a.syncSvc.SetLastPullSeq(pullResult.ServerSequence); err != nil { + _ = a.updateSyncError(fmt.Sprintf("save pull cursor at sequence %d: %v", pullResult.ServerSequence, err)) + return pulled, nextCursor, pullResult.ServerSequence, fmt.Errorf("save pull cursor: %w", err) + } + nextCursor = pullResult.ServerSequence + } + return pulled, nextCursor, pullResult.ServerSequence, nil +} + +func syncOperationPath(op syncsvc.Op) string { + if op.EntityType == syncsvc.EntityWorkspace { + if payload, err := parseSyncWorkspacePayload(op.PayloadJSON); err == nil && payload.Path != "" { + return payload.Path + } + return op.EntityID + } + payload, _ := parseSyncFilePayload(op.PayloadJSON) + if path := syncPayloadPath(op, payload); path != "" { + return path + } + return op.EntityID +} + // PluginSyncNow triggers sync for a plugin with sync permission. func (a *App) PluginSyncNow(pluginID string) (map[string]interface{}, string) { if err := a.requirePluginSyncAccess(pluginID, true); err != nil { @@ -3368,12 +3601,23 @@ func (a *App) updateSyncSuccess(lastSyncAt string) error { } func (a *App) applyRemoteOp(op syncsvc.Op) error { + return a.applyRemoteOpForReconciliation(op, false) +} + +func (a *App) applyRemoteOpForReconciliation(op syncsvc.Op, initialReconciliation bool) error { if a.debug { log.Printf("[sync] applyRemoteOp: type=%s entity=%s/%s", op.OpType, op.EntityType, op.EntityID) } if op.DeviceID != "" && op.DeviceID == a.localSyncDeviceID() { return nil } + if op.EntityType == syncsvc.EntityWorkspace { + payload, err := parseSyncWorkspacePayload(op.PayloadJSON) + if err != nil { + return err + } + return a.applyRemoteWorkspaceOp(op, payload) + } if a.files == nil { return fmt.Errorf("files service not initialized") } @@ -3384,20 +3628,29 @@ func (a *App) applyRemoteOp(op syncsvc.Op) error { } switch op.EntityType { case syncsvc.EntityFile: - return a.applyRemoteFileOp(op, payload) + return a.applyRemoteFileOp(op, payload, initialReconciliation) case syncsvc.EntityFolder: - return a.applyRemoteFolderOp(op, payload) + return a.applyRemoteFolderOp(op, payload, initialReconciliation) default: return fmt.Errorf("unsupported sync entity type: %s", op.EntityType) } } type syncFilePayload struct { - Path string `json:"path"` - Content string `json:"content"` - DataBase64 *string `json:"dataBase64"` - FromPath string `json:"fromPath"` - ToPath string `json:"toPath"` + Path string `json:"path"` + Content string `json:"content"` + DataBase64 *string `json:"dataBase64"` + ContentHash string `json:"contentHash"` + FromPath string `json:"fromPath"` + ToPath string `json:"toPath"` +} + +type syncWorkspacePayload struct { + WorkspaceID string `json:"workspaceId"` + Path string `json:"path"` + PreviousPath string `json:"previousPath,omitempty"` + Name string `json:"name"` + Metadata workspace.Metadata `json:"metadata"` } func parseSyncFilePayload(payloadJSON string) (syncFilePayload, error) { @@ -3411,13 +3664,64 @@ func parseSyncFilePayload(payloadJSON string) (syncFilePayload, error) { return payload, nil } -func (a *App) applyRemoteFileOp(op syncsvc.Op, payload syncFilePayload) error { +func parseSyncWorkspacePayload(payloadJSON string) (syncWorkspacePayload, error) { + if payloadJSON == "" { + return syncWorkspacePayload{}, fmt.Errorf("workspace sync payload is empty") + } + var payload syncWorkspacePayload + if err := json.Unmarshal([]byte(payloadJSON), &payload); err != nil { + return syncWorkspacePayload{}, fmt.Errorf("invalid workspace sync payload: %w", err) + } + if payload.WorkspaceID == "" { + return syncWorkspacePayload{}, fmt.Errorf("workspace sync payload is missing workspaceId") + } + if payload.Path == "" { + payload.Path = payload.Name + } + return payload, nil +} + +func (a *App) applyRemoteWorkspaceOp(op syncsvc.Op, payload syncWorkspacePayload) error { + if a.workspace == nil { + return fmt.Errorf("workspace service not initialized") + } + if payload.WorkspaceID != op.EntityID { + return fmt.Errorf("workspace identity mismatch: entity %s payload %s", op.EntityID, payload.WorkspaceID) + } + switch op.OpType { + case syncsvc.OpCreate: + _, err := a.workspace.CreateWorkspaceFromSync(payload.Path, payload.WorkspaceID, payload.Metadata) + return err + case syncsvc.OpRename: + return a.workspace.RenameWorkspaceFromSync(payload.WorkspaceID, payload.PreviousPath, payload.Path) + case syncsvc.OpTrash: + _, err := a.workspace.TrashWorkspaceFromSync(payload.WorkspaceID, payload.Path) + return err + case syncsvc.OpRestore: + _, err := a.workspace.RestoreWorkspaceFromSync(payload.WorkspaceID, payload.Path) + return err + default: + return fmt.Errorf("unsupported workspace sync op type: %s", op.OpType) + } +} + +func (a *App) applyRemoteFileOp(op syncsvc.Op, payload syncFilePayload, initialReconciliation bool) error { switch op.OpType { case syncsvc.OpCreate: path := syncPayloadPath(op, payload) if path == "" { return fmt.Errorf("missing file path") } + matches, exists, err := a.remoteFileMatches(path, payload) + if err != nil { + return err + } + if matches { + return nil + } + if exists { + return fmt.Errorf("conflict: remote create would replace local file %s", path) + } if payload.DataBase64 != nil { return a.files.WriteVaultFileBytes(path, *payload.DataBase64, corefiles.WriteOptions{CreateIfMissing: true}) } @@ -3427,15 +3731,44 @@ func (a *App) applyRemoteFileOp(op syncsvc.Op, payload syncFilePayload) error { if path == "" { return fmt.Errorf("missing file path") } - if payload.DataBase64 != nil { - return a.files.WriteVaultFileBytes(path, *payload.DataBase64, corefiles.WriteOptions{CreateIfMissing: true, Overwrite: true}) + matches, exists, err := a.remoteFileMatches(path, payload) + if err != nil { + return err } - return a.files.WriteVaultTextFile(path, payload.Content, corefiles.WriteOptions{CreateIfMissing: true, Overwrite: true}) + if matches { + return nil + } + if initialReconciliation && exists { + return fmt.Errorf("conflict: initial reconciliation would replace local file %s", path) + } + if pending, err := a.syncSvc.HasUnpushedPath(path); err != nil { + return err + } else if pending { + return fmt.Errorf("conflict: remote update would replace unpushed local file %s", path) + } + if payload.DataBase64 != nil { + return a.files.WriteVaultFileBytes(path, *payload.DataBase64, corefiles.WriteOptions{CreateIfMissing: !exists, Overwrite: exists}) + } + return a.files.WriteVaultTextFile(path, payload.Content, corefiles.WriteOptions{CreateIfMissing: !exists, Overwrite: exists}) case syncsvc.OpDelete: path := syncPayloadPath(op, payload) if path == "" { return fmt.Errorf("missing file path") } + if initialReconciliation { + exists, err := a.syncPathExists(path) + if err != nil { + return err + } + if exists { + return fmt.Errorf("conflict: initial reconciliation would delete local file %s", path) + } + } + if pending, err := a.syncSvc.HasUnpushedPath(path); err != nil { + return err + } else if pending { + return fmt.Errorf("conflict: remote delete would remove unpushed local file %s", path) + } _, err := a.files.TrashVaultPath(path) if isSyncNotFound(err) { return nil @@ -3449,6 +3782,24 @@ func (a *App) applyRemoteFileOp(op syncsvc.Op, payload syncFilePayload) error { if fromPath == "" || payload.ToPath == "" { return fmt.Errorf("missing file move path") } + if initialReconciliation { + fromExists, err := a.syncPathExists(fromPath) + if err != nil { + return err + } + toExists, err := a.syncPathExists(payload.ToPath) + if err != nil { + return err + } + if fromExists || toExists { + return fmt.Errorf("conflict: initial reconciliation would move local file %s", fromPath) + } + } + if pending, err := a.syncSvc.HasUnpushedPath(fromPath); err != nil { + return err + } else if pending { + return fmt.Errorf("conflict: remote move would replace unpushed local file %s", fromPath) + } err := a.files.MoveVaultPath(fromPath, payload.ToPath, corefiles.MoveOptions{}) if isSyncNotFound(err) { return nil @@ -3459,7 +3810,54 @@ func (a *App) applyRemoteFileOp(op syncsvc.Op, payload syncFilePayload) error { } } -func (a *App) applyRemoteFolderOp(op syncsvc.Op, payload syncFilePayload) error { +func (a *App) remoteFileMatches(path string, payload syncFilePayload) (matches, exists bool, err error) { + normalized, err := corefiles.NormalizeRelativeFile(path) + if err != nil { + return false, false, err + } + info, err := os.Lstat(filepath.Join(a.vaultPath(), filepath.FromSlash(normalized))) + if err != nil { + if os.IsNotExist(err) { + return false, false, nil + } + return false, false, err + } + if info.Mode()&os.ModeSymlink != 0 || !info.Mode().IsRegular() { + return false, true, fmt.Errorf("conflict: remote file target is not a regular file: %s", normalized) + } + want, err := remotePayloadHash(payload) + if err != nil { + return false, true, err + } + file, err := os.Open(filepath.Join(a.vaultPath(), filepath.FromSlash(normalized))) + if err != nil { + return false, true, err + } + defer file.Close() + hash := sha256.New() + if _, err := io.Copy(hash, file); err != nil { + return false, true, err + } + return fmt.Sprintf("%x", hash.Sum(nil)) == want, true, nil +} + +func remotePayloadHash(payload syncFilePayload) (string, error) { + if payload.ContentHash != "" { + return payload.ContentHash, nil + } + data := []byte(payload.Content) + if payload.DataBase64 != nil { + decoded, err := base64.StdEncoding.DecodeString(*payload.DataBase64) + if err != nil { + return "", fmt.Errorf("invalid remote base64 payload: %w", err) + } + data = decoded + } + sum := sha256.Sum256(data) + return fmt.Sprintf("%x", sum[:]), nil +} + +func (a *App) applyRemoteFolderOp(op syncsvc.Op, payload syncFilePayload, initialReconciliation bool) error { switch op.OpType { case syncsvc.OpCreate: path := syncPayloadPath(op, payload) @@ -3476,6 +3874,15 @@ func (a *App) applyRemoteFolderOp(op syncsvc.Op, payload syncFilePayload) error if path == "" { return fmt.Errorf("missing folder path") } + if initialReconciliation { + exists, err := a.syncPathExists(path) + if err != nil { + return err + } + if exists { + return fmt.Errorf("conflict: initial reconciliation would delete local folder %s", path) + } + } _, err := a.files.TrashVaultPath(path) if isSyncNotFound(err) { return nil @@ -3489,6 +3896,19 @@ func (a *App) applyRemoteFolderOp(op syncsvc.Op, payload syncFilePayload) error if fromPath == "" || payload.ToPath == "" { return fmt.Errorf("missing folder move path") } + if initialReconciliation { + fromExists, err := a.syncPathExists(fromPath) + if err != nil { + return err + } + toExists, err := a.syncPathExists(payload.ToPath) + if err != nil { + return err + } + if fromExists || toExists { + return fmt.Errorf("conflict: initial reconciliation would move local folder %s", fromPath) + } + } err := a.files.MoveVaultPath(fromPath, payload.ToPath, corefiles.MoveOptions{}) if isSyncNotFound(err) { return nil @@ -3499,6 +3919,21 @@ func (a *App) applyRemoteFolderOp(op syncsvc.Op, payload syncFilePayload) error } } +func (a *App) syncPathExists(path string) (bool, error) { + normalized, err := corefiles.NormalizeRelativeFile(path) + if err != nil { + return false, err + } + _, err = os.Lstat(filepath.Join(a.vaultPath(), filepath.FromSlash(normalized))) + if err == nil { + return true, nil + } + if os.IsNotExist(err) { + return false, nil + } + return false, err +} + func syncPayloadPath(op syncsvc.Op, payload syncFilePayload) string { if payload.Path != "" { return payload.Path diff --git a/internal/api/app_test.go b/internal/api/app_test.go index c611d4a..7016eeb 100644 --- a/internal/api/app_test.go +++ b/internal/api/app_test.go @@ -175,6 +175,12 @@ func newSyncFilesTestApp(t *testing.T, perms []string, deviceID string) (*App, s t.Helper() app, root := newFilesTestApp(t, perms) app.syncSvc = syncsvc.NewService(root, deviceID) + if _, err := app.syncSvc.ScanAndRecord(); err != nil { + t.Fatalf("initial sync snapshot: %v", err) + } + if err := app.syncSvc.SetBootstrapComplete(true); err != nil { + t.Fatalf("mark test sync bootstrap complete: %v", err) + } app.appSettings = appsettings.NewManager(filepath.Join(t.TempDir(), "config.json")) if err := app.appSettings.Load(); err != nil { t.Fatalf("settings Load: %v", err) @@ -1692,6 +1698,63 @@ func TestApplyRemoteOpSkipsLocalDevice(t *testing.T) { } } +func TestApplyRemoteWorkspaceLifecyclePreservesDurableIdentity(t *testing.T) { + app, root := newSyncFilesTestApp(t, []string{"files.read", "files.write", "files.delete"}, "local-device") + app.workspace = workspace.NewManager(root) + if err := app.workspace.Load(); err != nil { + t.Fatal(err) + } + workspaceID := "5f0f96d9-61c8-4b6b-8c3a-a1b9a0f40002" + payload := syncWorkspacePayload{ + WorkspaceID: workspaceID, + Path: "Remote", + Name: "Remote", + Metadata: workspace.Metadata{ + WorkspaceID: workspaceID, + WorkspaceName: "Remote", + Folders: map[string]string{"notes": "Notes"}, + Features: map[string]bool{"files": true}, + }, + } + encoded, err := json.Marshal(payload) + if err != nil { + t.Fatal(err) + } + create := syncsvc.Op{OpID: "workspace-create", DeviceID: "remote-device", EntityType: syncsvc.EntityWorkspace, EntityID: workspaceID, OpType: syncsvc.OpCreate, PayloadJSON: string(encoded)} + if err := app.applyRemoteOp(create); err != nil { + t.Fatalf("apply workspace create: %v", err) + } + if err := app.applyRemoteOp(create); err != nil { + t.Fatalf("replay workspace create: %v", err) + } + if identity, err := app.workspace.GetWorkspaceIdentity("Remote"); err != nil || identity.WorkspaceID != workspaceID { + t.Fatalf("remote workspace identity = %+v err=%v", identity, err) + } + + payload.PreviousPath = "Remote" + payload.Path = "Remote-Renamed" + payload.Name = "Remote-Renamed" + encoded, _ = json.Marshal(payload) + rename := syncsvc.Op{OpID: "workspace-rename", DeviceID: "remote-device", EntityType: syncsvc.EntityWorkspace, EntityID: workspaceID, OpType: syncsvc.OpRename, PayloadJSON: string(encoded)} + if err := app.applyRemoteOp(rename); err != nil { + t.Fatalf("apply workspace rename: %v", err) + } + trash := syncsvc.Op{OpID: "workspace-trash", DeviceID: "remote-device", EntityType: syncsvc.EntityWorkspace, EntityID: workspaceID, OpType: syncsvc.OpTrash, PayloadJSON: string(encoded)} + if err := app.applyRemoteOp(trash); err != nil { + t.Fatalf("apply workspace trash: %v", err) + } + payload.Path = "Remote-Restored" + payload.Name = "Remote-Restored" + encoded, _ = json.Marshal(payload) + restore := syncsvc.Op{OpID: "workspace-restore", DeviceID: "remote-device", EntityType: syncsvc.EntityWorkspace, EntityID: workspaceID, OpType: syncsvc.OpRestore, PayloadJSON: string(encoded)} + if err := app.applyRemoteOp(restore); err != nil { + t.Fatalf("apply workspace restore: %v", err) + } + if identity, err := app.workspace.GetWorkspaceIdentity("Remote-Restored"); err != nil || identity.WorkspaceID != workspaceID { + t.Fatalf("restored workspace identity = %+v err=%v", identity, err) + } +} + func TestFileBridgeRecordsSyncOps(t *testing.T) { app, _ := newSyncFilesTestApp(t, []string{"files.read", "files.write", "files.delete"}, "local-device") @@ -1718,8 +1781,8 @@ func TestFileBridgeRecordsSyncOps(t *testing.T) { if err != nil { t.Fatalf("GetUnpushedOps: %v", err) } - if len(ops) != 6 { - t.Fatalf("ops len = %d, want 6: %#v", len(ops), ops) + if len(ops) != 7 { + t.Fatalf("ops len = %d, want 7: %#v", len(ops), ops) } want := []struct { @@ -1732,7 +1795,10 @@ func TestFileBridgeRecordsSyncOps(t *testing.T) { {syncsvc.EntityFile, "Docs/one.txt", syncsvc.OpCreate, `"content":"hello"`}, {syncsvc.EntityFile, "Docs/one.txt", syncsvc.OpUpdate, `"content":"updated"`}, {syncsvc.EntityFile, "Docs/image.bin", syncsvc.OpCreate, `"dataBase64":"AQID"`}, - {syncsvc.EntityFile, "Docs/one.txt", syncsvc.OpMove, `"toPath":"Docs/two.txt"`}, + // Snapshot reconciliation represents external and API renames uniformly + // as a create at the destination followed by a delete at the source. + {syncsvc.EntityFile, "Docs/two.txt", syncsvc.OpCreate, `"content":"updated"`}, + {syncsvc.EntityFile, "Docs/one.txt", syncsvc.OpDelete, `"path":"Docs/one.txt"`}, {syncsvc.EntityFile, "Docs/two.txt", syncsvc.OpDelete, `"path":"Docs/two.txt"`}, } for i, w := range want { @@ -1854,8 +1920,8 @@ func TestSyncNowPushesLocalOpsAndAppliesPulledFileOps(t *testing.T) { t.Fatalf("pushed ops len = %d, want 2", len(pushedOps)) } for i, op := range pushedOps { - if op.LastSeenServerSeq != 0 { - t.Fatalf("pushed op[%d] last seen = %d, want 0", i, op.LastSeenServerSeq) + if op.LastSeenServerSeq != 2 { + t.Fatalf("pushed op[%d] last seen = %d, want 2 after initial pull", i, op.LastSeenServerSeq) } } @@ -1882,6 +1948,323 @@ func TestSyncNowPushesLocalOpsAndAppliesPulledFileOps(t *testing.T) { } } +func TestSyncNowStopsAtFailedRemoteOperationAndRetriesAfterRestart(t *testing.T) { + app, root := newSyncFilesTestApp(t, []string{"files.read", "files.write", "files.delete"}, "local-device") + pullCalls := 0 + server := newLocalHTTPTestServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") != "Bearer device-token" { + http.Error(w, "missing auth", http.StatusUnauthorized) + return + } + w.Header().Set("Content-Type", "application/json") + switch r.URL.Path { + case "/api/v1/sync/pull": + pullCalls++ + _ = json.NewEncoder(w).Encode(map[string]interface{}{ + "server_sequence": 3, + "ops": []map[string]interface{}{ + { + "op_id": "remote-folder", + "server_sequence": 1, + "device_id": "remote-device", + "entity_type": syncsvc.EntityFolder, + "entity_id": "Remote", + "op_type": syncsvc.OpCreate, + "payload_json": `{"path":"Remote"}`, + }, + { + "op_id": "blocked-file", + "server_sequence": 2, + "device_id": "remote-device", + "entity_type": syncsvc.EntityFile, + "entity_id": "Missing/blocked.txt", + "op_type": syncsvc.OpCreate, + "payload_json": `{"path":"Missing/blocked.txt","content":"blocked"}`, + }, + { + "op_id": "after-failure", + "server_sequence": 3, + "device_id": "remote-device", + "entity_type": syncsvc.EntityFile, + "entity_id": "Remote/after.txt", + "op_type": syncsvc.OpCreate, + "payload_json": `{"path":"Remote/after.txt","content":"must wait"}`, + }, + }, + }) + case "/api/v1/sync/push": + _ = json.NewEncoder(w).Encode(map[string]interface{}{"accepted": []string{}, "count": 0, "conflicts": []map[string]interface{}{}}) + default: + http.NotFound(w, r) + } + })) + defer server.Close() + if err := app.syncSvc.SetState(server.URL, ""); err != nil { + t.Fatal(err) + } + if err := syncsvc.SaveDeviceToken(root, "device-token"); err != nil { + t.Fatal(err) + } + + if _, err := app.syncNow(); err == nil || !strings.Contains(err.Error(), "sequence 2") { + t.Fatalf("first sync error = %v, want failed sequence", err) + } + assertLastPullSequence(t, app.syncSvc, 1) + if _, errStr := app.GetVaultFileMetadata("files.plugin", "Remote/after.txt"); !strings.Contains(errStr, "not-found") { + t.Fatalf("operation after failure applied early: %q", errStr) + } + cfg := app.appSettings.Get() + if cfg.Sync.LastError == "" || !strings.Contains(cfg.Sync.LastError, "sequence 2") || !strings.Contains(cfg.Sync.LastError, "Missing/blocked.txt") { + t.Fatalf("sync status after failed apply = %#v", cfg.Sync) + } + + if err := os.Mkdir(filepath.Join(root, "Missing"), 0o755); err != nil { + t.Fatal(err) + } + restarted := &App{ + vault: app.vault, + files: app.files, + plugins: app.plugins, + appSettings: app.appSettings, + syncSvc: syncsvc.NewService(root, ""), + } + if _, err := restarted.syncNow(); err != nil { + t.Fatalf("retry after restart: %v", err) + } + if pullCalls < 2 { + t.Fatalf("pull calls = %d, want retry", pullCalls) + } + assertLastPullSequence(t, restarted.syncSvc, 3) + expectText(t, restarted, "Missing/blocked.txt", "blocked") + expectText(t, restarted, "Remote/after.txt", "must wait") +} + +func TestSyncNowBootstrapsExistingLocalSnapshotAfterInitialPull(t *testing.T) { + app, root := newFilesTestApp(t, []string{"files.read", "files.write", "files.delete"}) + if err := os.Mkdir(filepath.Join(root, "Existing"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "Existing", "before-connect.txt"), []byte("local before connect"), 0o644); err != nil { + t.Fatal(err) + } + app.syncSvc = syncsvc.NewService(root, "local-device") + if _, err := app.syncSvc.ScanAndRecord(); err != nil { + t.Fatalf("initial baseline scan: %v", err) + } + app.appSettings = appsettings.NewManager(filepath.Join(t.TempDir(), "config.json")) + if err := app.appSettings.Load(); err != nil { + t.Fatal(err) + } + + var pushed []syncsvc.PushOp + server := newLocalHTTPTestServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") != "Bearer device-token" { + http.Error(w, "missing auth", http.StatusUnauthorized) + return + } + w.Header().Set("Content-Type", "application/json") + switch r.URL.Path { + case "/api/v1/sync/pull": + _ = json.NewEncoder(w).Encode(map[string]interface{}{"server_sequence": 0, "ops": []map[string]interface{}{}}) + case "/api/v1/sync/push": + var request struct { + Ops []syncsvc.PushOp `json:"ops"` + } + if err := json.NewDecoder(r.Body).Decode(&request); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + pushed = request.Ops + accepted := make([]string, 0, len(request.Ops)) + for _, op := range request.Ops { + accepted = append(accepted, op.OpID) + } + _ = json.NewEncoder(w).Encode(map[string]interface{}{"accepted": accepted, "count": len(accepted), "conflicts": []map[string]interface{}{}}) + default: + http.NotFound(w, r) + } + })) + defer server.Close() + if err := app.syncSvc.SetState(server.URL, ""); err != nil { + t.Fatal(err) + } + if err := syncsvc.SaveDeviceToken(root, "device-token"); err != nil { + t.Fatal(err) + } + + result, err := app.syncNow() + if err != nil { + t.Fatalf("syncNow: %v", err) + } + if result["pushed"] != len(pushed) || len(pushed) < 2 { + t.Fatalf("result=%#v pushed=%#v, want initial entries", result, pushed) + } + foundFolder, foundFile := false, false + for _, op := range pushed { + if op.EntityID == "Existing" && op.EntityType == syncsvc.EntityFolder && op.OpType == syncsvc.OpCreate { + foundFolder = true + } + if op.EntityID == "Existing/before-connect.txt" && op.EntityType == syncsvc.EntityFile && op.OpType == syncsvc.OpCreate { + foundFile = true + } + if strings.Contains(op.EntityID, "/.verstak/") { + t.Fatalf("ordinary bootstrap operation leaked internal path: %+v", op) + } + } + if !foundFolder || !foundFile { + t.Fatalf("bootstrap push = %#v, missing existing file or folder", pushed) + } + bootstrapped, err := app.syncSvc.BootstrapComplete() + if err != nil || !bootstrapped { + t.Fatalf("bootstrap complete = %v err=%v", bootstrapped, err) + } +} + +func TestInitialSyncConflictDoesNotOverwriteLocalFileOrAdvanceCursor(t *testing.T) { + app, root := newFilesTestApp(t, []string{"files.read", "files.write", "files.delete"}) + if err := os.WriteFile(filepath.Join(root, "same-name.txt"), []byte("local value"), 0o644); err != nil { + t.Fatal(err) + } + app.syncSvc = syncsvc.NewService(root, "local-device") + if _, err := app.syncSvc.ScanAndRecord(); err != nil { + t.Fatalf("initial baseline scan: %v", err) + } + app.appSettings = appsettings.NewManager(filepath.Join(t.TempDir(), "config.json")) + if err := app.appSettings.Load(); err != nil { + t.Fatal(err) + } + + pushCalls := 0 + server := newLocalHTTPTestServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") != "Bearer device-token" { + http.Error(w, "missing auth", http.StatusUnauthorized) + return + } + w.Header().Set("Content-Type", "application/json") + switch r.URL.Path { + case "/api/v1/sync/pull": + _ = json.NewEncoder(w).Encode(map[string]interface{}{ + "server_sequence": 1, + "ops": []map[string]interface{}{{ + "op_id": "remote-update", + "server_sequence": 1, + "device_id": "remote-device", + "entity_type": syncsvc.EntityFile, + "entity_id": "same-name.txt", + "op_type": syncsvc.OpUpdate, + "payload_json": `{"path":"same-name.txt","content":"remote value"}`, + }}, + }) + case "/api/v1/sync/push": + pushCalls++ + _ = json.NewEncoder(w).Encode(map[string]interface{}{"accepted": []string{}, "count": 0, "conflicts": []map[string]interface{}{}}) + default: + http.NotFound(w, r) + } + })) + defer server.Close() + if err := app.syncSvc.SetState(server.URL, ""); err != nil { + t.Fatal(err) + } + if err := syncsvc.SaveDeviceToken(root, "device-token"); err != nil { + t.Fatal(err) + } + + if _, err := app.syncNow(); err == nil || !strings.Contains(err.Error(), "initial reconciliation") || !strings.Contains(err.Error(), "same-name.txt") { + t.Fatalf("sync error = %v, want explicit initial conflict", err) + } + expectText(t, app, "same-name.txt", "local value") + assertLastPullSequence(t, app.syncSvc, 0) + if pushCalls != 0 { + t.Fatalf("push calls = %d, conflict must not overwrite remote state", pushCalls) + } + bootstrapped, err := app.syncSvc.BootstrapComplete() + if err != nil || bootstrapped { + t.Fatalf("bootstrap state = %v err=%v, conflict must remain unresolved", bootstrapped, err) + } +} + +func TestInitialSyncEmptyVaultRestoresRemoteDataWithoutDeletePush(t *testing.T) { + app, root := newFilesTestApp(t, []string{"files.read", "files.write", "files.delete"}) + app.syncSvc = syncsvc.NewService(root, "empty-device") + if _, err := app.syncSvc.ScanAndRecord(); err != nil { + t.Fatalf("initial empty baseline: %v", err) + } + app.appSettings = appsettings.NewManager(filepath.Join(t.TempDir(), "config.json")) + if err := app.appSettings.Load(); err != nil { + t.Fatal(err) + } + + var pushed []syncsvc.PushOp + server := newLocalHTTPTestServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.Header.Get("Authorization") != "Bearer device-token" { + http.Error(w, "missing auth", http.StatusUnauthorized) + return + } + w.Header().Set("Content-Type", "application/json") + switch r.URL.Path { + case "/api/v1/sync/pull": + _ = json.NewEncoder(w).Encode(map[string]interface{}{ + "server_sequence": 1, + "ops": []map[string]interface{}{{ + "op_id": "remote-file", + "server_sequence": 1, + "device_id": "remote-device", + "entity_type": syncsvc.EntityFile, + "entity_id": "restored.txt", + "op_type": syncsvc.OpCreate, + "payload_json": `{"path":"restored.txt","content":"from remote"}`, + }}, + }) + case "/api/v1/sync/push": + var request struct { + Ops []syncsvc.PushOp `json:"ops"` + } + if err := json.NewDecoder(r.Body).Decode(&request); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + pushed = append(pushed, request.Ops...) + _ = json.NewEncoder(w).Encode(map[string]interface{}{"accepted": []string{}, "count": 0, "conflicts": []map[string]interface{}{}}) + default: + http.NotFound(w, r) + } + })) + defer server.Close() + if err := app.syncSvc.SetState(server.URL, ""); err != nil { + t.Fatal(err) + } + if err := syncsvc.SaveDeviceToken(root, "device-token"); err != nil { + t.Fatal(err) + } + + result, err := app.syncNow() + if err != nil { + t.Fatalf("sync empty vault: %v", err) + } + expectText(t, app, "restored.txt", "from remote") + for _, op := range pushed { + if op.OpType == syncsvc.OpDelete { + t.Fatalf("empty vault published delete operation: %+v", op) + } + } + if result["pushed"] != 0 && len(pushed) == 0 { + t.Fatalf("result reports pushed operations without a push payload: %#v", result) + } + assertLastPullSequence(t, app.syncSvc, 1) +} + +func assertLastPullSequence(t *testing.T, service *syncsvc.Service, want int) { + t.Helper() + _, _, got, _, err := service.GetState() + if err != nil { + t.Fatalf("GetState: %v", err) + } + if got != want { + t.Fatalf("last pull sequence = %d, want %d", got, want) + } +} + func TestSyncConfigurePairsCurrentVaultID(t *testing.T) { app, _ := newSyncFilesTestApp(t, []string{"files.read", "files.write", "files.delete"}, "local-device") meta := app.vault.GetVaultMeta() @@ -1911,7 +2294,7 @@ func TestSyncConfigurePairsCurrentVaultID(t *testing.T) { })) defer server.Close() - if err := app.syncConfigure(server.URL, "alice", "secret"); err != nil { + if err := app.syncConfigure(server.URL, "alice", "secret", ""); err != nil { t.Fatalf("syncConfigure: %v", err) } if pairedVaultID != meta.VaultID { @@ -1919,6 +2302,79 @@ func TestSyncConfigurePairsCurrentVaultID(t *testing.T) { } } +func TestSyncConfigurePairsSpecifiedRemoteVaultID(t *testing.T) { + app, _ := newSyncFilesTestApp(t, []string{"files.read", "files.write", "files.delete"}, "local-device") + const remoteVaultID = "existing-remote-vault" + var pairedVaultID string + server := newLocalHTTPTestServer(t, http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { + if r.URL.Path != "/api/client/pair" { + http.NotFound(w, r) + return + } + var request struct { + VaultID string `json:"vault_id"` + } + if err := json.NewDecoder(r.Body).Decode(&request); err != nil { + http.Error(w, err.Error(), http.StatusBadRequest) + return + } + pairedVaultID = request.VaultID + w.Header().Set("Content-Type", "application/json") + _ = json.NewEncoder(w).Encode(map[string]string{ + "device_id": "paired-device", + "device_token": "paired-token", + }) + })) + defer server.Close() + + if err := app.syncConfigure(server.URL, "alice", "secret", remoteVaultID); err != nil { + t.Fatalf("syncConfigure: %v", err) + } + if pairedVaultID != remoteVaultID { + t.Fatalf("paired vault ID = %q, want %q", pairedVaultID, remoteVaultID) + } + storedRemoteVaultID, err := app.syncSvc.RemoteVaultID() + if err != nil || storedRemoteVaultID != remoteVaultID { + t.Fatalf("stored remote vault ID = %q err=%v, want %q", storedRemoteVaultID, err, remoteVaultID) + } + bootstrapped, err := app.syncSvc.BootstrapComplete() + if err != nil || bootstrapped { + t.Fatalf("bootstrap state after new pairing = %v err=%v, want false", bootstrapped, err) + } +} + +func TestSyncConfigureRefusesRemoteScopeChangeWithUnpushedOperations(t *testing.T) { + app, _ := newSyncFilesTestApp(t, []string{"files.read", "files.write", "files.delete"}, "local-device") + if err := app.syncSvc.SetState("https://old-sync.example.test", ""); err != nil { + t.Fatal(err) + } + if err := app.syncSvc.SetRemoteVaultID("old-remote-vault"); err != nil { + t.Fatal(err) + } + if err := app.syncSvc.RecordOp(syncsvc.EntityFile, "pending.txt", syncsvc.OpCreate, map[string]string{"path": "pending.txt", "content": "local"}); err != nil { + t.Fatal(err) + } + + err := app.syncConfigure("https://new-sync.example.test", "alice", "secret", "new-remote-vault") + if err == nil || !strings.Contains(err.Error(), "unpushed local operation") { + t.Fatalf("scope-change error = %v, want pending-operation refusal", err) + } +} + +func TestSyncStatusExposesPersistentScannerWarning(t *testing.T) { + app, _ := newSyncFilesTestApp(t, []string{"files.read", "files.write", "files.delete"}, "local-device") + if err := app.syncSvc.SetLastWarning("file-too-large: archive.bin"); err != nil { + t.Fatal(err) + } + status, err := app.syncStatus() + if err != nil { + t.Fatalf("syncStatus: %v", err) + } + if status.LastWarning != "file-too-large: archive.bin" { + t.Fatalf("last warning = %q", status.LastWarning) + } +} + func TestSyncNowHydratesLegacyVaultDeviceID(t *testing.T) { app, root := newSyncFilesTestApp(t, []string{"files.read", "files.write", "files.delete"}, "wrong-global-device") var pushedDeviceID string diff --git a/internal/api/sync_real_server_test.go b/internal/api/sync_real_server_test.go index b17a8da..1aa5060 100644 --- a/internal/api/sync_real_server_test.go +++ b/internal/api/sync_real_server_test.go @@ -2,10 +2,12 @@ package api import ( "os" + "path/filepath" "strings" "testing" corefiles "github.com/verstak/verstak-desktop/internal/core/files" + "github.com/verstak/verstak-desktop/internal/core/workspace" ) func TestSyncNowAgainstRealServerTwoVaults(t *testing.T) { @@ -18,8 +20,16 @@ func TestSyncNowAgainstRealServerTwoVaults(t *testing.T) { t.Skip("set VERSTAK_SYNC_SMOKE_* env vars to run the real sync-server smoke test") } - appA, _ := newSyncFilesTestApp(t, []string{"files.read", "files.write", "files.delete"}, deviceA) - appB, _ := newSyncFilesTestApp(t, []string{"files.read", "files.write", "files.delete"}, deviceB) + appA, rootA := newSyncFilesTestApp(t, []string{"files.read", "files.write", "files.delete"}, deviceA) + appB, rootB := newSyncFilesTestApp(t, []string{"files.read", "files.write", "files.delete"}, deviceB) + appA.workspace = workspace.NewManager(rootA) + appB.workspace = workspace.NewManager(rootB) + if err := appA.workspace.Load(); err != nil { + t.Fatalf("load workspace A: %v", err) + } + if err := appB.workspace.Load(); err != nil { + t.Fatalf("load workspace B: %v", err) + } if err := appA.syncSvc.SetState(serverURL, apiKeyA); err != nil { t.Fatalf("appA SetState: %v", err) } @@ -43,8 +53,8 @@ func TestSyncNowAgainstRealServerTwoVaults(t *testing.T) { if errStr := appB.MoveVaultPath("files.plugin", "Shared/one.txt", "Shared/two.txt", corefiles.MoveOptions{}); errStr != "" { t.Fatalf("appB move: %s", errStr) } - expectSyncCounts(t, appB, 2, 2) - expectSyncCounts(t, appA, 0, 2) + expectSyncCounts(t, appB, 3, 3) + expectSyncCounts(t, appA, 0, 3) expectText(t, appA, "Shared/two.txt", "from B") if _, errStr := appA.TrashVaultPath("files.plugin", "Shared/two.txt"); errStr != "" { @@ -68,8 +78,8 @@ func TestSyncNowAgainstRealServerTwoVaults(t *testing.T) { if errStr := appA.MoveVaultPath("files.plugin", "Shared/Folder", "Shared/Archive", corefiles.MoveOptions{}); errStr != "" { t.Fatalf("appA move folder: %s", errStr) } - expectSyncCounts(t, appA, 1, 1) - expectSyncCounts(t, appB, 0, 1) + expectSyncCounts(t, appA, 2, 2) + expectSyncCounts(t, appB, 0, 2) if _, errStr := appB.GetVaultFileMetadata("files.plugin", "Shared/Folder"); !strings.Contains(errStr, "not-found") { t.Fatalf("appB moved folder old metadata err = %q, want not-found", errStr) } @@ -85,6 +95,63 @@ func TestSyncNowAgainstRealServerTwoVaults(t *testing.T) { if _, errStr := appA.GetVaultFileMetadata("files.plugin", "Shared/Archive"); !strings.Contains(errStr, "not-found") { t.Fatalf("appA deleted folder metadata err = %q, want not-found", errStr) } + + if err := os.WriteFile(filepath.Join(rootA, "Shared", "external.txt"), []byte("external while running"), 0o644); err != nil { + t.Fatalf("external create: %v", err) + } + expectSyncCounts(t, appA, 1, 1) + expectSyncCounts(t, appB, 0, 1) + expectText(t, appB, "Shared/external.txt", "external while running") + assertNoUnpushedOps(t, appB) + + if err := os.WriteFile(filepath.Join(rootB, "Shared", "external.txt"), []byte("external while closed"), 0o644); err != nil { + t.Fatalf("offline external update: %v", err) + } + if err := os.WriteFile(filepath.Join(rootB, "Shared", "offline-created.txt"), []byte("created while closed"), 0o644); err != nil { + t.Fatalf("offline external create: %v", err) + } + if err := os.Remove(filepath.Join(rootB, "Shared", "external.txt")); err != nil { + t.Fatalf("offline external delete: %v", err) + } + expectSyncCounts(t, appB, 2, 2) + expectSyncCounts(t, appA, 0, 2) + if _, errStr := appA.GetVaultFileMetadata("files.plugin", "Shared/external.txt"); !strings.Contains(errStr, "not-found") { + t.Fatalf("offline deleted file remained on appA: %q", errStr) + } + expectText(t, appA, "Shared/offline-created.txt", "created while closed") + assertNoUnpushedOps(t, appA) + + deal, errStr := appA.CreateWorkspace("Synced Deal", "minimal") + if errStr != "" { + t.Fatalf("appA CreateWorkspace: %s", errStr) + } + expectSyncCounts(t, appA, 1, 1) + expectSyncCounts(t, appB, 0, 1) + assertWorkspaceIdentity(t, appB, "Synced Deal", deal.ID) + + if errStr := appA.RenameWorkspace("Synced Deal", "Renamed Deal"); errStr != "" { + t.Fatalf("appA RenameWorkspace: %s", errStr) + } + expectSyncCounts(t, appA, 1, 1) + expectSyncCounts(t, appB, 0, 1) + assertWorkspaceIdentity(t, appB, "Renamed Deal", deal.ID) + + trash, errStr := appA.TrashWorkspace("Renamed Deal") + if errStr != "" { + t.Fatalf("appA TrashWorkspace: %s", errStr) + } + expectSyncCounts(t, appA, 1, 1) + expectSyncCounts(t, appB, 0, 1) + if _, err := appB.workspace.GetWorkspaceIdentity("Renamed Deal"); err == nil { + t.Fatal("trashed workspace is still active on appB") + } + + if _, errStr := appA.RestoreWorkspaceTrash(trash.TrashID, "Restored Deal"); errStr != "" { + t.Fatalf("appA RestoreWorkspaceTrash: %s", errStr) + } + expectSyncCounts(t, appA, 1, 1) + expectSyncCounts(t, appB, 0, 1) + assertWorkspaceIdentity(t, appB, "Restored Deal", deal.ID) } func expectSyncCounts(t *testing.T, app *App, pushed, pulled int) { @@ -108,3 +175,25 @@ func expectText(t *testing.T, app *App, path, want string) { t.Fatalf("ReadVaultTextFile(%s) = %q, want %q", path, text, want) } } + +func assertNoUnpushedOps(t *testing.T, app *App) { + t.Helper() + ops, err := app.syncSvc.GetUnpushedOps() + if err != nil { + t.Fatalf("GetUnpushedOps: %v", err) + } + if len(ops) != 0 { + t.Fatalf("remote operations were echoed as local operations: %#v", ops) + } +} + +func assertWorkspaceIdentity(t *testing.T, app *App, name, wantID string) { + t.Helper() + identity, err := app.workspace.GetWorkspaceIdentity(name) + if err != nil { + t.Fatalf("GetWorkspaceIdentity(%s): %v", name, err) + } + if identity.WorkspaceID != wantID { + t.Fatalf("workspace %s ID = %s, want %s", name, identity.WorkspaceID, wantID) + } +} diff --git a/internal/core/files/path_policy.go b/internal/core/files/path_policy.go index 5b27df6..a96ecd6 100644 --- a/internal/core/files/path_policy.go +++ b/internal/core/files/path_policy.go @@ -25,8 +25,7 @@ func IsReservedPath(relativePath string) bool { if cleaned == "" { return false } - first := strings.Split(cleaned, "/")[0] - return strings.EqualFold(first, ".verstak") + return containsReservedSegment(cleaned) } func normalizeRelativePath(input string, allowRoot bool) (string, error) { @@ -67,8 +66,16 @@ func IsReservedPathNoNormalize(cleaned string) bool { if cleaned == "" { return false } - first := strings.Split(cleaned, "/")[0] - return strings.EqualFold(first, ".verstak") + return containsReservedSegment(cleaned) +} + +func containsReservedSegment(cleaned string) bool { + for _, segment := range strings.Split(cleaned, "/") { + if strings.EqualFold(segment, ".verstak") { + return true + } + } + return false } func looksAbsolute(input string) bool { diff --git a/internal/core/files/path_policy_test.go b/internal/core/files/path_policy_test.go index 285e5c5..66ce2c3 100644 --- a/internal/core/files/path_policy_test.go +++ b/internal/core/files/path_policy_test.go @@ -43,6 +43,7 @@ func TestNormalizeRelativeFileRejectsUnsafePaths(t *testing.T) { ".verstak/vault.json", "./.verstak", ".verstak/trash", + "Workspace/.verstak/workspace.json", "folder/../.verstak", ".Verstak", } @@ -70,6 +71,9 @@ func TestReservedPathPolicy(t *testing.T) { if IsReservedPath("Notes/.verstak.md") { t.Fatal("Notes/.verstak.md should not be reserved") } + if !IsReservedPath("Workspace/.verstak/workspace.json") { + t.Fatal("nested .verstak should be reserved") + } } func TestNormalizeRelativeFileAcceptsOnlySlashSeparatedRelativePaths(t *testing.T) { diff --git a/internal/core/filewatcher/service.go b/internal/core/filewatcher/service.go index 97cbf1d..d6e522f 100644 --- a/internal/core/filewatcher/service.go +++ b/internal/core/filewatcher/service.go @@ -35,11 +35,23 @@ type Service struct { bus *events.Bus interval time.Duration - mu sync.Mutex - root string - cancel chan struct{} - done chan struct{} - current map[string]snapshotEntry + mu sync.Mutex + root string + cancel chan struct{} + done chan struct{} + current map[string]snapshotEntry + onChange func() +} + +// SetOnChange installs a lightweight notification used by core services that +// need a debounced reconciliation after the watcher has observed a change. +func (s *Service) SetOnChange(callback func()) { + if s == nil { + return + } + s.mu.Lock() + s.onChange = callback + s.mu.Unlock() } // NewService creates a watcher. The interval parameter is mainly for tests. @@ -133,22 +145,30 @@ func (s *Service) poll(root string) { s.mu.Lock() prev := s.current s.current = next + callback := s.onChange s.mu.Unlock() + changed := false for path, entry := range next { old, ok := prev[path] if !ok { s.publish(path, "external.create", entry.kind) + changed = true continue } if entry.kind == entryFile && (entry.size != old.size || !entry.modTime.Equal(old.modTime)) { s.publish(path, "external.update", entry.kind) + changed = true } } for path, entry := range prev { if _, ok := next[path]; !ok { s.publish(path, "external.delete", entry.kind) + changed = true } } + if changed && callback != nil { + callback() + } } func (s *Service) publish(path, operation string, kind entryKind) { @@ -217,8 +237,12 @@ func kindFromInfo(info fs.FileInfo) entryKind { } func isReserved(rel string) bool { - first := strings.Split(filepath.ToSlash(rel), "/")[0] - return strings.EqualFold(first, ".verstak") + for _, segment := range strings.Split(filepath.ToSlash(rel), "/") { + if strings.EqualFold(segment, ".verstak") { + return true + } + } + return false } func workspaceRoot(path string) string { diff --git a/internal/core/filewatcher/service_test.go b/internal/core/filewatcher/service_test.go index dc7c51f..e7201ad 100644 --- a/internal/core/filewatcher/service_test.go +++ b/internal/core/filewatcher/service_test.go @@ -74,6 +74,26 @@ func TestServiceIgnoresReservedVerstakPaths(t *testing.T) { } } +func TestServiceCallsChangeCallbackForExternalChanges(t *testing.T) { + root := t.TempDir() + service := NewService(events.NewBus(), 10*time.Millisecond) + changed := make(chan struct{}, 1) + service.SetOnChange(func() { changed <- struct{}{} }) + if err := service.Start(root); err != nil { + t.Fatalf("Start: %v", err) + } + t.Cleanup(service.Stop) + + if err := os.WriteFile(filepath.Join(root, "external.txt"), []byte("change"), 0o644); err != nil { + t.Fatal(err) + } + select { + case <-changed: + case <-time.After(500 * time.Millisecond): + t.Fatal("timed out waiting for watcher callback") + } +} + func waitForEvent(t *testing.T, eventCh <-chan events.Event) events.Event { t.Helper() select { diff --git a/internal/core/sync/service.go b/internal/core/sync/service.go index 18212d8..fd36725 100644 --- a/internal/core/sync/service.go +++ b/internal/core/sync/service.go @@ -5,25 +5,30 @@ import ( "fmt" "os" "path/filepath" + "strings" "time" "github.com/google/uuid" ) const ( - EntityNode = "node" - EntityNote = "note" - EntityFile = "file" - EntityFolder = "folder" - EntityAction = "action" - EntityWorklog = "worklog" + EntityNode = "node" + EntityNote = "note" + EntityFile = "file" + EntityFolder = "folder" + EntityWorkspace = "workspace" + EntityAction = "action" + EntityWorklog = "worklog" ) const ( - OpCreate = "create" - OpUpdate = "update" - OpDelete = "delete" - OpMove = "move" + OpCreate = "create" + OpUpdate = "update" + OpDelete = "delete" + OpMove = "move" + OpRename = "rename" + OpTrash = "trash" + OpRestore = "restore" ) // Op represents a sync operation. @@ -45,11 +50,14 @@ type Op struct { // syncState persists connection state to JSON file. type syncState struct { - ServerURL string `json:"server_url"` - APIKey string `json:"api_key"` - DeviceID string `json:"device_id"` - LastPullSeq int `json:"last_pull_seq"` - LastSyncAt string `json:"last_sync_at"` + ServerURL string `json:"server_url"` + APIKey string `json:"api_key"` + DeviceID string `json:"device_id"` + LastPullSeq int `json:"last_pull_seq"` + LastSyncAt string `json:"last_sync_at"` + BootstrapComplete bool `json:"bootstrap_complete"` + LastWarning string `json:"last_warning"` + RemoteVaultID string `json:"remote_vault_id"` } // Service records and manages sync operations using JSON file storage. @@ -81,6 +89,14 @@ func (s *Service) statePath() string { return filepath.Join(s.syncDir(), "state.json") } +func (s *Service) snapshotPath() string { + return filepath.Join(s.syncDir(), "snapshot.json") +} + +func (s *Service) scanJournalPath() string { + return filepath.Join(s.syncDir(), "scan-journal.json") +} + func (s *Service) ensureDir() error { return os.MkdirAll(s.syncDir(), 0o755) } @@ -113,11 +129,33 @@ func (s *Service) RecordOp(entityType, entityID, opType string, payload interfac CreatedAt: now, } + return s.recordOps([]Op{op}) +} + +// recordOps is idempotent by op ID so a scanner recovery journal can safely +// resume after a crash between recording operations and replacing its snapshot. +func (s *Service) recordOps(newOps []Op) error { + if err := s.ensureDir(); err != nil { + return err + } ops, err := s.loadOps() if err != nil { return err } - ops = append(ops, op) + existing := make(map[string]bool, len(ops)) + for _, op := range ops { + existing[op.OpID] = true + } + for _, op := range newOps { + if op.OpID == "" || existing[op.OpID] { + continue + } + if op.ID == "" { + op.ID = op.OpID + } + ops = append(ops, op) + existing[op.OpID] = true + } return s.saveOps(ops) } @@ -160,6 +198,42 @@ func (s *Service) GetUnpushedOps() ([]Op, error) { return unpushed, nil } +// HasUnpushedPath reports whether a local operation still owns a path (or one +// of its descendants). Pull uses it to turn an incoming overwrite/delete into +// a visible conflict instead of silently replacing a local external edit. +func (s *Service) HasUnpushedPath(path string) (bool, error) { + ops, err := s.GetUnpushedOps() + if err != nil { + return false, err + } + for _, op := range ops { + if syncPathsOverlap(path, op.EntityID) { + return true, nil + } + var payload struct { + Path string `json:"path"` + FromPath string `json:"fromPath"` + ToPath string `json:"toPath"` + } + if op.PayloadJSON == "" || json.Unmarshal([]byte(op.PayloadJSON), &payload) != nil { + continue + } + if syncPathsOverlap(path, payload.Path) || syncPathsOverlap(path, payload.FromPath) || syncPathsOverlap(path, payload.ToPath) { + return true, nil + } + } + return false, nil +} + +func syncPathsOverlap(left, right string) bool { + left = strings.Trim(left, "/") + right = strings.Trim(right, "/") + if left == "" || right == "" { + return false + } + return left == right || strings.HasPrefix(left, right+"/") || strings.HasPrefix(right, left+"/") +} + // MarkPushed marks ops as pushed to server. func (s *Service) MarkPushed(opIDs []string) error { ops, err := s.loadOps() @@ -244,6 +318,66 @@ func (s *Service) SetLastSyncAt(t string) error { return s.saveState(st) } +// BootstrapComplete reports whether the initial pull/reconcile/bootstrap cycle +// finished successfully for this vault connection. +func (s *Service) BootstrapComplete() (bool, error) { + st, err := s.loadState() + if err != nil { + return false, err + } + return st.BootstrapComplete, nil +} + +// SetBootstrapComplete marks the initial reconciliation as complete only after +// all remote operations were applied and the local initial snapshot was queued. +func (s *Service) SetBootstrapComplete(done bool) error { + st, err := s.loadState() + if err != nil { + return err + } + st.BootstrapComplete = done + return s.saveState(st) +} + +// LastWarning returns the persistent scanner warning shown by sync status. +func (s *Service) LastWarning() (string, error) { + st, err := s.loadState() + if err != nil { + return "", err + } + return st.LastWarning, nil +} + +// SetLastWarning persists an unresolved scanner condition. An empty string +// clears the warning once a later complete scan no longer reports it. +func (s *Service) SetLastWarning(message string) error { + st, err := s.loadState() + if err != nil { + return err + } + st.LastWarning = message + return s.saveState(st) +} + +// RemoteVaultID returns the optional target vault chosen while pairing a new +// local vault for restore. Empty means this vault's own durable ID was used. +func (s *Service) RemoteVaultID() (string, error) { + st, err := s.loadState() + if err != nil { + return "", err + } + return st.RemoteVaultID, nil +} + +func (s *Service) SetRemoteVaultID(vaultID string) error { + st, err := s.loadState() + if err != nil { + return err + } + st.RemoteVaultID = vaultID + return s.saveState(st) +} + // GetDeviceID returns the device ID used by this service. func (s *Service) GetDeviceID() string { return s.deviceID @@ -288,7 +422,7 @@ func (s *Service) saveOps(ops []Op) error { if err != nil { return fmt.Errorf("marshal ops: %w", err) } - return os.WriteFile(s.opsPath(), data, 0o644) + return atomicWriteFile(s.opsPath(), data, 0o600) } func (s *Service) loadState() (*syncState, error) { @@ -311,5 +445,42 @@ func (s *Service) saveState(st *syncState) error { if err != nil { return fmt.Errorf("marshal state: %w", err) } - return os.WriteFile(s.statePath(), data, 0o644) + return atomicWriteFile(s.statePath(), data, 0o600) +} + +func atomicWriteFile(path string, data []byte, perm os.FileMode) error { + if err := os.MkdirAll(filepath.Dir(path), 0o750); err != nil { + return err + } + tmp, err := os.CreateTemp(filepath.Dir(path), ".verstak-sync-*") + if err != nil { + return err + } + tmpPath := tmp.Name() + cleanup := true + defer func() { + if cleanup { + _ = os.Remove(tmpPath) + } + }() + if err := tmp.Chmod(perm); err != nil { + _ = tmp.Close() + return err + } + if _, err := tmp.Write(data); err != nil { + _ = tmp.Close() + return err + } + if err := tmp.Sync(); err != nil { + _ = tmp.Close() + return err + } + if err := tmp.Close(); err != nil { + return err + } + if err := os.Rename(tmpPath, path); err != nil { + return err + } + cleanup = false + return nil } diff --git a/internal/core/sync/snapshot.go b/internal/core/sync/snapshot.go new file mode 100644 index 0000000..8946034 --- /dev/null +++ b/internal/core/sync/snapshot.go @@ -0,0 +1,789 @@ +package sync + +import ( + "crypto/sha256" + "encoding/base64" + "encoding/hex" + "encoding/json" + "fmt" + "io" + "io/fs" + "os" + "path/filepath" + "sort" + "strings" + "time" + "unicode" + "unicode/utf8" + + "github.com/google/uuid" + corefiles "github.com/verstak/verstak-desktop/internal/core/files" +) + +const snapshotVersion = 1 +const maxOperationFileBytes = corefiles.MaxBinaryReadBytes + +// Snapshot is the durable local view of the synchronizable part of a vault. +// Its entries are only files and folders whose latest state was successfully +// represented by an operation or intentionally accepted as an initial baseline. +type Snapshot struct { + Version int `json:"version"` + Entries map[string]SnapshotEntry `json:"entries"` + Workspaces map[string]WorkspaceSnapshot `json:"workspaces,omitempty"` + TrashedWorkspaces map[string]WorkspaceSnapshot `json:"trashedWorkspaces,omitempty"` + WorkspacesInitialized bool `json:"workspacesInitialized,omitempty"` + Unresolved map[string]string `json:"unresolved,omitempty"` +} + +// SnapshotEntry stores only stable filesystem facts needed for reconciliation. +type SnapshotEntry struct { + Path string `json:"path"` + Type string `json:"type"` + Size int64 `json:"size"` + ModifiedAt string `json:"modifiedAt"` + Hash string `json:"hash,omitempty"` +} + +// WorkspaceSnapshot keeps the core-owned identity and creation metadata of a +// top-level workspace. The marker itself remains excluded from normal file +// sync and is never exposed through the Files API. +type WorkspaceSnapshot struct { + WorkspaceID string `json:"workspaceId"` + Path string `json:"path"` + Metadata json.RawMessage `json:"metadata,omitempty"` + Entries map[string]SnapshotEntry `json:"entries,omitempty"` +} + +type scanJournal struct { + Snapshot Snapshot `json:"snapshot"` + Ops []Op `json:"ops"` +} + +type scannedVault struct { + Entries map[string]SnapshotEntry + Workspaces map[string]WorkspaceSnapshot + Unresolved map[string]string +} + +// LoadSnapshot returns the current durable scanner snapshot. A missing +// snapshot is represented by an empty snapshot, which is useful to callers +// that only need to inspect it. +func (s *Service) LoadSnapshot() (Snapshot, error) { + snapshot, exists, err := s.loadSnapshot() + if err != nil { + return Snapshot{}, err + } + if !exists { + return newSnapshot(), nil + } + return snapshot, nil +} + +// ScanAndRecord scans the whole synchronizable vault and records exactly the +// detected local changes. On its first run it writes a baseline and deliberately +// produces no operations; bootstrap decides what can safely be published. +func (s *Service) ScanAndRecord() ([]string, error) { + if err := s.recoverScanJournal(); err != nil { + return nil, err + } + previous, exists, err := s.loadSnapshot() + if err != nil { + return nil, err + } + current, warnings, err := scanVault(s.vaultRoot, previous) + if err != nil { + return nil, err + } + next := snapshotFromScan(current, previous, exists) + if !exists { + if err := s.saveSnapshot(next); err != nil { + return nil, err + } + return warnings, nil + } + + ops, next, err := diffSnapshots(previous, next, s.deviceID, s.vaultRoot) + if err != nil { + return nil, err + } + if len(ops) == 0 { + if err := s.saveSnapshot(next); err != nil { + return nil, err + } + return warnings, nil + } + if err := s.commitScanTransaction(next, ops); err != nil { + return nil, err + } + return warnings, nil +} + +// RecordBootstrapOps records creates for a pre-pull local snapshot. It is used +// only after a successful initial pull, so an empty new vault never turns into +// remote delete operations. Callers may pass the snapshot captured before the +// pull; remote-only entries added during reconciliation are therefore not +// reflected back to the server as local creates. +func (s *Service) RecordBootstrapOps(initial Snapshot) error { + if err := s.recoverScanJournal(); err != nil { + return err + } + current, exists, err := s.loadSnapshot() + if err != nil { + return err + } + if !exists { + return fmt.Errorf("bootstrap requires an initial snapshot") + } + ops, _, err := diffSnapshots(newSnapshot(), initial, s.deviceID, s.vaultRoot) + if err != nil { + return err + } + existing, err := s.GetUnpushedOps() + if err != nil { + return err + } + existingCreates := make(map[string]bool, len(existing)) + for _, op := range existing { + if op.OpType == OpCreate { + existingCreates[op.EntityType+"\x00"+op.EntityID] = true + } + } + filtered := ops[:0] + for _, op := range ops { + if op.OpType == OpCreate && existingCreates[op.EntityType+"\x00"+op.EntityID] { + continue + } + filtered = append(filtered, op) + } + ops = filtered + if len(ops) == 0 { + return nil + } + return s.commitScanTransaction(current, ops) +} + +// RebaseSnapshot accepts filesystem changes that were applied from a remote +// operation without producing any outgoing operation for them. +func (s *Service) RebaseSnapshot() ([]string, error) { + if err := s.recoverScanJournal(); err != nil { + return nil, err + } + previous, exists, err := s.loadSnapshot() + if err != nil { + return nil, err + } + current, warnings, err := scanVault(s.vaultRoot, previous) + if err != nil { + return nil, err + } + next := snapshotFromScan(current, previous, exists) + acceptWorkspaceLifecycle(previous, &next) + if err := s.saveSnapshot(next); err != nil { + return nil, err + } + return warnings, nil +} + +func newSnapshot() Snapshot { + return Snapshot{ + Version: snapshotVersion, + Entries: make(map[string]SnapshotEntry), + Workspaces: make(map[string]WorkspaceSnapshot), + TrashedWorkspaces: make(map[string]WorkspaceSnapshot), + WorkspacesInitialized: true, + Unresolved: make(map[string]string), + } +} + +func (s *Service) loadSnapshot() (Snapshot, bool, error) { + data, err := os.ReadFile(s.snapshotPath()) + if err != nil { + if os.IsNotExist(err) { + return Snapshot{}, false, nil + } + return Snapshot{}, false, fmt.Errorf("read snapshot: %w", err) + } + var snapshot Snapshot + if err := json.Unmarshal(data, &snapshot); err != nil { + return Snapshot{}, false, fmt.Errorf("parse snapshot: %w", err) + } + if snapshot.Version != snapshotVersion { + return Snapshot{}, false, fmt.Errorf("unsupported snapshot version: %d", snapshot.Version) + } + if snapshot.Entries == nil { + snapshot.Entries = make(map[string]SnapshotEntry) + } + if snapshot.Workspaces == nil { + snapshot.Workspaces = make(map[string]WorkspaceSnapshot) + } + if snapshot.TrashedWorkspaces == nil { + snapshot.TrashedWorkspaces = make(map[string]WorkspaceSnapshot) + } + if snapshot.Unresolved == nil { + snapshot.Unresolved = make(map[string]string) + } + return snapshot, true, nil +} + +func (s *Service) saveSnapshot(snapshot Snapshot) error { + data, err := json.MarshalIndent(snapshot, "", " ") + if err != nil { + return fmt.Errorf("marshal snapshot: %w", err) + } + return atomicWriteFile(s.snapshotPath(), data, 0o600) +} + +func (s *Service) loadScanJournal() (scanJournal, bool, error) { + data, err := os.ReadFile(s.scanJournalPath()) + if err != nil { + if os.IsNotExist(err) { + return scanJournal{}, false, nil + } + return scanJournal{}, false, fmt.Errorf("read scan journal: %w", err) + } + var journal scanJournal + if err := json.Unmarshal(data, &journal); err != nil { + return scanJournal{}, false, fmt.Errorf("parse scan journal: %w", err) + } + return journal, true, nil +} + +func (s *Service) saveScanJournal(journal scanJournal) error { + data, err := json.MarshalIndent(journal, "", " ") + if err != nil { + return fmt.Errorf("marshal scan journal: %w", err) + } + return atomicWriteFile(s.scanJournalPath(), data, 0o600) +} + +func (s *Service) recoverScanJournal() error { + journal, exists, err := s.loadScanJournal() + if err != nil || !exists { + return err + } + if err := s.recordOps(journal.Ops); err != nil { + return err + } + if err := s.saveSnapshot(journal.Snapshot); err != nil { + return err + } + if err := os.Remove(s.scanJournalPath()); err != nil && !os.IsNotExist(err) { + return err + } + return nil +} + +func (s *Service) commitScanTransaction(snapshot Snapshot, ops []Op) error { + journal := scanJournal{Snapshot: snapshot, Ops: ops} + if err := s.saveScanJournal(journal); err != nil { + return err + } + if err := s.recordOps(ops); err != nil { + return err + } + if err := s.saveSnapshot(snapshot); err != nil { + return err + } + if err := os.Remove(s.scanJournalPath()); err != nil && !os.IsNotExist(err) { + return err + } + return nil +} + +func scanVault(root string, previous Snapshot) (scannedVault, []string, error) { + result := scannedVault{ + Entries: make(map[string]SnapshotEntry), + Workspaces: make(map[string]WorkspaceSnapshot), + Unresolved: make(map[string]string), + } + var warnings []string + err := filepath.WalkDir(root, func(path string, entry fs.DirEntry, walkErr error) error { + if walkErr != nil { + return walkErr + } + if path == root { + return nil + } + rel, err := filepath.Rel(root, path) + if err != nil { + return err + } + rel = filepath.ToSlash(rel) + if excludedFromSync(rel) { + if entry.IsDir() { + return filepath.SkipDir + } + return nil + } + if entry.Type()&os.ModeSymlink != 0 { + if entry.IsDir() { + return filepath.SkipDir + } + return nil + } + info, err := entry.Info() + if err != nil { + return err + } + if info.IsDir() { + result.Entries[rel] = SnapshotEntry{ + Path: rel, + Type: EntityFolder, + ModifiedAt: info.ModTime().UTC().Format(time.RFC3339Nano), + } + return nil + } + if !info.Mode().IsRegular() { + return nil + } + if info.Size() > maxOperationFileBytes { + message := fmt.Sprintf("file-too-large: %s (%d bytes exceeds %d bytes)", rel, info.Size(), maxOperationFileBytes) + result.Unresolved[rel] = message + warnings = append(warnings, message) + return nil + } + hash, err := sha256File(path) + if err != nil { + return fmt.Errorf("hash %s: %w", rel, err) + } + result.Entries[rel] = SnapshotEntry{ + Path: rel, + Type: EntityFile, + Size: info.Size(), + ModifiedAt: info.ModTime().UTC().Format(time.RFC3339Nano), + Hash: hash, + } + return nil + }) + if err != nil { + return scannedVault{}, nil, err + } + workspaces, workspaceWarnings, err := scanWorkspaceSnapshots(root, previous.Workspaces) + if err != nil { + return scannedVault{}, nil, err + } + for workspaceID, workspace := range workspaces { + result.Workspaces[workspaceID] = workspace + } + for _, warning := range workspaceWarnings { + warnings = append(warnings, warning) + if strings.HasPrefix(warning, "duplicate-workspace-id: ") { + path := strings.TrimPrefix(warning, "duplicate-workspace-id: ") + result.Unresolved[path] = warning + removeEntriesUnder(result.Entries, path) + } + } + sort.Strings(warnings) + return result, warnings, nil +} + +func scanWorkspaceSnapshots(root string, preferred map[string]WorkspaceSnapshot) (map[string]WorkspaceSnapshot, []string, error) { + entries, err := os.ReadDir(root) + if err != nil { + return nil, nil, err + } + candidates := make(map[string][]WorkspaceSnapshot) + var warnings []string + for _, entry := range entries { + if !entry.IsDir() || entry.Type()&os.ModeSymlink != 0 || strings.EqualFold(entry.Name(), ".verstak") { + continue + } + path := filepath.Join(root, entry.Name(), ".verstak", "workspace.json") + data, err := os.ReadFile(path) + if err != nil { + if os.IsNotExist(err) { + continue + } + return nil, nil, fmt.Errorf("read workspace marker %s: %w", entry.Name(), err) + } + var marker struct { + WorkspaceID string `json:"workspaceId"` + } + if err := json.Unmarshal(data, &marker); err != nil { + warnings = append(warnings, fmt.Sprintf("invalid-workspace-id: %s", entry.Name())) + continue + } + if _, err := uuid.Parse(marker.WorkspaceID); err != nil { + warnings = append(warnings, fmt.Sprintf("invalid-workspace-id: %s", entry.Name())) + continue + } + metadata, err := readWorkspaceMetadataSnapshot(root, entry.Name()) + if err != nil { + return nil, nil, err + } + candidates[marker.WorkspaceID] = append(candidates[marker.WorkspaceID], WorkspaceSnapshot{ + WorkspaceID: marker.WorkspaceID, + Path: entry.Name(), + Metadata: metadata, + }) + } + workspaces := make(map[string]WorkspaceSnapshot, len(candidates)) + for workspaceID, choices := range candidates { + sort.Slice(choices, func(i, j int) bool { return choices[i].Path < choices[j].Path }) + selected := choices[0] + if old, ok := preferred[workspaceID]; ok { + for _, choice := range choices { + if choice.Path == old.Path { + selected = choice + break + } + } + } + workspaces[workspaceID] = selected + for _, choice := range choices { + if choice.Path != selected.Path { + warnings = append(warnings, "duplicate-workspace-id: "+choice.Path) + } + } + } + return workspaces, warnings, nil +} + +func readWorkspaceMetadataSnapshot(root, name string) (json.RawMessage, error) { + path := filepath.Join(root, ".verstak", "workspaces", name, "metadata.json") + data, err := os.ReadFile(path) + if err != nil { + if os.IsNotExist(err) { + return nil, nil + } + return nil, fmt.Errorf("read workspace metadata %s: %w", name, err) + } + if !json.Valid(data) { + return nil, fmt.Errorf("invalid workspace metadata: %s", name) + } + return json.RawMessage(append([]byte(nil), data...)), nil +} + +func excludedFromSync(rel string) bool { + rel = filepath.ToSlash(rel) + for _, segment := range strings.Split(rel, "/") { + if strings.EqualFold(segment, ".verstak") { + return true + } + } + base := filepath.Base(rel) + return strings.HasPrefix(base, ".verstak-write-") || strings.HasSuffix(base, ".tmp") || strings.HasSuffix(base, ".swp") || strings.HasSuffix(base, "~") +} + +func sha256File(path string) (string, error) { + file, err := os.Open(path) + if err != nil { + return "", err + } + defer file.Close() + hash := sha256.New() + if _, err := io.Copy(hash, file); err != nil { + return "", err + } + return hex.EncodeToString(hash.Sum(nil)), nil +} + +func snapshotFromScan(current scannedVault, previous Snapshot, previousExists bool) Snapshot { + next := newSnapshot() + for path, entry := range current.Entries { + next.Entries[path] = entry + } + for workspaceID, workspace := range current.Workspaces { + next.Workspaces[workspaceID] = workspace + } + for workspaceID, workspace := range previous.TrashedWorkspaces { + next.TrashedWorkspaces[workspaceID] = workspace + } + for path, message := range current.Unresolved { + next.Unresolved[path] = message + copyEntriesUnder(next.Entries, previous.Entries, path) + } + if !previousExists { + return next + } + for path, message := range previous.Unresolved { + if _, supported := current.Entries[path]; supported { + continue + } + if _, stillUnsupported := current.Unresolved[path]; stillUnsupported { + continue + } + next.Unresolved[path] = "unresolved sync file disappeared before it could be synchronized: " + message + copyEntriesUnder(next.Entries, previous.Entries, path) + } + return next +} + +func diffSnapshots(previous, next Snapshot, deviceID, vaultRoot string) ([]Op, Snapshot, error) { + workspaceOps, err := diffWorkspaceSnapshots(&previous, &next, deviceID) + if err != nil { + return nil, next, err + } + var createsOrUpdates []Op + var deletes []Op + for path, entry := range next.Entries { + old, existed := previous.Entries[path] + if existed && entriesEqual(old, entry) { + continue + } + if unresolvedPath(next.Unresolved, path) { + continue + } + opType := OpCreate + if existed { + opType = OpUpdate + } + payload, err := payloadForEntry(vaultRoot, entry) + if err != nil { + return nil, next, err + } + createsOrUpdates = append(createsOrUpdates, newSnapshotOp(deviceID, entry.Type, path, opType, payload)) + } + for path, old := range previous.Entries { + if _, exists := next.Entries[path]; exists { + continue + } + if unresolvedPath(previous.Unresolved, path) { + continue + } + deletes = append(deletes, newSnapshotOp(deviceID, old.Type, path, OpDelete, map[string]string{"path": path})) + } + sort.Slice(createsOrUpdates, func(i, j int) bool { + left, right := createsOrUpdates[i], createsOrUpdates[j] + leftDepth, rightDepth := pathDepth(left.EntityID), pathDepth(right.EntityID) + if leftDepth != rightDepth { + return leftDepth < rightDepth + } + if left.EntityType != right.EntityType { + return left.EntityType == EntityFolder + } + return left.EntityID < right.EntityID + }) + sort.Slice(deletes, func(i, j int) bool { + left, right := deletes[i], deletes[j] + leftDepth, rightDepth := pathDepth(left.EntityID), pathDepth(right.EntityID) + if leftDepth != rightDepth { + return leftDepth > rightDepth + } + if left.EntityType != right.EntityType { + return left.EntityType == EntityFile + } + return left.EntityID < right.EntityID + }) + return append(workspaceOps, append(createsOrUpdates, deletes...)...), next, nil +} + +func diffWorkspaceSnapshots(previous, next *Snapshot, deviceID string) ([]Op, error) { + var ops []Op + if !previous.WorkspacesInitialized { + return ops, nil + } + for workspaceID, oldWorkspace := range previous.Workspaces { + currentWorkspace, active := next.Workspaces[workspaceID] + if active { + if currentWorkspace.Path != oldWorkspace.Path { + ops = append(ops, newWorkspaceSnapshotOp(deviceID, workspaceID, OpRename, currentWorkspace, oldWorkspace.Path)) + remapEntriesPrefix(previous.Entries, oldWorkspace.Path, currentWorkspace.Path) + } + continue + } + ops = append(ops, newWorkspaceSnapshotOp(deviceID, workspaceID, OpTrash, oldWorkspace, "")) + oldWorkspace.Entries = entriesUnder(previous.Entries, oldWorkspace.Path) + removeEntriesUnder(previous.Entries, oldWorkspace.Path) + next.TrashedWorkspaces[workspaceID] = oldWorkspace + } + for workspaceID, currentWorkspace := range next.Workspaces { + if _, alreadyActive := previous.Workspaces[workspaceID]; alreadyActive { + continue + } + if trashedWorkspace, wasTrashed := previous.TrashedWorkspaces[workspaceID]; wasTrashed { + ops = append(ops, newWorkspaceSnapshotOp(deviceID, workspaceID, OpRestore, currentWorkspace, "")) + copyRemappedEntries(previous.Entries, trashedWorkspace.Entries, trashedWorkspace.Path, currentWorkspace.Path) + delete(next.TrashedWorkspaces, workspaceID) + continue + } + ops = append(ops, newWorkspaceSnapshotOp(deviceID, workspaceID, OpCreate, currentWorkspace, "")) + delete(next.Entries, currentWorkspace.Path) + } + sort.Slice(ops, func(i, j int) bool { + if ops[i].OpType != ops[j].OpType { + return workspaceOpOrder(ops[i].OpType) < workspaceOpOrder(ops[j].OpType) + } + return ops[i].EntityID < ops[j].EntityID + }) + return ops, nil +} + +func workspaceOpOrder(opType string) int { + switch opType { + case OpCreate: + return 0 + case OpRename: + return 1 + case OpRestore: + return 2 + case OpTrash: + return 3 + default: + return 4 + } +} + +type snapshotWorkspacePayload struct { + WorkspaceID string `json:"workspaceId"` + Path string `json:"path"` + PreviousPath string `json:"previousPath,omitempty"` + Name string `json:"name"` + Metadata json.RawMessage `json:"metadata,omitempty"` +} + +func newWorkspaceSnapshotOp(deviceID, workspaceID, opType string, workspace WorkspaceSnapshot, previousPath string) Op { + payload, _ := json.Marshal(snapshotWorkspacePayload{ + WorkspaceID: workspaceID, + Path: workspace.Path, + PreviousPath: previousPath, + Name: workspace.Path, + Metadata: workspace.Metadata, + }) + return newSnapshotOp(deviceID, EntityWorkspace, workspaceID, opType, json.RawMessage(payload)) +} + +func acceptWorkspaceLifecycle(previous Snapshot, next *Snapshot) { + for workspaceID, oldWorkspace := range previous.Workspaces { + if _, stillActive := next.Workspaces[workspaceID]; !stillActive { + oldWorkspace.Entries = entriesUnder(previous.Entries, oldWorkspace.Path) + next.TrashedWorkspaces[workspaceID] = oldWorkspace + } + } + for workspaceID := range next.Workspaces { + delete(next.TrashedWorkspaces, workspaceID) + } +} + +func unresolvedPath(unresolved map[string]string, path string) bool { + for unresolvedPath := range unresolved { + if path == unresolvedPath || strings.HasPrefix(path, unresolvedPath+"/") { + return true + } + } + return false +} + +func copyEntriesUnder(destination, source map[string]SnapshotEntry, root string) { + for path, entry := range source { + if path == root || strings.HasPrefix(path, root+"/") { + destination[path] = entry + } + } +} + +func removeEntriesUnder(entries map[string]SnapshotEntry, root string) { + for path := range entries { + if path == root || strings.HasPrefix(path, root+"/") { + delete(entries, path) + } + } +} + +func remapEntriesPrefix(entries map[string]SnapshotEntry, oldPrefix, newPrefix string) { + type remappedEntry struct { + oldPath string + entry SnapshotEntry + } + var remapped []remappedEntry + for path, entry := range entries { + if path != oldPrefix && !strings.HasPrefix(path, oldPrefix+"/") { + continue + } + suffix := strings.TrimPrefix(path, oldPrefix) + entry.Path = newPrefix + suffix + remapped = append(remapped, remappedEntry{oldPath: path, entry: entry}) + } + for _, item := range remapped { + delete(entries, item.oldPath) + entries[item.entry.Path] = item.entry + } +} + +func entriesUnder(entries map[string]SnapshotEntry, root string) map[string]SnapshotEntry { + result := make(map[string]SnapshotEntry) + copyEntriesUnder(result, entries, root) + return result +} + +func copyRemappedEntries(destination, source map[string]SnapshotEntry, oldPrefix, newPrefix string) { + for path, entry := range source { + if path != oldPrefix && !strings.HasPrefix(path, oldPrefix+"/") { + continue + } + suffix := strings.TrimPrefix(path, oldPrefix) + entry.Path = newPrefix + suffix + destination[entry.Path] = entry + } +} + +func entriesEqual(left, right SnapshotEntry) bool { + return left.Type == right.Type && left.Size == right.Size && left.Hash == right.Hash +} + +func payloadForEntry(vaultRoot string, entry SnapshotEntry) (map[string]string, error) { + payload := map[string]string{"path": entry.Path, "contentHash": entry.Hash} + if entry.Type == EntityFolder { + return payload, nil + } + data, err := os.ReadFile(filepath.Join(vaultRoot, filepath.FromSlash(entry.Path))) + if err != nil { + return nil, err + } + if int64(len(data)) > maxOperationFileBytes { + return nil, fmt.Errorf("file-too-large: %s", entry.Path) + } + if hash, err := sha256File(filepath.Join(vaultRoot, filepath.FromSlash(entry.Path))); err != nil { + return nil, err + } else if hash != entry.Hash { + return nil, fmt.Errorf("file changed during scan: %s", entry.Path) + } + return filePayload(entry.Path, data, entry.Hash), nil +} + +func newSnapshotOp(deviceID, entityType, entityID, opType string, payload interface{}) Op { + data, _ := json.Marshal(payload) + id := uuid.NewString() + return Op{ + ID: id, + OpID: id, + DeviceID: deviceID, + EntityType: entityType, + EntityID: entityID, + OpType: opType, + PayloadJSON: string(data), + CreatedAt: time.Now().UTC().Format(time.RFC3339Nano), + } +} + +func pathDepth(path string) int { + if path == "" { + return 0 + } + return strings.Count(path, "/") + 1 +} + +// filePayload encodes an already bounded file without changing its content +// representation. UTF-8 files use plain text up to the existing text limit; +// other supported files use the current bounded base64 transport. +func filePayload(path string, data []byte, hash string) map[string]string { + payload := map[string]string{"path": path, "contentHash": hash} + if int64(len(data)) <= corefiles.MaxTextFileBytes && isSyncText(data) { + payload["content"] = string(data) + return payload + } + payload["dataBase64"] = base64.StdEncoding.EncodeToString(data) + return payload +} + +func isSyncText(data []byte) bool { + if !utf8.Valid(data) { + return false + } + for _, r := range string(data) { + if unicode.IsControl(r) && r != '\n' && r != '\r' && r != '\t' { + return false + } + } + return true +} diff --git a/internal/core/sync/snapshot_test.go b/internal/core/sync/snapshot_test.go new file mode 100644 index 0000000..5cd16b0 --- /dev/null +++ b/internal/core/sync/snapshot_test.go @@ -0,0 +1,406 @@ +package sync + +import ( + "encoding/json" + "os" + "path/filepath" + "runtime" + "strings" + "testing" + + "github.com/google/uuid" +) + +func TestScanAndRecordTracksExternalWorkspaceLifecycleByIdentity(t *testing.T) { + root := t.TempDir() + workspaceID := uuid.NewString() + createSnapshotWorkspace(t, root, "Project", workspaceID) + if err := os.Mkdir(filepath.Join(root, "Project", "Files"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "Project", "Files", "note.txt"), []byte("one"), 0o644); err != nil { + t.Fatal(err) + } + service := NewService(root, "device-a") + if _, err := service.ScanAndRecord(); err != nil { + t.Fatalf("baseline: %v", err) + } + + if err := os.Rename(filepath.Join(root, "Project"), filepath.Join(root, "Renamed")); err != nil { + t.Fatal(err) + } + if _, err := service.ScanAndRecord(); err != nil { + t.Fatalf("scan rename: %v", err) + } + assertWorkspaceSnapshotOp(t, unpushedOps(t, service), 0, OpRename, workspaceID, "Renamed", "Project") + if _, err := service.ScanAndRecord(); err != nil { + t.Fatalf("unchanged renamed scan: %v", err) + } + if got := len(unpushedOps(t, service)); got != 1 { + t.Fatalf("unchanged rename produced %d operations, want 1", got) + } + + trashPath := filepath.Join(root, ".verstak", "trash", "workspaces", "external-trash", "Renamed") + if err := os.MkdirAll(filepath.Dir(trashPath), 0o755); err != nil { + t.Fatal(err) + } + if err := os.Rename(filepath.Join(root, "Renamed"), trashPath); err != nil { + t.Fatal(err) + } + if _, err := service.ScanAndRecord(); err != nil { + t.Fatalf("scan trash: %v", err) + } + assertWorkspaceSnapshotOp(t, unpushedOps(t, service), 1, OpTrash, workspaceID, "Renamed", "") + + if err := os.Rename(trashPath, filepath.Join(root, "Restored")); err != nil { + t.Fatal(err) + } + if _, err := service.ScanAndRecord(); err != nil { + t.Fatalf("scan restore: %v", err) + } + assertWorkspaceSnapshotOp(t, unpushedOps(t, service), 2, OpRestore, workspaceID, "Restored", "") + + createSnapshotWorkspace(t, root, "Copied", workspaceID) + warnings, err := service.ScanAndRecord() + if err != nil { + t.Fatalf("scan duplicate identity: %v", err) + } + if len(warnings) != 1 || !strings.Contains(warnings[0], "duplicate-workspace-id: Copied") { + t.Fatalf("duplicate identity warnings = %v", warnings) + } + if got := len(unpushedOps(t, service)); got != 3 { + t.Fatalf("duplicate identity created operations: %d, want 3", got) + } +} + +func createSnapshotWorkspace(t *testing.T, root, name, workspaceID string) { + t.Helper() + if err := os.MkdirAll(filepath.Join(root, name, ".verstak"), 0o755); err != nil { + t.Fatal(err) + } + marker, err := json.Marshal(map[string]string{"workspaceId": workspaceID}) + if err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, name, ".verstak", "workspace.json"), marker, 0o600); err != nil { + t.Fatal(err) + } + metadataPath := filepath.Join(root, ".verstak", "workspaces", name, "metadata.json") + if err := os.MkdirAll(filepath.Dir(metadataPath), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(metadataPath, []byte(`{"workspaceId":"`+workspaceID+`","workspaceName":"`+name+`"}`), 0o600); err != nil { + t.Fatal(err) + } +} + +func assertWorkspaceSnapshotOp(t *testing.T, ops []Op, index int, opType, workspaceID, path, previousPath string) { + t.Helper() + if len(ops) <= index { + t.Fatalf("operations = %#v, want index %d", ops, index) + } + op := ops[index] + if op.EntityType != EntityWorkspace || op.EntityID != workspaceID || op.OpType != opType { + t.Fatalf("workspace operation = %+v", op) + } + var payload snapshotWorkspacePayload + if err := json.Unmarshal([]byte(op.PayloadJSON), &payload); err != nil { + t.Fatalf("decode workspace payload: %v", err) + } + if payload.Path != path || payload.PreviousPath != previousPath || payload.WorkspaceID != workspaceID { + t.Fatalf("workspace payload = %+v", payload) + } +} + +func TestScanAndRecordBaselinesThenRecordsExternalChanges(t *testing.T) { + root := t.TempDir() + service := NewService(root, "device-a") + + warnings, err := service.ScanAndRecord() + if err != nil { + t.Fatalf("initial ScanAndRecord: %v", err) + } + if len(warnings) != 0 { + t.Fatalf("initial warnings = %v, want none", warnings) + } + assertUnpushedCount(t, service, 0) + + if err := os.Mkdir(filepath.Join(root, "Docs"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "Docs", "note.txt"), []byte("one"), 0o644); err != nil { + t.Fatal(err) + } + + warnings, err = service.ScanAndRecord() + if err != nil { + t.Fatalf("scan create: %v", err) + } + if len(warnings) != 0 { + t.Fatalf("create warnings = %v", warnings) + } + ops := unpushedOps(t, service) + if len(ops) != 2 { + t.Fatalf("create ops = %#v, want folder and file", ops) + } + if ops[0].EntityType != EntityFolder || ops[0].EntityID != "Docs" || ops[0].OpType != OpCreate { + t.Fatalf("folder op = %+v", ops[0]) + } + if ops[1].EntityType != EntityFile || ops[1].EntityID != "Docs/note.txt" || ops[1].OpType != OpCreate { + t.Fatalf("file op = %+v", ops[1]) + } + var createPayload map[string]interface{} + if err := json.Unmarshal([]byte(ops[1].PayloadJSON), &createPayload); err != nil { + t.Fatalf("decode file payload: %v", err) + } + if createPayload["content"] != "one" || createPayload["contentHash"] == "" { + t.Fatalf("file payload = %#v", createPayload) + } + + if _, err := service.ScanAndRecord(); err != nil { + t.Fatalf("unchanged scan: %v", err) + } + assertUnpushedCount(t, service, 2) + + if err := os.WriteFile(filepath.Join(root, "Docs", "note.txt"), []byte("two"), 0o644); err != nil { + t.Fatal(err) + } + if _, err := service.ScanAndRecord(); err != nil { + t.Fatalf("scan update: %v", err) + } + ops = unpushedOps(t, service) + if len(ops) != 3 || ops[2].EntityType != EntityFile || ops[2].OpType != OpUpdate { + t.Fatalf("update ops = %#v", ops) + } + + if err := os.Remove(filepath.Join(root, "Docs", "note.txt")); err != nil { + t.Fatal(err) + } + if err := os.Remove(filepath.Join(root, "Docs")); err != nil { + t.Fatal(err) + } + if _, err := service.ScanAndRecord(); err != nil { + t.Fatalf("scan delete: %v", err) + } + ops = unpushedOps(t, service) + if len(ops) != 5 { + t.Fatalf("delete ops = %#v", ops) + } + if ops[3].EntityType != EntityFile || ops[3].OpType != OpDelete || ops[4].EntityType != EntityFolder || ops[4].OpType != OpDelete { + t.Fatalf("delete ordering = %#v", ops[3:]) + } +} + +func TestScanAndRecordNeverTreatsInitialFilesAsDeletes(t *testing.T) { + root := t.TempDir() + if err := os.Mkdir(filepath.Join(root, "Existing"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "Existing", "before-sync.txt"), []byte("local"), 0o644); err != nil { + t.Fatal(err) + } + service := NewService(root, "device-a") + + if _, err := service.ScanAndRecord(); err != nil { + t.Fatalf("initial ScanAndRecord: %v", err) + } + assertUnpushedCount(t, service, 0) + snapshot, err := service.LoadSnapshot() + if err != nil { + t.Fatalf("LoadSnapshot: %v", err) + } + if snapshot.Entries["Existing/before-sync.txt"].Hash == "" { + t.Fatalf("snapshot = %#v, expected content hash", snapshot.Entries) + } +} + +func TestScanAndRecordFindsChangesMadeWhileDesktopWasClosed(t *testing.T) { + root := t.TempDir() + initial := NewService(root, "device-a") + if _, err := initial.ScanAndRecord(); err != nil { + t.Fatalf("initial baseline: %v", err) + } + if err := os.WriteFile(filepath.Join(root, "offline.txt"), []byte("created while closed"), 0o644); err != nil { + t.Fatal(err) + } + + restarted := NewService(root, "device-a") + if _, err := restarted.ScanAndRecord(); err != nil { + t.Fatalf("scan after offline create: %v", err) + } + ops := unpushedOps(t, restarted) + if len(ops) != 1 || ops[0].OpType != OpCreate || ops[0].EntityID != "offline.txt" { + t.Fatalf("offline create operations = %#v", ops) + } + + if err := os.WriteFile(filepath.Join(root, "offline.txt"), []byte("updated while closed"), 0o644); err != nil { + t.Fatal(err) + } + restarted = NewService(root, "device-a") + if _, err := restarted.ScanAndRecord(); err != nil { + t.Fatalf("scan after offline update: %v", err) + } + if err := os.Remove(filepath.Join(root, "offline.txt")); err != nil { + t.Fatal(err) + } + restarted = NewService(root, "device-a") + if _, err := restarted.ScanAndRecord(); err != nil { + t.Fatalf("scan after offline delete: %v", err) + } + ops = unpushedOps(t, restarted) + if len(ops) != 3 || ops[1].OpType != OpUpdate || ops[2].OpType != OpDelete { + t.Fatalf("offline lifecycle operations = %#v", ops) + } +} + +func TestRecordBootstrapOpsPublishesExistingFilesWithoutDeletes(t *testing.T) { + root := t.TempDir() + if err := os.Mkdir(filepath.Join(root, "Existing"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "Existing", "before-sync.txt"), []byte("local"), 0o644); err != nil { + t.Fatal(err) + } + service := NewService(root, "device-a") + if _, err := service.ScanAndRecord(); err != nil { + t.Fatal(err) + } + initial, err := service.LoadSnapshot() + if err != nil { + t.Fatal(err) + } + if err := service.RecordBootstrapOps(initial); err != nil { + t.Fatalf("RecordBootstrapOps: %v", err) + } + ops := unpushedOps(t, service) + if len(ops) != 2 { + t.Fatalf("bootstrap ops = %#v, want create folder and file", ops) + } + for _, op := range ops { + if op.OpType != OpCreate { + t.Fatalf("bootstrap op = %+v, initial scan must not create delete", op) + } + } + + empty := newSnapshot() + if err := service.RecordBootstrapOps(empty); err != nil { + t.Fatalf("empty bootstrap: %v", err) + } + if got := len(unpushedOps(t, service)); got != 2 { + t.Fatalf("empty bootstrap added operations = %d, want 2", got) + } +} + +func TestSyncBootstrapAndWarningStateSurviveRestart(t *testing.T) { + root := t.TempDir() + service := NewService(root, "device-a") + if err := service.SetBootstrapComplete(true); err != nil { + t.Fatalf("SetBootstrapComplete: %v", err) + } + if err := service.SetLastWarning("file-too-large: archive.bin"); err != nil { + t.Fatalf("SetLastWarning: %v", err) + } + + restarted := NewService(root, "") + bootstrapped, err := restarted.BootstrapComplete() + if err != nil { + t.Fatalf("BootstrapComplete: %v", err) + } + if !bootstrapped { + t.Fatal("bootstrap state was lost after restart") + } + warning, err := restarted.LastWarning() + if err != nil { + t.Fatalf("LastWarning: %v", err) + } + if warning != "file-too-large: archive.bin" { + t.Fatalf("warning = %q", warning) + } +} + +func TestScanAndRecordSkipsReservedTemporaryAndSymlinkPaths(t *testing.T) { + root := t.TempDir() + if err := os.MkdirAll(filepath.Join(root, ".verstak", "sync"), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, ".verstak", "sync", "state.json"), []byte("{}"), 0o600); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, ".verstak-write-local"), []byte("temporary"), 0o600); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "draft.tmp"), []byte("temporary"), 0o600); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(filepath.Join(root, "normal.txt"), []byte("normal"), 0o600); err != nil { + t.Fatal(err) + } + if runtime.GOOS != "windows" { + if err := os.Symlink(filepath.Join(root, "normal.txt"), filepath.Join(root, "normal-link.txt")); err != nil { + t.Skipf("symlink unavailable: %v", err) + } + } + + service := NewService(root, "device-a") + if _, err := service.ScanAndRecord(); err != nil { + t.Fatalf("baseline: %v", err) + } + snapshot, err := service.LoadSnapshot() + if err != nil { + t.Fatal(err) + } + for _, path := range []string{".verstak/sync/state.json", ".verstak-write-local", "draft.tmp", "normal-link.txt"} { + if _, ok := snapshot.Entries[path]; ok { + t.Fatalf("reserved path %q was included in snapshot %#v", path, snapshot.Entries) + } + } + if _, ok := snapshot.Entries["normal.txt"]; !ok { + t.Fatalf("normal file missing from snapshot %#v", snapshot.Entries) + } +} + +func TestScanAndRecordKeepsUnsupportedFileUnresolved(t *testing.T) { + root := t.TempDir() + path := filepath.Join(root, "too-large.bin") + if err := os.WriteFile(path, make([]byte, maxOperationFileBytes+1), 0o600); err != nil { + t.Fatal(err) + } + service := NewService(root, "device-a") + + warnings, err := service.ScanAndRecord() + if err != nil { + t.Fatalf("scan unsupported file: %v", err) + } + if len(warnings) != 1 || !strings.Contains(warnings[0], "too-large.bin") || !strings.Contains(warnings[0], "file-too-large") { + t.Fatalf("warnings = %v", warnings) + } + assertUnpushedCount(t, service, 0) + snapshot, err := service.LoadSnapshot() + if err != nil { + t.Fatal(err) + } + if _, ok := snapshot.Entries["too-large.bin"]; ok { + t.Fatalf("unsupported file was marked synchronized: %#v", snapshot.Entries) + } + + warnings, err = service.ScanAndRecord() + if err != nil || len(warnings) != 1 { + t.Fatalf("second scan warnings=%v err=%v, unresolved file must remain visible", warnings, err) + } +} + +func unpushedOps(t *testing.T, service *Service) []Op { + t.Helper() + ops, err := service.GetUnpushedOps() + if err != nil { + t.Fatalf("GetUnpushedOps: %v", err) + } + return ops +} + +func assertUnpushedCount(t *testing.T, service *Service, want int) { + t.Helper() + if got := len(unpushedOps(t, service)); got != want { + t.Fatalf("unpushed ops = %d, want %d", got, want) + } +} diff --git a/internal/core/workspace/manager.go b/internal/core/workspace/manager.go index 77e1a24..c25e25c 100644 --- a/internal/core/workspace/manager.go +++ b/internal/core/workspace/manager.go @@ -470,25 +470,35 @@ func (m *Manager) CreateWorkspace(name, templateID string) (Workspace, error) { func writeWorkspaceIdentity(workspacePath string) (string, error) { workspaceID := uuid.NewString() - data, err := json.Marshal(workspaceIdentityMarker{WorkspaceID: workspaceID}) - if err != nil { - return "", err - } - markerPath := filepath.Join(workspacePath, filepath.FromSlash(workspaceIdentityRelativePath)) - if err := os.MkdirAll(filepath.Dir(markerPath), 0o755); err != nil { - return "", err - } - tmpPath := markerPath + ".tmp" - if err := os.WriteFile(tmpPath, data, 0o644); err != nil { - return "", err - } - if err := os.Rename(tmpPath, markerPath); err != nil { - _ = os.Remove(tmpPath) + if err := writeWorkspaceIdentityValue(workspacePath, workspaceID); err != nil { return "", err } return workspaceID, nil } +func writeWorkspaceIdentityValue(workspacePath, workspaceID string) error { + if _, err := uuid.Parse(workspaceID); err != nil { + return fmt.Errorf("invalid workspace identity: %w", err) + } + data, err := json.Marshal(workspaceIdentityMarker{WorkspaceID: workspaceID}) + if err != nil { + return err + } + markerPath := filepath.Join(workspacePath, filepath.FromSlash(workspaceIdentityRelativePath)) + if err := os.MkdirAll(filepath.Dir(markerPath), 0o755); err != nil { + return err + } + tmpPath := markerPath + ".tmp" + if err := os.WriteFile(tmpPath, data, 0o644); err != nil { + return err + } + if err := os.Rename(tmpPath, markerPath); err != nil { + _ = os.Remove(tmpPath) + return err + } + return nil +} + func ensureWorkspaceIdentity(workspacePath string) (string, error) { workspaceID, err := readWorkspaceIdentity(workspacePath) if os.IsNotExist(err) { @@ -739,6 +749,213 @@ func (m *Manager) RestoreWorkspaceTrash(trashID, targetName string) (Workspace, return Workspace{ID: workspaceID, Name: targetName, RootPath: targetName}, nil } +// CreateWorkspaceFromSync creates a workspace from the core-only sync contract. +// It deliberately does not apply a live template: child folders and files are +// represented by normal file operations, while the captured metadata preserves +// the historical template snapshot that matters for restoration. +func (m *Manager) CreateWorkspaceFromSync(name, workspaceID string, meta Metadata) (Workspace, error) { + name = strings.TrimSpace(name) + if err := validateWorkspaceName(name); err != nil { + return Workspace{}, err + } + if _, err := uuid.Parse(workspaceID); err != nil { + return Workspace{}, fmt.Errorf("invalid workspace identity: %w", err) + } + if existing, found, err := m.findActiveWorkspaceByID(workspaceID); err != nil { + return Workspace{}, err + } else if found { + if existing.Name == name { + return existing, nil + } + return Workspace{}, fmt.Errorf("conflict: workspace identity %s already belongs to %s", workspaceID, existing.Name) + } + + full := filepath.Join(m.vaultDir, name) + if _, err := os.Lstat(full); err == nil { + return Workspace{}, fmt.Errorf("conflict: %s", name) + } else if !os.IsNotExist(err) { + return Workspace{}, err + } + if err := os.Mkdir(full, 0o755); err != nil { + return Workspace{}, err + } + created := true + defer func() { + if created { + _ = os.RemoveAll(full) + } + }() + if err := writeWorkspaceIdentityValue(full, workspaceID); err != nil { + return Workspace{}, err + } + meta.WorkspaceID = workspaceID + meta.WorkspaceName = name + if meta.Features == nil { + meta.Features = map[string]bool{"files": true} + } + if meta.Folders == nil { + meta.Folders = defaultFolders() + } + if meta.UpdatedAt == "" { + meta.UpdatedAt = time.Now().UTC().Format(time.RFC3339Nano) + } + for _, folder := range meta.Folders { + if strings.TrimSpace(folder) == "" { + continue + } + if err := validateWorkspaceFolderPath(folder); err != nil { + return Workspace{}, err + } + if err := os.MkdirAll(filepath.Join(full, folder), 0o755); err != nil { + return Workspace{}, err + } + } + if err := m.writeMetadata(name, meta); err != nil { + return Workspace{}, err + } + created = false + return Workspace{ID: workspaceID, Name: name, RootPath: name}, nil +} + +func validateWorkspaceFolderPath(folder string) error { + folder = strings.TrimSpace(folder) + if folder == "" || filepath.IsAbs(folder) || strings.Contains(folder, `\`) { + return fmt.Errorf("invalid workspace folder path") + } + cleaned := filepath.ToSlash(filepath.Clean(filepath.FromSlash(folder))) + if cleaned == "." || cleaned == ".." || strings.HasPrefix(cleaned, "../") { + return fmt.Errorf("invalid workspace folder path") + } + for _, segment := range strings.Split(cleaned, "/") { + if strings.EqualFold(segment, ".verstak") { + return fmt.Errorf("invalid workspace folder path") + } + } + return nil +} + +// RenameWorkspaceFromSync applies an idempotent rename only to the durable +// identity named by the operation. A different local path is a visible conflict. +func (m *Manager) RenameWorkspaceFromSync(workspaceID, oldName, newName string) error { + newName = strings.TrimSpace(newName) + if err := validateWorkspaceName(newName); err != nil { + return err + } + workspace, found, err := m.findActiveWorkspaceByID(workspaceID) + if err != nil { + return err + } + if !found { + return fmt.Errorf("not-found: workspace identity %s", workspaceID) + } + if workspace.Name == newName { + return nil + } + if strings.TrimSpace(oldName) != "" && workspace.Name != oldName { + return fmt.Errorf("conflict: workspace identity %s is at %s, expected %s", workspaceID, workspace.Name, oldName) + } + return m.RenameWorkspace(workspace.Name, newName) +} + +// TrashWorkspaceFromSync moves the identified active workspace into this +// device's local trash. Local trash IDs are intentionally not synchronized. +func (m *Manager) TrashWorkspaceFromSync(workspaceID, name string) (TrashResult, error) { + workspace, found, err := m.findActiveWorkspaceByID(workspaceID) + if err != nil { + return TrashResult{}, err + } + if !found { + if _, trashed, err := m.findTrashedWorkspaceByID(workspaceID); err != nil { + return TrashResult{}, err + } else if trashed { + return TrashResult{WorkspaceID: workspaceID}, nil + } + return TrashResult{}, fmt.Errorf("not-found: workspace identity %s", workspaceID) + } + if strings.TrimSpace(name) != "" && workspace.Name != name { + return TrashResult{}, fmt.Errorf("conflict: workspace identity %s is at %s, expected %s", workspaceID, workspace.Name, name) + } + return m.TrashWorkspace(workspace.Name) +} + +// RestoreWorkspaceFromSync restores a workspace by durable identity. It is +// idempotent at the requested target and rejects a workspace of another ID. +func (m *Manager) RestoreWorkspaceFromSync(workspaceID, targetName string) (Workspace, error) { + targetName = strings.TrimSpace(targetName) + if err := validateWorkspaceName(targetName); err != nil { + return Workspace{}, err + } + if active, found, err := m.findActiveWorkspaceByID(workspaceID); err != nil { + return Workspace{}, err + } else if found { + if active.Name == targetName { + return active, nil + } + return Workspace{}, fmt.Errorf("conflict: workspace identity %s is already active at %s", workspaceID, active.Name) + } + trashID, found, err := m.findTrashedWorkspaceByID(workspaceID) + if err != nil { + return Workspace{}, err + } + if !found { + return Workspace{}, fmt.Errorf("not-found: trashed workspace identity %s", workspaceID) + } + return m.RestoreWorkspaceTrash(trashID, targetName) +} + +func (m *Manager) findActiveWorkspaceByID(workspaceID string) (Workspace, bool, error) { + if _, err := uuid.Parse(workspaceID); err != nil { + return Workspace{}, false, fmt.Errorf("invalid workspace identity: %w", err) + } + workspaces, err := m.ListWorkspaces() + if err != nil { + return Workspace{}, false, err + } + var match Workspace + for _, workspace := range workspaces { + if workspace.ID != workspaceID { + continue + } + if match.ID != "" { + return Workspace{}, false, fmt.Errorf("conflict: duplicated workspace identity %s", workspaceID) + } + match = workspace + } + return match, match.ID != "", nil +} + +func (m *Manager) findTrashedWorkspaceByID(workspaceID string) (string, bool, error) { + if _, err := uuid.Parse(workspaceID); err != nil { + return "", false, fmt.Errorf("invalid workspace identity: %w", err) + } + trashRoot := filepath.Join(m.vaultDir, ".verstak", "trash", "workspaces") + entries, err := os.ReadDir(trashRoot) + if err != nil { + if os.IsNotExist(err) { + return "", false, nil + } + return "", false, err + } + var match string + for _, entry := range entries { + if !entry.IsDir() { + continue + } + identity, err := m.GetWorkspaceTrashIdentity(entry.Name()) + if err != nil { + continue + } + if identity.WorkspaceID != workspaceID { + continue + } + if match != "" { + return "", false, fmt.Errorf("conflict: duplicated trashed workspace identity %s", workspaceID) + } + match = entry.Name() + } + return match, match != "", nil +} + func validateWorkspaceTrashID(trashID string) error { if trashID == "" || strings.ContainsAny(trashID, `/\\`) || filepath.Clean(trashID) != trashID { return fmt.Errorf("invalid workspace trash ID") diff --git a/internal/core/workspace/manager_test.go b/internal/core/workspace/manager_test.go index 1797ec0..1323047 100644 --- a/internal/core/workspace/manager_test.go +++ b/internal/core/workspace/manager_test.go @@ -7,6 +7,8 @@ import ( "runtime" "strings" "testing" + + "github.com/google/uuid" ) func TestListWorkspacesReadsTopLevelPhysicalFolders(t *testing.T) { @@ -435,6 +437,93 @@ func TestRestoreWorkspaceTrashPreservesIdentity(t *testing.T) { } } +func TestApplySyncedWorkspaceLifecycleKeepsIdentityAndRejectsNameConflict(t *testing.T) { + vaultDir := newVaultDir(t) + m := NewManager(vaultDir) + workspaceID := "5f0f96d9-61c8-4b6b-8c3a-a1b9a0f40001" + meta := Metadata{ + WorkspaceID: workspaceID, + WorkspaceName: "Remote", + CreatedFromTemplate: &TemplateSnapshot{ + TemplateID: "minimal", + TemplateName: "Minimal", + TemplateVersion: 1, + AppliedAt: "2026-07-17T00:00:00Z", + }, + Features: map[string]bool{"files": true}, + Folders: map[string]string{"notes": "Notes"}, + } + + created, err := m.CreateWorkspaceFromSync("Remote", workspaceID, meta) + if err != nil { + t.Fatalf("CreateWorkspaceFromSync: %v", err) + } + if created.ID != workspaceID || created.Name != "Remote" { + t.Fatalf("created = %+v", created) + } + if _, err := m.CreateWorkspaceFromSync("Remote", workspaceID, meta); err != nil { + t.Fatalf("replayed create: %v", err) + } + + if err := m.RenameWorkspaceFromSync(workspaceID, "Remote", "Remote-Renamed"); err != nil { + t.Fatalf("RenameWorkspaceFromSync: %v", err) + } + if err := m.RenameWorkspaceFromSync(workspaceID, "Remote", "Remote-Renamed"); err != nil { + t.Fatalf("replayed rename: %v", err) + } + trashed, err := m.TrashWorkspaceFromSync(workspaceID, "Remote-Renamed") + if err != nil { + t.Fatalf("TrashWorkspaceFromSync: %v", err) + } + if trashed.WorkspaceID != workspaceID { + t.Fatalf("trashed = %+v", trashed) + } + if _, err := m.TrashWorkspaceFromSync(workspaceID, "Remote-Renamed"); err != nil { + t.Fatalf("replayed trash: %v", err) + } + restored, err := m.RestoreWorkspaceFromSync(workspaceID, "Remote-Restored") + if err != nil { + t.Fatalf("RestoreWorkspaceFromSync: %v", err) + } + if restored.ID != workspaceID || restored.Name != "Remote-Restored" { + t.Fatalf("restored = %+v", restored) + } + if _, err := m.RestoreWorkspaceFromSync(workspaceID, "Remote-Restored"); err != nil { + t.Fatalf("replayed restore: %v", err) + } + + if _, err := m.CreateWorkspace("Taken", "minimal"); err != nil { + t.Fatalf("CreateWorkspace Taken: %v", err) + } + if err := m.RenameWorkspaceFromSync(workspaceID, "Remote-Restored", "Taken"); err == nil || !strings.Contains(err.Error(), "conflict") { + t.Fatalf("rename conflict error = %v, want conflict", err) + } + if _, err := m.CreateWorkspaceFromSync("Copied", workspaceID, meta); err == nil || !strings.Contains(err.Error(), "conflict") { + t.Fatalf("duplicate workspace ID error = %v, want conflict", err) + } + + stored, err := m.GetWorkspaceMetadata("Remote-Restored") + if err != nil { + t.Fatalf("GetWorkspaceMetadata: %v", err) + } + if stored.WorkspaceID != workspaceID || stored.CreatedFromTemplate == nil || stored.CreatedFromTemplate.TemplateID != "minimal" { + t.Fatalf("stored metadata = %+v", stored) + } +} + +func TestCreateWorkspaceFromSyncRejectsUnsafeMetadataFolder(t *testing.T) { + vaultDir := newVaultDir(t) + m := NewManager(vaultDir) + if _, err := m.CreateWorkspaceFromSync("Remote", uuid.NewString(), Metadata{ + Folders: map[string]string{"files": "../escaped"}, + }); err == nil || !strings.Contains(err.Error(), "invalid workspace folder") { + t.Fatalf("unsafe synced metadata error = %v", err) + } + if _, err := os.Stat(filepath.Join(vaultDir, "escaped")); !os.IsNotExist(err) { + t.Fatalf("unsafe synced metadata created path outside workspace: %v", err) + } +} + func TestPurgeWorkspaceTrashRemovesPayload(t *testing.T) { vaultDir := newVaultDir(t) m := NewManager(vaultDir) diff --git a/scripts/smoke-real-sync.sh b/scripts/smoke-real-sync.sh index 3d66aca..d288618 100755 --- a/scripts/smoke-real-sync.sh +++ b/scripts/smoke-real-sync.sh @@ -30,7 +30,10 @@ fi ) >"$LOG_FILE" 2>&1 & SERVER_PID="$!" -for _ in $(seq 1 80); do +# A cold Go cache can take longer than 20 seconds to compile `go run`; wait +# for the real listener rather than treating a still-running compiler as a +# healthy server. +for _ in $(seq 1 300); do if curl -fsS "http://127.0.0.1:$PORT/api/v1/health" >/dev/null 2>&1; then break fi @@ -45,10 +48,12 @@ curl -fsS "http://127.0.0.1:$PORT/api/v1/health" >/dev/null NOW="$(date -u +%Y-%m-%dT%H:%M:%SZ)" sqlite3 "$DATA_DIR/server.db" " -INSERT INTO server_devices (id, name, api_key, last_seen, created_at) -VALUES ('smoke-device-a', 'Smoke Device A', 'smoke-key-a', '$NOW', '$NOW'); -INSERT INTO server_devices (id, name, api_key, last_seen, created_at) -VALUES ('smoke-device-b', 'Smoke Device B', 'smoke-key-b', '$NOW', '$NOW'); +INSERT INTO server_users (id, username, email, password_hash, confirmed, created_at) +VALUES ('smoke-user', 'smoke-user', 'smoke@example.test', 'unused', 1, '$NOW'); +INSERT INTO server_devices (id, name, api_key, user_id, vault_id, last_seen, created_at) +VALUES ('smoke-device-a', 'Smoke Device A', 'smoke-key-a', 'smoke-user', 'smoke-vault', '$NOW', '$NOW'); +INSERT INTO server_devices (id, name, api_key, user_id, vault_id, last_seen, created_at) +VALUES ('smoke-device-b', 'Smoke Device B', 'smoke-key-b', 'smoke-user', 'smoke-vault', '$NOW', '$NOW'); " (