diff --git a/README.md b/README.md index bc1c4b7..93e22ed 100644 --- a/README.md +++ b/README.md @@ -33,7 +33,7 @@ As of Jan 2022, the plugin is considered in BETA stage. **DO NOT USE IT for any ## Limitations -- **"deletion" operation can only be triggered from local device.** It's because of the "[minimal intrusive design](./docs/minimal_intrusive_design.md)". May be changed in the future. +- **To support deltions sync, extra metadata will also be uploaded.** See [Minimal Intrusive](./docs/minimal_intrusive_design.md). - **No Conflict resolution. No content-diff-and-patch algorithm.** All files and folders are compared using their local and remote "last modified time" and those with later "last modified time" wins. - **Cloud services cost you money.** Always be aware of the costs and pricing. - **All files or folder starting with `.` (dot) or `_` (underscore) are treated as hidden files, and would NOT be synced.** It's useful if you have some files just staying locally. But this strategy also means that themes / other plugins / settings of this plugin would neither be synced. diff --git a/docs/minimal_intrusive_design.md b/docs/minimal_intrusive_design.md index 5d9f96b..68bd86c 100644 --- a/docs/minimal_intrusive_design.md +++ b/docs/minimal_intrusive_design.md @@ -1,36 +1,21 @@ # Minimal Intrusive Design -The plugin tries to avoid saving additional meta data remotely. +Before version 0.3.0, the plugin did not upload additional meta data to the remote. + +From and after version 0.3.0, the plugin just upload minimal extra necessary meta data to the remote. ## Benefits -Then the plugin doesn't make any assumptions about information on the remote endpoint. +Then the plugin doesn't make more-than-necessary assumptions about information on the remote endpoint. For example, it's possbile for a uses to manually upload a file to s3, and next time the plugin can download that file to the local device. And it's also possible to combine another "sync-to-s3" solution (like, another software) on desktops, and this plugin on mobile devices, together. -## Flaws +## Necessarity Of Uploading Extra Metadata The main issue comes from deletions (and renamings which is actually interpreted as "deletion-then-creation"). -Consider this: +If we don't upload any extra info to the remote, there's usually no way for the second device to know what files / folders have been deleted on the first device. -1. The user create and sync a file to the cloud on the 1st device. -2. Then download this file to the 2nd device. -3. And then delete this file on the 1st device. -4. And sync on the 1st device. The file on the cloud is also deleted. -5. And sync on the 2nd device. **The 2nd device would upload the file again to the cloud.** - -In step 4, the file is marked "deleted" on the 1st device, and the 1st device send the command "delete this file on the cloud" to the cloud sevice (e.g. s3). Then the file on the cloud is also deleted. So far so good. - -But, in step 5, because no meta data are saved on the cloud, the 2nd device doesn't know that the file are deleted. Instead, it thinks "the file was not synced to the cloud last time, so it's uploaded this time". So an unintentional upload occurs. - -Currently no way to fix this if no meta data are saved remotely. The only workarounds are: - -1. Delete the file on the 1st device, **before** syncing it to the cloud. Then the file never show up on the cloud or on the 2nd device. -2. Or, manually delete the file on 2nd device **before** step 5 in above situation. - -## Future - -This design may be changed in the feature, considering the flaws described above. +To overcome this issue, from and after version 0.3.0, the plugin uploads extra metadata files `_remotely-save-metadata-on-remote.{json,bin}` to users' configured cloud services. Those files contain some info about what has been deleted on the first device, so that the second device can read the list to apply the deletions to itself. Some other necessary meta info would also be written into the extra files. diff --git a/docs/sync_algorithm.md b/docs/sync_algorithm_v1.md similarity index 100% rename from docs/sync_algorithm.md rename to docs/sync_algorithm_v1.md diff --git a/docs/sync_algorithm_v2.md b/docs/sync_algorithm_v2.md new file mode 100644 index 0000000..4b8405f --- /dev/null +++ b/docs/sync_algorithm_v2.md @@ -0,0 +1,61 @@ +# Sync Algorithm V2 + +## Sources + +We have 4 record sources: + +1. Local files. By scanning all files in the vault locally. Actually Obsidian provides an api directly returning this. +2. Remote files. By scanning all files on the remote service. Some services provide an api directly returning this, and some other services require the plugin scanning the folders recursively. +3. Local "delete-or-rename" history. It's recorded by using Obsidian's tracking api. So if users delete or rename files/folders outside Obsidian, we could do nothing. +4. Remote "delete" history. It's uploaded by the plugin in each sync. + +Assuming all sources are reliable. + +## Deal with them + +We list all combinations mutually exclusive and collectively exhaustive. + +### Files + +| t1 | t2 | t3 | t4 | local file to do | remote file to do | local del history to do | remote del history to do | equal to sync v2 branch | +| -------------- | -------------- | -------------- | -------------- | ---------------- | ----------------- | ----------------------- | ------------------------ | ----------------------- | +| mtime_remote | mtime_local | deltime_remote | deltime_local | del_if_exists | del_if_exists | clean | upload_local_del_history | | +| mtime_local | mtime_remote | deltime_remote | deltime_local | del_if_exists | del_if_exists | clean | upload_local_del_history | | +| mtime_remote | deltime_remote | mtime_local | deltime_local | del_if_exists | del_if_exists | clean | upload_local_del_history | | +| deltime_remote | mtime_remote | mtime_local | deltime_local | del_if_exists | del_if_exists | clean | upload_local_del_history | | +| mtime_local | deltime_remote | mtime_remote | deltime_local | del_if_exists | del_if_exists | clean | upload_local_del_history | | +| deltime_remote | mtime_local | mtime_remote | deltime_local | del_if_exists | del_if_exists | clean | upload_local_del_history | 8 | +| mtime_remote | mtime_local | deltime_local | deltime_remote | del_if_exists | del_if_exists | clean | keep | | +| mtime_local | mtime_remote | deltime_local | deltime_remote | del_if_exists | del_if_exists | clean | keep | | +| mtime_remote | deltime_local | mtime_local | deltime_remote | del_if_exists | del_if_exists | clean | keep | | +| deltime_local | mtime_remote | mtime_local | deltime_remote | del_if_exists | del_if_exists | clean | keep | | +| mtime_local | deltime_local | mtime_remote | deltime_remote | del_if_exists | del_if_exists | clean | keep | | +| deltime_local | mtime_local | mtime_remote | deltime_remote | del_if_exists | del_if_exists | clean | keep | | +| mtime_remote | deltime_remote | deltime_local | mtime_local | skip | upload_local | clean | clean | | +| deltime_remote | mtime_remote | deltime_local | mtime_local | skip | upload_local | clean | clean | 10 | +| mtime_remote | deltime_local | deltime_remote | mtime_local | skip | upload_local | clean | clean | | +| deltime_local | mtime_remote | deltime_remote | mtime_local | skip | upload_local | clean | clean | | +| deltime_remote | deltime_local | mtime_remote | mtime_local | skip | upload_local | clean | clean | 2;3;4;5;6 | +| deltime_local | deltime_remote | mtime_remote | mtime_local | skip | upload_local | clean | clean | | +| mtime_local | deltime_remote | deltime_local | mtime_remote | download_remote | skip | clean | clean | | +| deltime_remote | mtime_local | deltime_local | mtime_remote | download_remote | skip | clean | clean | 7;9 | +| mtime_local | deltime_local | deltime_remote | mtime_remote | download_remote | skip | clean | clean | | +| deltime_local | mtime_local | deltime_remote | mtime_remote | download_remote | skip | clean | clean | | +| deltime_remote | deltime_local | mtime_local | mtime_remote | download_remote | skip | clean | clean | 1;9 | +| deltime_local | deltime_remote | mtime_local | mtime_remote | download_remote | skip | clean | clean | | + +### Folders + +We actually do not use any folders' metadata. Thus the only relevent info is their names, while the mtime is actually ignorable. + +1. Firstly generate all the files' plan. If any files exist, then it's parent folders all should exist. If the should-exist folder doesn't exist locally, the local should create it recursively. If the should-exist folder doesn't exist remotely, the remote should create it recursively. +2. Then, a folder is deletable, if and only if all the following conditions meet: + + - it shows up in the remote deletion history + - it's empty, or all its sub-folders are deletable + + Some examples: + + - A user deletes the folder in device 1, then syncs from the device 1, then creates the same-name folder in device 2, then syncs from the device 2. The folder is deleted (again), on device 2. + - A user deletes the folder in device 1, then syncs from the device 1, then creates the same-name folder in device 2, **then create a new file inside it,** then syncs from the device 2. The folder is **kept** instead of deleted because of the new file, on device 2. + - A user deletes the folder in device 1, then syncs from the device 1, then do not touch the same-name folder in device 2, then syncs from the device 2. The folder and its untouched sub-files should be deleted on device 2. diff --git a/manifest.json b/manifest.json index c65bbcb..9ba4c09 100644 --- a/manifest.json +++ b/manifest.json @@ -1,7 +1,7 @@ { "id": "remotely-save", "name": "Remotely Save", - "version": "0.2.14", + "version": "0.3.0", "minAppVersion": "0.12.15", "description": "Yet another unofficial plugin allowing users to synchronize notes between local device and the cloud service.", "author": "fyears", diff --git a/package.json b/package.json index bd50de2..e03a850 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "remotely-save", - "version": "0.2.14", + "version": "0.3.0", "description": "This is yet another sync plugin for Obsidian app.", "scripts": { "dev2": "node esbuild.config.mjs", diff --git a/src/baseTypes.ts b/src/baseTypes.ts index 946744b..9fe2992 100644 --- a/src/baseTypes.ts +++ b/src/baseTypes.ts @@ -80,3 +80,35 @@ export interface UriParams { // 80 days export const OAUTH2_FORCE_EXPIRE_MILLISECONDS = 1000 * 60 * 60 * 24 * 80; + +type DecisionTypeForFile = + | "skipUploading" // special, mtimeLocal === mtimeRemote + | "uploadLocalDelHistToRemote" // "delLocalIfExists && delRemoteIfExists && cleanLocalDelHist && uploadLocalDelHistToRemote" + | "keepRemoteDelHist" // "delLocalIfExists && delRemoteIfExists && cleanLocalDelHist && keepRemoteDelHist" + | "uploadLocalToRemote" // "skipLocal && uploadLocalToRemote && cleanLocalDelHist && cleanRemoteDelHist" + | "downloadRemoteToLocal"; // "downloadRemoteToLocal && skipRemote && cleanLocalDelHist && cleanRemoteDelHist" + +type DecisionTypeForFolder = + | "createFolder" + | "uploadLocalDelHistToRemoteFolder" + | "keepRemoteDelHistFolder" + | "skipFolder"; + +export type DecisionType = DecisionTypeForFile | DecisionTypeForFolder; + +export interface FileOrFolderMixedState { + key: string; + existLocal?: boolean; + existRemote?: boolean; + mtimeLocal?: number; + mtimeRemote?: number; + deltimeLocal?: number; + deltimeRemote?: number; + sizeLocal?: number; + sizeRemote?: number; + changeMtimeUsingMapping?: boolean; + decision?: DecisionType; + decisionBranch?: number; + syncDone?: "done"; + remoteEncryptedKey?: string; +} diff --git a/src/main.ts b/src/main.ts index 8d65721..6a02f17 100644 --- a/src/main.ts +++ b/src/main.ts @@ -34,11 +34,13 @@ import { import { DEFAULT_S3_CONFIG } from "./remoteForS3"; import { DEFAULT_WEBDAV_CONFIG } from "./remoteForWebdav"; import { RemotelySaveSettingTab } from "./settings"; -import type { SyncStatusType } from "./sync"; +import { fetchMetadataFile, parseRemoteItems, SyncStatusType } from "./sync"; import { doActualSync, getSyncPlan, isPasswordOk } from "./sync"; import { messyConfigToNormal, normalConfigToMessy } from "./configPersist"; import * as origLog from "loglevel"; +import { DeletionOnRemote, MetadataOnRemote } from "./metadataOnRemote"; +import { SyncAlgoV2Modal } from "./syncAlgoV2Notice"; const log = origLog.getLogger("rs-default"); const DEFAULT_SETTINGS: RemotelySavePluginSettings = { @@ -51,6 +53,7 @@ const DEFAULT_SETTINGS: RemotelySavePluginSettings = { currLogLevel: "info", vaultRandomID: "", autoRunEveryMilliseconds: -1, + agreeToUploadExtraMetadata: false, }; interface OAuth2Info { @@ -61,7 +64,7 @@ interface OAuth2Info { revokeAuthSetting?: Setting; } -type SyncTriggerSourceType = "manual" | "auto"; +type SyncTriggerSourceType = "manual" | "auto" | "dry"; const iconNameSyncWait = `remotely-save-sync-wait`; const iconNameSyncRunning = `remotely-save-sync-running`; @@ -89,7 +92,7 @@ export default class RemotelySavePlugin extends Plugin { const getNotice = (x: string) => { // only show notices in manual mode // no notice in auto mode - if (triggerSource === "manual") { + if (triggerSource === "manual" || triggerSource === "dry") { new Notice(x); } }; @@ -122,14 +125,22 @@ export default class RemotelySavePlugin extends Plugin { ); } + const MAX_STEPS = 8; + + if (triggerSource === "dry") { + getNotice( + `0/${MAX_STEPS} Remotely Save running in dry mode, not actual file changes would happen.` + ); + } + //log.info(`huh ${this.settings.password}`) getNotice( - `1/7 Remotely Save Sync Preparing (${this.settings.serviceType})` + `1/${MAX_STEPS} Remotely Save Sync Preparing (${this.settings.serviceType})` ); this.syncStatus = "preparing"; - getNotice("2/7 Starting to fetch remote meta data."); - this.syncStatus = "getting_remote_meta"; + getNotice(`2/${MAX_STEPS} Starting to fetch remote meta data.`); + this.syncStatus = "getting_remote_files_list"; const self = this; const client = new RemoteClient( this.settings.serviceType, @@ -143,17 +154,7 @@ export default class RemotelySavePlugin extends Plugin { const remoteRsp = await client.listFromRemote(); log.info(remoteRsp); - getNotice("3/7 Starting to fetch local meta data."); - this.syncStatus = "getting_local_meta"; - const local = this.app.vault.getAllLoadedFiles(); - const localHistory = await loadDeleteRenameHistoryTableByVault( - this.db, - this.settings.vaultRandomID - ); - // log.info(local); - // log.info(localHistory); - - getNotice("4/7 Checking password correct or not."); + getNotice(`3/${MAX_STEPS} Checking password correct or not.`); this.syncStatus = "checking_password"; const passwordCheckResult = await isPasswordOk( remoteRsp.Contents, @@ -164,43 +165,81 @@ export default class RemotelySavePlugin extends Plugin { throw Error(passwordCheckResult.reason); } - getNotice("5/7 Starting to generate sync plan."); - this.syncStatus = "generating_plan"; - const syncPlan = await getSyncPlan( + getNotice(`4/${MAX_STEPS} Trying to fetch extra meta data from remote.`); + this.syncStatus = "getting_remote_extra_meta"; + const { remoteStates, metadataFile } = await parseRemoteItems( remoteRsp.Contents, - local, - localHistory, this.db, this.settings.vaultRandomID, client.serviceType, this.settings.password ); - log.info(syncPlan.mixedStates); // for debugging - await insertSyncPlanRecordByVault( + const origMetadataOnRemote = await fetchMetadataFile( + metadataFile, + client, + this.app.vault, + this.settings.password + ); + + getNotice(`5/${MAX_STEPS} Starting to fetch local meta data.`); + this.syncStatus = "getting_local_meta"; + const local = this.app.vault.getAllLoadedFiles(); + const localHistory = await loadDeleteRenameHistoryTableByVault( this.db, - syncPlan, this.settings.vaultRandomID ); + // log.info(local); + // log.info(localHistory); - // The operations above are read only and kind of safe. + getNotice(`6/${MAX_STEPS} Starting to generate sync plan.`); + this.syncStatus = "generating_plan"; + const { plan, sortedKeys, deletions } = await getSyncPlan( + remoteStates, + local, + origMetadataOnRemote.deletions, + localHistory, + client.serviceType, + this.settings.password + ); + log.info(plan.mixedStates); // for debugging + if (triggerSource !== "dry") { + await insertSyncPlanRecordByVault( + this.db, + plan, + this.settings.vaultRandomID + ); + } + + // The operations above are almost read only and kind of safe. // The operations below begins to write or delete (!!!) something. - getNotice("6/7 Remotely Save Sync data exchanging!"); + if (triggerSource !== "dry") { + getNotice(`7/${MAX_STEPS} Remotely Save Sync data exchanging!`); - this.syncStatus = "syncing"; - await doActualSync( - client, - this.db, - this.settings.vaultRandomID, - this.app.vault, - syncPlan, - this.settings.password, - (i: number, totalCount: number, pathName: string, decision: string) => - self.setCurrSyncMsg(i, totalCount, pathName, decision) - ); + this.syncStatus = "syncing"; + await doActualSync( + client, + this.db, + this.settings.vaultRandomID, + this.app.vault, + plan, + sortedKeys, + metadataFile, + origMetadataOnRemote, + deletions, + (key: string) => self.trash(key), + this.settings.password, + (i: number, totalCount: number, pathName: string, decision: string) => + self.setCurrSyncMsg(i, totalCount, pathName, decision) + ); + } else { + this.syncStatus = "syncing"; + getNotice( + `7/${MAX_STEPS} Remotely Save real sync is skipped in dry run mode.` + ); + } - getNotice("7/7 Remotely Save finish!"); - this.currSyncMsg = ""; + getNotice(`8/${MAX_STEPS} Remotely Save finish!`); this.syncStatus = "finish"; this.syncStatus = "idle"; @@ -474,22 +513,24 @@ export default class RemotelySavePlugin extends Plugin { }, }); + this.addCommand({ + id: "start-sync-dry-run", + name: "start sync (dry run only)", + icon: iconNameSyncWait, + callback: async () => { + this.syncRun("dry"); + }, + }); + this.addSettingTab(new RemotelySaveSettingTab(this.app, this)); // this.registerDomEvent(document, "click", (evt: MouseEvent) => { // log.info("click", evt); // }); - if ( - this.settings.autoRunEveryMilliseconds !== undefined && - this.settings.autoRunEveryMilliseconds !== null && - this.settings.autoRunEveryMilliseconds > 0 - ) { - const intervalID = window.setInterval(() => { - this.syncRun("auto"); - }, this.settings.autoRunEveryMilliseconds); - this.autoRunIntervalID = intervalID; - this.registerInterval(intervalID); + if (!this.settings.agreeToUploadExtraMetadata) { + const syncAlgoV2Modal = new SyncAlgoV2Modal(this.app, this); + syncAlgoV2Modal.open(); } } @@ -602,10 +643,35 @@ export default class RemotelySavePlugin extends Plugin { } } + async trash(x: string) { + if (!(await this.app.vault.adapter.trashSystem(x))) { + await this.app.vault.adapter.trashLocal(x); + } + } + async prepareDB() { this.db = await prepareDBs(this.settings.vaultRandomID); } + enableAutoSyncIfSet() { + if ( + this.settings.autoRunEveryMilliseconds !== undefined && + this.settings.autoRunEveryMilliseconds !== null && + this.settings.autoRunEveryMilliseconds > 0 + ) { + const intervalID = window.setInterval(() => { + this.syncRun("auto"); + }, this.settings.autoRunEveryMilliseconds); + this.autoRunIntervalID = intervalID; + this.registerInterval(intervalID); + } + } + + async saveAgreeToUseNewSyncAlgorithm() { + this.settings.agreeToUploadExtraMetadata = true; + await this.saveSettings(); + } + destroyDBs() { /* destroyDBs(this.db); */ } diff --git a/src/metadataOnRemote.ts b/src/metadataOnRemote.ts new file mode 100644 index 0000000..66d45fe --- /dev/null +++ b/src/metadataOnRemote.ts @@ -0,0 +1,87 @@ +import isEqual from "lodash/isEqual"; +import { base64url } from "rfc4648"; +import { reverseString } from "./misc"; + +const DEFAULT_README_FOR_METADATAONREMOTE = + "Do NOT edit or delete the file manually. This file is for the plugin remotely-save to store some necessary meta data on the remote services. Its content is slightly obfuscated."; + +const DEFAULT_VERSION_FOR_METADATAONREMOTE = "20220220"; + +export const DEFAULT_FILE_NAME_FOR_METADATAONREMOTE = + "_remotely-save-metadata-on-remote.json"; + +export const DEFAULT_FILE_NAME_FOR_METADATAONREMOTE2 = + "_remotely-save-metadata-on-remote.bin"; + +export interface DeletionOnRemote { + key: string; + actionWhen: number; +} + +export interface MetadataOnRemote { + version?: string; + generatedWhen?: number; + deletions?: DeletionOnRemote[]; +} + +export const isEqualMetadataOnRemote = ( + a: MetadataOnRemote, + b: MetadataOnRemote +) => { + const m1 = a === undefined ? { deletions: [] } : a; + const m2 = b === undefined ? { deletions: [] } : b; + + // we only need to compare deletions + const d1 = m1.deletions === undefined ? [] : m1.deletions; + const d2 = m2.deletions === undefined ? [] : m2.deletions; + return isEqual(d1, d2); +}; + +export const serializeMetadataOnRemote = (x: MetadataOnRemote) => { + const y = x; + + if (y["version"] === undefined) { + y["version"] === DEFAULT_VERSION_FOR_METADATAONREMOTE; + } + if (y["generatedWhen"] === undefined) { + y["generatedWhen"] = Date.now(); + } + if (y["deletions"] === undefined) { + y["deletions"] = []; + } + + const z = { + readme: DEFAULT_README_FOR_METADATAONREMOTE, + d: reverseString( + base64url.stringify(Buffer.from(JSON.stringify(x), "utf-8"), { + pad: false, + }) + ), + }; + + return JSON.stringify(z, null, 2); +}; + +export const deserializeMetadataOnRemote = (x: string | ArrayBuffer) => { + let y1 = ""; + if (typeof x === "string") { + y1 = x; + } else { + y1 = new TextDecoder().decode(x); + } + const y2: any = JSON.parse(y1); + + if (!("readme" in y2 && "d" in y2)) { + throw Error("invalid remote meta data file!"); + } + + const y3 = JSON.parse( + ( + base64url.parse(reverseString(y2["d"]), { + out: Buffer.allocUnsafe as any, + loose: true, + }) as Buffer + ).toString("utf-8") + ) as MetadataOnRemote; + return y3; +}; diff --git a/src/misc.ts b/src/misc.ts index b143f2f..ed00ca7 100644 --- a/src/misc.ts +++ b/src/misc.ts @@ -38,7 +38,7 @@ export const isHiddenPath = (item: string, loose: boolean = true) => { * @param x string * @returns string[] might be empty */ -export const getFolderLevels = (x: string) => { +export const getFolderLevels = (x: string, addEndingSlash: boolean = false) => { const res: string[] = []; if (x === "" || x === "/") { @@ -48,10 +48,14 @@ export const getFolderLevels = (x: string) => { const y1 = x.split("/"); let i = 0; for (let index = 0; index + 1 < y1.length; index++) { - const k = y1.slice(0, index + 1).join("/"); - if (k !== "" && k !== "/") { - res.push(k); + let k = y1.slice(0, index + 1).join("/"); + if (k === "" || k === "/") { + continue; } + if (addEndingSlash) { + k = `${k}/`; + } + res.push(k); } return res; }; @@ -157,6 +161,24 @@ export const getPathFolder = (a: string) => { return b.endsWith("/") ? b : `${b}/`; }; +/** + * If input is already a folder, returns its folder; + * And if input is a file, returns its direname. + * @param a + * @returns + */ +export const getParentFolder = (a: string) => { + const b = path.posix.dirname(a); + if (b === "." || b === "/") { + // the root + return "/"; + } + if (b.endsWith("/")) { + return b; + } + return `${b}/`; +}; + /** * https://stackoverflow.com/questions/54511144 * @param a diff --git a/src/remote.ts b/src/remote.ts index 6057bc2..008944b 100644 --- a/src/remote.ts +++ b/src/remote.ts @@ -102,7 +102,9 @@ export class RemoteClient { isRecursively: boolean = false, password: string = "", remoteEncryptedKey: string = "", - foldersCreatedBefore: Set | undefined = undefined + foldersCreatedBefore: Set | undefined = undefined, + uploadRaw: boolean = false, + rawContent: string | ArrayBuffer = "" ) => { if (this.serviceType === "s3") { return await s3.uploadToRemote( @@ -112,7 +114,9 @@ export class RemoteClient { vault, isRecursively, password, - remoteEncryptedKey + remoteEncryptedKey, + uploadRaw, + rawContent ); } else if (this.serviceType === "webdav") { return await webdav.uploadToRemote( @@ -121,7 +125,9 @@ export class RemoteClient { vault, isRecursively, password, - remoteEncryptedKey + remoteEncryptedKey, + uploadRaw, + rawContent ); } else if (this.serviceType === "dropbox") { return await dropbox.uploadToRemote( @@ -131,7 +137,9 @@ export class RemoteClient { isRecursively, password, remoteEncryptedKey, - foldersCreatedBefore + foldersCreatedBefore, + uploadRaw, + rawContent ); } else if (this.serviceType === "onedrive") { return await onedrive.uploadToRemote( @@ -141,7 +149,9 @@ export class RemoteClient { isRecursively, password, remoteEncryptedKey, - foldersCreatedBefore + foldersCreatedBefore, + uploadRaw, + rawContent ); } else { throw Error(`not supported service type ${this.serviceType}`); @@ -167,7 +177,8 @@ export class RemoteClient { vault: Vault, mtime: number, password: string = "", - remoteEncryptedKey: string = "" + remoteEncryptedKey: string = "", + skipSaving: boolean = false ) => { if (this.serviceType === "s3") { return await s3.downloadFromRemote( @@ -177,7 +188,8 @@ export class RemoteClient { vault, mtime, password, - remoteEncryptedKey + remoteEncryptedKey, + skipSaving ); } else if (this.serviceType === "webdav") { return await webdav.downloadFromRemote( @@ -186,7 +198,8 @@ export class RemoteClient { vault, mtime, password, - remoteEncryptedKey + remoteEncryptedKey, + skipSaving ); } else if (this.serviceType === "dropbox") { return await dropbox.downloadFromRemote( @@ -195,7 +208,8 @@ export class RemoteClient { vault, mtime, password, - remoteEncryptedKey + remoteEncryptedKey, + skipSaving ); } else if (this.serviceType === "onedrive") { return await onedrive.downloadFromRemote( @@ -204,7 +218,8 @@ export class RemoteClient { vault, mtime, password, - remoteEncryptedKey + remoteEncryptedKey, + skipSaving ); } else { throw Error(`not supported service type ${this.serviceType}`); diff --git a/src/remoteForDropbox.ts b/src/remoteForDropbox.ts index 08f7627..67dc8bf 100644 --- a/src/remoteForDropbox.ts +++ b/src/remoteForDropbox.ts @@ -400,7 +400,9 @@ export const uploadToRemote = async ( isRecursively: boolean = false, password: string = "", remoteEncryptedKey: string = "", - foldersCreatedBefore: Set | undefined = undefined + foldersCreatedBefore: Set | undefined = undefined, + uploadRaw: boolean = false, + rawContent: string | ArrayBuffer = "" ) => { await client.init(); @@ -415,6 +417,9 @@ export const uploadToRemote = async ( if (isFolder && isRecursively) { throw Error("upload function doesn't implement recursive function yet!"); } else if (isFolder && !isRecursively) { + if (uploadRaw) { + throw Error(`you specify uploadRaw, but you also provide a folder key!`); + } // folder if (password === "") { // if not encrypted, mkdir a remote folder @@ -448,7 +453,16 @@ export const uploadToRemote = async ( } else { // file // we ignore isRecursively parameter here - const localContent = await vault.adapter.readBinary(fileOrFolderPath); + let localContent = undefined; + if (uploadRaw) { + if (typeof rawContent === "string") { + localContent = new TextEncoder().encode(rawContent); + } else { + localContent = rawContent; + } + } else { + localContent = await vault.adapter.readBinary(fileOrFolderPath); + } let remoteContent = localContent; if (password !== "") { remoteContent = await encryptArrayBuffer(localContent, password); @@ -551,13 +565,16 @@ export const downloadFromRemote = async ( vault: Vault, mtime: number, password: string = "", - remoteEncryptedKey: string = "" + remoteEncryptedKey: string = "", + skipSaving: boolean = false ) => { await client.init(); const isFolder = fileOrFolderPath.endsWith("/"); - await mkdirpInVault(fileOrFolderPath, vault); + if (!skipSaving) { + await mkdirpInVault(fileOrFolderPath, vault); + } // the file is always local file // we need to encrypt it @@ -565,6 +582,7 @@ export const downloadFromRemote = async ( if (isFolder) { // mkdirp locally is enough // do nothing here + return new ArrayBuffer(0); } else { let downloadFile = fileOrFolderPath; if (password !== "") { @@ -576,9 +594,12 @@ export const downloadFromRemote = async ( if (password !== "") { localContent = await decryptArrayBuffer(remoteContent, password); } - await vault.adapter.writeBinary(fileOrFolderPath, localContent, { - mtime: mtime, - }); + if (!skipSaving) { + await vault.adapter.writeBinary(fileOrFolderPath, localContent, { + mtime: mtime, + }); + } + return localContent; } }; diff --git a/src/remoteForOnedrive.ts b/src/remoteForOnedrive.ts index 5e82bef..5a40f47 100644 --- a/src/remoteForOnedrive.ts +++ b/src/remoteForOnedrive.ts @@ -624,7 +624,9 @@ export const uploadToRemote = async ( isRecursively: boolean = false, password: string = "", remoteEncryptedKey: string = "", - foldersCreatedBefore: Set | undefined = undefined + foldersCreatedBefore: Set | undefined = undefined, + uploadRaw: boolean = false, + rawContent: string | ArrayBuffer = "" ) => { await client.init(); @@ -640,6 +642,9 @@ export const uploadToRemote = async ( if (isFolder && isRecursively) { throw Error("upload function doesn't implement recursive function yet!"); } else if (isFolder && !isRecursively) { + if (uploadRaw) { + throw Error(`you specify uploadRaw, but you also provide a folder key!`); + } // folder if (password === "") { // if not encrypted, mkdir a remote folder @@ -682,7 +687,16 @@ export const uploadToRemote = async ( } else { // file // we ignore isRecursively parameter here - const localContent = await vault.adapter.readBinary(fileOrFolderPath); + let localContent = undefined; + if (uploadRaw) { + if (typeof rawContent === "string") { + localContent = new TextEncoder().encode(rawContent); + } else { + localContent = rawContent; + } + } else { + localContent = await vault.adapter.readBinary(fileOrFolderPath); + } let remoteContent = localContent; if (password !== "") { remoteContent = await encryptArrayBuffer(localContent, password); @@ -751,17 +765,21 @@ export const downloadFromRemote = async ( vault: Vault, mtime: number, password: string = "", - remoteEncryptedKey: string = "" + remoteEncryptedKey: string = "", + skipSaving: boolean = false ) => { await client.init(); const isFolder = fileOrFolderPath.endsWith("/"); - await mkdirpInVault(fileOrFolderPath, vault); + if (!skipSaving) { + await mkdirpInVault(fileOrFolderPath, vault); + } if (isFolder) { // mkdirp locally is enough // do nothing here + return new ArrayBuffer(0); } else { let downloadFile = fileOrFolderPath; if (password !== "") { @@ -773,9 +791,12 @@ export const downloadFromRemote = async ( if (password !== "") { localContent = await decryptArrayBuffer(remoteContent, password); } - await vault.adapter.writeBinary(fileOrFolderPath, localContent, { - mtime: mtime, - }); + if (!skipSaving) { + await vault.adapter.writeBinary(fileOrFolderPath, localContent, { + mtime: mtime, + }); + } + return localContent; } }; diff --git a/src/remoteForS3.ts b/src/remoteForS3.ts index 42c7698..2fbf123 100644 --- a/src/remoteForS3.ts +++ b/src/remoteForS3.ts @@ -99,7 +99,9 @@ export const uploadToRemote = async ( vault: Vault, isRecursively: boolean = false, password: string = "", - remoteEncryptedKey: string = "" + remoteEncryptedKey: string = "", + uploadRaw: boolean = false, + rawContent: string | ArrayBuffer = "" ) => { let uploadFile = fileOrFolderPath; if (password !== "") { @@ -112,6 +114,9 @@ export const uploadToRemote = async ( if (isFolder && isRecursively) { throw Error("upload function doesn't implement recursive function yet!"); } else if (isFolder && !isRecursively) { + if (uploadRaw) { + throw Error(`you specify uploadRaw, but you also provide a folder key!`); + } // folder const contentType = DEFAULT_CONTENT_TYPE; await s3Client.send( @@ -133,7 +138,16 @@ export const uploadToRemote = async ( mime.lookup(fileOrFolderPath) || DEFAULT_CONTENT_TYPE ) || DEFAULT_CONTENT_TYPE; } - const localContent = await vault.adapter.readBinary(fileOrFolderPath); + let localContent = undefined; + if (uploadRaw) { + if (typeof rawContent === "string") { + localContent = new TextEncoder().encode(rawContent); + } else { + localContent = rawContent; + } + } else { + localContent = await vault.adapter.readBinary(fileOrFolderPath); + } let remoteContent = localContent; if (password !== "") { remoteContent = await encryptArrayBuffer(localContent, password); @@ -252,11 +266,14 @@ export const downloadFromRemote = async ( vault: Vault, mtime: number, password: string = "", - remoteEncryptedKey: string = "" + remoteEncryptedKey: string = "", + skipSaving: boolean = false ) => { const isFolder = fileOrFolderPath.endsWith("/"); - await mkdirpInVault(fileOrFolderPath, vault); + if (!skipSaving) { + await mkdirpInVault(fileOrFolderPath, vault); + } // the file is always local file // we need to encrypt it @@ -264,6 +281,7 @@ export const downloadFromRemote = async ( if (isFolder) { // mkdirp locally is enough // do nothing here + return new ArrayBuffer(0); } else { let downloadFile = fileOrFolderPath; if (password !== "") { @@ -278,9 +296,12 @@ export const downloadFromRemote = async ( if (password !== "") { localContent = await decryptArrayBuffer(remoteContent, password); } - await vault.adapter.writeBinary(fileOrFolderPath, localContent, { - mtime: mtime, - }); + if (!skipSaving) { + await vault.adapter.writeBinary(fileOrFolderPath, localContent, { + mtime: mtime, + }); + } + return localContent; } }; diff --git a/src/remoteForWebdav.ts b/src/remoteForWebdav.ts index c6ea009..bc8724f 100644 --- a/src/remoteForWebdav.ts +++ b/src/remoteForWebdav.ts @@ -138,7 +138,9 @@ export const uploadToRemote = async ( vault: Vault, isRecursively: boolean = false, password: string = "", - remoteEncryptedKey: string = "" + remoteEncryptedKey: string = "", + uploadRaw: boolean = false, + rawContent: string | ArrayBuffer = "" ) => { await client.init(); let uploadFile = fileOrFolderPath; @@ -152,6 +154,9 @@ export const uploadToRemote = async ( if (isFolder && isRecursively) { throw Error("upload function doesn't implement recursive function yet!"); } else if (isFolder && !isRecursively) { + if (uploadRaw) { + throw Error(`you specify uploadRaw, but you also provide a folder key!`); + } // folder if (password === "") { // if not encrypted, mkdir a remote folder @@ -174,7 +179,16 @@ export const uploadToRemote = async ( } else { // file // we ignore isRecursively parameter here - const localContent = await vault.adapter.readBinary(fileOrFolderPath); + let localContent = undefined; + if (uploadRaw) { + if (typeof rawContent === "string") { + localContent = new TextEncoder().encode(rawContent); + } else { + localContent = rawContent; + } + } else { + localContent = await vault.adapter.readBinary(fileOrFolderPath); + } let remoteContent = localContent; if (password !== "") { remoteContent = await encryptArrayBuffer(localContent, password); @@ -277,13 +291,16 @@ export const downloadFromRemote = async ( vault: Vault, mtime: number, password: string = "", - remoteEncryptedKey: string = "" + remoteEncryptedKey: string = "", + skipSaving: boolean = false ) => { await client.init(); const isFolder = fileOrFolderPath.endsWith("/"); - await mkdirpInVault(fileOrFolderPath, vault); + if (!skipSaving) { + await mkdirpInVault(fileOrFolderPath, vault); + } // the file is always local file // we need to encrypt it @@ -291,6 +308,7 @@ export const downloadFromRemote = async ( if (isFolder) { // mkdirp locally is enough // do nothing here + return new ArrayBuffer(0); } else { let downloadFile = fileOrFolderPath; if (password !== "") { @@ -302,9 +320,12 @@ export const downloadFromRemote = async ( if (password !== "") { localContent = await decryptArrayBuffer(remoteContent, password); } - await vault.adapter.writeBinary(fileOrFolderPath, localContent, { - mtime: mtime, - }); + if (!skipSaving) { + await vault.adapter.writeBinary(fileOrFolderPath, localContent, { + mtime: mtime, + }); + } + return localContent; } }; diff --git a/src/sync.ts b/src/sync.ts index 8816722..b4081e0 100644 --- a/src/sync.ts +++ b/src/sync.ts @@ -1,5 +1,10 @@ import { TAbstractFile, TFile, TFolder, Vault } from "obsidian"; -import type { RemoteItem, SUPPORTED_SERVICES_TYPE } from "./baseTypes"; +import type { + RemoteItem, + SUPPORTED_SERVICES_TYPE, + DecisionType, + FileOrFolderMixedState, +} from "./baseTypes"; import { decryptBase32ToString, decryptBase64urlToString, @@ -13,49 +18,40 @@ import { getSyncMetaMappingByRemoteKeyAndVault, upsertSyncMetaMappingDataByVault, } from "./localdb"; -import { isHiddenPath, isVaildText, mkdirpInVault } from "./misc"; +import { + isHiddenPath, + isVaildText, + mkdirpInVault, + getFolderLevels, + getParentFolder, +} from "./misc"; import { RemoteClient } from "./remote"; +import { + MetadataOnRemote, + DeletionOnRemote, + serializeMetadataOnRemote, + deserializeMetadataOnRemote, + DEFAULT_FILE_NAME_FOR_METADATAONREMOTE, + DEFAULT_FILE_NAME_FOR_METADATAONREMOTE2, + isEqualMetadataOnRemote, +} from "./metadataOnRemote"; import * as origLog from "loglevel"; +import { padEnd } from "lodash"; const log = origLog.getLogger("rs-default"); export type SyncStatusType = | "idle" | "preparing" - | "getting_remote_meta" + | "getting_remote_files_list" + | "getting_remote_extra_meta" | "getting_local_meta" | "checking_password" | "generating_plan" | "syncing" + | "cleaning" | "finish"; -type DecisionType = - | "undecided" - | "unknown" - | "upload_clearhist" - | "download_clearhist" - | "delremote_clearhist" - | "download" - | "upload" - | "clearhist" - | "mkdirplocal" - | "skip"; - -interface FileOrFolderMixedState { - key: string; - exist_local?: boolean; - exist_remote?: boolean; - mtime_local?: number; - mtime_remote?: number; - delete_time_local?: number; - size_local?: number; - size_remote?: number; - decision?: DecisionType; - syncDone?: "done"; - decision_branch?: number; - remote_encrypted_key?: string; -} - export interface SyncPlanType { ts: number; remoteType: SUPPORTED_SERVICES_TYPE; @@ -165,73 +161,125 @@ export const isPasswordOk = async ( } }; -const ensembleMixedStates = async ( +export const parseRemoteItems = async ( remote: RemoteItem[], - local: TAbstractFile[], - deleteHistory: FileFolderHistoryRecord[], db: InternalDBs, vaultRandomID: string, remoteType: SUPPORTED_SERVICES_TYPE, password: string = "" ) => { - const results = {} as Record; + const remoteStates = [] as FileOrFolderMixedState[]; + let metadataFile: FileOrFolderMixedState = undefined; + if (remote === undefined) { + return { + remoteStates: remoteStates, + metadataFile: metadataFile, + }; + } - if (remote !== undefined) { - for (const entry of remote) { - const remoteEncryptedKey = entry.key; - let key = remoteEncryptedKey; - if (password !== "") { - if (remoteEncryptedKey.startsWith(MAGIC_ENCRYPTED_PREFIX_BASE32)) { - key = await decryptBase32ToString(remoteEncryptedKey, password); - } else if ( - remoteEncryptedKey.startsWith(MAGIC_ENCRYPTED_PREFIX_BASE64URL) - ) { - key = await decryptBase64urlToString(remoteEncryptedKey, password); - } else { - throw Error(`unexpected key=${remoteEncryptedKey}`); - } - } - const backwardMapping = await getSyncMetaMappingByRemoteKeyAndVault( - remoteType, - db, - key, - entry.lastModified, - entry.etag, - vaultRandomID - ); - - let r = {} as FileOrFolderMixedState; - if (backwardMapping !== undefined) { - key = backwardMapping.localKey; - r = { - key: key, - exist_remote: true, - mtime_remote: backwardMapping.localMtime || entry.lastModified, - size_remote: backwardMapping.localSize || entry.size, - remote_encrypted_key: remoteEncryptedKey, - }; + for (const entry of remote) { + const remoteEncryptedKey = entry.key; + let key = remoteEncryptedKey; + if (password !== "") { + if (remoteEncryptedKey.startsWith(MAGIC_ENCRYPTED_PREFIX_BASE32)) { + key = await decryptBase32ToString(remoteEncryptedKey, password); + } else if ( + remoteEncryptedKey.startsWith(MAGIC_ENCRYPTED_PREFIX_BASE64URL) + ) { + key = await decryptBase64urlToString(remoteEncryptedKey, password); } else { - r = { - key: key, - exist_remote: true, - mtime_remote: entry.lastModified, - size_remote: entry.size, - remote_encrypted_key: remoteEncryptedKey, - }; - } - if (isHiddenPath(key)) { - continue; - } - if (results.hasOwnProperty(key)) { - results[key].key = r.key; - results[key].exist_remote = r.exist_remote; - results[key].mtime_remote = r.mtime_remote; - results[key].size_remote = r.size_remote; - results[key].remote_encrypted_key = r.remote_encrypted_key; - } else { - results[key] = r; + throw Error(`unexpected key=${remoteEncryptedKey}`); } } + const backwardMapping = await getSyncMetaMappingByRemoteKeyAndVault( + remoteType, + db, + key, + entry.lastModified, + entry.etag, + vaultRandomID + ); + + let r = {} as FileOrFolderMixedState; + if (backwardMapping !== undefined) { + key = backwardMapping.localKey; + r = { + key: key, + existRemote: true, + mtimeRemote: backwardMapping.localMtime || entry.lastModified, + sizeRemote: backwardMapping.localSize || entry.size, + remoteEncryptedKey: remoteEncryptedKey, + changeMtimeUsingMapping: true, + }; + } else { + r = { + key: key, + existRemote: true, + mtimeRemote: entry.lastModified, + sizeRemote: entry.size, + remoteEncryptedKey: remoteEncryptedKey, + changeMtimeUsingMapping: false, + }; + } + + if (r.key === DEFAULT_FILE_NAME_FOR_METADATAONREMOTE) { + metadataFile = Object.assign({}, r); + } + if (r.key === DEFAULT_FILE_NAME_FOR_METADATAONREMOTE2) { + throw Error( + `A reserved file name ${r.key} has been found. You may upgrade the plugin to latest version to try to deal with it.` + ); + } + + remoteStates.push(r); + } + return { + remoteStates: remoteStates, + metadataFile: metadataFile, + }; +}; + +export const fetchMetadataFile = async ( + metadataFile: FileOrFolderMixedState, + client: RemoteClient, + vault: Vault, + password: string = "" +) => { + if (metadataFile === undefined) { + log.debug("no metadata file, so no fetch"); + return { + deletions: [], + } as MetadataOnRemote; + } + + const buf = await client.downloadFromRemote( + metadataFile.key, + vault, + metadataFile.mtimeRemote, + password, + metadataFile.remoteEncryptedKey, + true + ); + const metadata = deserializeMetadataOnRemote(buf); + return metadata; +}; + +const ensembleMixedStates = async ( + remoteStates: FileOrFolderMixedState[], + local: TAbstractFile[], + remoteDeleteHistory: DeletionOnRemote[], + localDeleteHistory: FileFolderHistoryRecord[] +) => { + const results = {} as Record; + + for (const r of remoteStates) { + const key = r.key; + + if (isHiddenPath(key)) { + continue; + } + results[key] = r; + results[key].existLocal = false; } for (const entry of local) { @@ -244,17 +292,17 @@ const ensembleMixedStates = async ( } else if (entry instanceof TFile) { r = { key: entry.path, - exist_local: true, - mtime_local: entry.stat.mtime, - size_local: entry.stat.size, + existLocal: true, + mtimeLocal: entry.stat.mtime, + sizeLocal: entry.stat.size, }; } else if (entry instanceof TFolder) { key = `${entry.path}/`; r = { key: key, - exist_local: true, - mtime_local: undefined, - size_local: 0, + existLocal: true, + mtimeLocal: undefined, + sizeLocal: 0, }; } else { throw Error(`unexpected ${entry}`); @@ -265,15 +313,34 @@ const ensembleMixedStates = async ( } if (results.hasOwnProperty(key)) { results[key].key = r.key; - results[key].exist_local = r.exist_local; - results[key].mtime_local = r.mtime_local; - results[key].size_local = r.size_local; + results[key].existLocal = r.existLocal; + results[key].mtimeLocal = r.mtimeLocal; + results[key].sizeLocal = r.sizeLocal; } else { results[key] = r; + results[key].existRemote = false; } } - for (const entry of deleteHistory) { + for (const entry of remoteDeleteHistory) { + const key = entry.key; + const r = { + key: key, + deltimeRemote: entry.actionWhen, + } as FileOrFolderMixedState; + + if (results.hasOwnProperty(key)) { + results[key].key = r.key; + results[key].deltimeRemote = r.deltimeRemote; + } else { + results[key] = r; + + results[key].existLocal = false; + results[key].existRemote = false; + } + } + + for (const entry of localDeleteHistory) { let key = entry.key; if (entry.keyType === "folder") { if (!entry.key.endsWith("/")) { @@ -287,7 +354,7 @@ const ensembleMixedStates = async ( const r = { key: key, - delete_time_local: entry.actionWhen, + deltimeLocal: entry.actionWhen, } as FileOrFolderMixedState; if (isHiddenPath(key)) { @@ -295,193 +362,363 @@ const ensembleMixedStates = async ( } if (results.hasOwnProperty(key)) { results[key].key = r.key; - results[key].delete_time_local = r.delete_time_local; + results[key].deltimeLocal = r.deltimeLocal; } else { results[key] = r; + + results[key].existLocal = false; + results[key].existRemote = false; } } return results; }; -const getOperation = ( +const assignOperationToFileInplace = ( origRecord: FileOrFolderMixedState, - inplace: boolean = false, + keptFolder: Set, password: string = "" ) => { let r = origRecord; - if (!inplace) { - r = Object.assign({}, origRecord); + + // files and folders are treated differently + // here we only check files + if (r.key.endsWith("/")) { + return r; } - if (r.mtime_local === 0) { - r.mtime_local = undefined; - } - if (r.mtime_remote === 0) { - r.mtime_remote = undefined; - } - if (r.delete_time_local === 0) { - r.delete_time_local = undefined; - } - if (r.exist_local === undefined) { - r.exist_local = false; - } - if (r.exist_remote === undefined) { - r.exist_remote = false; - } - r.decision = "unknown"; + // we find the max date from four sources - if ( - r.exist_remote && - r.exist_local && - r.mtime_remote !== undefined && - r.mtime_local !== undefined && - r.mtime_remote > r.mtime_local - ) { - r.decision = "download_clearhist"; - r.decision_branch = 1; - } else if ( - r.exist_remote && - r.exist_local && - r.mtime_remote !== undefined && - r.mtime_local !== undefined && - r.mtime_remote < r.mtime_local - ) { - r.decision = "upload_clearhist"; - r.decision_branch = 2; - } else if ( - r.exist_remote && - r.exist_local && - r.mtime_remote !== undefined && - r.mtime_local !== undefined && - r.mtime_remote === r.mtime_local && - password === "" && - r.size_local === r.size_remote - ) { - r.decision = "skip"; - r.decision_branch = 3; - } else if ( - r.exist_remote && - r.exist_local && - r.mtime_remote !== undefined && - r.mtime_local !== undefined && - r.mtime_remote === r.mtime_local && - password === "" && - r.size_local !== r.size_remote - ) { - r.decision = "upload_clearhist"; - r.decision_branch = 4; - } else if ( - r.exist_remote && - r.exist_local && - r.mtime_remote !== undefined && - r.mtime_local !== undefined && - r.mtime_remote === r.mtime_local && - password !== "" - ) { - // if we have encryption, - // the size is always unequal - // only mtime(s) are reliable - r.decision = "skip"; - r.decision_branch = 5; - } else if (r.exist_remote && r.exist_local && r.mtime_local === undefined) { - // this must be a folder! - if (!r.key.endsWith("/")) { - throw Error(`${r.key} is not a folder but lacks local mtime`); + // 0. find anything inconsistent + if (r.existLocal && (r.mtimeLocal === undefined || r.mtimeLocal <= 0)) { + throw Error( + `Error: File ${r.key} has a last modified time <=0 or undefined in the local file system. It's abnormal and the plugin stops.` + ); + } + if (r.existRemote && (r.mtimeRemote === undefined || r.mtimeRemote <= 0)) { + throw Error( + `Error: File ${r.key} has a last modified time <=0 or undefined on the remote service. It's abnormal and the plugin stops.` + ); + } + if (r.deltimeLocal !== undefined && r.deltimeLocal <= 0) { + throw Error( + `Error: File ${r.key} has a local deletion time <=0. It's abnormal and the plugin stops.` + ); + } + if (r.deltimeRemote !== undefined && r.deltimeRemote <= 0) { + throw Error( + `Error: File ${r.key} has a remote deletion time <=0. It's abnormal and the plugin stops.` + ); + } + + // 1. mtimeLocal + if (r.existLocal) { + const mtimeRemote = r.existRemote ? r.mtimeRemote : -1; + const deltime_remote = r.deltimeRemote !== undefined ? r.deltimeRemote : -1; + const deltimeLocal = r.deltimeLocal !== undefined ? r.deltimeLocal : -1; + if ( + r.mtimeLocal >= mtimeRemote && + r.mtimeLocal >= deltimeLocal && + r.mtimeLocal >= deltime_remote + ) { + if (r.mtimeLocal === r.mtimeRemote) { + // mtime the same + if (password === "") { + // no password, we can also compare the sizes! + if (r.sizeLocal === r.sizeRemote) { + r.decision = "skipUploading"; + r.decisionBranch = 1; + } else { + r.decision = "uploadLocalToRemote"; + r.decisionBranch = 2; + } + } else { + // we have password, then the sizes are always unequal + // we can only rely on mtime + r.decision = "skipUploading"; + r.decisionBranch = 3; + } + } else { + r.decision = "uploadLocalToRemote"; + r.decisionBranch = 4; + } + keptFolder.add(getParentFolder(r.key)); + return r; } - r.decision = "skip"; - r.decision_branch = 6; - } else if ( - r.exist_remote && - !r.exist_local && - r.mtime_remote !== undefined && - r.mtime_local === undefined && - r.delete_time_local !== undefined && - r.mtime_remote >= r.delete_time_local - ) { - r.decision = "download_clearhist"; - r.decision_branch = 7; - } else if ( - r.exist_remote && - !r.exist_local && - r.mtime_remote !== undefined && - r.mtime_local === undefined && - r.delete_time_local !== undefined && - r.mtime_remote < r.delete_time_local - ) { - r.decision = "delremote_clearhist"; - r.decision_branch = 8; - } else if ( - r.exist_remote && - !r.exist_local && - r.mtime_remote !== undefined && - r.mtime_local === undefined && - r.delete_time_local == undefined - ) { - r.decision = "download"; - r.decision_branch = 9; - } else if (!r.exist_remote && r.exist_local && r.mtime_remote === undefined) { - r.decision = "upload_clearhist"; - r.decision_branch = 10; - } else if ( - !r.exist_remote && - !r.exist_local && - r.mtime_remote === undefined && - r.mtime_local === undefined - ) { - r.decision = "clearhist"; - r.decision_branch = 11; } - if (r.decision === "unknown") { - throw Error(`unknown decision for ${JSON.stringify(r)}`); + // 2. mtimeRemote + if (r.existRemote) { + const mtimeLocal = r.existLocal ? r.mtimeLocal : -1; + const deltime_remote = r.deltimeRemote !== undefined ? r.deltimeRemote : -1; + const deltimeLocal = r.deltimeLocal !== undefined ? r.deltimeLocal : -1; + if ( + r.mtimeRemote > mtimeLocal && + r.mtimeRemote >= deltimeLocal && + r.mtimeRemote >= deltime_remote + ) { + r.decision = "downloadRemoteToLocal"; + r.decisionBranch = 5; + keptFolder.add(getParentFolder(r.key)); + return r; + } } + // 3. deltimeLocal + if (r.deltimeLocal !== undefined && r.deltimeLocal !== 0) { + const mtimeLocal = r.existLocal ? r.mtimeLocal : -1; + const mtimeRemote = r.existRemote ? r.mtimeRemote : -1; + const deltime_remote = r.deltimeRemote !== undefined ? r.deltimeRemote : -1; + if ( + r.deltimeLocal >= mtimeLocal && + r.deltimeLocal >= mtimeRemote && + r.deltimeLocal >= deltime_remote + ) { + r.decision = "uploadLocalDelHistToRemote"; + r.decisionBranch = 6; + if (r.existLocal || r.existRemote) { + // actual deletion would happen + } + return r; + } + } + + // 4. deltime_remote + if (r.deltimeRemote !== undefined && r.deltimeRemote !== 0) { + const mtimeLocal = r.existLocal ? r.mtimeLocal : -1; + const mtimeRemote = r.existRemote ? r.mtimeRemote : -1; + const deltimeLocal = r.deltimeLocal !== undefined ? r.deltimeLocal : -1; + if ( + r.deltimeRemote >= mtimeLocal && + r.deltimeRemote >= mtimeRemote && + r.deltimeRemote >= deltimeLocal + ) { + r.decision = "keepRemoteDelHist"; + r.decisionBranch = 7; + if (r.existLocal || r.existRemote) { + // actual deletion would happen + } + return r; + } + } + + throw Error(`no decision for ${JSON.stringify(r)}`); +}; + +const assignOperationToFolderInplace = ( + origRecord: FileOrFolderMixedState, + keptFolder: Set, + password: string = "" +) => { + let r = origRecord; + + // files and folders are treated differently + // here we only check folders + if (!r.key.endsWith("/")) { + return r; + } + + if (!keptFolder.has(r.key)) { + // the folder does NOT have any must-be-kept children! + + if (r.deltimeLocal !== undefined || r.deltimeRemote !== undefined) { + // it has some deletion "commands" + if ( + r.deltimeLocal !== undefined && + r.deltimeLocal >= (r.deltimeRemote !== undefined ? r.deltimeRemote : -1) + ) { + r.decision = "uploadLocalDelHistToRemoteFolder"; + r.decisionBranch = 8; + } else { + r.decision = "keepRemoteDelHistFolder"; + r.decisionBranch = 9; + } + } else { + // it does not have any deletion commands + // keep it as is, and create it if necessary + keptFolder.add(getParentFolder(r.key)); + if (r.existLocal && r.existRemote) { + r.decision = "skipFolder"; + r.decisionBranch = 10; + } else if (r.existLocal || r.existRemote) { + r.decision = "createFolder"; + r.decisionBranch = 11; + } else { + throw Error( + `Error: Folder ${r.key} doesn't exist locally and remotely but is marked must be kept. Abort.` + ); + } + } + } else { + // the folder has some must be kept children! + // so itself and its parent folder must be kept + keptFolder.add(getParentFolder(r.key)); + if (r.existLocal && r.existRemote) { + r.decision = "skipFolder"; + r.decisionBranch = 12; + } else if (r.existLocal || r.existRemote) { + r.decision = "createFolder"; + r.decisionBranch = 13; + } else { + throw Error( + `Error: Folder ${r.key} doesn't exist locally and remotely but is marked must be kept. Abort.` + ); + } + } + + // save the memory, save the world! + // we have dealt with it, so we don't need it any more. + keptFolder.delete(r.key); return r; }; +const DELETION_DECISIONS: Set = new Set([ + "uploadLocalDelHistToRemote", + "keepRemoteDelHist", + "uploadLocalDelHistToRemoteFolder", + "keepRemoteDelHistFolder", +]); + export const getSyncPlan = async ( - remote: RemoteItem[], + remoteStates: FileOrFolderMixedState[], local: TAbstractFile[], - deleteHistory: FileFolderHistoryRecord[], - db: InternalDBs, - vaultRandomID: string, + remoteDeleteHistory: DeletionOnRemote[], + localDeleteHistory: FileFolderHistoryRecord[], remoteType: SUPPORTED_SERVICES_TYPE, password: string = "" ) => { const mixedStates = await ensembleMixedStates( - remote, + remoteStates, local, - deleteHistory, - db, - vaultRandomID, - remoteType, - password + remoteDeleteHistory, + localDeleteHistory ); - for (const [key, val] of Object.entries(mixedStates)) { - getOperation(val, true, password); + + const sortedKeys = Object.keys(mixedStates).sort( + (k1, k2) => k2.length - k1.length + ); + + const deletions: DeletionOnRemote[] = []; + + const keptFolder = new Set(); + for (let i = 0; i < sortedKeys.length; ++i) { + const key = sortedKeys[i]; + const val = mixedStates[key]; + + if (key.endsWith("/")) { + // decide some folders + // because the keys are sorted by length + // so all the children must have been shown up before in the iteration + assignOperationToFolderInplace(val, keptFolder, password); + } else { + // get all operations of files + // and at the same time get some helper info for folders + assignOperationToFileInplace(val, keptFolder, password); + } + + if (DELETION_DECISIONS.has(val.decision)) { + if (val.decision === "uploadLocalDelHistToRemote") { + deletions.push({ + key: key, + actionWhen: val.deltimeLocal, + }); + } else if (val.decision === "keepRemoteDelHist") { + deletions.push({ + key: key, + actionWhen: val.deltimeRemote, + }); + } else if (val.decision === "uploadLocalDelHistToRemoteFolder") { + deletions.push({ + key: key, + actionWhen: val.deltimeLocal, + }); + } else if (val.decision === "keepRemoteDelHistFolder") { + deletions.push({ + key: key, + actionWhen: val.deltimeRemote, + }); + } else { + throw Error(`do not know how to delete for decision ${val.decision}`); + } + } } + const plan = { ts: Date.now(), remoteType: remoteType, mixedStates: mixedStates, } as SyncPlanType; - return plan; + return { + plan: plan, + sortedKeys: sortedKeys, + deletions: deletions, + }; +}; + +const uploadExtraMeta = async ( + client: RemoteClient, + metadataFile: FileOrFolderMixedState | undefined, + origMetadata: MetadataOnRemote | undefined, + deletions: DeletionOnRemote[], + password: string = "" +) => { + if (deletions === undefined || deletions.length === 0) { + return; + } + + const key = DEFAULT_FILE_NAME_FOR_METADATAONREMOTE; + let remoteEncryptedKey = key; + + if (password !== "") { + if (metadataFile === undefined) { + remoteEncryptedKey = undefined; + } else { + remoteEncryptedKey = metadataFile.remoteEncryptedKey; + } + if (remoteEncryptedKey === undefined || remoteEncryptedKey === "") { + // remoteEncryptedKey = await encryptStringToBase32(key, password); + remoteEncryptedKey = await encryptStringToBase64url(key, password); + } + } + + const newMetadata: MetadataOnRemote = { + deletions: deletions, + }; + + if (isEqualMetadataOnRemote(origMetadata, newMetadata)) { + log.debug( + "metadata are the same, no need to re-generate and re-upload it." + ); + return; + } + + const resultText = serializeMetadataOnRemote(newMetadata); + + await client.uploadToRemote( + key, + undefined, + false, + password, + remoteEncryptedKey, + undefined, + true, + resultText + ); }; const dispatchOperationToActual = async ( key: string, vaultRandomID: string, - state: FileOrFolderMixedState, + r: FileOrFolderMixedState, client: RemoteClient, db: InternalDBs, vault: Vault, - password: string = "", - foldersCreatedBefore: Set | undefined = undefined + localDeleteFunc: any, + password: string = "" ) => { let remoteEncryptedKey = key; if (password !== "") { - remoteEncryptedKey = state.remote_encrypted_key; + remoteEncryptedKey = r.remoteEncryptedKey; if (remoteEncryptedKey === undefined || remoteEncryptedKey === "") { // the old version uses base32 // remoteEncryptedKey = await encryptStringToBase32(key, password); @@ -490,94 +727,114 @@ const dispatchOperationToActual = async ( } } - if ( - state.decision === undefined || - state.decision === "unknown" || - state.decision === "undecided" - ) { - throw Error(`unknown decision in ${JSON.stringify(state)}`); - } else if (state.decision === "skip") { - // do nothing - } else if ( - client.serviceType === "onedrive" && - state.size_local === 0 && - !state.key.endsWith("/") && - password === "" && - (state.decision === "upload" || state.decision === "upload_clearhist") - ) { - // TODO: it's ugly, any other way to deal with empty file for onedrive? - // do nothing, skip empty file without encryption - // if it's empty folder, or it's encrypted file/folder, it continues to be uploaded. - // this branch should be earlier than normal upload / upload_clearhist branches. - log.debug(`skip empty file ${state.key} uploading for OneDrive`); - } else if (state.decision === "download_clearhist") { + if (r.decision === undefined) { + throw Error(`unknown decision in ${JSON.stringify(r)}`); + } else if (r.decision === "skipUploading") { + // do nothing! + } else if (r.decision === "uploadLocalDelHistToRemote") { + if (r.existLocal) { + await localDeleteFunc(r.key); + } + if (r.existRemote) { + await client.deleteFromRemote(r.key, password, remoteEncryptedKey); + } + await clearDeleteRenameHistoryOfKeyAndVault(db, r.key, vaultRandomID); + } else if (r.decision === "keepRemoteDelHist") { + if (r.existLocal) { + await localDeleteFunc(r.key); + } + if (r.existRemote) { + await client.deleteFromRemote(r.key, password, remoteEncryptedKey); + } + await clearDeleteRenameHistoryOfKeyAndVault(db, r.key, vaultRandomID); + } else if (r.decision === "uploadLocalToRemote") { + if ( + client.serviceType === "onedrive" && + r.sizeLocal === 0 && + password === "" + ) { + // special treatment for empty files for OneDrive + // TODO: it's ugly, any other way? + // special treatment for OneDrive: do nothing, skip empty file without encryption + // if it's empty folder, or it's encrypted file/folder, it continues to be uploaded. + } else { + const remoteObjMeta = await client.uploadToRemote( + r.key, + vault, + false, + password, + remoteEncryptedKey + ); + await upsertSyncMetaMappingDataByVault( + client.serviceType, + db, + r.key, + r.mtimeLocal, + r.sizeLocal, + r.key, + remoteObjMeta.lastModified, + remoteObjMeta.size, + remoteObjMeta.etag, + vaultRandomID + ); + } + await clearDeleteRenameHistoryOfKeyAndVault(db, r.key, vaultRandomID); + } else if (r.decision === "downloadRemoteToLocal") { + await mkdirpInVault(r.key, vault); /* should be unnecessary */ await client.downloadFromRemote( - state.key, + r.key, vault, - state.mtime_remote, + r.mtimeRemote, password, remoteEncryptedKey ); - await clearDeleteRenameHistoryOfKeyAndVault(db, state.key, vaultRandomID); - } else if (state.decision === "upload_clearhist") { - const remoteObjMeta = await client.uploadToRemote( - state.key, - vault, - false, - password, - remoteEncryptedKey, - foldersCreatedBefore - ); - await upsertSyncMetaMappingDataByVault( - client.serviceType, - db, - state.key, - state.mtime_local, - state.size_local, - state.key, - remoteObjMeta.lastModified, - remoteObjMeta.size, - remoteObjMeta.etag, - vaultRandomID - ); - await clearDeleteRenameHistoryOfKeyAndVault(db, state.key, vaultRandomID); - } else if (state.decision === "download") { - await mkdirpInVault(state.key, vault); - await client.downloadFromRemote( - state.key, - vault, - state.mtime_remote, - password, - remoteEncryptedKey - ); - } else if (state.decision === "delremote_clearhist") { - await client.deleteFromRemote(state.key, password, remoteEncryptedKey); - await clearDeleteRenameHistoryOfKeyAndVault(db, state.key, vaultRandomID); - } else if (state.decision === "upload") { - const remoteObjMeta = await client.uploadToRemote( - state.key, - vault, - false, - password, - remoteEncryptedKey, - foldersCreatedBefore - ); - await upsertSyncMetaMappingDataByVault( - client.serviceType, - db, - state.key, - state.mtime_local, - state.size_local, - state.key, - remoteObjMeta.lastModified, - remoteObjMeta.size, - remoteObjMeta.etag, - vaultRandomID - ); - } else if (state.decision === "clearhist") { - await clearDeleteRenameHistoryOfKeyAndVault(db, state.key, vaultRandomID); + await clearDeleteRenameHistoryOfKeyAndVault(db, r.key, vaultRandomID); + } else if (r.decision === "createFolder") { + if (!r.existLocal) { + await mkdirpInVault(r.key, vault); + } + if (!r.existRemote) { + const remoteObjMeta = await client.uploadToRemote( + r.key, + vault, + false, + password, + remoteEncryptedKey + ); + await upsertSyncMetaMappingDataByVault( + client.serviceType, + db, + r.key, + r.mtimeLocal, + r.sizeLocal, + r.key, + remoteObjMeta.lastModified, + remoteObjMeta.size, + remoteObjMeta.etag, + vaultRandomID + ); + } + await clearDeleteRenameHistoryOfKeyAndVault(db, r.key, vaultRandomID); + } else if (r.decision === "uploadLocalDelHistToRemoteFolder") { + if (r.existLocal) { + await localDeleteFunc(r.key); + } + if (r.existRemote) { + await client.deleteFromRemote(r.key, password, remoteEncryptedKey); + } + await clearDeleteRenameHistoryOfKeyAndVault(db, r.key, vaultRandomID); + } else if (r.decision === "keepRemoteDelHistFolder") { + if (r.existLocal) { + await localDeleteFunc(r.key); + } + if (r.existRemote) { + await client.deleteFromRemote(r.key, password, remoteEncryptedKey); + } + await clearDeleteRenameHistoryOfKeyAndVault(db, r.key, vaultRandomID); + } else if (r.decision === "skipFolder") { + // do nothing! } else { - throw Error("this should never happen!"); + throw Error(`unknown decision in ${JSON.stringify(r)}`); } }; @@ -587,47 +844,63 @@ export const doActualSync = async ( vaultRandomID: string, vault: Vault, syncPlan: SyncPlanType, + sortedKeys: string[], + metadataFile: FileOrFolderMixedState, + origMetadata: MetadataOnRemote, + deletions: DeletionOnRemote[], + localDeleteFunc: any, password: string = "", callbackSyncProcess?: any ) => { - const keyStates = syncPlan.mixedStates; - const foldersCreatedBefore = new Set(); + const mixedStates = syncPlan.mixedStates; let i = 0; - const totalCount = Object.keys(keyStates).length || 0; - for (const [k, v] of Object.entries(keyStates).sort( - ([k1, v1], [k2, v2]) => k2.length - k1.length - )) { - i += 1; - const k2 = k as string; - const v2 = v as FileOrFolderMixedState; - log.debug(`start syncing "${k2}" with plan ${JSON.stringify(v2)}`); + const totalCount = sortedKeys.length || 0; + + log.debug(`start syncing extra data firstly`); + await uploadExtraMeta( + client, + metadataFile, + origMetadata, + deletions, + password + ); + log.debug(`finish syncing extra data firstly`); + + for (let i = 0; i < sortedKeys.length; ++i) { + const key = sortedKeys[i]; + const val = mixedStates[key]; + + log.debug(`start syncing "${key}" with plan ${JSON.stringify(val)}`); + if (callbackSyncProcess !== undefined) { - await callbackSyncProcess(i, totalCount, k2, v2.decision); + await callbackSyncProcess(i, totalCount, key, val.decision); } + await dispatchOperationToActual( - k2, + key, vaultRandomID, - v2, + val, client, db, vault, - password, - foldersCreatedBefore + localDeleteFunc, + password ); - log.info(`finished ${k2}`); + log.debug(`finished ${key}`); + + // await Promise.all( + // Object.entries(mixedStates).map(async ([k, v]) => + // dispatchOperationToActual( + // k as string, + // vaultRandomID, + // v as FileOrFolderMixedState, + // client, + // db, + // vault, + // localDeleteFunc, + // password + // ) + // ) + // ); } - // await Promise.all( - // Object.entries(keyStates) - // .map(async ([k, v]) => - // dispatchOperationToActual( - // k as string, - // v as FileOrFolderMixedState, - // client, - // db, - // vault, - // password, - // foldersCreatedBefore - // ) - // ) - // ); }; diff --git a/src/syncAlgoV2Notice.ts b/src/syncAlgoV2Notice.ts new file mode 100644 index 0000000..bc877ee --- /dev/null +++ b/src/syncAlgoV2Notice.ts @@ -0,0 +1,68 @@ +import { App, Modal, Notice, PluginSettingTab, Setting } from "obsidian"; +import type RemotelySavePlugin from "./main"; // unavoidable +import * as origLog from "loglevel"; +const log = origLog.getLogger("rs-default"); + +export class SyncAlgoV2Modal extends Modal { + agree: boolean; + readonly plugin: RemotelySavePlugin; + constructor(app: App, plugin: RemotelySavePlugin) { + super(app); + this.plugin = plugin; + this.agree = false; + } + onOpen() { + let { contentEl } = this; + contentEl.createEl("h2", { + text: "Remotely Save has a better sync algorithm", + }); + + const texts = [ + "Welcome to use Remotely Save!", + + "From this version 0.3.0, a new algorithm has been developed, but it needs uploading extra meta data files _remotely-save-metadata-on-remote.{json,bin} to YOUR configured cloud destinations, besides your notes.", + + "So that, for example, the second device can know that what files/folders have been deleted on the first device by reading those files.", + + 'If you agree, plase click the button "agree", and enjoy the plugin! AND PLEASE REMEMBER TO BACKUP YOUR VAULT FIRSTLY!', + + 'If you do not agree, you should stop using the current and later versions of Remotely Save. You could consider manually install the old version 0.2.14 which uses old algorithm and does not upload any extra meta data files. By clicking the "Do not agree" button, the plugin will unload itself, and you need to manually disable it in Obsidian settings.', + ]; + + const ul = contentEl.createEl("ul"); + + for (const t of texts) { + ul.createEl("li", { + text: t, + }); + } + + new Setting(contentEl) + .addButton((button) => { + button.setButtonText("Agree"); + button.onClick(async () => { + this.agree = true; + this.close(); + }); + }) + .addButton((button) => { + button.setButtonText("Do not agree"); + button.onClick(() => { + this.close(); + }); + }); + } + + onClose() { + let { contentEl } = this; + contentEl.empty(); + if (this.agree) { + log.info("agree to use the new algorithm"); + this.plugin.saveAgreeToUseNewSyncAlgorithm(); + this.plugin.enableAutoSyncIfSet(); + } else { + log.info("do not agree to use the new algorithm"); + this.plugin.unload(); + } + } +} diff --git a/tests/metadataOnRemote.test.ts b/tests/metadataOnRemote.test.ts new file mode 100644 index 0000000..35eb5fe --- /dev/null +++ b/tests/metadataOnRemote.test.ts @@ -0,0 +1,86 @@ +import * as chai from "chai"; +import chaiAsPromised from "chai-as-promised"; + +import { + isEqualMetadataOnRemote, + MetadataOnRemote, +} from "../src/metadataOnRemote"; + +chai.use(chaiAsPromised); +const expect = chai.expect; + +describe("Metadata operations tests", () => { + it("should compare objects deeply", async () => { + const a: MetadataOnRemote = { + deletions: [ + { key: "xxx", actionWhen: 1 }, + { key: "yyy", actionWhen: 2 }, + ], + }; + const b: MetadataOnRemote = { + deletions: [ + { key: "xxx", actionWhen: 1 }, + { key: "yyy", actionWhen: 2 }, + ], + }; + + expect(isEqualMetadataOnRemote(a, b)); + }); + + it("should find diff", async () => { + const a: MetadataOnRemote = { + deletions: [ + { key: "xxxx", actionWhen: 1 }, + { key: "yyy", actionWhen: 2 }, + ], + }; + const b: MetadataOnRemote = { + deletions: [ + { key: "xxx", actionWhen: 1 }, + { key: "yyy", actionWhen: 2 }, + ], + }; + + expect(!isEqualMetadataOnRemote(a, b)); + }); + + it("should treat undefined correctly", async () => { + const a: MetadataOnRemote = undefined; + let b: MetadataOnRemote = { + deletions: [ + { key: "xxx", actionWhen: 1 }, + { key: "yyy", actionWhen: 2 }, + ], + }; + + expect(!isEqualMetadataOnRemote(a, b)); + + b = { deletions: [] }; + expect(isEqualMetadataOnRemote(a, b)); + + b = { deletions: undefined }; + expect(isEqualMetadataOnRemote(a, b)); + + b = undefined; + expect(isEqualMetadataOnRemote(a, b)); + }); + + it("should ignore generated at fields", async () => { + const a: MetadataOnRemote = { + deletions: [ + { key: "xxxx", actionWhen: 1 }, + { key: "yyy", actionWhen: 2 }, + ], + generatedWhen: 1, + }; + const b: MetadataOnRemote = { + deletions: [ + { key: "xxx", actionWhen: 1 }, + { key: "yyy", actionWhen: 2 }, + ], + generatedWhen: 2, + }; + + expect(isEqualMetadataOnRemote(a, b)); + }); +}); diff --git a/tests/misc.test.ts b/tests/misc.test.ts index e4c892e..11caf76 100644 --- a/tests/misc.test.ts +++ b/tests/misc.test.ts @@ -68,6 +68,20 @@ describe("Misc: get folder levels", () => { expect(misc.getFolderLevels(item3)).to.deep.equal(res3); }); + it("should correctly add ending slash if required", () => { + const item = "xxx/yyy/zzz.md"; + const res = ["xxx/", "xxx/yyy/"]; + expect(misc.getFolderLevels(item, true)).to.deep.equal(res); + + const item2 = "xxx/yyy/zzz"; + const res2 = ["xxx/", "xxx/yyy/"]; + expect(misc.getFolderLevels(item2, true)).to.deep.equal(res2); + + const item3 = "xxx/yyy/zzz/"; + const res3 = ["xxx/", "xxx/yyy/", "xxx/yyy/zzz/"]; + expect(misc.getFolderLevels(item3, true)).to.deep.equal(res3); + }); + it("should treat path starting with / correctly", () => { const item = "/xxx/yyy/zzz.md"; const res = ["/xxx", "/xxx/yyy"]; @@ -91,6 +105,27 @@ describe("Misc: get folder levels", () => { }); }); +describe("Misc: get parent folder", () => { + it("should treat empty path correctly", () => { + const item = ""; + expect(misc.getParentFolder(item)).equals("/"); + }); + + it("should treat one level path correctly", () => { + let item = "abc/"; + expect(misc.getParentFolder(item)).equals("/"); + item = "/efg/"; + expect(misc.getParentFolder(item)).equals("/"); + }); + + it("should treat more levels path correctly", () => { + let item = "abc/efg"; + expect(misc.getParentFolder(item)).equals("abc/"); + item = "/hij/klm/"; + expect(misc.getParentFolder(item)).equals("/hij/"); + }); +}); + describe("Misc: vaild file name tests", () => { it("should treat no ascii correctly", async () => { const x = misc.isVaildText("😄🍎 apple 苹果"); diff --git a/versions.json b/versions.json index 50228a6..6d49518 100644 --- a/versions.json +++ b/versions.json @@ -1,3 +1,3 @@ { - "0.2.14": "0.12.15" + "0.3.0": "0.12.15" }