bump to 0.3.23

README.md

@@ -92,6 +92,8 @@ Additionally, the plugin author may occasionally visit Obsidian official forum a
   - If you are using Obsidian desktop < 0.13.25 or iOS < 1.1.1 or any Android version:
     - The webdav server has to be enabled CORS for requests from `app://obsidian.md` and `capacitor://localhost` and `http://localhost`, **AND** all webdav HTTP methods, **AND** all webdav headers. These are required, because Obsidian mobile works like a browser and mobile plugins are limited by CORS policies unless under a upgraded Obsidian version.
     - Popular software NextCloud, OwnCloud, `rclone serve webdav` do **NOT** enable CORS by default. If you are using any of them, you should evaluate the risk, and find a way to enable CORS, before using this plugin, or use a upgraded Obsidian version.
+      - **Unofficial** workaround: NextCloud users can **evaluate the risk by themselves**, and if decide to accept the risk, they can install [WebAppPassword](https://apps.nextcloud.com/apps/webapppassword) app, and add `app://obsidian.md`, `capacitor://localhost`, `http://localhost` to `Allowed origins`
+      - **Unofficial** workaround: OwnCloud users can **evaluate the risk by themselves**, and if decide to accept the risk, they can download `.tar.gz` of `WebAppPassword` above and manually install and configure it on their instances.
     - The plugin is tested successfully under python package [`wsgidav` (version 4.0)](https://github.com/mar10/wsgidav). See [this issue](https://github.com/mar10/wsgidav/issues/239) for some details.
 - Your data would be synced to a `${vaultName}` sub folder on your webdav server.
 - Password-based end-to-end encryption is also supported. But please be aware that **the vault name itself is not encrypted**.

docs/apache_cors_configure.md

@@ -0,0 +1,25 @@
+# How To Configure Apache CORS Rules for Webdav
+
+This method is [contributed by community](https://github.com/remotely-save/remotely-save/pull/31).
+
+You should evaluate the risk by yourself before trying this method.
+
+**You are strongly advised to backup your original Apache configuration files before making any changes.**
+
+```apacheconf
+<IfModule mod_headers.c>
+  # Obsidian webdav
+  SetEnvIf Origin "^app://obsidian.md$" IS_OBSIDIAN
+  SetEnvIf Origin "^capacitor://localhost$" IS_OBSIDIAN
+  SetEnvIf Origin "^http://localhost$" IS_OBSIDIAN
+  Header always set Access-Control-Allow-Origin "*" env=IS_OBSIDIAN
+  Header always set Access-Control-Allow-Methods "GET, HEAD, POST, PUT, OPTIONS, MOVE, DELETE, COPY, LOCK, UNLOCK, PROPFIND" env=IS_OBSIDIAN
+  Header always set Access-Control-Allow-Headers "Authorization, Depth, DNT, User-Agent, Keep-Alive, Content-Type, accept, origin, X-Requested-With" env=IS_OBSIDIAN
+  Header always set Access-Control-Expose-Headers "etag, dav" env=IS_OBSIDIAN
+
+  # Allow OPTION request without authentication and respond with status 200
+  RewriteCond %{ENV:IS_OBSIDIAN} 1
+  RewriteCond %{REQUEST_METHOD} OPTIONS
+  RewriteRule ^(.*)$ $1 [R=200,L]
+</IfModule>
+```

docs/browser_env_cors.md

@@ -4,6 +4,11 @@ The plugin is developed for the browser environment. The "fake" browser behind t
 
 [MDN has a doc about CORS.](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS)
 
-From Obsidian desktop >= 0.13.25 or mobile >= 1.1.1, Obsidian [provides a new API `requiestUrl`](https://forum.obsidian.md/t/obsidian-release-v0-13-25-insider-build/32701), that allows the plugin to fully bypass the CORS issue. As of Mar 2022, the latest public-released Obsidian desktop has supported this API, but the Obsidian mobile still stays in insider.
+1. From Obsidian desktop >= 0.13.25 or mobile >= 1.1.1, Obsidian [provides a new API `requiestUrl`](https://forum.obsidian.md/t/obsidian-release-v0-13-25-insider-build/32701), that allows the plugin to fully bypass the CORS issue. As of Mar 2022, the latest public-released Obsidian desktop has supported this API, but the Obsidian mobile still stays in insider.
 
-For using this plugin in Obsidian desktop < 0.13.25 or mobile < 1.1.1, we need to configure the server side to return the header `Access-Control-Allow-Origin` allowing the origins `app://obsidian.md` and `capacitor://localhost` and `http://localhost`. Here is an example [configuration for Amazon S3](./s3_cors_configure.md).
+2. For using this plugin in Obsidian desktop < 0.13.25 or mobile < 1.1.1, we need to configure the server side to return the header `Access-Control-Allow-Origin` allowing the origins `app://obsidian.md` and `capacitor://localhost` and `http://localhost`.
+
+   Example configurations:
+
+   - [Amazon S3](./s3_cors_configure.md)
+   - [Apache](./apache_cors_configure.md) ([contributed by community](https://github.com/remotely-save/remotely-save/pull/31))

docs/sync_ignoring_large_files.md

@@ -0,0 +1,17 @@
+# Sync Ignoring Large Files
+
+Initially, the plugin does not ignore large files.
+
+From the new version in May 2022, it can ignore all files with some sizes. But we need some rules to make the function compatible with existing conditions.
+
+1. If users are using E2E password mode, then the file sizes are compared on the **encrypted sizes**, rather than the original unencripted file sizes. The reasons are: the encrypted ones are in transferations, and the encrypted sizes can be computed from unencrypted sizes but not the reverse.
+
+2. Assuming the file A, is already synced between local device and remote service before.
+
+   - If the local size and remote size are both below the threshold, then the file can be synced normally.
+   - If the local size and remote size are both above the threshold, then the file will be ignored normally.
+   - If the local size is below the threshold, and the remote size is above the threshold, then the plugin **rejects** the sync, and throws the error to the user.
+   - If the local size is above the threshold, and the remote size is below the threshold, then the plugin **rejects** the sync, and throws the error to the user.
+   - When it somes to deletions, the same rules apply.
+
+   The main point is that, if the file sizes "cross the line", the plugin does not introduce any further trouble and just reject to work for this file.

esbuild.config.mjs

@@ -31,11 +31,11 @@ esbuild
       "obsidian",
       "electron",
       "fs",
-      "crypto",
       "tls",
       "net",
       // ...builtins
     ],
+    inject: ["./esbuild.injecthelper.mjs"],
     format: "cjs",
     watch: !prod,
     target: "es2016",
@@ -48,6 +48,9 @@ esbuild
       "process.env.DEFAULT_DROPBOX_APP_KEY": `"${DEFAULT_DROPBOX_APP_KEY}"`,
       "process.env.DEFAULT_ONEDRIVE_CLIENT_ID": `"${DEFAULT_ONEDRIVE_CLIENT_ID}"`,
       "process.env.DEFAULT_ONEDRIVE_AUTHORITY": `"${DEFAULT_ONEDRIVE_AUTHORITY}"`,
+      global: "window",
+      "process.env.NODE_DEBUG": `undefined`, // ugly fix
+      "process.env.DEBUG": `undefined`, // ugly fix
     },
   })
   .catch(() => process.exit(1));

esbuild.injecthelper.mjs

@@ -0,0 +1,2 @@
+export let Buffer = require("buffer").Buffer;
+export let process = require("process/browser");

manifest.json

@@ -1,7 +1,7 @@
 {
   "id": "remotely-save",
   "name": "Remotely Save",
-  "version": "0.3.18",
+  "version": "0.3.23",
   "minAppVersion": "0.13.21",
   "description": "Yet another unofficial plugin allowing users to synchronize notes between local device and the cloud service.",
   "author": "fyears",

package.json

@@ -1,6 +1,6 @@
 {
   "name": "remotely-save",
-  "version": "0.3.18",
+  "version": "0.3.23",
   "description": "This is yet another sync plugin for Obsidian app.",
   "scripts": {
     "dev2": "node esbuild.config.mjs",
@@ -23,71 +23,73 @@
   "author": "",
   "license": "Apache-2.0",
   "devDependencies": {
-    "@microsoft/microsoft-graph-types": "^2.11.0",
+    "@microsoft/microsoft-graph-types": "^2.19.0",
-    "@types/chai": "^4.2.22",
+    "@types/chai": "^4.3.1",
-    "@types/chai-as-promised": "^7.1.4",
+    "@types/chai-as-promised": "^7.1.5",
-    "@types/jsdom": "^16.2.13",
+    "@types/jsdom": "^16.2.14",
-    "@types/lodash": "^4.14.178",
+    "@types/lodash": "^4.14.182",
     "@types/mime-types": "^2.1.1",
-    "@types/mocha": "^9.0.0",
+    "@types/mocha": "^9.1.1",
     "@types/mustache": "^4.1.2",
-    "@types/node": "^14.14.37",
+    "@types/node": "^17.0.30",
-    "@types/qrcode": "^1.4.1",
+    "@types/qrcode": "^1.4.2",
     "builtin-modules": "^3.2.0",
-    "chai": "^4.3.4",
+    "chai": "^4.3.6",
     "chai-as-promised": "^7.1.1",
     "cross-env": "^7.0.3",
-    "dotenv": "^10.0.0",
+    "dotenv": "^16.0.0",
-    "esbuild": "^0.14.27",
+    "esbuild": "^0.14.38",
     "jsdom": "^19.0.0",
-    "mocha": "^9.1.3",
+    "mocha": "^9.2.2",
-    "prettier": "^2.4.1",
+    "prettier": "^2.6.2",
-    "ts-loader": "^9.2.6",
+    "ts-loader": "^9.2.9",
-    "ts-node": "^10.4.0",
+    "ts-node": "^10.7.0",
-    "tslib": "^2.2.0",
+    "tslib": "^2.4.0",
-    "typescript": "^4.4.4",
+    "typescript": "^4.6.4",
     "webdav-server": "^2.6.2",
-    "webpack": "^5.64.4",
+    "webpack": "^5.72.0",
-    "webpack-cli": "^4.9.1"
+    "webpack-cli": "^4.9.2"
   },
   "dependencies": {
-    "@aws-sdk/client-s3": "^3.67.0",
+    "@aws-sdk/client-s3": "^3.81.0",
-    "@aws-sdk/fetch-http-handler": "^3.58.0",
+    "@aws-sdk/fetch-http-handler": "^3.78.0",
-    "@aws-sdk/lib-storage": "^3.67.0",
+    "@aws-sdk/lib-storage": "^3.81.0",
-    "@aws-sdk/protocol-http": "^3.58.0",
+    "@aws-sdk/protocol-http": "^3.78.0",
-    "@aws-sdk/querystring-builder": "^3.55.0",
+    "@aws-sdk/querystring-builder": "^3.78.0",
-    "@aws-sdk/signature-v4-crt": "^3.66.0",
+    "@aws-sdk/signature-v4-crt": "^3.78.0",
-    "@aws-sdk/types": "^3.55.0",
+    "@aws-sdk/types": "^3.78.0",
-    "@azure/msal-node": "^1.6.0",
+    "@azure/msal-node": "^1.8.0",
     "@fyears/tsqueue": "^1.0.1",
-    "@microsoft/microsoft-graph-client": "^3.0.1",
+    "@microsoft/microsoft-graph-client": "^3.0.2",
-    "acorn": "^8.5.0",
+    "acorn": "^8.7.1",
     "aggregate-error": "^4.0.0",
     "assert": "^2.0.0",
-    "aws-crt": "^1.10.1",
+    "aws-crt": "^1.12.1",
     "buffer": "^6.0.3",
     "crypto-browserify": "^3.12.0",
-    "dropbox": "^10.22.0",
+    "delay": "^5.0.0",
+    "dropbox": "^10.28.0",
+    "emoji-regex": "^10.1.0",
     "http-status-codes": "^2.2.0",
     "localforage": "^1.10.0",
     "lodash": "^4.17.21",
     "loglevel": "^1.8.0",
-    "lucide": "^0.17.12",
+    "lucide": "^0.35.0",
-    "mime-types": "^2.1.33",
+    "mime-types": "^2.1.35",
     "mustache": "^4.2.0",
-    "nanoid": "^3.1.30",
+    "nanoid": "^3.3.3",
-    "obsidian": "^0.13.26",
+    "obsidian": "^0.14.6",
     "p-queue": "^7.2.0",
     "path-browserify": "^1.0.1",
     "process": "^0.11.10",
     "qrcode": "^1.5.0",
-    "rfc4648": "^1.5.0",
+    "rfc4648": "^1.5.1",
     "rimraf": "^3.0.2",
     "stream-browserify": "^3.0.0",
     "url": "^0.11.0",
     "util": "^0.12.4",
-    "webdav": "^4.7.0",
+    "webdav": "^4.9.0",
-    "webdav-fs": "^4.0.0",
+    "webdav-fs": "^4.0.1",
     "xregexp": "^5.1.0"
   }
 }

src/baseTypes.ts

@@ -85,6 +85,7 @@ export interface RemotelySavePluginSettings {
   syncUnderscoreItems?: boolean;
   lang?: LangTypeAndAuto;
   logToDB?: boolean;
+  skipSizeLargerThan?: number;
 
   /**
    * @deprecated
@@ -122,13 +123,24 @@ type DecisionTypeForFile =
   | "uploadLocalToRemote" // "skipLocal && uploadLocalToRemote && cleanLocalDelHist && cleanRemoteDelHist"
   | "downloadRemoteToLocal"; // "downloadRemoteToLocal && skipRemote && cleanLocalDelHist && cleanRemoteDelHist"
 
+type DecisionTypeForFileSize =
+  | "skipUploadingTooLarge"
+  | "skipDownloadingTooLarge"
+  | "skipUsingLocalDelTooLarge"
+  | "skipUsingRemoteDelTooLarge"
+  | "errorLocalTooLargeConflictRemote"
+  | "errorRemoteTooLargeConflictLocal";
+
 type DecisionTypeForFolder =
   | "createFolder"
   | "uploadLocalDelHistToRemoteFolder"
   | "keepRemoteDelHistFolder"
   | "skipFolder";
 
-export type DecisionType = DecisionTypeForFile | DecisionTypeForFolder;
+export type DecisionType =
+  | DecisionTypeForFile
+  | DecisionTypeForFileSize
+  | DecisionTypeForFolder;
 
 export interface FileOrFolderMixedState {
   key: string;
@@ -139,7 +151,9 @@ export interface FileOrFolderMixedState {
   deltimeLocal?: number;
   deltimeRemote?: number;
   sizeLocal?: number;
+  sizeLocalEnc?: number;
   sizeRemote?: number;
+  sizeRemoteEnc?: number;
   changeRemoteMtimeUsingMapping?: boolean;
   changeLocalMtimeUsingMapping?: boolean;
   decision?: DecisionType;
@@ -154,11 +168,16 @@ export interface FileOrFolderMixedState {
 }
 
 export const API_VER_STAT_FOLDER = "0.13.27";
-export const API_VER_REQURL = "0.13.26";
+export const API_VER_REQURL = "0.13.26"; // desktop ver 0.13.26, iOS ver 1.1.1
+export const API_VER_REQURL_ANDROID = "0.14.6"; // Android ver 1.2.1
+
 export const VALID_REQURL =
-  requireApiVersion(API_VER_REQURL) && !Platform.isAndroidApp;
+  (!Platform.isAndroidApp && requireApiVersion(API_VER_REQURL)) ||
+  (Platform.isAndroidApp && requireApiVersion(API_VER_REQURL_ANDROID));
 
 export const DEFAULT_DEBUG_FOLDER = "_debug_remotely_save/";
 export const DEFAULT_SYNC_PLANS_HISTORY_FILE_PREFIX =
   "sync_plans_hist_exported_on_";
 export const DEFAULT_LOG_HISTORY_FILE_PREFIX = "log_hist_exported_on_";
+
+export type SyncTriggerSourceType = "manual" | "auto" | "dry" | "autoOnceInit";

src/debugMode.ts

@@ -26,11 +26,13 @@ const turnSyncPlanToTable = (record: string) => {
     "remoteEncryptedKey",
     "existLocal",
     "sizeLocal",
+    "sizeLocalEnc",
     "mtimeLocal",
     "deltimeLocal",
     "changeLocalMtimeUsingMapping",
     "existRemote",
     "sizeRemote",
+    "sizeRemoteEnc",
     "mtimeRemote",
     "deltimeRemote",
     "changeRemoteMtimeUsingMapping",

src/main.ts

@@ -9,7 +9,11 @@ import {
 } from "obsidian";
 import cloneDeep from "lodash/cloneDeep";
 import { createElement, RotateCcw, RefreshCcw, FileText } from "lucide";
-import type { RemotelySavePluginSettings } from "./baseTypes";
+import type {
+  FileOrFolderMixedState,
+  RemotelySavePluginSettings,
+  SyncTriggerSourceType,
+} from "./baseTypes";
 import {
   COMMAND_CALLBACK,
   COMMAND_CALLBACK_ONEDRIVE,
@@ -61,6 +65,7 @@ import {
   exportVaultLoggerOutputToFiles,
   exportVaultSyncPlansToFiles,
 } from "./debugMode";
+import { SizesConflictModal } from "./syncSizesConflictNotice";
 
 const DEFAULT_SETTINGS: RemotelySavePluginSettings = {
   s3: DEFAULT_S3_CONFIG,
@@ -79,6 +84,7 @@ const DEFAULT_SETTINGS: RemotelySavePluginSettings = {
   syncUnderscoreItems: false,
   lang: "auto",
   logToDB: false,
+  skipSizeLargerThan: -1,
 };
 
 interface OAuth2Info {
@@ -89,8 +95,6 @@ interface OAuth2Info {
   revokeAuthSetting?: Setting;
 }
 
-type SyncTriggerSourceType = "manual" | "auto" | "dry" | "autoOnceInit";
-
 const iconNameSyncWait = `remotely-save-sync-wait`;
 const iconNameSyncRunning = `remotely-save-sync-running`;
 const iconNameLogs = `remotely-save-logs`;
@@ -279,23 +283,23 @@ export default class RemotelySavePlugin extends Plugin {
         })
       );
       this.syncStatus = "generating_plan";
-      const { plan, sortedKeys, deletions } = await getSyncPlan(
+      const { plan, sortedKeys, deletions, sizesGoWrong } = await getSyncPlan(
         remoteStates,
         local,
         localConfigDirContents,
         origMetadataOnRemote.deletions,
         localHistory,
         client.serviceType,
+        triggerSource,
         this.app.vault,
         this.settings.syncConfigDir,
         this.app.vault.configDir,
         this.settings.syncUnderscoreItems,
+        this.settings.skipSizeLargerThan,
         this.settings.password
       );
       log.info(plan.mixedStates); // for debugging
-      if (triggerSource !== "dry") {
       await insertSyncPlanRecordByVault(this.db, plan, this.vaultRandomID);
-      }
 
       // The operations above are almost read only and kind of safe.
       // The operations below begins to write or delete (!!!) something.
@@ -317,10 +321,20 @@ export default class RemotelySavePlugin extends Plugin {
           sortedKeys,
           metadataFile,
           origMetadataOnRemote,
+          sizesGoWrong,
           deletions,
           (key: string) => self.trash(key),
           this.settings.password,
           this.settings.concurrency,
+          (ss: FileOrFolderMixedState[]) => {
+            new SizesConflictModal(
+              self.app,
+              self,
+              this.settings.skipSizeLargerThan,
+              ss,
+              this.settings.password !== ""
+            ).open();
+          },
           (i: number, totalCount: number, pathName: string, decision: string) =>
             self.setCurrSyncMsg(i, totalCount, pathName, decision)
         );

src/metadataOnRemote.ts

@@ -1,6 +1,7 @@
 import isEqual from "lodash/isEqual";
 import { base64url } from "rfc4648";
 import { reverseString } from "./misc";
+import { log } from "./moreOnLog";
 
 const DEFAULT_README_FOR_METADATAONREMOTE =
   "Do NOT edit or delete the file manually. This file is for the plugin remotely-save to store some necessary meta data on the remote services. Its content is slightly obfuscated.";
@@ -69,19 +70,44 @@ export const deserializeMetadataOnRemote = (x: string | ArrayBuffer) => {
   } else {
     y1 = new TextDecoder().decode(x);
   }
-  const y2: any = JSON.parse(y1);
 
-  if (!("readme" in y2 && "d" in y2)) {
+  let y2: any;
-    throw Error("invalid remote meta data file!");
+  try {
+    y2 = JSON.parse(y1);
+  } catch (e) {
+    throw new Error(
+      `invalid remote meta data file with first few chars: ${y1.slice(0, 5)}`
+    );
   }
 
-  const y3 = JSON.parse(
+  if (!("readme" in y2 && "d" in y2)) {
-    (
+    throw new Error(
+      'invalid remote meta data file (no "readme" or "d" fields)!'
+    );
+  }
+
+  let y3: string;
+  try {
+    y3 = (
       base64url.parse(reverseString(y2["d"]), {
         out: Buffer.allocUnsafe as any,
         loose: true,
       }) as Buffer
-    ).toString("utf-8")
+    ).toString("utf-8");
-  ) as MetadataOnRemote;
+  } catch (e) {
-  return y3;
+    throw new Error('invalid remote meta data file (invalid "d" field)!');
+  }
+
+  let y4: MetadataOnRemote;
+  try {
+    y4 = JSON.parse(y3) as MetadataOnRemote;
+  } catch (e) {
+    throw new Error(
+      `invalid remote meta data file with \"d\" field with first few chars: ${y3.slice(
+        0,
+        5
+      )}`
+    );
+  }
+  return y4;
 };

src/misc.ts

@@ -3,6 +3,7 @@ import * as path from "path";
 
 import { base32, base64url } from "rfc4648";
 import XRegExp from "xregexp";
+import emojiRegex from "emoji-regex";
 
 import { log } from "./moreOnLog";
 
@@ -162,6 +163,34 @@ export const isVaildText = (a: string) => {
   );
 };
 
+/**
+ * Use regex to detect a text contains emoji or not.
+ * @param a
+ * @returns
+ */
+export const hasEmojiInText = (a: string) => {
+  const regex = emojiRegex();
+  return regex.test(a);
+};
+
+/**
+ * Convert the headers to a normal object.
+ * @param h
+ * @param toLower
+ * @returns
+ */
+export const headersToRecord = (h: Headers, toLower: boolean = true) => {
+  const res: Record<string, string> = {};
+  h.forEach((v, k) => {
+    if (toLower) {
+      res[k.toLowerCase()] = v;
+    } else {
+      res[k] = v;
+    }
+  });
+  return res;
+};
+
 /**
  * If input is already a folder, returns it as is;
  * And if input is a file, returns its direname.

src/remote.ts

@@ -15,7 +15,6 @@ import { log } from "./moreOnLog";
 
 export class RemoteClient {
   readonly serviceType: SUPPORTED_SERVICES_TYPE;
-  readonly s3Client?: s3.S3Client;
   readonly s3Config?: S3Config;
   readonly webdavClient?: webdav.WrappedWebdavClient;
   readonly webdavConfig?: WebdavConfig;
@@ -38,7 +37,6 @@ export class RemoteClient {
     // so we use a ref not copy of config here
     if (serviceType === "s3") {
       this.s3Config = s3Config;
-      this.s3Client = s3.getS3Client(this.s3Config);
     } else if (serviceType === "webdav") {
       if (vaultName === undefined || saveUpdatedConfigFunc === undefined) {
         throw Error(
@@ -86,7 +84,7 @@ export class RemoteClient {
   getRemoteMeta = async (fileOrFolderPath: string) => {
     if (this.serviceType === "s3") {
       return await s3.getRemoteMeta(
-        this.s3Client,
+        s3.getS3Client(this.s3Config),
         this.s3Config,
         fileOrFolderPath
       );
@@ -116,7 +114,7 @@ export class RemoteClient {
   ) => {
     if (this.serviceType === "s3") {
       return await s3.uploadToRemote(
-        this.s3Client,
+        s3.getS3Client(this.s3Config),
         this.s3Config,
         fileOrFolderPath,
         vault,
@@ -168,7 +166,11 @@ export class RemoteClient {
 
   listFromRemote = async (prefix?: string) => {
     if (this.serviceType === "s3") {
-      return await s3.listFromRemote(this.s3Client, this.s3Config, prefix);
+      return await s3.listFromRemote(
+        s3.getS3Client(this.s3Config),
+        this.s3Config,
+        prefix
+      );
     } else if (this.serviceType === "webdav") {
       return await webdav.listFromRemote(this.webdavClient, prefix);
     } else if (this.serviceType === "dropbox") {
@@ -190,7 +192,7 @@ export class RemoteClient {
   ) => {
     if (this.serviceType === "s3") {
       return await s3.downloadFromRemote(
-        this.s3Client,
+        s3.getS3Client(this.s3Config),
         this.s3Config,
         fileOrFolderPath,
         vault,
@@ -241,7 +243,7 @@ export class RemoteClient {
   ) => {
     if (this.serviceType === "s3") {
       return await s3.deleteFromRemote(
-        this.s3Client,
+        s3.getS3Client(this.s3Config),
         this.s3Config,
         fileOrFolderPath,
         password,
@@ -276,7 +278,7 @@ export class RemoteClient {
   checkConnectivity = async (callbackFunc?: any) => {
     if (this.serviceType === "s3") {
       return await s3.checkConnectivity(
-        this.s3Client,
+        s3.getS3Client(this.s3Config),
         this.s3Config,
         callbackFunc
       );

src/remoteForDropbox.ts

@@ -1,4 +1,6 @@
-import { Dropbox, DropboxAuth, files } from "dropbox";
+import delay from "delay";
+import { Dropbox, DropboxAuth } from "dropbox";
+import type { files, DropboxResponseError, DropboxResponse } from "dropbox";
 import { Vault } from "obsidian";
 import * as path from "path";
 import {
@@ -8,7 +10,13 @@ import {
   OAUTH2_FORCE_EXPIRE_MILLISECONDS,
 } from "./baseTypes";
 import { decryptArrayBuffer, encryptArrayBuffer } from "./encrypt";
-import { bufferToArrayBuffer, getFolderLevels, mkdirpInVault } from "./misc";
+import {
+  bufferToArrayBuffer,
+  getFolderLevels,
+  hasEmojiInText,
+  headersToRecord,
+  mkdirpInVault,
+} from "./misc";
 
 export { Dropbox } from "dropbox";
 
@@ -270,6 +278,73 @@ export const setConfigBySuccessfullAuthInplace = async (
 // Other usual common methods
 ////////////////////////////////////////////////////////////////////////////////
 
+interface ErrSubType {
+  error: {
+    retry_after: number;
+  };
+}
+
+async function retryReq<T>(
+  reqFunc: () => Promise<DropboxResponse<T>>,
+  extraHint: string = ""
+): Promise<DropboxResponse<T>> {
+  const waitSeconds = [1, 2, 4, 8]; // hard code exponential backoff
+  for (let idx = 0; idx < waitSeconds.length; ++idx) {
+    try {
+      if (idx !== 0) {
+        log.warn(
+          `${extraHint === "" ? "" : extraHint + ": "}The ${
+            idx + 1
+          }-th try starts at time ${Date.now()}`
+        );
+      }
+      return await reqFunc();
+    } catch (e: unknown) {
+      const err = e as DropboxResponseError<ErrSubType>;
+      if (err.status === undefined) {
+        // then the err is not DropboxResponseError
+        throw err;
+      }
+      if (err.status !== 429) {
+        // then the err is not "too many requests", give up
+        throw err;
+      }
+
+      if (idx === waitSeconds.length - 1) {
+        // the last retry also failed, give up
+        throw new Error(
+          `${
+            extraHint === "" ? "" : extraHint + ": "
+          }"429 too many requests", after retrying for ${
+            idx + 1
+          } times still failed.`
+        );
+      }
+
+      const headers = headersToRecord(err.headers);
+      const svrSec =
+        err.error.error.retry_after ||
+        parseInt(headers["retry-after"] || "1") ||
+        1;
+      const fallbackSec = waitSeconds[idx];
+      const secMin = Math.max(svrSec, fallbackSec);
+      const secMax = Math.max(secMin * 1.8, 2);
+      log.warn(
+        `${
+          extraHint === "" ? "" : extraHint + ": "
+        }We have "429 too many requests" error of ${
+          idx + 1
+        }-th try, at time ${Date.now()}, and wait for ${secMin} ~ ${secMax} seconds to retry. Original info: ${JSON.stringify(
+          err.error,
+          null,
+          2
+        )}`
+      );
+      await delay.range(secMin * 1000, secMax * 1000);
+    }
+  }
+}
+
 export class WrappedDropboxClient {
   dropboxConfig: DropboxConfig;
   remoteBaseDir: string;
@@ -296,9 +371,13 @@ export class WrappedDropboxClient {
       throw Error("The user has not manually auth yet.");
     }
     const currentTs = Date.now();
+    const customHeaders = {
+      "Cache-Control": "no-cache",
+    };
     if (this.dropboxConfig.accessTokenExpiresAtTime > currentTs) {
       this.dropbox = new Dropbox({
         accessToken: this.dropboxConfig.accessToken,
+        customHeaders: customHeaders,
       });
     } else {
       if (this.dropboxConfig.refreshToken === "") {
@@ -318,6 +397,7 @@ export class WrappedDropboxClient {
       );
       this.dropbox = new Dropbox({
         accessToken: this.dropboxConfig.accessToken,
+        customHeaders: customHeaders,
       });
     }
 
@@ -338,6 +418,13 @@ export class WrappedDropboxClient {
       }
       if (!this.vaultFolderExists) {
         log.info(`remote does not have folder /${this.remoteBaseDir}`);
+
+        if (hasEmojiInText(`/${this.remoteBaseDir}`)) {
+          throw new Error(
+            `/${this.remoteBaseDir}: Error: Dropbox does not support emoji in folder names.`
+          );
+        }
+
         await this.dropbox.filesCreateFolderV2({
           path: `/${this.remoteBaseDir}`,
         });
@@ -377,10 +464,12 @@ export const getRemoteMeta = async (
     // filesGetMetadata doesn't support root folder
     // we instead try to list files
     // if no error occurs, we ensemble a fake result.
-    const rsp = await client.dropbox.filesListFolder({
+    const rsp = await retryReq(() =>
+      client.dropbox.filesListFolder({
         path: `/${client.remoteBaseDir}`,
         recursive: false, // don't need to recursive here
-    });
+      })
+    );
     if (rsp.status !== 200) {
       throw Error(JSON.stringify(rsp));
     }
@@ -395,9 +484,11 @@ export const getRemoteMeta = async (
 
   const key = getDropboxPath(fileOrFolderPath, client.remoteBaseDir);
 
-  const rsp = await client.dropbox.filesGetMetadata({
+  const rsp = await retryReq(() =>
+    client.dropbox.filesGetMetadata({
       path: key,
-  });
+    })
+  );
   if (rsp.status !== 200) {
     throw Error(JSON.stringify(rsp));
   }
@@ -423,6 +514,12 @@ export const uploadToRemote = async (
   }
   uploadFile = getDropboxPath(uploadFile, client.remoteBaseDir);
 
+  if (hasEmojiInText(uploadFile)) {
+    throw new Error(
+      `${uploadFile}: Error: Dropbox does not support emoji in file / folder names.`
+    );
+  }
+
   const isFolder = fileOrFolderPath.endsWith("/");
 
   if (isFolder && isRecursively) {
@@ -438,11 +535,19 @@ export const uploadToRemote = async (
         // created, pass
       } else {
         try {
-          await client.dropbox.filesCreateFolderV2({
+          await retryReq(
+            () =>
+              client.dropbox.filesCreateFolderV2({
                 path: uploadFile,
-          });
+              }),
+            fileOrFolderPath
+          );
           foldersCreatedBefore?.add(uploadFile);
-        } catch (err) {
+        } catch (e: unknown) {
+          const err = e as DropboxResponseError<files.CreateFolderError>;
+          if (err.status === undefined) {
+            throw err;
+          }
           if (err.status === 409) {
             // pass
             foldersCreatedBefore?.add(uploadFile);
@@ -455,10 +560,14 @@ export const uploadToRemote = async (
       return res;
     } else {
       // if encrypted, upload a fake file with the encrypted file name
-      await client.dropbox.filesUpload({
+      await retryReq(
+        () =>
+          client.dropbox.filesUpload({
             path: uploadFile,
             contents: "",
-      });
+          }),
+        fileOrFolderPath
+      );
       return await getRemoteMeta(client, uploadFile);
     }
   } else {
@@ -480,13 +589,19 @@ export const uploadToRemote = async (
     }
     // in dropbox, we don't need to create folders before uploading! cool!
     // TODO: filesUploadSession for larger files (>=150 MB)
-    await client.dropbox.filesUpload({
+
+    await retryReq(
+      () =>
+        client.dropbox.filesUpload({
           path: uploadFile,
           contents: remoteContent,
           mode: {
             ".tag": "overwrite",
           },
-    });
+        }),
+      fileOrFolderPath
+    );
+
     // we want to mark that parent folders are created
     if (foldersCreatedBefore !== undefined) {
       const dirs = getFolderLevels(uploadFile).map((x) =>
@@ -554,9 +669,13 @@ const downloadFromRemoteRaw = async (
 ) => {
   await client.init();
   const key = getDropboxPath(fileOrFolderPath, client.remoteBaseDir);
-  const rsp = await client.dropbox.filesDownload({
+  const rsp = await retryReq(
+    () =>
+      client.dropbox.filesDownload({
         path: key,
-  });
+      }),
+    fileOrFolderPath
+  );
   if ((rsp.result as any).fileBlob !== undefined) {
     // we get a Blob
     const content = (rsp.result as any).fileBlob as Blob;
@@ -631,9 +750,13 @@ export const deleteFromRemote = async (
 
   await client.init();
   try {
-    await client.dropbox.filesDeleteV2({
+    await retryReq(
+      () =>
+        client.dropbox.filesDeleteV2({
           path: remoteFileName,
-    });
+        }),
+      fileOrFolderPath
+    );
   } catch (err) {
     console.error("some error while deleting");
     console.error(err);

src/remoteForOnedrive.ts

@@ -8,7 +8,7 @@ import type {
 import cloneDeep from "lodash/cloneDeep";
 import { request, requestUrl, requireApiVersion, Vault } from "obsidian";
 import {
-  API_VER_REQURL,
+  VALID_REQURL,
   COMMAND_CALLBACK_ONEDRIVE,
   DEFAULT_CONTENT_TYPE,
   OAUTH2_FORCE_EXPIRE_MILLISECONDS,
@@ -444,6 +444,7 @@ export class WrappedOnedriveClient {
         contentType: "application/json",
         headers: {
           Authorization: `Bearer ${await this.authGetter.getAccessToken()}`,
+          "Cache-Control": "no-cache",
         },
       })
     );
@@ -484,7 +485,7 @@ export class WrappedOnedriveClient {
   deleteJson = async (pathFragOrig: string) => {
     const theUrl = this.buildUrl(pathFragOrig);
     log.debug(`deleteJson, theUrl=${theUrl}`);
-    if (requireApiVersion(API_VER_REQURL)) {
+    if (VALID_REQURL) {
       await requestUrl({
         url: theUrl,
         method: "DELETE",
@@ -508,7 +509,7 @@ export class WrappedOnedriveClient {
     // TODO:
     // 20220401: On Android, requestUrl has issue that text becomes base64.
     // Use fetch everywhere instead!
-    if (false /*requireApiVersion(API_VER_REQURL)*/) {
+    if (false /*VALID_REQURL*/) {
       await requestUrl({
         url: theUrl,
         method: "PUT",
@@ -556,7 +557,7 @@ export class WrappedOnedriveClient {
     // TODO:
     // 20220401: On Android, requestUrl has issue that text becomes base64.
     // Use fetch everywhere instead!
-    if (false /*requireApiVersion(API_VER_REQURL)*/) {
+    if (false /*VALID_REQURL*/) {
       const res = await requestUrl({
         url: theUrl,
         method: "PUT",
@@ -565,6 +566,7 @@ export class WrappedOnedriveClient {
         headers: {
           // no "Content-Length" allowed here
           "Content-Range": `bytes ${rangeStart}-${rangeEnd - 1}/${size}`,
+          /* "Cache-Control": "no-cache", not allowed here!!! */
         },
       });
       return res.json as DriveItem | UploadSession;
@@ -576,6 +578,7 @@ export class WrappedOnedriveClient {
           "Content-Length": `${rangeEnd - rangeStart}`,
           "Content-Range": `bytes ${rangeStart}-${rangeEnd - 1}/${size}`,
           "Content-Type": DEFAULT_CONTENT_TYPE,
+          /* "Cache-Control": "no-cache", not allowed here!!! */
         },
       });
       return (await res.json()) as DriveItem | UploadSession;
@@ -808,11 +811,19 @@ const downloadFromRemoteRaw = async (
     `${key}?$select=@microsoft.graph.downloadUrl`
   );
   const downloadUrl: string = rsp["@microsoft.graph.downloadUrl"];
-  if (requireApiVersion(API_VER_REQURL)) {
+  if (VALID_REQURL) {
-    const content = (await requestUrl({ url: downloadUrl })).arrayBuffer;
+    const content = (
+      await requestUrl({
+        url: downloadUrl,
+        headers: { "Cache-Control": "no-cache" },
+      })
+    ).arrayBuffer;
     return content;
   } else {
-    const content = await (await fetch(downloadUrl)).arrayBuffer();
+    const content = await // cannot set no-cache here, will have cors error
+    (
+      await fetch(downloadUrl)
+    ).arrayBuffer();
     return content;
   }
 };

src/remoteForS3.ts

@@ -24,6 +24,7 @@ import { Buffer } from "buffer";
 import * as mime from "mime-types";
 import { Vault, requestUrl, RequestUrlParam } from "obsidian";
 import { Readable } from "stream";
+import AggregateError from "aggregate-error";
 import {
   DEFAULT_CONTENT_TYPE,
   RemoteItem,
@@ -83,11 +84,14 @@ class ObsHttpHandler extends FetchHttpHandler {
     const body =
       method === "GET" || method === "HEAD" ? undefined : request.body;
 
-    const transformedHeaders = { ...request.headers };
+    const transformedHeaders: Record<string, string> = {};
-    delete transformedHeaders["host"];
+    for (const key of Object.keys(request.headers)) {
-    delete transformedHeaders["Host"];
+      const keyLower = key.toLowerCase();
-    delete transformedHeaders["content-length"];
+      if (keyLower === "host" || keyLower === "content-length") {
-    delete transformedHeaders["Content-Length"];
+        continue;
+      }
+      transformedHeaders[keyLower] = request.headers[key];
+    }
 
     let contentType: string = undefined;
     if (transformedHeaders["content-type"] !== undefined) {
@@ -192,8 +196,10 @@ export const getS3Client = (s3Config: S3Config) => {
     endpoint = `https://${endpoint}`;
   }
 
+  let s3Client: S3Client;
+
   if (VALID_REQURL && s3Config.bypassCorsLocally) {
-    const s3Client = new S3Client({
+    s3Client = new S3Client({
       region: s3Config.s3Region,
       endpoint: endpoint,
       forcePathStyle: s3Config.forcePathStyle,
@@ -203,9 +209,8 @@ export const getS3Client = (s3Config: S3Config) => {
       },
       requestHandler: new ObsHttpHandler(),
     });
-    return s3Client;
   } else {
-    const s3Client = new S3Client({
+    s3Client = new S3Client({
       region: s3Config.s3Region,
       endpoint: endpoint,
       forcePathStyle: s3Config.forcePathStyle,
@@ -214,8 +219,19 @@ export const getS3Client = (s3Config: S3Config) => {
         secretAccessKey: s3Config.s3SecretAccessKey,
       },
     });
-    return s3Client;
   }
+
+  s3Client.middlewareStack.add(
+    (next, context) => (args) => {
+      (args.request as any).headers["cache-control"] = "no-cache";
+      return next(args);
+    },
+    {
+      step: "build",
+    }
+  );
+
+  return s3Client;
 };
 
 export const getRemoteMeta = async (
@@ -522,8 +538,19 @@ export const checkConnectivity = async (
   } catch (err) {
     log.debug(err);
     if (callbackFunc !== undefined) {
+      if (s3Config.s3Endpoint.contains(s3Config.s3BucketName)) {
+        const err2 = new AggregateError([
+          err,
+          new Error(
+            "Maybe you've included the bucket name inside the endpoint setting. Please remove the bucket name and try again."
+          ),
+        ]);
+        callbackFunc(err2);
+      } else {
         callbackFunc(err);
       }
+    }
+
     return false;
   }
 };

src/remoteForWebdav.ts

@@ -183,6 +183,9 @@ export class WrappedWebdavClient {
 
   init = async () => {
     // init client if not inited
+    const headers = {
+      "Cache-Control": "no-cache",
+    };
     if (this.client === undefined) {
       if (
         this.webdavConfig.username !== "" &&
@@ -191,6 +194,7 @@ export class WrappedWebdavClient {
         this.client = createClient(this.webdavConfig.address, {
           username: this.webdavConfig.username,
           password: this.webdavConfig.password,
+          headers: headers,
           authType:
             this.webdavConfig.authType === "digest"
               ? AuthType.Digest
@@ -198,7 +202,9 @@ export class WrappedWebdavClient {
         });
       } else {
         log.info("no password");
-        this.client = createClient(this.webdavConfig.address);
+        this.client = createClient(this.webdavConfig.address, {
+          headers: headers,
+        });
       }
     }
 

src/settings.ts

@@ -7,6 +7,8 @@ import {
   Platform,
   requireApiVersion,
 } from "obsidian";
+import type { TextComponent } from "obsidian";
+import { createElement, Eye, EyeOff } from "lucide";
 import {
   API_VER_REQURL,
   DEFAULT_DEBUG_FOLDER,
@@ -627,8 +629,34 @@ class ExportSettingsQrCodeModal extends Modal {
   }
 }
 
+const getEyesElements = () => {
+  const eyeEl = createElement(Eye);
+  const eyeOffEl = createElement(EyeOff);
+  return {
+    eye: eyeEl.outerHTML,
+    eyeOff: eyeOffEl.outerHTML,
+  };
+};
+
+const wrapTextWithPasswordHide = (text: TextComponent) => {
+  const { eye, eyeOff } = getEyesElements();
+  const hider = text.inputEl.insertAdjacentElement("afterend", createSpan());
+  // the init type of hider is "hidden" === eyeOff === password
+  hider.innerHTML = eyeOff;
+  hider.addEventListener("click", (e) => {
+    const isText = text.inputEl.getAttribute("type") === "text";
+    hider.innerHTML = isText ? eyeOff : eye;
+    text.inputEl.setAttribute("type", isText ? "password" : "text");
+    text.inputEl.focus();
+  });
+
+  // the init type of text el is password
+  text.inputEl.setAttribute("type", "password");
+  return text;
+};
+
 export class RemotelySaveSettingTab extends PluginSettingTab {
-  plugin: RemotelySavePlugin;
+  readonly plugin: RemotelySavePlugin;
 
   constructor(app: App, plugin: RemotelySavePlugin) {
     super(app, plugin);
@@ -658,14 +686,15 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
     new Setting(passwordDiv)
       .setName(t("settings_password"))
       .setDesc(t("settings_password_desc"))
-      .addText((text) =>
+      .addText((text) => {
+        wrapTextWithPasswordHide(text);
         text
           .setPlaceholder("")
           .setValue(`${this.plugin.settings.password}`)
           .onChange(async (value) => {
             newPassword = value.trim();
+          });
       })
-      )
       .addButton(async (button) => {
         button.setButtonText(t("confirm"));
         button.onClick(async () => {
@@ -717,6 +746,10 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
       .setDesc(t("settings_runoncestartup_desc"))
       .addDropdown((dropdown) => {
         dropdown.addOption("-1", t("settings_runoncestartup_notset"));
+        dropdown.addOption(
+          `${1000 * 1 * 1}`,
+          t("settings_runoncestartup_1sec")
+        );
         dropdown.addOption(
           `${1000 * 10 * 1}`,
           t("settings_runoncestartup_10sec")
@@ -734,6 +767,25 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
           });
       });
 
+    const skipLargeFilesDiv = generalDiv.createEl("div");
+    new Setting(skipLargeFilesDiv)
+      .setName(t("settings_skiplargefiles"))
+      .setDesc(t("settings_skiplargefiles_desc"))
+      .addDropdown((dropdown) => {
+        dropdown.addOption("-1", t("settings_skiplargefiles_notset"));
+
+        const mbs = [1, 5, 10, 50, 100, 500, 1000];
+        for (const mb of mbs) {
+          dropdown.addOption(`${mb * 1000 * 1000}`, `${mb} MB`);
+        }
+        dropdown
+          .setValue(`${this.plugin.settings.skipSizeLargerThan}`)
+          .onChange(async (val) => {
+            this.plugin.settings.skipSizeLargerThan = parseInt(val);
+            await this.plugin.saveSettings();
+          });
+      });
+
     //////////////////////////////////////////////////
     // below for general chooser (part 1/2)
     //////////////////////////////////////////////////
@@ -816,29 +868,31 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
 
     new Setting(s3Div)
       .setName(t("settings_s3_accesskeyid"))
-      .setDesc(t("settings_s3_accesskeyid"))
+      .setDesc(t("settings_s3_accesskeyid_desc"))
-      .addText((text) =>
+      .addText((text) => {
+        wrapTextWithPasswordHide(text);
         text
           .setPlaceholder("")
           .setValue(`${this.plugin.settings.s3.s3AccessKeyID}`)
           .onChange(async (value) => {
             this.plugin.settings.s3.s3AccessKeyID = value.trim();
             await this.plugin.saveSettings();
-          })
+          });
-      );
+      });
 
     new Setting(s3Div)
       .setName(t("settings_s3_secretaccesskey"))
-      .setDesc(t("settings_s3_secretaccesskey"))
+      .setDesc(t("settings_s3_secretaccesskey_desc"))
-      .addText((text) =>
+      .addText((text) => {
+        wrapTextWithPasswordHide(text);
         text
           .setPlaceholder("")
           .setValue(`${this.plugin.settings.s3.s3SecretAccessKey}`)
           .onChange(async (value) => {
             this.plugin.settings.s3.s3SecretAccessKey = value.trim();
             await this.plugin.saveSettings();
-          })
+          });
-      );
+      });
 
     new Setting(s3Div)
       .setName(t("settings_s3_bucketname"))
@@ -877,11 +931,7 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
     if (VALID_REQURL) {
       new Setting(s3Div)
         .setName(t("settings_s3_bypasscorslocally"))
-        .setDesc(
+        .setDesc(t("settings_s3_bypasscorslocally_desc"))
-          t("settings_s3_bypasscorslocally_desc", {
-            ver: API_VER_REQURL,
-          })
-        )
         .addDropdown((dropdown) => {
           dropdown
             .addOption("disable", t("disable"))
@@ -1297,15 +1347,9 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
     });
 
     if (!VALID_REQURL) {
-      if (Platform.isAndroidApp) {
       webdavDiv.createEl("p", {
-          text: t("settings_webdav_cors_android"),
+        text: t("settings_webdav_cors_os"),
       });
-      } else {
-        webdavDiv.createEl("p", {
-          text: t("settings_webdav_cors_otheros"),
-        });
-      }
 
       webdavDiv.createEl("p", {
         text: t("settings_webdav_cors"),
@@ -1346,7 +1390,8 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
     new Setting(webdavDiv)
       .setName(t("settings_webdav_user"))
       .setDesc(t("settings_webdav_user_desc"))
-      .addText((text) =>
+      .addText((text) => {
+        wrapTextWithPasswordHide(text);
         text
           .setPlaceholder("")
           .setValue(this.plugin.settings.webdav.username)
@@ -1359,13 +1404,14 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
               this.plugin.settings.webdav.depth = "auto_unknown";
             }
             await this.plugin.saveSettings();
-          })
+          });
-      );
+      });
 
     new Setting(webdavDiv)
       .setName(t("settings_webdav_password"))
       .setDesc(t("settings_webdav_password_desc"))
-      .addText((text) =>
+      .addText((text) => {
+        wrapTextWithPasswordHide(text);
         text
           .setPlaceholder("")
           .setValue(this.plugin.settings.webdav.password)
@@ -1378,8 +1424,8 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
               this.plugin.settings.webdav.depth = "auto_unknown";
             }
             await this.plugin.saveSettings();
-          })
+          });
-      );
+      });
 
     new Setting(webdavDiv)
       .setName(t("settings_webdav_auth"))

src/sync.ts

@@ -7,17 +7,19 @@ import {
 } from "obsidian";
 import AggregateError from "aggregate-error";
 import PQueue from "p-queue";
-import {
+import type {
   RemoteItem,
-  SUPPORTED_SERVICES_TYPE,
+  SyncTriggerSourceType,
   DecisionType,
   FileOrFolderMixedState,
-  API_VER_STAT_FOLDER,
+  SUPPORTED_SERVICES_TYPE,
 } from "./baseTypes";
+import { API_VER_STAT_FOLDER } from "./baseTypes";
 import {
   decryptBase32ToString,
   decryptBase64urlToString,
   encryptStringToBase64url,
+  getSizeFromOrigToEnc,
   MAGIC_ENCRYPTED_PREFIX_BASE32,
   MAGIC_ENCRYPTED_PREFIX_BASE64URL,
 } from "./encrypt";
@@ -65,6 +67,7 @@ export type SyncStatusType =
 export interface SyncPlanType {
   ts: number;
   tsFmt?: string;
+  syncTriggerSource?: SyncTriggerSourceType;
   remoteType: SUPPORTED_SERVICES_TYPE;
   mixedStates: Record<string, FileOrFolderMixedState>;
 }
@@ -215,22 +218,29 @@ export const parseRemoteItems = async (
     if (backwardMapping !== undefined) {
       key = backwardMapping.localKey;
       const mtimeRemote = backwardMapping.localMtime || entry.lastModified;
+
+      // the backwardMapping.localSize is the file BEFORE encryption
+      // we want to split two sizes for comparation later
+
       r = {
         key: key,
         existRemote: true,
         mtimeRemote: mtimeRemote,
         mtimeRemoteFmt: unixTimeToStr(mtimeRemote),
-        sizeRemote: backwardMapping.localSize || entry.size,
+        sizeRemote: backwardMapping.localSize,
+        sizeRemoteEnc: password === "" ? undefined : entry.size,
         remoteEncryptedKey: remoteEncryptedKey,
         changeRemoteMtimeUsingMapping: true,
       };
     } else {
+      // do not have backwardMapping
       r = {
         key: key,
         existRemote: true,
         mtimeRemote: entry.lastModified,
         mtimeRemoteFmt: unixTimeToStr(entry.lastModified),
-        sizeRemote: entry.size,
+        sizeRemote: password === "" ? entry.size : undefined,
+        sizeRemoteEnc: password === "" ? undefined : entry.size,
         remoteEncryptedKey: remoteEncryptedKey,
         changeRemoteMtimeUsingMapping: false,
       };
@@ -303,7 +313,8 @@ const ensembleMixedStates = async (
   localFileHistory: FileFolderHistoryRecord[],
   syncConfigDir: boolean,
   configDir: string,
-  syncUnderscoreItems: boolean
+  syncUnderscoreItems: boolean,
+  password: string
 ) => {
   const results = {} as Record<string, FileOrFolderMixedState>;
 
@@ -332,6 +343,8 @@ const ensembleMixedStates = async (
         mtimeLocal: mtimeLocal,
         mtimeLocalFmt: unixTimeToStr(mtimeLocal),
         sizeLocal: entry.stat.size,
+        sizeLocalEnc:
+          password === "" ? undefined : getSizeFromOrigToEnc(entry.stat.size),
       };
     } else if (entry instanceof TFolder) {
       key = `${entry.path}/`;
@@ -341,6 +354,7 @@ const ensembleMixedStates = async (
         mtimeLocal: undefined,
         mtimeLocalFmt: undefined,
         sizeLocal: 0,
+        sizeLocalEnc: password === "" ? undefined : getSizeFromOrigToEnc(0),
       };
     } else {
       throw Error(`unexpected ${entry}`);
@@ -356,6 +370,7 @@ const ensembleMixedStates = async (
       results[key].mtimeLocal = r.mtimeLocal;
       results[key].mtimeLocalFmt = r.mtimeLocalFmt;
       results[key].sizeLocal = r.sizeLocal;
+      results[key].sizeLocalEnc = r.sizeLocalEnc;
     } else {
       results[key] = r;
       results[key].existRemote = false;
@@ -372,6 +387,8 @@ const ensembleMixedStates = async (
         mtimeLocal: mtimeLocal,
         mtimeLocalFmt: unixTimeToStr(mtimeLocal),
         sizeLocal: entry.size,
+        sizeLocalEnc:
+          password === "" ? undefined : getSizeFromOrigToEnc(entry.size),
       };
 
       if (results.hasOwnProperty(key)) {
@@ -380,6 +397,7 @@ const ensembleMixedStates = async (
         results[key].mtimeLocal = r.mtimeLocal;
         results[key].mtimeLocalFmt = r.mtimeLocalFmt;
         results[key].sizeLocal = r.sizeLocal;
+        results[key].sizeLocalEnc = r.sizeLocalEnc;
       } else {
         results[key] = r;
         results[key].existRemote = false;
@@ -459,9 +477,15 @@ const ensembleMixedStates = async (
         results[key].changeLocalMtimeUsingMapping =
           r.changeLocalMtimeUsingMapping;
       } else {
-        results[key] = r;
+        // So, the file doesn't exist,
-        results[key].existLocal = false; // we have already checked local
+        // except that it existed in the "renamed to" history records.
-        results[key].existRemote = false; // we have already checked remote
+        // Most likely because that the user deleted the file while Obsidian was closed,
+        // so Obsidian could not track the deletions.
+        // We are not sure how to deal with this, so do not generate anything here!
+        // // // The following 3 lines are of old logic, and have been removed:
+        // // results[key] = r;
+        // // results[key].existLocal = false; // we have already checked local
+        // // results[key].existRemote = false; // we have already checked remote
       }
     } else {
       throw Error(
@@ -476,6 +500,7 @@ const ensembleMixedStates = async (
 const assignOperationToFileInplace = (
   origRecord: FileOrFolderMixedState,
   keptFolder: Set<string>,
+  skipSizeLargerThan: number,
   password: string = ""
 ) => {
   let r = origRecord;
@@ -518,6 +543,18 @@ const assignOperationToFileInplace = (
     );
   }
 
+  if (
+    (r.existLocal && password !== "" && r.sizeLocalEnc === undefined) ||
+    (r.existRemote && password !== "" && r.sizeRemoteEnc === undefined)
+  ) {
+    throw new Error(
+      `Error: No encryption sizes: ${JSON.stringify(r, null, 2)}`
+    );
+  }
+
+  const sizeLocalComp = password === "" ? r.sizeLocal : r.sizeLocalEnc;
+  const sizeRemoteComp = password === "" ? r.sizeRemote : r.sizeRemoteEnc;
+
   // 1. mtimeLocal
   if (r.existLocal) {
     const mtimeRemote = r.existRemote ? r.mtimeRemote : -1;
@@ -528,26 +565,79 @@ const assignOperationToFileInplace = (
       r.mtimeLocal >= deltimeLocal &&
       r.mtimeLocal >= deltimeRemote
     ) {
+      if (sizeLocalComp === undefined) {
+        throw new Error(
+          `Error: no local size but has local mtime: ${JSON.stringify(
+            r,
+            null,
+            2
+          )}`
+        );
+      }
       if (r.mtimeLocal === r.mtimeRemote) {
-        // mtime the same
+        // local and remote both exist and mtimes are the same
-        if (password === "") {
+        if (sizeLocalComp === sizeRemoteComp) {
-          // no password, we can also compare the sizes!
+          // do not need to consider skipSizeLargerThan in this case
-          if (r.sizeLocal === r.sizeRemote) {
           r.decision = "skipUploading";
           r.decisionBranch = 1;
         } else {
+          if (skipSizeLargerThan <= 0) {
             r.decision = "uploadLocalToRemote";
             r.decisionBranch = 2;
+          } else {
+            // limit the sizes
+            if (sizeLocalComp <= skipSizeLargerThan) {
+              if (sizeRemoteComp <= skipSizeLargerThan) {
+                r.decision = "uploadLocalToRemote";
+                r.decisionBranch = 18;
+              } else {
+                r.decision = "errorRemoteTooLargeConflictLocal";
+                r.decisionBranch = 19;
               }
             } else {
-          // we have password, then the sizes are always unequal
+              if (sizeRemoteComp <= skipSizeLargerThan) {
-          // we can only rely on mtime
+                r.decision = "errorLocalTooLargeConflictRemote";
-          r.decision = "skipUploading";
+                r.decisionBranch = 20;
-          r.decisionBranch = 3;
+              } else {
+                r.decision = "skipUploadingTooLarge";
+                r.decisionBranch = 21;
+              }
+            }
+          }
         }
       } else {
+        // we have local laregest mtime,
+        // and the remote not existing or smaller mtime
+        if (skipSizeLargerThan <= 0) {
+          // no need to consider sizes
           r.decision = "uploadLocalToRemote";
           r.decisionBranch = 4;
+        } else {
+          // need to consider sizes
+          if (sizeLocalComp <= skipSizeLargerThan) {
+            if (sizeRemoteComp === undefined) {
+              r.decision = "uploadLocalToRemote";
+              r.decisionBranch = 22;
+            } else if (sizeRemoteComp <= skipSizeLargerThan) {
+              r.decision = "uploadLocalToRemote";
+              r.decisionBranch = 23;
+            } else {
+              r.decision = "errorRemoteTooLargeConflictLocal";
+              r.decisionBranch = 24;
+            }
+          } else {
+            if (sizeRemoteComp === undefined) {
+              r.decision = "skipUploadingTooLarge";
+              r.decisionBranch = 25;
+            } else if (sizeRemoteComp <= skipSizeLargerThan) {
+              r.decision = "errorLocalTooLargeConflictRemote";
+              r.decisionBranch = 26;
+            } else {
+              r.decision = "skipUploadingTooLarge";
+              r.decisionBranch = 27;
+            }
+          }
+        }
       }
       keptFolder.add(getParentFolder(r.key));
       return r;
@@ -564,8 +654,49 @@ const assignOperationToFileInplace = (
       r.mtimeRemote >= deltimeLocal &&
       r.mtimeRemote >= deltimeRemote
     ) {
+      // we have remote laregest mtime,
+      // and the local not existing or smaller mtime
+      if (sizeRemoteComp === undefined) {
+        throw new Error(
+          `Error: no remote size but has remote mtime: ${JSON.stringify(
+            r,
+            null,
+            2
+          )}`
+        );
+      }
+
+      if (skipSizeLargerThan <= 0) {
+        // no need to consider sizes
         r.decision = "downloadRemoteToLocal";
         r.decisionBranch = 5;
+      } else {
+        // need to consider sizes
+        if (sizeRemoteComp <= skipSizeLargerThan) {
+          if (sizeLocalComp === undefined) {
+            r.decision = "downloadRemoteToLocal";
+            r.decisionBranch = 28;
+          } else if (sizeLocalComp <= skipSizeLargerThan) {
+            r.decision = "downloadRemoteToLocal";
+            r.decisionBranch = 29;
+          } else {
+            r.decision = "errorLocalTooLargeConflictRemote";
+            r.decisionBranch = 30;
+          }
+        } else {
+          if (sizeLocalComp === undefined) {
+            r.decision = "skipDownloadingTooLarge";
+            r.decisionBranch = 31;
+          } else if (sizeLocalComp <= skipSizeLargerThan) {
+            r.decision = "errorRemoteTooLargeConflictLocal";
+            r.decisionBranch = 32;
+          } else {
+            r.decision = "skipDownloadingTooLarge";
+            r.decisionBranch = 33;
+          }
+        }
+      }
+
       keptFolder.add(getParentFolder(r.key));
       return r;
     }
@@ -581,11 +712,45 @@ const assignOperationToFileInplace = (
       r.deltimeLocal >= mtimeRemote &&
       r.deltimeLocal >= deltimeRemote
     ) {
+      if (skipSizeLargerThan <= 0) {
         r.decision = "uploadLocalDelHistToRemote";
         r.decisionBranch = 6;
         if (r.existLocal || r.existRemote) {
           // actual deletion would happen
         }
+      } else {
+        const localTooLargeToDelete =
+          r.existLocal && sizeLocalComp > skipSizeLargerThan;
+        const remoteTooLargeToDelete =
+          r.existRemote && sizeRemoteComp > skipSizeLargerThan;
+        if (localTooLargeToDelete) {
+          if (remoteTooLargeToDelete) {
+            r.decision = "skipUsingLocalDelTooLarge";
+            r.decisionBranch = 34;
+          } else {
+            if (r.existRemote) {
+              r.decision = "errorLocalTooLargeConflictRemote";
+              r.decisionBranch = 35;
+            } else {
+              r.decision = "skipUsingLocalDelTooLarge";
+              r.decisionBranch = 36;
+            }
+          }
+        } else {
+          if (remoteTooLargeToDelete) {
+            if (r.existLocal) {
+              r.decision = "errorLocalTooLargeConflictRemote";
+              r.decisionBranch = 37;
+            } else {
+              r.decision = "skipUsingLocalDelTooLarge";
+              r.decisionBranch = 38;
+            }
+          } else {
+            r.decision = "uploadLocalDelHistToRemote";
+            r.decisionBranch = 39;
+          }
+        }
+      }
       return r;
     }
   }
@@ -600,11 +765,45 @@ const assignOperationToFileInplace = (
       r.deltimeRemote >= mtimeRemote &&
       r.deltimeRemote >= deltimeLocal
     ) {
+      if (skipSizeLargerThan <= 0) {
         r.decision = "keepRemoteDelHist";
         r.decisionBranch = 7;
         if (r.existLocal || r.existRemote) {
           // actual deletion would happen
         }
+      } else {
+        const localTooLargeToDelete =
+          r.existLocal && sizeLocalComp > skipSizeLargerThan;
+        const remoteTooLargeToDelete =
+          r.existRemote && sizeRemoteComp > skipSizeLargerThan;
+        if (localTooLargeToDelete) {
+          if (remoteTooLargeToDelete) {
+            r.decision = "skipUsingRemoteDelTooLarge";
+            r.decisionBranch = 40;
+          } else {
+            if (r.existRemote) {
+              r.decision = "errorLocalTooLargeConflictRemote";
+              r.decisionBranch = 41;
+            } else {
+              r.decision = "skipUsingRemoteDelTooLarge";
+              r.decisionBranch = 42;
+            }
+          }
+        } else {
+          if (remoteTooLargeToDelete) {
+            if (r.existLocal) {
+              r.decision = "errorLocalTooLargeConflictRemote";
+              r.decisionBranch = 43;
+            } else {
+              r.decision = "skipUsingRemoteDelTooLarge";
+              r.decisionBranch = 44;
+            }
+          } else {
+            r.decision = "keepRemoteDelHist";
+            r.decisionBranch = 45;
+          }
+        }
+      }
       return r;
     }
   }
@@ -738,6 +937,10 @@ const DELETION_DECISIONS: Set<DecisionType> = new Set([
   "uploadLocalDelHistToRemoteFolder",
   "keepRemoteDelHistFolder",
 ]);
+const SIZES_GO_WRONG_DECISIONS: Set<DecisionType> = new Set([
+  "errorLocalTooLargeConflictRemote",
+  "errorRemoteTooLargeConflictLocal",
+]);
 
 export const getSyncPlan = async (
   remoteStates: FileOrFolderMixedState[],
@@ -746,10 +949,12 @@ export const getSyncPlan = async (
   remoteDeleteHistory: DeletionOnRemote[],
   localFileHistory: FileFolderHistoryRecord[],
   remoteType: SUPPORTED_SERVICES_TYPE,
+  triggerSource: SyncTriggerSourceType,
   vault: Vault,
   syncConfigDir: boolean,
   configDir: string,
   syncUnderscoreItems: boolean,
+  skipSizeLargerThan: number,
   password: string = ""
 ) => {
   const mixedStates = await ensembleMixedStates(
@@ -760,13 +965,15 @@ export const getSyncPlan = async (
     localFileHistory,
     syncConfigDir,
     configDir,
-    syncUnderscoreItems
+    syncUnderscoreItems,
+    password
   );
 
   const sortedKeys = Object.keys(mixedStates).sort(
     (k1, k2) => k2.length - k1.length
   );
 
+  const sizesGoWrong: FileOrFolderMixedState[] = [];
   const deletions: DeletionOnRemote[] = [];
 
   const keptFolder = new Set<string>();
@@ -782,7 +989,16 @@ export const getSyncPlan = async (
     } else {
       // get all operations of files
       // and at the same time get some helper info for folders
-      assignOperationToFileInplace(val, keptFolder, password);
+      assignOperationToFileInplace(
+        val,
+        keptFolder,
+        skipSizeLargerThan,
+        password
+      );
+    }
+
+    if (SIZES_GO_WRONG_DECISIONS.has(val.decision)) {
+      sizesGoWrong.push(val);
     }
 
     if (DELETION_DECISIONS.has(val.decision)) {
@@ -818,12 +1034,14 @@ export const getSyncPlan = async (
     ts: currTs,
     tsFmt: currTsFmt,
     remoteType: remoteType,
+    syncTriggerSource: triggerSource,
     mixedStates: mixedStates,
   } as SyncPlanType;
   return {
     plan: plan,
     sortedKeys: sortedKeys,
     deletions: deletions,
+    sizesGoWrong: sizesGoWrong,
   };
 };
 
@@ -1005,6 +1223,14 @@ const dispatchOperationToActual = async (
     await clearDeleteRenameHistoryOfKeyAndVault(db, r.key, vaultRandomID);
   } else if (r.decision === "skipFolder") {
     // do nothing!
+  } else if (r.decision === "skipUploadingTooLarge") {
+    // do nothing!
+  } else if (r.decision === "skipDownloadingTooLarge") {
+    // do nothing!
+  } else if (r.decision === "skipUsingLocalDelTooLarge") {
+    // do nothing!
+  } else if (r.decision === "skipUsingRemoteDelTooLarge") {
+    // do nothing!
   } else {
     throw Error(`unknown decision in ${JSON.stringify(r)}`);
   }
@@ -1023,7 +1249,14 @@ const splitThreeSteps = (syncPlan: SyncPlanType, sortedKeys: string[]) => {
     const key = sortedKeys[i];
     const val: FileOrFolderMixedState = Object.assign({}, mixedStates[key]); // copy to avoid issue
 
-    if (val.decision === "skipFolder" || val.decision === "skipUploading") {
+    if (
+      val.decision === "skipFolder" ||
+      val.decision === "skipUploading" ||
+      val.decision === "skipDownloadingTooLarge" ||
+      val.decision === "skipUploadingTooLarge" ||
+      val.decision === "skipUsingLocalDelTooLarge" ||
+      val.decision === "skipUsingRemoteDelTooLarge"
+    ) {
       // pass
     } else if (val.decision === "createFolder") {
       const level = atWhichLevel(key);
@@ -1083,15 +1316,23 @@ export const doActualSync = async (
   sortedKeys: string[],
   metadataFile: FileOrFolderMixedState,
   origMetadata: MetadataOnRemote,
+  sizesGoWrong: FileOrFolderMixedState[],
   deletions: DeletionOnRemote[],
   localDeleteFunc: any,
   password: string = "",
   concurrency: number = 1,
+  callbackSizesGoWrong?: any,
   callbackSyncProcess?: any
 ) => {
   const mixedStates = syncPlan.mixedStates;
   const totalCount = sortedKeys.length || 0;
 
+  if (sizesGoWrong.length > 0) {
+    log.debug(`some sizes are larger than the threshold, abort and show hints`);
+    callbackSizesGoWrong(sizesGoWrong);
+    return;
+  }
+
   log.debug(`start syncing extra data firstly`);
   await uploadExtraMeta(
     client,

src/syncSizesConflictNotice.ts

@@ -0,0 +1,90 @@
+import { App, Modal, Notice, PluginSettingTab, Setting } from "obsidian";
+import type RemotelySavePlugin from "./main"; // unavoidable
+import type { TransItemType } from "./i18n";
+import type { FileOrFolderMixedState } from "./baseTypes";
+
+import { log } from "./moreOnLog";
+
+export class SizesConflictModal extends Modal {
+  readonly plugin: RemotelySavePlugin;
+  readonly skipSizeLargerThan: number;
+  readonly sizesGoWrong: FileOrFolderMixedState[];
+  readonly hasPassword: boolean;
+  constructor(
+    app: App,
+    plugin: RemotelySavePlugin,
+    skipSizeLargerThan: number,
+    sizesGoWrong: FileOrFolderMixedState[],
+    hasPassword: boolean
+  ) {
+    super(app);
+    this.plugin = plugin;
+    this.skipSizeLargerThan = skipSizeLargerThan;
+    this.sizesGoWrong = sizesGoWrong;
+    this.hasPassword = hasPassword;
+  }
+  onOpen() {
+    let { contentEl } = this;
+    const t = (x: TransItemType, vars?: any) => {
+      return this.plugin.i18n.t(x, vars);
+    };
+
+    contentEl.createEl("h2", {
+      text: t("modal_sizesconflict_title"),
+    });
+
+    t("modal_sizesconflict_desc", {
+      thresholdMB: `${this.skipSizeLargerThan / 1000 / 1000}`,
+      thresholdBytes: `${this.skipSizeLargerThan}`,
+    })
+      .split("\n")
+      .forEach((val) => {
+        contentEl.createEl("p", { text: val });
+      });
+
+    const info = this.serialize();
+
+    contentEl.createDiv().createEl(
+      "button",
+      {
+        text: t("modal_sizesconflict_copybutton"),
+      },
+      (el) => {
+        el.onclick = async () => {
+          await navigator.clipboard.writeText(info);
+          new Notice(t("modal_sizesconflict_copynotice"));
+        };
+      }
+    );
+
+    contentEl.createEl("pre", {
+      text: info,
+    });
+  }
+
+  serialize() {
+    return this.sizesGoWrong
+      .map((x) => {
+        return [
+          x.key,
+          this.hasPassword
+            ? `encrypted name: ${x.remoteEncryptedKey}`
+            : undefined,
+          `local ${this.hasPassword ? "encrypted " : ""}bytes: ${
+            this.hasPassword ? x.sizeLocalEnc : x.sizeLocal
+          }`,
+          `remote ${this.hasPassword ? "encrypted " : ""}bytes: ${
+            this.hasPassword ? x.sizeRemoteEnc : x.sizeRemote
+          }`,
+        ]
+          .filter((tmp) => tmp !== undefined)
+          .join("\n");
+      })
+      .join("\n\n");
+  }
+
+  onClose() {
+    let { contentEl } = this;
+    contentEl.empty();
+  }
+}

versions.json

@@ -1,4 +1,4 @@
 {
   "0.3.2": "0.12.15",
-  "0.3.18": "0.13.21"
+  "0.3.23": "0.13.21"
 }