update node ci

bump to 0.5.16
upgrade check connections
2024-07-27 23:08:49 +08:00 · 2024-07-27 22:58:36 +08:00 · 2024-07-27 22:58:05 +08:00 · 2024-07-27 22:55:04 +08:00 · 2024-07-27 22:08:22 +08:00 · 2024-07-27 19:38:25 +08:00
27 changed files with 524 additions and 106 deletions
@@ -34,7 +34,7 @@ jobs:

    strategy:
      matrix:
-        node-version: [16.x]
+        node-version: [20.x]
        # See supported Node.js release schedule at https://nodejs.org/en/about/releases/

    steps:
@@ -54,7 +54,7 @@ jobs:
      - name: Git LFS Pull
        run: git lfs pull
      - name: Use Node.js ${{ matrix.node-version }}
-        uses: actions/setup-node@v2
+        uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm install
@@ -38,7 +38,7 @@ jobs:

    strategy:
      matrix:
-        node-version: [16.x]
+        node-version: [20.x]

    steps:
      - name: Checkout codes
@@ -57,7 +57,7 @@ jobs:
      - name: Git LFS Pull
        run: git lfs pull
      - name: Use Node.js ${{ matrix.node-version }}
-        uses: actions/setup-node@v2
+        uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm install
@@ -46,6 +46,8 @@ This is yet another unofficial sync plugin for Obsidian. If you like it or find
 - **You should protect your `data.json` file.** The file contains sensitive information.
  - It's strongly advised **NOT** to share your `data.json` file to anyone.
  - It's usually **NOT** a good idea to check the file into version control. By default, the plugin tries to create a `.gitignore` file inside the plugin directory if it doesn't exist, for ignoring `data.json` in the `git` version control. If you know exactly what it means and want to remove the setting, please modify the `.gitignore` file or set it to be empty.
+- **Obsidian API on Mobile has performance issues syncing large files (>= 50 MB).**
+  - Setting the "Skip Large Files" option can help resolve issues related to syncing large files.

 ## Questions, Suggestions, Or Bugs

@@ -112,6 +114,7 @@ Additionally, the plugin author may occasionally visit Obsidian official forum a
  - [ownCloud](./docs/remote_services/webdav_owncloud/README.md)
  - [InfiniCloud](./docs/remote_services/webdav_infinicloud_teracloud/README.md)
  - [Synology webdav server](./docs/remote_services/webdav_synology_webdav_server/README.md) | [群晖 webdav server](./docs/remote_services/webdav_synology_webdav_server/README.zh-cn.md)
+  - [dufs](./docs/remote_services/webdav_dufs/README.md)
  - [AList（中文）](./docs/remote_services/webdav_alist/README.zh-cn.md) | [AList (English)](./docs/remote_services/webdav_alist/README.md)
  - [坚果云](./docs/remote_services/webdav_jianguoyun/README.zh-cn.md) | [JianGuoYun/NutStore](./docs/remote_services/webdav_jianguoyun/README.md)
  - [Open Media Vault](./docs/remote_services/webdav_openmediavault/README.md)
@@ -161,7 +164,7 @@ PRO (paid) feature "sync with Azure Blob Storage" allows users to to sync with A

 Basic (free) version can detect conflicts, but users have to choose to keep newer version or larger version of the files.

-PRO (paid) feature "Smart Conflict" gives users one more option: merge small markdown files, or duplicate large markdown files or non-markdown files.
+PRO (paid) feature "Smart Conflict" gives users one more option: merge small markdown files, or duplicate large markdown files or any-size non-markdown files.

 See documents [here](./docs/pro/README.md)

@@ -176,11 +179,19 @@ See documents [here](./docs/pro/README.md)
 - You can configure sync on save in settings.
 - In sync on save mode, if any error occurs, the plugin would **fail silently**.

-## How To Deal With Hidden Files Or Folders
+## Config Folder / Files and Bookmarks
+
+By default, the plugin doesn't sync config folder / files of obsidian (normally `.obsidian` folder) because it's one of the hidden files. 
+
+However, in the latest version, you can enable syncing config folder in settings. Please be aware that it's an experimental feature. Notably some config files' modified times are changed everytime when Obsidian is opened, and this destroys the assumptions of the plugin, so some information may not be synced properly.
+
+Moreover, the bookmarks of Obsidian are actually stored in `.obsidian/bookmarks.json` file. You can also set syncing this file (instead of the whole config folder) in settings. The plugin will keep the latest modified one across devices.
+
+## Hidden Files Or Folders

 **By default, all files or folder starting with `.` (dot) or `_` (underscore) are treated as hidden files, and would NOT be synced.** It's useful if you have some files just staying locally. But this strategy also means that themes / other plugins / settings of this plugin would neither be synced.

-In the latest version, you can change the settings to allow syncing `_` files or folders, as well as `.obsidian` special config folder (but not any other `.` files or folders).
+In the latest version, you can change the settings to allow syncing `_` files or folders, as well as `.obsidian` special config folder as described above (but not any other `.` files or folders).

 ## PRO Features

@@ -1,5 +1,5 @@
 {
-  "$schema": "https://biomejs.dev/schemas/1.7.3/schema.json",
+  "$schema": "https://biomejs.dev/schemas/1.8.3/schema.json",
  "organizeImports": {
    "enabled": true
  },
@@ -24,7 +24,7 @@
      "jsxQuoteStyle": "double",
      "quoteProperties": "asNeeded",
      "semicolons": "always",
-      "trailingComma": "es5"
+      "trailingCommas": "es5"
    }
  },
  "linter": {
@@ -6,12 +6,18 @@ Ideally, users do not need to debug anything. But if something bad happened, we

 See [here](./export_sync_plans.md).

-## Advanced: Check Console Output Directly
+## Medium: Desktop: Check Console Output Directly

 ...of desktop and Android.

 See [here](./check_console_output.md).

+## Medium: Mobile: Use `Obsidian vConsole` to Check Console Ouput Directly
+
+...of mobile (iOS and Android).
+
+See [here](./check_vconsole_output.md).
+
 ## Advanced: Use `Logstravaganza` to export logs

 This method works for desktop and mobile devices (iOS, Android), especially useful for iOS.
@@ -18,6 +18,8 @@ Go to the plugin settings, scroll down to the section "Debug" -> "alter console

 - If you are using Android

-  You need to [enable USB debugging](https://developer.android.com/studio/debug/dev-options#enable) on your Android, then connect your Android to a computer using USB, then open the **desktop** Chrome browser and go to the special web page <chrome://inspect>. You shall be able to see the "inspect" link inside the special page, then click the link to open the console. After debugging, remember to turn off USB debugging.
+  If you want to check the console **with the help of desktop Chrome**, you need to [enable USB debugging](https://developer.android.com/studio/debug/dev-options#enable) on your Android, then connect your Android to a computer using USB, then open the **desktop** Chrome browser and go to the special web page <chrome://inspect>. You shall be able to see the "inspect" link inside the special page, then click the link to open the console. After debugging, remember to turn off USB debugging.
+
+  If you want to check the console **directly on the mobile device**, check out [this doc](./check_vconsole_output.md).

 Trigger the sync manually (by clicking the icon on the ribbon sidebar). Something (hopefully) helpful should show up in the console. You could understand what happened and what goes wrong more explictly by checking the output.
@@ -0,0 +1,21 @@
+# Check "Obsidian vConsole" Output
+
+It's quite hard to debug on mobile. But fortunately you can use another plugin to rescue!
+
+This applies to both iOS (iPhone / iPad) and Android.
+
+## Disable Auto Sync Firstly
+
+You should disable auto sync to avoid any unexpected running.
+
+## Set The Output Level To Debug
+
+Go to the plugin settings, scroll down to the section "Debug" -> "alter console log level", and change it from "info" to "debug".
+
+## Use "Obsidian vConsole"
+
+1. Install the third-party plugin ["Obsidian vConsole"](https://github.com/zhouhua/obsidian-vconsole).
+2. Enable "Obsidian vConsole". You shall see a green button in the right buttom corner of the interface of your mobile Obsidian. You can drag this green button to where you like.
+3. Sync! And quickly click the vConsole green button, you will see the console logs in its panel! Moreover, you can check browser network and Storage (LocalStorage) in the panel! 
+4. You can copy (by clicking the small "disk" icon beside each line of the log) or take a screenshot of the logs, and report bugs in Remotely Save GitHub. You can click "Hide" in the panel to hide the panel.
+5. If you don't need to debug any more, you can disable Obsidian vConsole plugin, then the green button should disappear.
@@ -0,0 +1,17 @@
+# dufs webdav
+
+## Link
+
+<https://github.com/sigoden/dufs>
+
+## Steps
+
+1. Download the latest version: <https://github.com/sigoden/dufs/releases> and save it to a folder
+2. Add the path environment variable.
+3. Start the server in terminal, you can change the address (`127.0.0.1`) and username (`user1`) and password (`pass1`) accordingly:
+    ```bash
+    dufs -A --enable-cors --bind 127.0.0.1 --port 8080 --auth user1:pass1@/:rw
+    ```
+4. In remotely-save setting page, select webdav type, then input the address/account/**webdav password**(not your account password).
+5. In remotely-save setting page, click "Check Connectivity".
+6. Sync!
@@ -1,7 +1,7 @@
 {
  "id": "remotely-save",
  "name": "Remotely Save",
-  "version": "0.5.14",
+  "version": "0.5.16",
  "minAppVersion": "0.13.21",
  "description": "Yet another unofficial plugin allowing users to synchronize notes between local device and the cloud service.",
  "author": "fyears",
@@ -1,7 +1,7 @@
 {
  "id": "remotely-save",
  "name": "Remotely Save",
-  "version": "0.5.14",
+  "version": "0.5.16",
  "minAppVersion": "0.13.21",
  "description": "Yet another unofficial plugin allowing users to synchronize notes between local device and the cloud service.",
  "author": "fyears",
@@ -1,15 +1,15 @@
 {
  "name": "remotely-save",
-  "version": "0.5.14",
+  "version": "0.5.16",
  "description": "This is yet another sync plugin for Obsidian app.",
  "scripts": {
    "dev2": "node esbuild.config.mjs --watch",
    "build2": "tsc -noEmit -skipLibCheck && node esbuild.config.mjs production",
    "build": "webpack --mode production",
    "dev": "webpack --mode development --watch",
-    "format": "npx @biomejs/biome check --apply .",
+    "format": "npx @biomejs/biome check --write .",
    "clean": "npx rimraf main.js",
-    "test": "cross-env TS_NODE_COMPILER_OPTIONS={\\\"module\\\":\\\"commonjs\\\"} mocha -r ts-node/register 'tests/**/*.ts' 'pro/tests/**/*.ts'"
+    "test": "mocha --import=tsx 'tests/**/*.ts' 'pro/tests/**/*.ts'"
  },
  "browser": {
    "path": "path-browserify",
@@ -25,55 +25,55 @@
  "author": "",
  "license": "SEE LICENSE IN LICENSE",
  "devDependencies": {
-    "@biomejs/biome": "1.7.3",
+    "@biomejs/biome": "1.8.3",
    "@microsoft/microsoft-graph-types": "^2.40.0",
-    "@types/chai": "^4.3.14",
+    "@types/chai": "^4.3.16",
    "@types/chai-as-promised": "^7.1.8",
-    "@types/jsdom": "^21.1.6",
-    "@types/lodash": "^4.17.0",
+    "@types/jsdom": "^21.1.7",
+    "@types/lodash": "^4.17.7",
    "@types/mime-types": "^2.1.4",
-    "@types/mocha": "^10.0.6",
+    "@types/mocha": "^10.0.7",
    "@types/mustache": "^4.2.5",
-    "@types/node": "^20.12.7",
+    "@types/node": "^20.14.12",
    "@types/qrcode": "^1.5.5",
-    "builtin-modules": "^3.3.0",
+    "builtin-modules": "^4.0.0",
    "cross-env": "^7.0.3",
    "dotenv": "^16.4.5",
-    "esbuild": "^0.20.2",
+    "esbuild": "^0.23.0",
    "esbuild-plugin-inline-worker": "^0.1.1",
-    "jsdom": "^24.0.0",
-    "mocha": "^10.4.0",
+    "jsdom": "^24.1.1",
+    "mocha": "^10.7.0",
    "npm-check-updates": "^16.14.20",
    "obsidian": "^1.5.7",
-    "openapi-typescript": "^7.0.0-rc.0",
+    "openapi-typescript": "^7.1.0",
    "ts-loader": "^9.5.1",
-    "ts-node": "^10.9.2",
-    "tslib": "^2.6.2",
-    "typescript": "^5.4.5",
+    "tslib": "^2.6.3",
+    "tsx": "^4.16.2",
+    "typescript": "^5.5.4",
    "webdav-server": "^2.6.2",
-    "webpack": "^5.91.0",
+    "webpack": "^5.93.0",
    "webpack-cli": "^5.1.4",
    "worker-loader": "^3.0.8"
  },
  "dependencies": {
-    "@aws-sdk/client-s3": "^3.563.0",
-    "@aws-sdk/lib-storage": "^3.563.0",
-    "@aws-sdk/signature-v4-crt": "^3.556.0",
-    "@aws-sdk/types": "^3.535.0",
-    "@azure/msal-node": "^2.7.0",
-    "@azure/storage-blob": "^12.23.0",
+    "@aws-sdk/client-s3": "^3.620.0",
+    "@aws-sdk/lib-storage": "^3.620.0",
+    "@aws-sdk/signature-v4-crt": "^3.620.0",
+    "@aws-sdk/types": "^3.609.0",
+    "@azure/msal-node": "^2.12.0",
+    "@azure/storage-blob": "^12.24.0",
    "@fyears/rclone-crypt": "^0.0.7",
    "@fyears/tsqueue": "^1.0.1",
    "@microsoft/microsoft-graph-client": "^3.0.7",
    "@sanity/diff-match-patch": "^3.1.1",
-    "@smithy/fetch-http-handler": "^2.5.0",
-    "@smithy/protocol-http": "^3.3.0",
-    "@smithy/querystring-builder": "^2.2.0",
-    "acorn": "^8.11.3",
+    "@smithy/fetch-http-handler": "^3.2.4",
+    "@smithy/protocol-http": "^4.1.0",
+    "@smithy/querystring-builder": "^3.0.3",
+    "acorn": "^8.12.1",
    "aggregate-error": "^5.0.0",
    "assert": "^2.1.0",
-    "aws-crt": "^1.21.2",
-    "box-typescript-sdk-gen": "^1.0.0",
+    "aws-crt": "^1.21.3",
+    "box-typescript-sdk-gen": "^1.3.0",
    "buffer": "^6.0.3",
    "crypto-browserify": "^3.12.0",
    "dropbox": "^10.34.0",
@@ -83,21 +83,21 @@
    "localforage-getitems": "^1.4.2",
    "localforage-removeitems": "^1.4.0",
    "lodash": "^4.17.21",
-    "lucide": "^0.376.1",
+    "lucide": "^0.416.0",
    "mime-types": "^2.1.35",
    "mustache": "^4.2.0",
    "nanoid": "^5.0.7",
    "node-diff3": "^3.1.2",
-    "openapi-fetch": "^0.9.7",
+    "openapi-fetch": "^0.10.2",
    "p-queue": "^8.0.1",
    "path-browserify": "^1.0.1",
    "pcloud-sdk-js": "^2.0.0",
    "process": "^0.11.10",
    "qrcode": "^1.5.3",
    "rfc4648": "^1.5.3",
-    "rimraf": "^5.0.5",
+    "rimraf": "^6.0.1",
    "stream-browserify": "^3.0.0",
-    "url": "^0.11.3",
+    "url": "^0.11.4",
    "util": "^0.12.5",
    "webdav": "^5.6.0",
    "xregexp": "^5.1.1"
@@ -38,9 +38,11 @@ export function isMergable(a: Entity, b?: Entity) {
 * @param b
 */
 function mergeDigInModified(a: string, o: string, b: string) {
-  const { conflict, result } = mergeDigIn(a, o, b);
+  const { conflict, result } = mergeDigIn(a, o, b, {
+    stringSeparator: /\n/,
+  });
  for (let index = 0; index < result.length; ++index) {
-    if (["<<<<<<<", "=======", ">>>>>>>"].contains(result[index])) {
+    if (["<<<<<<<", "=======", ">>>>>>>"].includes(result[index])) {
      result[index] = "`" + result[index] + "`";
    }
  }
@@ -72,7 +74,7 @@ function getLCSText(a: string, b: string) {
 * @param b
 * @returns
 */
-function twoWayMerge(a: string, b: string): string {
+export function twoWayMerge(a: string, b: string): string {
  const aa = a.trim();
  const bb = b.trim();
  if (aa === "" && bb === "") {
@@ -88,7 +90,10 @@ function twoWayMerge(a: string, b: string): string {
  // const c = getLCSText(a, b);
  // const patches = makePatches(c, a);
  // const [d] = applyPatches(patches, b);
-  const c = getLCSText(a, b);
+  const c = getLCSText(a, b); //.trim();
+  // console.debug(`(start) LCS Text:`);
+  // console.debug(c);
+  // console.debug(`(end) LCS Text.`);
  const d = mergeDigInModified(a, c, b).result.join("\n");
  return d;
 }
@@ -100,7 +105,7 @@ function twoWayMerge(a: string, b: string): string {
 * @param orig
 * @returns
 */
-function threeWayMerge(a: string, b: string, orig: string) {
+export function threeWayMerge(a: string, b: string, orig: string) {
  return mergeDigInModified(a, orig, b).result.join("\n");
 }

@@ -320,12 +320,12 @@ export class FakeFsAzureBlobStorage extends FakeFs {
    // if we can walk, we can connect
    try {
      await this.walkPartial();
-      return true;
    } catch (err) {
      console.debug(err);
      callbackFunc?.(err);
      return false;
    }
+    return await this.checkConnectCommonOps(callbackFunc);
  }

  async getUserDisplayName(): Promise<string> {
@@ -892,12 +892,12 @@ export class FakeFsBox extends FakeFs {
    // if we can init, we can connect
    try {
      await this._init();
-      return true;
    } catch (err) {
      console.debug(err);
      callbackFunc?.(err);
      return false;
    }
+    return await this.checkConnectCommonOps(callbackFunc);
  }
  async getUserDisplayName(): Promise<string> {
    throw new Error("Method not implemented.");
@@ -728,12 +728,12 @@ export class FakeFsGoogleDrive extends FakeFs {
    // if we can init, we can connect
    try {
      await this._init();
-      return true;
    } catch (err) {
      console.debug(err);
      callbackFunc?.(err);
      return false;
    }
+    return await this.checkConnectCommonOps(callbackFunc);
  }

  async getUserDisplayName(): Promise<string> {
@@ -263,7 +263,7 @@ const getAuthMiddleware = (
  saveUpdatedConfigFunc: any
 ) => {
  const authMiddleware: Middleware = {
-    async onRequest(req) {
+    async onRequest({ request }) {
      const getAccessToken = async () => {
        if (koofrConfig.refreshToken === "") {
          throw Error("The user has not manually auth yet.");
@@ -291,8 +291,8 @@ const getAuthMiddleware = (
      };

      const access = await getAccessToken();
-      req.headers.set("Authorization", `Bearer ${access}`);
-      return req;
+      request.headers.set("Authorization", `Bearer ${access}`);
+      return request;
    },
  };
  return authMiddleware;
@@ -555,12 +555,12 @@ export class FakeFsKoofr extends FakeFs {
    // if we can init, we can connect
    try {
      await this._init();
-      return true;
    } catch (err) {
      console.debug(err);
      callbackFunc?.(err);
      return false;
    }
+    return await this.checkConnectCommonOps(callbackFunc);
  }

  async getUserDisplayName(): Promise<string> {
@@ -290,8 +290,12 @@ const fromDriveItemToEntity = (x: DriveItem, remoteBaseDir: string): Entity => {
    key = `${key}/`;
  }

-  const mtimeSvr = Date.parse(x?.fileSystemInfo!.lastModifiedDateTime!);
-  const mtimeCli = Date.parse(x?.fileSystemInfo!.lastModifiedDateTime!);
+  const mtimeTry = x?.fileSystemInfo?.lastModifiedDateTime;
+  if (mtimeTry === undefined || mtimeTry === null) {
+    throw Error(`onedrive cannot parse mtime: ${JSON.stringify(x, null, 2)}`);
+  }
+  const mtimeSvr = Date.parse(mtimeTry);
+  const mtimeCli = Date.parse(mtimeTry);
  return {
    key: key,
    keyRaw: key,
@@ -912,12 +916,15 @@ export class FakeFsOnedriveFull extends FakeFs {
  async checkConnect(callbackFunc?: any): Promise<boolean> {
    try {
      const k = await this.getUserDisplayName();
-      return k !== "<unknown display name>";
+      if (k === "<unknown display name>") {
+        throw Error(`unknown display name!`);
+      }
    } catch (err) {
      console.debug(err);
      callbackFunc?.(err);
      return false;
    }
+    return await this.checkConnectCommonOps(callbackFunc);
  }

  async getUserDisplayName() {
@@ -604,12 +604,12 @@ export class FakeFsPCloud extends FakeFs {
    // if we can init, we can connect
    try {
      await this._init();
-      return true;
    } catch (err) {
      console.debug(err);
      callbackFunc?.(err);
      return false;
    }
+    return await this.checkConnectCommonOps(callbackFunc);
  }

  async getUserDisplayName(): Promise<string> {
@@ -480,12 +480,12 @@ export class FakeFsYandexDisk extends FakeFs {
    // if we can init, we can connect
    try {
      await this._init();
-      return true;
    } catch (err) {
      console.debug(err);
      callbackFunc?.(err);
      return false;
    }
+    return await this.checkConnectCommonOps(callbackFunc);
  }

  async getUserDisplayName(): Promise<string> {
@@ -1,5 +1,9 @@
 import { deepStrictEqual, rejects, throws } from "assert";
-import { getFileRenameForDup } from "../src/conflictLogic";
+import {
+  getFileRenameForDup,
+  threeWayMerge,
+  twoWayMerge,
+} from "../src/conflictLogic";

 describe("New name is generated", () => {
  it("should throw for empty file", async () => {
@@ -69,3 +73,224 @@ describe("New name is generated", () => {
    );
  });
 });
+
+describe("Two way merge", () => {
+  it("should correctly merge from zero files", async () => {
+    const a = "aaa";
+    const b = "bbb";
+    const res = twoWayMerge(a, b);
+    const expected = `\`<<<<<<<\`
+aaa
+\`=======\`
+bbb
+\`>>>>>>>\``;
+    deepStrictEqual(expected, res);
+  });
+
+  it("should correctly merge from common lines", async () => {
+    const a = `
+Something is cool. 中文1
+Other thing is cooler. 哈哈！
+`;
+    const b = `
+Anything is cool. 中文2
+Other thing is cooler. 哈哈！
+`;
+    const res = twoWayMerge(a, b);
+    // console.log(res);
+    const expected = `
+\`<<<<<<<\`
+Something is cool. 中文1
+\`=======\`
+Anything is cool. 中文2
+\`>>>>>>>\`
+Other thing is cooler. 哈哈！
+`;
+    deepStrictEqual(expected, res);
+  });
+
+  it("should merge by lines", async () => {
+    const a = `
+Something is cool. 中文1
+`;
+    const b = `
+Something is cooler. 中文2
+`;
+    const res = twoWayMerge(a, b);
+    // console.log(res);
+    const expected = `
+\`<<<<<<<\`
+Something is cool. 中文1
+\`=======\`
+Something is cooler. 中文2
+\`>>>>>>>\`
+`;
+    deepStrictEqual(expected, res);
+  });
+});
+
+describe("Three way merge", () => {
+  it("should correctly merge from zero files", async () => {
+    const orig = "";
+    const a = "aaa";
+    const b = "bbb";
+    const res = threeWayMerge(a, b, orig);
+    const expected = `\`<<<<<<<\`
+aaa
+\`=======\`
+bbb
+\`>>>>>>>\``;
+    deepStrictEqual(expected, res);
+  });
+
+  it("should correctly merge after adding lines on both sides", async () => {
+    const orig = `
+* [ ] A1
+* [ ] A2
+* [ ] A3
+`;
+    const a = `
+* [ ] A1
+* [ ] new line after A1
+* [ ] A2
+* [ ] A3
+`;
+    const b = `
+* [ ] A1
+* [ ] A2
+* [ ] New line after A2
+* [ ] A3
+`;
+    const res = threeWayMerge(a, b, orig);
+    // console.log(res);
+    const expected = `
+* [ ] A1
+* [ ] new line after A1
+* [ ] A2
+* [ ] New line after A2
+* [ ] A3
+`;
+    deepStrictEqual(expected, res);
+  });
+
+  it("should correctly merge after adding lines on both sides (again)", async () => {
+    const orig = `
+* [ ] 中文
+* [ ] にほんご／にっぽんご
+* [ ] A3
+`;
+    const a = `
+* [ ] 中文
+* [ ] new line after 中文
+* [ ] にほんご／にっぽんご
+* [ ] A3
+`;
+    const b = `
+* [ ] 中文
+* [ ] にほんご／にっぽんご
+* [ ] New line after にほんご／にっぽんご
+* [ ] A3
+`;
+    const res = threeWayMerge(a, b, orig);
+    // console.log(res);
+    const expected = `
+* [ ] 中文
+* [ ] new line after 中文
+* [ ] にほんご／にっぽんご
+* [ ] New line after にほんご／にっぽんご
+* [ ] A3
+`;
+    deepStrictEqual(expected, res);
+  });
+
+  it("should correctly merge after deleting lines on both sides", async () => {
+    const orig = `
+* [ ] 中文
+* [ ] にほんご／にっぽんご
+* [ ] A3
+* [ ] A4
+`;
+    const a = `
+* [ ] にほんご／にっぽんご
+* [ ] A3
+* [ ] A4
+`;
+    const b = `
+* [ ] 中文
+* [ ] A3
+* [ ] A4
+`;
+    const res = threeWayMerge(a, b, orig);
+    // console.log(res);
+    const expected = `
+\`<<<<<<<\`
+* [ ] にほんご／にっぽんご
+\`=======\`
+* [ ] 中文
+\`>>>>>>>\`
+* [ ] A3
+* [ ] A4
+`;
+    deepStrictEqual(expected, res);
+  });
+
+  it("should correctly merge after adding on one side and deleting on other side", async () => {
+    const orig = `
+* [ ] 中文
+* [ ] A3
+* [ ] A4
+`;
+    const a = `
+* [ ] 中文
+* [ ] にほんご／にっぽんご
+* [ ] A3
+* [ ] A4
+`;
+    const b = `
+* [ ] A3
+* [ ] A4
+`;
+    const res = threeWayMerge(a, b, orig);
+    // console.log(res);
+    const expected = `
+\`<<<<<<<\`
+* [ ] 中文
+* [ ] にほんご／にっぽんご
+\`=======\`
+\`>>>>>>>\`
+* [ ] A3
+* [ ] A4
+`;
+    deepStrictEqual(expected, res);
+  });
+
+  it("should correctly merge after adding on one side and deleting on other side (again)", async () => {
+    const orig = `
+* [ ] 中文
+* [ ] A3
+* [ ] A4
+`;
+    const a = `
+* [ ] A3
+* [ ] A4
+`;
+    const b = `
+* [ ] 中文
+* [ ] にほんご／にっぽんご
+* [ ] A3
+* [ ] A4
+`;
+    const res = threeWayMerge(a, b, orig);
+    // console.log(res);
+    const expected = `
+\`<<<<<<<\`
+\`=======\`
+* [ ] 中文
+* [ ] にほんご／にっぽんご
+\`>>>>>>>\`
+* [ ] A3
+* [ ] A4
+`;
+    deepStrictEqual(expected, res);
+  });
+});
@@ -1,3 +1,5 @@
+import isEqual from "lodash/isEqual";
+import { nanoid } from "nanoid";
 import type { Entity } from "./baseTypes";

 export abstract class FakeFs {
@@ -16,6 +18,49 @@ export abstract class FakeFs {
  abstract rename(key1: string, key2: string): Promise<void>;
  abstract rm(key: string): Promise<void>;
  abstract checkConnect(callbackFunc?: any): Promise<boolean>;
+  async checkConnectCommonOps(callbackFunc?: any) {
+    try {
+      console.info(`check connect: create folder`);
+      const folderName = `rs-test-folder-${nanoid()}/`;
+      await this.mkdir(folderName);
+      // await delay(3000);
+
+      console.info(`check connect: upload file`);
+      const filename = `${folderName}/rs-test-file-${nanoid()}`;
+      const ctime = Date.now();
+      const mtime1 = Date.now();
+      const content1 = new ArrayBuffer(100);
+      await this.writeFile(filename, content1, mtime1, ctime);
+      // await delay(3000);
+
+      console.info(`check connect: overwrite file`);
+      const mtime2 = Date.now();
+      const content2 = new ArrayBuffer(200);
+      await this.writeFile(filename, content2, mtime2, ctime);
+      // await delay(3000);
+
+      console.info(`check connect: download file`);
+      const content3 = await this.readFile(filename);
+      if (!isEqual(content2, content3)) {
+        throw Error(`downloaded file is not equal with uploaded file!`);
+      }
+      // await delay(3000);
+
+      console.info(`check connect: delete file`);
+      await this.rm(filename);
+      // await delay(3000);
+
+      console.info(`check connect: delete folder`);
+      await this.rm(folderName);
+      // await delay(3000);
+
+      return true;
+    } catch (err) {
+      console.error(err);
+      callbackFunc?.(err);
+      return false;
+    }
+  }
  abstract getUserDisplayName(): Promise<string>;
  abstract revokeAuth(): Promise<any>;
  abstract allowEmptyFile(): boolean;
@@ -741,14 +741,15 @@ export class FakeFsDropbox extends FakeFs {
      await this._init();
      const results = await this._statFromRoot(`/${this.remoteBaseDir}`);
      if (results === undefined) {
-        return false;
+        throw Error(`cannot check root vault folder!`);
      }
-      return true;
    } catch (err) {
      console.debug(err);
      callbackFunc?.(err);
      return false;
    }
+
+    return await this.checkConnectCommonOps(callbackFunc);
  }

  async getUserDisplayName() {
@@ -47,7 +47,7 @@ export class FakeFsMock extends FakeFs {
  }

  async checkConnect(callbackFunc?: any): Promise<boolean> {
-    return true;
+    return await this.checkConnectCommonOps(callbackFunc);
  }

  async getUserDisplayName(): Promise<string> {
@@ -399,8 +399,12 @@ ${constructFromDriveItemToEntityError(x)}`
    key = `${key}/`;
  }

-  const mtimeSvr = Date.parse(x?.fileSystemInfo!.lastModifiedDateTime!);
-  const mtimeCli = Date.parse(x?.fileSystemInfo!.lastModifiedDateTime!);
+  const mtimeTry = x?.fileSystemInfo?.lastModifiedDateTime;
+  if (mtimeTry === undefined || mtimeTry === null) {
+    throw Error(`onedrive cannot parse mtime: ${JSON.stringify(x, null, 2)}`);
+  }
+  const mtimeSvr = Date.parse(mtimeTry);
+  const mtimeCli = Date.parse(mtimeTry);
  return {
    key: key,
    keyRaw: key,
@@ -1032,12 +1036,16 @@ export class FakeFsOnedrive extends FakeFs {
  async checkConnect(callbackFunc?: any): Promise<boolean> {
    try {
      const k = await this.getUserDisplayName();
-      return k !== "<unknown display name>";
+      if (k === "<unknown display name>") {
+        throw Error(`unknown display name!`);
+      }
    } catch (err) {
      console.debug(err);
      callbackFunc?.(err);
      return false;
    }
+
+    return await this.checkConnectCommonOps(callbackFunc);
  }

  async getUserDisplayName() {
@@ -831,14 +831,11 @@ export class FakeFsS3 extends FakeFs {
        results.$metadata === undefined ||
        results.$metadata.httpStatusCode === undefined
      ) {
-        const err = "results or $metadata or httStatusCode is undefined";
-        console.debug(err);
-        if (callbackFunc !== undefined) {
-          callbackFunc(err);
-        }
-        return false;
+        throw Error("results or $metadata or httStatusCode is undefined");
+      }
+      if (results.$metadata.httpStatusCode !== 200) {
+        throw Error(`not 200 httpStatusCode`);
      }
-      return results.$metadata.httpStatusCode === 200;
    } catch (err: any) {
      console.debug(err);
      if (callbackFunc !== undefined) {
@@ -854,9 +851,10 @@ export class FakeFsS3 extends FakeFs {
          callbackFunc(err);
        }
      }
-
      return false;
    }
+
+    return await this.checkConnectCommonOps(callbackFunc);
  }

  async getUserDisplayName(): Promise<string> {
@@ -4,8 +4,6 @@ import { getReasonPhrase } from "http-status-codes/build/cjs/utils-functions";
 import chunk from "lodash/chunk";
 import cloneDeep from "lodash/cloneDeep";
 import flatten from "lodash/flatten";
-import isString from "lodash/isString";
-import { nanoid } from "nanoid";
 import { Platform, type RequestUrlParam, requestUrl } from "obsidian";
 import type {
  FileStat,
@@ -140,6 +138,7 @@ if (VALID_REQURL) {
  );
 }

+import isEqual from "lodash/isEqual";
 // @ts-ignore
 // biome-ignore lint: we want to ts-ignore the next line
 import { AuthType, BufferLike, createClient } from "webdav/dist/web/index.js";
@@ -209,6 +208,13 @@ const tryEncodeURI = (x: string) => {
  return encodeURI(x);
 };

+const tryEncodeUsernamePassword = (x: string) => {
+  if (onlyAscii(x)) {
+    return x;
+  }
+  return unescape(encodeURIComponent(x));
+};
+
 const parseCustomHeaders = (x: string): Record<string, string> => {
  const y = x.trim();
  if (y === "") {
@@ -241,7 +247,8 @@ export class FakeFsWebdav extends FakeFs {
  vaultFolderExists: boolean;
  saveUpdatedConfigFunc: () => Promise<any>;

-  supportNativePartial: boolean;
+  supportApachePartial: boolean;
+  supportSabrePartial: boolean;
  isNextcloud: boolean;
  nextcloudUploadServerAddress: string;

@@ -258,7 +265,8 @@ export class FakeFsWebdav extends FakeFs {
    this.vaultFolderExists = false;
    this.saveUpdatedConfigFunc = saveUpdatedConfigFunc;

-    this.supportNativePartial = false;
+    this.supportApachePartial = false;
+    this.supportSabrePartial = false;
    this.isNextcloud = false;
    this.nextcloudUploadServerAddress = "";
  }
@@ -282,8 +290,8 @@ export class FakeFsWebdav extends FakeFs {
      this.webdavConfig.password !== ""
    ) {
      this.client = createClient(this.webdavConfig.address, {
-        username: this.webdavConfig.username,
-        password: this.webdavConfig.password,
+        username: tryEncodeUsernamePassword(this.webdavConfig.username),
+        password: tryEncodeUsernamePassword(this.webdavConfig.password),
        headers: headers,
        authType:
          this.webdavConfig.authType === "digest"
@@ -392,15 +400,20 @@ export class FakeFsWebdav extends FakeFs {
    // taken from https://github.com/perry-mitchell/webdav-client/blob/master/source/operations/partialUpdateFileContents.ts
    // which is under MIT license
    if (
-      (compliance.server.includes("Apache") &&
-        compliance.compliance.includes(
-          "<http://apache.org/dav/propset/fs/1>"
-        )) ||
-      compliance.compliance.includes("sabredav-partialupdate")
+      compliance.server.includes("Apache") &&
+      compliance.compliance.includes("<http://apache.org/dav/propset/fs/1>")
    ) {
-      this.supportNativePartial = true;
+      this.supportApachePartial = true;
      console.debug(
-        `supportNativePartial=true, compliance=${JSON.stringify(compliance)}`
+        `supportApachePartial=true, compliance=${JSON.stringify(compliance)}`
+      );
+      return true;
+    }
+
+    if (compliance.compliance.includes("sabredav-partialupdate")) {
+      this.supportSabrePartial = true;
+      console.debug(
+        `supportSabrePartial=true, compliance=${JSON.stringify(compliance)}`
      );
      return true;
    }
@@ -417,7 +430,9 @@ export class FakeFsWebdav extends FakeFs {
      this.webdavConfig.depth === "auto_unknown" ||
      this.webdavConfig.depth === "auto_1" ||
      this.webdavConfig.depth === "auto_infinity" /* don't trust auto now */ ||
-      this.webdavConfig.depth === "manual_1"
+      this.webdavConfig.depth === "manual_1" ||
+      this.webdavConfig.address.includes("jianguoyun.com") ||
+      this.webdavConfig.address.includes("teracloud.jp")
    ) {
      // the remote doesn't support infinity propfind,
      // we need to do a bfs here
@@ -570,7 +585,11 @@ export class FakeFsWebdav extends FakeFs {
    }

    // larger than 10 MB
-    if (!this.isNextcloud && !this.supportNativePartial) {
+    if (
+      !this.isNextcloud &&
+      !this.supportApachePartial &&
+      !this.supportSabrePartial
+    ) {
      // give up and upload by whole, and directly return
      return await this._writeFileFromRootFull(
        key,
@@ -591,8 +610,16 @@ export class FakeFsWebdav extends FakeFs {
          ctime,
          origKey
        );
-      } else if (this.supportNativePartial) {
-        return await this._writeFileFromRootNativePartial(
+      } else if (this.supportApachePartial) {
+        return await this._writeFileFromRootApachePartial(
+          key,
+          content,
+          mtime,
+          ctime,
+          origKey
+        );
+      } else if (this.supportSabrePartial) {
+        return await this._writeFileFromRootSabrePartial(
          key,
          content,
          mtime,
@@ -608,7 +635,7 @@ export class FakeFsWebdav extends FakeFs {
      console.error(e);
      throw e;
      // this.isNextcloud = false;
-      // this.supportNativePartial = false;
+      // this.supportApachePartial = false;
      // return await this._writeFileFromRootFull(
      //   key,
      //   content,
@@ -676,8 +703,8 @@ export class FakeFsWebdav extends FakeFs {
    console.debug(`tmpFolderName=${tmpFolderName}`);

    const clientForUpload = createClient(uploadServerAddress, {
-      username: this.webdavConfig.username,
-      password: this.webdavConfig.password,
+      username: tryEncodeUsernamePassword(this.webdavConfig.username),
+      password: tryEncodeUsernamePassword(this.webdavConfig.password),
      headers: {
        "Cache-Control": "no-cache",
      },
@@ -753,7 +780,7 @@ export class FakeFsWebdav extends FakeFs {
    return k;
  }

-  async _writeFileFromRootNativePartial(
+  async _writeFileFromRootApachePartial(
    key: string,
    content: ArrayBuffer,
    mtime: number,
@@ -775,6 +802,8 @@ export class FakeFsWebdav extends FakeFs {
      content.byteLength,
      sizePerChunk
    );
+
+    // TODO: parallel
    for (let i = 0; i < chunkRanges.length; ++i) {
      const { start, end } = chunkRanges[i];
      await this.client.partialUpdateFileContents(
@@ -789,6 +818,47 @@ export class FakeFsWebdav extends FakeFs {
    return await this.stat(origKey);
  }

+  async _writeFileFromRootSabrePartial(
+    key: string,
+    content: ArrayBuffer,
+    mtime: number,
+    ctime: number,
+    origKey: string
+  ): Promise<Entity> {
+    // firstly upload a 0-byte data
+    await this._writeFileFromRootFull(
+      key,
+      new ArrayBuffer(0),
+      mtime,
+      ctime,
+      origKey
+    );
+
+    // then "update" by chunks
+    const sizePerChunk = 5 * 1024 * 1024; // 5 mb
+    const chunkRanges = splitFileSizeToChunkRanges(
+      content.byteLength,
+      sizePerChunk
+    );
+
+    // diff from apachePartial: we use "append" header here for dufs...
+    // we cannot parallel here
+    for (let i = 0; i < chunkRanges.length; ++i) {
+      const { start, end } = chunkRanges[i];
+      await this.client.customRequest(key, {
+        method: "PATCH",
+        headers: {
+          "X-Update-Range": "append",
+          "Content-Type": "application/x-sabredav-partialupdate",
+        },
+        data: content.slice(start, end + 1),
+      });
+    }
+
+    // lastly return
+    return await this.stat(origKey);
+  }
+
  async readFile(key: string): Promise<ArrayBuffer> {
    if (key.endsWith("/")) {
      throw Error(`you should not call readFile on ${key}`);
@@ -853,17 +923,15 @@ export class FakeFsWebdav extends FakeFs {
      await this._init();
      const results = await this._statFromRoot(`/${this.remoteBaseDir}/`);
      if (results === undefined) {
-        const err = "results is undefined";
-        console.error(err);
-        callbackFunc?.(err);
-        return false;
+        throw Error("cannot stat root vault folder!");
      }
-      return true;
    } catch (err) {
      console.error(err);
      callbackFunc?.(err);
      return false;
    }
+
+    return await this.checkConnectCommonOps(callbackFunc);
  }

  async getUserDisplayName(): Promise<string> {
@@ -252,12 +252,16 @@ export class FakeFsWebdis extends FakeFs {
      const k = await (
        await this._fetchCommand("GET", "PING/helloworld")
      ).json();
-      return isEqual(k, { PING: "helloworld" });
+      if (!isEqual(k, { PING: "helloworld" })) {
+        throw Error(`no correct ping response`);
+      }
    } catch (err: any) {
      console.error(err);
      callbackFunc?.(err);
      return false;
    }
+    return await this.checkConnectCommonOps(callbackFunc);
+    // return true;
  }

  async getUserDisplayName(): Promise<string> {
Author	SHA1	Message	Date
fyears	d42b31cea2	update node ci Release A New Version / build (20.x) (push) Failing after 51s Details	2024-07-27 23:08:49 +08:00
fyears	69e0c77444	bump to 0.5.16	2024-07-27 22:58:36 +08:00
fyears	ee4394873b	upgrade check connections	2024-07-27 22:58:05 +08:00
fyears	f2845cdcbf	finally we can switch from ts-node to tsx	2024-07-27 22:55:04 +08:00
fyears	bbee873b64	remove debug log	2024-07-27 22:08:22 +08:00
fyears	f5fee16130	bump to 0.5.15	2024-07-27 19:38:25 +08:00
fyears	f4b82204c1	update packages	2024-07-27 19:37:55 +08:00
fyears	f1a06a3f52	add append mode for sabre	2024-07-27 19:22:57 +08:00
fyears	ae4e67f3b4	fix smart conflict	2024-07-27 19:09:42 +08:00
fyears	254231e4bd	add onedrive mtime error	2024-07-27 15:02:21 +08:00
fyears	871e8dc3cf	encode username	2024-07-27 14:41:37 +08:00
fyears	85aed7cfb9	dufs	2024-07-27 14:41:26 +08:00
fyears	3f7a6dbf6a	Merge branch 'master' of https://github.com/fyears/remotely-save	2024-07-27 12:37:31 +08:00
fyears	13cca9025e	hard code famous website	2024-07-27 12:30:17 +08:00
Vincent Whizin	a81af9ccec	Update docs on "large" files sync issues (#753 )	2024-07-18 13:14:06 +08:00
fyears	db86a75534	Add vConsole tutorial	2024-07-14 22:31:08 +08:00
fyears	72e059d5cc	description of config folder and bookmarks	2024-07-14 21:09:09 +08:00