0.0.11

check password
second confirm for password changes
2021-11-08 00:40:45 +08:00 · 2021-11-08 00:39:17 +08:00 · 2021-11-08 00:17:42 +08:00 · 2021-11-07 17:15:06 +08:00 · 2021-11-07 16:02:03 +08:00 · 2021-11-07 15:44:14 +08:00
27 changed files with 2235 additions and 369 deletions
@@ -0,0 +1,28 @@
+# EditorConfig is awesome: https://EditorConfig.org
+
+# top-most EditorConfig file
+root = true
+
+# Unix-style newlines with a newline ending every file
+[*]
+end_of_line = lf
+insert_final_newline = true
+
+# Matches multiple files with brace expansion notation
+# Set default charset
+[*.{js,jsx,mjs,cjs,ts,tsx,json,py}]
+charset = utf-8
+
+# 4 space indentation
+[*.py]
+indent_style = space
+indent_size = 4
+
+# Tab indentation (no size specified)
+[Makefile]
+indent_style = tab
+
+# Indentation override for all JS under lib directory
+[*.{js,jsx,mjs,cjs,ts,tsx,json,py}]
+indent_style = space
+indent_size = 2
@@ -0,0 +1,5 @@
+*.png filter=lfs diff=lfs merge=lfs -text
+*.jpg filter=lfs diff=lfs merge=lfs -text
+*.jpeg filter=lfs diff=lfs merge=lfs -text
+*.pdf filter=lfs diff=lfs merge=lfs -text
+*.enc filter=lfs diff=lfs merge=lfs -text
@@ -0,0 +1,40 @@
+# This workflow will do a clean install of node dependencies, cache/restore them, build the source code and run tests across different versions of node
+# For more information see: https://help.github.com/actions/language-and-framework-guides/using-nodejs-with-github-actions
+
+name: BuildCI
+
+on:
+  push:
+    branches: [master]
+  pull_request:
+    branches: [master]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    strategy:
+      matrix:
+        node-version: [16.x]
+        # See supported Node.js release schedule at https://nodejs.org/en/about/releases/
+
+    steps:
+      - name: Checkout codes
+        uses: actions/checkout@v2
+        with:
+          lfs: true
+      - name: Checkout LFS
+        run: git lfs checkout
+      - name: Use Node.js ${{ matrix.node-version }}
+        uses: actions/setup-node@v2
+        with:
+          node-version: ${{ matrix.node-version }}
+      - run: npm install
+      - run: npm test
+      - run: npm run build
+      - uses: actions/upload-artifact@v2
+        with:
+          name: my-dist
+          path: |
+            main.js
+            manifest.json
@@ -0,0 +1,2 @@
+node_modules/
+main.js
@@ -2,6 +2,14 @@

 This is yet another sync plugin for Obsidian.

+## Download and Install
+
+[![BuildCI](https://github.com/fyears/obsidian-save-remote/actions/workflows/auto-build.yml/badge.svg)](https://github.com/fyears/obsidian-save-remote/actions/workflows/auto-build.yml)
+
+Every artifacts are placed in the "Summary" under every successful builds.
+
+Besides manually download the files, you can also use [Obsidian42 - BRAT](https://github.com/TfTHacker/obsidian42-brat) to install this plugin.
+
 ## Disclaimer

 **This is NOT the official sync service provided by Obsidian.**
@@ -14,5 +22,5 @@ As of October 2021, the plugin is under development. **DO NOT USE IT for any ser

 Remote service in support list:

- [ ] AWS S3
+- [x] AWS S3
 - [ ] webdav
@@ -0,0 +1,29 @@
+# Algorithm
+
+## Sources
+
+We have three record sources:
+
+1. Local files
+2. Remote files
+3. Local "delete-or-rename" history.
+
+Assuming all sources are reliable.
+
+## Deal with them
+
+We list all combinations mutually exclusive and collectively exhaustive.
+
+| ID  | Remote Files | Local files | Local delete rename history | Extra                                                                         | Decision                                                                                                 |
+| --- | ------------ | ----------- | --------------------------- | ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
+| 1   | exist        | exist       | ignore                      | mtime_remote > mtime_local                                                    | download remote file, create local folder if not exists, clear local history if exists                   |
+| 2   | exist        | exist       | ignore                      | mtime_remote < mtime_local                                                    | upload local file, create remote folder if not exists, clear local history if exists                     |
+| 3   | exist        | exist       | ignore                      | mtime_remote === mtime_local && password === "" && size_remote === size_local | clear local history if exists (the file was synced and no changes after last sync)                       |
+| 4   | exist        | exist       | ignore                      | mtime_remote === mtime_local && password === "" && size_remote !== size_local | upload local file, clear local history if exists (we always prefer local to remote)                      |
+| 5   | exist        | exist       | ignore                      | mtime_remote === mtime_local && password !== ""                               | clear local history if exists (in encryption mode, file sizes are unequal. we can only rely on mtime(s)) |
+| 6   | exist        | exist       | ignore                      | If local is a folder. mtime_local === undefined                               | clear local history if exists. TODO: what if a folder and a previous file share the same name?           |
+| 7   | exist        | not exist   | exist                       | mtime_remote >= delete_time_local                                             | download remote file, create folder if not exists                                                        |
+| 8   | exist        | not exist   | exist                       | mtime_remote < delete_time_local                                              | delete remote file, clear local history                                                                  |
+| 9   | exist        | not exist   | not exist                   |                                                                               | download remote file, create folder if not exists                                                        |
+| 10  | not exist    | exist       | ignore                      | local may be folder or file                                                   | upload local files recursively, create remote folder if not exists, clear local history if exists        |
+| 11  | not exist    | not exist   | ignore                      |                                                                               | clear local history if exists                                                                            |
@@ -0,0 +1,13 @@
+# Code Design
+
+## Code Organization
+
+1. Every function except `main.ts` should be pure. Pass any stateful information in parameters.
+
+2. `misc.ts` should not depend on any other written code.
+
+3. Each storage code should not depend on `sync.ts`.
+
+## File and Folder Representation
+
+While writing sync codes, folders are always represented by a string ending with `/`.
@@ -1,340 +0,0 @@
-import * as path from "path";
-import * as fs from "fs";
-import { Buffer } from "buffer";
-import { Readable } from "stream";
-import * as mime from "mime-types";
-import {
-  App,
-  Modal,
-  Notice,
-  Plugin,
-  PluginSettingTab,
-  Setting,
-  request,
-  Platform,
-} from "obsidian";
-import * as CodeMirror from "codemirror";
-
-import {
-  S3Client,
-  ListObjectsCommand,
-  PutObjectCommand,
-  GetObjectCommand,
-} from "@aws-sdk/client-s3";
-
-interface SaveRemotePluginSettings {
-  s3Endpoint: string;
-  s3Region: string;
-  s3AccessKeyID: string;
-  s3SecretAccessKey: string;
-  s3BucketName: string;
-}
-
-const DEFAULT_SETTINGS: SaveRemotePluginSettings = {
-  s3Endpoint: "",
-  s3Region: "",
-  s3AccessKeyID: "",
-  s3SecretAccessKey: "",
-  s3BucketName: "",
-};
-
-const ignoreHiddenFiles = (item: string) => {
-  const basename = path.basename(item);
-  return basename === "." || basename[0] !== ".";
-};
-
-/**
- * Util func for mkdir -p based on the "path" of original file or folder
- * "a/b/c/" => ["a", "a/b", "a/b/c"]
- * "a/b/c/d/e.txt" => ["a", "a/b", "a/b/c", "a/b/c/d"]
- * @param x string
- * @returns string[] might be empty
- */
-const getFolderLevels = (x: string) => {
-  const res: string[] = [];
-
-  if (x === "" || x === "/") {
-    return res;
-  }
-
-  const y1 = x.split("/");
-  let i = 0;
-  for (let index = 0; index + 1 < y1.length; index++) {
-    res.push(y1.slice(0, index + 1).join("/"));
-  }
-  return res;
-};
-
-/**
- * https://stackoverflow.com/questions/8609289
- * @param b Buffer
- * @returns ArrayBuffer
- */
-const bufferToArrayBuffer = (b: Buffer) => {
-  return b.buffer.slice(b.byteOffset, b.byteOffset + b.byteLength);
-};
-
-/**
- * The Body of resp of aws GetObject has mix types
- * and we want to get ArrayBuffer here.
- * See https://github.com/aws/aws-sdk-js-v3/issues/1877
- * @param b The Body of GetObject
- * @returns Promise<ArrayBuffer>
- */
-const getObjectBodyToArrayBuffer = async (
-  b: Readable | ReadableStream | Blob
-) => {
-  if (b instanceof Readable) {
-    const chunks: Uint8Array[] = [];
-    for await (let chunk of b) {
-      chunks.push(chunk);
-    }
-    const buf = Buffer.concat(chunks);
-    return bufferToArrayBuffer(buf);
-  } else if (b instanceof ReadableStream) {
-    return await new Response(b, {}).arrayBuffer();
-  } else if (b instanceof Blob) {
-    return await b.arrayBuffer();
-  } else {
-    throw TypeError(`The type of ${b} is not one of the supported types`);
-  }
-};
-
-export default class SaveRemotePlugin extends Plugin {
-  settings: SaveRemotePluginSettings;
-  cm: CodeMirror.Editor;
-
-  async onload() {
-    console.log("loading plugin obsidian-save-remote");
-
-    await this.loadSettings();
-
-    this.addRibbonIcon("right-arrow-with-tail", "Upload", async () => {
-      // console.log(this.app.vault.getFiles());
-      // console.log(this.app.vault.getAllLoadedFiles());
-      new Notice(`Upload begun.`);
-      const allFilesAndFolders = this.app.vault.getAllLoadedFiles();
-
-      const s3Client = new S3Client({
-        region: this.settings.s3Region,
-        endpoint: this.settings.s3Endpoint,
-        credentials: {
-          accessKeyId: this.settings.s3AccessKeyID,
-          secretAccessKey: this.settings.s3SecretAccessKey,
-        },
-      });
-
-      try {
-        for (const fileOrFolder of allFilesAndFolders) {
-          if (fileOrFolder.path === "/") {
-            console.log('ignore "/"');
-          } else if ("children" in fileOrFolder) {
-            // folder
-            console.log(`folder ${fileOrFolder.path}/`);
-            new Notice(`folder ${fileOrFolder.path}/`);
-
-            const results = await s3Client.send(
-              new PutObjectCommand({
-                Bucket: this.settings.s3BucketName,
-                Key: `${fileOrFolder.path}/`,
-                Body: "",
-              })
-            );
-          } else {
-            // file
-            console.log(`file ${fileOrFolder.path}`);
-            const arrContent = await this.app.vault.adapter.readBinary(
-              fileOrFolder.path
-            );
-            new Notice(`file ${fileOrFolder.path}`);
-            const contentType =
-              mime.contentType(
-                mime.lookup(`${fileOrFolder.path}`) || undefined
-              ) || undefined;
-            // console.log(contentType);
-            const results = await s3Client.send(
-              new PutObjectCommand({
-                Bucket: this.settings.s3BucketName,
-                Key: `${fileOrFolder.path}`,
-                Body: Buffer.from(arrContent),
-                ContentType: contentType,
-              })
-            );
-          }
-        }
-        new Notice("Upload finished!");
-      } catch (err) {
-        console.log("Error", err);
-        new Notice(`${err}`);
-      }
-    });
-
-    this.addRibbonIcon("left-arrow-with-tail", "Download", async () => {
-      const allFilesAndFolders = this.app.vault.getAllLoadedFiles();
-
-      const s3Client = new S3Client({
-        region: this.settings.s3Region,
-        endpoint: this.settings.s3Endpoint,
-        credentials: {
-          accessKeyId: this.settings.s3AccessKeyID,
-          secretAccessKey: this.settings.s3SecretAccessKey,
-        },
-      });
-
-      try {
-        const listObj = await s3Client.send(
-          new ListObjectsCommand({ Bucket: this.settings.s3BucketName })
-        );
-
-        for (const singleContent of listObj.Contents) {
-          const foldersToBuild = getFolderLevels(singleContent.Key);
-          for (const folder of foldersToBuild) {
-            const r = await this.app.vault.adapter.exists(folder);
-            if (!r) {
-              console.log(`mkdir ${folder}`);
-              new Notice(`mkdir ${folder}`);
-              await this.app.vault.adapter.mkdir(folder);
-            }
-          }
-
-          if (singleContent.Key.endsWith("/")) {
-            // kind of a folder
-            // pass
-          } else {
-            // kind of a file
-            // download
-
-            console.log(`download file ${singleContent.Key}`);
-            new Notice(`download file ${singleContent.Key}`);
-
-            const data = await s3Client.send(
-              new GetObjectCommand({
-                Bucket: this.settings.s3BucketName,
-                Key: singleContent.Key,
-              })
-            );
-            const bodyContents = await getObjectBodyToArrayBuffer(data.Body);
-            await this.app.vault.adapter.writeBinary(
-              singleContent.Key,
-              bodyContents
-            );
-          }
-        }
-
-        new Notice("Download finished!");
-      } catch (err) {
-        console.log("Error", err);
-        new Notice(`${err}`);
-      }
-    });
-
-    this.addSettingTab(new SaveRemoteSettingTab(this.app, this));
-
-    this.registerCodeMirror((cm: CodeMirror.Editor) => {
-      this.cm = cm;
-      console.log("codemirror registered.");
-    });
-
-    // this.registerDomEvent(document, "click", (evt: MouseEvent) => {
-    //   console.log("click", evt);
-    // });
-
-    // this.registerInterval(
-    //   window.setInterval(() => console.log("setInterval"), 5 * 60 * 1000)
-    // );
-  }
-
-  onunload() {
-    console.log("unloading plugin obsidian-save-remote");
-  }
-
-  async loadSettings() {
-    this.settings = Object.assign({}, DEFAULT_SETTINGS, await this.loadData());
-  }
-
-  async saveSettings() {
-    await this.saveData(this.settings);
-  }
-}
-
-class SaveRemoteSettingTab extends PluginSettingTab {
-  plugin: SaveRemotePlugin;
-
-  constructor(app: App, plugin: SaveRemotePlugin) {
-    super(app, plugin);
-    this.plugin = plugin;
-  }
-
-  display(): void {
-    let { containerEl } = this;
-
-    containerEl.empty();
-
-    containerEl.createEl("h2", { text: "Settings for Save Remote" });
-
-    new Setting(containerEl)
-      .setName("s3Endpoint")
-      .setDesc("s3Endpoint")
-      .addText((text) =>
-        text
-          .setPlaceholder("")
-          .setValue(this.plugin.settings.s3Endpoint)
-          .onChange(async (value) => {
-            this.plugin.settings.s3Endpoint = value;
-            await this.plugin.saveSettings();
-          })
-      );
-
-    new Setting(containerEl)
-      .setName("s3Region")
-      .setDesc("s3Region")
-      .addText((text) =>
-        text
-          .setPlaceholder("")
-          .setValue(`${this.plugin.settings.s3Region}`)
-          .onChange(async (value) => {
-            this.plugin.settings.s3Region = value;
-            await this.plugin.saveSettings();
-          })
-      );
-
-    new Setting(containerEl)
-      .setName("s3AccessKeyID")
-      .setDesc("s3AccessKeyID")
-      .addText((text) =>
-        text
-          .setPlaceholder("")
-          .setValue(`${this.plugin.settings.s3AccessKeyID}`)
-          .onChange(async (value) => {
-            this.plugin.settings.s3AccessKeyID = value;
-            await this.plugin.saveSettings();
-          })
-      );
-
-    new Setting(containerEl)
-      .setName("s3SecretAccessKey")
-      .setDesc("s3SecretAccessKey")
-      .addText((text) =>
-        text
-          .setPlaceholder("")
-          .setValue(`${this.plugin.settings.s3SecretAccessKey}`)
-          .onChange(async (value) => {
-            this.plugin.settings.s3SecretAccessKey = value;
-            await this.plugin.saveSettings();
-          })
-      );
-
-    new Setting(containerEl)
-      .setName("s3BucketName")
-      .setDesc("s3BucketName")
-      .addText((text) =>
-        text
-          .setPlaceholder("")
-          .setValue(`${this.plugin.settings.s3BucketName}`)
-          .onChange(async (value) => {
-            this.plugin.settings.s3BucketName = value;
-            await this.plugin.saveSettings();
-          })
-      );
-  }
-}
@@ -1,7 +1,7 @@
 {
  "id": "obsdian-save-remote",
  "name": "Save remote",
-  "version": "0.0.1",
+  "version": "0.0.11",
  "minAppVersion": "0.12.15",
  "description": "This is yet another plugin allowing users to sync notes between local device and the cloud.",
  "author": "fyears",
@@ -1,23 +1,30 @@
 {
  "name": "obsidian-save-remote",
-  "version": "0.0.1",
+  "version": "0.0.11",
  "description": "This is yet another sync plugin for Obsidian app.",
  "scripts": {
    "dev": "webpack --mode development --watch",
    "build": "webpack --mode production",
    "format": "npx prettier --write .",
-    "clean": "npx rimraf main.js"
+    "clean": "npx rimraf main.js",
+    "test": "cross-env TS_NODE_COMPILER_OPTIONS={\\\"module\\\":\\\"commonjs\\\"} mocha -r ts-node/register 'tests/**/*.ts'"
  },
  "source": "main.ts",
  "keywords": [],
  "author": "",
  "license": "Apache-2.0",
  "devDependencies": {
+    "@types/chai": "^4.2.22",
+    "@types/mime-types": "^2.1.1",
+    "@types/mocha": "^9.0.0",
    "@types/node": "^14.14.37",
-    "parcel": "^2.0.0",
+    "chai": "^4.3.4",
+    "cross-env": "^7.0.3",
+    "mocha": "^9.1.3",
    "prettier": "^2.4.1",
    "terser-webpack-plugin": "^5.2.4",
    "ts-loader": "^9.2.6",
+    "ts-node": "^10.4.0",
    "tslib": "^2.2.0",
    "typescript": "^4.4.4",
    "webdav-server": "^2.6.2",
@@ -27,12 +34,18 @@
  "dependencies": {
    "@aws-sdk/client-s3": "^3.37.0",
    "@aws-sdk/signature-v4-crt": "^3.37.0",
-    "@types/mime-types": "^2.1.1",
+    "acorn": "^8.5.0",
    "aws-crt": "^1.10.1",
+    "buffer": "^6.0.3",
    "codemirror": "^5.63.1",
+    "lovefield-ts": "^0.7.0",
    "mime-types": "^2.1.33",
    "obsidian": "^0.12.0",
+    "path-browserify": "^1.0.1",
+    "process": "^0.11.10",
+    "rfc4648": "^1.5.0",
    "rimraf": "^3.0.2",
+    "stream-browserify": "^3.0.0",
    "webdav": "^4.7.0",
    "webdav-fs": "^4.0.0"
  }
@@ -0,0 +1,37 @@
+import { TAbstractFile, TFolder, TFile, Vault } from "obsidian";
+
+import * as lf from "lovefield-ts/dist/es6/lf.js";
+
+import type { SyncPlanType } from "./sync";
+import {
+  insertSyncPlanRecord,
+  clearAllSyncPlanRecords,
+  readAllSyncPlanRecordTexts,
+} from "./localdb";
+import { mkdirpInVault } from "./misc";
+
+const DEFAULT_DEBUG_FOLDER = "_debug_save_remote/";
+const DEFAULT_SYNC_PLANS_HISTORY_FILE_PREFIX = "sync_plans_hist_exported_on_";
+
+export const exportSyncPlansToFiles = async (
+  db: lf.DatabaseConnection,
+  vault: Vault
+) => {
+  console.log("exporting");
+  await mkdirpInVault(DEFAULT_DEBUG_FOLDER, vault);
+  const records = await readAllSyncPlanRecordTexts(db);
+  let md = "";
+  if (records.length === 0) {
+    md = "No sync plans history found";
+  } else {
+    md =
+      "Sync plans found:\n\n" +
+      records.map((x) => "```json\n" + x + "\n```\n").join("\n");
+  }
+  const ts = Date.now();
+  const filePath = `${DEFAULT_DEBUG_FOLDER}${DEFAULT_SYNC_PLANS_HISTORY_FILE_PREFIX}${ts}.md`;
+  await vault.create(filePath, md, {
+    mtime: ts,
+  });
+  console.log("finish exporting");
+};
@@ -0,0 +1,138 @@
+import { base32, base64 } from "rfc4648";
+import {
+  bufferToArrayBuffer,
+  arrayBufferToBuffer,
+  hexStringToTypedArray,
+  arrayBufferToHex,
+} from "./misc";
+
+const DEFAULT_ITER = 10000;
+
+// base32.stringify(Buffer.from('Salted__'))
+export const MAGIC_ENCRYPTED_PREFIX_BASE32 = "KNQWY5DFMRPV";
+
+const getKeyIVFromPassword = async (
+  salt: Uint8Array,
+  password: string,
+  rounds: number = DEFAULT_ITER
+) => {
+  const k1 = await window.crypto.subtle.importKey(
+    "raw",
+    new TextEncoder().encode(password),
+    { name: "PBKDF2" },
+    false,
+    ["deriveKey", "deriveBits"]
+  );
+
+  const k2 = await window.crypto.subtle.deriveBits(
+    {
+      name: "PBKDF2",
+      salt: salt,
+      iterations: rounds,
+      hash: "SHA-256",
+    },
+    k1,
+    256 + 128
+  );
+
+  return k2;
+};
+
+export const encryptArrayBuffer = async (
+  arrBuf: ArrayBuffer,
+  password: string,
+  rounds: number = DEFAULT_ITER,
+  saltHex: string = ""
+) => {
+  let salt: Uint8Array;
+  if (saltHex !== "") {
+    salt = hexStringToTypedArray(saltHex);
+  } else {
+    salt = window.crypto.getRandomValues(new Uint8Array(8));
+  }
+
+  const derivedKey = await getKeyIVFromPassword(salt, password, rounds);
+  const key = derivedKey.slice(0, 32);
+  const iv = derivedKey.slice(32, 32 + 16);
+
+  const keyCrypt = await window.crypto.subtle.importKey(
+    "raw",
+    key,
+    { name: "AES-CBC" },
+    false,
+    ["encrypt", "decrypt"]
+  );
+
+  const enc = (await window.crypto.subtle.encrypt(
+    { name: "AES-CBC", iv },
+    keyCrypt,
+    arrBuf
+  )) as ArrayBuffer;
+
+  const prefix = new TextEncoder().encode("Salted__");
+
+  const res = new Uint8Array([...prefix, ...salt, ...new Uint8Array(enc)]);
+
+  return bufferToArrayBuffer(res);
+};
+
+export const decryptArrayBuffer = async (
+  arrBuf: ArrayBuffer,
+  password: string,
+  rounds: number = DEFAULT_ITER
+) => {
+  const prefix = arrBuf.slice(0, 8);
+  const salt = arrBuf.slice(8, 16);
+  const derivedKey = await getKeyIVFromPassword(
+    new Uint8Array(salt),
+    password,
+    rounds
+  );
+  const key = derivedKey.slice(0, 32);
+  const iv = derivedKey.slice(32, 32 + 16);
+
+  const keyCrypt = await window.crypto.subtle.importKey(
+    "raw",
+    key,
+    { name: "AES-CBC" },
+    false,
+    ["encrypt", "decrypt"]
+  );
+
+  const dec = (await window.crypto.subtle.decrypt(
+    { name: "AES-CBC", iv },
+    keyCrypt,
+    arrBuf.slice(16)
+  )) as ArrayBuffer;
+
+  return dec;
+};
+
+export const encryptStringToBase32 = async (
+  text: string,
+  password: string,
+  rounds: number = DEFAULT_ITER,
+  saltHex: string = ""
+) => {
+  const enc = await encryptArrayBuffer(
+    bufferToArrayBuffer(new TextEncoder().encode(text)),
+    password,
+    rounds,
+    saltHex
+  );
+  return base32.stringify(new Uint8Array(enc));
+};
+
+export const decryptBase32ToString = async (
+  text: string,
+  password: string,
+  rounds: number = DEFAULT_ITER
+) => {
+  return new TextDecoder().decode(
+    await decryptArrayBuffer(
+      bufferToArrayBuffer(base32.parse(text)),
+      password,
+      rounds
+    )
+  );
+};
@@ -0,0 +1,316 @@
+import * as lf from "lovefield-ts/dist/es6/lf.js";
+import { TAbstractFile, TFile, TFolder } from "obsidian";
+
+import type { SUPPORTED_SERVICES_TYPE } from "./misc";
+import type { SyncPlanType } from "./sync";
+
+export type DatabaseConnection = lf.DatabaseConnection;
+
+export const DEFAULT_DB_NAME = "saveremotedb";
+export const DEFAULT_TBL_DELETE_HISTORY = "filefolderoperationhistory";
+export const DEFAULT_TBL_SYNC_MAPPING = "syncmetadatahistory";
+export const DEFAULT_SYNC_PLANS_HISTORY = "syncplanshistory";
+
+export interface FileFolderHistoryRecord {
+  key: string;
+  ctime: number;
+  mtime: number;
+  size: number;
+  action_when: number;
+  action_type: "delete" | "rename";
+  key_type: "folder" | "file";
+  rename_to: string;
+}
+
+export interface SyncMetaMappingRecord {
+  local_key: string;
+  remote_key: string;
+  local_size: number;
+  remote_size: number;
+  local_mtime: number;
+  remote_mtime: number;
+  remote_extra_key: string;
+  remote_type: SUPPORTED_SERVICES_TYPE;
+  key_type: "folder" | "file";
+}
+
+interface SyncPlanRecord {
+  ts: number;
+  remote_type: string;
+  sync_plan: string;
+}
+
+export const prepareDBs = async () => {
+  const schemaBuilder = lf.schema.create(DEFAULT_DB_NAME, 1);
+  schemaBuilder
+    .createTable(DEFAULT_TBL_DELETE_HISTORY)
+    .addColumn("id", lf.Type.INTEGER)
+    .addColumn("key", lf.Type.STRING)
+    .addColumn("ctime", lf.Type.INTEGER)
+    .addColumn("mtime", lf.Type.INTEGER)
+    .addColumn("size", lf.Type.INTEGER)
+    .addColumn("action_when", lf.Type.INTEGER)
+    .addColumn("action_type", lf.Type.STRING)
+    .addColumn("key_type", lf.Type.STRING)
+    .addPrimaryKey(["id"], true)
+    .addIndex("idxKey", ["key"]);
+
+  schemaBuilder
+    .createTable(DEFAULT_TBL_SYNC_MAPPING)
+    .addColumn("id", lf.Type.INTEGER)
+    .addColumn("local_key", lf.Type.STRING)
+    .addColumn("remote_key", lf.Type.STRING)
+    .addColumn("local_size", lf.Type.INTEGER)
+    .addColumn("remote_size", lf.Type.INTEGER)
+    .addColumn("local_mtime", lf.Type.INTEGER)
+    .addColumn("remote_mtime", lf.Type.INTEGER)
+    .addColumn("key_type", lf.Type.STRING)
+    .addColumn("remote_extra_key", lf.Type.STRING)
+    .addColumn("remote_type", lf.Type.STRING)
+    .addNullable([
+      "remote_extra_key",
+      "remote_mtime",
+      "remote_size",
+      "local_mtime",
+    ])
+    .addPrimaryKey(["id"], true)
+    .addIndex("idxkey", ["local_key", "remote_key"]);
+
+  schemaBuilder
+    .createTable(DEFAULT_SYNC_PLANS_HISTORY)
+    .addColumn("id", lf.Type.INTEGER)
+    .addColumn("ts", lf.Type.INTEGER)
+    .addColumn("remote_type", lf.Type.STRING)
+    .addColumn("sync_plan", lf.Type.STRING)
+    .addPrimaryKey(["id"], true)
+    .addIndex("tskey", ["ts"]);
+
+  const db = await schemaBuilder.connect({
+    storeType: lf.DataStoreType.INDEXED_DB,
+  });
+  console.log("db connected");
+  return db;
+};
+
+export const destroyDBs = async (db: lf.DatabaseConnection) => {
+  db.close();
+  const req = indexedDB.deleteDatabase(DEFAULT_DB_NAME);
+  req.onsuccess = (event) => {
+    console.log("db deleted");
+  };
+  req.onblocked = (event) => {
+    console.warn("trying to delete db but it was blocked");
+  };
+  req.onerror = (event) => {
+    console.error("tried to delete db but something bad!");
+    console.error(event);
+  };
+};
+
+export const loadDeleteRenameHistoryTable = async (
+  db: lf.DatabaseConnection
+) => {
+  const schema = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
+  const tbl = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
+
+  const records = await db
+    .select()
+    .from(schema)
+    .orderBy(schema.col("action_when"), lf.Order.ASC)
+    .exec();
+
+  return records as FileFolderHistoryRecord[];
+};
+
+export const clearDeleteRenameHistoryOfKey = async (
+  db: lf.DatabaseConnection,
+  key: string
+) => {
+  const schema = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
+  const tbl = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
+
+  await db.delete().from(tbl).where(tbl.col("key").eq(key)).exec();
+};
+
+export const insertDeleteRecord = async (
+  db: lf.DatabaseConnection,
+  fileOrFolder: TAbstractFile
+) => {
+  const schema = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
+  const tbl = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
+  // console.log(fileOrFolder);
+  let k: FileFolderHistoryRecord;
+  if (fileOrFolder instanceof TFile) {
+    k = {
+      key: fileOrFolder.path,
+      ctime: fileOrFolder.stat.ctime,
+      mtime: fileOrFolder.stat.mtime,
+      size: fileOrFolder.stat.size,
+      action_when: Date.now(),
+      action_type: "delete",
+      key_type: "file",
+      rename_to: "",
+    };
+  } else if (fileOrFolder instanceof TFolder) {
+    // key should endswith "/"
+    const key = fileOrFolder.path.endsWith("/")
+      ? fileOrFolder.path
+      : `${fileOrFolder.path}/`;
+    k = {
+      key: key,
+      ctime: 0,
+      mtime: 0,
+      size: 0,
+      action_when: Date.now(),
+      action_type: "delete",
+      key_type: "folder",
+      rename_to: "",
+    };
+  }
+  const row = tbl.createRow(k);
+  await db.insertOrReplace().into(tbl).values([row]).exec();
+};
+
+export const insertRenameRecord = async (
+  db: lf.DatabaseConnection,
+  fileOrFolder: TAbstractFile,
+  oldPath: string
+) => {
+  const schema = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
+  const tbl = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
+  // console.log(fileOrFolder);
+  let k: FileFolderHistoryRecord;
+  if (fileOrFolder instanceof TFile) {
+    k = {
+      key: oldPath,
+      ctime: fileOrFolder.stat.ctime,
+      mtime: fileOrFolder.stat.mtime,
+      size: fileOrFolder.stat.size,
+      action_when: Date.now(),
+      action_type: "rename",
+      key_type: "file",
+      rename_to: fileOrFolder.path,
+    };
+  } else if (fileOrFolder instanceof TFolder) {
+    const key = oldPath.endsWith("/") ? oldPath : `${oldPath}/`;
+    const renameTo = fileOrFolder.path.endsWith("/")
+      ? fileOrFolder.path
+      : `${fileOrFolder.path}/`;
+    k = {
+      key: key,
+      ctime: 0,
+      mtime: 0,
+      size: 0,
+      action_when: Date.now(),
+      action_type: "rename",
+      key_type: "folder",
+      rename_to: renameTo,
+    };
+  }
+  const row = tbl.createRow(k);
+  await db.insertOrReplace().into(tbl).values([row]).exec();
+};
+
+export const getAllDeleteRenameRecords = async (db: lf.DatabaseConnection) => {
+  const schema = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
+  const res1 = await db.select().from(schema).exec();
+  const res2 = res1 as FileFolderHistoryRecord[];
+  return res2;
+};
+
+export const upsertSyncMetaMappingDataS3 = async (
+  db: lf.DatabaseConnection,
+  localKey: string,
+  localMTime: number,
+  localSize: number,
+  remoteKey: string,
+  remoteMTime: number,
+  remoteSize: number,
+  remoteExtraKey: string /* ETag from s3 */
+) => {
+  const schema = db.getSchema().table(DEFAULT_TBL_SYNC_MAPPING);
+  const aggregratedInfo: SyncMetaMappingRecord = {
+    local_key: localKey,
+    local_mtime: localMTime,
+    local_size: localSize,
+    remote_key: remoteKey,
+    remote_mtime: remoteMTime,
+    remote_size: remoteSize,
+    remote_extra_key: remoteExtraKey,
+    remote_type: "s3",
+    key_type: localKey.endsWith("/") ? "folder" : "file",
+  };
+  const row = schema.createRow(aggregratedInfo);
+  await db.insertOrReplace().into(schema).values([row]).exec();
+};
+
+export const getSyncMetaMappingByRemoteKeyS3 = async (
+  db: lf.DatabaseConnection,
+  remoteKey: string,
+  remoteMTime: number,
+  remoteExtraKey: string
+) => {
+  const schema = db.getSchema().table(DEFAULT_TBL_SYNC_MAPPING);
+  const tbl = db.getSchema().table(DEFAULT_TBL_SYNC_MAPPING);
+  const res = (await db
+    .select()
+    .from(tbl)
+    .where(
+      lf.op.and(
+        tbl.col("remote_key").eq(remoteKey),
+        tbl.col("remote_mtime").eq(remoteMTime),
+        tbl.col("remote_extra_key").eq(remoteExtraKey),
+        tbl.col("remote_type").eq("s3")
+      )
+    )
+    .exec()) as SyncMetaMappingRecord[];
+
+  if (res.length === 1) {
+    return res[0];
+  }
+
+  if (res.length === 0) {
+    return undefined;
+  }
+
+  throw Error("something bad in sync meta mapping!");
+};
+
+export const clearAllSyncMetaMapping = async (db: lf.DatabaseConnection) => {
+  const tbl = db.getSchema().table(DEFAULT_TBL_SYNC_MAPPING);
+  await db.delete().from(tbl).exec();
+};
+
+export const insertSyncPlanRecord = async (
+  db: lf.DatabaseConnection,
+  syncPlan: SyncPlanType
+) => {
+  const schema = db.getSchema().table(DEFAULT_SYNC_PLANS_HISTORY);
+  const row = schema.createRow({
+    ts: syncPlan.ts,
+    remote_type: syncPlan.remoteType,
+    sync_plan: JSON.stringify(syncPlan, null, 2),
+  } as SyncPlanRecord);
+  await db.insertOrReplace().into(schema).values([row]).exec();
+};
+
+export const clearAllSyncPlanRecords = async (db: lf.DatabaseConnection) => {
+  const tbl = db.getSchema().table(DEFAULT_SYNC_PLANS_HISTORY);
+  await db.delete().from(tbl).exec();
+};
+
+export const readAllSyncPlanRecordTexts = async (db: lf.DatabaseConnection) => {
+  const schema = db.getSchema().table(DEFAULT_SYNC_PLANS_HISTORY);
+
+  const records = (await db
+    .select()
+    .from(schema)
+    .orderBy(schema.col("ts"), lf.Order.DESC)
+    .exec()) as SyncPlanRecord[];
+
+  if (records === undefined) {
+    return [] as string[];
+  } else {
+    return records.map((x) => x.sync_plan);
+  }
+};
@@ -0,0 +1,419 @@
+import {
+  App,
+  Modal,
+  Notice,
+  Plugin,
+  PluginSettingTab,
+  Setting,
+  request,
+  Platform,
+  TFile,
+  TFolder,
+} from "obsidian";
+import * as CodeMirror from "codemirror";
+import {
+  clearAllSyncPlanRecords,
+  clearAllSyncMetaMapping,
+  DatabaseConnection,
+} from "./localdb";
+import {
+  prepareDBs,
+  destroyDBs,
+  loadDeleteRenameHistoryTable,
+  insertDeleteRecord,
+  insertRenameRecord,
+  getAllDeleteRenameRecords,
+  insertSyncPlanRecord,
+} from "./localdb";
+
+import type { SyncStatusType, PasswordCheckType } from "./sync";
+import { isPasswordOk, getSyncPlan, doActualSync } from "./sync";
+import {
+  DEFAULT_S3_CONFIG,
+  getS3Client,
+  listFromRemote,
+  S3Config,
+  checkS3Connectivity,
+} from "./s3";
+import { exportSyncPlansToFiles } from "./debugMode";
+
+interface SaveRemotePluginSettings {
+  s3?: S3Config;
+  password?: string;
+}
+
+const DEFAULT_SETTINGS: SaveRemotePluginSettings = {
+  s3: DEFAULT_S3_CONFIG,
+  password: "",
+};
+
+export default class SaveRemotePlugin extends Plugin {
+  settings: SaveRemotePluginSettings;
+  cm: CodeMirror.Editor;
+  db: DatabaseConnection;
+  syncStatus: SyncStatusType;
+
+  async onload() {
+    console.log("loading plugin obsidian-save-remote");
+
+    await this.loadSettings();
+
+    await this.prepareDB();
+
+    this.syncStatus = "idle";
+
+    this.registerEvent(
+      this.app.vault.on("delete", async (fileOrFolder) => {
+        await insertDeleteRecord(this.db, fileOrFolder);
+      })
+    );
+
+    this.registerEvent(
+      this.app.vault.on("rename", async (fileOrFolder, oldPath) => {
+        await insertRenameRecord(this.db, fileOrFolder, oldPath);
+      })
+    );
+
+    this.addRibbonIcon("switch", "Save Remote", async () => {
+      if (this.syncStatus !== "idle") {
+        new Notice(`Save Remote already running in stage ${this.syncStatus}!`);
+        return;
+      }
+
+      try {
+        //console.log(`huh ${this.settings.password}`)
+        new Notice("1/6 Save Remote Sync Preparing");
+        this.syncStatus = "preparing";
+
+        new Notice("2/6 Starting to fetch remote meta data.");
+        this.syncStatus = "getting_remote_meta";
+        const s3Client = getS3Client(this.settings.s3);
+        const remoteRsp = await listFromRemote(s3Client, this.settings.s3);
+
+        new Notice("3/6 Starting to fetch local meta data.");
+        this.syncStatus = "getting_local_meta";
+        const local = this.app.vault.getAllLoadedFiles();
+        const localHistory = await loadDeleteRenameHistoryTable(this.db);
+        // console.log(remoteRsp);
+        // console.log(local);
+        // console.log(localHistory);
+
+        new Notice("4/7 Checking password correct or not.");
+        this.syncStatus = "checking_password";
+        const passwordCheckResult = await isPasswordOk(
+          remoteRsp.Contents,
+          this.settings.password
+        );
+        if (!passwordCheckResult.ok) {
+          throw Error(passwordCheckResult.reason);
+        }
+
+        new Notice("5/7 Starting to generate sync plan.");
+        this.syncStatus = "generating_plan";
+        const syncPlan = await getSyncPlan(
+          remoteRsp.Contents,
+          local,
+          localHistory,
+          this.db,
+          this.settings.password
+        );
+        console.log(syncPlan.mixedStates); // for debugging
+        await insertSyncPlanRecord(this.db, syncPlan);
+
+        // The operations above are read only and kind of safe.
+        // The operations below begins to write or delete (!!!) something.
+
+        new Notice("6/7 Save Remote Sync data exchanging!");
+
+        this.syncStatus = "syncing";
+        await doActualSync(
+          s3Client,
+          this.settings.s3,
+          this.db,
+          this.app.vault,
+          syncPlan,
+          this.settings.password
+        );
+
+        new Notice("7/7 Save Remote finish!");
+        this.syncStatus = "finish";
+        this.syncStatus = "idle";
+      } catch (error) {
+        const msg = `Save Remote error while ${this.syncStatus}`;
+        console.log(msg);
+        console.log(error);
+        new Notice(msg);
+        new Notice(error.message);
+        this.syncStatus = "idle";
+      }
+    });
+
+    this.addSettingTab(new SaveRemoteSettingTab(this.app, this));
+
+    this.registerCodeMirror((cm: CodeMirror.Editor) => {
+      this.cm = cm;
+      console.log("codemirror registered.");
+    });
+
+    // this.registerDomEvent(document, "click", (evt: MouseEvent) => {
+    //   console.log("click", evt);
+    // });
+
+    // this.registerInterval(
+    //   window.setInterval(() => console.log("setInterval"), 5 * 60 * 1000)
+    // );
+  }
+
+  onunload() {
+    console.log("unloading plugin obsidian-save-remote");
+    this.destroyDBs();
+  }
+
+  async loadSettings() {
+    this.settings = Object.assign({}, DEFAULT_SETTINGS, await this.loadData());
+  }
+
+  async saveSettings() {
+    await this.saveData(this.settings);
+  }
+
+  async prepareDB() {
+    this.db = await prepareDBs();
+  }
+
+  destroyDBs() {
+    /* destroyDBs(this.db); */
+  }
+}
+
+export class PasswordModal extends Modal {
+  plugin: SaveRemotePlugin;
+  newPassword: string;
+  constructor(app: App, plugin: SaveRemotePlugin, newPassword: string) {
+    super(app);
+    this.plugin = plugin;
+    this.newPassword = newPassword;
+  }
+
+  onOpen() {
+    let { contentEl } = this;
+    // contentEl.setText("Add Or change password.");
+    contentEl.createEl("h2", { text: "Hold on and PLEASE READ ON..." });
+    contentEl.createEl("p", {
+      text: "This password allows you encrypt your files before sending to remote services.",
+    });
+    contentEl.createEl("p", { text: "Empty means no password." });
+
+    contentEl.createEl("p", {
+      text: "Attention 1/3: The password setting itself is stored in PLAIN TEXT LOCALLY (because the plugin needs to use the password to encrypt the files) (and the password would not be sent to remote by this plugin).",
+    });
+    contentEl.createEl("p", {
+      text: "Attention 2/3: If you change the password. You should make sure the remote service (s3/webdav/...) IS EMPTY, or REMOTE FILES WERE ENCRYPTED BY THAT NEW PASSWORD. OTHERWISE SOMETHING BAD WOULD HAPPEN!",
+    });
+    contentEl.createEl("p", {
+      text: "Attention 3/3: The longer the password, the better.",
+    });
+
+    new Setting(contentEl)
+      .addButton((button) => {
+        button.setButtonText("The Second Confirm to change password.");
+        button.onClick(async () => {
+          this.plugin.settings.password = this.newPassword;
+          await this.plugin.saveSettings();
+          new Notice("New password saved!");
+          this.close();
+        });
+        button.setClass("password_second_confirm");
+      })
+      .addButton((button) => {
+        button.setButtonText("Cancel (password not changed.)");
+        button.onClick(() => {
+          this.close();
+        });
+      });
+  }
+
+  onClose() {
+    let { contentEl } = this;
+    contentEl.empty();
+  }
+}
+
+class SaveRemoteSettingTab extends PluginSettingTab {
+  plugin: SaveRemotePlugin;
+
+  constructor(app: App, plugin: SaveRemotePlugin) {
+    super(app, plugin);
+    this.plugin = plugin;
+  }
+
+  display(): void {
+    let { containerEl } = this;
+
+    containerEl.empty();
+
+    containerEl.createEl("h1", { text: "Save Remote" });
+
+    containerEl.createEl("h2", { text: "S3" });
+
+    new Setting(containerEl)
+      .setName("s3Endpoint")
+      .setDesc("s3Endpoint")
+      .addText((text) =>
+        text
+          .setPlaceholder("")
+          .setValue(this.plugin.settings.s3.s3Endpoint)
+          .onChange(async (value) => {
+            this.plugin.settings.s3.s3Endpoint = value.trim();
+            await this.plugin.saveSettings();
+          })
+      );
+
+    new Setting(containerEl)
+      .setName("s3Region")
+      .setDesc("s3Region")
+      .addText((text) =>
+        text
+          .setPlaceholder("")
+          .setValue(`${this.plugin.settings.s3.s3Region}`)
+          .onChange(async (value) => {
+            this.plugin.settings.s3.s3Region = value.trim();
+            await this.plugin.saveSettings();
+          })
+      );
+
+    new Setting(containerEl)
+      .setName("s3AccessKeyID")
+      .setDesc("s3AccessKeyID")
+      .addText((text) =>
+        text
+          .setPlaceholder("")
+          .setValue(`${this.plugin.settings.s3.s3AccessKeyID}`)
+          .onChange(async (value) => {
+            this.plugin.settings.s3.s3AccessKeyID = value.trim();
+            await this.plugin.saveSettings();
+          })
+      );
+
+    new Setting(containerEl)
+      .setName("s3SecretAccessKey")
+      .setDesc("s3SecretAccessKey")
+      .addText((text) =>
+        text
+          .setPlaceholder("")
+          .setValue(`${this.plugin.settings.s3.s3SecretAccessKey}`)
+          .onChange(async (value) => {
+            this.plugin.settings.s3.s3SecretAccessKey = value.trim();
+            await this.plugin.saveSettings();
+          })
+      );
+
+    new Setting(containerEl)
+      .setName("s3BucketName")
+      .setDesc("s3BucketName")
+      .addText((text) =>
+        text
+          .setPlaceholder("")
+          .setValue(`${this.plugin.settings.s3.s3BucketName}`)
+          .onChange(async (value) => {
+            this.plugin.settings.s3.s3BucketName = value.trim();
+            await this.plugin.saveSettings();
+          })
+      );
+
+    new Setting(containerEl)
+      .setName("check connectivity")
+      .setDesc("check connectivity")
+      .addButton(async (button) => {
+        button.setButtonText("Check");
+        button.onClick(async () => {
+          new Notice("Checking...");
+          const s3Client = getS3Client(this.plugin.settings.s3);
+          const res = await checkS3Connectivity(
+            s3Client,
+            this.plugin.settings.s3
+          );
+          if (res) {
+            new Notice("Great! The bucket can be accessed.");
+          } else {
+            new Notice("The S3 bucket cannot be reached.");
+          }
+        });
+      });
+
+    containerEl.createEl("h2", { text: "General" });
+
+    let newPassword = `${this.plugin.settings.password}`;
+    new Setting(containerEl)
+      .setName("encryption password")
+      .setDesc(
+        'Password for E2E encryption. Empty for no password. You need to click "Confirm".'
+      )
+      .addText((text) =>
+        text
+          .setPlaceholder("")
+          .setValue(`${this.plugin.settings.password}`)
+          .onChange(async (value) => {
+            newPassword = value.trim();
+          })
+      )
+      .addButton(async (button) => {
+        button.setButtonText("Confirm");
+        button.onClick(async () => {
+          new PasswordModal(this.app, this.plugin, newPassword).open();
+        });
+      });
+
+    containerEl.createEl("h2", { text: "Debug" });
+
+    const syncPlanDiv = containerEl.createEl("div");
+    syncPlanDiv.createEl("p", {
+      text: "Sync plans are created every time after you trigger sync and before the actual sync.",
+    });
+    syncPlanDiv.createEl("p", {
+      text: "They are useful to know what would actually happen in those sync.",
+    });
+
+    new Setting(containerEl)
+      .setName("export sync plans")
+      .setDesc("export sync plans")
+      .addButton(async (button) => {
+        button.setButtonText("Export");
+        button.onClick(async () => {
+          await exportSyncPlansToFiles(this.plugin.db, this.app.vault);
+          new Notice("sync plans history exported");
+        });
+      });
+
+    new Setting(containerEl)
+      .setName("delete sync plans history in db")
+      .setDesc("delete sync plans history in db")
+      .addButton(async (button) => {
+        button.setButtonText("Delete History");
+        button.onClick(async () => {
+          await clearAllSyncPlanRecords(this.plugin.db);
+          new Notice("sync plans history (in db) deleted");
+        });
+      });
+
+    const syncMappingDiv = containerEl.createEl("div");
+    syncMappingDiv.createEl("p", {
+      text: "Sync mappings history stores the actual LOCAL last modified time of the REMOTE objects.",
+    });
+
+    syncMappingDiv.createEl("p", {
+      text: "If the sync mappings history are deleted, unnecessary data exchanges may occur in next-time syncing, because whether a remote object and local object with same name are equivalent or not could not be determined correctly by comparing last modified times.",
+    });
+
+    new Setting(containerEl)
+      .setName("delete sync mappings history in db")
+      .setDesc("delete sync mappings history in db")
+      .addButton(async (button) => {
+        button.setButtonText("Delete Sync Mappings");
+        button.onClick(async () => {
+          await clearAllSyncMetaMapping(this.plugin.db);
+          new Notice("sync mappings history (in local db) deleted");
+        });
+      });
+  }
+}
@@ -0,0 +1,113 @@
+import { Vault } from "obsidian";
+import * as path from "path";
+
+import { base32 } from "rfc4648";
+
+export type SUPPORTED_SERVICES_TYPE = "s3" | "webdav" | "ftp";
+
+/**
+ * If any part of the file starts with '.' or '_' then it's a hidden file.
+ * @param item
+ * @param loose
+ * @returns
+ */
+export const isHiddenPath = (item: string, loose: boolean = true) => {
+  const k = path.posix.normalize(item); // TODO: only unix path now
+  const k2 = k.split("/"); // TODO: only unix path now
+  // console.log(k2)
+  for (const singlePart of k2) {
+    if (singlePart === "." || singlePart === ".." || singlePart === "") {
+      continue;
+    }
+    if (singlePart[0] === ".") {
+      return true;
+    }
+    if (loose && singlePart[0] === "_") {
+      return true;
+    }
+  }
+  return false;
+};
+
+/**
+ * Util func for mkdir -p based on the "path" of original file or folder
+ * "a/b/c/" => ["a", "a/b", "a/b/c"]
+ * "a/b/c/d/e.txt" => ["a", "a/b", "a/b/c", "a/b/c/d"]
+ * @param x string
+ * @returns string[] might be empty
+ */
+export const getFolderLevels = (x: string) => {
+  const res: string[] = [];
+
+  if (x === "" || x === "/") {
+    return res;
+  }
+
+  const y1 = x.split("/");
+  let i = 0;
+  for (let index = 0; index + 1 < y1.length; index++) {
+    res.push(y1.slice(0, index + 1).join("/"));
+  }
+  return res;
+};
+
+export const mkdirpInVault = async (thePath: string, vault: Vault) => {
+  // console.log(thePath);
+  const foldersToBuild = getFolderLevels(thePath);
+  // console.log(foldersToBuild);
+  for (const folder of foldersToBuild) {
+    const r = await vault.adapter.exists(folder);
+    // console.log(r);
+    if (!r) {
+      console.log(`mkdir ${folder}`);
+      await vault.adapter.mkdir(folder);
+    }
+  }
+};
+
+/**
+ * https://stackoverflow.com/questions/8609289
+ * @param b Buffer
+ * @returns ArrayBuffer
+ */
+export const bufferToArrayBuffer = (b: Buffer | Uint8Array) => {
+  return b.buffer.slice(b.byteOffset, b.byteOffset + b.byteLength);
+};
+
+/**
+ * Simple func.
+ * @param b
+ * @returns
+ */
+export const arrayBufferToBuffer = (b: ArrayBuffer) => {
+  return Buffer.from(b);
+};
+
+export const arrayBufferToBase64 = (b: ArrayBuffer) => {
+  return arrayBufferToBuffer(b).toString("base64");
+};
+
+export const arrayBufferToHex = (b: ArrayBuffer) => {
+  return arrayBufferToBuffer(b).toString("hex");
+};
+
+export const base64ToArrayBuffer = (b64text: string) => {
+  return bufferToArrayBuffer(Buffer.from(b64text, "base64"));
+};
+
+/**
+ * https://stackoverflow.com/questions/43131242
+ * @param hex
+ * @returns
+ */
+export const hexStringToTypedArray = (hex: string) => {
+  return new Uint8Array(
+    hex.match(/[\da-f]{2}/gi).map(function (h) {
+      return parseInt(h, 16);
+    })
+  );
+};
+
+export const base64ToBase32 = (a: string) => {
+  return base32.stringify(Buffer.from(a, "base64"));
+};
@@ -0,0 +1,324 @@
+import { Buffer } from "buffer";
+import { Readable } from "stream";
+
+import { Vault } from "obsidian";
+
+import {
+  S3Client,
+  ListObjectsV2Command,
+  PutObjectCommand,
+  GetObjectCommand,
+  DeleteObjectCommand,
+  HeadObjectCommand,
+  HeadBucketCommand,
+  ListObjectsV2CommandInput,
+  ListObjectsV2CommandOutput,
+} from "@aws-sdk/client-s3";
+
+import type { _Object } from "@aws-sdk/client-s3";
+
+import {
+  arrayBufferToBuffer,
+  bufferToArrayBuffer,
+  mkdirpInVault,
+} from "./misc";
+import * as mime from "mime-types";
+import { decryptArrayBuffer, encryptArrayBuffer } from "./encrypt";
+
+export interface S3Config {
+  s3Endpoint: string;
+  s3Region: string;
+  s3AccessKeyID: string;
+  s3SecretAccessKey: string;
+  s3BucketName: string;
+}
+
+export const DEFAULT_S3_CONFIG = {
+  s3Endpoint: "",
+  s3Region: "",
+  s3AccessKeyID: "",
+  s3SecretAccessKey: "",
+  s3BucketName: "",
+};
+
+export type S3ObjectType = _Object;
+
+export const getS3Client = (s3Config: S3Config) => {
+  const s3Client = new S3Client({
+    region: s3Config.s3Region,
+    endpoint: s3Config.s3Endpoint,
+    credentials: {
+      accessKeyId: s3Config.s3AccessKeyID,
+      secretAccessKey: s3Config.s3SecretAccessKey,
+    },
+  });
+  return s3Client;
+};
+
+export const getRemoteMeta = async (
+  s3Client: S3Client,
+  s3Config: S3Config,
+  fileOrFolderPath: string
+) => {
+  return await s3Client.send(
+    new HeadObjectCommand({
+      Bucket: s3Config.s3BucketName,
+      Key: fileOrFolderPath,
+    })
+  );
+};
+
+export const uploadToRemote = async (
+  s3Client: S3Client,
+  s3Config: S3Config,
+  fileOrFolderPath: string,
+  vault: Vault,
+  isRecursively: boolean = false,
+  password: string = "",
+  remoteEncryptedKey: string = ""
+) => {
+  let uploadFile = fileOrFolderPath;
+  if (password !== "") {
+    uploadFile = remoteEncryptedKey;
+  }
+  const isFolder = fileOrFolderPath.endsWith("/");
+
+  const DEFAULT_CONTENT_TYPE = "application/octet-stream";
+
+  if (isFolder && isRecursively) {
+    throw Error("upload function doesn't implement recursive function yet!");
+  } else if (isFolder && !isRecursively) {
+    // folder
+    const contentType = DEFAULT_CONTENT_TYPE;
+    await s3Client.send(
+      new PutObjectCommand({
+        Bucket: s3Config.s3BucketName,
+        Key: uploadFile,
+        Body: "",
+        ContentType: contentType,
+      })
+    );
+    return await getRemoteMeta(s3Client, s3Config, fileOrFolderPath);
+  } else {
+    // file
+    // we ignore isRecursively parameter here
+    let contentType = DEFAULT_CONTENT_TYPE;
+    if (password === "") {
+      contentType =
+        mime.contentType(
+          mime.lookup(fileOrFolderPath) || DEFAULT_CONTENT_TYPE
+        ) || DEFAULT_CONTENT_TYPE;
+    }
+    const localContent = await vault.adapter.readBinary(fileOrFolderPath);
+    let remoteContent = localContent;
+    if (password !== "") {
+      remoteContent = await encryptArrayBuffer(localContent, password);
+    }
+    const body = arrayBufferToBuffer(remoteContent);
+    await s3Client.send(
+      new PutObjectCommand({
+        Bucket: s3Config.s3BucketName,
+        Key: uploadFile,
+        Body: body,
+        ContentType: contentType,
+      })
+    );
+    return await getRemoteMeta(s3Client, s3Config, uploadFile);
+  }
+};
+
+export const listFromRemote = async (
+  s3Client: S3Client,
+  s3Config: S3Config,
+  prefix?: string
+) => {
+  const confCmd = {
+    Bucket: s3Config.s3BucketName,
+  } as ListObjectsV2CommandInput;
+  if (prefix !== undefined) {
+    confCmd.Prefix = prefix;
+  }
+
+  const contents = [] as _Object[];
+
+  let isTruncated = true;
+  let continuationToken = "";
+  do {
+    const rsp = await s3Client.send(new ListObjectsV2Command(confCmd));
+
+    if (rsp.$metadata.httpStatusCode !== 200) {
+      throw Error("some thing bad while listing remote!");
+    }
+    contents.push(...rsp.Contents);
+
+    isTruncated = rsp.IsTruncated;
+    confCmd.ContinuationToken = rsp.NextContinuationToken;
+    if (
+      isTruncated &&
+      (continuationToken === undefined || continuationToken === "")
+    ) {
+      throw Error("isTruncated is true but no continuationToken provided");
+    }
+  } while (isTruncated);
+
+  // ensemble fake rsp
+  return {
+    "$.metadata": {
+      httpStatusCode: 200,
+    },
+    Contents: contents,
+  };
+};
+
+/**
+ * The Body of resp of aws GetObject has mix types
+ * and we want to get ArrayBuffer here.
+ * See https://github.com/aws/aws-sdk-js-v3/issues/1877
+ * @param b The Body of GetObject
+ * @returns Promise<ArrayBuffer>
+ */
+const getObjectBodyToArrayBuffer = async (
+  b: Readable | ReadableStream | Blob
+) => {
+  if (b instanceof Readable) {
+    const chunks: Uint8Array[] = [];
+    for await (let chunk of b) {
+      chunks.push(chunk);
+    }
+    const buf = Buffer.concat(chunks);
+    return bufferToArrayBuffer(buf);
+  } else if (b instanceof ReadableStream) {
+    return await new Response(b, {}).arrayBuffer();
+  } else if (b instanceof Blob) {
+    return await b.arrayBuffer();
+  } else {
+    throw TypeError(`The type of ${b} is not one of the supported types`);
+  }
+};
+
+export const downloadFromRemoteRaw = async (
+  s3Client: S3Client,
+  s3Config: S3Config,
+  fileOrFolderPath: string
+) => {
+  const data = await s3Client.send(
+    new GetObjectCommand({
+      Bucket: s3Config.s3BucketName,
+      Key: fileOrFolderPath,
+    })
+  );
+  const bodyContents = await getObjectBodyToArrayBuffer(data.Body);
+  return bodyContents;
+};
+
+export const downloadFromRemote = async (
+  s3Client: S3Client,
+  s3Config: S3Config,
+  fileOrFolderPath: string,
+  vault: Vault,
+  mtime: number,
+  password: string = "",
+  remoteEncryptedKey: string = ""
+) => {
+  const isFolder = fileOrFolderPath.endsWith("/");
+
+  await mkdirpInVault(fileOrFolderPath, vault);
+
+  // the file is always local file
+  // we need to encrypt it
+
+  if (isFolder) {
+    // mkdirp locally is enough
+    // do nothing here
+  } else {
+    let downloadFile = fileOrFolderPath;
+    if (password !== "") {
+      downloadFile = remoteEncryptedKey;
+    }
+    const remoteContent = await downloadFromRemoteRaw(
+      s3Client,
+      s3Config,
+      downloadFile
+    );
+    let localContent = remoteContent;
+    if (password !== "") {
+      localContent = await decryptArrayBuffer(remoteContent, password);
+    }
+    await vault.adapter.writeBinary(fileOrFolderPath, localContent, {
+      mtime: mtime,
+    });
+  }
+};
+
+/**
+ * This function deals with file normally and "folder" recursively.
+ * @param s3Client
+ * @param s3Config
+ * @param fileOrFolderPath
+ * @returns
+ */
+export const deleteFromRemote = async (
+  s3Client: S3Client,
+  s3Config: S3Config,
+  fileOrFolderPath: string,
+  password: string = "",
+  remoteEncryptedKey: string = ""
+) => {
+  if (fileOrFolderPath === "/") {
+    return;
+  }
+  let remoteFileName = fileOrFolderPath;
+  if (password !== "") {
+    remoteFileName = remoteEncryptedKey;
+  }
+  await s3Client.send(
+    new DeleteObjectCommand({
+      Bucket: s3Config.s3BucketName,
+      Key: remoteFileName,
+    })
+  );
+
+  if (fileOrFolderPath.endsWith("/") && password === "") {
+    const x = await listFromRemote(s3Client, s3Config, fileOrFolderPath);
+    x.Contents.forEach(async (element) => {
+      await s3Client.send(
+        new DeleteObjectCommand({
+          Bucket: s3Config.s3BucketName,
+          Key: element.Key,
+        })
+      );
+    });
+  } else if (fileOrFolderPath.endsWith("/") && password !== "") {
+    // TODO
+  } else {
+    // pass
+  }
+};
+
+/**
+ * Check the config of S3 by heading bucket
+ * https://stackoverflow.com/questions/50842835
+ * @param s3Client
+ * @param s3Config
+ * @returns
+ */
+export const checkS3Connectivity = async (
+  s3Client: S3Client,
+  s3Config: S3Config
+) => {
+  try {
+    const results = await s3Client.send(
+      new HeadBucketCommand({ Bucket: s3Config.s3BucketName })
+    );
+    if (
+      results === undefined ||
+      results.$metadata === undefined ||
+      results.$metadata.httpStatusCode === undefined
+    ) {
+      return false;
+    }
+    return results.$metadata.httpStatusCode === 200;
+  } catch (err) {
+    return false;
+  }
+};
@@ -0,0 +1,523 @@
+import { TAbstractFile, TFolder, TFile, Vault } from "obsidian";
+
+import { S3Client } from "@aws-sdk/client-s3";
+import * as lf from "lovefield-ts/dist/es6/lf.js";
+
+import {
+  clearDeleteRenameHistoryOfKey,
+  FileFolderHistoryRecord,
+  upsertSyncMetaMappingDataS3,
+  getSyncMetaMappingByRemoteKeyS3,
+} from "./localdb";
+import {
+  S3Config,
+  S3ObjectType,
+  uploadToRemote,
+  deleteFromRemote,
+  downloadFromRemote,
+} from "./s3";
+import { mkdirpInVault, SUPPORTED_SERVICES_TYPE, isHiddenPath } from "./misc";
+import {
+  decryptBase32ToString,
+  encryptStringToBase32,
+  MAGIC_ENCRYPTED_PREFIX_BASE32,
+} from "./encrypt";
+
+export type SyncStatusType =
+  | "idle"
+  | "preparing"
+  | "getting_remote_meta"
+  | "getting_local_meta"
+  | "checking_password"
+  | "generating_plan"
+  | "syncing"
+  | "finish";
+
+type DecisionType =
+  | "undecided"
+  | "unknown"
+  | "upload_clearhist"
+  | "download_clearhist"
+  | "delremote_clearhist"
+  | "download"
+  | "upload"
+  | "clearhist"
+  | "mkdirplocal"
+  | "skip";
+
+interface FileOrFolderMixedState {
+  key: string;
+  exist_local?: boolean;
+  exist_remote?: boolean;
+  mtime_local?: number;
+  mtime_remote?: number;
+  delete_time_local?: number;
+  size_local?: number;
+  size_remote?: number;
+  decision?: DecisionType;
+  syncDone?: "done";
+  decision_branch?: number;
+  remote_encrypted_key?: string;
+}
+
+export interface SyncPlanType {
+  ts: number;
+  remoteType: SUPPORTED_SERVICES_TYPE;
+  mixedStates: Record<string, FileOrFolderMixedState>;
+}
+
+export interface PasswordCheckType {
+  ok: boolean;
+  reason:
+    | "ok"
+    | "empty_remote"
+    | "remote_encrypted_local_no_password"
+    | "password_matched"
+    | "password_not_matched"
+    | "remote_not_encrypted_local_has_password"
+    | "no_password_both_sides";
+}
+
+export const isPasswordOk = async (
+  remote: S3ObjectType[],
+  password: string = ""
+) => {
+  if (remote === undefined || remote.length === 0) {
+    // remote empty
+    return {
+      ok: true,
+      reason: "empty_remote",
+    } as PasswordCheckType;
+  }
+  const santyCheckKey = remote[0].Key;
+  if (santyCheckKey.startsWith(MAGIC_ENCRYPTED_PREFIX_BASE32)) {
+    // this is encrypted!
+    // try to decrypt it using the provided password.
+    if (password === "") {
+      return {
+        ok: false,
+        reason: "remote_encrypted_local_no_password",
+      } as PasswordCheckType;
+    }
+    try {
+      const res = await decryptBase32ToString(santyCheckKey, password);
+      return {
+        ok: true,
+        reason: "password_matched",
+      } as PasswordCheckType;
+    } catch (error) {
+      return {
+        ok: false,
+        reason: "password_not_matched",
+      } as PasswordCheckType;
+    }
+  } else {
+    // it is not encrypted!
+    if (password !== "") {
+      return {
+        ok: false,
+        reason: "remote_not_encrypted_local_has_password",
+      } as PasswordCheckType;
+    }
+    return {
+      ok: true,
+      reason: "no_password_both_sides",
+    } as PasswordCheckType;
+  }
+};
+
+const ensembleMixedStates = async (
+  remote: S3ObjectType[],
+  local: TAbstractFile[],
+  deleteHistory: FileFolderHistoryRecord[],
+  db: lf.DatabaseConnection,
+  password: string = ""
+) => {
+  const results = {} as Record<string, FileOrFolderMixedState>;
+
+  if (remote !== undefined) {
+    for (const entry of remote) {
+      const remoteEncryptedKey = entry.Key;
+      let key = remoteEncryptedKey;
+      if (password !== "") {
+        key = await decryptBase32ToString(remoteEncryptedKey, password);
+      }
+      const backwardMapping = await getSyncMetaMappingByRemoteKeyS3(
+        db,
+        key,
+        entry.LastModified.valueOf(),
+        entry.ETag
+      );
+
+      let r = {} as FileOrFolderMixedState;
+      if (backwardMapping !== undefined) {
+        key = backwardMapping.local_key;
+        r = {
+          key: key,
+          exist_remote: true,
+          mtime_remote: backwardMapping.local_mtime,
+          size_remote: backwardMapping.local_size,
+          remote_encrypted_key: remoteEncryptedKey,
+        };
+      } else {
+        r = {
+          key: key,
+          exist_remote: true,
+          mtime_remote: entry.LastModified.valueOf(),
+          size_remote: entry.Size,
+          remote_encrypted_key: remoteEncryptedKey,
+        };
+      }
+      if (isHiddenPath(key)) {
+        continue;
+      }
+      if (results.hasOwnProperty(key)) {
+        results[key].key = r.key;
+        results[key].exist_remote = r.exist_remote;
+        results[key].mtime_remote = r.mtime_remote;
+        results[key].size_remote = r.size_remote;
+        results[key].remote_encrypted_key = r.remote_encrypted_key;
+      } else {
+        results[key] = r;
+      }
+    }
+  }
+
+  for (const entry of local) {
+    let r = {} as FileOrFolderMixedState;
+    let key = entry.path;
+
+    if (entry.path === "/") {
+      // ignore
+      continue;
+    } else if (entry instanceof TFile) {
+      r = {
+        key: entry.path,
+        exist_local: true,
+        mtime_local: entry.stat.mtime,
+        size_local: entry.stat.size,
+      };
+    } else if (entry instanceof TFolder) {
+      key = `${entry.path}/`;
+      r = {
+        key: key,
+        exist_local: true,
+        mtime_local: undefined,
+        size_local: 0,
+      };
+    } else {
+      throw Error(`unexpected ${entry}`);
+    }
+
+    if (isHiddenPath(key)) {
+      continue;
+    }
+    if (results.hasOwnProperty(key)) {
+      results[key].key = r.key;
+      results[key].exist_local = r.exist_local;
+      results[key].mtime_local = r.mtime_local;
+      results[key].size_local = r.size_local;
+    } else {
+      results[key] = r;
+    }
+  }
+
+  for (const entry of deleteHistory) {
+    let key = entry.key;
+    if (entry.key_type === "folder") {
+      if (!entry.key.endsWith("/")) {
+        key = `${entry.key}/`;
+      }
+    } else if (entry.key_type === "file") {
+      // pass
+    } else {
+      throw Error(`unexpected ${entry}`);
+    }
+
+    const r = {
+      key: key,
+      delete_time_local: entry.action_when,
+    } as FileOrFolderMixedState;
+
+    if (isHiddenPath(key)) {
+      continue;
+    }
+    if (results.hasOwnProperty(key)) {
+      results[key].key = r.key;
+      results[key].delete_time_local = r.delete_time_local;
+    } else {
+      results[key] = r;
+    }
+  }
+
+  return results;
+};
+
+const getOperation = (
+  origRecord: FileOrFolderMixedState,
+  inplace: boolean = false,
+  password: string = ""
+) => {
+  let r = origRecord;
+  if (!inplace) {
+    r = Object.assign({}, origRecord);
+  }
+
+  if (r.mtime_local === 0) {
+    r.mtime_local = undefined;
+  }
+  if (r.mtime_remote === 0) {
+    r.mtime_remote = undefined;
+  }
+  if (r.delete_time_local === 0) {
+    r.delete_time_local = undefined;
+  }
+  if (r.exist_local === undefined) {
+    r.exist_local = false;
+  }
+  if (r.exist_remote === undefined) {
+    r.exist_remote = false;
+  }
+  r.decision = "unknown";
+
+  if (
+    r.exist_remote &&
+    r.exist_local &&
+    r.mtime_remote !== undefined &&
+    r.mtime_local !== undefined &&
+    r.mtime_remote > r.mtime_local
+  ) {
+    r.decision = "download_clearhist";
+    r.decision_branch = 1;
+  } else if (
+    r.exist_remote &&
+    r.exist_local &&
+    r.mtime_remote !== undefined &&
+    r.mtime_local !== undefined &&
+    r.mtime_remote < r.mtime_local
+  ) {
+    r.decision = "upload_clearhist";
+    r.decision_branch = 2;
+  } else if (
+    r.exist_remote &&
+    r.exist_local &&
+    r.mtime_remote !== undefined &&
+    r.mtime_local !== undefined &&
+    r.mtime_remote === r.mtime_local &&
+    password === "" &&
+    r.size_local === r.size_remote
+  ) {
+    r.decision = "skip";
+    r.decision_branch = 3;
+  } else if (
+    r.exist_remote &&
+    r.exist_local &&
+    r.mtime_remote !== undefined &&
+    r.mtime_local !== undefined &&
+    r.mtime_remote === r.mtime_local &&
+    password === "" &&
+    r.size_local !== r.size_remote
+  ) {
+    r.decision = "upload_clearhist";
+    r.decision_branch = 4;
+  } else if (
+    r.exist_remote &&
+    r.exist_local &&
+    r.mtime_remote !== undefined &&
+    r.mtime_local !== undefined &&
+    r.mtime_remote === r.mtime_local &&
+    password !== ""
+  ) {
+    // if we have encryption,
+    // the size is always unequal
+    // only mtime(s) are reliable
+    r.decision = "skip";
+    r.decision_branch = 5;
+  } else if (r.exist_remote && r.exist_local && r.mtime_local === undefined) {
+    // this must be a folder!
+    if (!r.key.endsWith("/")) {
+      throw Error(`${r.key} is not a folder but lacks local mtime`);
+    }
+    r.decision = "skip";
+    r.decision_branch = 6;
+  } else if (
+    r.exist_remote &&
+    !r.exist_local &&
+    r.mtime_remote !== undefined &&
+    r.mtime_local === undefined &&
+    r.delete_time_local !== undefined &&
+    r.mtime_remote >= r.delete_time_local
+  ) {
+    r.decision = "download_clearhist";
+    r.decision_branch = 7;
+  } else if (
+    r.exist_remote &&
+    !r.exist_local &&
+    r.mtime_remote !== undefined &&
+    r.mtime_local === undefined &&
+    r.delete_time_local !== undefined &&
+    r.mtime_remote < r.delete_time_local
+  ) {
+    r.decision = "delremote_clearhist";
+    r.decision_branch = 8;
+  } else if (
+    r.exist_remote &&
+    !r.exist_local &&
+    r.mtime_remote !== undefined &&
+    r.mtime_local === undefined &&
+    r.delete_time_local == undefined
+  ) {
+    r.decision = "download";
+    r.decision_branch = 9;
+  } else if (!r.exist_remote && r.exist_local && r.mtime_remote === undefined) {
+    r.decision = "upload_clearhist";
+    r.decision_branch = 10;
+  } else if (
+    !r.exist_remote &&
+    !r.exist_local &&
+    r.mtime_remote === undefined &&
+    r.mtime_local === undefined
+  ) {
+    r.decision = "clearhist";
+    r.decision_branch = 11;
+  }
+
+  return r;
+};
+
+export const getSyncPlan = async (
+  remote: S3ObjectType[],
+  local: TAbstractFile[],
+  deleteHistory: FileFolderHistoryRecord[],
+  db: lf.DatabaseConnection,
+  password: string = ""
+) => {
+  const mixedStates = await ensembleMixedStates(
+    remote,
+    local,
+    deleteHistory,
+    db,
+    password
+  );
+  for (const [key, val] of Object.entries(mixedStates)) {
+    getOperation(val, true, password);
+  }
+  const plan = {
+    ts: Date.now(),
+    remoteType: "s3",
+    mixedStates: mixedStates,
+  } as SyncPlanType;
+  return plan;
+};
+
+export const doActualSync = async (
+  s3Client: S3Client,
+  s3Config: S3Config,
+  db: lf.DatabaseConnection,
+  vault: Vault,
+  syncPlan: SyncPlanType,
+  password: string = ""
+) => {
+  const keyStates = syncPlan.mixedStates;
+  await Promise.all(
+    Object.entries(keyStates)
+      .sort((k, v) => -(k as string).length)
+      .map(async ([k, v]) => {
+        const key = k as string;
+        const state = v as FileOrFolderMixedState;
+        let remoteEncryptedKey = key;
+        if (password !== "") {
+          remoteEncryptedKey = state.remote_encrypted_key;
+          if (remoteEncryptedKey === undefined || remoteEncryptedKey === "") {
+            remoteEncryptedKey = await encryptStringToBase32(key, password);
+          }
+        }
+
+        if (
+          state.decision === undefined ||
+          state.decision === "unknown" ||
+          state.decision === "undecided"
+        ) {
+          throw Error(`unknown decision in ${JSON.stringify(state)}`);
+        } else if (state.decision === "skip") {
+          // do nothing
+        } else if (state.decision === "download_clearhist") {
+          await downloadFromRemote(
+            s3Client,
+            s3Config,
+            state.key,
+            vault,
+            state.mtime_remote,
+            password,
+            remoteEncryptedKey
+          );
+          await clearDeleteRenameHistoryOfKey(db, state.key);
+        } else if (state.decision === "upload_clearhist") {
+          const remoteObjMeta = await uploadToRemote(
+            s3Client,
+            s3Config,
+            state.key,
+            vault,
+            false,
+            password,
+            remoteEncryptedKey
+          );
+          await upsertSyncMetaMappingDataS3(
+            db,
+            state.key,
+            state.mtime_local,
+            state.size_local,
+            state.key,
+            remoteObjMeta.LastModified.valueOf(),
+            remoteObjMeta.ContentLength,
+            remoteObjMeta.ETag
+          );
+          await clearDeleteRenameHistoryOfKey(db, state.key);
+        } else if (state.decision === "download") {
+          await mkdirpInVault(state.key, vault);
+          await downloadFromRemote(
+            s3Client,
+            s3Config,
+            state.key,
+            vault,
+            state.mtime_remote,
+            password,
+            remoteEncryptedKey
+          );
+        } else if (state.decision === "delremote_clearhist") {
+          await deleteFromRemote(
+            s3Client,
+            s3Config,
+            state.key,
+            password,
+            remoteEncryptedKey
+          );
+          await clearDeleteRenameHistoryOfKey(db, state.key);
+        } else if (state.decision === "upload") {
+          const remoteObjMeta = await uploadToRemote(
+            s3Client,
+            s3Config,
+            state.key,
+            vault,
+            false,
+            password,
+            remoteEncryptedKey
+          );
+          await upsertSyncMetaMappingDataS3(
+            db,
+            state.key,
+            state.mtime_local,
+            state.size_local,
+            state.key,
+            remoteObjMeta.LastModified.valueOf(),
+            remoteObjMeta.ContentLength,
+            remoteObjMeta.ETag
+          );
+        } else if (state.decision === "clearhist") {
+          await clearDeleteRenameHistoryOfKey(db, state.key);
+        } else {
+          throw Error("this should never happen!");
+        }
+      })
+  );
+};
@@ -1 +1,5 @@
 /* set the styles */
+
+.password_second_confirm {
+  font-weight: bold;
+}
@@ -0,0 +1,98 @@
+import * as fs from "fs";
+import * as path from "path";
+import { expect } from "chai";
+import { base64ToBase32, bufferToArrayBuffer } from "../src/misc";
+import {
+  decryptArrayBuffer,
+  decryptBase32ToString,
+  encryptArrayBuffer,
+  encryptStringToBase32,
+} from "../src/encrypt";
+
+describe("Encryption tests", () => {
+  beforeEach(function () {
+    global.window = {
+      crypto: require("crypto").webcrypto,
+    } as any;
+  });
+
+  it("should encrypt string", async () => {
+    const k = "dkjdhkfhdkjgsdklxxd";
+    const password = "hey";
+    expect(await encryptStringToBase32(k, password)).to.not.equal(k);
+  });
+
+  it("should encrypt and decrypt string and get the same result returned", async () => {
+    const k = "jfkkjkjbce7983ycdeknkkjckooAIUHIDIBIE((*BII)njD/d/dd/d/sjxhux";
+    const password = "hfiuibec989###oiu982bj1`";
+    const enc = await encryptStringToBase32(k, password);
+    // console.log(enc);
+    const dec = await decryptBase32ToString(enc, password);
+    // console.log(dec);
+    expect(dec).equal(k);
+  });
+
+  it("should encrypt text file and get the same result as openssl", async () => {
+    const fileContent = (
+      await fs.readFileSync(
+        path.join(__dirname, "static_assets", "sometext.txt")
+      )
+    ).toString("utf-8");
+    const password = "somepassword";
+    const saltHex = "8302F586FAB491EC";
+    const enc = await encryptStringToBase32(
+      fileContent,
+      password,
+      undefined,
+      saltHex
+    );
+
+    // two command returns same result:
+    // cat ./sometext.txt | openssl enc -p -aes-256-cbc -S 8302F586FAB491EC -pbkdf2 -iter 10000 -base64 -pass pass:somepassword
+    // openssl enc -p -aes-256-cbc -S 8302F586FAB491EC -pbkdf2 -iter 10000 -base64 -pass pass:somepassword -in ./sometext.txt
+    const opensslBase64Res =
+      "U2FsdGVkX1+DAvWG+rSR7MSa+yJav1zCE7SSXiBooqwI5Q+LMpIthpk/pXkLj+25";
+    // we output base32, so we need some transformation
+    const opensslBase32Res = base64ToBase32(opensslBase64Res);
+
+    expect(enc).equal(opensslBase32Res);
+  });
+
+  it("should encrypt binary file and get the same result as openssl", async () => {
+    const testFolder = path.join(__dirname, "static_assets", "mona_lisa");
+    const testFileName =
+      "1374px-Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched.jpg";
+    const fileArrBuf = bufferToArrayBuffer(
+      await fs.readFileSync(path.join(testFolder, testFileName))
+    );
+    const password = "somepassword";
+    const saltHex = "8302F586FAB491EC";
+    const enc = await encryptArrayBuffer(
+      fileArrBuf,
+      password,
+      undefined,
+      saltHex
+    );
+    const opensslArrBuf = bufferToArrayBuffer(
+      await fs.readFileSync(path.join(testFolder, testFileName + ".enc"))
+    );
+
+    expect(Buffer.from(enc).equals(Buffer.from(opensslArrBuf))).to.be.true;
+  });
+
+  it("should descypt binary file and get the same result as openssl", async () => {
+    const testFolder = path.join(__dirname, "static_assets", "mona_lisa");
+    const testFileName =
+      "1374px-Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched.jpg";
+    const fileArrBuf = bufferToArrayBuffer(
+      await fs.readFileSync(path.join(testFolder, testFileName + ".enc"))
+    );
+    const password = "somepassword";
+    const dec = await decryptArrayBuffer(fileArrBuf, password);
+    const opensslArrBuf = bufferToArrayBuffer(
+      await fs.readFileSync(path.join(testFolder, testFileName))
+    );
+
+    expect(Buffer.from(dec).equals(Buffer.from(opensslArrBuf))).to.be.true;
+  });
+});
@@ -0,0 +1,72 @@
+import * as fs from "fs";
+import * as path from "path";
+import { expect } from "chai";
+
+import * as misc from "../src/misc";
+
+describe("Misc: hidden file", () => {
+  it("should find hidden file correctly", () => {
+    let item = "";
+    expect(misc.isHiddenPath(item)).to.be.false;
+
+    item = ".";
+    expect(misc.isHiddenPath(item)).to.be.false;
+
+    item = "..";
+    expect(misc.isHiddenPath(item)).to.be.false;
+
+    item = "/x/y/z/../././../a/b/c";
+    expect(misc.isHiddenPath(item)).to.be.false;
+
+    item = ".hidden";
+    expect(misc.isHiddenPath(item)).to.be.true;
+
+    item = "_hidden_loose";
+    expect(misc.isHiddenPath(item)).to.be.true;
+    expect(misc.isHiddenPath(item, false)).to.be.false;
+
+    item = "/sdd/_hidden_loose";
+    expect(misc.isHiddenPath(item)).to.be.true;
+
+    item = "what/../_hidden_loose/what/what/what";
+    expect(misc.isHiddenPath(item)).to.be.true;
+
+    item = "what/../_hidden_loose/what/what/what";
+    expect(misc.isHiddenPath(item, false)).to.be.false;
+
+    item = "what/../_hidden_loose/../.hidden/what/what/what";
+    expect(misc.isHiddenPath(item, false)).to.be.true;
+  });
+});
+
+describe("Misc: get folder levels", () => {
+  it("should ignore empty path", () => {
+    const item = "";
+    expect(misc.getFolderLevels(item)).to.be.empty;
+  });
+
+  it("should ignore single file", () => {
+    const item = "xxx";
+    expect(misc.getFolderLevels(item)).to.be.empty;
+  });
+
+  it("should detect path ending with /", () => {
+    const item = "xxx/";
+    const res = ["xxx"];
+    expect(misc.getFolderLevels(item)).to.deep.equal(res);
+  });
+
+  it("should correctly split folders and files", () => {
+    const item = "xxx/yyy/zzz.md";
+    const res = ["xxx", "xxx/yyy"];
+    expect(misc.getFolderLevels(item)).to.deep.equal(res);
+
+    const item2 = "xxx/yyy/zzz";
+    const res2 = ["xxx", "xxx/yyy"];
+    expect(misc.getFolderLevels(item2)).to.deep.equal(res2);
+
+    const item3 = "xxx/yyy/zzz/";
+    const res3 = ["xxx", "xxx/yyy", "xxx/yyy/zzz"];
+    expect(misc.getFolderLevels(item3)).to.deep.equal(res3);
+  });
+});
@@ -0,0 +1,3 @@
+The file 1374px-Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched.jpg is downloaded from Wikimedia Commons: https://commons.wikimedia.org/wiki/File:Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched.jpg .
+
+Quoted from the web address: ... This photographic reproduction is therefore also considered to be in the public domain in the United States. In other jurisdictions, re-use of this content may be restricted ...
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b332c298be98be9968715952822431a051f9d07e67460cdcf91a5a3a305b5df1
+size 1141550
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cf67abd791f1e230bce57dd47150f262424247cbefb8be16efcd57d13955576a
+size 1141568
@@ -0,0 +1,3 @@
+# The encryption file is produced by the following command.
+# A salt is explictly provided because we need reproducible output in tests.
+openssl enc -p -aes-256-cbc -S 8302F586FAB491EC -pbkdf2 -iter 10000 -pass pass:somepassword -in '1374px-Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched.jpg' -out 1374px-Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched.jpg.enc
@@ -0,0 +1 @@
+A secret text 你好世界
@@ -1,3 +1,3 @@
 {
-  "0.0.1": "0.12.15"
+  "0.0.11": "0.12.15"
 }
@@ -1,14 +1,25 @@
 const path = require("path");
+const webpack = require("webpack");
 const TerserPlugin = require("terser-webpack-plugin");

 module.exports = {
-  entry: "./main.ts",
+  entry: "./src/main.ts",
  target: "web",
  output: {
    filename: "main.js",
    path: __dirname,
    libraryTarget: "commonjs",
  },
+  plugins: [
+    // Work around for Buffer is undefined:
+    // https://github.com/webpack/changelog-v5/issues/10
+    new webpack.ProvidePlugin({
+      Buffer: ["buffer", "Buffer"],
+    }),
+    new webpack.ProvidePlugin({
+      process: "process/browser",
+    }),
+  ],
  module: {
    rules: [
      {
@@ -20,31 +31,31 @@ module.exports = {
  },
  resolve: {
    extensions: [".tsx", ".ts", ".js"],
-    mainFields: ["module", "main"],
+    mainFields: ["browser", "module", "main"],
    fallback: {
-      assert: require.resolve("assert"),
-      buffer: require.resolve("buffer/"),
-      console: require.resolve("console-browserify"),
-      constants: require.resolve("constants-browserify"),
-      crypto: require.resolve("crypto-browserify"),
-      domain: require.resolve("domain-browser"),
-      events: require.resolve("events"),
-      http: require.resolve("stream-http"),
-      https: require.resolve("https-browserify"),
-      os: require.resolve("os-browserify/browser"),
+      // assert: require.resolve("assert"),
+      // buffer: require.resolve("buffer/"),
+      // console: require.resolve("console-browserify"),
+      // constants: require.resolve("constants-browserify"),
+      // crypto: require.resolve("crypto-browserify"),
+      // domain: require.resolve("domain-browser"),
+      // events: require.resolve("events"),
+      // http: require.resolve("stream-http"),
+      // https: require.resolve("https-browserify"),
+      // os: require.resolve("os-browserify/browser"),
      path: require.resolve("path-browserify"),
-      punycode: require.resolve("punycode"),
+      // punycode: require.resolve("punycode"),
      process: require.resolve("process/browser"),
-      querystring: require.resolve("querystring-es3"),
+      // querystring: require.resolve("querystring-es3"),
      stream: require.resolve("stream-browserify"),
-      string_decoder: require.resolve("string_decoder"),
-      sys: require.resolve("util"),
-      timers: require.resolve("timers-browserify"),
-      tty: require.resolve("tty-browserify"),
-      url: require.resolve("url"),
-      util: require.resolve("util"),
-      vm: require.resolve("vm-browserify"),
-      zlib: require.resolve("browserify-zlib"),
+      // string_decoder: require.resolve("string_decoder"),
+      // sys: require.resolve("util"),
+      // timers: require.resolve("timers-browserify"),
+      // tty: require.resolve("tty-browserify"),
+      // url: require.resolve("url"),
+      // util: require.resolve("util"),
+      // vm: require.resolve("vm-browserify"),
+      // zlib: require.resolve("browserify-zlib"),
    },
  },
  externals: {
Author	SHA1	Message	Date
fyears	fde4c9a294	0.0.11	2021-11-08 00:40:45 +08:00
fyears	7fc38a942a	check password	2021-11-08 00:39:17 +08:00
fyears	cc0c4fe224	second confirm for password changes	2021-11-08 00:17:42 +08:00
fyears	2ab5640c12	list all objects	2021-11-07 17:15:06 +08:00
fyears	b7a0376a0f	allow clearing sync mappings	2021-11-07 16:02:03 +08:00
fyears	d7d1c47e2a	long desc for debug	2021-11-07 15:44:14 +08:00
fyears	a65f5eb004	head bucket	2021-11-07 15:43:57 +08:00
fyears	2ce34379ad	format doc	2021-11-07 15:27:41 +08:00
fyears	40d2ee95b7	0.0.10	2021-11-07 15:01:12 +08:00
fyears	5632e28331	make decision works	2021-11-07 15:00:22 +08:00
fyears	2b53a1bf5e	to 0.0.9	2021-11-07 14:54:25 +08:00
fyears	ccd9e3116c	add cross-env	2021-11-07 14:51:00 +08:00
fyears	e8d6a50793	update algo	2021-11-07 14:24:24 +08:00
fyears	c43b36e0e7	0.0.8	2021-11-07 14:09:22 +08:00
fyears	bcd92d135f	remove unused log	2021-11-07 14:05:03 +08:00
fyears	62aea9d330	debug mode: export sync plans	2021-11-07 13:58:51 +08:00
fyears	b9c17076b4	reorg misc tests	2021-11-07 13:13:40 +08:00
fyears	385290c1cc	find hidden files	2021-11-07 12:14:14 +08:00
fyears	371089e3ee	rename tests	2021-11-07 12:02:36 +08:00
fyears	254790cd61	abstract getSyncPlan	2021-11-07 00:39:57 +08:00
fyears	d3b812c8fa	split settings items	2021-11-07 00:24:02 +08:00
fyears	3a14f86880	fix error msg	2021-11-07 00:07:10 +08:00
fyears	887f948dc3	trim settings	2021-11-07 00:01:15 +08:00
fyears	035beb7d97	not destroy dbs while unloading	2021-11-06 23:48:15 +08:00
fyears	f334198241	more log	2021-11-06 23:41:55 +08:00
fyears	bcca2b4446	more accurate stages of sync	2021-11-06 23:31:49 +08:00
fyears	659deb9830	prettier way	2021-11-06 23:02:54 +08:00
fyears	b6df3f7ed5	promise all sync	2021-11-06 22:53:52 +08:00
fyears	2012758383	update readme	2021-11-06 13:15:54 +08:00
fyears	bea0e694f6	try lfs on ci	2021-11-06 13:06:17 +08:00
fyears	8fbbae6c2e	test for enc	2021-11-06 12:57:40 +08:00
fyears	ec2fec9bf8	no need console.log	2021-11-06 00:26:44 +08:00
fyears	09d33523b6	bump to 0.0.7	2021-11-06 00:25:05 +08:00
fyears	170cbbfb4f	add test	2021-11-06 00:23:59 +08:00
fyears	9d5fb7e966	test only on main branch	2021-11-05 01:31:18 +08:00
fyears	dc11d21d72	test on ci	2021-11-05 01:30:36 +08:00
fyears	774e31ef27	fix test	2021-11-05 01:29:44 +08:00
fyears	7c4a981434	allow tests	2021-11-05 01:21:58 +08:00
fyears	358311e8ac	spread Uint8Arrays	2021-11-05 01:21:33 +08:00
fyears	2551462003	clean unnecessary packages	2021-11-04 23:58:20 +08:00
fyears	60d5b78df0	fix length	2021-11-04 10:16:07 +08:00
fyears	f04017e05b	init web crypto	2021-11-04 10:10:05 +08:00
fyears	cb0098ec85	fix delete	2021-11-03 12:04:11 +08:00
fyears	d007d5f230	bump 0.0.6	2021-11-03 01:05:16 +08:00
fyears	07c6fbe747	change password position	2021-11-03 01:05:07 +08:00
fyears	5e3d413b33	bump ver	2021-11-03 00:58:48 +08:00
fyears	bcf8586fbf	use hash-wasm	2021-11-03 00:57:57 +08:00
fyears	6431182ec3	promisify data	2021-11-03 00:15:23 +08:00
fyears	cef623f493	bump ver	2021-11-02 10:14:11 +08:00
fyears	0c00c1a2e3	no need cryptojs?	2021-11-02 10:14:02 +08:00
fyears	ab8805f272	use default enc	2021-11-02 10:11:45 +08:00
fyears	da180fcc92	overwrite buffer	2021-11-02 10:07:00 +08:00
fyears	a9d80bef8f	yes encryption!	2021-11-02 02:39:24 +08:00
fyears	010561596e	format	2021-11-02 00:50:12 +08:00
fyears	9fc67e37f6	Merge pull request #2 from fyears/enc merge enc to master	2021-10-28 01:54:03 +08:00
fyears	151ae50a07	trigger ci	2021-10-28 01:53:16 +08:00
fyears	b0c6165fee	latest node	2021-10-28 01:50:56 +08:00
fyears	6a3ed65e06	no cache yet	2021-10-28 01:48:31 +08:00
fyears	4395ab508f	Create auto-build.yml	2021-10-28 01:47:10 +08:00
fyears	a681ceb4f5	builable	2021-10-27 10:15:14 +08:00
fyears	58cd51d776	basic encrypt	2021-10-27 00:33:35 +08:00
fyears	b450fa5c44	bump ver	2021-10-27 00:32:56 +08:00
fyears	d9ccd2c7eb	deal with empty remote	2021-10-25 01:01:04 +08:00
fyears	f58a71c1c4	bump version	2021-10-25 00:46:03 +08:00
fyears	69bc1f0e03	add mapping of local and remote	2021-10-25 00:41:13 +08:00
fyears	5ef032523a	skip same file locally	2021-10-24 22:12:32 +08:00
fyears	5a56526de1	avoid rewriting dist main.js	2021-10-24 20:40:30 +08:00
fyears	a26158055d	basically working 2 way sync!	2021-10-24 20:38:04 +08:00
fyears	13e5af0c34	prettier docs	2021-10-24 20:36:40 +08:00
fyears	29145604be	add code design	2021-10-24 13:12:45 +08:00
fyears	21f8789b27	more situations	2021-10-24 13:02:51 +08:00
fyears	0b26898c99	split func to misc	2021-10-23 16:51:46 +08:00
fyears	fa11b3fe7c	polish algo	2021-10-23 16:44:04 +08:00
fyears	8800c16b91	force editors	2021-10-23 15:48:42 +08:00
fyears	a5155b06d8	note down algorithm	2021-10-23 12:34:45 +08:00
fyears	e43a23a93d	no parcel	2021-10-23 12:12:24 +08:00
fyears	c75962aad5	track local history	2021-10-23 12:02:03 +08:00