manually delete indexeddb

more debug
0.0.15
2021-11-14 20:51:29 +08:00 · 2021-11-14 20:47:47 +08:00 · 2021-11-14 20:25:20 +08:00 · 2021-11-14 20:24:33 +08:00 · 2021-11-10 01:06:00 +08:00 · 2021-11-10 01:04:09 +08:00
23 changed files with 1277 additions and 477 deletions
@@ -0,0 +1,5 @@
+*.png filter=lfs diff=lfs merge=lfs -text
+*.jpg filter=lfs diff=lfs merge=lfs -text
+*.jpeg filter=lfs diff=lfs merge=lfs -text
+*.pdf filter=lfs diff=lfs merge=lfs -text
+*.enc filter=lfs diff=lfs merge=lfs -text
@@ -5,9 +5,9 @@ name: BuildCI

 on:
  push:
-    branches: [master, enc]
+    branches: [master]
  pull_request:
-    branches: [master, enc]
+    branches: [master]

 jobs:
  build:
@@ -19,12 +19,18 @@ jobs:
        # See supported Node.js release schedule at https://nodejs.org/en/about/releases/

    steps:
-      - uses: actions/checkout@v2
+      - name: Checkout codes
+        uses: actions/checkout@v2
+        with:
+          lfs: true
+      - name: Checkout LFS
+        run: git lfs checkout
      - name: Use Node.js ${{ matrix.node-version }}
        uses: actions/setup-node@v2
        with:
          node-version: ${{ matrix.node-version }}
      - run: npm install
+      - run: npm test
      - run: npm run build
      - uses: actions/upload-artifact@v2
        with:
@@ -32,3 +38,4 @@ jobs:
          path: |
            main.js
            manifest.json
+            styles.css
@@ -2,6 +2,14 @@

 This is yet another sync plugin for Obsidian.

+## Download and Install
+
+[![BuildCI](https://github.com/fyears/obsidian-save-remote/actions/workflows/auto-build.yml/badge.svg)](https://github.com/fyears/obsidian-save-remote/actions/workflows/auto-build.yml)
+
+Every artifacts are placed in the "Summary" under every successful builds.
+
+Besides manually download the files, you can also use [Obsidian42 - BRAT](https://github.com/TfTHacker/obsidian42-brat) to install this plugin.
+
 ## Disclaimer

 **This is NOT the official sync service provided by Obsidian.**
@@ -14,5 +22,5 @@ As of October 2021, the plugin is under development. **DO NOT USE IT for any ser

 Remote service in support list:

- [ ] AWS S3
+- [x] AWS S3
 - [ ] webdav
@@ -14,16 +14,16 @@ Assuming all sources are reliable.

 We list all combinations mutually exclusive and collectively exhaustive.

-| ID  | Remote Files | Local files | Local delete rename history | Extra                                                      | Decision                                                                                          |
-| --- | ------------ | ----------- | --------------------------- | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------- |
-| 1   | exist        | exist       | ignore                      | mtime_remote > mtime_local                                 | download remote file, create local folder if not exists, clear local history if exists            |
-| 2   | exist        | exist       | ignore                      | mtime_remote < mtime_local                                 | upload local file, create remote folder if not exists, clear local history if exists              |
-| 3   | exist        | exist       | ignore                      | mtime_remote === mtime_local && size_remote === size_local | clear local history if exists (the file was synced and no changes after last sync)                |
-| 4   | exist        | exist       | ignore                      | mtime_remote === mtime_local && size_remote !== size_local | upload local file, clear local history if exists (we always prefer local to remote)               |
-| 5   | exist        | exist       | ignore                      | If local is a folder. mtime_local === undefined            | clear local history if exists. TODO: what if a folder and a previous file share the same name?    |
-| 6   | exist        | not exist   | exist                       | mtime_remote >= delete_time_local                          | download remote file, create folder if not exists                                                 |
-| 7   | exist        | not exist   | exist                       | mtime_remote < delete_time_local                           | delete remote file, clear local history                                                           |
-| 8   | exist        | not exist   | not exist                   |                                                            | download remote file, create folder if not exists                                                 |
-| 9   | not exist    | exist       | ignore                      | If local is a single file.                                 | upload local file, create remote folder if not exists, clear local history if exists              |
-| 10  | not exist    | exist       | ignore                      | If local is a folder.                                      | upload local files recursively, create remote folder if not exists, clear local history if exists |
-| 11  | not exist    | not exist   | ignore                      |                                                            | clear local history if exists                                                                     |
+| ID  | Remote Files | Local files | Local delete rename history | Extra                                                                         | Decision                                                                                                 |
+| --- | ------------ | ----------- | --------------------------- | ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
+| 1   | exist        | exist       | ignore                      | mtime_remote > mtime_local                                                    | download remote file, create local folder if not exists, clear local history if exists                   |
+| 2   | exist        | exist       | ignore                      | mtime_remote < mtime_local                                                    | upload local file, create remote folder if not exists, clear local history if exists                     |
+| 3   | exist        | exist       | ignore                      | mtime_remote === mtime_local && password === "" && size_remote === size_local | clear local history if exists (the file was synced and no changes after last sync)                       |
+| 4   | exist        | exist       | ignore                      | mtime_remote === mtime_local && password === "" && size_remote !== size_local | upload local file, clear local history if exists (we always prefer local to remote)                      |
+| 5   | exist        | exist       | ignore                      | mtime_remote === mtime_local && password !== ""                               | clear local history if exists (in encryption mode, file sizes are unequal. we can only rely on mtime(s)) |
+| 6   | exist        | exist       | ignore                      | If local is a folder. mtime_local === undefined                               | clear local history if exists. TODO: what if a folder and a previous file share the same name?           |
+| 7   | exist        | not exist   | exist                       | mtime_remote >= delete_time_local                                             | download remote file, create folder if not exists                                                        |
+| 8   | exist        | not exist   | exist                       | mtime_remote < delete_time_local                                              | delete remote file, clear local history                                                                  |
+| 9   | exist        | not exist   | not exist                   |                                                                               | download remote file, create folder if not exists                                                        |
+| 10  | not exist    | exist       | ignore                      | local may be folder or file                                                   | upload local files recursively, create remote folder if not exists, clear local history if exists        |
+| 11  | not exist    | not exist   | ignore                      |                                                                               | clear local history if exists                                                                            |
@@ -1,7 +1,7 @@
 {
-  "id": "obsdian-save-remote",
+  "id": "obsidian-save-remote",
  "name": "Save remote",
-  "version": "0.0.4",
+  "version": "0.0.15",
  "minAppVersion": "0.12.15",
  "description": "This is yet another plugin allowing users to sync notes between local device and the cloud.",
  "author": "fyears",
@@ -1,22 +1,32 @@
 {
  "name": "obsidian-save-remote",
-  "version": "0.0.4",
+  "version": "0.0.15",
  "description": "This is yet another sync plugin for Obsidian app.",
  "scripts": {
    "dev": "webpack --mode development --watch",
    "build": "webpack --mode production",
    "format": "npx prettier --write .",
-    "clean": "npx rimraf main.js"
+    "clean": "npx rimraf main.js",
+    "test": "cross-env TS_NODE_COMPILER_OPTIONS={\\\"module\\\":\\\"commonjs\\\"} mocha -r ts-node/register 'tests/**/*.ts'"
  },
  "source": "main.ts",
  "keywords": [],
  "author": "",
  "license": "Apache-2.0",
  "devDependencies": {
+    "@types/chai": "^4.2.22",
+    "@types/chai-as-promised": "^7.1.4",
+    "@types/mime-types": "^2.1.1",
+    "@types/mocha": "^9.0.0",
    "@types/node": "^14.14.37",
+    "chai": "^4.3.4",
+    "chai-as-promised": "^7.1.1",
+    "cross-env": "^7.0.3",
+    "mocha": "^9.1.3",
    "prettier": "^2.4.1",
    "terser-webpack-plugin": "^5.2.4",
    "ts-loader": "^9.2.6",
+    "ts-node": "^10.4.0",
    "tslib": "^2.2.0",
    "typescript": "^4.4.4",
    "webdav-server": "^2.6.2",
@@ -25,39 +35,22 @@
  },
  "dependencies": {
    "@aws-sdk/client-s3": "^3.37.0",
+    "@aws-sdk/lib-storage": "^3.40.1",
    "@aws-sdk/signature-v4-crt": "^3.37.0",
-    "@types/mime-types": "^2.1.1",
    "acorn": "^8.5.0",
-    "assert": "^2.0.0",
    "aws-crt": "^1.10.1",
-    "browserify-zlib": "^0.2.0",
    "buffer": "^6.0.3",
    "codemirror": "^5.63.1",
-    "console-browserify": "^1.2.0",
-    "constants-browserify": "^1.0.0",
-    "crypto-browserify": "^3.12.0",
-    "domain-browser": "^4.22.0",
-    "events": "^3.3.0",
-    "hi-base32": "^0.5.1",
-    "https-browserify": "^1.0.0",
-    "lovefield-ts": "^0.7.0",
+    "localforage": "^1.10.0",
    "mime-types": "^2.1.33",
    "obsidian": "^0.12.0",
-    "os-browserify": "^0.3.0",
    "path-browserify": "^1.0.1",
    "process": "^0.11.10",
-    "punycode": "^2.1.1",
-    "querystring-es3": "^0.2.1",
+    "rfc4648": "^1.5.0",
    "rimraf": "^3.0.2",
    "stream-browserify": "^3.0.0",
-    "stream-http": "^3.2.0",
-    "string_decoder": "^1.3.0",
-    "timers-browserify": "^2.0.12",
-    "tty-browserify": "0.0.1",
-    "url": "^0.11.0",
-    "util": "^0.12.4",
-    "vm-browserify": "^1.1.2",
    "webdav": "^4.7.0",
-    "webdav-fs": "^4.0.0"
+    "webdav-fs": "^4.0.0",
+    "xregexp": "^5.1.0"
  }
 }
@@ -0,0 +1,29 @@
+import { TAbstractFile, TFolder, TFile, Vault } from "obsidian";
+
+import type { SyncPlanType } from "./sync";
+import { readAllSyncPlanRecordTexts } from "./localdb";
+import type { InternalDBs } from "./localdb";
+import { mkdirpInVault } from "./misc";
+
+const DEFAULT_DEBUG_FOLDER = "_debug_save_remote/";
+const DEFAULT_SYNC_PLANS_HISTORY_FILE_PREFIX = "sync_plans_hist_exported_on_";
+
+export const exportSyncPlansToFiles = async (db: InternalDBs, vault: Vault) => {
+  console.log("exporting");
+  await mkdirpInVault(DEFAULT_DEBUG_FOLDER, vault);
+  const records = await readAllSyncPlanRecordTexts(db);
+  let md = "";
+  if (records.length === 0) {
+    md = "No sync plans history found";
+  } else {
+    md =
+      "Sync plans found:\n\n" +
+      records.map((x) => "```json\n" + x + "\n```\n").join("\n");
+  }
+  const ts = Date.now();
+  const filePath = `${DEFAULT_DEBUG_FOLDER}${DEFAULT_SYNC_PLANS_HISTORY_FILE_PREFIX}${ts}.md`;
+  await vault.create(filePath, md, {
+    mtime: ts,
+  });
+  console.log("finish exporting");
+};
@@ -1,91 +1,138 @@
-import * as crypto from "crypto";
-import * as base32 from "hi-base32";
-import { bufferToArrayBuffer, arrayBufferToBuffer } from "./misc";
+import { base32, base64 } from "rfc4648";
+import {
+  bufferToArrayBuffer,
+  arrayBufferToBuffer,
+  hexStringToTypedArray,
+  arrayBufferToHex,
+} from "./misc";

 const DEFAULT_ITER = 10000;

-export const encryptBuffer = (
-  buf: Buffer,
+// base32.stringify(Buffer.from('Salted__'))
+export const MAGIC_ENCRYPTED_PREFIX_BASE32 = "KNQWY5DFMRPV";
+
+const getKeyIVFromPassword = async (
+  salt: Uint8Array,
  password: string,
  rounds: number = DEFAULT_ITER
 ) => {
-  const salt = crypto.randomBytes(8);
-  const derivedKey = crypto.pbkdf2Sync(
-    password,
-    salt,
-    rounds,
-    32 + 16,
-    "sha256"
+  const k1 = await window.crypto.subtle.importKey(
+    "raw",
+    new TextEncoder().encode(password),
+    { name: "PBKDF2" },
+    false,
+    ["deriveKey", "deriveBits"]
  );
-  const key = derivedKey.slice(0, 32);
-  const iv = derivedKey.slice(32, 32 + 16);
-  const cipher = crypto.createCipheriv("aes-256-cbc", key, iv);
-  cipher.write(buf);
-  cipher.end();
-  const encrypted = cipher.read();
-  const res = Buffer.concat([Buffer.from("Salted__"), salt, encrypted]);
-  return res;
+
+  const k2 = await window.crypto.subtle.deriveBits(
+    {
+      name: "PBKDF2",
+      salt: salt,
+      iterations: rounds,
+      hash: "SHA-256",
+    },
+    k1,
+    256 + 128
+  );
+
+  return k2;
 };

-export const decryptBuffer = (
-  buf: Buffer,
+export const encryptArrayBuffer = async (
+  arrBuf: ArrayBuffer,
  password: string,
-  rounds: number = DEFAULT_ITER
+  rounds: number = DEFAULT_ITER,
+  saltHex: string = ""
 ) => {
-  const prefix = buf.slice(0, 8);
-  const salt = buf.slice(8, 16);
-  const derivedKey = crypto.pbkdf2Sync(
-    password,
-    salt,
-    rounds,
-    32 + 16,
-    "sha256"
-  );
+  let salt: Uint8Array;
+  if (saltHex !== "") {
+    salt = hexStringToTypedArray(saltHex);
+  } else {
+    salt = window.crypto.getRandomValues(new Uint8Array(8));
+  }
+
+  const derivedKey = await getKeyIVFromPassword(salt, password, rounds);
  const key = derivedKey.slice(0, 32);
  const iv = derivedKey.slice(32, 32 + 16);
-  const decipher = crypto.createDecipheriv("aes-256-cbc", key, iv);
-  decipher.write(buf.slice(16));
-  decipher.end();
-  const decrypted = decipher.read();
-  return decrypted as Buffer;
+
+  const keyCrypt = await window.crypto.subtle.importKey(
+    "raw",
+    key,
+    { name: "AES-CBC" },
+    false,
+    ["encrypt", "decrypt"]
+  );
+
+  const enc = (await window.crypto.subtle.encrypt(
+    { name: "AES-CBC", iv },
+    keyCrypt,
+    arrBuf
+  )) as ArrayBuffer;
+
+  const prefix = new TextEncoder().encode("Salted__");
+
+  const res = new Uint8Array([...prefix, ...salt, ...new Uint8Array(enc)]);
+
+  return bufferToArrayBuffer(res);
 };

-export const encryptArrayBuffer = (
+export const decryptArrayBuffer = async (
  arrBuf: ArrayBuffer,
  password: string,
  rounds: number = DEFAULT_ITER
 ) => {
-  return bufferToArrayBuffer(
-    encryptBuffer(arrayBufferToBuffer(arrBuf), password, rounds)
-  );
-};
-
-export const decryptArrayBuffer = (
-  arrBuf: ArrayBuffer,
-  password: string,
-  rounds: number = DEFAULT_ITER
-) => {
-  return bufferToArrayBuffer(
-    decryptBuffer(arrayBufferToBuffer(arrBuf), password, rounds)
-  );
-};
-
-export const encryptStringToBase32 = (
-  text: string,
-  password: string,
-  rounds: number = DEFAULT_ITER
-) => {
-  return base32.encode(encryptBuffer(Buffer.from(text), password, rounds));
-};
-
-export const decryptBase32ToString = (
-  text: string,
-  password: string,
-  rounds: number = DEFAULT_ITER
-) => {
-  return decryptBuffer(
-    Buffer.from(base32.decode.asBytes(text)),
+  const prefix = arrBuf.slice(0, 8);
+  const salt = arrBuf.slice(8, 16);
+  const derivedKey = await getKeyIVFromPassword(
+    new Uint8Array(salt),
    password,
    rounds
-  ).toString();
+  );
+  const key = derivedKey.slice(0, 32);
+  const iv = derivedKey.slice(32, 32 + 16);
+
+  const keyCrypt = await window.crypto.subtle.importKey(
+    "raw",
+    key,
+    { name: "AES-CBC" },
+    false,
+    ["encrypt", "decrypt"]
+  );
+
+  const dec = (await window.crypto.subtle.decrypt(
+    { name: "AES-CBC", iv },
+    keyCrypt,
+    arrBuf.slice(16)
+  )) as ArrayBuffer;
+
+  return dec;
+};
+
+export const encryptStringToBase32 = async (
+  text: string,
+  password: string,
+  rounds: number = DEFAULT_ITER,
+  saltHex: string = ""
+) => {
+  const enc = await encryptArrayBuffer(
+    bufferToArrayBuffer(new TextEncoder().encode(text)),
+    password,
+    rounds,
+    saltHex
+  );
+  return base32.stringify(new Uint8Array(enc));
+};
+
+export const decryptBase32ToString = async (
+  text: string,
+  password: string,
+  rounds: number = DEFAULT_ITER
+) => {
+  return new TextDecoder().decode(
+    await decryptArrayBuffer(
+      bufferToArrayBuffer(base32.parse(text)),
+      password,
+      rounds
+    )
+  );
 };
@@ -1,82 +1,92 @@
-import * as lf from "lovefield-ts/dist/es6/lf.js";
+import localforage from "localforage";
 import { TAbstractFile, TFile, TFolder } from "obsidian";

 import type { SUPPORTED_SERVICES_TYPE } from "./misc";
+import type { SyncPlanType } from "./sync";

-export type DatabaseConnection = lf.DatabaseConnection;
+export type LocalForage = typeof localforage;

+export const DEFAULT_DB_VERSION_NUMBER: number = 20211114;
 export const DEFAULT_DB_NAME = "saveremotedb";
+export const DEFAULT_TBL_VERSION = "schemaversion";
 export const DEFAULT_TBL_DELETE_HISTORY = "filefolderoperationhistory";
 export const DEFAULT_TBL_SYNC_MAPPING = "syncmetadatahistory";
+export const DEFAULT_SYNC_PLANS_HISTORY = "syncplanshistory";

 export interface FileFolderHistoryRecord {
  key: string;
  ctime: number;
  mtime: number;
  size: number;
-  action_when: number;
-  action_type: "delete" | "rename";
-  key_type: "folder" | "file";
-  rename_to: string;
+  actionWhen: number;
+  actionType: "delete" | "rename";
+  keyType: "folder" | "file";
+  renameTo: string;
 }

-export interface SyncMetaMappingRecord {
-  local_key: string;
-  remote_key: string;
-  local_size: number;
-  remote_size: number;
-  local_mtime: number;
-  remote_mtime: number;
-  remote_extra_key: string;
-  remote_type: SUPPORTED_SERVICES_TYPE;
-  key_type: "folder" | "file";
+interface SyncMetaMappingRecord {
+  localKey: string;
+  remoteKey: string;
+  localSize: number;
+  remoteSize: number;
+  localMtime: number;
+  remoteMtime: number;
+  remoteExtraKey: string;
+  remoteType: SUPPORTED_SERVICES_TYPE;
+  keyType: "folder" | "file";
+}
+
+interface SyncPlanRecord {
+  ts: number;
+  remoteType: string;
+  syncPlan: string;
+}
+
+export interface InternalDBs {
+  versionTbl: LocalForage;
+  deleteHistoryTbl: LocalForage;
+  syncMappingTbl: LocalForage;
+  syncPlansTbl: LocalForage;
 }

 export const prepareDBs = async () => {
-  const schemaBuilder = lf.schema.create(DEFAULT_DB_NAME, 1);
-  schemaBuilder
-    .createTable(DEFAULT_TBL_DELETE_HISTORY)
-    .addColumn("id", lf.Type.INTEGER)
-    .addColumn("key", lf.Type.STRING)
-    .addColumn("ctime", lf.Type.INTEGER)
-    .addColumn("mtime", lf.Type.INTEGER)
-    .addColumn("size", lf.Type.INTEGER)
-    .addColumn("action_when", lf.Type.INTEGER)
-    .addColumn("action_type", lf.Type.STRING)
-    .addColumn("key_type", lf.Type.STRING)
-    .addPrimaryKey(["id"], true)
-    .addIndex("idxKey", ["key"]);
+  const db = {
+    versionTbl: localforage.createInstance({
+      name: DEFAULT_DB_NAME,
+      storeName: DEFAULT_TBL_VERSION,
+    }),
+    deleteHistoryTbl: localforage.createInstance({
+      name: DEFAULT_DB_NAME,
+      storeName: DEFAULT_TBL_DELETE_HISTORY,
+    }),
+    syncMappingTbl: localforage.createInstance({
+      name: DEFAULT_DB_NAME,
+      storeName: DEFAULT_TBL_SYNC_MAPPING,
+    }),
+    syncPlansTbl: localforage.createInstance({
+      name: DEFAULT_DB_NAME,
+      storeName: DEFAULT_SYNC_PLANS_HISTORY,
+    }),
+  } as InternalDBs;

-  schemaBuilder
-    .createTable(DEFAULT_TBL_SYNC_MAPPING)
-    .addColumn("id", lf.Type.INTEGER)
-    .addColumn("local_key", lf.Type.STRING)
-    .addColumn("remote_key", lf.Type.STRING)
-    .addColumn("local_size", lf.Type.INTEGER)
-    .addColumn("remote_size", lf.Type.INTEGER)
-    .addColumn("local_mtime", lf.Type.INTEGER)
-    .addColumn("remote_mtime", lf.Type.INTEGER)
-    .addColumn("key_type", lf.Type.STRING)
-    .addColumn("remote_extra_key", lf.Type.STRING)
-    .addColumn("remote_type", lf.Type.STRING)
-    .addNullable([
-      "remote_extra_key",
-      "remote_mtime",
-      "remote_size",
-      "local_mtime",
-    ])
-    .addPrimaryKey(["id"], true)
-    .addIndex("idxkey", ["local_key", "remote_key"]);
+  const originalVersion = (await db.versionTbl.getItem("version")) as number;
+  if (originalVersion === null) {
+    await db.versionTbl.setItem("version", DEFAULT_DB_VERSION_NUMBER);
+  } else if (originalVersion === DEFAULT_DB_VERSION_NUMBER) {
+    // do nothing
+  } else {
+    await migrateDBs(db, originalVersion, DEFAULT_DB_VERSION_NUMBER);
+  }

-  const db = await schemaBuilder.connect({
-    storeType: lf.DataStoreType.INDEXED_DB,
-  });
  console.log("db connected");
  return db;
 };

-export const destroyDBs = async (db: lf.DatabaseConnection) => {
-  db.close();
+export const destroyDBs = async () => {
+  // await localforage.dropInstance({
+  //   name: DEFAULT_DB_NAME,
+  // });
+  // console.log("db deleted");
  const req = indexedDB.deleteDatabase(DEFAULT_DB_NAME);
  req.onsuccess = (event) => {
    console.log("db deleted");
@@ -90,37 +100,34 @@ export const destroyDBs = async (db: lf.DatabaseConnection) => {
  };
 };

-export const loadDeleteRenameHistoryTable = async (
-  db: lf.DatabaseConnection
-) => {
-  const schema = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
-  const tbl = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
+const migrateDBs = async (db: InternalDBs, oldVer: number, newVer: number) => {
+  if (oldVer === newVer) {
+    return;
+  }
+  // not implemented
+  throw Error(`not supported internal db changes from ${oldVer} to ${newVer}`);
+};

-  const records = await db
-    .select()
-    .from(schema)
-    .orderBy(schema.col("action_when"), lf.Order.ASC)
-    .exec();
-
-  return records as FileFolderHistoryRecord[];
+export const loadDeleteRenameHistoryTable = async (db: InternalDBs) => {
+  const records = [] as FileFolderHistoryRecord[];
+  await db.deleteHistoryTbl.iterate((value, key, iterationNumber) => {
+    records.push(value as FileFolderHistoryRecord);
+  });
+  records.sort((a, b) => a.actionWhen - b.actionWhen); // ascending
+  return records;
 };

 export const clearDeleteRenameHistoryOfKey = async (
-  db: lf.DatabaseConnection,
+  db: InternalDBs,
  key: string
 ) => {
-  const schema = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
-  const tbl = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
-
-  await db.delete().from(tbl).where(tbl.col("key").eq(key)).exec();
+  await db.deleteHistoryTbl.removeItem(key);
 };

 export const insertDeleteRecord = async (
-  db: lf.DatabaseConnection,
+  db: InternalDBs,
  fileOrFolder: TAbstractFile
 ) => {
-  const schema = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
-  const tbl = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
  // console.log(fileOrFolder);
  let k: FileFolderHistoryRecord;
  if (fileOrFolder instanceof TFile) {
@@ -129,10 +136,10 @@ export const insertDeleteRecord = async (
      ctime: fileOrFolder.stat.ctime,
      mtime: fileOrFolder.stat.mtime,
      size: fileOrFolder.stat.size,
-      action_when: Date.now(),
-      action_type: "delete",
-      key_type: "file",
-      rename_to: "",
+      actionWhen: Date.now(),
+      actionType: "delete",
+      keyType: "file",
+      renameTo: "",
    };
  } else if (fileOrFolder instanceof TFolder) {
    // key should endswith "/"
@@ -144,23 +151,20 @@ export const insertDeleteRecord = async (
      ctime: 0,
      mtime: 0,
      size: 0,
-      action_when: Date.now(),
-      action_type: "delete",
-      key_type: "folder",
-      rename_to: "",
+      actionWhen: Date.now(),
+      actionType: "delete",
+      keyType: "folder",
+      renameTo: "",
    };
  }
-  const row = tbl.createRow(k);
-  await db.insertOrReplace().into(tbl).values([row]).exec();
+  await db.deleteHistoryTbl.setItem(k.key, k);
 };

 export const insertRenameRecord = async (
-  db: lf.DatabaseConnection,
+  db: InternalDBs,
  fileOrFolder: TAbstractFile,
  oldPath: string
 ) => {
-  const schema = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
-  const tbl = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
  // console.log(fileOrFolder);
  let k: FileFolderHistoryRecord;
  if (fileOrFolder instanceof TFile) {
@@ -169,10 +173,10 @@ export const insertRenameRecord = async (
      ctime: fileOrFolder.stat.ctime,
      mtime: fileOrFolder.stat.mtime,
      size: fileOrFolder.stat.size,
-      action_when: Date.now(),
-      action_type: "rename",
-      key_type: "file",
-      rename_to: fileOrFolder.path,
+      actionWhen: Date.now(),
+      actionType: "rename",
+      keyType: "file",
+      renameTo: fileOrFolder.path,
    };
  } else if (fileOrFolder instanceof TFolder) {
    const key = oldPath.endsWith("/") ? oldPath : `${oldPath}/`;
@@ -184,25 +188,17 @@ export const insertRenameRecord = async (
      ctime: 0,
      mtime: 0,
      size: 0,
-      action_when: Date.now(),
-      action_type: "rename",
-      key_type: "folder",
-      rename_to: renameTo,
+      actionWhen: Date.now(),
+      actionType: "rename",
+      keyType: "folder",
+      renameTo: renameTo,
    };
  }
-  const row = tbl.createRow(k);
-  await db.insertOrReplace().into(tbl).values([row]).exec();
-};
-
-export const getAllDeleteRenameRecords = async (db: lf.DatabaseConnection) => {
-  const schema = db.getSchema().table(DEFAULT_TBL_DELETE_HISTORY);
-  const res1 = await db.select().from(schema).exec();
-  const res2 = res1 as FileFolderHistoryRecord[];
-  return res2;
+  await db.deleteHistoryTbl.setItem(k.key, k);
 };

 export const upsertSyncMetaMappingDataS3 = async (
-  db: lf.DatabaseConnection,
+  db: InternalDBs,
  localKey: string,
  localMTime: number,
  localSize: number,
@@ -211,50 +207,78 @@ export const upsertSyncMetaMappingDataS3 = async (
  remoteSize: number,
  remoteExtraKey: string /* ETag from s3 */
 ) => {
-  const schema = db.getSchema().table(DEFAULT_TBL_SYNC_MAPPING);
  const aggregratedInfo: SyncMetaMappingRecord = {
-    local_key: localKey,
-    local_mtime: localMTime,
-    local_size: localSize,
-    remote_key: remoteKey,
-    remote_mtime: remoteMTime,
-    remote_size: remoteSize,
-    remote_extra_key: remoteExtraKey,
-    remote_type: "s3",
-    key_type: localKey.endsWith("/") ? "folder" : "file",
+    localKey: localKey,
+    localMtime: localMTime,
+    localSize: localSize,
+    remoteKey: remoteKey,
+    remoteMtime: remoteMTime,
+    remoteSize: remoteSize,
+    remoteExtraKey: remoteExtraKey,
+    remoteType: "s3",
+    keyType: localKey.endsWith("/") ? "folder" : "file",
  };
-  const row = schema.createRow(aggregratedInfo);
-  await db.insertOrReplace().into(schema).values([row]).exec();
+  await db.syncMappingTbl.setItem(remoteKey, aggregratedInfo);
 };

 export const getSyncMetaMappingByRemoteKeyS3 = async (
-  db: lf.DatabaseConnection,
+  db: InternalDBs,
  remoteKey: string,
  remoteMTime: number,
  remoteExtraKey: string
 ) => {
-  const schema = db.getSchema().table(DEFAULT_TBL_SYNC_MAPPING);
-  const tbl = db.getSchema().table(DEFAULT_TBL_SYNC_MAPPING);
-  const res = (await db
-    .select()
-    .from(tbl)
-    .where(
-      lf.op.and(
-        tbl.col("remote_key").eq(remoteKey),
-        tbl.col("remote_mtime").eq(remoteMTime),
-        tbl.col("remote_extra_key").eq(remoteExtraKey),
-        tbl.col("remote_type").eq("s3")
-      )
-    )
-    .exec()) as SyncMetaMappingRecord[];
+  const potentialItem = (await db.syncMappingTbl.getItem(
+    remoteKey
+  )) as SyncMetaMappingRecord;

-  if (res.length === 1) {
-    return res[0];
-  }
-
-  if (res.length === 0) {
+  if (potentialItem === null) {
+    // no result was found
    return undefined;
  }

-  throw Error("something bad in sync meta mapping!");
+  if (
+    potentialItem.remoteKey === remoteKey &&
+    potentialItem.remoteMtime === remoteMTime &&
+    potentialItem.remoteExtraKey === remoteExtraKey &&
+    potentialItem.remoteType === "s3"
+  ) {
+    // the result was found
+    return potentialItem;
+  } else {
+    return undefined;
+  }
+};
+
+export const clearAllSyncMetaMapping = async (db: InternalDBs) => {
+  await db.syncMappingTbl.clear();
+};
+
+export const insertSyncPlanRecord = async (
+  db: InternalDBs,
+  syncPlan: SyncPlanType
+) => {
+  const record = {
+    ts: syncPlan.ts,
+    remoteType: syncPlan.remoteType,
+    syncPlan: JSON.stringify(syncPlan /* directly stringify */, null, 2),
+  } as SyncPlanRecord;
+  await db.syncPlansTbl.setItem(`${syncPlan.ts}`, record);
+};
+
+export const clearAllSyncPlanRecords = async (db: InternalDBs) => {
+  await db.syncPlansTbl.clear();
+};
+
+export const readAllSyncPlanRecordTexts = async (db: InternalDBs) => {
+  const records = [] as SyncPlanRecord[];
+  await db.syncPlansTbl.iterate((value, key, iterationNumber) => {
+    records.push(value as SyncPlanRecord);
+  });
+  records.sort((a, b) => -(a.ts - b.ts)); // descending
+
+  if (records === undefined) {
+    return [] as string[];
+  } else {
+    return records.map((x) => x.syncPlan);
+  }
 };
@@ -11,19 +11,28 @@ import {
  TFolder,
 } from "obsidian";
 import * as CodeMirror from "codemirror";
-import type { DatabaseConnection } from "./localdb";
 import {
  prepareDBs,
  destroyDBs,
  loadDeleteRenameHistoryTable,
+  clearAllSyncPlanRecords,
+  clearAllSyncMetaMapping,
  insertDeleteRecord,
  insertRenameRecord,
-  getAllDeleteRenameRecords,
+  insertSyncPlanRecord,
 } from "./localdb";
+import type { InternalDBs } from "./localdb";

-import type { SyncStatusType } from "./sync";
-import { ensembleMixedStates, getOperation, doActualSync } from "./sync";
-import { DEFAULT_S3_CONFIG, getS3Client, listFromRemote, S3Config } from "./s3";
+import type { SyncStatusType, PasswordCheckType } from "./sync";
+import { isPasswordOk, getSyncPlan, doActualSync } from "./sync";
+import {
+  DEFAULT_S3_CONFIG,
+  getS3Client,
+  listFromRemote,
+  S3Config,
+  checkS3Connectivity,
+} from "./s3";
+import { exportSyncPlansToFiles } from "./debugMode";

 interface SaveRemotePluginSettings {
  s3?: S3Config;
@@ -38,7 +47,7 @@ const DEFAULT_SETTINGS: SaveRemotePluginSettings = {
 export default class SaveRemotePlugin extends Plugin {
  settings: SaveRemotePluginSettings;
  cm: CodeMirror.Editor;
-  db: DatabaseConnection;
+  db: InternalDBs;
  syncStatus: SyncStatusType;

  async onload() {
@@ -62,59 +71,79 @@ export default class SaveRemotePlugin extends Plugin {
      })
    );

-    // this.addRibbonIcon("dice", "Misc", async () => {
-    //   const a = this.app.vault.getAllLoadedFiles();
-    //   console.log(a);
-    //   const h = await getAllRecords(this.db);
-    //   console.log(h);
-    // });
-
    this.addRibbonIcon("switch", "Save Remote", async () => {
      if (this.syncStatus !== "idle") {
-        new Notice("Save Remote already running!");
+        new Notice(`Save Remote already running in stage ${this.syncStatus}!`);
        return;
      }

-      new Notice("Save Remote Sync Preparing");
-      this.syncStatus = "preparing";
-      const s3Client = getS3Client(this.settings.s3);
-      const remoteRsp = await listFromRemote(s3Client, this.settings.s3);
-      const local = this.app.vault.getAllLoadedFiles();
-      const localHistory = await loadDeleteRenameHistoryTable(this.db);
-      // console.log(remoteRsp);
-      // console.log(local);
-      // console.log(localHistory);
+      try {
+        //console.log(`huh ${this.settings.password}`)
+        new Notice("1/6 Save Remote Sync Preparing");
+        this.syncStatus = "preparing";

-      const mixedStates = await ensembleMixedStates(
-        remoteRsp.Contents,
-        local,
-        localHistory,
-        this.db,
-        this.settings.password
-      );
+        new Notice("2/6 Starting to fetch remote meta data.");
+        this.syncStatus = "getting_remote_meta";
+        const s3Client = getS3Client(this.settings.s3);
+        const remoteRsp = await listFromRemote(s3Client, this.settings.s3);

-      for (const [key, val] of Object.entries(mixedStates)) {
-        getOperation(val, true);
+        new Notice("3/6 Starting to fetch local meta data.");
+        this.syncStatus = "getting_local_meta";
+        const local = this.app.vault.getAllLoadedFiles();
+        const localHistory = await loadDeleteRenameHistoryTable(this.db);
+        // console.log(remoteRsp);
+        // console.log(local);
+        // console.log(localHistory);
+
+        new Notice("4/7 Checking password correct or not.");
+        this.syncStatus = "checking_password";
+        const passwordCheckResult = await isPasswordOk(
+          remoteRsp.Contents,
+          this.settings.password
+        );
+        if (!passwordCheckResult.ok) {
+          new Notice("something goes wrong while checking password");
+          throw Error(passwordCheckResult.reason);
+        }
+
+        new Notice("5/7 Starting to generate sync plan.");
+        this.syncStatus = "generating_plan";
+        const syncPlan = await getSyncPlan(
+          remoteRsp.Contents,
+          local,
+          localHistory,
+          this.db,
+          this.settings.password
+        );
+        console.log(syncPlan.mixedStates); // for debugging
+        await insertSyncPlanRecord(this.db, syncPlan);
+
+        // The operations above are read only and kind of safe.
+        // The operations below begins to write or delete (!!!) something.
+
+        new Notice("6/7 Save Remote Sync data exchanging!");
+
+        this.syncStatus = "syncing";
+        await doActualSync(
+          s3Client,
+          this.settings.s3,
+          this.db,
+          this.app.vault,
+          syncPlan,
+          this.settings.password
+        );
+
+        new Notice("7/7 Save Remote finish!");
+        this.syncStatus = "finish";
+        this.syncStatus = "idle";
+      } catch (error) {
+        const msg = `Save Remote error while ${this.syncStatus}`;
+        console.log(msg);
+        console.log(error);
+        new Notice(msg);
+        new Notice(error.message);
+        this.syncStatus = "idle";
      }
-
-      console.log(mixedStates);
-
-      // The operations above are read only and kind of safe.
-      // The operations below begins to write or delete (!!!) something.
-
-      new Notice("Save Remote Sync data exchanging!");
-
-      await doActualSync(
-        s3Client,
-        this.settings.s3,
-        this.db,
-        this.app.vault,
-        mixedStates,
-        this.settings.password
-      );
-
-      new Notice("Save Remote finish!");
-      this.syncStatus = "idle";
    });

    this.addSettingTab(new SaveRemoteSettingTab(this.app, this));
@@ -151,7 +180,65 @@ export default class SaveRemotePlugin extends Plugin {
  }

  destroyDBs() {
-    destroyDBs(this.db);
+    /* destroyDBs(this.db); */
+  }
+}
+
+export class PasswordModal extends Modal {
+  plugin: SaveRemotePlugin;
+  newPassword: string;
+  constructor(app: App, plugin: SaveRemotePlugin, newPassword: string) {
+    super(app);
+    this.plugin = plugin;
+    this.newPassword = newPassword;
+  }
+
+  onOpen() {
+    let { contentEl } = this;
+    // contentEl.setText("Add Or change password.");
+    contentEl.createEl("h2", { text: "Hold on and PLEASE READ ON..." });
+    contentEl.createEl("p", {
+      text: "If the field is not empty, files are enctrypted using the password locally before sent to remote.",
+    });
+    contentEl.createEl("p", {
+      text: "If the field is empty, then no password is used, and files would be sent without encryption.",
+    });
+
+    contentEl.createEl("p", {
+      text: "Attention 1/4: The password itself is stored in PLAIN TEXT LOCALLY and would not be sent to remote by this plugin.",
+    });
+    contentEl.createEl("p", {
+      text: "Attention 2/4: Non-empty file contents are encrypted using openssl format. File/directory path are also encrypted then applied base32. BUT, some metadata such as file sizes and directory structures are not encrypted or can be easily guessed, and directory path are stored as 0-byte-size object remotely.",
+    });
+    contentEl.createEl("p", {
+      text: "Attention 3/4: Before changing password, you should make sure the remote store (s3/webdav/...) IS EMPTY, or REMOTE FILES WERE ENCRYPTED BY THAT NEW PASSWORD. OTHERWISE SOMETHING BAD WOULD HAPPEN!",
+    });
+    contentEl.createEl("p", {
+      text: "Attention 4/4: The longer the password, the better.",
+    });
+
+    new Setting(contentEl)
+      .addButton((button) => {
+        button.setButtonText("The Second Confirm to change password.");
+        button.onClick(async () => {
+          this.plugin.settings.password = this.newPassword;
+          await this.plugin.saveSettings();
+          new Notice("New password saved!");
+          this.close();
+        });
+        button.setClass("password_second_confirm");
+      })
+      .addButton((button) => {
+        button.setButtonText("Go Back");
+        button.onClick(() => {
+          this.close();
+        });
+      });
+  }
+
+  onClose() {
+    let { contentEl } = this;
+    contentEl.empty();
  }
 }

@@ -168,7 +255,9 @@ class SaveRemoteSettingTab extends PluginSettingTab {

    containerEl.empty();

-    containerEl.createEl("h2", { text: "Settings for Save Remote" });
+    containerEl.createEl("h1", { text: "Save Remote" });
+
+    containerEl.createEl("h2", { text: "S3" });

    new Setting(containerEl)
      .setName("s3Endpoint")
@@ -178,7 +267,7 @@ class SaveRemoteSettingTab extends PluginSettingTab {
          .setPlaceholder("")
          .setValue(this.plugin.settings.s3.s3Endpoint)
          .onChange(async (value) => {
-            this.plugin.settings.s3.s3Endpoint = value;
+            this.plugin.settings.s3.s3Endpoint = value.trim();
            await this.plugin.saveSettings();
          })
      );
@@ -191,7 +280,7 @@ class SaveRemoteSettingTab extends PluginSettingTab {
          .setPlaceholder("")
          .setValue(`${this.plugin.settings.s3.s3Region}`)
          .onChange(async (value) => {
-            this.plugin.settings.s3.s3Region = value;
+            this.plugin.settings.s3.s3Region = value.trim();
            await this.plugin.saveSettings();
          })
      );
@@ -204,7 +293,7 @@ class SaveRemoteSettingTab extends PluginSettingTab {
          .setPlaceholder("")
          .setValue(`${this.plugin.settings.s3.s3AccessKeyID}`)
          .onChange(async (value) => {
-            this.plugin.settings.s3.s3AccessKeyID = value;
+            this.plugin.settings.s3.s3AccessKeyID = value.trim();
            await this.plugin.saveSettings();
          })
      );
@@ -217,19 +306,7 @@ class SaveRemoteSettingTab extends PluginSettingTab {
          .setPlaceholder("")
          .setValue(`${this.plugin.settings.s3.s3SecretAccessKey}`)
          .onChange(async (value) => {
-            this.plugin.settings.s3.s3SecretAccessKey = value;
-            await this.plugin.saveSettings();
-          })
-      );
-    new Setting(containerEl)
-      .setName("password")
-      .setDesc("password")
-      .addText((text) =>
-        text
-          .setPlaceholder("")
-          .setValue(`${this.plugin.settings.password}`)
-          .onChange(async (value) => {
-            this.plugin.settings.password = value;
+            this.plugin.settings.s3.s3SecretAccessKey = value.trim();
            await this.plugin.saveSettings();
          })
      );
@@ -242,9 +319,114 @@ class SaveRemoteSettingTab extends PluginSettingTab {
          .setPlaceholder("")
          .setValue(`${this.plugin.settings.s3.s3BucketName}`)
          .onChange(async (value) => {
-            this.plugin.settings.s3.s3BucketName = value;
+            this.plugin.settings.s3.s3BucketName = value.trim();
            await this.plugin.saveSettings();
          })
      );
+
+    new Setting(containerEl)
+      .setName("check connectivity")
+      .setDesc("check connectivity")
+      .addButton(async (button) => {
+        button.setButtonText("Check");
+        button.onClick(async () => {
+          new Notice("Checking...");
+          const s3Client = getS3Client(this.plugin.settings.s3);
+          const res = await checkS3Connectivity(
+            s3Client,
+            this.plugin.settings.s3
+          );
+          if (res) {
+            new Notice("Great! The bucket can be accessed.");
+          } else {
+            new Notice("The S3 bucket cannot be reached.");
+          }
+        });
+      });
+
+    containerEl.createEl("h2", { text: "General" });
+
+    let newPassword = `${this.plugin.settings.password}`;
+    new Setting(containerEl)
+      .setName("encryption password")
+      .setDesc(
+        'Password for E2E encryption. Empty for no password. You need to click "Confirm".'
+      )
+      .addText((text) =>
+        text
+          .setPlaceholder("")
+          .setValue(`${this.plugin.settings.password}`)
+          .onChange(async (value) => {
+            newPassword = value.trim();
+          })
+      )
+      .addButton(async (button) => {
+        button.setButtonText("Confirm");
+        button.onClick(async () => {
+          new PasswordModal(this.app, this.plugin, newPassword).open();
+        });
+      });
+
+    containerEl.createEl("h2", { text: "Debug" });
+
+    const syncPlanDiv = containerEl.createEl("div");
+    syncPlanDiv.createEl("p", {
+      text: "Sync plans are created every time after you trigger sync and before the actual sync. Useful to know what would actually happen in those sync.",
+    });
+
+    new Setting(syncPlanDiv)
+      .setName("export sync plans")
+      .setDesc("export sync plans")
+      .addButton(async (button) => {
+        button.setButtonText("Export");
+        button.onClick(async () => {
+          await exportSyncPlansToFiles(this.plugin.db, this.app.vault);
+          new Notice("sync plans history exported");
+        });
+      });
+
+    new Setting(syncPlanDiv)
+      .setName("delete sync plans history in db")
+      .setDesc("delete sync plans history in db")
+      .addButton(async (button) => {
+        button.setButtonText("Delete History");
+        button.onClick(async () => {
+          await clearAllSyncPlanRecords(this.plugin.db);
+          new Notice("sync plans history (in db) deleted");
+        });
+      });
+
+    const syncMappingDiv = containerEl.createEl("div");
+    syncMappingDiv.createEl("p", {
+      text: "Sync mappings history stores the actual LOCAL last modified time of the REMOTE objects. Clearing it may cause unnecessary data exchanges in next-time sync.",
+    });
+
+    new Setting(syncMappingDiv)
+      .setName("delete sync mappings history in db")
+      .setDesc("delete sync mappings history in db")
+      .addButton(async (button) => {
+        button.setButtonText("Delete Sync Mappings");
+        button.onClick(async () => {
+          await clearAllSyncMetaMapping(this.plugin.db);
+          new Notice("sync mappings history (in local db) deleted");
+        });
+      });
+
+    const dbsResetDiv = containerEl.createEl("div");
+    syncMappingDiv.createEl("p", {
+      text: "Reset local internal caches/databases (for debugging purposes). You would want to reload the plugin after resetting this. This option will not empty the {s3, password...} settings.",
+    });
+    new Setting(syncMappingDiv)
+      .setName("reset local internal cache/databases")
+      .setDesc("reset local internal cache/databases")
+      .addButton(async (button) => {
+        button.setButtonText("Reset");
+        button.onClick(async () => {
+          await destroyDBs();
+          new Notice(
+            "Local internal cache/databases deleted. Please manually reload the plugin."
+          );
+        });
+      });
  }
 }
@@ -1,11 +1,33 @@
 import { Vault } from "obsidian";
 import * as path from "path";

+import { base32 } from "rfc4648";
+import XRegExp from "xregexp";
+
 export type SUPPORTED_SERVICES_TYPE = "s3" | "webdav" | "ftp";

-export const ignoreHiddenFiles = (item: string) => {
-  const basename = path.basename(item);
-  return basename === "." || basename[0] !== ".";
+/**
+ * If any part of the file starts with '.' or '_' then it's a hidden file.
+ * @param item
+ * @param loose
+ * @returns
+ */
+export const isHiddenPath = (item: string, loose: boolean = true) => {
+  const k = path.posix.normalize(item); // TODO: only unix path now
+  const k2 = k.split("/"); // TODO: only unix path now
+  // console.log(k2)
+  for (const singlePart of k2) {
+    if (singlePart === "." || singlePart === ".." || singlePart === "") {
+      continue;
+    }
+    if (singlePart[0] === ".") {
+      return true;
+    }
+    if (loose && singlePart[0] === "_") {
+      return true;
+    }
+  }
+  return false;
 };

 /**
@@ -31,9 +53,12 @@ export const getFolderLevels = (x: string) => {
 };

 export const mkdirpInVault = async (thePath: string, vault: Vault) => {
+  // console.log(thePath);
  const foldersToBuild = getFolderLevels(thePath);
+  // console.log(foldersToBuild);
  for (const folder of foldersToBuild) {
    const r = await vault.adapter.exists(folder);
+    // console.log(r);
    if (!r) {
      console.log(`mkdir ${folder}`);
      await vault.adapter.mkdir(folder);
@@ -46,7 +71,7 @@ export const mkdirpInVault = async (thePath: string, vault: Vault) => {
 * @param b Buffer
 * @returns ArrayBuffer
 */
-export const bufferToArrayBuffer = (b: Buffer) => {
+export const bufferToArrayBuffer = (b: Buffer | Uint8Array) => {
  return b.buffer.slice(b.byteOffset, b.byteOffset + b.byteLength);
 };

@@ -63,6 +88,45 @@ export const arrayBufferToBase64 = (b: ArrayBuffer) => {
  return arrayBufferToBuffer(b).toString("base64");
 };

+export const arrayBufferToHex = (b: ArrayBuffer) => {
+  return arrayBufferToBuffer(b).toString("hex");
+};
+
 export const base64ToArrayBuffer = (b64text: string) => {
  return bufferToArrayBuffer(Buffer.from(b64text, "base64"));
 };
+
+/**
+ * https://stackoverflow.com/questions/43131242
+ * @param hex
+ * @returns
+ */
+export const hexStringToTypedArray = (hex: string) => {
+  return new Uint8Array(
+    hex.match(/[\da-f]{2}/gi).map(function (h) {
+      return parseInt(h, 16);
+    })
+  );
+};
+
+export const base64ToBase32 = (a: string) => {
+  return base32.stringify(Buffer.from(a, "base64"));
+};
+
+/**
+ * iOS Safari could decrypt string with invalid password!
+ * So we need an extra way to test the decrypted result.
+ * One simple way is testing the result are "valid", printable chars or not.
+ *
+ * https://stackoverflow.com/questions/6198986
+ * https://www.regular-expressions.info/unicode.html
+ * Manual test shows that emojis like '🍎' match '\\p{Cs}',
+ * so we need to write the regrex in a form that \p{C} minus \p{Cs}
+ * @param a
+ */
+export const isVaildText = (a: string) => {
+  // If the regex matches, the string is invalid.
+  return !XRegExp("\\p{Cc}|\\p{Cf}|\\p{Co}|\\p{Cn}|\\p{Zl}|\\p{Zp}", "A").test(
+    a
+  );
+};
@@ -3,6 +3,7 @@ import { Readable } from "stream";

 import { Vault } from "obsidian";

+import { Upload } from "@aws-sdk/lib-storage";
 import {
  S3Client,
  ListObjectsV2Command,
@@ -10,6 +11,9 @@ import {
  GetObjectCommand,
  DeleteObjectCommand,
  HeadObjectCommand,
+  HeadBucketCommand,
+  ListObjectsV2CommandInput,
+  ListObjectsV2CommandOutput,
 } from "@aws-sdk/client-s3";

 import type { _Object } from "@aws-sdk/client-s3";
@@ -95,7 +99,7 @@ export const uploadToRemote = async (
        ContentType: contentType,
      })
    );
-    return await getRemoteMeta(s3Client, s3Config, fileOrFolderPath);
+    return await getRemoteMeta(s3Client, s3Config, uploadFile);
  } else {
    // file
    // we ignore isRecursively parameter here
@@ -109,17 +113,27 @@ export const uploadToRemote = async (
    const localContent = await vault.adapter.readBinary(fileOrFolderPath);
    let remoteContent = localContent;
    if (password !== "") {
-      remoteContent = encryptArrayBuffer(localContent, password);
+      remoteContent = await encryptArrayBuffer(localContent, password);
    }
    const body = arrayBufferToBuffer(remoteContent);
-    await s3Client.send(
-      new PutObjectCommand({
+
+    const upload = new Upload({
+      client: s3Client,
+      queueSize: 20, // concurrency
+      partSize: 5242880, // minimal 5MB by default
+      leavePartsOnError: false,
+      params: {
        Bucket: s3Config.s3BucketName,
        Key: uploadFile,
        Body: body,
        ContentType: contentType,
-      })
-    );
+      },
+    });
+    upload.on("httpUploadProgress", (progress) => {
+      // console.log(progress);
+    });
+    await upload.done();
+
    return await getRemoteMeta(s3Client, s3Config, uploadFile);
  }
 };
@@ -129,17 +143,45 @@ export const listFromRemote = async (
  s3Config: S3Config,
  prefix?: string
 ) => {
+  const confCmd = {
+    Bucket: s3Config.s3BucketName,
+  } as ListObjectsV2CommandInput;
  if (prefix !== undefined) {
-    return await s3Client.send(
-      new ListObjectsV2Command({
-        Bucket: s3Config.s3BucketName,
-        Prefix: prefix,
-      })
-    );
+    confCmd.Prefix = prefix;
  }
-  return await s3Client.send(
-    new ListObjectsV2Command({ Bucket: s3Config.s3BucketName })
-  );
+
+  const contents = [] as _Object[];
+
+  let isTruncated = true;
+  let continuationToken = "";
+  do {
+    const rsp = await s3Client.send(new ListObjectsV2Command(confCmd));
+
+    if (rsp.$metadata.httpStatusCode !== 200) {
+      throw Error("some thing bad while listing remote!");
+    }
+    if (rsp.Contents === undefined) {
+      break;
+    }
+    contents.push(...rsp.Contents);
+
+    isTruncated = rsp.IsTruncated;
+    confCmd.ContinuationToken = rsp.NextContinuationToken;
+    if (
+      isTruncated &&
+      (continuationToken === undefined || continuationToken === "")
+    ) {
+      throw Error("isTruncated is true but no continuationToken provided");
+    }
+  } while (isTruncated);
+
+  // ensemble fake rsp
+  return {
+    "$.metadata": {
+      httpStatusCode: 200,
+    },
+    Contents: contents,
+  };
 };

 /**
@@ -214,7 +256,7 @@ export const downloadFromRemote = async (
    );
    let localContent = remoteContent;
    if (password !== "") {
-      localContent = decryptArrayBuffer(remoteContent, password);
+      localContent = await decryptArrayBuffer(remoteContent, password);
    }
    await vault.adapter.writeBinary(fileOrFolderPath, localContent, {
      mtime: mtime,
@@ -266,3 +308,31 @@ export const deleteFromRemote = async (
    // pass
  }
 };
+
+/**
+ * Check the config of S3 by heading bucket
+ * https://stackoverflow.com/questions/50842835
+ * @param s3Client
+ * @param s3Config
+ * @returns
+ */
+export const checkS3Connectivity = async (
+  s3Client: S3Client,
+  s3Config: S3Config
+) => {
+  try {
+    const results = await s3Client.send(
+      new HeadBucketCommand({ Bucket: s3Config.s3BucketName })
+    );
+    if (
+      results === undefined ||
+      results.$metadata === undefined ||
+      results.$metadata.httpStatusCode === undefined
+    ) {
+      return false;
+    }
+    return results.$metadata.httpStatusCode === 200;
+  } catch (err) {
+    return false;
+  }
+};
@@ -1,14 +1,14 @@
 import { TAbstractFile, TFolder, TFile, Vault } from "obsidian";

 import { S3Client } from "@aws-sdk/client-s3";
-import * as lf from "lovefield-ts/dist/es6/lf.js";

 import {
  clearDeleteRenameHistoryOfKey,
-  FileFolderHistoryRecord,
  upsertSyncMetaMappingDataS3,
  getSyncMetaMappingByRemoteKeyS3,
 } from "./localdb";
+import type { FileFolderHistoryRecord, InternalDBs } from "./localdb";
+
 import {
  S3Config,
  S3ObjectType,
@@ -16,8 +16,27 @@ import {
  deleteFromRemote,
  downloadFromRemote,
 } from "./s3";
-import { mkdirpInVault } from "./misc";
-import { decryptBase32ToString, encryptStringToBase32 } from "./encrypt";
+import {
+  mkdirpInVault,
+  SUPPORTED_SERVICES_TYPE,
+  isHiddenPath,
+  isVaildText,
+} from "./misc";
+import {
+  decryptBase32ToString,
+  encryptStringToBase32,
+  MAGIC_ENCRYPTED_PREFIX_BASE32,
+} from "./encrypt";
+
+export type SyncStatusType =
+  | "idle"
+  | "preparing"
+  | "getting_remote_meta"
+  | "getting_local_meta"
+  | "checking_password"
+  | "generating_plan"
+  | "syncing"
+  | "finish";

 type DecisionType =
  | "undecided"
@@ -31,8 +50,6 @@ type DecisionType =
  | "mkdirplocal"
  | "skip";

-export type SyncStatusType = "idle" | "preparing" | "syncing";
-
 interface FileOrFolderMixedState {
  key: string;
  exist_local?: boolean;
@@ -48,11 +65,88 @@ interface FileOrFolderMixedState {
  remote_encrypted_key?: string;
 }

-export const ensembleMixedStates = async (
+export interface SyncPlanType {
+  ts: number;
+  remoteType: SUPPORTED_SERVICES_TYPE;
+  mixedStates: Record<string, FileOrFolderMixedState>;
+}
+
+export interface PasswordCheckType {
+  ok: boolean;
+  reason:
+    | "ok"
+    | "empty_remote"
+    | "remote_encrypted_local_no_password"
+    | "password_matched"
+    | "password_not_matched"
+    | "invalid_text_after_decryption"
+    | "remote_not_encrypted_local_has_password"
+    | "no_password_both_sides";
+}
+
+export const isPasswordOk = async (
+  remote: S3ObjectType[],
+  password: string = ""
+) => {
+  if (remote === undefined || remote.length === 0) {
+    // remote empty
+    return {
+      ok: true,
+      reason: "empty_remote",
+    } as PasswordCheckType;
+  }
+  const santyCheckKey = remote[0].Key;
+  if (santyCheckKey.startsWith(MAGIC_ENCRYPTED_PREFIX_BASE32)) {
+    // this is encrypted!
+    // try to decrypt it using the provided password.
+    if (password === "") {
+      return {
+        ok: false,
+        reason: "remote_encrypted_local_no_password",
+      } as PasswordCheckType;
+    }
+    try {
+      const res = await decryptBase32ToString(santyCheckKey, password);
+
+      // additional test
+      // because iOS Safari bypasses decryption with wrong password!
+      if (isVaildText(res)) {
+        return {
+          ok: true,
+          reason: "password_matched",
+        } as PasswordCheckType;
+      } else {
+        return {
+          ok: false,
+          reason: "invalid_text_after_decryption",
+        } as PasswordCheckType;
+      }
+    } catch (error) {
+      return {
+        ok: false,
+        reason: "password_not_matched",
+      } as PasswordCheckType;
+    }
+  } else {
+    // it is not encrypted!
+    if (password !== "") {
+      return {
+        ok: false,
+        reason: "remote_not_encrypted_local_has_password",
+      } as PasswordCheckType;
+    }
+    return {
+      ok: true,
+      reason: "no_password_both_sides",
+    } as PasswordCheckType;
+  }
+};
+
+const ensembleMixedStates = async (
  remote: S3ObjectType[],
  local: TAbstractFile[],
  deleteHistory: FileFolderHistoryRecord[],
-  db: lf.DatabaseConnection,
+  db: InternalDBs,
  password: string = ""
 ) => {
  const results = {} as Record<string, FileOrFolderMixedState>;
@@ -62,7 +156,7 @@ export const ensembleMixedStates = async (
      const remoteEncryptedKey = entry.Key;
      let key = remoteEncryptedKey;
      if (password !== "") {
-        key = decryptBase32ToString(remoteEncryptedKey, password);
+        key = await decryptBase32ToString(remoteEncryptedKey, password);
      }
      const backwardMapping = await getSyncMetaMappingByRemoteKeyS3(
        db,
@@ -73,12 +167,12 @@ export const ensembleMixedStates = async (

      let r = {} as FileOrFolderMixedState;
      if (backwardMapping !== undefined) {
-        key = backwardMapping.local_key;
+        key = backwardMapping.localKey;
        r = {
          key: key,
          exist_remote: true,
-          mtime_remote: backwardMapping.local_mtime,
-          size_remote: backwardMapping.local_size,
+          mtime_remote: backwardMapping.localMtime,
+          size_remote: backwardMapping.localSize,
          remote_encrypted_key: remoteEncryptedKey,
        };
      } else {
@@ -90,6 +184,9 @@ export const ensembleMixedStates = async (
          remote_encrypted_key: remoteEncryptedKey,
        };
      }
+      if (isHiddenPath(key)) {
+        continue;
+      }
      if (results.hasOwnProperty(key)) {
        results[key].key = r.key;
        results[key].exist_remote = r.exist_remote;
@@ -128,6 +225,9 @@ export const ensembleMixedStates = async (
      throw Error(`unexpected ${entry}`);
    }

+    if (isHiddenPath(key)) {
+      continue;
+    }
    if (results.hasOwnProperty(key)) {
      results[key].key = r.key;
      results[key].exist_local = r.exist_local;
@@ -140,11 +240,11 @@ export const ensembleMixedStates = async (

  for (const entry of deleteHistory) {
    let key = entry.key;
-    if (entry.key_type === "folder") {
+    if (entry.keyType === "folder") {
      if (!entry.key.endsWith("/")) {
        key = `${entry.key}/`;
      }
-    } else if (entry.key_type === "file") {
+    } else if (entry.keyType === "file") {
      // pass
    } else {
      throw Error(`unexpected ${entry}`);
@@ -152,9 +252,12 @@ export const ensembleMixedStates = async (

    const r = {
      key: key,
-      delete_time_local: entry.action_when,
+      delete_time_local: entry.actionWhen,
    } as FileOrFolderMixedState;

+    if (isHiddenPath(key)) {
+      continue;
+    }
    if (results.hasOwnProperty(key)) {
      results[key].key = r.key;
      results[key].delete_time_local = r.delete_time_local;
@@ -166,9 +269,10 @@ export const ensembleMixedStates = async (
  return results;
 };

-export const getOperation = (
+const getOperation = (
  origRecord: FileOrFolderMixedState,
-  inplace: boolean = false
+  inplace: boolean = false,
+  password: string = ""
 ) => {
  let r = origRecord;
  if (!inplace) {
@@ -216,6 +320,7 @@ export const getOperation = (
    r.mtime_remote !== undefined &&
    r.mtime_local !== undefined &&
    r.mtime_remote === r.mtime_local &&
+    password === "" &&
    r.size_local === r.size_remote
  ) {
    r.decision = "skip";
@@ -226,17 +331,31 @@ export const getOperation = (
    r.mtime_remote !== undefined &&
    r.mtime_local !== undefined &&
    r.mtime_remote === r.mtime_local &&
+    password === "" &&
    r.size_local !== r.size_remote
  ) {
    r.decision = "upload_clearhist";
    r.decision_branch = 4;
+  } else if (
+    r.exist_remote &&
+    r.exist_local &&
+    r.mtime_remote !== undefined &&
+    r.mtime_local !== undefined &&
+    r.mtime_remote === r.mtime_local &&
+    password !== ""
+  ) {
+    // if we have encryption,
+    // the size is always unequal
+    // only mtime(s) are reliable
+    r.decision = "skip";
+    r.decision_branch = 5;
  } else if (r.exist_remote && r.exist_local && r.mtime_local === undefined) {
    // this must be a folder!
    if (!r.key.endsWith("/")) {
      throw Error(`${r.key} is not a folder but lacks local mtime`);
    }
    r.decision = "skip";
-    r.decision_branch = 5;
+    r.decision_branch = 6;
  } else if (
    r.exist_remote &&
    !r.exist_local &&
@@ -246,7 +365,7 @@ export const getOperation = (
    r.mtime_remote >= r.delete_time_local
  ) {
    r.decision = "download_clearhist";
-    r.decision_branch = 6;
+    r.decision_branch = 7;
  } else if (
    r.exist_remote &&
    !r.exist_local &&
@@ -256,7 +375,7 @@ export const getOperation = (
    r.mtime_remote < r.delete_time_local
  ) {
    r.decision = "delremote_clearhist";
-    r.decision_branch = 7;
+    r.decision_branch = 8;
  } else if (
    r.exist_remote &&
    !r.exist_local &&
@@ -265,10 +384,10 @@ export const getOperation = (
    r.delete_time_local == undefined
  ) {
    r.decision = "download";
-    r.decision_branch = 8;
+    r.decision_branch = 9;
  } else if (!r.exist_remote && r.exist_local && r.mtime_remote === undefined) {
    r.decision = "upload_clearhist";
-    r.decision_branch = 9;
+    r.decision_branch = 10;
  } else if (
    !r.exist_remote &&
    !r.exist_local &&
@@ -276,111 +395,145 @@ export const getOperation = (
    r.mtime_local === undefined
  ) {
    r.decision = "clearhist";
-    r.decision_branch = 10;
+    r.decision_branch = 11;
  }

  return r;
 };

+export const getSyncPlan = async (
+  remote: S3ObjectType[],
+  local: TAbstractFile[],
+  deleteHistory: FileFolderHistoryRecord[],
+  db: InternalDBs,
+  password: string = ""
+) => {
+  const mixedStates = await ensembleMixedStates(
+    remote,
+    local,
+    deleteHistory,
+    db,
+    password
+  );
+  for (const [key, val] of Object.entries(mixedStates)) {
+    getOperation(val, true, password);
+  }
+  const plan = {
+    ts: Date.now(),
+    remoteType: "s3",
+    mixedStates: mixedStates,
+  } as SyncPlanType;
+  return plan;
+};
+
 export const doActualSync = async (
  s3Client: S3Client,
  s3Config: S3Config,
-  db: lf.DatabaseConnection,
+  db: InternalDBs,
  vault: Vault,
-  keyStates: Record<string, FileOrFolderMixedState>,
+  syncPlan: SyncPlanType,
  password: string = ""
 ) => {
-  Object.entries(keyStates)
-    .sort((k, v) => -(k as string).length)
-    .map(async ([k, v]) => {
-      const key = k as string;
-      const state = v as FileOrFolderMixedState;
-      let remoteEncryptedKey = key;
-      if (password !== "") {
-        remoteEncryptedKey = state.remote_encrypted_key;
-        if (remoteEncryptedKey === undefined || remoteEncryptedKey === "") {
-          remoteEncryptedKey = encryptStringToBase32(key, password);
+  const keyStates = syncPlan.mixedStates;
+  await Promise.all(
+    Object.entries(keyStates)
+      .sort((k, v) => -(k as string).length)
+      .map(async ([k, v]) => {
+        const key = k as string;
+        const state = v as FileOrFolderMixedState;
+        let remoteEncryptedKey = key;
+        if (password !== "") {
+          remoteEncryptedKey = state.remote_encrypted_key;
+          if (remoteEncryptedKey === undefined || remoteEncryptedKey === "") {
+            remoteEncryptedKey = await encryptStringToBase32(key, password);
+          }
        }
-      }

-      if (
-        state.decision === undefined ||
-        state.decision === "unknown" ||
-        state.decision === "undecided"
-      ) {
-        throw Error(`unknown decision in ${JSON.stringify(state)}`);
-      } else if (state.decision === "skip") {
-        // do nothing
-      } else if (state.decision === "download_clearhist") {
-        await downloadFromRemote(
-          s3Client,
-          s3Config,
-          state.key,
-          vault,
-          state.mtime_remote,
-          password,
-          remoteEncryptedKey
-        );
-        await clearDeleteRenameHistoryOfKey(db, state.key);
-      } else if (state.decision === "upload_clearhist") {
-        const remoteObjMeta = await uploadToRemote(
-          s3Client,
-          s3Config,
-          state.key,
-          vault,
-          false,
-          password,
-          remoteEncryptedKey
-        );
-        await upsertSyncMetaMappingDataS3(
-          db,
-          state.key,
-          state.mtime_local,
-          state.size_local,
-          state.key,
-          remoteObjMeta.LastModified.valueOf(),
-          remoteObjMeta.ContentLength,
-          remoteObjMeta.ETag
-        );
-        await clearDeleteRenameHistoryOfKey(db, state.key);
-      } else if (state.decision === "download") {
-        await mkdirpInVault(state.key, vault);
-        await downloadFromRemote(
-          s3Client,
-          s3Config,
-          state.key,
-          vault,
-          state.mtime_remote,
-          password,
-          remoteEncryptedKey
-        );
-      } else if (state.decision === "delremote_clearhist") {
-        await deleteFromRemote(s3Client, s3Config, state.key);
-        await clearDeleteRenameHistoryOfKey(db, state.key);
-      } else if (state.decision === "upload") {
-        const remoteObjMeta = await uploadToRemote(
-          s3Client,
-          s3Config,
-          state.key,
-          vault,
-          false,
-          password,
-          remoteEncryptedKey
-        );
-        await upsertSyncMetaMappingDataS3(
-          db,
-          state.key,
-          state.mtime_local,
-          state.size_local,
-          state.key,
-          remoteObjMeta.LastModified.valueOf(),
-          remoteObjMeta.ContentLength,
-          remoteObjMeta.ETag
-        );
-      } else if (state.decision === "clearhist") {
-        await clearDeleteRenameHistoryOfKey(db, state.key);
-      } else {
-        throw Error("this should never happen!");
-      }
-    });
+        if (
+          state.decision === undefined ||
+          state.decision === "unknown" ||
+          state.decision === "undecided"
+        ) {
+          throw Error(`unknown decision in ${JSON.stringify(state)}`);
+        } else if (state.decision === "skip") {
+          // do nothing
+        } else if (state.decision === "download_clearhist") {
+          await downloadFromRemote(
+            s3Client,
+            s3Config,
+            state.key,
+            vault,
+            state.mtime_remote,
+            password,
+            remoteEncryptedKey
+          );
+          await clearDeleteRenameHistoryOfKey(db, state.key);
+        } else if (state.decision === "upload_clearhist") {
+          const remoteObjMeta = await uploadToRemote(
+            s3Client,
+            s3Config,
+            state.key,
+            vault,
+            false,
+            password,
+            remoteEncryptedKey
+          );
+          await upsertSyncMetaMappingDataS3(
+            db,
+            state.key,
+            state.mtime_local,
+            state.size_local,
+            state.key,
+            remoteObjMeta.LastModified.valueOf(),
+            remoteObjMeta.ContentLength,
+            remoteObjMeta.ETag
+          );
+          await clearDeleteRenameHistoryOfKey(db, state.key);
+        } else if (state.decision === "download") {
+          await mkdirpInVault(state.key, vault);
+          await downloadFromRemote(
+            s3Client,
+            s3Config,
+            state.key,
+            vault,
+            state.mtime_remote,
+            password,
+            remoteEncryptedKey
+          );
+        } else if (state.decision === "delremote_clearhist") {
+          await deleteFromRemote(
+            s3Client,
+            s3Config,
+            state.key,
+            password,
+            remoteEncryptedKey
+          );
+          await clearDeleteRenameHistoryOfKey(db, state.key);
+        } else if (state.decision === "upload") {
+          const remoteObjMeta = await uploadToRemote(
+            s3Client,
+            s3Config,
+            state.key,
+            vault,
+            false,
+            password,
+            remoteEncryptedKey
+          );
+          await upsertSyncMetaMappingDataS3(
+            db,
+            state.key,
+            state.mtime_local,
+            state.size_local,
+            state.key,
+            remoteObjMeta.LastModified.valueOf(),
+            remoteObjMeta.ContentLength,
+            remoteObjMeta.ETag
+          );
+        } else if (state.decision === "clearhist") {
+          await clearDeleteRenameHistoryOfKey(db, state.key);
+        } else {
+          throw Error("this should never happen!");
+        }
+      })
+  );
 };
@@ -1 +1,5 @@
 /* set the styles */
+
+.password_second_confirm {
+  font-weight: bold;
+}
@@ -0,0 +1,110 @@
+import * as fs from "fs";
+import * as path from "path";
+import * as chai from "chai";
+import chaiAsPromised from "chai-as-promised";
+import { base64ToBase32, bufferToArrayBuffer } from "../src/misc";
+import {
+  decryptArrayBuffer,
+  decryptBase32ToString,
+  encryptArrayBuffer,
+  encryptStringToBase32,
+} from "../src/encrypt";
+
+chai.use(chaiAsPromised);
+const expect = chai.expect;
+
+describe("Encryption tests", () => {
+  beforeEach(function () {
+    global.window = {
+      crypto: require("crypto").webcrypto,
+    } as any;
+  });
+
+  it("should encrypt string", async () => {
+    const k = "dkjdhkfhdkjgsdklxxd";
+    const password = "hey";
+    expect(await encryptStringToBase32(k, password)).to.not.equal(k);
+  });
+
+  it("should raise error using different password", async () => {
+    const k = "secret text";
+    const password = "hey";
+    const password2 = "hey2";
+    const enc = await encryptStringToBase32(k, password);
+    await expect(decryptBase32ToString(enc, password2)).to.be.rejected;
+  });
+
+  it("should encrypt and decrypt string and get the same result returned", async () => {
+    const k = "jfkkjkjbce7983ycdeknkkjckooAIUHIDIBIE((*BII)njD/d/dd/d/sjxhux";
+    const password = "hfiuibec989###oiu982bj1`";
+    const enc = await encryptStringToBase32(k, password);
+    // console.log(enc);
+    const dec = await decryptBase32ToString(enc, password);
+    // console.log(dec);
+    expect(dec).equal(k);
+  });
+
+  it("should encrypt text file and get the same result as openssl", async () => {
+    const fileContent = (
+      await fs.readFileSync(
+        path.join(__dirname, "static_assets", "sometext.txt")
+      )
+    ).toString("utf-8");
+    const password = "somepassword";
+    const saltHex = "8302F586FAB491EC";
+    const enc = await encryptStringToBase32(
+      fileContent,
+      password,
+      undefined,
+      saltHex
+    );
+
+    // two command returns same result:
+    // cat ./sometext.txt | openssl enc -p -aes-256-cbc -S 8302F586FAB491EC -pbkdf2 -iter 10000 -base64 -pass pass:somepassword
+    // openssl enc -p -aes-256-cbc -S 8302F586FAB491EC -pbkdf2 -iter 10000 -base64 -pass pass:somepassword -in ./sometext.txt
+    const opensslBase64Res =
+      "U2FsdGVkX1+DAvWG+rSR7MSa+yJav1zCE7SSXiBooqwI5Q+LMpIthpk/pXkLj+25";
+    // we output base32, so we need some transformation
+    const opensslBase32Res = base64ToBase32(opensslBase64Res);
+
+    expect(enc).equal(opensslBase32Res);
+  });
+
+  it("should encrypt binary file and get the same result as openssl", async () => {
+    const testFolder = path.join(__dirname, "static_assets", "mona_lisa");
+    const testFileName =
+      "1374px-Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched.jpg";
+    const fileArrBuf = bufferToArrayBuffer(
+      await fs.readFileSync(path.join(testFolder, testFileName))
+    );
+    const password = "somepassword";
+    const saltHex = "8302F586FAB491EC";
+    const enc = await encryptArrayBuffer(
+      fileArrBuf,
+      password,
+      undefined,
+      saltHex
+    );
+    const opensslArrBuf = bufferToArrayBuffer(
+      await fs.readFileSync(path.join(testFolder, testFileName + ".enc"))
+    );
+
+    expect(Buffer.from(enc).equals(Buffer.from(opensslArrBuf))).to.be.true;
+  });
+
+  it("should descypt binary file and get the same result as openssl", async () => {
+    const testFolder = path.join(__dirname, "static_assets", "mona_lisa");
+    const testFileName =
+      "1374px-Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched.jpg";
+    const fileArrBuf = bufferToArrayBuffer(
+      await fs.readFileSync(path.join(testFolder, testFileName + ".enc"))
+    );
+    const password = "somepassword";
+    const dec = await decryptArrayBuffer(fileArrBuf, password);
+    const opensslArrBuf = bufferToArrayBuffer(
+      await fs.readFileSync(path.join(testFolder, testFileName))
+    );
+
+    expect(Buffer.from(dec).equals(Buffer.from(opensslArrBuf))).to.be.true;
+  });
+});
@@ -0,0 +1,91 @@
+import * as fs from "fs";
+import * as path from "path";
+import { expect } from "chai";
+
+import * as misc from "../src/misc";
+
+describe("Misc: hidden file", () => {
+  it("should find hidden file correctly", () => {
+    let item = "";
+    expect(misc.isHiddenPath(item)).to.be.false;
+
+    item = ".";
+    expect(misc.isHiddenPath(item)).to.be.false;
+
+    item = "..";
+    expect(misc.isHiddenPath(item)).to.be.false;
+
+    item = "/x/y/z/../././../a/b/c";
+    expect(misc.isHiddenPath(item)).to.be.false;
+
+    item = ".hidden";
+    expect(misc.isHiddenPath(item)).to.be.true;
+
+    item = "_hidden_loose";
+    expect(misc.isHiddenPath(item)).to.be.true;
+    expect(misc.isHiddenPath(item, false)).to.be.false;
+
+    item = "/sdd/_hidden_loose";
+    expect(misc.isHiddenPath(item)).to.be.true;
+
+    item = "what/../_hidden_loose/what/what/what";
+    expect(misc.isHiddenPath(item)).to.be.true;
+
+    item = "what/../_hidden_loose/what/what/what";
+    expect(misc.isHiddenPath(item, false)).to.be.false;
+
+    item = "what/../_hidden_loose/../.hidden/what/what/what";
+    expect(misc.isHiddenPath(item, false)).to.be.true;
+  });
+});
+
+describe("Misc: get folder levels", () => {
+  it("should ignore empty path", () => {
+    const item = "";
+    expect(misc.getFolderLevels(item)).to.be.empty;
+  });
+
+  it("should ignore single file", () => {
+    const item = "xxx";
+    expect(misc.getFolderLevels(item)).to.be.empty;
+  });
+
+  it("should detect path ending with /", () => {
+    const item = "xxx/";
+    const res = ["xxx"];
+    expect(misc.getFolderLevels(item)).to.deep.equal(res);
+  });
+
+  it("should correctly split folders and files", () => {
+    const item = "xxx/yyy/zzz.md";
+    const res = ["xxx", "xxx/yyy"];
+    expect(misc.getFolderLevels(item)).to.deep.equal(res);
+
+    const item2 = "xxx/yyy/zzz";
+    const res2 = ["xxx", "xxx/yyy"];
+    expect(misc.getFolderLevels(item2)).to.deep.equal(res2);
+
+    const item3 = "xxx/yyy/zzz/";
+    const res3 = ["xxx", "xxx/yyy", "xxx/yyy/zzz"];
+    expect(misc.getFolderLevels(item3)).to.deep.equal(res3);
+  });
+});
+
+describe("Misc: vaild file name tests", () => {
+  it("should treat no ascii correctly", async () => {
+    const x = misc.isVaildText("😄🍎 apple 苹果");
+    // console.log(x)
+    expect(x).to.be.true;
+  });
+
+  it("should find not-printable chars correctly", async () => {
+    const x = misc.isVaildText("😄🍎 apple 苹果\u0000");
+    // console.log(x)
+    expect(x).to.be.false;
+  });
+
+  it("should allow spaces/slashes/...", async () => {
+    const x = misc.isVaildText("😄🍎 apple 苹果/-_=/\\*%^&@#$`");
+    expect(x).to.be.true;
+  });
+});
@@ -0,0 +1,3 @@
+The file 1374px-Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched.jpg is downloaded from Wikimedia Commons: https://commons.wikimedia.org/wiki/File:Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched.jpg .
+
+Quoted from the web address: ... This photographic reproduction is therefore also considered to be in the public domain in the United States. In other jurisdictions, re-use of this content may be restricted ...
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b332c298be98be9968715952822431a051f9d07e67460cdcf91a5a3a305b5df1
+size 1141550
@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cf67abd791f1e230bce57dd47150f262424247cbefb8be16efcd57d13955576a
+size 1141568
@@ -0,0 +1,3 @@
+# The encryption file is produced by the following command.
+# A salt is explictly provided because we need reproducible output in tests.
+openssl enc -p -aes-256-cbc -S 8302F586FAB491EC -pbkdf2 -iter 10000 -pass pass:somepassword -in '1374px-Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched.jpg' -out 1374px-Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched.jpg.enc
@@ -0,0 +1 @@
+A secret text 你好世界
@@ -1,3 +1,3 @@
 {
-  "0.0.4": "0.12.15"
+  "0.0.15": "0.12.15"
 }
@@ -33,29 +33,29 @@ module.exports = {
    extensions: [".tsx", ".ts", ".js"],
    mainFields: ["browser", "module", "main"],
    fallback: {
-      assert: require.resolve("assert"),
-      buffer: require.resolve("buffer/"),
-      console: require.resolve("console-browserify"),
-      constants: require.resolve("constants-browserify"),
-      crypto: require.resolve("crypto-browserify"),
-      domain: require.resolve("domain-browser"),
-      events: require.resolve("events"),
-      http: require.resolve("stream-http"),
-      https: require.resolve("https-browserify"),
-      os: require.resolve("os-browserify/browser"),
+      // assert: require.resolve("assert"),
+      // buffer: require.resolve("buffer/"),
+      // console: require.resolve("console-browserify"),
+      // constants: require.resolve("constants-browserify"),
+      // crypto: require.resolve("crypto-browserify"),
+      // domain: require.resolve("domain-browser"),
+      // events: require.resolve("events"),
+      // http: require.resolve("stream-http"),
+      // https: require.resolve("https-browserify"),
+      // os: require.resolve("os-browserify/browser"),
      path: require.resolve("path-browserify"),
-      punycode: require.resolve("punycode"),
+      // punycode: require.resolve("punycode"),
      process: require.resolve("process/browser"),
-      querystring: require.resolve("querystring-es3"),
+      // querystring: require.resolve("querystring-es3"),
      stream: require.resolve("stream-browserify"),
-      string_decoder: require.resolve("string_decoder"),
-      sys: require.resolve("util"),
-      timers: require.resolve("timers-browserify"),
-      tty: require.resolve("tty-browserify"),
-      url: require.resolve("url"),
-      util: require.resolve("util"),
-      vm: require.resolve("vm-browserify"),
-      zlib: require.resolve("browserify-zlib"),
+      // string_decoder: require.resolve("string_decoder"),
+      // sys: require.resolve("util"),
+      // timers: require.resolve("timers-browserify"),
+      // tty: require.resolve("tty-browserify"),
+      // url: require.resolve("url"),
+      // util: require.resolve("util"),
+      // vm: require.resolve("vm-browserify"),
+      // zlib: require.resolve("browserify-zlib"),
    },
  },
  externals: {
Author	SHA1	Message	Date
fyears	59645e620a	manually delete indexeddb	2021-11-14 20:51:29 +08:00
fyears	125da9d734	more debug	2021-11-14 20:47:47 +08:00
fyears	808ae709fb	0.0.15	2021-11-14 20:25:20 +08:00
fyears	16c3c11fd8	use localforage	2021-11-14 20:24:33 +08:00
fyears	be2b168c6c	artifact has styles	2021-11-10 01:06:00 +08:00
fyears	6b296d6110	bump to 0.0.14	2021-11-10 01:04:09 +08:00
fyears	6757348ac9	use multipart-upload	2021-11-10 01:03:33 +08:00
fyears	a7e6bed071	fix typo	2021-11-10 00:30:30 +08:00
fyears	8b8c5bdb98	fix letter	2021-11-10 00:30:11 +08:00
fyears	5b77b23e9f	xregexp should be dep not dev-dep	2021-11-10 00:16:24 +08:00
fyears	05313f9530	0.0.13	2021-11-09 10:01:18 +08:00
fyears	ce4e55fcc7	more tests	2021-11-09 10:00:44 +08:00
fyears	b26d6a7ffb	more clear notice	2021-11-09 10:00:14 +08:00
fyears	ea12a14d7f	0.0.12	2021-11-08 10:00:40 +08:00
fyears	3609a78465	test enc	2021-11-08 10:00:06 +08:00
fyears	7ec2db4006	correctly fetch remote data	2021-11-08 09:38:39 +08:00
fyears	5352c7b828	check undefined	2021-11-08 09:37:51 +08:00
fyears	2a49bb53b2	more attention	2021-11-08 00:57:18 +08:00
fyears	fde4c9a294	0.0.11	2021-11-08 00:40:45 +08:00
fyears	7fc38a942a	check password	2021-11-08 00:39:17 +08:00
fyears	cc0c4fe224	second confirm for password changes	2021-11-08 00:17:42 +08:00
fyears	2ab5640c12	list all objects	2021-11-07 17:15:06 +08:00
fyears	b7a0376a0f	allow clearing sync mappings	2021-11-07 16:02:03 +08:00
fyears	d7d1c47e2a	long desc for debug	2021-11-07 15:44:14 +08:00
fyears	a65f5eb004	head bucket	2021-11-07 15:43:57 +08:00
fyears	2ce34379ad	format doc	2021-11-07 15:27:41 +08:00
fyears	40d2ee95b7	0.0.10	2021-11-07 15:01:12 +08:00
fyears	5632e28331	make decision works	2021-11-07 15:00:22 +08:00
fyears	2b53a1bf5e	to 0.0.9	2021-11-07 14:54:25 +08:00
fyears	ccd9e3116c	add cross-env	2021-11-07 14:51:00 +08:00
fyears	e8d6a50793	update algo	2021-11-07 14:24:24 +08:00
fyears	c43b36e0e7	0.0.8	2021-11-07 14:09:22 +08:00
fyears	bcd92d135f	remove unused log	2021-11-07 14:05:03 +08:00
fyears	62aea9d330	debug mode: export sync plans	2021-11-07 13:58:51 +08:00
fyears	b9c17076b4	reorg misc tests	2021-11-07 13:13:40 +08:00
fyears	385290c1cc	find hidden files	2021-11-07 12:14:14 +08:00
fyears	371089e3ee	rename tests	2021-11-07 12:02:36 +08:00
fyears	254790cd61	abstract getSyncPlan	2021-11-07 00:39:57 +08:00
fyears	d3b812c8fa	split settings items	2021-11-07 00:24:02 +08:00
fyears	3a14f86880	fix error msg	2021-11-07 00:07:10 +08:00
fyears	887f948dc3	trim settings	2021-11-07 00:01:15 +08:00
fyears	035beb7d97	not destroy dbs while unloading	2021-11-06 23:48:15 +08:00
fyears	f334198241	more log	2021-11-06 23:41:55 +08:00
fyears	bcca2b4446	more accurate stages of sync	2021-11-06 23:31:49 +08:00
fyears	659deb9830	prettier way	2021-11-06 23:02:54 +08:00
fyears	b6df3f7ed5	promise all sync	2021-11-06 22:53:52 +08:00
fyears	2012758383	update readme	2021-11-06 13:15:54 +08:00
fyears	bea0e694f6	try lfs on ci	2021-11-06 13:06:17 +08:00
fyears	8fbbae6c2e	test for enc	2021-11-06 12:57:40 +08:00
fyears	ec2fec9bf8	no need console.log	2021-11-06 00:26:44 +08:00
fyears	09d33523b6	bump to 0.0.7	2021-11-06 00:25:05 +08:00
fyears	170cbbfb4f	add test	2021-11-06 00:23:59 +08:00
fyears	9d5fb7e966	test only on main branch	2021-11-05 01:31:18 +08:00
fyears	dc11d21d72	test on ci	2021-11-05 01:30:36 +08:00
fyears	774e31ef27	fix test	2021-11-05 01:29:44 +08:00
fyears	7c4a981434	allow tests	2021-11-05 01:21:58 +08:00
fyears	358311e8ac	spread Uint8Arrays	2021-11-05 01:21:33 +08:00
fyears	2551462003	clean unnecessary packages	2021-11-04 23:58:20 +08:00
fyears	60d5b78df0	fix length	2021-11-04 10:16:07 +08:00
fyears	f04017e05b	init web crypto	2021-11-04 10:10:05 +08:00
fyears	cb0098ec85	fix delete	2021-11-03 12:04:11 +08:00
fyears	d007d5f230	bump 0.0.6	2021-11-03 01:05:16 +08:00
fyears	07c6fbe747	change password position	2021-11-03 01:05:07 +08:00
fyears	5e3d413b33	bump ver	2021-11-03 00:58:48 +08:00
fyears	bcf8586fbf	use hash-wasm	2021-11-03 00:57:57 +08:00
fyears	6431182ec3	promisify data	2021-11-03 00:15:23 +08:00