Compare commits

...
35 Commits
Author SHA1 Message Date
fyears 2a3e0524c5 fix terrible typo 2021-12-04 01:21:37 +08:00
fyears 2f20ca61ce rebranding 2021-12-04 01:21:15 +08:00
fyears 68ec8b5454 bump version and update name 2021-12-04 01:18:20 +08:00
fyears 2bbbd569c5 fix typo 2021-12-04 01:11:17 +08:00
fyears 62936c4473 attempt to reduce size and use native crypto 2021-12-04 00:55:31 +08:00
fyears 789d07261e 0.1.10 2021-12-01 10:11:20 +08:00
fyears ccd3b49b4b inject env var in webpack 2021-12-01 10:10:39 +08:00
fyears 2930fe5350 0.1.9 2021-12-01 01:50:00 +08:00
fyears a9abf6d834 webpack is slow but works 2021-12-01 01:49:29 +08:00
fyears bdf09e24c3 remove assert 2021-12-01 01:49:07 +08:00
fyears 8dc4b835c9 0.1.8 2021-12-01 00:34:02 +08:00
fyears 2e800af4f8 dropbox should use sub folder 2021-12-01 00:33:24 +08:00
fyears e9e9e2b6eb fix typo 2021-11-30 00:44:26 +08:00
fyears 274e0b8b28 bump to 0.1.7 2021-11-30 00:42:05 +08:00
fyears 2147332cf4 more about dropbox 2021-11-30 00:41:41 +08:00
fyears 262bb9a1fe more disclaimer 2021-11-30 00:32:43 +08:00
fyears 2669a3e8ad fix auth dropbox 2021-11-30 00:21:15 +08:00
fyears bdac16ee24 define clientID or AppKey by env var 2021-11-29 01:03:17 +08:00
fyears 0a7de6588a ugly way to refresh token 2021-11-29 00:04:24 +08:00
fyears 1effcdd7f7 basic dropbox auth flow 2021-11-28 16:28:52 +08:00
fyears 00c840a758 basically working dropbox 2021-11-28 12:20:38 +08:00
fyears 2eb1545f9d more misc 2021-11-28 12:18:44 +08:00
fyears 0d75ea92b1 polish get parent folder 2021-11-28 10:45:46 +08:00
fyears d6cbf4260a esbuild is way faster 2021-11-27 14:49:07 +08:00
fyears 9389039c7a avoid using for await 2021-11-27 14:43:19 +08:00
fyears aa3aecb074 0.1.6 2021-11-27 00:51:01 +08:00
fyears 9128f9ed5b add support for ios 5-times click 2021-11-27 00:49:21 +08:00
fyears b581ef3f31 this is 0.1.5 correctly 2021-11-22 00:12:21 +08:00
fyears d8c42adc55 update readme 2021-11-22 00:06:50 +08:00
fyears 45b18ca63d 0.1.5 2021-11-21 23:55:43 +08:00
fyears 52e9d3298a hide webdav yet 2021-11-21 23:54:50 +08:00
fyears e82dfc1e88 fix unknown decision 2021-11-21 22:54:31 +08:00
fyears 0be19b677b selective menu for s3 and webdav 2021-11-21 16:46:01 +08:00
fyears d1839706af basically workable webdav 2021-11-21 15:33:14 +08:00
fyears ce0cc232c8 init methods for webdav 2021-11-20 23:51:20 +08:00
18 changed files with 1938 additions and 199 deletions
+19 -3
View File
@@ -4,7 +4,7 @@ This is yet another unofficial sync plugin for Obsidian.
## Disclaimer ## Disclaimer
- **This is NOT the [official sync service](https://github.com/fyears/obsidian-remotely-save.git) provided by Obsidian.** - **This is NOT the [official sync service](https://obsidian.md/sync) provided by Obsidian.**
## !!!Caution!!! ## !!!Caution!!!
@@ -12,7 +12,7 @@ As of November 2021, the plugin is considered in BETA stage. **DO NOT USE IT for
## Features ## Features
- **Amazon S3 or S3-compatible services are supported.** Webdav supports on the plan. - **Amazon S3 or S3-compatible, and Dropbox services are supported.** Webdav supports on the half way.
- **Obsidiain Mobile supported.** Vaults can be synced across mobile and desktop devices with the cloud service as the "broker". - **Obsidiain Mobile supported.** Vaults can be synced across mobile and desktop devices with the cloud service as the "broker".
- **[End-to-end encryption](./docs/encryption.md) supported.** Files would be encrypted using openssl format before being sent to the cloud **if** user specify a password. - **[End-to-end encryption](./docs/encryption.md) supported.** Files would be encrypted using openssl format before being sent to the cloud **if** user specify a password.
- **[Minimal Intrusive](./docs/minimal_intrusive_design.md).** - **[Minimal Intrusive](./docs/minimal_intrusive_design.md).**
@@ -29,12 +29,14 @@ As of November 2021, the plugin is considered in BETA stage. **DO NOT USE IT for
## Download and Install ## Download and Install
- Option #1: [![BuildCI](https://github.com/fyears/obsidian-remotely-save/actions/workflows/auto-build.yml/badge.svg)](https://github.com/fyears/obsidian-remotely-save/actions/workflows/auto-build.yml) Every artifacts are placed in the "Summary" under every successful builds. - Option #1: [![BuildCI](https://github.com/fyears/remotely-save/actions/workflows/auto-build.yml/badge.svg)](https://github.com/fyears/remotely-save/actions/workflows/auto-build.yml) Every artifacts are placed in the "Summary" under every successful builds.
- Option #2: Besides manually downloading the files, you can also use [Obsidian42 - BRAT](https://github.com/TfTHacker/obsidian42-brat) to install this plugin. - Option #2: Besides manually downloading the files, you can also use [Obsidian42 - BRAT](https://github.com/TfTHacker/obsidian42-brat) to install this plugin.
- Option #3: The pluin would be submitted to the official "community plugin list" in near future. - Option #3: The pluin would be submitted to the official "community plugin list" in near future.
## Usage ## Usage
### s3
- Prepare your S3 (-compatible) service information: [endpoint, region](https://docs.aws.amazon.com/general/latest/gr/s3.html), [access key id, secret access key](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/getting-your-credentials.html), bucket name. The bucket should be empty and solely for syncing a vault. - Prepare your S3 (-compatible) service information: [endpoint, region](https://docs.aws.amazon.com/general/latest/gr/s3.html), [access key id, secret access key](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/getting-your-credentials.html), bucket name. The bucket should be empty and solely for syncing a vault.
- Configure (enable) [CORS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html) for requests from `app://obsidian.md` and `capacitor://localhost` and `http://localhost`. It's unfortunately required, because the plugin sends requests from a browser-like envirement. And those addresses are tested and found on desktop and ios and android. - Configure (enable) [CORS](https://docs.aws.amazon.com/AmazonS3/latest/userguide/enabling-cors-examples.html) for requests from `app://obsidian.md` and `capacitor://localhost` and `http://localhost`. It's unfortunately required, because the plugin sends requests from a browser-like envirement. And those addresses are tested and found on desktop and ios and android.
- Download and enable this plugin. - Download and enable this plugin.
@@ -42,3 +44,17 @@ As of November 2021, the plugin is considered in BETA stage. **DO NOT USE IT for
- If you want to enable end-to-end encryption, also set a password in settings. If you do not specify a password, the files and folders are synced in plain, original content to the cloud. - If you want to enable end-to-end encryption, also set a password in settings. If you do not specify a password, the files and folders are synced in plain, original content to the cloud.
- Click the new "switch" icon on the ribbon (the left sidebar), **every time** you want to sync your vault between local and remote. (No "auto sync" yet.) - Click the new "switch" icon on the ribbon (the left sidebar), **every time** you want to sync your vault between local and remote. (No "auto sync" yet.)
- **Be patient while syncing.** Especially in the first-time sync. - **Be patient while syncing.** Especially in the first-time sync.
### Dropbox
- **This plugin's function for Dropbox is not as mature as functions for S3.**
- **This plugin is NOT an official Dropbox product.** The plugin just uses Dropbox's public API.
- After the authorization, the plugin can read your name and email (which cannot be unselected on Dropbox api), and read and write files in your Dropbox's `/Apps/remotely-save` folder.
- If you decide to authorize this plugin to connect to Dropbox, please go to plugin's settings, and choose Dropbox then follow the instructions.
- Password-based end-to-end encryption is also supported.
### webdav
- **webdav support is buggy (as of now, 20211122) and considered experimental, so it's hidden by default.** Highly recommend to use the more stable s3.
- If you decide to give it a try, open settings, and click "Choose service" area five times, then a Notice should show up. Close and open settings again then you will be able to select webdav.
- Currently webdav server should enable CORS for requests, because of technical limitations of mobile.
+42
View File
@@ -0,0 +1,42 @@
import dotenv from "dotenv/config";
import esbuild from "esbuild";
import process from "process";
// import builtins from 'builtin-modules'
const banner = `/*
THIS IS A GENERATED/BUNDLED FILE BY ESBUILD
if you want to view the source, please visit the github repository of this plugin
*/
`;
const prod = process.argv[2] === "production";
const DEFAULT_DROPBOX_APP_KEY = process.env.DROPBOX_APP_KEY || "";
esbuild
.build({
banner: {
js: banner,
},
entryPoints: ["./src/main.ts"],
bundle: true,
external: [
"obsidian",
"electron",
"fs",
"crypto",
// ...builtins
],
format: "cjs",
watch: !prod,
target: "es2016",
logLevel: "info",
sourcemap: prod ? false : "inline",
treeShaking: true,
minify: prod,
outfile: "main.js",
define: {
"process.env.DEFAULT_DROPBOX_APP_KEY": `"${DEFAULT_DROPBOX_APP_KEY}"`,
},
})
.catch(() => process.exit(1));
+2 -2
View File
@@ -1,7 +1,7 @@
{ {
"id": "obsidian-remotely-save", "id": "remotely-save",
"name": "Remotely Save", "name": "Remotely Save",
"version": "0.1.3", "version": "0.2.0",
"minAppVersion": "0.12.15", "minAppVersion": "0.12.15",
"description": "Yet another unofficial plugin allowing users to sync notes between local device and the cloud service.", "description": "Yet another unofficial plugin allowing users to sync notes between local device and the cloud service.",
"author": "fyears", "author": "fyears",
+17 -5
View File
@@ -1,14 +1,20 @@
{ {
"name": "obsidian-remotely-save", "name": "remotely-save",
"version": "0.1.3", "version": "0.2.0",
"description": "This is yet another sync plugin for Obsidian app.", "description": "This is yet another sync plugin for Obsidian app.",
"scripts": { "scripts": {
"dev": "webpack --mode development --watch", "dev": "node esbuild.config.mjs",
"build2": "tsc -noEmit -skipLibCheck && node esbuild.config.mjs production",
"build": "webpack --mode production", "build": "webpack --mode production",
"format": "npx prettier --write .", "format": "npx prettier --write .",
"clean": "npx rimraf main.js", "clean": "npx rimraf main.js",
"test": "cross-env TS_NODE_COMPILER_OPTIONS={\\\"module\\\":\\\"commonjs\\\"} mocha -r ts-node/register 'tests/**/*.ts'" "test": "cross-env TS_NODE_COMPILER_OPTIONS={\\\"module\\\":\\\"commonjs\\\"} mocha -r ts-node/register 'tests/**/*.ts'"
}, },
"browser": {
"path": "path-browserify",
"process": "process/browser",
"stream": "stream-browserify"
},
"source": "main.ts", "source": "main.ts",
"keywords": [], "keywords": [],
"author": "", "author": "",
@@ -19,18 +25,20 @@
"@types/mime-types": "^2.1.1", "@types/mime-types": "^2.1.1",
"@types/mocha": "^9.0.0", "@types/mocha": "^9.0.0",
"@types/node": "^14.14.37", "@types/node": "^14.14.37",
"builtin-modules": "^3.2.0",
"chai": "^4.3.4", "chai": "^4.3.4",
"chai-as-promised": "^7.1.1", "chai-as-promised": "^7.1.1",
"cross-env": "^7.0.3", "cross-env": "^7.0.3",
"dotenv": "^10.0.0",
"esbuild": "^0.14.0",
"mocha": "^9.1.3", "mocha": "^9.1.3",
"prettier": "^2.4.1", "prettier": "^2.4.1",
"terser-webpack-plugin": "^5.2.4",
"ts-loader": "^9.2.6", "ts-loader": "^9.2.6",
"ts-node": "^10.4.0", "ts-node": "^10.4.0",
"tslib": "^2.2.0", "tslib": "^2.2.0",
"typescript": "^4.4.4", "typescript": "^4.4.4",
"webdav-server": "^2.6.2", "webdav-server": "^2.6.2",
"webpack": "^5.58.2", "webpack": "^5.64.4",
"webpack-cli": "^4.9.1" "webpack-cli": "^4.9.1"
}, },
"dependencies": { "dependencies": {
@@ -38,9 +46,12 @@
"@aws-sdk/lib-storage": "^3.40.1", "@aws-sdk/lib-storage": "^3.40.1",
"@aws-sdk/signature-v4-crt": "^3.37.0", "@aws-sdk/signature-v4-crt": "^3.37.0",
"acorn": "^8.5.0", "acorn": "^8.5.0",
"assert": "^2.0.0",
"aws-crt": "^1.10.1", "aws-crt": "^1.10.1",
"buffer": "^6.0.3", "buffer": "^6.0.3",
"codemirror": "^5.63.1", "codemirror": "^5.63.1",
"crypto-browserify": "^3.12.0",
"dropbox": "^10.22.0",
"localforage": "^1.10.0", "localforage": "^1.10.0",
"mime-types": "^2.1.33", "mime-types": "^2.1.33",
"obsidian": "^0.12.0", "obsidian": "^0.12.0",
@@ -49,6 +60,7 @@
"rfc4648": "^1.5.0", "rfc4648": "^1.5.0",
"rimraf": "^3.0.2", "rimraf": "^3.0.2",
"stream-browserify": "^3.0.0", "stream-browserify": "^3.0.0",
"util": "^0.12.4",
"webdav": "^4.7.0", "webdav": "^4.7.0",
"webdav-fs": "^4.0.0", "webdav-fs": "^4.0.0",
"xregexp": "^5.1.0" "xregexp": "^5.1.0"
+13
View File
@@ -0,0 +1,13 @@
/**
* Only type defs here.
*/
export type SUPPORTED_SERVICES_TYPE = "s3" | "webdav" | "dropbox";
export interface RemoteItem {
key: string;
lastModified: number;
size: number;
remoteType: SUPPORTED_SERVICES_TYPE;
etag?: string;
}
+7 -5
View File
@@ -1,7 +1,7 @@
import localforage from "localforage"; import localforage from "localforage";
import { TAbstractFile, TFile, TFolder } from "obsidian"; import { TAbstractFile, TFile, TFolder } from "obsidian";
import type { SUPPORTED_SERVICES_TYPE } from "./misc"; import type { SUPPORTED_SERVICES_TYPE } from "./baseTypes";
import type { SyncPlanType } from "./sync"; import type { SyncPlanType } from "./sync";
export type LocalForage = typeof localforage; export type LocalForage = typeof localforage;
@@ -197,7 +197,8 @@ export const insertRenameRecord = async (
await db.deleteHistoryTbl.setItem(k.key, k); await db.deleteHistoryTbl.setItem(k.key, k);
}; };
export const upsertSyncMetaMappingDataS3 = async ( export const upsertSyncMetaMappingData = async (
serviceType: SUPPORTED_SERVICES_TYPE,
db: InternalDBs, db: InternalDBs,
localKey: string, localKey: string,
localMTime: number, localMTime: number,
@@ -215,13 +216,14 @@ export const upsertSyncMetaMappingDataS3 = async (
remoteMtime: remoteMTime, remoteMtime: remoteMTime,
remoteSize: remoteSize, remoteSize: remoteSize,
remoteExtraKey: remoteExtraKey, remoteExtraKey: remoteExtraKey,
remoteType: "s3", remoteType: serviceType,
keyType: localKey.endsWith("/") ? "folder" : "file", keyType: localKey.endsWith("/") ? "folder" : "file",
}; };
await db.syncMappingTbl.setItem(remoteKey, aggregratedInfo); await db.syncMappingTbl.setItem(remoteKey, aggregratedInfo);
}; };
export const getSyncMetaMappingByRemoteKeyS3 = async ( export const getSyncMetaMappingByRemoteKey = async (
serviceType: SUPPORTED_SERVICES_TYPE,
db: InternalDBs, db: InternalDBs,
remoteKey: string, remoteKey: string,
remoteMTime: number, remoteMTime: number,
@@ -240,7 +242,7 @@ export const getSyncMetaMappingByRemoteKeyS3 = async (
potentialItem.remoteKey === remoteKey && potentialItem.remoteKey === remoteKey &&
potentialItem.remoteMtime === remoteMTime && potentialItem.remoteMtime === remoteMTime &&
potentialItem.remoteExtraKey === remoteExtraKey && potentialItem.remoteExtraKey === remoteExtraKey &&
potentialItem.remoteType === "s3" potentialItem.remoteType === serviceType
) { ) {
// the result was found // the result was found
return potentialItem; return potentialItem;
+465 -33
View File
@@ -11,6 +11,7 @@ import {
TFolder, TFolder,
} from "obsidian"; } from "obsidian";
import * as CodeMirror from "codemirror"; import * as CodeMirror from "codemirror";
import process from "process";
import { import {
prepareDBs, prepareDBs,
destroyDBs, destroyDBs,
@@ -25,23 +26,37 @@ import type { InternalDBs } from "./localdb";
import type { SyncStatusType, PasswordCheckType } from "./sync"; import type { SyncStatusType, PasswordCheckType } from "./sync";
import { isPasswordOk, getSyncPlan, doActualSync } from "./sync"; import { isPasswordOk, getSyncPlan, doActualSync } from "./sync";
import { S3Config, DEFAULT_S3_CONFIG } from "./s3";
import { WebdavConfig, DEFAULT_WEBDAV_CONFIG, WebdavAuthType } from "./webdav";
import { import {
DEFAULT_S3_CONFIG, DropboxConfig,
getS3Client, DEFAULT_DROPBOX_CONFIG,
listFromRemote, getAuthUrlAndVerifier,
S3Config, sendAuthReq,
checkS3Connectivity, setConfigBySuccessfullAuthInplace,
} from "./s3"; } from "./remoteForDropbox";
import { RemoteClient } from "./remote";
import { exportSyncPlansToFiles } from "./debugMode"; import { exportSyncPlansToFiles } from "./debugMode";
import { SUPPORTED_SERVICES_TYPE } from "./baseTypes";
interface RemotelySavePluginSettings { interface RemotelySavePluginSettings {
s3?: S3Config; s3: S3Config;
password?: string; webdav: WebdavConfig;
dropbox: DropboxConfig;
password: string;
serviceType: SUPPORTED_SERVICES_TYPE;
enableExperimentService: boolean;
} }
const DEFAULT_SETTINGS: RemotelySavePluginSettings = { const DEFAULT_SETTINGS: RemotelySavePluginSettings = {
s3: DEFAULT_S3_CONFIG, s3: DEFAULT_S3_CONFIG,
webdav: DEFAULT_WEBDAV_CONFIG,
dropbox: DEFAULT_DROPBOX_CONFIG,
password: "", password: "",
serviceType: "s3",
enableExperimentService: false,
}; };
export default class RemotelySavePlugin extends Plugin { export default class RemotelySavePlugin extends Plugin {
@@ -51,7 +66,7 @@ export default class RemotelySavePlugin extends Plugin {
syncStatus: SyncStatusType; syncStatus: SyncStatusType;
async onload() { async onload() {
console.log("loading plugin obsidian-remotely-save"); console.log("loading plugin remotely-save");
await this.loadSettings(); await this.loadSettings();
@@ -81,19 +96,29 @@ export default class RemotelySavePlugin extends Plugin {
try { try {
//console.log(`huh ${this.settings.password}`) //console.log(`huh ${this.settings.password}`)
new Notice("1/6 Remotely Save Sync Preparing"); new Notice(
`1/6 Remotely Save Sync Preparing (${this.settings.serviceType})`
);
this.syncStatus = "preparing"; this.syncStatus = "preparing";
new Notice("2/6 Starting to fetch remote meta data."); new Notice("2/6 Starting to fetch remote meta data.");
this.syncStatus = "getting_remote_meta"; this.syncStatus = "getting_remote_meta";
const s3Client = getS3Client(this.settings.s3); const self = this;
const remoteRsp = await listFromRemote(s3Client, this.settings.s3); const client = new RemoteClient(
this.settings.serviceType,
this.settings.s3,
this.settings.webdav,
this.settings.dropbox,
this.app.vault.getName(),
() => self.saveSettings()
);
const remoteRsp = await client.listFromRemote();
// console.log(remoteRsp);
new Notice("3/6 Starting to fetch local meta data."); new Notice("3/6 Starting to fetch local meta data.");
this.syncStatus = "getting_local_meta"; this.syncStatus = "getting_local_meta";
const local = this.app.vault.getAllLoadedFiles(); const local = this.app.vault.getAllLoadedFiles();
const localHistory = await loadDeleteRenameHistoryTable(this.db); const localHistory = await loadDeleteRenameHistoryTable(this.db);
// console.log(remoteRsp);
// console.log(local); // console.log(local);
// console.log(localHistory); // console.log(localHistory);
@@ -115,6 +140,7 @@ export default class RemotelySavePlugin extends Plugin {
local, local,
localHistory, localHistory,
this.db, this.db,
client.serviceType,
this.settings.password this.settings.password
); );
console.log(syncPlan.mixedStates); // for debugging console.log(syncPlan.mixedStates); // for debugging
@@ -127,8 +153,7 @@ export default class RemotelySavePlugin extends Plugin {
this.syncStatus = "syncing"; this.syncStatus = "syncing";
await doActualSync( await doActualSync(
s3Client, client,
this.settings.s3,
this.db, this.db,
this.app.vault, this.app.vault,
syncPlan, syncPlan,
@@ -165,12 +190,19 @@ export default class RemotelySavePlugin extends Plugin {
} }
onunload() { onunload() {
console.log("unloading plugin obsidian-remotely-save"); console.log("unloading plugin remotely-save");
this.destroyDBs(); this.destroyDBs();
} }
async loadSettings() { async loadSettings() {
this.settings = Object.assign({}, DEFAULT_SETTINGS, await this.loadData()); this.settings = Object.assign(
{},
JSON.parse(JSON.stringify(DEFAULT_SETTINGS)) /* copy an object */,
await this.loadData()
);
if (this.settings.dropbox.clientID === "") {
this.settings.dropbox.clientID = DEFAULT_SETTINGS.dropbox.clientID;
}
} }
async saveSettings() { async saveSettings() {
@@ -244,6 +276,110 @@ export class PasswordModal extends Modal {
} }
} }
export class DropboxAuthModal extends Modal {
readonly plugin: RemotelySavePlugin;
readonly authDiv: HTMLDivElement;
readonly revokeAuthDiv: HTMLDivElement;
readonly revokeAuthSetting: Setting;
constructor(
app: App,
plugin: RemotelySavePlugin,
authDiv: HTMLDivElement,
revokeAuthDiv: HTMLDivElement,
revokeAuthSetting: Setting
) {
super(app);
this.plugin = plugin;
this.authDiv = authDiv;
this.revokeAuthDiv = revokeAuthDiv;
this.revokeAuthSetting = revokeAuthSetting;
}
async onOpen() {
let { contentEl } = this;
const { authUrl, verifier } = await getAuthUrlAndVerifier(
this.plugin.settings.dropbox.clientID
);
contentEl.createEl("p", {
text: "Step 1: Visit the following address in a browser, and follow the steps on the web page to authorize.",
});
contentEl.createEl("p").createEl("a", {
href: authUrl,
text: authUrl,
});
contentEl.createEl("p", {
text: 'Step 2: In the end of the web flow, you obtain a long code. Paste it here then click "Submit".',
});
let authCode = "";
new Setting(contentEl)
.setName("Auth Code from web page")
.setDesc('You need to click "Confirm".')
.addText((text) =>
text
.setPlaceholder("")
.setValue("")
.onChange((val) => {
authCode = val.trim();
})
)
.addButton(async (button) => {
button.setButtonText("Confirm");
button.onClick(async () => {
new Notice("Trying to connect to Dropbox");
try {
const authRes = await sendAuthReq(
this.plugin.settings.dropbox.clientID,
verifier,
authCode
);
const self = this;
setConfigBySuccessfullAuthInplace(
this.plugin.settings.dropbox,
authRes,
() => self.plugin.saveSettings()
);
const client = new RemoteClient(
"dropbox",
undefined,
undefined,
this.plugin.settings.dropbox,
this.app.vault.getName(),
() => self.plugin.saveSettings()
);
const username = await client.getUser();
this.plugin.settings.dropbox.username = username;
await this.plugin.saveSettings();
new Notice(`Good! We've connected to Dropbox as user ${username}!`);
this.authDiv.toggleClass(
"dropbox-auth-button-hide",
this.plugin.settings.dropbox.username !== ""
);
this.revokeAuthDiv.toggleClass(
"dropbox-revoke-auth-button-hide",
this.plugin.settings.dropbox.username === ""
);
this.revokeAuthSetting.setDesc(
`You've connected as user ${this.plugin.settings.dropbox.username}. If you want to disconnect, click this button`
);
this.close();
} catch (err) {
console.error(err);
new Notice("Something goes wrong while connecting to Dropbox.");
}
});
});
}
onClose() {
let { contentEl } = this;
contentEl.empty();
}
}
class RemotelySaveSettingTab extends PluginSettingTab { class RemotelySaveSettingTab extends PluginSettingTab {
plugin: RemotelySavePlugin; plugin: RemotelySavePlugin;
@@ -259,7 +395,69 @@ class RemotelySaveSettingTab extends PluginSettingTab {
containerEl.createEl("h1", { text: "Remotely Save" }); containerEl.createEl("h1", { text: "Remotely Save" });
const s3Div = containerEl.createEl("div"); const generalDiv = containerEl.createEl("div");
generalDiv.createEl("h2", { text: "General" });
const passwordDiv = generalDiv.createEl("div");
let newPassword = `${this.plugin.settings.password}`;
new Setting(passwordDiv)
.setName("encryption password")
.setDesc(
'Password for E2E encryption. Empty for no password. You need to click "Confirm".'
)
.addText((text) =>
text
.setPlaceholder("")
.setValue(`${this.plugin.settings.password}`)
.onChange(async (value) => {
newPassword = value.trim();
})
)
.addButton(async (button) => {
button.setButtonText("Confirm");
button.onClick(async () => {
new PasswordModal(this.app, this.plugin, newPassword).open();
});
});
// we need to create the div in advance of any other service divs
const serviceChooserDiv = generalDiv.createEl("div");
let clickChooserTimes = 0;
serviceChooserDiv.onClickEvent(async (x) => {
if (Platform.isIosApp) {
// downgrade the experiment
// because iOS doesn't support x.detail
clickChooserTimes += 1;
setTimeout(function () {
clickChooserTimes = 0;
}, 2000);
}
if ((Platform.isIosApp && clickChooserTimes === 5) || x.detail === 5) {
if (this.plugin.settings.serviceType === "webdav") {
new Notice(
"You've enabled hidden unstable experimental webdav support before. Nothing changes."
);
} else if (!this.plugin.settings.enableExperimentService) {
this.plugin.settings.enableExperimentService = true;
await this.plugin.saveSettings();
new Notice(
"You've enabled hidden unstable experimental webdav support. Reopen settings again and try webdav with caution."
);
} else if (this.plugin.settings.enableExperimentService) {
this.plugin.settings.enableExperimentService = false;
await this.plugin.saveSettings();
new Notice(
"You've disabled hidden unstable experimental webdav support. Reopen settings again."
);
}
}
x.preventDefault();
});
const s3Div = containerEl.createEl("div", { cls: "s3-hide" });
s3Div.toggleClass("s3-hide", this.plugin.settings.serviceType !== "s3");
s3Div.createEl("h2", { text: "S3 (-compatible) Service" }); s3Div.createEl("h2", { text: "S3 (-compatible) Service" });
s3Div.createEl("p", { s3Div.createEl("p", {
@@ -368,11 +566,12 @@ class RemotelySaveSettingTab extends PluginSettingTab {
button.setButtonText("Check"); button.setButtonText("Check");
button.onClick(async () => { button.onClick(async () => {
new Notice("Checking..."); new Notice("Checking...");
const s3Client = getS3Client(this.plugin.settings.s3); const client = new RemoteClient(
const res = await checkS3Connectivity( "s3",
s3Client, this.plugin.settings.s3,
this.plugin.settings.s3 undefined
); );
const res = await client.checkConnectivity();
if (res) { if (res) {
new Notice("Great! The bucket can be accessed."); new Notice("Great! The bucket can be accessed.");
} else { } else {
@@ -381,28 +580,261 @@ class RemotelySaveSettingTab extends PluginSettingTab {
}); });
}); });
const generalDiv = containerEl.createEl("div"); const dropboxDiv = containerEl.createEl("div", { cls: "dropbox-hide" });
generalDiv.createEl("h2", { text: "General" }); dropboxDiv.toggleClass(
"dropbox-hide",
this.plugin.settings.serviceType !== "dropbox"
);
dropboxDiv.createEl("h2", { text: "for Dropbox" });
dropboxDiv.createEl("p", {
text: "Disclaimer: Sync support for Dropbox are more experimental, and s3 functions are more stable now.",
cls: "dropbox-disclaimer",
});
dropboxDiv.createEl("p", {
text: "Disclaimer: This app is NOT an official Dropbox product. It just uses Dropbox open api.",
cls: "dropbox-disclaimer",
});
dropboxDiv.createEl("p", {
text: "Disclaimer: The infomation is stored in PLAIN TEXT locally. Other malicious/harmful/faulty plugins may or may not be able to read the info. If you see any unintentional access to your Dropbox, please immediately disconnect this app on https://www.dropbox.com/account/connected_apps .",
cls: "dropbox-disclaimer",
});
dropboxDiv.createEl("p", {
text: "We will create a folder Apps/remotely-save on your Dropbox. All files/folders sync would happen inside this folder.",
});
const passwordDiv = generalDiv.createEl("div"); const dropboxSelectAuthDiv = dropboxDiv.createDiv();
let newPassword = `${this.plugin.settings.password}`; const dropboxAuthDiv = dropboxSelectAuthDiv.createDiv({
new Setting(passwordDiv) cls: "dropbox-auth-button-hide",
.setName("encryption password") });
const dropboxRevokeAuthDiv = dropboxSelectAuthDiv.createDiv({
cls: "dropbox-revoke-auth-button-hide",
});
const revokeAuthSetting = new Setting(dropboxRevokeAuthDiv)
.setName("Revoke Auth")
.setDesc( .setDesc(
'Password for E2E encryption. Empty for no password. You need to click "Confirm".' `You've connected as user ${this.plugin.settings.dropbox.username}. If you want to disconnect, click this button`
) )
.addButton(async (button) => {
button.setButtonText("Revoke Auth");
button.onClick(async () => {
try {
const self = this;
const client = new RemoteClient(
"dropbox",
undefined,
undefined,
this.plugin.settings.dropbox,
this.app.vault.getName(),
() => self.plugin.saveSettings()
);
await client.revokeAuth();
this.plugin.settings.dropbox = JSON.parse(
JSON.stringify(DEFAULT_DROPBOX_CONFIG)
);
await this.plugin.saveSettings();
dropboxAuthDiv.toggleClass(
"dropbox-auth-button-hide",
this.plugin.settings.dropbox.username !== ""
);
dropboxRevokeAuthDiv.toggleClass(
"dropbox-revoke-auth-button-hide",
this.plugin.settings.dropbox.username === ""
);
new Notice("Revoked!");
} catch (err) {
console.error(err);
new Notice("Something goes wrong while revoking");
}
});
});
new Setting(dropboxAuthDiv)
.setName("Auth")
.setDesc("Auth")
.addButton(async (button) => {
button.setButtonText("Auth");
button.onClick(async () => {
new DropboxAuthModal(
this.app,
this.plugin,
dropboxAuthDiv,
dropboxRevokeAuthDiv,
revokeAuthSetting
).open();
});
});
dropboxAuthDiv.toggleClass(
"dropbox-auth-button-hide",
this.plugin.settings.dropbox.username !== ""
);
dropboxRevokeAuthDiv.toggleClass(
"dropbox-revoke-auth-button-hide",
this.plugin.settings.dropbox.username === ""
);
new Setting(dropboxDiv)
.setName("check connectivity")
.setDesc("check connectivity")
.addButton(async (button) => {
button.setButtonText("Check");
button.onClick(async () => {
new Notice("Checking...");
const self = this;
const client = new RemoteClient(
"dropbox",
undefined,
undefined,
this.plugin.settings.dropbox,
this.app.vault.getName(),
() => self.plugin.saveSettings()
);
const res = await client.checkConnectivity();
if (res) {
new Notice("Great! We can connect to Dropbox!");
} else {
new Notice("We cannot connect to Dropbox.");
}
});
});
const webdavDiv = containerEl.createEl("div", { cls: "webdav-hide" });
webdavDiv.toggleClass(
"webdav-hide",
this.plugin.settings.serviceType !== "webdav"
);
webdavDiv.createEl("h2", { text: "Webdav Service" });
webdavDiv.createEl("p", {
text: "Disclaimer: Webdav functions are more experimental, and s3 functions are more stable now.",
cls: "webdav-disclaimer",
});
webdavDiv.createEl("p", {
text: "Disclaimer: The infomation is stored in PLAIN TEXT locally. Other malicious/harmful/faulty plugins may or may not be able to read the info. If you see any unintentional access to your webdav server, please immediately change the username and/or password to stop further accessment.",
cls: "webdav-disclaimer",
});
webdavDiv.createEl("p", {
text: "You need to configure CORS to allow requests from origin app://obsidian.md and capacitor://localhost and http://localhost",
});
new Setting(webdavDiv)
.setName("server address")
.setDesc("server address")
.addText((text) => .addText((text) =>
text text
.setPlaceholder("") .setPlaceholder("")
.setValue(`${this.plugin.settings.password}`) .setValue(this.plugin.settings.webdav.address)
.onChange(async (value) => { .onChange(async (value) => {
newPassword = value.trim(); this.plugin.settings.webdav.address = value.trim();
await this.plugin.saveSettings();
}) })
);
new Setting(webdavDiv)
.setName("server username")
.setDesc("server username")
.addText((text) =>
text
.setPlaceholder("")
.setValue(this.plugin.settings.webdav.username)
.onChange(async (value) => {
this.plugin.settings.webdav.username = value.trim();
await this.plugin.saveSettings();
})
);
new Setting(webdavDiv)
.setName("server password")
.setDesc("server password")
.addText((text) =>
text
.setPlaceholder("")
.setValue(this.plugin.settings.webdav.password)
.onChange(async (value) => {
this.plugin.settings.webdav.password = value.trim();
await this.plugin.saveSettings();
})
);
new Setting(webdavDiv)
.setName("server auth type")
.setDesc(
"Server auth type. If you do not set password, this option would be ignored."
) )
.addDropdown((dropdown) => {
dropdown.addOption("basic", "basic");
dropdown.addOption("digest", "digest");
dropdown
.setValue(this.plugin.settings.webdav.authType)
.onChange(async (val: WebdavAuthType) => {
this.plugin.settings.webdav.authType = val;
await this.plugin.saveSettings();
});
});
new Setting(webdavDiv)
.setName("check connectivity")
.setDesc("check connectivity")
.addButton(async (button) => { .addButton(async (button) => {
button.setButtonText("Confirm"); button.setButtonText("Check");
button.onClick(async () => { button.onClick(async () => {
new PasswordModal(this.app, this.plugin, newPassword).open(); new Notice("Checking...");
const client = new RemoteClient(
"webdav",
undefined,
this.plugin.settings.webdav
);
const res = await client.checkConnectivity();
if (res) {
new Notice("Great! The webdav server can be accessed.");
} else {
new Notice("The webdav server cannot be reached.");
}
});
});
// we need to create chooser
// after s3Div and webdavDiv being created
new Setting(serviceChooserDiv)
.setName("Choose service")
.setDesc("Choose a service, by default s3")
.addDropdown(async (dropdown) => {
const currService = this.plugin.settings.serviceType;
const enableExperimentService =
this.plugin.settings.enableExperimentService;
dropdown.addOption("s3", "s3 (-compatible)");
dropdown.addOption("dropbox", "Dropbox");
if (currService === "webdav" || enableExperimentService) {
dropdown.addOption("webdav", "webdav (experimental)");
if (!enableExperimentService) {
this.plugin.settings.enableExperimentService = true;
await this.plugin.saveSettings();
}
}
dropdown
.setValue(this.plugin.settings.serviceType)
.onChange(async (val: SUPPORTED_SERVICES_TYPE) => {
this.plugin.settings.serviceType = val;
s3Div.toggleClass(
"s3-hide",
this.plugin.settings.serviceType !== "s3"
);
dropboxDiv.toggleClass(
"dropbox-hide",
this.plugin.settings.serviceType !== "dropbox"
);
webdavDiv.toggleClass(
"webdav-hide",
this.plugin.settings.serviceType !== "webdav"
);
await this.plugin.saveSettings();
}); });
}); });
+28 -3
View File
@@ -4,8 +4,6 @@ import * as path from "path";
import { base32 } from "rfc4648"; import { base32 } from "rfc4648";
import XRegExp from "xregexp"; import XRegExp from "xregexp";
export type SUPPORTED_SERVICES_TYPE = "s3" | "webdav" | "ftp";
/** /**
* If any part of the file starts with '.' or '_' then it's a hidden file. * If any part of the file starts with '.' or '_' then it's a hidden file.
* @param item * @param item
@@ -47,7 +45,10 @@ export const getFolderLevels = (x: string) => {
const y1 = x.split("/"); const y1 = x.split("/");
let i = 0; let i = 0;
for (let index = 0; index + 1 < y1.length; index++) { for (let index = 0; index + 1 < y1.length; index++) {
res.push(y1.slice(0, index + 1).join("/")); const k = y1.slice(0, index + 1).join("/");
if (k !== "" && k !== "/") {
res.push(k);
}
} }
return res; return res;
}; };
@@ -130,3 +131,27 @@ export const isVaildText = (a: string) => {
a a
); );
}; };
/**
* If input is already a folder, returns it as is;
* And if input is a file, returns its direname.
* @param a
* @returns
*/
export const getPathFolder = (a: string) => {
if (a.endsWith("/")) {
return a;
}
const b = path.posix.dirname(a);
return b.endsWith("/") ? b : `${b}/`;
};
/**
* https://stackoverflow.com/questions/54511144
* @param a
* @param delimiter
* @returns
*/
export const setToString = (a: Set<string>, delimiter: string = ",") => {
return [...a].join(delimiter);
};
+220
View File
@@ -0,0 +1,220 @@
import { Vault } from "obsidian";
import type { SUPPORTED_SERVICES_TYPE } from "./baseTypes";
import * as s3 from "./s3";
import * as webdav from "./webdav";
import * as dropbox from "./remoteForDropbox";
export class RemoteClient {
readonly serviceType: SUPPORTED_SERVICES_TYPE;
readonly s3Client?: s3.S3Client;
readonly s3Config?: s3.S3Config;
readonly webdavClient?: webdav.WebDAVClient;
readonly webdavConfig?: webdav.WebdavConfig;
readonly dropboxClient?: dropbox.WrappedDropboxClient;
readonly dropboxConfig?: dropbox.DropboxConfig;
constructor(
serviceType: SUPPORTED_SERVICES_TYPE,
s3Config?: s3.S3Config,
webdavConfig?: webdav.WebdavConfig,
dropboxConfig?: dropbox.DropboxConfig,
vaultName?: string,
saveUpdatedConfigFunc?: () => Promise<any>
) {
this.serviceType = serviceType;
// the client may modify the config inplace,
// so we use a ref not copy of config here
if (serviceType === "s3") {
this.s3Config = s3Config;
this.s3Client = s3.getS3Client(this.s3Config);
} else if (serviceType === "webdav") {
this.webdavConfig = webdavConfig;
this.webdavClient = webdav.getWebdavClient(this.webdavConfig);
} else if (serviceType === "dropbox") {
if (vaultName === undefined || saveUpdatedConfigFunc === undefined) {
throw Error(
"remember to provide vault name and callback while init dropbox client"
);
}
this.dropboxConfig = dropboxConfig;
this.dropboxClient = dropbox.getDropboxClient(
this.dropboxConfig,
vaultName,
saveUpdatedConfigFunc
);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
}
getRemoteMeta = async (fileOrFolderPath: string) => {
if (this.serviceType === "s3") {
return await s3.getRemoteMeta(
this.s3Client,
this.s3Config,
fileOrFolderPath
);
} else if (this.serviceType === "webdav") {
return await webdav.getRemoteMeta(this.webdavClient, fileOrFolderPath);
} else if (this.serviceType === "dropbox") {
return await dropbox.getRemoteMeta(this.dropboxClient, fileOrFolderPath);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
};
uploadToRemote = async (
fileOrFolderPath: string,
vault: Vault,
isRecursively: boolean = false,
password: string = "",
remoteEncryptedKey: string = "",
foldersCreatedBefore: Set<string> | undefined = undefined
) => {
if (this.serviceType === "s3") {
return await s3.uploadToRemote(
this.s3Client,
this.s3Config,
fileOrFolderPath,
vault,
isRecursively,
password,
remoteEncryptedKey
);
} else if (this.serviceType === "webdav") {
return await webdav.uploadToRemote(
this.webdavClient,
fileOrFolderPath,
vault,
isRecursively,
password,
remoteEncryptedKey
);
} else if (this.serviceType === "dropbox") {
return await dropbox.uploadToRemote(
this.dropboxClient,
fileOrFolderPath,
vault,
isRecursively,
password,
remoteEncryptedKey,
foldersCreatedBefore
);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
};
listFromRemote = async (prefix?: string) => {
if (this.serviceType === "s3") {
return await s3.listFromRemote(this.s3Client, this.s3Config, prefix);
} else if (this.serviceType === "webdav") {
return await webdav.listFromRemote(this.webdavClient, prefix);
} else if (this.serviceType === "dropbox") {
return await dropbox.listFromRemote(this.dropboxClient, prefix);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
};
downloadFromRemote = async (
fileOrFolderPath: string,
vault: Vault,
mtime: number,
password: string = "",
remoteEncryptedKey: string = ""
) => {
if (this.serviceType === "s3") {
return await s3.downloadFromRemote(
this.s3Client,
this.s3Config,
fileOrFolderPath,
vault,
mtime,
password,
remoteEncryptedKey
);
} else if (this.serviceType === "webdav") {
return await webdav.downloadFromRemote(
this.webdavClient,
fileOrFolderPath,
vault,
mtime,
password,
remoteEncryptedKey
);
} else if (this.serviceType === "dropbox") {
return await dropbox.downloadFromRemote(
this.dropboxClient,
fileOrFolderPath,
vault,
mtime,
password,
remoteEncryptedKey
);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
};
deleteFromRemote = async (
fileOrFolderPath: string,
password: string = "",
remoteEncryptedKey: string = ""
) => {
if (this.serviceType === "s3") {
return await s3.deleteFromRemote(
this.s3Client,
this.s3Config,
fileOrFolderPath,
password,
remoteEncryptedKey
);
} else if (this.serviceType === "webdav") {
return await webdav.deleteFromRemote(
this.webdavClient,
fileOrFolderPath,
password,
remoteEncryptedKey
);
} else if (this.serviceType === "dropbox") {
return await dropbox.deleteFromRemote(
this.dropboxClient,
fileOrFolderPath,
password,
remoteEncryptedKey
);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
};
checkConnectivity = async () => {
if (this.serviceType === "s3") {
return await s3.checkConnectivity(this.s3Client, this.s3Config);
} else if (this.serviceType === "webdav") {
return await webdav.checkConnectivity(this.webdavClient);
} else if (this.serviceType === "dropbox") {
return await dropbox.checkConnectivity(this.dropboxClient);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
};
getUser = async () => {
if (this.serviceType === "dropbox") {
return await dropbox.getUserDisplayName(this.dropboxClient);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
};
revokeAuth = async () => {
if (this.serviceType === "dropbox") {
return await dropbox.revokeAuth(this.dropboxClient);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
};
}
+609
View File
@@ -0,0 +1,609 @@
import * as path from "path";
import { FileStats, Vault } from "obsidian";
import { Dropbox, DropboxAuth, DropboxResponse, files } from "dropbox";
export { Dropbox } from "dropbox";
import { RemoteItem } from "./baseTypes";
import {
arrayBufferToBuffer,
bufferToArrayBuffer,
mkdirpInVault,
getPathFolder,
getFolderLevels,
setToString,
} from "./misc";
import { decryptArrayBuffer, encryptArrayBuffer } from "./encrypt";
export interface DropboxConfig {
accessToken: string;
clientID: string;
refreshToken: string;
accessTokenExpiresInSeconds: number;
accessTokenExpiresAtTime: number;
accountID: string;
username: string;
}
export const DEFAULT_DROPBOX_CONFIG = {
accessToken: "",
clientID: process.env.DEFAULT_DROPBOX_APP_KEY,
refreshToken: "",
accessTokenExpiresInSeconds: 0,
accessTokenExpiresAtTime: 0,
accountID: "",
username: "",
};
export const getDropboxPath = (fileOrFolderPath: string, vaultName: string) => {
let key = fileOrFolderPath;
if (!fileOrFolderPath.startsWith("/")) {
// then this is original path in Obsidian
key = `/${vaultName}/${fileOrFolderPath}`;
}
if (key.endsWith("/")) {
key = key.slice(0, key.length - 1);
}
return key;
};
const getNormPath = (fileOrFolderPath: string, vaultName: string) => {
if (
!(
fileOrFolderPath === `/${vaultName}` ||
fileOrFolderPath.startsWith(`/${vaultName}/`)
)
) {
throw Error(`"${fileOrFolderPath}" doesn't starts with "/${vaultName}/"`);
}
return fileOrFolderPath.slice(`/${vaultName}/`.length);
};
const fromDropboxItemToRemoteItem = (
x:
| files.FileMetadataReference
| files.FolderMetadataReference
| files.DeletedMetadataReference,
vaultName: string
): RemoteItem => {
let key = getNormPath(x.path_display, vaultName);
if (x[".tag"] === "folder" && !key.endsWith("/")) {
key = `${key}/`;
}
if (x[".tag"] === "folder") {
return {
key: key,
lastModified: undefined,
size: 0,
remoteType: "dropbox",
etag: `${x.id}\t`,
} as RemoteItem;
} else if (x[".tag"] === "file") {
return {
key: key,
lastModified: Date.parse(x.server_modified).valueOf(),
size: x.size,
remoteType: "dropbox",
etag: `${x.id}\t${x.content_hash}`,
} as RemoteItem;
} else if (x[".tag"] === "deleted") {
throw Error("do not support deleted tag");
}
};
/**
* Dropbox api doesn't return mtime for folders.
* This is a try to assign mtime by using files in folder.
* @param allFilesFolders
* @returns
*/
const fixLastModifiedTimeInplace = (allFilesFolders: RemoteItem[]) => {
if (allFilesFolders.length === 0) {
return;
}
// sort by longer to shorter
allFilesFolders.sort((a, b) => b.key.length - a.key.length);
// a "map" from dir to mtime
let potentialMTime = {} as Record<string, number>;
// first sort pass, from buttom to up
for (const item of allFilesFolders) {
if (item.key.endsWith("/")) {
// itself is a folder, and initially doesn't have mtime
if (item.lastModified === undefined && item.key in potentialMTime) {
// previously we gathered all sub info of this folder
item.lastModified = potentialMTime[item.key];
}
}
const parent = `${path.posix.dirname(item.key)}/`;
if (item.lastModified !== undefined) {
if (parent in potentialMTime) {
potentialMTime[parent] = Math.max(
potentialMTime[parent],
item.lastModified
);
} else {
potentialMTime[parent] = item.lastModified;
}
}
}
// second pass, from up to buttom.
// fill mtime by parent folder or Date.Now() if still not available.
// this is only possible if no any sub-folder-files recursively.
// we do not sort the array again, just iterate over it by reverse
// using good old for loop.
for (let i = allFilesFolders.length - 1; i >= 0; --i) {
const item = allFilesFolders[i];
if (!item.key.endsWith("/")) {
continue; // skip files
}
if (item.lastModified !== undefined) {
continue; // don't need to deal with it
}
const parent = `${path.posix.dirname(item.key)}/`;
if (parent in potentialMTime) {
item.lastModified = potentialMTime[parent];
} else {
item.lastModified = Date.now().valueOf();
potentialMTime[item.key] = item.lastModified;
}
}
return allFilesFolders;
};
////////////////////////////////////////////////////////////////////////////////
// Dropbox authorization using PKCE
// see https://dropbox.tech/developers/pkce--what-and-why-
////////////////////////////////////////////////////////////////////////////////
export const getAuthUrlAndVerifier = async (appKey: string) => {
const auth = new DropboxAuth({
clientId: appKey,
});
const authUrl = (
await auth.getAuthenticationUrl(
undefined,
undefined,
"code",
"offline",
undefined,
"none",
true
)
).toString();
const verifier = auth.getCodeVerifier();
return {
authUrl: authUrl,
verifier: verifier,
};
};
export interface DropboxSuccessAuthRes {
access_token: string;
token_type: "bearer";
expires_in: string;
refresh_token?: string;
scope?: string;
uid?: string;
account_id?: string;
}
export const sendAuthReq = async (
appKey: string,
verifier: string,
authCode: string
) => {
const resp1 = await fetch("https://api.dropboxapi.com/oauth2/token", {
method: "POST",
body: new URLSearchParams({
code: authCode,
grant_type: "authorization_code",
code_verifier: verifier,
client_id: appKey,
}),
});
const resp2 = (await resp1.json()) as DropboxSuccessAuthRes;
return resp2;
};
export const sendRefreshTokenReq = async (
appKey: string,
refreshToken: string
) => {
console.log("start auto getting refreshed Dropbox access token.");
const resp1 = await fetch("https://api.dropboxapi.com/oauth2/token", {
method: "POST",
body: new URLSearchParams({
grant_type: "refresh_token",
refresh_token: refreshToken,
client_id: appKey,
}),
});
const resp2 = (await resp1.json()) as DropboxSuccessAuthRes;
console.log("finish auto getting refreshed Dropbox access token.");
return resp2;
};
export const setConfigBySuccessfullAuthInplace = async (
config: DropboxConfig,
authRes: DropboxSuccessAuthRes,
saveUpdatedConfigFunc: () => Promise<any> | undefined
) => {
console.log("start updating local info of Dropbox token");
config.accessToken = authRes.access_token;
config.accessTokenExpiresInSeconds = parseInt(authRes.expires_in);
config.accessTokenExpiresAtTime =
Date.now() + parseInt(authRes.expires_in) * 1000 - 10 * 1000;
if (authRes.refresh_token !== undefined) {
config.refreshToken = authRes.refresh_token;
}
if (authRes.refresh_token !== undefined) {
config.accountID = authRes.account_id;
}
if (saveUpdatedConfigFunc !== undefined) {
await saveUpdatedConfigFunc();
}
console.log("finish updating local info of Dropbox token");
};
////////////////////////////////////////////////////////////////////////////////
// Other usual common methods
////////////////////////////////////////////////////////////////////////////////
export class WrappedDropboxClient {
dropboxConfig: DropboxConfig;
vaultName: string;
saveUpdatedConfigFunc: () => Promise<any>;
dropbox: Dropbox;
vaultFolderExists: boolean;
constructor(
dropboxConfig: DropboxConfig,
vaultName: string,
saveUpdatedConfigFunc: () => Promise<any>
) {
this.dropboxConfig = dropboxConfig;
this.vaultName = vaultName;
this.saveUpdatedConfigFunc = saveUpdatedConfigFunc;
this.vaultFolderExists = false;
}
init = async () => {
// check token
if (
this.dropboxConfig.accessToken === "" ||
this.dropboxConfig.refreshToken === ""
) {
throw Error("The user has not manually auth yet.");
}
const currentTs = Date.now();
if (this.dropboxConfig.accessTokenExpiresAtTime > currentTs) {
this.dropbox = new Dropbox({
accessToken: this.dropboxConfig.accessToken,
});
} else {
if (this.dropboxConfig.refreshToken === "") {
throw Error(
"We need to automatically refresh token but none is stored."
);
}
const resp = await sendRefreshTokenReq(
this.dropboxConfig.clientID,
this.dropboxConfig.refreshToken
);
setConfigBySuccessfullAuthInplace(
this.dropboxConfig,
resp,
this.saveUpdatedConfigFunc
);
this.dropbox = new Dropbox({
accessToken: this.dropboxConfig.accessToken,
});
}
// check vault folder
// console.log(`checking remote has folder /${this.vaultName}`);
if (this.vaultFolderExists) {
// console.log(`already checked, /${this.vaultName} exist before`)
} else {
const res = await this.dropbox.filesListFolder({
path: "",
recursive: false,
});
for (const item of res.result.entries) {
if (item.path_display === `/${this.vaultName}`) {
this.vaultFolderExists = true;
break;
}
}
if (!this.vaultFolderExists) {
console.log(`remote does not have folder /${this.vaultName}`);
await this.dropbox.filesCreateFolderV2({
path: `/${this.vaultName}`,
});
console.log(`remote folder /${this.vaultName} created`);
} else {
// console.log(`remote folder /${this.vaultName} exists`);
}
}
return this.dropbox;
};
}
/**
* @param dropboxConfig
* @returns
*/
export const getDropboxClient = (
dropboxConfig: DropboxConfig,
vaultName: string,
saveUpdatedConfigFunc: () => Promise<any>
) => {
return new WrappedDropboxClient(
dropboxConfig,
vaultName,
saveUpdatedConfigFunc
);
};
export const getRemoteMeta = async (
client: WrappedDropboxClient,
fileOrFolderPath: string
) => {
await client.init();
if (fileOrFolderPath === "" || fileOrFolderPath === "/") {
// filesGetMetadata doesn't support root folder
// we instead try to list files
// if no error occurs, we ensemble a fake result.
const rsp = await client.dropbox.filesListFolder({
path: `/${client.vaultName}`,
recursive: false, // don't need to recursive here
});
if (rsp.status !== 200) {
throw Error(JSON.stringify(rsp));
}
return {
key: fileOrFolderPath,
lastModified: undefined,
size: 0,
remoteType: "dropbox",
etag: undefined,
} as RemoteItem;
}
const key = getDropboxPath(fileOrFolderPath, client.vaultName);
const rsp = await client.dropbox.filesGetMetadata({
path: key,
});
if (rsp.status !== 200) {
throw Error(JSON.stringify(rsp));
}
return fromDropboxItemToRemoteItem(rsp.result, client.vaultName);
};
export const uploadToRemote = async (
client: WrappedDropboxClient,
fileOrFolderPath: string,
vault: Vault,
isRecursively: boolean = false,
password: string = "",
remoteEncryptedKey: string = "",
foldersCreatedBefore: Set<string> | undefined = undefined
) => {
await client.init();
let uploadFile = fileOrFolderPath;
if (password !== "") {
uploadFile = remoteEncryptedKey;
}
uploadFile = getDropboxPath(uploadFile, client.vaultName);
const isFolder = fileOrFolderPath.endsWith("/");
if (isFolder && isRecursively) {
throw Error("upload function doesn't implement recursive function yet!");
} else if (isFolder && !isRecursively) {
// folder
if (password === "") {
// if not encrypted, mkdir a remote folder
if (foldersCreatedBefore?.has(uploadFile)) {
// created, pass
} else {
try {
await client.dropbox.filesCreateFolderV2({
path: uploadFile,
});
foldersCreatedBefore?.add(uploadFile);
} catch (err) {
if (err.status === 409) {
// pass
foldersCreatedBefore?.add(uploadFile);
} else {
throw err;
}
}
}
const res = await getRemoteMeta(client, uploadFile);
return res;
} else {
// if encrypted, upload a fake file with the encrypted file name
await client.dropbox.filesUpload({
path: uploadFile,
contents: "",
});
return await getRemoteMeta(client, uploadFile);
}
} else {
// file
// we ignore isRecursively parameter here
const localContent = await vault.adapter.readBinary(fileOrFolderPath);
let remoteContent = localContent;
if (password !== "") {
remoteContent = await encryptArrayBuffer(localContent, password);
}
// in dropbox, we don't need to create folders before uploading! cool!
// TODO: filesUploadSession for larger files (>=150 MB)
await client.dropbox.filesUpload({
path: uploadFile,
contents: remoteContent,
mode: {
".tag": "overwrite",
},
});
// we want to mark that parent folders are created
if (foldersCreatedBefore !== undefined) {
const dirs = getFolderLevels(uploadFile).map((x) =>
getDropboxPath(x, client.vaultName)
);
for (const dir of dirs) {
foldersCreatedBefore?.add(dir);
}
}
return await getRemoteMeta(client, uploadFile);
}
};
export const listFromRemote = async (
client: WrappedDropboxClient,
prefix?: string
) => {
if (prefix !== undefined) {
throw Error("prefix not supported (yet)");
}
await client.init();
const res = await client.dropbox.filesListFolder({
path: `/${client.vaultName}`,
recursive: true,
});
if (res.status !== 200) {
throw Error(JSON.stringify(res));
}
// console.log(res);
const contents = res.result.entries;
const unifiedContents = contents
.filter((x) => x[".tag"] !== "deleted")
.filter((x) => x.path_display !== `/${client.vaultName}`)
.map((x) => fromDropboxItemToRemoteItem(x, client.vaultName));
fixLastModifiedTimeInplace(unifiedContents);
return {
Contents: unifiedContents,
};
};
const downloadFromRemoteRaw = async (
client: WrappedDropboxClient,
fileOrFolderPath: string
) => {
await client.init();
const key = getDropboxPath(fileOrFolderPath, client.vaultName);
const rsp = await client.dropbox.filesDownload({
path: key,
});
if ((rsp.result as any).fileBlob !== undefined) {
// we get a Blob
const content = (rsp.result as any).fileBlob as Blob;
return await content.arrayBuffer();
} else if ((rsp.result as any).fileBinary !== undefined) {
// we get a Buffer
const content = (rsp.result as any).fileBinary as Buffer;
return bufferToArrayBuffer(content);
} else {
throw Error(`unknown rsp from dropbox download: ${rsp}`);
}
};
export const downloadFromRemote = async (
client: WrappedDropboxClient,
fileOrFolderPath: string,
vault: Vault,
mtime: number,
password: string = "",
remoteEncryptedKey: string = ""
) => {
const isFolder = fileOrFolderPath.endsWith("/");
await mkdirpInVault(fileOrFolderPath, vault);
// the file is always local file
// we need to encrypt it
if (isFolder) {
// mkdirp locally is enough
// do nothing here
} else {
let downloadFile = fileOrFolderPath;
if (password !== "") {
downloadFile = remoteEncryptedKey;
}
downloadFile = getDropboxPath(downloadFile, client.vaultName);
const remoteContent = await downloadFromRemoteRaw(client, downloadFile);
let localContent = remoteContent;
if (password !== "") {
localContent = await decryptArrayBuffer(remoteContent, password);
}
await vault.adapter.writeBinary(fileOrFolderPath, localContent, {
mtime: mtime,
});
}
};
export const deleteFromRemote = async (
client: WrappedDropboxClient,
fileOrFolderPath: string,
password: string = "",
remoteEncryptedKey: string = ""
) => {
if (fileOrFolderPath === "/") {
return;
}
let remoteFileName = fileOrFolderPath;
if (password !== "") {
remoteFileName = remoteEncryptedKey;
}
remoteFileName = getDropboxPath(remoteFileName, client.vaultName);
await client.init();
try {
await client.dropbox.filesDeleteV2({
path: remoteFileName,
});
} catch (err) {
console.error("some error while deleting");
console.error(err);
}
};
export const checkConnectivity = async (client: WrappedDropboxClient) => {
try {
const results = await getRemoteMeta(client, "/");
if (results === undefined) {
return false;
}
return true;
} catch (err) {
console.error("dropbox connectivity error:");
console.error(err);
return false;
}
};
export const getUserDisplayName = async (client: WrappedDropboxClient) => {
await client.init();
const acct = await client.dropbox.usersGetCurrentAccount();
return acct.result.name.display_name;
};
export const revokeAuth = async (client: WrappedDropboxClient) => {
await client.init();
await client.dropbox.authTokenRevoke();
};
+39 -13
View File
@@ -14,7 +14,9 @@ import {
HeadBucketCommand, HeadBucketCommand,
ListObjectsV2CommandInput, ListObjectsV2CommandInput,
ListObjectsV2CommandOutput, ListObjectsV2CommandOutput,
HeadObjectCommandOutput,
} from "@aws-sdk/client-s3"; } from "@aws-sdk/client-s3";
export { S3Client } from "@aws-sdk/client-s3";
import type { _Object } from "@aws-sdk/client-s3"; import type { _Object } from "@aws-sdk/client-s3";
@@ -24,6 +26,8 @@ import {
mkdirpInVault, mkdirpInVault,
} from "./misc"; } from "./misc";
import * as mime from "mime-types"; import * as mime from "mime-types";
import { RemoteItem } from "./baseTypes";
import { decryptArrayBuffer, encryptArrayBuffer } from "./encrypt"; import { decryptArrayBuffer, encryptArrayBuffer } from "./encrypt";
export interface S3Config { export interface S3Config {
@@ -44,6 +48,29 @@ export const DEFAULT_S3_CONFIG = {
export type S3ObjectType = _Object; export type S3ObjectType = _Object;
const fromS3ObjectToRemoteItem = (x: S3ObjectType) => {
return {
key: x.Key,
lastModified: x.LastModified.valueOf(),
size: x.Size,
remoteType: "s3",
etag: x.ETag,
} as RemoteItem;
};
const fromS3HeadObjectToRemoteItem = (
key: string,
x: HeadObjectCommandOutput
) => {
return {
key: key,
lastModified: x.LastModified.valueOf(),
size: x.ContentLength,
remoteType: "s3",
etag: x.ETag,
} as RemoteItem;
};
export const getS3Client = (s3Config: S3Config) => { export const getS3Client = (s3Config: S3Config) => {
let endpoint = s3Config.s3Endpoint; let endpoint = s3Config.s3Endpoint;
if (!(endpoint.startsWith("http://") || endpoint.startsWith("https://"))) { if (!(endpoint.startsWith("http://") || endpoint.startsWith("https://"))) {
@@ -65,12 +92,14 @@ export const getRemoteMeta = async (
s3Config: S3Config, s3Config: S3Config,
fileOrFolderPath: string fileOrFolderPath: string
) => { ) => {
return await s3Client.send( const res = await s3Client.send(
new HeadObjectCommand({ new HeadObjectCommand({
Bucket: s3Config.s3BucketName, Bucket: s3Config.s3BucketName,
Key: fileOrFolderPath, Key: fileOrFolderPath,
}) })
); );
return fromS3HeadObjectToRemoteItem(fileOrFolderPath, res);
}; };
export const uploadToRemote = async ( export const uploadToRemote = async (
@@ -181,10 +210,7 @@ export const listFromRemote = async (
// ensemble fake rsp // ensemble fake rsp
return { return {
"$.metadata": { Contents: contents.map((x) => fromS3ObjectToRemoteItem(x)),
httpStatusCode: 200,
},
Contents: contents,
}; };
}; };
@@ -199,12 +225,12 @@ const getObjectBodyToArrayBuffer = async (
b: Readable | ReadableStream | Blob b: Readable | ReadableStream | Blob
) => { ) => {
if (b instanceof Readable) { if (b instanceof Readable) {
return (await new Promise((resolve, reject) => {
const chunks: Uint8Array[] = []; const chunks: Uint8Array[] = [];
for await (let chunk of b) { b.on("data", (chunk) => chunks.push(chunk));
chunks.push(chunk); b.on("error", reject);
} b.on("end", () => resolve(bufferToArrayBuffer(Buffer.concat(chunks))));
const buf = Buffer.concat(chunks); })) as ArrayBuffer;
return bufferToArrayBuffer(buf);
} else if (b instanceof ReadableStream) { } else if (b instanceof ReadableStream) {
return await new Response(b, {}).arrayBuffer(); return await new Response(b, {}).arrayBuffer();
} else if (b instanceof Blob) { } else if (b instanceof Blob) {
@@ -214,7 +240,7 @@ const getObjectBodyToArrayBuffer = async (
} }
}; };
export const downloadFromRemoteRaw = async ( const downloadFromRemoteRaw = async (
s3Client: S3Client, s3Client: S3Client,
s3Config: S3Config, s3Config: S3Config,
fileOrFolderPath: string fileOrFolderPath: string
@@ -302,7 +328,7 @@ export const deleteFromRemote = async (
await s3Client.send( await s3Client.send(
new DeleteObjectCommand({ new DeleteObjectCommand({
Bucket: s3Config.s3BucketName, Bucket: s3Config.s3BucketName,
Key: element.Key, Key: element.key,
}) })
); );
}); });
@@ -320,7 +346,7 @@ export const deleteFromRemote = async (
* @param s3Config * @param s3Config
* @returns * @returns
*/ */
export const checkS3Connectivity = async ( export const checkConnectivity = async (
s3Client: S3Client, s3Client: S3Client,
s3Config: S3Config s3Config: S3Config
) => { ) => {
+91 -72
View File
@@ -1,27 +1,15 @@
import { TAbstractFile, TFolder, TFile, Vault } from "obsidian"; import { TAbstractFile, TFolder, TFile, Vault } from "obsidian";
import { S3Client } from "@aws-sdk/client-s3";
import { import {
clearDeleteRenameHistoryOfKey, clearDeleteRenameHistoryOfKey,
upsertSyncMetaMappingDataS3, upsertSyncMetaMappingData,
getSyncMetaMappingByRemoteKeyS3, getSyncMetaMappingByRemoteKey,
} from "./localdb"; } from "./localdb";
import type { FileFolderHistoryRecord, InternalDBs } from "./localdb"; import type { FileFolderHistoryRecord, InternalDBs } from "./localdb";
import { import { RemoteClient } from "./remote";
S3Config, import type { SUPPORTED_SERVICES_TYPE, RemoteItem } from "./baseTypes";
S3ObjectType, import { mkdirpInVault, isHiddenPath, isVaildText, setToString } from "./misc";
uploadToRemote,
deleteFromRemote,
downloadFromRemote,
} from "./s3";
import {
mkdirpInVault,
SUPPORTED_SERVICES_TYPE,
isHiddenPath,
isVaildText,
} from "./misc";
import { import {
decryptBase32ToString, decryptBase32ToString,
encryptStringToBase32, encryptStringToBase32,
@@ -85,7 +73,7 @@ export interface PasswordCheckType {
} }
export const isPasswordOk = async ( export const isPasswordOk = async (
remote: S3ObjectType[], remote: RemoteItem[],
password: string = "" password: string = ""
) => { ) => {
if (remote === undefined || remote.length === 0) { if (remote === undefined || remote.length === 0) {
@@ -95,7 +83,7 @@ export const isPasswordOk = async (
reason: "empty_remote", reason: "empty_remote",
} as PasswordCheckType; } as PasswordCheckType;
} }
const santyCheckKey = remote[0].Key; const santyCheckKey = remote[0].key;
if (santyCheckKey.startsWith(MAGIC_ENCRYPTED_PREFIX_BASE32)) { if (santyCheckKey.startsWith(MAGIC_ENCRYPTED_PREFIX_BASE32)) {
// this is encrypted! // this is encrypted!
// try to decrypt it using the provided password. // try to decrypt it using the provided password.
@@ -143,26 +131,28 @@ export const isPasswordOk = async (
}; };
const ensembleMixedStates = async ( const ensembleMixedStates = async (
remote: S3ObjectType[], remote: RemoteItem[],
local: TAbstractFile[], local: TAbstractFile[],
deleteHistory: FileFolderHistoryRecord[], deleteHistory: FileFolderHistoryRecord[],
db: InternalDBs, db: InternalDBs,
remoteType: SUPPORTED_SERVICES_TYPE,
password: string = "" password: string = ""
) => { ) => {
const results = {} as Record<string, FileOrFolderMixedState>; const results = {} as Record<string, FileOrFolderMixedState>;
if (remote !== undefined) { if (remote !== undefined) {
for (const entry of remote) { for (const entry of remote) {
const remoteEncryptedKey = entry.Key; const remoteEncryptedKey = entry.key;
let key = remoteEncryptedKey; let key = remoteEncryptedKey;
if (password !== "") { if (password !== "") {
key = await decryptBase32ToString(remoteEncryptedKey, password); key = await decryptBase32ToString(remoteEncryptedKey, password);
} }
const backwardMapping = await getSyncMetaMappingByRemoteKeyS3( const backwardMapping = await getSyncMetaMappingByRemoteKey(
remoteType,
db, db,
key, key,
entry.LastModified.valueOf(), entry.lastModified,
entry.ETag entry.etag
); );
let r = {} as FileOrFolderMixedState; let r = {} as FileOrFolderMixedState;
@@ -171,16 +161,16 @@ const ensembleMixedStates = async (
r = { r = {
key: key, key: key,
exist_remote: true, exist_remote: true,
mtime_remote: backwardMapping.localMtime, mtime_remote: backwardMapping.localMtime || entry.lastModified,
size_remote: backwardMapping.localSize, size_remote: backwardMapping.localSize || entry.size,
remote_encrypted_key: remoteEncryptedKey, remote_encrypted_key: remoteEncryptedKey,
}; };
} else { } else {
r = { r = {
key: key, key: key,
exist_remote: true, exist_remote: true,
mtime_remote: entry.LastModified.valueOf(), mtime_remote: entry.lastModified,
size_remote: entry.Size, size_remote: entry.size,
remote_encrypted_key: remoteEncryptedKey, remote_encrypted_key: remoteEncryptedKey,
}; };
} }
@@ -398,14 +388,19 @@ const getOperation = (
r.decision_branch = 11; r.decision_branch = 11;
} }
if (r.decision === "unknown") {
throw Error(`unknown decision for ${JSON.stringify(r)}`);
}
return r; return r;
}; };
export const getSyncPlan = async ( export const getSyncPlan = async (
remote: S3ObjectType[], remote: RemoteItem[],
local: TAbstractFile[], local: TAbstractFile[],
deleteHistory: FileFolderHistoryRecord[], deleteHistory: FileFolderHistoryRecord[],
db: InternalDBs, db: InternalDBs,
remoteType: SUPPORTED_SERVICES_TYPE,
password: string = "" password: string = ""
) => { ) => {
const mixedStates = await ensembleMixedStates( const mixedStates = await ensembleMixedStates(
@@ -413,6 +408,7 @@ export const getSyncPlan = async (
local, local,
deleteHistory, deleteHistory,
db, db,
remoteType,
password password
); );
for (const [key, val] of Object.entries(mixedStates)) { for (const [key, val] of Object.entries(mixedStates)) {
@@ -420,27 +416,21 @@ export const getSyncPlan = async (
} }
const plan = { const plan = {
ts: Date.now(), ts: Date.now(),
remoteType: "s3", remoteType: remoteType,
mixedStates: mixedStates, mixedStates: mixedStates,
} as SyncPlanType; } as SyncPlanType;
return plan; return plan;
}; };
export const doActualSync = async ( const dispatchOperationToActual = async (
s3Client: S3Client, key: string,
s3Config: S3Config, state: FileOrFolderMixedState,
client: RemoteClient,
db: InternalDBs, db: InternalDBs,
vault: Vault, vault: Vault,
syncPlan: SyncPlanType, password: string = "",
password: string = "" foldersCreatedBefore: Set<string> | undefined = undefined
) => { ) => {
const keyStates = syncPlan.mixedStates;
await Promise.all(
Object.entries(keyStates)
.sort((k, v) => -(k as string).length)
.map(async ([k, v]) => {
const key = k as string;
const state = v as FileOrFolderMixedState;
let remoteEncryptedKey = key; let remoteEncryptedKey = key;
if (password !== "") { if (password !== "") {
remoteEncryptedKey = state.remote_encrypted_key; remoteEncryptedKey = state.remote_encrypted_key;
@@ -458,9 +448,7 @@ export const doActualSync = async (
} else if (state.decision === "skip") { } else if (state.decision === "skip") {
// do nothing // do nothing
} else if (state.decision === "download_clearhist") { } else if (state.decision === "download_clearhist") {
await downloadFromRemote( await client.downloadFromRemote(
s3Client,
s3Config,
state.key, state.key,
vault, vault,
state.mtime_remote, state.mtime_remote,
@@ -469,31 +457,29 @@ export const doActualSync = async (
); );
await clearDeleteRenameHistoryOfKey(db, state.key); await clearDeleteRenameHistoryOfKey(db, state.key);
} else if (state.decision === "upload_clearhist") { } else if (state.decision === "upload_clearhist") {
const remoteObjMeta = await uploadToRemote( const remoteObjMeta = await client.uploadToRemote(
s3Client,
s3Config,
state.key, state.key,
vault, vault,
false, false,
password, password,
remoteEncryptedKey remoteEncryptedKey,
foldersCreatedBefore
); );
await upsertSyncMetaMappingDataS3( await upsertSyncMetaMappingData(
client.serviceType,
db, db,
state.key, state.key,
state.mtime_local, state.mtime_local,
state.size_local, state.size_local,
state.key, state.key,
remoteObjMeta.LastModified.valueOf(), remoteObjMeta.lastModified,
remoteObjMeta.ContentLength, remoteObjMeta.size,
remoteObjMeta.ETag remoteObjMeta.etag
); );
await clearDeleteRenameHistoryOfKey(db, state.key); await clearDeleteRenameHistoryOfKey(db, state.key);
} else if (state.decision === "download") { } else if (state.decision === "download") {
await mkdirpInVault(state.key, vault); await mkdirpInVault(state.key, vault);
await downloadFromRemote( await client.downloadFromRemote(
s3Client,
s3Config,
state.key, state.key,
vault, vault,
state.mtime_remote, state.mtime_remote,
@@ -501,39 +487,72 @@ export const doActualSync = async (
remoteEncryptedKey remoteEncryptedKey
); );
} else if (state.decision === "delremote_clearhist") { } else if (state.decision === "delremote_clearhist") {
await deleteFromRemote( await client.deleteFromRemote(state.key, password, remoteEncryptedKey);
s3Client,
s3Config,
state.key,
password,
remoteEncryptedKey
);
await clearDeleteRenameHistoryOfKey(db, state.key); await clearDeleteRenameHistoryOfKey(db, state.key);
} else if (state.decision === "upload") { } else if (state.decision === "upload") {
const remoteObjMeta = await uploadToRemote( const remoteObjMeta = await client.uploadToRemote(
s3Client,
s3Config,
state.key, state.key,
vault, vault,
false, false,
password, password,
remoteEncryptedKey remoteEncryptedKey,
foldersCreatedBefore
); );
await upsertSyncMetaMappingDataS3( await upsertSyncMetaMappingData(
client.serviceType,
db, db,
state.key, state.key,
state.mtime_local, state.mtime_local,
state.size_local, state.size_local,
state.key, state.key,
remoteObjMeta.LastModified.valueOf(), remoteObjMeta.lastModified,
remoteObjMeta.ContentLength, remoteObjMeta.size,
remoteObjMeta.ETag remoteObjMeta.etag
); );
} else if (state.decision === "clearhist") { } else if (state.decision === "clearhist") {
await clearDeleteRenameHistoryOfKey(db, state.key); await clearDeleteRenameHistoryOfKey(db, state.key);
} else { } else {
throw Error("this should never happen!"); throw Error("this should never happen!");
} }
}) };
export const doActualSync = async (
client: RemoteClient,
db: InternalDBs,
vault: Vault,
syncPlan: SyncPlanType,
password: string = ""
) => {
const keyStates = syncPlan.mixedStates;
const foldersCreatedBefore = new Set<string>();
for (const [k, v] of Object.entries(keyStates).sort(
([k1, v1], [k2, v2]) => k2.length - k1.length
)) {
const k2 = k as string;
const v2 = v as FileOrFolderMixedState;
await dispatchOperationToActual(
k as string,
v as FileOrFolderMixedState,
client,
db,
vault,
password,
foldersCreatedBefore
); );
// console.log(`finished ${k}, with ${setToString(foldersCreatedBefore)}`);
}
// await Promise.all(
// Object.entries(keyStates)
// .map(async ([k, v]) =>
// dispatchOperationToActual(
// k as string,
// v as FileOrFolderMixedState,
// client,
// db,
// vault,
// password,
// foldersCreatedBefore
// )
// )
// );
}; };
+246
View File
@@ -0,0 +1,246 @@
import { Buffer } from "buffer";
import { FileStats, Vault } from "obsidian";
import { AuthType, BufferLike, createClient } from "webdav/web";
import type { WebDAVClient, ResponseDataDetailed, FileStat } from "webdav/web";
export type { WebDAVClient } from "webdav/web";
import type { RemoteItem } from "./baseTypes";
import {
arrayBufferToBuffer,
bufferToArrayBuffer,
mkdirpInVault,
getPathFolder,
} from "./misc";
import { decryptArrayBuffer, encryptArrayBuffer } from "./encrypt";
export type WebdavAuthType = "digest" | "basic";
export interface WebdavConfig {
address: string;
username: string;
password: string;
authType: WebdavAuthType;
}
export const DEFAULT_WEBDAV_CONFIG = {
address: "",
username: "",
password: "",
authType: "basic",
} as WebdavConfig;
const getWebdavPath = (fileOrFolderPath: string) => {
if (!fileOrFolderPath.startsWith("/")) {
return `/${fileOrFolderPath}`;
}
return fileOrFolderPath;
};
const getNormPath = (fileOrFolderPath: string) => {
if (fileOrFolderPath.startsWith("/")) {
return fileOrFolderPath.slice(1);
}
return fileOrFolderPath;
};
const fromWebdavItemToRemoteItem = (x: FileStat) => {
let key = getNormPath(x.filename);
if (x.type === "directory" && !key.endsWith("/")) {
key = `${key}/`;
}
return {
key: key,
lastModified: Date.parse(x.lastmod).valueOf(),
size: x.size,
remoteType: "webdav",
etag: x.etag || undefined,
} as RemoteItem;
};
export const getWebdavClient = (webdavConfig: WebdavConfig) => {
if (webdavConfig.username !== "" && webdavConfig.password !== "") {
return createClient(webdavConfig.address, {
username: webdavConfig.username,
password: webdavConfig.password,
authType:
webdavConfig.authType === "digest"
? AuthType.Digest
: AuthType.Password,
});
} else {
console.log("no password");
return createClient(webdavConfig.address);
}
};
export const getRemoteMeta = async (
client: WebDAVClient,
fileOrFolderPath: string
) => {
const res = (await client.stat(getWebdavPath(fileOrFolderPath), {
details: false,
})) as FileStat;
return fromWebdavItemToRemoteItem(res);
};
export const uploadToRemote = async (
client: WebDAVClient,
fileOrFolderPath: string,
vault: Vault,
isRecursively: boolean = false,
password: string = "",
remoteEncryptedKey: string = ""
) => {
let uploadFile = fileOrFolderPath;
if (password !== "") {
uploadFile = remoteEncryptedKey;
}
uploadFile = getWebdavPath(uploadFile);
const isFolder = fileOrFolderPath.endsWith("/");
if (isFolder && isRecursively) {
throw Error("upload function doesn't implement recursive function yet!");
} else if (isFolder && !isRecursively) {
// folder
if (password === "") {
// if not encrypted, mkdir a remote folder
await client.createDirectory(uploadFile, {
recursive: true,
});
const res = await getRemoteMeta(client, uploadFile);
return res;
} else {
// if encrypted, upload a fake file with the encrypted file name
await client.putFileContents(uploadFile, "", {
overwrite: true,
onUploadProgress: (progress) => {
console.log(`Uploaded ${progress.loaded} bytes of ${progress.total}`);
},
});
return await getRemoteMeta(client, uploadFile);
}
} else {
// file
// we ignore isRecursively parameter here
const localContent = await vault.adapter.readBinary(fileOrFolderPath);
let remoteContent = localContent;
if (password !== "") {
remoteContent = await encryptArrayBuffer(localContent, password);
}
// we need to create folders before uploading
const dir = getPathFolder(uploadFile);
if (dir !== "/" && dir !== "") {
await client.createDirectory(dir, { recursive: true });
}
await client.putFileContents(uploadFile, remoteContent, {
overwrite: true,
onUploadProgress: (progress) => {
console.log(`Uploaded ${progress.loaded} bytes of ${progress.total}`);
},
});
return await getRemoteMeta(client, uploadFile);
}
};
export const listFromRemote = async (client: WebDAVClient, prefix?: string) => {
if (prefix !== undefined) {
throw Error("prefix not supported");
}
const contents = (await client.getDirectoryContents("/", {
deep: true,
details: false /* no need for verbose details here */,
glob: "/**" /* avoid dot files by using glob */,
})) as FileStat[];
return {
Contents: contents.map((x) => fromWebdavItemToRemoteItem(x)),
};
};
const downloadFromRemoteRaw = async (
client: WebDAVClient,
fileOrFolderPath: string
) => {
const buff = (await client.getFileContents(
getWebdavPath(fileOrFolderPath)
)) as BufferLike;
if (buff instanceof ArrayBuffer) {
return buff;
} else if (buff instanceof Buffer) {
return bufferToArrayBuffer(buff);
}
throw Error(`unexpected file content result with type ${typeof buff}`);
};
export const downloadFromRemote = async (
client: WebDAVClient,
fileOrFolderPath: string,
vault: Vault,
mtime: number,
password: string = "",
remoteEncryptedKey: string = ""
) => {
const isFolder = fileOrFolderPath.endsWith("/");
await mkdirpInVault(fileOrFolderPath, vault);
// the file is always local file
// we need to encrypt it
if (isFolder) {
// mkdirp locally is enough
// do nothing here
} else {
let downloadFile = fileOrFolderPath;
if (password !== "") {
downloadFile = remoteEncryptedKey;
}
downloadFile = getWebdavPath(downloadFile);
const remoteContent = await downloadFromRemoteRaw(client, downloadFile);
let localContent = remoteContent;
if (password !== "") {
localContent = await decryptArrayBuffer(remoteContent, password);
}
await vault.adapter.writeBinary(fileOrFolderPath, localContent, {
mtime: mtime,
});
}
};
export const deleteFromRemote = async (
client: WebDAVClient,
fileOrFolderPath: string,
password: string = "",
remoteEncryptedKey: string = ""
) => {
if (fileOrFolderPath === "/") {
return;
}
let remoteFileName = fileOrFolderPath;
if (password !== "") {
remoteFileName = remoteEncryptedKey;
}
remoteFileName = getWebdavPath(remoteFileName);
try {
await client.deleteFile(remoteFileName);
console.log(`delete ${remoteFileName} succeeded`);
} catch (err) {
console.error("some error while deleting");
console.log(err);
}
};
export const checkConnectivity = async (client: WebDAVClient) => {
try {
const results = await getRemoteMeta(client, "/");
if (results === undefined) {
return false;
}
return true;
} catch (err) {
return false;
}
};
+25
View File
@@ -7,3 +7,28 @@
.s3-disclaimer { .s3-disclaimer {
font-weight: bold; font-weight: bold;
} }
.s3-hide {
display: none;
}
.dropbox-disclaimer {
font-weight: bold;
}
.dropbox-hide {
display: none;
}
.dropbox-auth-button-hide {
display: none;
}
.dropbox-revoke-auth-button-hide {
display: none;
}
.webdav-disclaimer {
font-weight: bold;
}
.webdav-hide {
display: none;
}
+44
View File
@@ -69,6 +69,28 @@ describe("Misc: get folder levels", () => {
const res3 = ["xxx", "xxx/yyy", "xxx/yyy/zzz"]; const res3 = ["xxx", "xxx/yyy", "xxx/yyy/zzz"];
expect(misc.getFolderLevels(item3)).to.deep.equal(res3); expect(misc.getFolderLevels(item3)).to.deep.equal(res3);
}); });
it("should treat path starting with / correctly", () => {
const item = "/xxx/yyy/zzz.md";
const res = ["/xxx", "/xxx/yyy"];
expect(misc.getFolderLevels(item)).to.deep.equal(res);
const item2 = "/xxx/yyy/zzz";
const res2 = ["/xxx", "/xxx/yyy"];
expect(misc.getFolderLevels(item2)).to.deep.equal(res2);
const item3 = "/xxx/yyy/zzz/";
const res3 = ["/xxx", "/xxx/yyy", "/xxx/yyy/zzz"];
expect(misc.getFolderLevels(item3)).to.deep.equal(res3);
const item4 = "/xxx";
const res4 = [] as string[];
expect(misc.getFolderLevels(item4)).to.deep.equal(res4);
const item5 = "/";
const res5 = [] as string[];
expect(misc.getFolderLevels(item5)).to.deep.equal(res5);
});
}); });
describe("Misc: vaild file name tests", () => { describe("Misc: vaild file name tests", () => {
@@ -89,3 +111,25 @@ describe("Misc: vaild file name tests", () => {
expect(x).to.be.true; expect(x).to.be.true;
}); });
}); });
describe("Misc: get dirname", () => {
it("should return itself for folder", async () => {
const x = misc.getPathFolder("ssss/");
// console.log(x)
expect(x).to.equal("ssss/");
});
it("should return folder for file", async () => {
const x = misc.getPathFolder("sss/yyy");
// console.log(x)
expect(x).to.equal("sss/");
});
it("should treat / specially", async () => {
const x = misc.getPathFolder("/");
expect(x).to.equal("/");
const y = misc.getPathFolder("/abc");
expect(y).to.equal("/");
});
});
+1
View File
@@ -11,6 +11,7 @@
// "allowSyntheticDefaultImports": true, // "allowSyntheticDefaultImports": true,
"esModuleInterop": true, "esModuleInterop": true,
"importHelpers": true, "importHelpers": true,
"isolatedModules": true,
"lib": ["dom", "es5", "scripthost", "es2015"] "lib": ["dom", "es5", "scripthost", "es2015"]
}, },
"include": ["**/*.ts"] "include": ["**/*.ts"]
+1 -1
View File
@@ -1,3 +1,3 @@
{ {
"0.1.3": "0.12.15" "0.2.0": "0.12.15"
} }
+7
View File
@@ -1,7 +1,10 @@
require("dotenv").config();
const path = require("path"); const path = require("path");
const webpack = require("webpack"); const webpack = require("webpack");
const TerserPlugin = require("terser-webpack-plugin"); const TerserPlugin = require("terser-webpack-plugin");
const DEFAULT_DROPBOX_APP_KEY = process.env.DROPBOX_APP_KEY || "";
module.exports = { module.exports = {
entry: "./src/main.ts", entry: "./src/main.ts",
target: "web", target: "web",
@@ -11,6 +14,9 @@ module.exports = {
libraryTarget: "commonjs", libraryTarget: "commonjs",
}, },
plugins: [ plugins: [
new webpack.DefinePlugin({
"process.env.DEFAULT_DROPBOX_APP_KEY": `"${DEFAULT_DROPBOX_APP_KEY}"`,
}),
// Work around for Buffer is undefined: // Work around for Buffer is undefined:
// https://github.com/webpack/changelog-v5/issues/10 // https://github.com/webpack/changelog-v5/issues/10
new webpack.ProvidePlugin({ new webpack.ProvidePlugin({
@@ -38,6 +44,7 @@ module.exports = {
// console: require.resolve("console-browserify"), // console: require.resolve("console-browserify"),
// constants: require.resolve("constants-browserify"), // constants: require.resolve("constants-browserify"),
// crypto: require.resolve("crypto-browserify"), // crypto: require.resolve("crypto-browserify"),
crypto: false,
// domain: require.resolve("domain-browser"), // domain: require.resolve("domain-browser"),
// events: require.resolve("events"), // events: require.resolve("events"),
// http: require.resolve("stream-http"), // http: require.resolve("stream-http"),