Compare commits

..
25 Commits
Author SHA1 Message Date
fyears 9ea7c8e858 bump to 0.4.19
Release A New Version / build (16.x) (push) Failing after 33s
2024-04-28 00:31:14 +08:00
fyears 61a3fab219 webdis 2024-04-27 23:10:36 +08:00
fyears 5340e38eac format of doc 2024-04-27 17:11:59 +08:00
fyears 8d5868b8d8 fix process counting 2024-04-27 17:11:48 +08:00
fyears 55175a6d06 bump to 0.4.18
Release A New Version / build (16.x) (push) Failing after 37s
2024-04-27 12:34:10 +08:00
fyears f60bd25490 add enc for upyun 2024-04-27 12:21:31 +08:00
fyears 1d8463a3ed add upyun
Release A New Version / build (16.x) (push) Failing after 49s
2024-04-27 12:13:09 +08:00
fyears 9fe1b1d5e6 clean both folder if empty 2024-04-27 12:03:36 +08:00
fyears df7b6e1848 allowing s3 synth folder 2024-04-27 12:01:30 +08:00
fyears a9126e5947 fix and optimize tests 2024-04-27 03:28:39 +08:00
fyears 3bb7355db3 update package 2024-04-27 02:35:38 +08:00
fyears f33fa26c03 a large semi-rewrite of fs logic 2024-04-27 02:27:24 +08:00
Kira Kawaiandras0q 5ce350ba41 💄 setings: overflow-wrap: break-word (#597)
Co-authored-by: ras0q <ras0q@users.noreply.github.com>
2024-04-26 23:23:34 +08:00
lyiton 1d32f1242d Update zh_cn.json (#598) 2024-04-19 23:33:03 +08:00
fyears d8ab054da1 change css on mobile to make multiple buttons happy 2024-04-05 11:53:44 +08:00
fyears b877228415 revert back the webdav path func 2024-04-05 11:37:03 +08:00
fyears 28b99557a8 clean up reverse proxy 2024-04-05 11:06:16 +08:00
fyears ae28cf9183 format 2024-04-05 10:48:53 +08:00
Yesterday17andfyears 8f68ac4ded handle relative path correctly (#226)
Co-authored-by: fyears <1142836+fyears@users.noreply.github.com>
2024-04-05 10:45:23 +08:00
220fd07a8b feat: add reserve proxy url (#479)
Co-authored-by: Adens <dwang@senparc.com>
Co-authored-by: fyears <1142836+fyears@users.noreply.github.com>
2024-04-05 10:36:58 +08:00
fyears fe572b41c9 bump to 0.4.16
Release A New Version / build (16.x) (push) Failing after 40s
2024-04-05 00:23:56 +08:00
fyears ca5ca5f576 fix the time mapping 2024-04-05 00:23:05 +08:00
fyears 0e6182b887 bulk load db and remove some verbose output, to speed up 2024-04-04 23:37:09 +08:00
fyears c5a7085fda format 2024-04-04 22:20:44 +08:00
fyears 640f1e56f1 save the writing 2024-04-04 22:20:36 +08:00
43 changed files with 4370 additions and 3994 deletions
+8
View File
@@ -66,6 +66,7 @@ Additionally, the plugin author may occasionally visit Obsidian official forum a
- [Storj](./docs/remote_services/s3_storj_io/README.md)
- [腾讯云 COS](./docs/remote_services/s3_tencent_cloud_cos/README.zh-cn.md) | [Tencent Cloud COS](./docs/remote_services/s3_tencent_cloud_cos/README.md)
- [MinIO](./docs/remote_services/s3_minio/README.md)
- [又拍云](./docs/remote_services/s3_upyun/README.zh-cn.md)
- Prepare your S3 (-compatible) service information: [endpoint, region](https://docs.aws.amazon.com/general/latest/gr/s3.html), [access key id, secret access key](https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/getting-your-credentials.html), bucket name. The bucket should be empty and solely for syncing a vault.
- If you are using AWS S3, create [policy and user](./docs/remote_services/s3_general/s3_user_policy.md).
- Very old version of Obsidian needs [configuring CORS](./docs/remote_services/s3_general/s3_cors_configure.md).
@@ -107,6 +108,13 @@ Additionally, the plugin author may occasionally visit Obsidian official forum a
- Password-based end-to-end encryption is also supported. But please be aware that **the vault name itself is not encrypted**.
- If you want to sync the files across multiple devices, **your vault name should be the same** while using default settings.
### Webdis
- Tutorials:
- [Webdis](./docs/remote_services/webdis/README.md)
- Mostly experimental.
- You have to setup and protect your web server by yourself.
## Scheduled Auto Sync
- You can configure auto syncing every N minutes in settings.
@@ -0,0 +1,24 @@
# 又拍云
## 链接
- 官网 <https://www.upyun.com/>
- 官网的 S3 文档 <https://help.upyun.com/knowledge-base/aws-s3%e5%85%bc%e5%ae%b9/>
## 注意!!!!!
又拍云似乎(?)文件都是默认公开的,强烈建议注意隐私问题,强烈建议设置插件加密。
## 步骤
1. 注册,新建对象存储。
2. 参考官网文档 <https://help.upyun.com/knowledge-base/aws-s3%e5%85%bc%e5%ae%b9/>,创建操作员然后创建 S3 访问凭证。
3. 在 Remotely Save 设置以下:
- 服务地址(Endpoint):`s3.api.upyun.com` **一定是这个域名**
- 区域(Region):`us-east-1`
- Acccess Key ID:您获取到的访问凭证的 AccessKey
- Secret Access Key:您获取到的访问凭证的 SecretAccessKey
- 存储桶(Bucket)的名字:您创建的“服务名”
- 是否生成文件夹 Object:不生成(默认) **一定要选择不生成**
4. 可以在插件设置里,加上密码。
5. 同步。
+35
View File
@@ -0,0 +1,35 @@
# Webdis
## Links
- Webdis: <https://github.com/nicolasff/webdis>
- Redis®: <https://redis.io/>
## Explanation and Background
I like the Redis® software very much, and would like to experiment using it as a "file storage". It seems to be nature by using path as the key and the content as the value (Sort of..., see below).
However, Redis® works by using TCP connections, and browser js cannot establish raw TCP connections. We need a HTTP gateway, to provide the HTTP api. Wedis seems to be the most famous open source one.
And of course, this method should work for Redis® alternatives: Valkey, Redict, KeyDB, Dragonfly, Garnet, ...
## Disclaimer
This app is NOT an official Redis® Ltd / Redis® OSS / Webdis product. Redis is a registered trademark of Redis Ltd.
**Never expose your Redis® or Webdis to public without security protection!!!** You are response for protecting your server.
## Usage
1. Install Redis®.
2. Install Webdis.
3. In `webdis.json`, configure the ACL for using password and username, and / or ip filters. **Never expose your Redis® or Webdis to public without security protection!!!**.
4. Install and configure reverse proxy, firewall, https, etc. (You have to configure HTTPS correctly if you want to use it on iOS)
5. In Remotely Save settings, enter your server address, username, password, and adjust the "base dir". Check connection.
6. Sync!
7. Serveral keys and values will be generated in your Redis® database:
```
rs:fs:v1:${encodeURIComponent(vaultName+'/'+folderStructure+'/'+fileName)}:meta # you can HGETALL it
rs:fs:v1:${encodeURIComponent(vaultName+'/'+folderStructure+'/'+fileName)}:content # you can GET it
```
+1
View File
@@ -36,6 +36,7 @@ esbuild
"net",
"http",
"https",
"vm",
// ...builtins
],
inject: ["./esbuild.injecthelper.mjs"],
+1 -1
View File
@@ -1,7 +1,7 @@
{
"id": "remotely-save",
"name": "Remotely Save",
"version": "0.4.15",
"version": "0.4.19",
"minAppVersion": "0.13.21",
"description": "Yet another unofficial plugin allowing users to synchronize notes between local device and the cloud service.",
"author": "fyears",
+1 -1
View File
@@ -1,7 +1,7 @@
{
"id": "remotely-save",
"name": "Remotely Save",
"version": "0.4.15",
"version": "0.4.19",
"minAppVersion": "0.13.21",
"description": "Yet another unofficial plugin allowing users to synchronize notes between local device and the cloud service.",
"author": "fyears",
+28 -27
View File
@@ -1,6 +1,6 @@
{
"name": "remotely-save",
"version": "0.4.15",
"version": "0.4.19",
"description": "This is yet another sync plugin for Obsidian app.",
"scripts": {
"dev2": "node esbuild.config.mjs --watch",
@@ -16,7 +16,9 @@
"process": "process/browser",
"stream": "stream-browserify",
"crypto": "crypto-browserify",
"url": "url/"
"url": "url/",
"fs": false,
"vm": false
},
"source": "main.ts",
"keywords": [],
@@ -27,60 +29,59 @@
"@types/chai": "^4.3.14",
"@types/chai-as-promised": "^7.1.8",
"@types/jsdom": "^21.1.6",
"@types/lodash": "^4.14.202",
"@types/lodash": "^4.17.0",
"@types/mime-types": "^2.1.4",
"@types/mocha": "^10.0.6",
"@types/mustache": "^4.2.5",
"@types/node": "^20.10.4",
"@types/node": "^20.12.7",
"@types/qrcode": "^1.5.5",
"builtin-modules": "^3.3.0",
"chai": "^4.4.1",
"chai-as-promised": "^7.1.1",
"cross-env": "^7.0.3",
"dotenv": "^16.3.1",
"esbuild": "^0.19.9",
"dotenv": "^16.4.5",
"esbuild": "^0.20.2",
"esbuild-plugin-inline-worker": "^0.1.1",
"jsdom": "^23.0.1",
"jsdom": "^24.0.0",
"mocha": "^10.4.0",
"npm-check-updates": "^16.14.12",
"obsidian": "^1.4.11",
"prettier": "^3.1.1",
"npm-check-updates": "^16.14.20",
"obsidian": "^1.5.7",
"prettier": "^3.2.5",
"ts-loader": "^9.5.1",
"ts-node": "^10.9.2",
"tslib": "^2.6.2",
"typescript": "^5.3.3",
"typescript": "^5.4.5",
"webdav-server": "^2.6.2",
"webpack": "^5.89.0",
"webpack": "^5.91.0",
"webpack-cli": "^5.1.4",
"worker-loader": "^3.0.8"
},
"dependencies": {
"@aws-sdk/client-s3": "^3.474.0",
"@aws-sdk/lib-storage": "^3.474.0",
"@aws-sdk/signature-v4-crt": "^3.474.0",
"@aws-sdk/types": "^3.468.0",
"@azure/msal-node": "^2.6.0",
"@aws-sdk/client-s3": "^3.563.0",
"@aws-sdk/lib-storage": "^3.563.0",
"@aws-sdk/signature-v4-crt": "^3.556.0",
"@aws-sdk/types": "^3.535.0",
"@azure/msal-node": "^2.7.0",
"@fyears/rclone-crypt": "^0.0.7",
"@fyears/tsqueue": "^1.0.1",
"@microsoft/microsoft-graph-client": "^3.0.7",
"@smithy/fetch-http-handler": "^2.3.1",
"@smithy/protocol-http": "^3.0.11",
"@smithy/querystring-builder": "^2.0.15",
"acorn": "^8.11.2",
"@smithy/fetch-http-handler": "^2.5.0",
"@smithy/protocol-http": "^3.3.0",
"@smithy/querystring-builder": "^2.2.0",
"acorn": "^8.11.3",
"aggregate-error": "^5.0.0",
"assert": "^2.1.0",
"aws-crt": "^1.20.0",
"aws-crt": "^1.21.2",
"buffer": "^6.0.3",
"crypto-browserify": "^3.12.0",
"dropbox": "^10.34.0",
"emoji-regex": "^10.3.0",
"http-status-codes": "^2.3.0",
"localforage": "^1.10.0",
"localforage-getitems": "^1.4.2",
"lodash": "^4.17.21",
"lucide": "^0.298.0",
"lucide": "^0.376.1",
"mime-types": "^2.1.35",
"mustache": "^4.2.0",
"nanoid": "^5.0.4",
"nanoid": "^5.0.7",
"p-queue": "^8.0.1",
"path-browserify": "^1.0.1",
"process": "^0.11.10",
@@ -90,7 +91,7 @@
"stream-browserify": "^3.0.0",
"url": "^0.11.3",
"util": "^0.12.5",
"webdav": "^5.3.1",
"webdav": "^5.6.0",
"xregexp": "^5.1.1"
}
}
+19 -12
View File
@@ -3,17 +3,22 @@
* To avoid circular dependency.
*/
import { Platform, requireApiVersion } from "obsidian";
import type { LangType, LangTypeAndAuto } from "./i18n";
export const DEFAULT_CONTENT_TYPE = "application/octet-stream";
export type SUPPORTED_SERVICES_TYPE = "s3" | "webdav" | "dropbox" | "onedrive";
export type SUPPORTED_SERVICES_TYPE =
| "s3"
| "webdav"
| "dropbox"
| "onedrive"
| "webdis";
export type SUPPORTED_SERVICES_TYPE_WITH_REMOTE_BASE_DIR =
| "webdav"
| "dropbox"
| "onedrive";
| "onedrive"
| "webdis";
export interface S3Config {
s3Endpoint: string;
@@ -27,6 +32,9 @@ export interface S3Config {
remotePrefix?: string;
useAccurateMTime?: boolean;
reverseProxyNoSignUrl?: string;
generateFolderObject?: boolean;
/**
* @deprecated
@@ -83,6 +91,13 @@ export interface OnedriveConfig {
remoteBaseDir?: string;
}
export interface WebdisConfig {
address: string;
username?: string;
password?: string;
remoteBaseDir?: string;
}
export type SyncDirectionType =
| "bidirectional"
| "incremental_pull_only"
@@ -97,6 +112,7 @@ export interface RemotelySavePluginSettings {
webdav: WebdavConfig;
dropbox: DropboxConfig;
onedrive: OnedriveConfig;
webdis: WebdisConfig;
password: string;
serviceType: SUPPORTED_SERVICES_TYPE;
currLogLevel?: string;
@@ -258,15 +274,6 @@ export interface FileOrFolderMixedState {
deltimeRemoteFmt?: string;
}
export const API_VER_STAT_FOLDER = "0.13.27";
export const API_VER_REQURL = "0.13.26"; // desktop ver 0.13.26, iOS ver 1.1.1
export const API_VER_REQURL_ANDROID = "0.14.6"; // Android ver 1.2.1
export const API_VER_ENSURE_REQURL_OK = "1.0.0"; // always bypass CORS here
export const VALID_REQURL =
(!Platform.isAndroidApp && requireApiVersion(API_VER_REQURL)) ||
(Platform.isAndroidApp && requireApiVersion(API_VER_REQURL_ANDROID));
export const DEFAULT_DEBUG_FOLDER = "_debug_remotely_save/";
export const DEFAULT_SYNC_PLANS_HISTORY_FILE_PREFIX =
"sync_plans_hist_exported_on_";
+14
View File
@@ -0,0 +1,14 @@
/**
* Every utils requiring Obsidian is placed here.
*/
import { Platform, requireApiVersion } from "obsidian";
export const API_VER_STAT_FOLDER = "0.13.27";
export const API_VER_REQURL = "0.13.26"; // desktop ver 0.13.26, iOS ver 1.1.1
export const API_VER_REQURL_ANDROID = "0.14.6"; // Android ver 1.2.1
export const API_VER_ENSURE_REQURL_OK = "1.0.0"; // always bypass CORS here
export const VALID_REQURL =
(!Platform.isAndroidApp && requireApiVersion(API_VER_REQURL)) ||
(Platform.isAndroidApp && requireApiVersion(API_VER_REQURL_ANDROID));
-215
View File
@@ -1,215 +0,0 @@
import { CipherMethodType } from "./baseTypes";
import * as openssl from "./encryptOpenSSL";
import * as rclone from "./encryptRClone";
import { isVaildText } from "./misc";
export class Cipher {
readonly password: string;
readonly method: CipherMethodType;
cipherRClone?: rclone.CipherRclone;
constructor(password: string, method: CipherMethodType) {
this.password = password ?? "";
this.method = method;
if (method === "rclone-base64") {
this.cipherRClone = new rclone.CipherRclone(password, 5);
}
}
closeResources() {
if (this.method === "rclone-base64" && this.cipherRClone !== undefined) {
this.cipherRClone.closeResources();
}
}
isPasswordEmpty() {
return this.password === "";
}
isFolderAware() {
if (this.method === "openssl-base64") {
return false;
}
if (this.method === "rclone-base64") {
return true;
}
throw Error(`no idea about isFolderAware for method=${this.method}`);
}
async encryptContent(content: ArrayBuffer) {
// console.debug("start encryptContent");
if (this.password === "") {
return content;
}
if (this.method === "openssl-base64") {
const res = await openssl.encryptArrayBuffer(content, this.password);
if (res === undefined) {
throw Error(`cannot encrypt content`);
}
return res;
} else if (this.method === "rclone-base64") {
const res =
await this.cipherRClone!.encryptContentByCallingWorker(content);
if (res === undefined) {
throw Error(`cannot encrypt content`);
}
return res;
} else {
throw Error(`not supported encrypt method=${this.method}`);
}
}
async decryptContent(content: ArrayBuffer) {
// console.debug("start decryptContent");
if (this.password === "") {
return content;
}
if (this.method === "openssl-base64") {
const res = await openssl.decryptArrayBuffer(content, this.password);
if (res === undefined) {
throw Error(`cannot decrypt content`);
}
return res;
} else if (this.method === "rclone-base64") {
const res =
await this.cipherRClone!.decryptContentByCallingWorker(content);
if (res === undefined) {
throw Error(`cannot decrypt content`);
}
return res;
} else {
throw Error(`not supported decrypt method=${this.method}`);
}
}
async encryptName(name: string) {
// console.debug("start encryptName");
if (this.password === "") {
return name;
}
if (this.method === "openssl-base64") {
const res = await openssl.encryptStringToBase64url(name, this.password);
if (res === undefined) {
throw Error(`cannot encrypt name=${name}`);
}
return res;
} else if (this.method === "rclone-base64") {
const res = await this.cipherRClone!.encryptNameByCallingWorker(name);
if (res === undefined) {
throw Error(`cannot encrypt name=${name}`);
}
return res;
} else {
throw Error(`not supported encrypt method=${this.method}`);
}
}
async decryptName(name: string): Promise<string> {
// console.debug("start decryptName");
if (this.password === "") {
return name;
}
if (this.method === "openssl-base64") {
if (name.startsWith(openssl.MAGIC_ENCRYPTED_PREFIX_BASE32)) {
// backward compitable with the openssl-base32
try {
const res = await openssl.decryptBase32ToString(name, this.password);
if (res !== undefined && isVaildText(res)) {
return res;
} else {
throw Error(`cannot decrypt name=${name}`);
}
} catch (error) {
throw Error(`cannot decrypt name=${name}`);
}
} else if (name.startsWith(openssl.MAGIC_ENCRYPTED_PREFIX_BASE64URL)) {
try {
const res = await openssl.decryptBase64urlToString(
name,
this.password
);
if (res !== undefined && isVaildText(res)) {
return res;
} else {
throw Error(`cannot decrypt name=${name}`);
}
} catch (error) {
throw Error(`cannot decrypt name=${name}`);
}
} else {
throw Error(
`method=${this.method} but the name=${name}, likely mismatch`
);
}
} else if (this.method === "rclone-base64") {
const res = await this.cipherRClone!.decryptNameByCallingWorker(name);
if (res === undefined) {
throw Error(`cannot decrypt name=${name}`);
}
return res;
} else {
throw Error(`not supported decrypt method=${this.method}`);
}
}
getSizeFromOrigToEnc(x: number) {
if (this.password === "") {
return x;
}
if (this.method === "openssl-base64") {
return openssl.getSizeFromOrigToEnc(x);
} else if (this.method === "rclone-base64") {
return rclone.getSizeFromOrigToEnc(x);
} else {
throw Error(`not supported encrypt method=${this.method}`);
}
}
/**
* quick guess, no actual decryption here
* @param name
* @returns
*/
static isLikelyOpenSSLEncryptedName(name: string): boolean {
if (
name.startsWith(openssl.MAGIC_ENCRYPTED_PREFIX_BASE32) ||
name.startsWith(openssl.MAGIC_ENCRYPTED_PREFIX_BASE64URL)
) {
return true;
}
return false;
}
/**
* quick guess, no actual decryption here
* @param name
* @returns
*/
static isLikelyEncryptedName(name: string): boolean {
return Cipher.isLikelyOpenSSLEncryptedName(name);
}
/**
* quick guess, no actual decryption here, only openssl can be guessed here
* @param name
* @returns
*/
static isLikelyEncryptedNameNotMatchMethod(
name: string,
method: CipherMethodType
): boolean {
if (
Cipher.isLikelyOpenSSLEncryptedName(name) &&
method !== "openssl-base64"
) {
return true;
}
if (
!Cipher.isLikelyOpenSSLEncryptedName(name) &&
method === "openssl-base64"
) {
return true;
}
return false;
}
}
+19
View File
@@ -0,0 +1,19 @@
import { Entity } from "./baseTypes";
export abstract class FakeFs {
abstract kind: string;
abstract walk(): Promise<Entity[]>;
abstract stat(key: string): Promise<Entity>;
abstract mkdir(key: string, mtime?: number, ctime?: number): Promise<Entity>;
abstract writeFile(
key: string,
content: ArrayBuffer,
mtime: number,
ctime: number
): Promise<Entity>;
abstract readFile(key: string): Promise<ArrayBuffer>;
abstract rm(key: string): Promise<void>;
abstract checkConnect(callbackFunc?: any): Promise<boolean>;
abstract getUserDisplayName(): Promise<string>;
abstract revokeAuth(): Promise<any>;
}
+308 -345
View File
@@ -1,25 +1,21 @@
import { FakeFs } from "./fsAll";
import { Dropbox, DropboxAuth } from "dropbox";
import type { files, DropboxResponseError, DropboxResponse } from "dropbox";
import { Vault } from "obsidian";
import * as path from "path";
import {
DropboxConfig,
Entity,
COMMAND_CALLBACK_DROPBOX,
OAUTH2_FORCE_EXPIRE_MILLISECONDS,
UploadedType,
Entity,
} from "./baseTypes";
import random from "lodash/random";
import {
bufferToArrayBuffer,
delay,
fixEntityListCasesInplace,
getFolderLevels,
getParentFolder,
hasEmojiInText,
headersToRecord,
mkdirpInVault,
} from "./misc";
import { Cipher } from "./encryptUnified";
import { random } from "lodash";
export { Dropbox } from "dropbox";
@@ -34,10 +30,7 @@ export const DEFAULT_DROPBOX_CONFIG: DropboxConfig = {
credentialsShouldBeDeletedAtTime: 0,
};
export const getDropboxPath = (
fileOrFolderPath: string,
remoteBaseDir: string
) => {
const getDropboxPath = (fileOrFolderPath: string, remoteBaseDir: string) => {
let key = fileOrFolderPath;
if (fileOrFolderPath === "/" || fileOrFolderPath === "") {
// special
@@ -84,20 +77,22 @@ const fromDropboxItemToEntity = (
if (x[".tag"] === "folder") {
return {
key: key,
keyRaw: key,
size: 0,
sizeRaw: 0,
etag: `${x.id}\t`,
} as Entity;
} else if (x[".tag"] === "file") {
const mtimeCli = Date.parse(x.client_modified).valueOf();
const mtimeSvr = Date.parse(x.server_modified).valueOf();
return {
key: key,
keyRaw: key,
mtimeCli: mtimeCli,
mtimeSvr: mtimeSvr,
size: x.size,
sizeRaw: x.size,
hash: x.content_hash,
etag: `${x.id}\t${x.content_hash}`,
} as Entity;
} else {
// x[".tag"] === "deleted"
@@ -105,6 +100,132 @@ const fromDropboxItemToEntity = (
}
};
/**
* https://github.com/remotely-save/remotely-save/issues/567
* https://www.dropboxforum.com/t5/Dropbox-API-Support-Feedback/Case-Sensitivity-in-API-2/td-p/191279
* @param entities
*/
export const fixEntityListCasesInplace = (entities: { key?: string }[]) => {
for (const iterator of entities) {
if (iterator.key === undefined) {
throw Error(`dropbox list should all have key, but meet undefined`);
}
}
entities.sort((a, b) => a.key!.length - b.key!.length);
// console.log(JSON.stringify(entities,null,2));
const caseMapping: Record<string, string> = { "": "" };
for (const e of entities) {
// console.log(`looking for: ${JSON.stringify(e, null, 2)}`);
let parentFolder = getParentFolder(e.key!);
if (parentFolder === "/") {
parentFolder = "";
}
const parentFolderLower = parentFolder.toLocaleLowerCase();
const segs = e.key!.split("/");
if (e.key!.endsWith("/")) {
// folder
if (caseMapping.hasOwnProperty(parentFolderLower)) {
const newKey = `${caseMapping[parentFolderLower]}${segs
.slice(-2)
.join("/")}`;
caseMapping[newKey.toLocaleLowerCase()] = newKey;
e.key = newKey;
// console.log(JSON.stringify(caseMapping,null,2));
continue;
} else {
throw Error(`${parentFolder} doesn't have cases record??`);
}
} else {
// file
if (caseMapping.hasOwnProperty(parentFolderLower)) {
const newKey = `${caseMapping[parentFolderLower]}${segs
.slice(-1)
.join("/")}`;
e.key = newKey;
continue;
} else {
throw Error(`${parentFolder} doesn't have cases record??`);
}
}
}
return entities;
};
////////////////////////////////////////////////////////////////////////////////
// Other usual common methods
////////////////////////////////////////////////////////////////////////////////
interface ErrSubType {
error: {
retry_after: number;
};
}
async function retryReq<T>(
reqFunc: () => Promise<DropboxResponse<T>>,
extraHint: string = ""
): Promise<DropboxResponse<T> | undefined> {
const waitSeconds = [1, 2, 4, 8]; // hard code exponential backoff
for (let idx = 0; idx < waitSeconds.length; ++idx) {
try {
if (idx !== 0) {
console.warn(
`${extraHint === "" ? "" : extraHint + ": "}The ${
idx + 1
}-th try starts at time ${Date.now()}`
);
}
return await reqFunc();
} catch (e: unknown) {
const err = e as DropboxResponseError<ErrSubType>;
if (err.status === undefined) {
// then the err is not DropboxResponseError
throw err;
}
if (err.status !== 429) {
// then the err is not "too many requests", give up
throw err;
}
if (idx === waitSeconds.length - 1) {
// the last retry also failed, give up
throw new Error(
`${
extraHint === "" ? "" : extraHint + ": "
}"429 too many requests", after retrying for ${
idx + 1
} times still failed.`
);
}
const headers = headersToRecord(err.headers);
const svrSec =
err.error.error.retry_after ||
parseInt(headers["retry-after"] || "1") ||
1;
const fallbackSec = waitSeconds[idx];
const secMin = Math.max(svrSec, fallbackSec);
const secMax = Math.max(secMin * 1.8, 2);
console.warn(
`${
extraHint === "" ? "" : extraHint + ": "
}We have "429 too many requests" error of ${
idx + 1
}-th try, at time ${Date.now()}, and wait for ${secMin} ~ ${secMax} seconds to retry. Original info: ${JSON.stringify(
err.error,
null,
2
)}`
);
await delay(random(secMin * 1000, secMax * 1000));
}
}
}
////////////////////////////////////////////////////////////////////////////////
// Dropbox authorization using PKCE
// see https://dropbox.tech/developers/pkce--what-and-why-
@@ -228,94 +349,33 @@ export const setConfigBySuccessfullAuthInplace = async (
};
////////////////////////////////////////////////////////////////////////////////
// Other usual common methods
// real exported interface
////////////////////////////////////////////////////////////////////////////////
interface ErrSubType {
error: {
retry_after: number;
};
}
async function retryReq<T>(
reqFunc: () => Promise<DropboxResponse<T>>,
extraHint: string = ""
): Promise<DropboxResponse<T> | undefined> {
const waitSeconds = [1, 2, 4, 8]; // hard code exponential backoff
for (let idx = 0; idx < waitSeconds.length; ++idx) {
try {
if (idx !== 0) {
console.warn(
`${extraHint === "" ? "" : extraHint + ": "}The ${
idx + 1
}-th try starts at time ${Date.now()}`
);
}
return await reqFunc();
} catch (e: unknown) {
const err = e as DropboxResponseError<ErrSubType>;
if (err.status === undefined) {
// then the err is not DropboxResponseError
throw err;
}
if (err.status !== 429) {
// then the err is not "too many requests", give up
throw err;
}
if (idx === waitSeconds.length - 1) {
// the last retry also failed, give up
throw new Error(
`${
extraHint === "" ? "" : extraHint + ": "
}"429 too many requests", after retrying for ${
idx + 1
} times still failed.`
);
}
const headers = headersToRecord(err.headers);
const svrSec =
err.error.error.retry_after ||
parseInt(headers["retry-after"] || "1") ||
1;
const fallbackSec = waitSeconds[idx];
const secMin = Math.max(svrSec, fallbackSec);
const secMax = Math.max(secMin * 1.8, 2);
console.warn(
`${
extraHint === "" ? "" : extraHint + ": "
}We have "429 too many requests" error of ${
idx + 1
}-th try, at time ${Date.now()}, and wait for ${secMin} ~ ${secMax} seconds to retry. Original info: ${JSON.stringify(
err.error,
null,
2
)}`
);
await delay(random(secMin * 1000, secMax * 1000));
}
}
}
export class WrappedDropboxClient {
export class FakeFsDropbox extends FakeFs {
kind: "dropbox";
dropboxConfig: DropboxConfig;
remoteBaseDir: string;
saveUpdatedConfigFunc: () => Promise<any>;
dropbox!: Dropbox;
vaultFolderExists: boolean;
foldersCreatedBefore: Set<string>;
constructor(
dropboxConfig: DropboxConfig,
remoteBaseDir: string,
vaultName: string,
saveUpdatedConfigFunc: () => Promise<any>
) {
super();
this.kind = "dropbox";
this.dropboxConfig = dropboxConfig;
this.remoteBaseDir = remoteBaseDir;
this.remoteBaseDir = this.dropboxConfig.remoteBaseDir || vaultName || "";
this.saveUpdatedConfigFunc = saveUpdatedConfigFunc;
this.vaultFolderExists = false;
this.foldersCreatedBefore = new Set();
}
init = async () => {
async _init() {
// check token
if (
this.dropboxConfig.accessToken === "" ||
@@ -388,38 +448,63 @@ export class WrappedDropboxClient {
}
}
return this.dropbox;
};
}
return this;
}
/**
* @param dropboxConfig
* @returns
*/
export const getDropboxClient = (
dropboxConfig: DropboxConfig,
remoteBaseDir: string,
saveUpdatedConfigFunc: () => Promise<any>
) => {
return new WrappedDropboxClient(
dropboxConfig,
remoteBaseDir,
saveUpdatedConfigFunc
);
};
async walk(): Promise<Entity[]> {
await this._init();
export const getRemoteMeta = async (
client: WrappedDropboxClient,
remotePath: string
) => {
await client.init();
// if (remotePath === "" || remotePath === "/") {
let res = await this.dropbox.filesListFolder({
path: `/${this.remoteBaseDir}`,
recursive: true,
include_deleted: false,
limit: 1000,
});
if (res.status !== 200) {
throw Error(JSON.stringify(res));
}
// console.info(res);
const contents = res.result.entries;
const unifiedContents = contents
.filter((x) => x[".tag"] !== "deleted")
.filter((x) => x.path_display !== `/${this.remoteBaseDir}`)
.map((x) => fromDropboxItemToEntity(x, this.remoteBaseDir));
while (res.result.has_more) {
res = await this.dropbox.filesListFolderContinue({
cursor: res.result.cursor,
});
if (res.status !== 200) {
throw Error(JSON.stringify(res));
}
const contents2 = res.result.entries;
const unifiedContents2 = contents2
.filter((x) => x[".tag"] !== "deleted")
.filter((x) => x.path_display !== `/${this.remoteBaseDir}`)
.map((x) => fromDropboxItemToEntity(x, this.remoteBaseDir));
unifiedContents.push(...unifiedContents2);
}
fixEntityListCasesInplace(unifiedContents);
return unifiedContents;
}
async stat(key: string): Promise<Entity> {
await this._init();
return await this._statFromRoot(getDropboxPath(key, this.remoteBaseDir));
}
async _statFromRoot(key: string): Promise<Entity> {
// if (key === "" || key === "/") {
// // filesGetMetadata doesn't support root folder
// // we instead try to list files
// // if no error occurs, we ensemble a fake result.
// const rsp = await retryReq(() =>
// client.dropbox.filesListFolder({
// path: `/${client.remoteBaseDir}`,
// path: `/${client.key}`,
// recursive: false, // don't need to recursive here
// })
// );
@@ -436,8 +521,8 @@ export const getRemoteMeta = async (
// }
const rsp = await retryReq(() =>
client.dropbox.filesGetMetadata({
path: remotePath,
this.dropbox.filesGetMetadata({
path: key,
})
);
if (rsp === undefined) {
@@ -446,73 +531,42 @@ export const getRemoteMeta = async (
if (rsp.status !== 200) {
throw Error(JSON.stringify(rsp));
}
return fromDropboxItemToEntity(rsp.result, client.remoteBaseDir);
};
export const uploadToRemote = async (
client: WrappedDropboxClient,
fileOrFolderPath: string,
vault: Vault | undefined,
isRecursively: boolean,
cipher: Cipher,
remoteEncryptedKey: string = "",
foldersCreatedBefore: Set<string> | undefined = undefined,
uploadRaw: boolean = false,
rawContent: string | ArrayBuffer = "",
rawContentMTime: number = 0,
rawContentCTime: number = 0
): Promise<UploadedType> => {
await client.init();
let uploadFile = fileOrFolderPath;
if (!cipher.isPasswordEmpty()) {
if (remoteEncryptedKey === undefined || remoteEncryptedKey === "") {
throw Error(
`uploadToRemote(dropbox) you have password but remoteEncryptedKey is empty!`
);
return fromDropboxItemToEntity(rsp.result, this.remoteBaseDir);
}
uploadFile = remoteEncryptedKey;
}
uploadFile = getDropboxPath(uploadFile, client.remoteBaseDir);
if (hasEmojiInText(uploadFile)) {
async mkdir(key: string, mtime?: number, ctime?: number): Promise<Entity> {
if (!key.endsWith("/")) {
throw Error(`you should not call mkdir on ${key}`);
}
await this._init();
const uploadFile = getDropboxPath(key, this.remoteBaseDir);
return await this._mkdirFromRoot(uploadFile, mtime, ctime);
}
async _mkdirFromRoot(
key: string,
mtime?: number,
ctime?: number
): Promise<Entity> {
if (hasEmojiInText(key)) {
throw new Error(
`${uploadFile}: Error: Dropbox does not support emoji in file / folder names.`
`${key}: Error: Dropbox does not support emoji in file / folder names.`
);
}
let mtime = 0;
let ctime = 0;
const s = await vault?.adapter?.stat(fileOrFolderPath);
if (s !== undefined && s !== null) {
mtime = Math.floor(s.mtime / 1000.0) * 1000;
ctime = Math.floor(s.ctime / 1000.0) * 1000;
}
const mtimeStr = new Date(mtime).toISOString().replace(/\.\d{3}Z$/, "Z");
const isFolder = fileOrFolderPath.endsWith("/");
if (isFolder && isRecursively) {
throw Error("upload function doesn't implement recursive function yet!");
} else if (isFolder && !isRecursively) {
if (uploadRaw) {
throw Error(`you specify uploadRaw, but you also provide a folder key!`);
}
// folder
if (cipher.isPasswordEmpty() || cipher.isFolderAware()) {
// if not encrypted, || encrypted isFolderAware, mkdir a remote folder
if (foldersCreatedBefore?.has(uploadFile)) {
if (this.foldersCreatedBefore?.has(key)) {
// created, pass
} else {
try {
await retryReq(
() =>
client.dropbox.filesCreateFolderV2({
path: uploadFile,
this.dropbox.filesCreateFolderV2({
path: key,
}),
fileOrFolderPath
key // just a hint
);
foldersCreatedBefore?.add(uploadFile);
this.foldersCreatedBefore?.add(key);
} catch (e: unknown) {
const err = e as DropboxResponseError<files.CreateFolderError>;
if (err.status === undefined) {
@@ -520,139 +574,100 @@ export const uploadToRemote = async (
}
if (err.status === 409) {
// pass
foldersCreatedBefore?.add(uploadFile);
this.foldersCreatedBefore?.add(key);
} else {
throw err;
}
}
}
const res = await getRemoteMeta(client, uploadFile);
return {
entity: res,
mtimeCli: mtime,
};
} else {
// if encrypted && !isFolderAware(),
// upload a fake file with the encrypted file name
await retryReq(
() =>
client.dropbox.filesUpload({
path: uploadFile,
contents: "",
client_modified: mtimeStr,
}),
fileOrFolderPath
return await this._statFromRoot(key);
}
async writeFile(
key: string,
content: ArrayBuffer,
mtime: number,
ctime: number
): Promise<Entity> {
if (key.endsWith("/")) {
throw Error(`you should not call writeFile on ${key}`);
}
await this._init();
const uploadFile = getDropboxPath(key, this.remoteBaseDir);
return await this._writeFileFromRoot(
uploadFile,
content,
mtime,
ctime,
key
);
return {
entity: await getRemoteMeta(client, uploadFile),
mtimeCli: mtime,
};
}
} else {
// file
// we ignore isRecursively parameter here
let localContent = undefined;
if (uploadRaw) {
if (typeof rawContent === "string") {
localContent = new TextEncoder().encode(rawContent).buffer;
} else {
localContent = rawContent;
}
} else {
if (vault === undefined) {
async _writeFileFromRoot(
key: string,
content: ArrayBuffer,
mtime: number,
ctime: number,
origKey: string
): Promise<Entity> {
if (hasEmojiInText(origKey)) {
throw new Error(
`the vault variable is not passed but we want to read ${fileOrFolderPath} for Dropbox`
`${origKey}: Error: Dropbox does not support emoji in file / folder names.`
);
}
localContent = await vault.adapter.readBinary(fileOrFolderPath);
}
let remoteContent = localContent;
if (!cipher.isPasswordEmpty()) {
remoteContent = await cipher.encryptContent(localContent);
}
const mtimeFixed = Math.floor(mtime / 1000.0) * 1000;
const ctimeFixed = Math.floor(ctime / 1000.0) * 1000;
const mtimeStr = new Date(mtimeFixed)
.toISOString()
.replace(/\.\d{3}Z$/, "Z");
// in dropbox, we don't need to create folders before uploading! cool!
// TODO: filesUploadSession for larger files (>=150 MB)
await retryReq(
() =>
client.dropbox.filesUpload({
path: uploadFile,
contents: remoteContent,
this.dropbox.filesUpload({
path: key,
contents: content,
mode: {
".tag": "overwrite",
},
client_modified: mtimeStr,
}),
fileOrFolderPath
origKey // hint
);
// we want to mark that parent folders are created
if (foldersCreatedBefore !== undefined) {
const dirs = getFolderLevels(uploadFile).map((x) =>
getDropboxPath(x, client.remoteBaseDir)
if (this.foldersCreatedBefore !== undefined) {
const dirs = getFolderLevels(origKey).map((x) =>
getDropboxPath(x, this.remoteBaseDir)
);
for (const dir of dirs) {
foldersCreatedBefore?.add(dir);
this.foldersCreatedBefore?.add(dir);
}
}
return {
entity: await getRemoteMeta(client, uploadFile),
mtimeCli: mtime,
};
}
};
export const listAllFromRemote = async (client: WrappedDropboxClient) => {
await client.init();
let res = await client.dropbox.filesListFolder({
path: `/${client.remoteBaseDir}`,
recursive: true,
include_deleted: false,
limit: 1000,
});
if (res.status !== 200) {
throw Error(JSON.stringify(res));
}
// console.info(res);
const contents = res.result.entries;
const unifiedContents = contents
.filter((x) => x[".tag"] !== "deleted")
.filter((x) => x.path_display !== `/${client.remoteBaseDir}`)
.map((x) => fromDropboxItemToEntity(x, client.remoteBaseDir));
while (res.result.has_more) {
res = await client.dropbox.filesListFolderContinue({
cursor: res.result.cursor,
});
if (res.status !== 200) {
throw Error(JSON.stringify(res));
return await this._statFromRoot(key);
}
const contents2 = res.result.entries;
const unifiedContents2 = contents2
.filter((x) => x[".tag"] !== "deleted")
.filter((x) => x.path_display !== `/${client.remoteBaseDir}`)
.map((x) => fromDropboxItemToEntity(x, client.remoteBaseDir));
unifiedContents.push(...unifiedContents2);
async readFile(key: string): Promise<ArrayBuffer> {
await this._init();
if (key.endsWith("/")) {
throw new Error(`you should not call readFile on folder ${key}`);
}
const downloadFile = getDropboxPath(key, this.remoteBaseDir);
return await this._readFileFromRoot(downloadFile);
}
fixEntityListCasesInplace(unifiedContents);
return unifiedContents;
};
const downloadFromRemoteRaw = async (
client: WrappedDropboxClient,
remotePath: string
) => {
await client.init();
async _readFileFromRoot(key: string): Promise<ArrayBuffer> {
const rsp = await retryReq(
() =>
client.dropbox.filesDownload({
path: remotePath,
this.dropbox.filesDownload({
path: key,
}),
`downloadFromRemoteRaw=${remotePath}`
`downloadFromRemoteRaw=${key}`
);
if (rsp === undefined) {
throw Error(`unknown rsp from dropbox download: ${rsp}`);
@@ -668,109 +683,57 @@ const downloadFromRemoteRaw = async (
} else {
throw Error(`unknown rsp from dropbox download: ${rsp}`);
}
};
export const downloadFromRemote = async (
client: WrappedDropboxClient,
fileOrFolderPath: string,
vault: Vault,
mtime: number,
cipher: Cipher,
remoteEncryptedKey: string = "",
skipSaving: boolean = false
) => {
await client.init();
const isFolder = fileOrFolderPath.endsWith("/");
if (!skipSaving) {
await mkdirpInVault(fileOrFolderPath, vault);
}
// the file is always local file
// we need to encrypt it
if (isFolder) {
// mkdirp locally is enough
// do nothing here
return new ArrayBuffer(0);
} else {
let downloadFile = fileOrFolderPath;
if (!cipher.isPasswordEmpty()) {
downloadFile = remoteEncryptedKey;
}
downloadFile = getDropboxPath(downloadFile, client.remoteBaseDir);
const remoteContent = await downloadFromRemoteRaw(client, downloadFile);
let localContent = remoteContent;
if (!cipher.isPasswordEmpty()) {
localContent = await cipher.decryptContent(remoteContent);
}
if (!skipSaving) {
await vault.adapter.writeBinary(fileOrFolderPath, localContent, {
mtime: mtime,
});
}
return localContent;
}
};
export const deleteFromRemote = async (
client: WrappedDropboxClient,
fileOrFolderPath: string,
cipher: Cipher,
remoteEncryptedKey: string = ""
) => {
if (fileOrFolderPath === "/") {
async rm(key: string): Promise<void> {
if (key === "/") {
return;
}
let remoteFileName = fileOrFolderPath;
if (!cipher.isPasswordEmpty()) {
remoteFileName = remoteEncryptedKey;
}
remoteFileName = getDropboxPath(remoteFileName, client.remoteBaseDir);
const remoteFileName = getDropboxPath(key, this.remoteBaseDir);
await client.init();
await this._init();
try {
await retryReq(
() =>
client.dropbox.filesDeleteV2({
this.dropbox.filesDeleteV2({
path: remoteFileName,
}),
fileOrFolderPath
key // just a hint here
);
} catch (err) {
console.error("some error while deleting");
console.error(err);
}
};
}
export const checkConnectivity = async (
client: WrappedDropboxClient,
callbackFunc?: any
) => {
async checkConnect(callbackFunc?: any): Promise<boolean> {
try {
await client.init();
const results = await getRemoteMeta(client, `/${client.remoteBaseDir}`);
await this._init();
const results = await this._statFromRoot(`/${this.remoteBaseDir}`);
if (results === undefined) {
return false;
}
return true;
} catch (err) {
console.debug(err);
if (callbackFunc !== undefined) {
callbackFunc(err);
}
callbackFunc?.(err);
return false;
}
};
}
export const getUserDisplayName = async (client: WrappedDropboxClient) => {
await client.init();
const acct = await client.dropbox.usersGetCurrentAccount();
async getUserDisplayName() {
await this._init();
const acct = await this.dropbox.usersGetCurrentAccount();
return acct.result.name.display_name;
};
}
export const revokeAuth = async (client: WrappedDropboxClient) => {
await client.init();
await client.dropbox.authTokenRevoke();
};
async revokeAuth() {
try {
await this._init();
await this.dropbox.authTokenRevoke();
return true;
} catch (e) {
return false;
}
}
}
+557
View File
@@ -0,0 +1,557 @@
import { CipherMethodType, Entity } from "./baseTypes";
import * as openssl from "./encryptOpenSSL";
import * as rclone from "./encryptRClone";
import { isVaildText } from "./misc";
import { FakeFs } from "./fsAll";
import cloneDeep from "lodash/cloneDeep";
/**
* quick guess, no actual decryption here
* @param name
* @returns
*/
function isLikelyOpenSSLEncryptedName(name: string): boolean {
if (
name.startsWith(openssl.MAGIC_ENCRYPTED_PREFIX_BASE32) ||
name.startsWith(openssl.MAGIC_ENCRYPTED_PREFIX_BASE64URL)
) {
return true;
}
return false;
}
/**
* quick guess, no actual decryption here
* @param name
* @returns
*/
function isLikelyEncryptedName(name: string): boolean {
return isLikelyOpenSSLEncryptedName(name);
}
/**
* quick guess, no actual decryption here, only openssl can be guessed here
* @param name
* @returns
*/
function isLikelyEncryptedNameNotMatchMethod(
name: string,
method: CipherMethodType
): boolean {
if (isLikelyOpenSSLEncryptedName(name) && method !== "openssl-base64") {
return true;
}
if (!isLikelyOpenSSLEncryptedName(name) && method === "openssl-base64") {
return true;
}
return false;
}
export interface PasswordCheckType {
ok: boolean;
reason:
| "empty_remote"
| "unknown_encryption_method"
| "remote_encrypted_local_no_password"
| "password_matched"
| "password_or_method_not_matched_or_remote_not_encrypted"
| "likely_no_password_both_sides"
| "encryption_method_not_matched";
}
/**
* Useful if isPasswordEmpty()
*/
function copyEntityAndCopyKeyEncSizeEnc(entity: Entity) {
const res = cloneDeep(entity);
res["keyEnc"] = res["keyRaw"];
res["sizeEnc"] = res["sizeRaw"];
return res;
}
export class FakeFsEncrypt extends FakeFs {
innerFs: FakeFs;
readonly password: string;
readonly method: CipherMethodType;
cipherRClone?: rclone.CipherRclone;
cacheMapOrigToEnc: Record<string, string>;
hasCacheMap: boolean;
kind: string;
innerWalkResultCache?: Entity[];
innerWalkResultCacheTime?: number;
constructor(innerFs: FakeFs, password: string, method: CipherMethodType) {
super();
this.innerFs = innerFs;
this.password = password ?? "";
this.method = method;
this.cacheMapOrigToEnc = {};
this.hasCacheMap = false;
this.kind = `encrypt(${this.innerFs.kind},${method})`;
if (method === "rclone-base64") {
this.cipherRClone = new rclone.CipherRclone(password, 5);
}
}
isPasswordEmpty() {
return this.password === "";
}
isFolderAware() {
if (this.method === "openssl-base64") {
return false;
}
if (this.method === "rclone-base64") {
return true;
}
throw Error(`no idea about isFolderAware for method=${this.method}`);
}
/**
* we want a little caching here.
*/
async _getInnerWalkResult(): Promise<Entity[]> {
let innerWalkResult: Entity[] | undefined = undefined;
if (
this.innerWalkResultCacheTime !== undefined &&
this.innerWalkResultCacheTime >= Date.now() - 1000
) {
innerWalkResult = this.innerWalkResultCache!;
} else {
innerWalkResult = await this.innerFs.walk();
this.innerWalkResultCache = innerWalkResult;
this.innerWalkResultCacheTime = Date.now();
}
return innerWalkResult;
}
async isPasswordOk(): Promise<PasswordCheckType> {
const innerWalkResult = await this._getInnerWalkResult();
if (innerWalkResult === undefined || innerWalkResult.length === 0) {
// remote empty
return {
ok: true,
reason: "empty_remote",
};
}
const santyCheckKey = innerWalkResult[0].keyRaw;
if (this.isPasswordEmpty()) {
// TODO: no way to distinguish remote rclone encrypted
// if local has no password??
if (isLikelyEncryptedName(santyCheckKey)) {
return {
ok: false,
reason: "remote_encrypted_local_no_password",
};
} else {
return {
ok: true,
reason: "likely_no_password_both_sides",
};
}
} else {
if (this.method === "unknown") {
return {
ok: false,
reason: "unknown_encryption_method",
};
}
if (isLikelyEncryptedNameNotMatchMethod(santyCheckKey, this.method)) {
return {
ok: false,
reason: "encryption_method_not_matched",
};
}
try {
const k = await this._decryptName(santyCheckKey);
if (k === undefined) {
throw Error(`decryption failed`);
}
return {
ok: true,
reason: "password_matched",
};
} catch (error) {
return {
ok: false,
reason: "password_or_method_not_matched_or_remote_not_encrypted",
};
}
}
}
async walk(): Promise<Entity[]> {
const innerWalkResult = await this._getInnerWalkResult();
const res: Entity[] = [];
if (this.isPasswordEmpty()) {
for (const innerEntity of innerWalkResult) {
res.push(copyEntityAndCopyKeyEncSizeEnc(innerEntity));
this.cacheMapOrigToEnc[innerEntity.key!] = innerEntity.key!;
}
this.hasCacheMap = true;
return res;
} else {
for (const innerEntity of innerWalkResult) {
const key = await this._decryptName(innerEntity.keyRaw);
const size = key.endsWith("/") ? 0 : undefined;
res.push({
key: key,
keyRaw: innerEntity.keyRaw,
keyEnc: innerEntity.key!,
mtimeCli: innerEntity.mtimeCli,
mtimeSvr: innerEntity.mtimeSvr,
size: size,
sizeEnc: innerEntity.size!,
sizeRaw: innerEntity.sizeRaw,
hash: undefined,
synthesizedFolder: innerEntity.synthesizedFolder,
});
this.cacheMapOrigToEnc[key] = innerEntity.keyRaw;
}
this.hasCacheMap = true;
return res;
}
}
async stat(key: string): Promise<Entity> {
if (!this.hasCacheMap) {
throw new Error("You have to build the cacheMap firstly for stat");
}
const keyEnc = this.cacheMapOrigToEnc[key];
if (keyEnc === undefined) {
throw new Error(`no encrypted key ${key} before!`);
}
const innerEntity = await this.innerFs.stat(keyEnc);
if (this.isPasswordEmpty()) {
return copyEntityAndCopyKeyEncSizeEnc(innerEntity);
} else {
return {
key: key,
keyRaw: innerEntity.keyRaw,
keyEnc: innerEntity.key!,
mtimeCli: innerEntity.mtimeCli,
mtimeSvr: innerEntity.mtimeSvr,
size: undefined,
sizeEnc: innerEntity.size!,
sizeRaw: innerEntity.sizeRaw,
hash: undefined,
synthesizedFolder: innerEntity.synthesizedFolder,
};
}
}
async mkdir(key: string, mtime?: number, ctime?: number): Promise<Entity> {
if (!this.hasCacheMap) {
throw new Error("You have to build the cacheMap firstly for mkdir");
}
if (!key.endsWith("/")) {
throw new Error(`should not call mkdir on ${key}`);
}
let keyEnc = this.cacheMapOrigToEnc[key];
if (keyEnc === undefined) {
if (this.isPasswordEmpty()) {
keyEnc = key;
} else {
keyEnc = await this._encryptName(key);
}
this.cacheMapOrigToEnc[key] = keyEnc;
}
if (this.isPasswordEmpty() || this.isFolderAware()) {
const innerEntity = await this.innerFs.mkdir(keyEnc, mtime, ctime);
return copyEntityAndCopyKeyEncSizeEnc(innerEntity);
} else {
const now = Date.now();
const innerEntity = await this.innerFs.writeFile(
keyEnc,
new ArrayBuffer(0),
mtime ?? now,
ctime ?? now
);
return {
key: key,
keyRaw: innerEntity.keyRaw,
keyEnc: innerEntity.key!,
mtimeCli: innerEntity.mtimeCli,
mtimeSvr: innerEntity.mtimeSvr,
size: 0,
sizeEnc: innerEntity.size!,
sizeRaw: innerEntity.sizeRaw,
hash: undefined,
synthesizedFolder: innerEntity.synthesizedFolder,
};
}
}
async writeFile(
key: string,
content: ArrayBuffer,
mtime: number,
ctime: number
): Promise<Entity> {
if (!this.hasCacheMap) {
throw new Error("You have to build the cacheMap firstly for readFile");
}
let keyEnc = this.cacheMapOrigToEnc[key];
if (keyEnc === undefined) {
if (this.isPasswordEmpty()) {
keyEnc = key;
} else {
keyEnc = await this._encryptName(key);
}
this.cacheMapOrigToEnc[key] = keyEnc;
}
if (this.isPasswordEmpty()) {
const innerEntity = await this.innerFs.writeFile(
keyEnc,
content,
mtime,
ctime
);
return copyEntityAndCopyKeyEncSizeEnc(innerEntity);
} else {
const contentEnc = await this._encryptContent(content);
const innerEntity = await this.innerFs.writeFile(
keyEnc,
contentEnc,
mtime,
ctime
);
return {
key: key,
keyRaw: innerEntity.keyRaw,
keyEnc: innerEntity.key!,
mtimeCli: innerEntity.mtimeCli,
mtimeSvr: innerEntity.mtimeSvr,
size: undefined,
sizeEnc: innerEntity.size!,
sizeRaw: innerEntity.sizeRaw,
hash: undefined,
synthesizedFolder: innerEntity.synthesizedFolder,
};
}
}
async readFile(key: string): Promise<ArrayBuffer> {
if (!this.hasCacheMap) {
throw new Error("You have to build the cacheMap firstly for readFile");
}
const keyEnc = this.cacheMapOrigToEnc[key];
if (keyEnc === undefined) {
throw new Error(`no encrypted key ${key} before! cannot readFile`);
}
const contentEnc = await this.innerFs.readFile(keyEnc);
if (this.isPasswordEmpty()) {
return contentEnc;
} else {
const res = await this._decryptContent(contentEnc);
return res;
}
}
async rm(key: string): Promise<void> {
if (!this.hasCacheMap) {
throw new Error("You have to build the cacheMap firstly for rm");
}
const keyEnc = this.cacheMapOrigToEnc[key];
if (keyEnc === undefined) {
throw new Error(`no encrypted key ${key} before! cannot rm`);
}
return await this.innerFs.rm(keyEnc);
}
async checkConnect(callbackFunc?: any): Promise<boolean> {
return await this.innerFs.checkConnect(callbackFunc);
}
async closeResources() {
if (this.method === "rclone-base64" && this.cipherRClone !== undefined) {
this.cipherRClone.closeResources();
}
}
async encryptEntity(input: Entity): Promise<Entity> {
if (input.key === undefined) {
// input.key should always have value
throw Error(`input ${input.keyRaw} is abnormal without key`);
}
if (this.isPasswordEmpty()) {
return copyEntityAndCopyKeyEncSizeEnc(input);
}
// below is for having password
const local = cloneDeep(input);
if (local.sizeEnc === undefined && local.size !== undefined) {
// it's not filled yet, we fill it
// local.size is possibly undefined if it's "prevSync" Entity
// but local.key should always have value
local.sizeEnc = this._getSizeFromOrigToEnc(local.size);
}
if (local.keyEnc === undefined || local.keyEnc === "") {
let keyEnc = this.cacheMapOrigToEnc[input.key];
if (keyEnc !== undefined && keyEnc !== "" && keyEnc !== local.key) {
// we can reuse remote encrypted key if any
local.keyEnc = keyEnc;
} else {
// we assign a new encrypted key because of no remote
keyEnc = await this._encryptName(input.key);
local.keyEnc = keyEnc;
// remember to add back to cache!
this.cacheMapOrigToEnc[input.key] = keyEnc;
}
}
return local;
}
async _encryptContent(content: ArrayBuffer) {
// console.debug("start encryptContent");
if (this.password === "") {
return content;
}
if (this.method === "openssl-base64") {
const res = await openssl.encryptArrayBuffer(content, this.password);
if (res === undefined) {
throw Error(`cannot encrypt content`);
}
return res;
} else if (this.method === "rclone-base64") {
const res =
await this.cipherRClone!.encryptContentByCallingWorker(content);
if (res === undefined) {
throw Error(`cannot encrypt content`);
}
return res;
} else {
throw Error(`not supported encrypt method=${this.method}`);
}
}
async _decryptContent(content: ArrayBuffer) {
// console.debug("start decryptContent");
if (this.password === "") {
return content;
}
if (this.method === "openssl-base64") {
const res = await openssl.decryptArrayBuffer(content, this.password);
if (res === undefined) {
throw Error(`cannot decrypt content`);
}
return res;
} else if (this.method === "rclone-base64") {
const res =
await this.cipherRClone!.decryptContentByCallingWorker(content);
if (res === undefined) {
throw Error(`cannot decrypt content`);
}
return res;
} else {
throw Error(`not supported decrypt method=${this.method}`);
}
}
async _encryptName(name: string) {
// console.debug("start encryptName");
if (this.password === "") {
return name;
}
if (this.method === "openssl-base64") {
const res = await openssl.encryptStringToBase64url(name, this.password);
if (res === undefined) {
throw Error(`cannot encrypt name=${name}`);
}
return res;
} else if (this.method === "rclone-base64") {
const res = await this.cipherRClone!.encryptNameByCallingWorker(name);
if (res === undefined) {
throw Error(`cannot encrypt name=${name}`);
}
return res;
} else {
throw Error(`not supported encrypt method=${this.method}`);
}
}
async _decryptName(name: string): Promise<string> {
// console.debug("start decryptName");
if (this.password === "") {
return name;
}
if (this.method === "openssl-base64") {
if (name.startsWith(openssl.MAGIC_ENCRYPTED_PREFIX_BASE32)) {
// backward compitable with the openssl-base32
try {
const res = await openssl.decryptBase32ToString(name, this.password);
if (res !== undefined && isVaildText(res)) {
return res;
} else {
throw Error(`cannot decrypt name=${name}`);
}
} catch (error) {
throw Error(`cannot decrypt name=${name}`);
}
} else if (name.startsWith(openssl.MAGIC_ENCRYPTED_PREFIX_BASE64URL)) {
try {
const res = await openssl.decryptBase64urlToString(
name,
this.password
);
if (res !== undefined && isVaildText(res)) {
return res;
} else {
throw Error(`cannot decrypt name=${name}`);
}
} catch (error) {
throw Error(`cannot decrypt name=${name}`);
}
} else {
throw Error(
`method=${this.method} but the name=${name}, likely mismatch`
);
}
} else if (this.method === "rclone-base64") {
const res = await this.cipherRClone!.decryptNameByCallingWorker(name);
if (res === undefined) {
throw Error(`cannot decrypt name=${name}`);
}
return res;
} else {
throw Error(`not supported decrypt method=${this.method}`);
}
}
_getSizeFromOrigToEnc(x: number) {
if (this.password === "") {
return x;
}
if (this.method === "openssl-base64") {
return openssl.getSizeFromOrigToEnc(x);
} else if (this.method === "rclone-base64") {
return rclone.getSizeFromOrigToEnc(x);
} else {
throw Error(`not supported encrypt method=${this.method}`);
}
}
async getUserDisplayName(): Promise<string> {
return await this.innerFs.getUserDisplayName();
}
async revokeAuth(): Promise<any> {
return await this.innerFs.revokeAuth();
}
}
+53
View File
@@ -0,0 +1,53 @@
import { RemotelySavePluginSettings } from "./baseTypes";
import { FakeFs } from "./fsAll";
import { FakeFsDropbox } from "./fsDropbox";
import { FakeFsOnedrive } from "./fsOnedrive";
import { FakeFsS3 } from "./fsS3";
import { FakeFsWebdav } from "./fsWebdav";
import { FakeFsWebdis } from "./fsWebdis";
/**
* To avoid circular dependency, we need a new file here.
*/
export function getClient(
settings: RemotelySavePluginSettings,
vaultName: string,
saveUpdatedConfigFunc: () => Promise<any>
): FakeFs {
switch (settings.serviceType) {
case "s3":
return new FakeFsS3(settings.s3);
break;
case "webdav":
return new FakeFsWebdav(
settings.webdav,
vaultName,
saveUpdatedConfigFunc
);
break;
case "dropbox":
return new FakeFsDropbox(
settings.dropbox,
vaultName,
saveUpdatedConfigFunc
);
break;
case "onedrive":
return new FakeFsOnedrive(
settings.onedrive,
vaultName,
saveUpdatedConfigFunc
);
break;
case "webdis":
return new FakeFsWebdis(
settings.webdis,
vaultName,
saveUpdatedConfigFunc
);
break;
default:
throw Error(`cannot init client for serviceType=${settings.serviceType}`);
break;
}
}
+171
View File
@@ -0,0 +1,171 @@
import { DEFAULT_DEBUG_FOLDER, Entity } from "./baseTypes";
import { FakeFs } from "./fsAll";
import { TFile, TFolder, type Vault } from "obsidian";
import { listFilesInObsFolder } from "./obsFolderLister";
import { Profiler } from "./profiler";
import { getFolderLevels, mkdirpInVault, statFix } from "./misc";
export class FakeFsLocal extends FakeFs {
vault: Vault;
syncConfigDir: boolean;
configDir: string;
pluginID: string;
profiler: Profiler;
deleteToWhere: "obsidian" | "system";
kind: "local";
constructor(
vault: Vault,
syncConfigDir: boolean,
configDir: string,
pluginID: string,
profiler: Profiler,
deleteToWhere: "obsidian" | "system"
) {
super();
this.vault = vault;
this.syncConfigDir = syncConfigDir;
this.configDir = configDir;
this.pluginID = pluginID;
this.profiler = profiler;
this.deleteToWhere = deleteToWhere;
this.kind = "local";
}
async walk(): Promise<Entity[]> {
this.profiler.addIndent();
this.profiler.insert("enter walk for local");
const local: Entity[] = [];
const localTAbstractFiles = this.vault.getAllLoadedFiles();
this.profiler.insert("finish getting walk for local");
for (const entry of localTAbstractFiles) {
let r: Entity | undefined = undefined;
let key = entry.path;
if (entry.path === "/") {
// ignore
continue;
} else if (entry instanceof TFile) {
let mtimeLocal: number | undefined = entry.stat.mtime;
if (mtimeLocal <= 0) {
mtimeLocal = entry.stat.ctime;
}
if (mtimeLocal === 0) {
mtimeLocal = undefined;
}
if (mtimeLocal === undefined) {
throw Error(
`Your file has last modified time 0: ${key}, don't know how to deal with it`
);
}
r = {
key: entry.path, // local always unencrypted
keyRaw: entry.path,
mtimeCli: mtimeLocal,
mtimeSvr: mtimeLocal,
size: entry.stat.size, // local always unencrypted
sizeRaw: entry.stat.size,
};
} else if (entry instanceof TFolder) {
key = `${entry.path}/`;
r = {
key: key,
keyRaw: key,
size: 0,
sizeRaw: 0,
};
} else {
throw Error(`unexpected ${entry}`);
}
if (r.keyRaw.startsWith(DEFAULT_DEBUG_FOLDER)) {
// skip listing the debug folder,
// which should always not involved in sync
continue;
} else {
local.push(r);
}
}
this.profiler.insert("finish transforming walk for local");
if (this.syncConfigDir) {
this.profiler.insert("into syncConfigDir");
const syncFiles = await listFilesInObsFolder(
this.configDir,
this.vault,
this.pluginID
);
for (const f of syncFiles) {
local.push(f);
}
this.profiler.insert("finish syncConfigDir");
}
this.profiler.insert("finish walk for local");
this.profiler.removeIndent();
return local;
}
async stat(key: string): Promise<Entity> {
const statRes = await statFix(this.vault, key);
if (statRes === undefined || statRes === null) {
throw Error(`${key} does not exist! cannot stat for local`);
}
const isFolder = statRes.type === "folder";
return {
key: isFolder ? `${key}/` : key, // local always unencrypted
keyRaw: isFolder ? `${key}/` : key,
mtimeCli: statRes.mtime,
mtimeSvr: statRes.mtime,
size: statRes.size, // local always unencrypted
sizeRaw: statRes.size,
};
}
async mkdir(key: string, mtime?: number, ctime?: number): Promise<Entity> {
// console.debug(`mkdir: ${key}`);
await mkdirpInVault(key, this.vault);
return await this.stat(key);
}
async writeFile(
key: string,
content: ArrayBuffer,
mtime: number,
ctime: number
): Promise<Entity> {
await this.vault.adapter.writeBinary(key, content, {
mtime: mtime,
});
return await this.stat(key);
}
async readFile(key: string): Promise<ArrayBuffer> {
return await this.vault.adapter.readBinary(key);
}
async rm(key: string): Promise<void> {
if (this.deleteToWhere === "obsidian") {
await this.vault.adapter.trashLocal(key);
} else {
// "system"
if (!(await this.vault.adapter.trashSystem(key))) {
await this.vault.adapter.trashLocal(key);
}
}
}
async checkConnect(callbackFunc?: any): Promise<boolean> {
return true;
}
async getUserDisplayName(): Promise<string> {
throw new Error("Method not implemented.");
}
async revokeAuth(): Promise<any> {
throw new Error("Method not implemented.");
}
}
+52
View File
@@ -0,0 +1,52 @@
import { Entity } from "./baseTypes";
import { FakeFs } from "./fsAll";
export class FakeFsMock extends FakeFs {
kind: "mock";
constructor() {
super();
this.kind = "mock";
}
async walk(): Promise<Entity[]> {
throw new Error("Method not implemented.");
}
async stat(key: string): Promise<Entity> {
throw new Error("Method not implemented.");
}
async mkdir(key: string, mtime: number, ctime: number): Promise<Entity> {
throw new Error("Method not implemented.");
}
async writeFile(
key: string,
content: ArrayBuffer,
mtime: number,
ctime: number
): Promise<Entity> {
throw new Error("Method not implemented.");
}
async readFile(key: string): Promise<ArrayBuffer> {
throw new Error("Method not implemented.");
}
async rm(key: string): Promise<void> {
throw new Error("Method not implemented.");
}
async checkConnect(callbackFunc?: any): Promise<boolean> {
return true;
}
async getUserDisplayName(): Promise<string> {
throw new Error("Method not implemented.");
}
async revokeAuth(): Promise<any> {
throw new Error("Method not implemented.");
}
}
+185 -310
View File
@@ -1,29 +1,23 @@
import { CryptoProvider, PublicClientApplication } from "@azure/msal-node";
import { AuthenticationProvider } from "@microsoft/microsoft-graph-client";
import type {
DriveItem,
FileSystemInfo,
UploadSession,
User,
} from "@microsoft/microsoft-graph-types";
import { CryptoProvider, PublicClientApplication } from "@azure/msal-node";
import { AuthenticationProvider } from "@microsoft/microsoft-graph-client";
import cloneDeep from "lodash/cloneDeep";
import { request, requestUrl, requireApiVersion, Vault } from "obsidian";
import { request, requestUrl } from "obsidian";
import {
VALID_REQURL,
COMMAND_CALLBACK_ONEDRIVE,
DEFAULT_CONTENT_TYPE,
Entity,
OAUTH2_FORCE_EXPIRE_MILLISECONDS,
OnedriveConfig,
Entity,
UploadedType,
} from "./baseTypes";
import {
bufferToArrayBuffer,
getRandomArrayBuffer,
getRandomIntInclusive,
mkdirpInVault,
} from "./misc";
import { Cipher } from "./encryptUnified";
import { VALID_REQURL } from "./baseTypesObs";
import { FakeFs } from "./fsAll";
import { bufferToArrayBuffer } from "./misc";
const SCOPES = ["User.Read", "Files.ReadWrite.AppFolder", "offline_access"];
const REDIRECT_URI = `obsidian://${COMMAND_CALLBACK_ONEDRIVE}`;
@@ -237,23 +231,6 @@ const getOnedrivePath = (fileOrFolderPath: string, remoteBaseDir: string) => {
return key;
};
const getNormPath = (fileOrFolderPath: string, remoteBaseDir: string) => {
const prefix = `/drive/special/approot:/${remoteBaseDir}`;
if (
!(fileOrFolderPath === prefix || fileOrFolderPath.startsWith(`${prefix}/`))
) {
throw Error(
`"${fileOrFolderPath}" doesn't starts with "${prefix}/" or equals to "${prefix}"`
);
}
if (fileOrFolderPath === prefix) {
return "/";
}
return fileOrFolderPath.slice(`${prefix}/`.length);
};
const constructFromDriveItemToEntityError = (x: DriveItem) => {
return `parentPath="${
x.parentReference?.path ?? "(no parentReference or path)"
@@ -361,15 +338,20 @@ const fromDriveItemToEntity = (x: DriveItem, remoteBaseDir: string): Entity => {
const mtimeSvr = Date.parse(x?.fileSystemInfo!.lastModifiedDateTime!);
const mtimeCli = Date.parse(x?.fileSystemInfo!.lastModifiedDateTime!);
return {
key: key,
keyRaw: key,
mtimeSvr: mtimeSvr,
mtimeCli: mtimeCli,
size: isFolder ? 0 : x.size!,
sizeRaw: isFolder ? 0 : x.size!,
// hash: ?? // TODO
etag: x.cTag || "", // do NOT use x.eTag because it changes if meta changes
};
};
////////////////////////////////////////////////////////////////////////////////
// The client.
////////////////////////////////////////////////////////////////////////////////
// to adapt to the required interface
class MyAuthProvider implements AuthenticationProvider {
onedriveConfig: OnedriveConfig;
@@ -381,7 +363,8 @@ class MyAuthProvider implements AuthenticationProvider {
this.onedriveConfig = onedriveConfig;
this.saveUpdatedConfigFunc = saveUpdatedConfigFunc;
}
getAccessToken = async () => {
async getAccessToken() {
if (
this.onedriveConfig.accessToken === "" ||
this.onedriveConfig.refreshToken === ""
@@ -415,7 +398,7 @@ class MyAuthProvider implements AuthenticationProvider {
console.info("Onedrive accessToken updated");
return this.onedriveConfig.accessToken;
}
};
}
}
/**
@@ -431,25 +414,31 @@ export const getShrinkedSettings = (onedriveConfig: OnedriveConfig) => {
return config;
};
export class WrappedOnedriveClient {
export class FakeFsOnedrive extends FakeFs {
kind: "onedrive";
onedriveConfig: OnedriveConfig;
remoteBaseDir: string;
vaultFolderExists: boolean;
authGetter: MyAuthProvider;
saveUpdatedConfigFunc: () => Promise<any>;
foldersCreatedBefore: Set<string>;
constructor(
onedriveConfig: OnedriveConfig,
remoteBaseDir: string,
vaultName: string,
saveUpdatedConfigFunc: () => Promise<any>
) {
super();
this.kind = "onedrive";
this.onedriveConfig = onedriveConfig;
this.remoteBaseDir = remoteBaseDir;
this.remoteBaseDir = this.onedriveConfig.remoteBaseDir || vaultName || "";
this.vaultFolderExists = false;
this.saveUpdatedConfigFunc = saveUpdatedConfigFunc;
this.authGetter = new MyAuthProvider(onedriveConfig, saveUpdatedConfigFunc);
this.foldersCreatedBefore = new Set();
}
init = async () => {
async _init() {
// check token
if (
this.onedriveConfig.accessToken === "" ||
@@ -463,14 +452,14 @@ export class WrappedOnedriveClient {
if (this.vaultFolderExists) {
// console.info(`already checked, /${this.remoteBaseDir} exist before`)
} else {
const k = await this.getJson("/drive/special/approot/children");
const k = await this._getJson("/drive/special/approot/children");
// console.debug(k);
this.vaultFolderExists =
(k.value as DriveItem[]).filter((x) => x.name === this.remoteBaseDir)
.length > 0;
if (!this.vaultFolderExists) {
console.info(`remote does not have folder /${this.remoteBaseDir}`);
await this.postJson("/drive/special/approot/children", {
await this._postJson("/drive/special/approot/children", {
name: `${this.remoteBaseDir}`,
folder: {},
"@microsoft.graph.conflictBehavior": "replace",
@@ -481,9 +470,9 @@ export class WrappedOnedriveClient {
// console.info(`remote folder /${this.remoteBaseDir} exists`);
}
}
};
}
buildUrl = (pathFragOrig: string) => {
_buildUrl(pathFragOrig: string) {
const API_PREFIX = "https://graph.microsoft.com/v1.0";
let theUrl = "";
if (
@@ -501,10 +490,10 @@ export class WrappedOnedriveClient {
theUrl = theUrl.replace(/#/g, "%23");
// console.debug(`building url: [${pathFragOrig}] => [${theUrl}]`)
return theUrl;
};
}
getJson = async (pathFragOrig: string) => {
const theUrl = this.buildUrl(pathFragOrig);
async _getJson(pathFragOrig: string) {
const theUrl = this._buildUrl(pathFragOrig);
console.debug(`getJson, theUrl=${theUrl}`);
return JSON.parse(
await request({
@@ -517,10 +506,10 @@ export class WrappedOnedriveClient {
},
})
);
};
}
postJson = async (pathFragOrig: string, payload: any) => {
const theUrl = this.buildUrl(pathFragOrig);
async _postJson(pathFragOrig: string, payload: any) {
const theUrl = this._buildUrl(pathFragOrig);
console.debug(`postJson, theUrl=${theUrl}`);
return JSON.parse(
await request({
@@ -533,10 +522,10 @@ export class WrappedOnedriveClient {
},
})
);
};
}
patchJson = async (pathFragOrig: string, payload: any) => {
const theUrl = this.buildUrl(pathFragOrig);
async _patchJson(pathFragOrig: string, payload: any) {
const theUrl = this._buildUrl(pathFragOrig);
console.debug(`patchJson, theUrl=${theUrl}`);
return JSON.parse(
await request({
@@ -549,10 +538,10 @@ export class WrappedOnedriveClient {
},
})
);
};
}
deleteJson = async (pathFragOrig: string) => {
const theUrl = this.buildUrl(pathFragOrig);
async _deleteJson(pathFragOrig: string) {
const theUrl = this._buildUrl(pathFragOrig);
console.debug(`deleteJson, theUrl=${theUrl}`);
if (VALID_REQURL) {
await requestUrl({
@@ -570,10 +559,10 @@ export class WrappedOnedriveClient {
},
});
}
};
}
putArrayBuffer = async (pathFragOrig: string, payload: ArrayBuffer) => {
const theUrl = this.buildUrl(pathFragOrig);
async _putArrayBuffer(pathFragOrig: string, payload: ArrayBuffer) {
const theUrl = this._buildUrl(pathFragOrig);
console.debug(`putArrayBuffer, theUrl=${theUrl}`);
// TODO:
// 20220401: On Android, requestUrl has issue that text becomes base64.
@@ -601,7 +590,7 @@ export class WrappedOnedriveClient {
});
return (await res.json()) as DriveItem | UploadSession;
}
};
}
/**
* A specialized function to upload large files by parts
@@ -611,14 +600,14 @@ export class WrappedOnedriveClient {
* @param rangeEnd the end, exclusive
* @param size
*/
putUint8ArrayByRange = async (
async _putUint8ArrayByRange(
pathFragOrig: string,
payload: Uint8Array,
rangeStart: number,
rangeEnd: number,
size: number
) => {
const theUrl = this.buildUrl(pathFragOrig);
) {
const theUrl = this._buildUrl(pathFragOrig);
console.debug(
`putUint8ArrayByRange, theUrl=${theUrl}, range=${rangeStart}-${
rangeEnd - 1
@@ -654,202 +643,141 @@ export class WrappedOnedriveClient {
});
return (await res.json()) as DriveItem | UploadSession;
}
};
}
}
export const getOnedriveClient = (
onedriveConfig: OnedriveConfig,
remoteBaseDir: string,
saveUpdatedConfigFunc: () => Promise<any>
) => {
return new WrappedOnedriveClient(
onedriveConfig,
remoteBaseDir,
saveUpdatedConfigFunc
);
};
/**
/**
* Use delta api to list all files and folders
* https://docs.microsoft.com/en-us/onedrive/developer/rest-api/api/driveitem_delta?view=odsp-graph-online
* @param client
*/
export const listAllFromRemote = async (client: WrappedOnedriveClient) => {
await client.init();
async walk(): Promise<Entity[]> {
await this._init();
const NEXT_LINK_KEY = "@odata.nextLink";
const DELTA_LINK_KEY = "@odata.deltaLink";
let res = await client.getJson(
`/drive/special/approot:/${client.remoteBaseDir}:/delta`
let res = await this._getJson(
`/drive/special/approot:/${this.remoteBaseDir}:/delta`
);
let driveItems = res.value as DriveItem[];
// console.debug(driveItems);
while (NEXT_LINK_KEY in res) {
res = await client.getJson(res[NEXT_LINK_KEY]);
res = await this._getJson(res[NEXT_LINK_KEY]);
driveItems.push(...cloneDeep(res.value as DriveItem[]));
}
// lastly we should have delta link?
if (DELTA_LINK_KEY in res) {
client.onedriveConfig.deltaLink = res[DELTA_LINK_KEY];
await client.saveUpdatedConfigFunc();
this.onedriveConfig.deltaLink = res[DELTA_LINK_KEY];
await this.saveUpdatedConfigFunc();
}
// unify everything to Entity
const unifiedContents = driveItems
.map((x) => fromDriveItemToEntity(x, client.remoteBaseDir))
.filter((x) => x.keyRaw !== "/");
.map((x) => fromDriveItemToEntity(x, this.remoteBaseDir))
.filter((x) => x.key !== "/");
return unifiedContents;
};
}
export const getRemoteMeta = async (
client: WrappedOnedriveClient,
remotePath: string
) => {
await client.init();
async stat(key: string): Promise<Entity> {
await this._init();
return await this._statFromRoot(getOnedrivePath(key, this.remoteBaseDir));
}
async _statFromRoot(key: string): Promise<Entity> {
// console.info(`remotePath=${remotePath}`);
const rsp = await client.getJson(
`${remotePath}?$select=cTag,eTag,fileSystemInfo,folder,file,name,parentReference,size`
const rsp = await this._getJson(
`${key}?$select=cTag,eTag,fileSystemInfo,folder,file,name,parentReference,size`
);
// console.info(rsp);
const driveItem = rsp as DriveItem;
const res = fromDriveItemToEntity(driveItem, client.remoteBaseDir);
const res = fromDriveItemToEntity(driveItem, this.remoteBaseDir);
// console.info(res);
return res;
};
export const uploadToRemote = async (
client: WrappedOnedriveClient,
fileOrFolderPath: string,
vault: Vault | undefined,
isRecursively: boolean,
cipher: Cipher,
remoteEncryptedKey: string = "",
foldersCreatedBefore: Set<string> | undefined = undefined,
uploadRaw: boolean = false,
rawContent: string | ArrayBuffer = ""
): Promise<UploadedType> => {
await client.init();
let uploadFile = fileOrFolderPath;
if (!cipher.isPasswordEmpty()) {
if (remoteEncryptedKey === undefined || remoteEncryptedKey === "") {
throw Error(
`uploadToRemote(onedrive) you have password but remoteEncryptedKey is empty!`
);
}
uploadFile = remoteEncryptedKey;
}
uploadFile = getOnedrivePath(uploadFile, client.remoteBaseDir);
console.debug(`uploadFile=${uploadFile}`);
let mtime = 0;
let ctime = 0;
const s = await vault?.adapter?.stat(fileOrFolderPath);
if (s !== undefined && s !== null) {
mtime = s.mtime;
ctime = s.ctime;
async mkdir(key: string, mtime?: number, ctime?: number): Promise<Entity> {
if (!key.endsWith("/")) {
throw Error(`you should not call mkdir on ${key}`);
}
const ctimeStr = new Date(ctime).toISOString();
const mtimeStr = new Date(mtime).toISOString();
const isFolder = fileOrFolderPath.endsWith("/");
if (isFolder && isRecursively) {
throw Error("upload function doesn't implement recursive function yet!");
} else if (isFolder && !isRecursively) {
if (uploadRaw) {
throw Error(`you specify uploadRaw, but you also provide a folder key!`);
await this._init();
const uploadFolder = getOnedrivePath(key, this.remoteBaseDir);
console.debug(`mkdir uploadFolder=${uploadFolder}`);
return await this._mkdirFromRoot(uploadFolder, mtime, ctime);
}
// folder
if (cipher.isPasswordEmpty() || cipher.isFolderAware()) {
// if not encrypted, || encrypted isFolderAware, mkdir a remote folder
if (foldersCreatedBefore?.has(uploadFile)) {
async _mkdirFromRoot(
key: string,
mtime?: number,
ctime?: number
): Promise<Entity> {
// console.debug(`foldersCreatedBefore=${Array.from(this.foldersCreatedBefore)}`);
if (this.foldersCreatedBefore.has(key)) {
// created, pass
// console.debug(`folder ${key} created.`)
} else {
// https://stackoverflow.com/questions/56479865/creating-nested-folders-in-one-go-onedrive-api
// use PATCH to create folder recursively!!!
let k: any = {
let playload: any = {
folder: {},
"@microsoft.graph.conflictBehavior": "replace",
};
if (mtime !== 0 && ctime !== 0) {
k = {
folder: {},
"@microsoft.graph.conflictBehavior": "replace",
fileSystemInfo: {
lastModifiedDateTime: mtimeStr,
createdDateTime: ctimeStr,
} as FileSystemInfo,
};
const fileSystemInfo: Record<string, string> = {};
if (mtime !== undefined && mtime !== 0) {
const mtimeStr = new Date(mtime).toISOString();
fileSystemInfo["lastModifiedDateTime"] = mtimeStr;
}
await client.patchJson(uploadFile, k);
if (ctime !== undefined && ctime !== 0) {
const ctimeStr = new Date(ctime).toISOString();
fileSystemInfo["createdDateTime"] = ctimeStr;
}
if (Object.keys(fileSystemInfo).length > 0) {
playload["fileSystemInfo"] = fileSystemInfo;
}
await this._patchJson(key, playload);
}
const res = await this._statFromRoot(key);
return res;
}
const res = await getRemoteMeta(client, uploadFile);
return {
entity: res,
mtimeCli: mtime,
};
} else {
// if encrypted && !isFolderAware(),
// upload a fake, random-size file
// with the encrypted file name
const byteLengthRandom = getRandomIntInclusive(
1,
65536 /* max allowed */
);
const arrBufRandom = await cipher.encryptContent(
getRandomArrayBuffer(byteLengthRandom)
);
// an encrypted folder is always small, we just use put here
await client.putArrayBuffer(
`${uploadFile}:/content?${new URLSearchParams({
"@microsoft.graph.conflictBehavior": "replace",
})}`,
arrBufRandom
);
if (mtime !== 0 && ctime !== 0) {
await client.patchJson(`${uploadFile}`, {
fileSystemInfo: {
lastModifiedDateTime: mtimeStr,
createdDateTime: ctimeStr,
} as FileSystemInfo,
});
async writeFile(
key: string,
content: ArrayBuffer,
mtime: number,
ctime: number
): Promise<Entity> {
if (key.endsWith("/")) {
throw Error(`you should not call writeFile on ${key}`);
}
// console.info(uploadResult)
const res = await getRemoteMeta(client, uploadFile);
return {
entity: res,
mtimeCli: mtime,
};
}
} else {
// file
// we ignore isRecursively parameter here
let localContent = undefined;
if (uploadRaw) {
if (typeof rawContent === "string") {
localContent = new TextEncoder().encode(rawContent).buffer;
} else {
localContent = rawContent;
}
} else {
if (vault === undefined) {
throw new Error(
`the vault variable is not passed but we want to read ${fileOrFolderPath} for OneDrive`
await this._init();
const uploadFile = getOnedrivePath(key, this.remoteBaseDir);
console.debug(`uploadFile=${uploadFile}`);
return await this._writeFileFromRoot(
uploadFile,
content,
mtime,
ctime,
key
);
}
localContent = await vault.adapter.readBinary(fileOrFolderPath);
}
let remoteContent = localContent;
if (!cipher.isPasswordEmpty()) {
remoteContent = await cipher.encryptContent(localContent);
async _writeFileFromRoot(
key: string,
content: ArrayBuffer,
mtime: number,
ctime: number,
origKey: string
): Promise<Entity> {
if (content.byteLength === 0) {
throw Error(
`${origKey}: Empty file is not allowed in OneDrive, and please write something in it.`
);
}
const ctimeStr = new Date(ctime).toISOString();
const mtimeStr = new Date(mtime).toISOString();
// no need to create parent folders firstly, cool!
// hard code range size
@@ -857,16 +785,16 @@ export const uploadToRemote = async (
const RANGE_SIZE = MIN_UNIT * 20; // about 6.5536 MB
const DIRECT_UPLOAD_MAX_SIZE = 1000 * 1000 * 4; // 4 Megabyte
if (remoteContent.byteLength < DIRECT_UPLOAD_MAX_SIZE) {
if (content.byteLength < DIRECT_UPLOAD_MAX_SIZE) {
// directly using put!
await client.putArrayBuffer(
`${uploadFile}:/content?${new URLSearchParams({
await this._putArrayBuffer(
`${key}:/content?${new URLSearchParams({
"@microsoft.graph.conflictBehavior": "replace",
})}`,
remoteContent
content
);
if (mtime !== 0 && ctime !== 0) {
await client.patchJson(`${uploadFile}`, {
await this._patchJson(key, {
fileSystemInfo: {
lastModifiedDateTime: mtimeStr,
createdDateTime: ctimeStr,
@@ -879,13 +807,13 @@ export const uploadToRemote = async (
// 1. create uploadSession
// uploadFile already starts with /drive/special/approot:/${remoteBaseDir}
let k: any = {
let playload: any = {
item: {
"@microsoft.graph.conflictBehavior": "replace",
},
};
if (mtime !== 0 && ctime !== 0) {
k = {
playload = {
item: {
"@microsoft.graph.conflictBehavior": "replace",
@@ -897,9 +825,9 @@ export const uploadToRemote = async (
},
};
}
const s: UploadSession = await client.postJson(
`${uploadFile}:/createUploadSession`,
k
const s: UploadSession = await this._postJson(
`${key}:/createUploadSession`,
playload
);
const uploadUrl = s.uploadUrl!;
console.debug("uploadSession = ");
@@ -907,12 +835,12 @@ export const uploadToRemote = async (
// 2. upload by ranges
// convert to uint8
const uint8 = new Uint8Array(remoteContent);
const uint8 = new Uint8Array(content);
// upload the ranges one by one
let rangeStart = 0;
while (rangeStart < uint8.byteLength) {
await client.putUint8ArrayByRange(
await this._putUint8ArrayByRange(
uploadUrl,
uint8,
rangeStart,
@@ -923,21 +851,22 @@ export const uploadToRemote = async (
}
}
const res = await getRemoteMeta(client, uploadFile);
return {
entity: res,
mtimeCli: mtime,
};
const res = await this._statFromRoot(key);
return res;
}
};
const downloadFromRemoteRaw = async (
client: WrappedOnedriveClient,
remotePath: string
): Promise<ArrayBuffer> => {
await client.init();
const rsp = await client.getJson(
`${remotePath}?$select=@microsoft.graph.downloadUrl`
async readFile(key: string): Promise<ArrayBuffer> {
await this._init();
if (key.endsWith("/")) {
throw new Error(`you should not call readFile on folder ${key}`);
}
const downloadFile = getOnedrivePath(key, this.remoteBaseDir);
return await this._readFileFromRoot(downloadFile);
}
async _readFileFromRoot(key: string): Promise<ArrayBuffer> {
const rsp = await this._getJson(
`${key}?$select=@microsoft.graph.downloadUrl`
);
const downloadUrl: string = rsp["@microsoft.graph.downloadUrl"];
if (VALID_REQURL) {
@@ -949,106 +878,52 @@ const downloadFromRemoteRaw = async (
).arrayBuffer;
return content;
} else {
const content = await // cannot set no-cache here, will have cors error
(await fetch(downloadUrl)).arrayBuffer();
// cannot set no-cache here, will have cors error
const content = await (await fetch(downloadUrl)).arrayBuffer();
return content;
}
};
export const downloadFromRemote = async (
client: WrappedOnedriveClient,
fileOrFolderPath: string,
vault: Vault,
mtime: number,
cipher: Cipher,
remoteEncryptedKey: string = "",
skipSaving: boolean = false
) => {
await client.init();
const isFolder = fileOrFolderPath.endsWith("/");
if (!skipSaving) {
await mkdirpInVault(fileOrFolderPath, vault);
}
if (isFolder) {
// mkdirp locally is enough
// do nothing here
return new ArrayBuffer(0);
} else {
let downloadFile = fileOrFolderPath;
if (!cipher.isPasswordEmpty()) {
downloadFile = remoteEncryptedKey;
}
downloadFile = getOnedrivePath(downloadFile, client.remoteBaseDir);
const remoteContent = await downloadFromRemoteRaw(client, downloadFile);
let localContent = remoteContent;
if (!cipher.isPasswordEmpty()) {
localContent = await cipher.decryptContent(remoteContent);
}
if (!skipSaving) {
await vault.adapter.writeBinary(fileOrFolderPath, localContent, {
mtime: mtime,
});
}
return localContent;
}
};
export const deleteFromRemote = async (
client: WrappedOnedriveClient,
fileOrFolderPath: string,
cipher: Cipher,
remoteEncryptedKey: string = ""
) => {
if (fileOrFolderPath === "/") {
async rm(key: string): Promise<void> {
if (key === "" || key === "/") {
return;
}
let remoteFileName = fileOrFolderPath;
if (!cipher.isPasswordEmpty()) {
remoteFileName = remoteEncryptedKey;
const remoteFileName = getOnedrivePath(key, this.remoteBaseDir);
await this._init();
await this._deleteJson(remoteFileName);
}
remoteFileName = getOnedrivePath(remoteFileName, client.remoteBaseDir);
await client.init();
await client.deleteJson(remoteFileName);
};
export const checkConnectivity = async (
client: WrappedOnedriveClient,
callbackFunc?: any
) => {
async checkConnect(callbackFunc?: any): Promise<boolean> {
try {
const k = await getUserDisplayName(client);
const k = await this.getUserDisplayName();
return k !== "<unknown display name>";
} catch (err) {
console.debug(err);
if (callbackFunc !== undefined) {
callbackFunc(err);
}
callbackFunc?.(err);
return false;
}
};
}
export const getUserDisplayName = async (client: WrappedOnedriveClient) => {
await client.init();
const res: User = await client.getJson("/me?$select=displayName");
async getUserDisplayName() {
await this._init();
const res: User = await this._getJson("/me?$select=displayName");
return res.displayName || "<unknown display name>";
};
}
/**
/**
*
* https://docs.microsoft.com/en-us/azure/active-directory/develop/v2-protocols-oidc#send-a-sign-out-request
* https://docs.microsoft.com/en-us/graph/api/user-revokesigninsessions
* https://docs.microsoft.com/en-us/graph/api/user-invalidateallrefreshtokens
* @param client
*/
// export const revokeAuth = async (client: WrappedOnedriveClient) => {
// await client.init();
// await client.postJson('/me/revokeSignInSessions', {});
// };
async revokeAuth() {
// await this._init();
// await this._postJson("/me/revokeSignInSessions", {});
throw new Error("Method not implemented.");
}
export const getRevokeAddr = async () => {
async getRevokeAddr() {
return "https://account.live.com/consent/Manage";
};
}
}
+813
View File
@@ -0,0 +1,813 @@
import type { _Object, PutObjectCommandInput } from "@aws-sdk/client-s3";
import {
DeleteObjectCommand,
GetObjectCommand,
HeadObjectCommand,
HeadObjectCommandOutput,
ListObjectsV2Command,
ListObjectsV2CommandInput,
PutObjectCommand,
S3Client,
} from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
import { HttpRequest, HttpResponse } from "@smithy/protocol-http";
import {
FetchHttpHandler,
FetchHttpHandlerOptions,
} from "@smithy/fetch-http-handler";
// @ts-ignore
import { requestTimeout } from "@smithy/fetch-http-handler/dist-es/request-timeout";
import { buildQueryString } from "@smithy/querystring-builder";
import { HttpHandlerOptions } from "@aws-sdk/types";
import { Buffer } from "buffer";
import * as mime from "mime-types";
import { Platform, requestUrl, RequestUrlParam } from "obsidian";
import { Readable } from "stream";
import * as path from "path";
import AggregateError from "aggregate-error";
import { DEFAULT_CONTENT_TYPE, S3Config } from "./baseTypes";
import { VALID_REQURL } from "./baseTypesObs";
import { bufferToArrayBuffer, getFolderLevels } from "./misc";
import PQueue from "p-queue";
import { Entity } from "./baseTypes";
import { FakeFs } from "./fsAll";
////////////////////////////////////////////////////////////////////////////////
// special handler using Obsidian requestUrl
////////////////////////////////////////////////////////////////////////////////
/**
* This is close to origin implementation of FetchHttpHandler
* https://github.com/aws/aws-sdk-js-v3/blob/main/packages/fetch-http-handler/src/fetch-http-handler.ts
* that is released under Apache 2 License.
* But this uses Obsidian requestUrl instead.
*/
class ObsHttpHandler extends FetchHttpHandler {
requestTimeoutInMs: number | undefined;
reverseProxyNoSignUrl: string | undefined;
constructor(
options?: FetchHttpHandlerOptions,
reverseProxyNoSignUrl?: string
) {
super(options);
this.requestTimeoutInMs =
options === undefined ? undefined : options.requestTimeout;
this.reverseProxyNoSignUrl = reverseProxyNoSignUrl;
}
async handle(
request: HttpRequest,
{ abortSignal }: HttpHandlerOptions = {}
): Promise<{ response: HttpResponse }> {
if (abortSignal?.aborted) {
const abortError = new Error("Request aborted");
abortError.name = "AbortError";
return Promise.reject(abortError);
}
let path = request.path;
if (request.query) {
const queryString = buildQueryString(request.query);
if (queryString) {
path += `?${queryString}`;
}
}
const { port, method } = request;
let url = `${request.protocol}//${request.hostname}${
port ? `:${port}` : ""
}${path}`;
if (
this.reverseProxyNoSignUrl !== undefined &&
this.reverseProxyNoSignUrl !== ""
) {
const urlObj = new URL(url);
urlObj.host = this.reverseProxyNoSignUrl;
url = urlObj.href;
}
const body =
method === "GET" || method === "HEAD" ? undefined : request.body;
const transformedHeaders: Record<string, string> = {};
for (const key of Object.keys(request.headers)) {
const keyLower = key.toLowerCase();
if (keyLower === "host" || keyLower === "content-length") {
continue;
}
transformedHeaders[keyLower] = request.headers[key];
}
let contentType: string | undefined = undefined;
if (transformedHeaders["content-type"] !== undefined) {
contentType = transformedHeaders["content-type"];
}
let transformedBody: any = body;
if (ArrayBuffer.isView(body)) {
transformedBody = bufferToArrayBuffer(body);
}
const param: RequestUrlParam = {
body: transformedBody,
headers: transformedHeaders,
method: method,
url: url,
contentType: contentType,
};
const raceOfPromises = [
requestUrl(param).then((rsp) => {
const headers = rsp.headers;
const headersLower: Record<string, string> = {};
for (const key of Object.keys(headers)) {
headersLower[key.toLowerCase()] = headers[key];
}
const stream = new ReadableStream<Uint8Array>({
start(controller) {
controller.enqueue(new Uint8Array(rsp.arrayBuffer));
controller.close();
},
});
return {
response: new HttpResponse({
headers: headersLower,
statusCode: rsp.status,
body: stream,
}),
};
}),
requestTimeout(this.requestTimeoutInMs),
];
if (abortSignal) {
raceOfPromises.push(
new Promise<never>((resolve, reject) => {
abortSignal.onabort = () => {
const abortError = new Error("Request aborted");
abortError.name = "AbortError";
reject(abortError);
};
})
);
}
return Promise.race(raceOfPromises);
}
}
////////////////////////////////////////////////////////////////////////////////
// other stuffs
////////////////////////////////////////////////////////////////////////////////
export const simpleTransRemotePrefix = (x: string) => {
if (x === undefined) {
return "";
}
let y = path.posix.normalize(x.trim());
if (y === undefined || y === "" || y === "/" || y === ".") {
return "";
}
if (y.startsWith("/")) {
y = y.slice(1);
}
if (!y.endsWith("/")) {
y = `${y}/`;
}
return y;
};
export const DEFAULT_S3_CONFIG: S3Config = {
s3Endpoint: "",
s3Region: "",
s3AccessKeyID: "",
s3SecretAccessKey: "",
s3BucketName: "",
bypassCorsLocally: true,
partsConcurrency: 20,
forcePathStyle: false,
remotePrefix: "",
useAccurateMTime: false, // it causes money, disable by default
reverseProxyNoSignUrl: "",
generateFolderObject: false, // new version, by default not generate folders
};
/**
* The Body of resp of aws GetObject has mix types
* and we want to get ArrayBuffer here.
* See https://github.com/aws/aws-sdk-js-v3/issues/1877
* @param b The Body of GetObject
* @returns Promise<ArrayBuffer>
*/
const getObjectBodyToArrayBuffer = async (
b: Readable | ReadableStream | Blob | undefined
) => {
if (b === undefined) {
throw Error(`ObjectBody is undefined and don't know how to deal with it`);
}
if (b instanceof Readable) {
return (await new Promise((resolve, reject) => {
const chunks: Uint8Array[] = [];
b.on("data", (chunk) => chunks.push(chunk));
b.on("error", reject);
b.on("end", () => resolve(bufferToArrayBuffer(Buffer.concat(chunks))));
})) as ArrayBuffer;
} else if (b instanceof ReadableStream) {
return await new Response(b, {}).arrayBuffer();
} else if (b instanceof Blob) {
return await b.arrayBuffer();
} else {
throw TypeError(`The type of ${b} is not one of the supported types`);
}
};
const getS3Client = (s3Config: S3Config) => {
let endpoint = s3Config.s3Endpoint;
if (!(endpoint.startsWith("http://") || endpoint.startsWith("https://"))) {
endpoint = `https://${endpoint}`;
}
let s3Client: S3Client;
if (VALID_REQURL && s3Config.bypassCorsLocally) {
s3Client = new S3Client({
region: s3Config.s3Region,
endpoint: endpoint,
forcePathStyle: s3Config.forcePathStyle,
credentials: {
accessKeyId: s3Config.s3AccessKeyID,
secretAccessKey: s3Config.s3SecretAccessKey,
},
requestHandler: new ObsHttpHandler(
undefined,
s3Config.reverseProxyNoSignUrl
),
});
} else {
s3Client = new S3Client({
region: s3Config.s3Region,
endpoint: endpoint,
forcePathStyle: s3Config.forcePathStyle,
credentials: {
accessKeyId: s3Config.s3AccessKeyID,
secretAccessKey: s3Config.s3SecretAccessKey,
},
});
}
s3Client.middlewareStack.add(
(next, context) => (args) => {
(args.request as any).headers["cache-control"] = "no-cache";
return next(args);
},
{
step: "build",
}
);
return s3Client;
};
const getLocalNoPrefixPath = (
fileOrFolderPathWithRemotePrefix: string,
remotePrefix: string
) => {
if (
!(
fileOrFolderPathWithRemotePrefix === `${remotePrefix}` ||
fileOrFolderPathWithRemotePrefix.startsWith(`${remotePrefix}`)
)
) {
throw Error(
`"${fileOrFolderPathWithRemotePrefix}" doesn't starts with "${remotePrefix}"`
);
}
return fileOrFolderPathWithRemotePrefix.slice(`${remotePrefix}`.length);
};
const getRemoteWithPrefixPath = (
fileOrFolderPath: string,
remotePrefix: string
) => {
if (remotePrefix === undefined || remotePrefix === "") {
return fileOrFolderPath;
}
let key = fileOrFolderPath;
if (fileOrFolderPath === "/" || fileOrFolderPath === "") {
// special
key = remotePrefix;
}
if (!fileOrFolderPath.startsWith("/")) {
key = `${remotePrefix}${fileOrFolderPath}`;
}
return key;
};
const fromS3ObjectToEntity = (
x: _Object,
remotePrefix: string,
mtimeRecords: Record<string, number>,
ctimeRecords: Record<string, number>
) => {
// console.debug(`fromS3ObjectToEntity: ${x.Key!}, ${JSON.stringify(x,null,2)}`);
// S3 officially only supports seconds precision!!!!!
const mtimeSvr = Math.floor(x.LastModified!.valueOf() / 1000.0) * 1000;
let mtimeCli = mtimeSvr;
if (x.Key! in mtimeRecords) {
const m2 = mtimeRecords[x.Key!];
if (m2 !== 0) {
// to be compatible with RClone, we read and store the time in seconds in new version!
if (m2 >= 1000000000000) {
// it's a millsecond, uploaded by old codes..
mtimeCli = m2;
} else {
// it's a second, uploaded by new codes of the plugin from March 24, 2024
mtimeCli = m2 * 1000;
}
}
}
const key = getLocalNoPrefixPath(x.Key!, remotePrefix); // we remove prefix here
const r: Entity = {
key: key, // from s3's repsective, the keyRaw is the key, we will change it in decyption
keyRaw: key,
mtimeSvr: mtimeSvr,
mtimeCli: mtimeCli,
sizeRaw: x.Size!,
size: x.Size!, // from s3's repsective, the sizeRaw is the size, we will change it in decyption
etag: x.ETag,
synthesizedFolder: false,
};
return r;
};
const fromS3HeadObjectToEntity = (
fileOrFolderPathWithRemotePrefix: string,
x: HeadObjectCommandOutput,
remotePrefix: string
) => {
// console.debug(`fromS3HeadObjectToEntity: ${fileOrFolderPathWithRemotePrefix}: ${JSON.stringify(x,null,2)}`);
// S3 officially only supports seconds precision!!!!!
const mtimeSvr = Math.floor(x.LastModified!.valueOf() / 1000.0) * 1000;
let mtimeCli = mtimeSvr;
if (x.Metadata !== undefined) {
const m2 = Math.floor(
parseFloat(x.Metadata.mtime || x.Metadata.MTime || "0")
);
if (m2 !== 0) {
// to be compatible with RClone, we read and store the time in seconds in new version!
if (m2 >= 1000000000000) {
// it's a millsecond, uploaded by old codes..
mtimeCli = m2;
} else {
// it's a second, uploaded by new codes of the plugin from March 24, 2024
mtimeCli = m2 * 1000;
}
}
}
// console.debug(
// `fromS3HeadObjectToEntity, fileOrFolderPathWithRemotePrefix=${fileOrFolderPathWithRemotePrefix}, remotePrefix=${remotePrefix}, x=${JSON.stringify(
// x
// )} `
// );
const key = getLocalNoPrefixPath(
fileOrFolderPathWithRemotePrefix,
remotePrefix
);
// console.debug(`fromS3HeadObjectToEntity, key=${key} after removing prefix`);
return {
key: key,
keyRaw: key,
mtimeSvr: mtimeSvr,
mtimeCli: mtimeCli,
sizeRaw: x.ContentLength,
size: x.ContentLength,
etag: x.ETag,
synthesizedFolder: false,
} as Entity;
};
export class FakeFsS3 extends FakeFs {
s3Config: S3Config;
s3Client: S3Client;
kind: "s3";
synthFoldersCache: Record<string, Entity>;
constructor(s3Config: S3Config) {
super();
this.s3Config = s3Config;
this.s3Client = getS3Client(s3Config);
this.kind = "s3";
this.synthFoldersCache = {};
}
async walk(): Promise<Entity[]> {
const res = (await this._walkFromRoot(this.s3Config.remotePrefix)).filter(
(x) => x.key !== "" && x.key !== "/"
);
return res;
}
/**
* the input key contains basedir (prefix),
* but the result doesn't contain it.
*/
async _walkFromRoot(prefixOfRawKeys: string | undefined) {
const confCmd = {
Bucket: this.s3Config.s3BucketName,
} as ListObjectsV2CommandInput;
if (prefixOfRawKeys !== undefined && prefixOfRawKeys !== "") {
confCmd.Prefix = prefixOfRawKeys;
}
const contents = [] as _Object[];
const mtimeRecords: Record<string, number> = {};
const ctimeRecords: Record<string, number> = {};
const queueHead = new PQueue({
concurrency: this.s3Config.partsConcurrency,
autoStart: true,
});
queueHead.on("error", (error) => {
queueHead.pause();
queueHead.clear();
throw error;
});
let isTruncated = true;
do {
const rsp = await this.s3Client.send(new ListObjectsV2Command(confCmd));
if (rsp.$metadata.httpStatusCode !== 200) {
throw Error("some thing bad while listing remote!");
}
if (rsp.Contents === undefined) {
break;
}
contents.push(...rsp.Contents);
if (this.s3Config.useAccurateMTime) {
// head requests of all objects, love it
for (const content of rsp.Contents) {
queueHead.add(async () => {
const rspHead = await this.s3Client.send(
new HeadObjectCommand({
Bucket: this.s3Config.s3BucketName,
Key: content.Key,
})
);
if (rspHead.$metadata.httpStatusCode !== 200) {
throw Error("some thing bad while heading single object!");
}
if (rspHead.Metadata === undefined) {
// pass
} else {
mtimeRecords[content.Key!] = Math.floor(
parseFloat(
rspHead.Metadata.mtime || rspHead.Metadata.MTime || "0"
)
);
ctimeRecords[content.Key!] = Math.floor(
parseFloat(
rspHead.Metadata.ctime || rspHead.Metadata.CTime || "0"
)
);
}
});
}
}
isTruncated = rsp.IsTruncated ?? false;
confCmd.ContinuationToken = rsp.NextContinuationToken;
if (
isTruncated &&
(confCmd.ContinuationToken === undefined ||
confCmd.ContinuationToken === "")
) {
throw Error("isTruncated is true but no continuationToken provided");
}
} while (isTruncated);
// wait for any head requests
await queueHead.onIdle();
// ensemble fake rsp
// in the end, we need to transform the response list
// back to the local contents-alike list
const res: Entity[] = [];
const realEnrities = new Set<string>();
for (const remoteObj of contents) {
const remoteEntity = fromS3ObjectToEntity(
remoteObj,
this.s3Config.remotePrefix ?? "",
mtimeRecords,
ctimeRecords
);
realEnrities.add(remoteEntity.key!);
res.push(remoteEntity);
for (const f of getFolderLevels(remoteEntity.key!, true)) {
if (realEnrities.has(f)) {
delete this.synthFoldersCache[f];
continue;
}
if (
!this.synthFoldersCache.hasOwnProperty(f) ||
remoteEntity.mtimeSvr! >= this.synthFoldersCache[f].mtimeSvr!
) {
this.synthFoldersCache[f] = {
key: f,
keyRaw: f,
size: 0,
sizeRaw: 0,
sizeEnc: 0,
mtimeSvr: remoteEntity.mtimeSvr,
mtimeSvrFmt: remoteEntity.mtimeSvrFmt,
mtimeCli: remoteEntity.mtimeCli,
mtimeCliFmt: remoteEntity.mtimeCliFmt,
synthesizedFolder: true,
};
}
}
}
for (const key of Object.keys(this.synthFoldersCache)) {
res.push(this.synthFoldersCache[key]);
}
return res;
}
async stat(key: string): Promise<Entity> {
if (this.synthFoldersCache.hasOwnProperty(key)) {
return this.synthFoldersCache[key];
}
let keyFullPath = key;
keyFullPath = getRemoteWithPrefixPath(
keyFullPath,
this.s3Config.remotePrefix ?? ""
);
return await this._statFromRoot(keyFullPath);
}
/**
* the input key contains basedir (prefix),
* but the result doesn't contain it.
*/
async _statFromRoot(key: string): Promise<Entity> {
if (
this.s3Config.remotePrefix !== undefined &&
this.s3Config.remotePrefix !== "" &&
!key.startsWith(this.s3Config.remotePrefix)
) {
throw Error(`_statFromRoot should only accept prefix-ed path`);
}
const res = await this.s3Client.send(
new HeadObjectCommand({
Bucket: this.s3Config.s3BucketName,
Key: key,
})
);
return fromS3HeadObjectToEntity(key, res, this.s3Config.remotePrefix ?? "");
}
async mkdir(key: string, mtime?: number, ctime?: number): Promise<Entity> {
if (!key.endsWith("/")) {
throw new Error(`You should not call mkdir on ${key}!`);
}
const generateFolderObject = this.s3Config.generateFolderObject ?? false;
if (!generateFolderObject) {
const synth = {
key: key,
keyRaw: key,
size: 0,
sizeRaw: 0,
sizeEnc: 0,
mtimeSvr: mtime,
mtimeCli: mtime,
synthesizedFolder: true,
};
this.synthFoldersCache[key] = synth;
return synth;
}
const uploadFile = getRemoteWithPrefixPath(
key,
this.s3Config.remotePrefix ?? ""
);
return await this._mkdirFromRoot(uploadFile, mtime, ctime);
}
async _mkdirFromRoot(key: string, mtime?: number, ctime?: number) {
if (
this.s3Config.remotePrefix !== undefined &&
this.s3Config.remotePrefix !== "" &&
!key.startsWith(this.s3Config.remotePrefix)
) {
throw Error(`_mkdirFromRoot should only accept prefix-ed path`);
}
const contentType = DEFAULT_CONTENT_TYPE;
const p: PutObjectCommandInput = {
Bucket: this.s3Config.s3BucketName,
Key: key,
Body: "",
ContentType: contentType,
ContentLength: 0, // interesting we need to set this to avoid the warning
};
const metadata: Record<string, string> = {};
if (mtime !== undefined && mtime !== 0) {
metadata["MTime"] = `${mtime / 1000.0}`;
}
if (ctime !== undefined && ctime !== 0) {
metadata["CTime"] = `${ctime / 1000.0}`;
}
if (Object.keys(metadata).length > 0) {
p["Metadata"] = metadata;
}
await this.s3Client.send(new PutObjectCommand(p));
return await this._statFromRoot(key);
}
async writeFile(
key: string,
content: ArrayBuffer,
mtime: number,
ctime: number
): Promise<Entity> {
const uploadFile = getRemoteWithPrefixPath(
key,
this.s3Config.remotePrefix ?? ""
);
const res = await this._writeFileFromRoot(
uploadFile,
content,
mtime,
ctime
);
return res;
}
/**
* the input key contains basedir (prefix),
* but the result doesn't contain it.
*/
async _writeFileFromRoot(
key: string,
content: ArrayBuffer,
mtime: number,
ctime: number
): Promise<Entity> {
if (
this.s3Config.remotePrefix !== undefined &&
this.s3Config.remotePrefix !== "" &&
!key.startsWith(this.s3Config.remotePrefix)
) {
throw Error(`_writeFileFromRoot should only accept prefix-ed path`);
}
const bytesIn5MB = 5242880;
const body = new Uint8Array(content);
let contentType = DEFAULT_CONTENT_TYPE;
contentType =
mime.contentType(mime.lookup(key) || DEFAULT_CONTENT_TYPE) ||
DEFAULT_CONTENT_TYPE;
const upload = new Upload({
client: this.s3Client,
queueSize: this.s3Config.partsConcurrency, // concurrency
partSize: bytesIn5MB, // minimal 5MB by default
leavePartsOnError: false,
params: {
Bucket: this.s3Config.s3BucketName,
Key: key,
Body: body,
ContentType: contentType,
Metadata: {
MTime: `${mtime / 1000.0}`,
CTime: `${ctime / 1000.0}`,
},
},
});
upload.on("httpUploadProgress", (progress) => {
// console.info(progress);
});
await upload.done();
return await this._statFromRoot(key);
}
async readFile(key: string): Promise<ArrayBuffer> {
if (key.endsWith("/")) {
throw new Error(`you should not call readFile on folder ${key}`);
}
const downloadFile = getRemoteWithPrefixPath(
key,
this.s3Config.remotePrefix ?? ""
);
return await this._readFileFromRoot(downloadFile);
}
async _readFileFromRoot(key: string): Promise<ArrayBuffer> {
if (
this.s3Config.remotePrefix !== undefined &&
this.s3Config.remotePrefix !== "" &&
!key.startsWith(this.s3Config.remotePrefix)
) {
throw Error(`_readFileFromRoot should only accept prefix-ed path`);
}
const data = await this.s3Client.send(
new GetObjectCommand({
Bucket: this.s3Config.s3BucketName,
Key: key,
})
);
const bodyContents = await getObjectBodyToArrayBuffer(data.Body);
return bodyContents;
}
async rm(key: string): Promise<void> {
if (key === "/") {
return;
}
if (this.synthFoldersCache.hasOwnProperty(key)) {
delete this.synthFoldersCache[key];
return;
}
const remoteFileName = getRemoteWithPrefixPath(
key,
this.s3Config.remotePrefix ?? ""
);
await this.s3Client.send(
new DeleteObjectCommand({
Bucket: this.s3Config.s3BucketName,
Key: remoteFileName,
})
);
// TODO: do we need to delete folder recursively?
// maybe we should not
// because the outer sync algorithm should do that
// (await this._walkFromRoot(remoteFileName)).map(...)
}
async checkConnect(callbackFunc?: any): Promise<boolean> {
try {
// TODO: no universal way now, just check this in connectivity
if (Platform.isIosApp && this.s3Config.s3Endpoint.startsWith("http://")) {
throw Error(
`Your s3 endpoint could only be https, not http, because of the iOS restriction.`
);
}
// const results = await this.s3Client.send(
// new HeadBucketCommand({ Bucket: this.s3Config.s3BucketName })
// );
// very simplified version of listing objects
const confCmd = {
Bucket: this.s3Config.s3BucketName,
} as ListObjectsV2CommandInput;
const results = await this.s3Client.send(
new ListObjectsV2Command(confCmd)
);
if (
results === undefined ||
results.$metadata === undefined ||
results.$metadata.httpStatusCode === undefined
) {
const err = "results or $metadata or httStatusCode is undefined";
console.debug(err);
if (callbackFunc !== undefined) {
callbackFunc(err);
}
return false;
}
return results.$metadata.httpStatusCode === 200;
} catch (err: any) {
console.debug(err);
if (callbackFunc !== undefined) {
if (this.s3Config.s3Endpoint.contains(this.s3Config.s3BucketName)) {
const err2 = new AggregateError([
err,
new Error(
"Maybe you've included the bucket name inside the endpoint setting. Please remove the bucket name and try again."
),
]);
callbackFunc(err2);
} else {
callbackFunc(err);
}
}
return false;
}
}
async getUserDisplayName(): Promise<string> {
throw new Error("Method not implemented.");
}
async revokeAuth() {
throw new Error("Method not implemented.");
}
}
+495
View File
@@ -0,0 +1,495 @@
import { getReasonPhrase } from "http-status-codes/build/cjs/utils-functions";
import { Buffer } from "buffer";
import cloneDeep from "lodash/cloneDeep";
import { Queue } from "@fyears/tsqueue";
import chunk from "lodash/chunk";
import flatten from "lodash/flatten";
import { Platform, requestUrl } from "obsidian";
import { FakeFs } from "./fsAll";
import { bufferToArrayBuffer } from "./misc";
import { Entity, WebdavConfig } from "./baseTypes";
import { VALID_REQURL } from "./baseTypesObs";
import type {
FileStat,
WebDAVClient,
RequestOptionsWithState,
// Response,
// ResponseDataDetailed,
} from "webdav";
/**
* https://stackoverflow.com/questions/32850898/how-to-check-if-a-string-has-any-non-iso-8859-1-characters-with-javascript
* @param str
* @returns true if all are iso 8859 1 chars
*/
function onlyAscii(str: string) {
return !/[^\u0000-\u00ff]/g.test(str);
}
/**
* https://stackoverflow.com/questions/12539574/
* @param obj
* @returns
*/
function objKeyToLower(obj: Record<string, string>) {
return Object.fromEntries(
Object.entries(obj).map(([k, v]) => [k.toLowerCase(), v])
);
}
// @ts-ignore
import { getPatcher } from "webdav/dist/web/index.js";
if (VALID_REQURL) {
getPatcher().patch(
"request",
async (options: RequestOptionsWithState): Promise<Response> => {
const transformedHeaders = objKeyToLower({ ...options.headers });
delete transformedHeaders["host"];
delete transformedHeaders["content-length"];
const reqContentType =
transformedHeaders["accept"] ?? transformedHeaders["content-type"];
const retractedHeaders = { ...transformedHeaders };
if (retractedHeaders.hasOwnProperty("authorization")) {
retractedHeaders["authorization"] = "<retracted>";
}
console.debug(`before request:`);
console.debug(`url: ${options.url}`);
console.debug(`method: ${options.method}`);
console.debug(`headers: ${JSON.stringify(retractedHeaders, null, 2)}`);
console.debug(`reqContentType: ${reqContentType}`);
let r = await requestUrl({
url: options.url,
method: options.method,
body: options.data as string | ArrayBuffer,
headers: transformedHeaders,
contentType: reqContentType,
throw: false,
});
if (
r.status === 401 &&
Platform.isIosApp &&
!options.url.endsWith("/") &&
!options.url.endsWith(".md") &&
options.method.toUpperCase() === "PROPFIND"
) {
// don't ask me why,
// some webdav servers have some mysterious behaviours,
// if a folder doesn't exist without slash, the servers return 401 instead of 404
// here is a dirty hack that works
console.debug(`so we have 401, try appending request url with slash`);
r = await requestUrl({
url: `${options.url}/`,
method: options.method,
body: options.data as string | ArrayBuffer,
headers: transformedHeaders,
contentType: reqContentType,
throw: false,
});
}
console.debug(`after request:`);
const rspHeaders = objKeyToLower({ ...r.headers });
console.debug(`rspHeaders: ${JSON.stringify(rspHeaders, null, 2)}`);
for (let key in rspHeaders) {
if (rspHeaders.hasOwnProperty(key)) {
// avoid the error:
// Failed to read the 'headers' property from 'ResponseInit': String contains non ISO-8859-1 code point.
// const possibleNonAscii = [
// "Content-Disposition",
// "X-Accel-Redirect",
// "X-Outfilename",
// "X-Sendfile"
// ];
// for (const p of possibleNonAscii) {
// if (key === p || key === p.toLowerCase()) {
// rspHeaders[key] = encodeURIComponent(rspHeaders[key]);
// }
// }
if (!onlyAscii(rspHeaders[key])) {
console.debug(`rspHeaders[key] needs encode: ${key}`);
rspHeaders[key] = encodeURIComponent(rspHeaders[key]);
}
}
}
let r2: Response | undefined = undefined;
const statusText = getReasonPhrase(r.status);
console.debug(`statusText: ${statusText}`);
if ([101, 103, 204, 205, 304].includes(r.status)) {
// A null body status is a status that is 101, 103, 204, 205, or 304.
// https://fetch.spec.whatwg.org/#statuses
// fix this: Failed to construct 'Response': Response with null body status cannot have body
r2 = new Response(null, {
status: r.status,
statusText: statusText,
headers: rspHeaders,
});
} else {
r2 = new Response(r.arrayBuffer, {
status: r.status,
statusText: statusText,
headers: rspHeaders,
});
}
return r2;
}
);
}
// @ts-ignore
import { AuthType, BufferLike, createClient } from "webdav/dist/web/index.js";
export const DEFAULT_WEBDAV_CONFIG = {
address: "",
username: "",
password: "",
authType: "basic",
manualRecursive: true,
depth: "manual_1",
remoteBaseDir: "",
} as WebdavConfig;
const getWebdavPath = (fileOrFolderPath: string, remoteBaseDir: string) => {
let key = fileOrFolderPath;
if (fileOrFolderPath === "/" || fileOrFolderPath === "") {
// special
key = `/${remoteBaseDir}/`;
} else if (fileOrFolderPath.startsWith("/")) {
console.warn(
`why the path ${fileOrFolderPath} starts with '/'? but we just go on.`
);
key = `/${remoteBaseDir}${fileOrFolderPath}`;
} else {
key = `/${remoteBaseDir}/${fileOrFolderPath}`;
}
return key;
};
const getNormPath = (fileOrFolderPath: string, remoteBaseDir: string) => {
if (
!(
fileOrFolderPath === `/${remoteBaseDir}` ||
fileOrFolderPath.startsWith(`/${remoteBaseDir}/`)
)
) {
throw Error(
`"${fileOrFolderPath}" doesn't starts with "/${remoteBaseDir}/"`
);
}
return fileOrFolderPath.slice(`/${remoteBaseDir}/`.length);
};
const fromWebdavItemToEntity = (x: FileStat, remoteBaseDir: string): Entity => {
let key = getNormPath(x.filename, remoteBaseDir);
if (x.type === "directory" && !key.endsWith("/")) {
key = `${key}/`;
}
const mtimeSvr = Date.parse(x.lastmod).valueOf();
return {
key: key,
keyRaw: key,
mtimeSvr: mtimeSvr,
mtimeCli: mtimeSvr, // TODO: no universal way to set mtime in webdav
size: x.size,
sizeRaw: x.size,
};
};
export class FakeFsWebdav extends FakeFs {
kind: "webdav";
webdavConfig: WebdavConfig;
remoteBaseDir: string;
client!: WebDAVClient;
vaultFolderExists: boolean;
saveUpdatedConfigFunc: () => Promise<any>;
constructor(
webdavConfig: WebdavConfig,
vaultName: string,
saveUpdatedConfigFunc: () => Promise<any>
) {
super();
this.kind = "webdav";
this.webdavConfig = cloneDeep(webdavConfig);
this.webdavConfig.address = encodeURI(this.webdavConfig.address);
this.remoteBaseDir = this.webdavConfig.remoteBaseDir || vaultName || "";
this.vaultFolderExists = false;
this.saveUpdatedConfigFunc = saveUpdatedConfigFunc;
}
async _init() {
// init client if not inited
if (this.client !== undefined) {
return;
}
if (Platform.isIosApp && !this.webdavConfig.address.startsWith("https")) {
throw Error(
`Your webdav address could only be https, not http, because of the iOS restriction.`
);
}
const headers = {
"Cache-Control": "no-cache",
};
if (
this.webdavConfig.username !== "" &&
this.webdavConfig.password !== ""
) {
this.client = createClient(this.webdavConfig.address, {
username: this.webdavConfig.username,
password: this.webdavConfig.password,
headers: headers,
authType:
this.webdavConfig.authType === "digest"
? AuthType.Digest
: AuthType.Password,
});
} else {
console.info("no password");
this.client = createClient(this.webdavConfig.address, {
headers: headers,
});
}
// check vault folder
if (this.vaultFolderExists) {
// pass
} else {
const res = await this.client.exists(`/${this.remoteBaseDir}/`);
if (res) {
// console.info("remote vault folder exits!");
this.vaultFolderExists = true;
} else {
console.info("remote vault folder not exists, creating");
await this.client.createDirectory(`/${this.remoteBaseDir}/`);
console.info("remote vault folder created!");
this.vaultFolderExists = true;
}
}
// adjust depth parameter
if (
this.webdavConfig.depth === "auto" ||
this.webdavConfig.depth === "auto_1" ||
this.webdavConfig.depth === "auto_infinity" ||
this.webdavConfig.depth === "auto_unknown"
) {
this.webdavConfig.depth = "manual_1";
this.webdavConfig.manualRecursive = true;
if (this.saveUpdatedConfigFunc !== undefined) {
await this.saveUpdatedConfigFunc();
console.info(
`webdav depth="auto_???" is changed to ${this.webdavConfig.depth}`
);
}
}
}
async walk(): Promise<Entity[]> {
await this._init();
let contents = [] as FileStat[];
if (
this.webdavConfig.depth === "auto" ||
this.webdavConfig.depth === "auto_unknown" ||
this.webdavConfig.depth === "auto_1" ||
this.webdavConfig.depth === "auto_infinity" /* don't trust auto now */ ||
this.webdavConfig.depth === "manual_1"
) {
// the remote doesn't support infinity propfind,
// we need to do a bfs here
const q = new Queue([`/${this.remoteBaseDir}`]);
const CHUNK_SIZE = 10;
while (q.length > 0) {
const itemsToFetch: string[] = [];
while (q.length > 0) {
itemsToFetch.push(q.pop()!);
}
const itemsToFetchChunks = chunk(itemsToFetch, CHUNK_SIZE);
// console.debug(itemsToFetchChunks);
const subContents = [] as FileStat[];
for (const singleChunk of itemsToFetchChunks) {
const r = singleChunk.map((x) => {
return this.client.getDirectoryContents(x, {
deep: false,
details: false /* no need for verbose details here */,
// TODO: to support .obsidian,
// we need to load all files including dot,
// anyway to reduce the resources?
// glob: "/**" /* avoid dot files by using glob */,
}) as Promise<FileStat[]>;
});
const r2 = flatten(await Promise.all(r));
subContents.push(...r2);
}
for (let i = 0; i < subContents.length; ++i) {
const f = subContents[i];
contents.push(f);
if (f.type === "directory") {
q.push(f.filename);
}
}
}
} else {
// the remote supports infinity propfind
contents = (await this.client.getDirectoryContents(
`/${this.remoteBaseDir}`,
{
deep: true,
details: false /* no need for verbose details here */,
// TODO: to support .obsidian,
// we need to load all files including dot,
// anyway to reduce the resources?
// glob: "/**" /* avoid dot files by using glob */,
}
)) as FileStat[];
}
return contents.map((x) => fromWebdavItemToEntity(x, this.remoteBaseDir));
}
async stat(key: string): Promise<Entity> {
await this._init();
const fullPath = getWebdavPath(key, this.remoteBaseDir);
return await this._statFromRoot(fullPath);
}
async _statFromRoot(key: string): Promise<Entity> {
const res = (await this.client.stat(key, {
details: false,
})) as FileStat;
return fromWebdavItemToEntity(res, this.remoteBaseDir);
}
async mkdir(key: string, mtime?: number, ctime?: number): Promise<Entity> {
if (!key.endsWith("/")) {
throw Error(`you should not call mkdir on ${key}`);
}
await this._init();
const uploadFile = getWebdavPath(key, this.remoteBaseDir);
return await this._mkdirFromRoot(uploadFile, mtime, ctime);
}
async _mkdirFromRoot(
key: string,
mtime?: number,
ctime?: number
): Promise<Entity> {
await this.client.createDirectory(key, {
recursive: true,
});
return await this._statFromRoot(key);
}
async writeFile(
key: string,
content: ArrayBuffer,
mtime: number,
ctime: number
): Promise<Entity> {
if (key.endsWith("/")) {
throw Error(`you should not call writeFile on ${key}`);
}
await this._init();
const uploadFile = getWebdavPath(key, this.remoteBaseDir);
return await this._writeFileFromRoot(uploadFile, content, mtime, ctime);
}
async _writeFileFromRoot(
key: string,
content: ArrayBuffer,
mtime: number,
ctime: number
): Promise<Entity> {
await this.client.putFileContents(key, content, {
overwrite: true,
onUploadProgress: (progress: any) => {
console.info(`Uploaded ${progress.loaded} bytes of ${progress.total}`);
},
});
return await this._statFromRoot(key);
}
async readFile(key: string): Promise<ArrayBuffer> {
if (key.endsWith("/")) {
throw Error(`you should not call readFile on ${key}`);
}
await this._init();
const downloadFile = getWebdavPath(key, this.remoteBaseDir);
return await this._readFileFromRoot(downloadFile);
}
async _readFileFromRoot(key: string): Promise<ArrayBuffer> {
const buff = (await this.client.getFileContents(key)) as BufferLike;
if (buff instanceof ArrayBuffer) {
return buff;
} else if (buff instanceof Buffer) {
return bufferToArrayBuffer(buff);
}
throw Error(`unexpected file content result with type ${typeof buff}`);
}
async rm(key: string): Promise<void> {
if (key === "/") {
return;
}
await this._init();
try {
const remoteFileName = getWebdavPath(key, this.remoteBaseDir);
await this.client.deleteFile(remoteFileName);
// console.info(`delete ${remoteFileName} succeeded`);
} catch (err) {
console.error("some error while deleting");
console.error(err);
}
}
async checkConnect(callbackFunc?: any): Promise<boolean> {
if (
!(
this.webdavConfig.address.startsWith("http://") ||
this.webdavConfig.address.startsWith("https://")
)
) {
const err =
"Error: the url should start with http(s):// but it does not!";
console.error(err);
if (callbackFunc !== undefined) {
callbackFunc(err);
}
return false;
}
try {
await this._init();
const results = await this._statFromRoot(`/${this.remoteBaseDir}/`);
if (results === undefined) {
const err = "results is undefined";
console.error(err);
callbackFunc?.(err);
return false;
}
return true;
} catch (err) {
console.error(err);
callbackFunc?.(err);
return false;
}
}
async getUserDisplayName(): Promise<string> {
throw new Error("Method not implemented.");
}
async revokeAuth() {
throw new Error("Method not implemented.");
}
}
+238
View File
@@ -0,0 +1,238 @@
import { cloneDeep, isEqual } from "lodash";
import { DEFAULT_CONTENT_TYPE, Entity, WebdisConfig } from "./baseTypes";
import { FakeFs } from "./fsAll";
export const DEFAULT_WEBDIS_CONFIG: WebdisConfig = {
address: "",
username: "",
password: "",
remoteBaseDir: "",
};
const getWebdisPath = (fileOrFolderPath: string, remoteBaseDir: string) => {
let key = fileOrFolderPath;
if (fileOrFolderPath === "/" || fileOrFolderPath === "") {
// special
key = `${remoteBaseDir}`;
} else if (fileOrFolderPath.startsWith("/")) {
console.warn(
`why the path ${fileOrFolderPath} starts with '/'? but we just go on.`
);
key = `${remoteBaseDir}${fileOrFolderPath}`;
} else {
key = `${remoteBaseDir}/${fileOrFolderPath}`;
}
return `rs:fs:v1:${encodeURIComponent(key)}`; // we should encode them!!!!
};
export const getOrigPath = (fullKey: string, remoteBaseDir: string) => {
const fullKeyDecoded = decodeURIComponent(fullKey);
const prefix = `rs:fs:v1:${remoteBaseDir}/`;
// console.debug(`prefix=${prefix}`);
const suffix1 = ":meta";
const suffix2 = ":content";
if (!fullKeyDecoded.startsWith(prefix)) {
throw Error(`you should not call getOrigEntity on ${fullKey}`);
}
let realKey = fullKeyDecoded.slice(prefix.length);
// console.debug(`realKey=${realKey}`);
if (realKey.endsWith(suffix1)) {
realKey = realKey.slice(0, -suffix1.length);
// console.debug(`realKey=${realKey}`);
} else if (realKey.endsWith(suffix2)) {
realKey = realKey.slice(0, -suffix2.length);
// console.debug(`realKey=${realKey}`);
}
// console.debug(`fullKey=${fullKey}, realKey=${realKey}`);
return realKey;
};
export class FakeFsWebdis extends FakeFs {
kind: "webdis";
webdisConfig: WebdisConfig;
remoteBaseDir: string;
saveUpdatedConfigFunc: () => Promise<any>;
constructor(
webdisConfig: WebdisConfig,
vaultName: string,
saveUpdatedConfigFunc: () => Promise<any>
) {
super();
this.kind = "webdis";
this.webdisConfig = webdisConfig;
this.remoteBaseDir = this.webdisConfig.remoteBaseDir || vaultName || "";
this.saveUpdatedConfigFunc = saveUpdatedConfigFunc;
}
async _fetchCommand(
method: "GET" | "POST" | "PUT",
urlPath: string,
content?: ArrayBuffer
) {
const address = this.webdisConfig.address;
if (!address.startsWith(`https://`) && !address.startsWith(`http://`)) {
throw Error(
`your webdis server address should start with https:// or http://`
);
}
if (address.endsWith("/")) {
throw Error(`your webdis server should not ends with /`);
}
if (content !== undefined && method !== "PUT") {
throw Error(`you can only "POST" ArrayBuffer, not using other methods`);
}
const fullUrl = `${address}/${urlPath}`;
// console.debug(`fullUrl=${fullUrl}`)
const username = this.webdisConfig.username ?? "";
const password = this.webdisConfig.password ?? "";
if (username !== "" && password !== "") {
return await fetch(fullUrl, {
method: method,
headers: {
Authorization: "Basic " + btoa(username + ":" + password),
},
body: content,
});
} else if (username === "" && password === "") {
return await fetch(fullUrl, {
method: method,
body: content,
});
} else {
throw Error(
`your username and password should both be empty or not empty!`
);
}
}
async walk(): Promise<Entity[]> {
let cursor = "0";
const res: Entity[] = [];
do {
const command = `SCAN/${cursor}/MATCH/rs:fs:v1:*:meta/COUNT/1000`;
const rsp = (await (await this._fetchCommand("GET", command)).json())[
"SCAN"
];
// console.debug(rsp);
cursor = rsp[0];
for (const fullKeyWithMeta of rsp[1]) {
const realKey = getOrigPath(fullKeyWithMeta, this.remoteBaseDir);
res.push(await this.stat(realKey));
}
} while (cursor !== "0");
// console.debug(`walk res:`);
// console.debug(res);
return res;
}
async stat(key: string): Promise<Entity> {
const fullKey = getWebdisPath(key, this.remoteBaseDir);
return await this._statFromRaw(fullKey);
}
async _statFromRaw(key: string): Promise<Entity> {
// console.debug(`_statFromRaw on ${key}`);
const command = `HGETALL/${key}:meta`;
const rsp = (await (await this._fetchCommand("GET", command)).json())[
"HGETALL"
];
// console.debug(`rsp: ${JSON.stringify(rsp, null, 2)}`);
if (isEqual(rsp, {})) {
// empty!
throw Error(`key ${key} doesn't exist!`);
}
const realKey = getOrigPath(key, this.remoteBaseDir);
return {
key: realKey,
keyRaw: realKey,
mtimeCli: parseInt(rsp["mtime"]),
mtimeSvr: parseInt(rsp["mtime"]),
size: parseInt(rsp["size"]),
sizeRaw: parseInt(rsp["size"]),
};
}
async mkdir(key: string, mtime?: number, ctime?: number): Promise<Entity> {
let command = `HSET/${getWebdisPath(key, this.remoteBaseDir)}:meta/size/0`;
if (mtime !== undefined && mtime !== 0) {
command = `${command}/mtime/${mtime}`;
}
if (ctime !== undefined && ctime !== 0) {
command = `${command}/ctime/${ctime}`;
}
const rsp = (await (await this._fetchCommand("GET", command)).json())[
"HSET"
];
return await this.stat(key);
}
async writeFile(
key: string,
content: ArrayBuffer,
mtime: number,
ctime: number
): Promise<Entity> {
const fullKey = getWebdisPath(key, this.remoteBaseDir);
// meta
let command1 = `HSET/${fullKey}:meta/size/${content.byteLength}`;
if (mtime !== undefined && mtime !== 0) {
command1 = `${command1}/mtime/${mtime}`;
}
if (ctime !== undefined && ctime !== 0) {
command1 = `${command1}/ctime/${ctime}`;
}
const rsp1 = (await (await this._fetchCommand("GET", command1)).json())[
"HSET"
];
// content
const command2 = `SET/${fullKey}:content`;
const rsp2 = (
await (await this._fetchCommand("PUT", command2, content)).json()
)["SET"];
// fetch meta
return await this.stat(key);
}
async readFile(key: string): Promise<ArrayBuffer> {
const fullKey = getWebdisPath(key, this.remoteBaseDir);
const command = `GET/${fullKey}:content?type=${DEFAULT_CONTENT_TYPE}`;
const rsp = await (await this._fetchCommand("GET", command)).arrayBuffer();
return rsp;
}
async rm(key: string): Promise<void> {
const fullKey = getWebdisPath(key, this.remoteBaseDir);
const command = `DEL/${fullKey}:meta/${fullKey}:content`;
const rsp = (await (await this._fetchCommand("PUT", command)).json())[
"DEL"
];
}
async checkConnect(callbackFunc?: any): Promise<boolean> {
try {
const k = await (
await this._fetchCommand("GET", "PING/helloworld")
).json();
return isEqual(k, { PING: "helloworld" });
} catch (err: any) {
console.error(err);
callbackFunc?.(err);
return false;
}
}
async getUserDisplayName(): Promise<string> {
return this.webdisConfig.username || "<no usernme>";
}
async revokeAuth(): Promise<any> {
throw new Error("Method not implemented.");
}
}
+1 -1
View File
@@ -7,7 +7,7 @@ import {
RemotelySavePluginSettings,
QRExportType,
} from "./baseTypes";
import { getShrinkedSettings } from "./remoteForOnedrive";
import { getShrinkedSettings } from "./fsOnedrive";
export const exportQrCodeUri = async (
settings: RemotelySavePluginSettings,
+27 -13
View File
@@ -9,9 +9,9 @@
"syncrun_syncingribbon": "{{pluginName}}: syncing from {{triggerSource}}",
"syncrun_step0": "0/8 Remotely Save is running in dry mode, thus not actual file changes would happen.",
"syncrun_step1": "1/8 Remotely Save is preparing ({{serviceType}})",
"syncrun_step2": "2/8 Starting to fetch remote meta data.",
"syncrun_step3": "3/8 Checking password correct or not.",
"syncrun_step2": "2/8 Starting to fetch remote meta data and check password.",
"syncrun_passworderr": "Something goes wrong while checking password.",
"syncrun_step3": "3/8 Startting to process remote data.",
"syncrun_step4": "4/8 Starting to fetch local meta data.",
"syncrun_step5": "5/8 Starting to fetch local prev sync data.",
"syncrun_step6": "6/8 Starting to generate sync plan.",
@@ -57,7 +57,6 @@
"statusbar_lastsync_label": "Last successful Sync on {{date}}",
"statusbar_lastsync_never": "Never Synced",
"statusbar_lastsync_never_label": "Never Synced before",
"modal_password_title": "Hold on and PLEASE READ ON...",
"modal_password_shortdesc": "If the field is not empty, files would be encrypted locally before being uploaded.\nIf the field is empty, then files would be uploaded without encryption.",
"modal_password_attn1": "Attention 1/5: The vault name is NOT encrypted. The plugin creates a folder with the vault name on some remote services.",
@@ -134,17 +133,13 @@
"settings_runoncestartup_1sec": "sync once after 1 second of start up",
"settings_runoncestartup_10sec": "sync once after 10 seconds of start up",
"settings_runoncestartup_30sec": "sync once after 30 seconds of start up",
"settings_saverun": "Sync On Save (experimental)",
"settings_saverun_desc": "A sync will be triggered if a file save action happened within a few seconds. Please pay attention that syncing is potentially a heavy action and battery may be impacted. (May need to reload the plugin or restart Obsidian after changing)",
"settings_saverun_notset": "(not set)",
"settings_saverun_1sec": "check every 1 second",
"settings_saverun_5sec": "check every 5 seconds",
"settings_saverun_10sec": "check every 10 seconds (recommended)",
"settings_saverun_1min": "check every 1 minute",
"settings_synconsave": "Sync On Save (experimental)",
"settings_synconsave_desc": "If you change your file, the plugin tries to trigger a sync.",
"settings_synconsave_disable": "Disable (default)",
"settings_synconsave_enable": "Enable",
"settings_skiplargefiles": "Skip Large Files",
"settings_skiplargefiles_desc": "Skip files with sizes larger than the threshold. Here 1 MB = 10^6 bytes.",
"settings_skiplargefiles_notset": "(not set)",
"settings_ignorepaths": "Regex Of Paths To Ignore",
"settings_ignorepaths_desc": "Regex of paths of folders or files to ignore. One regex per line. The path is relative to the vault root without leading slash.",
"settings_enablestatusbar_info": "Show Last Successful Sync In Status Bar",
@@ -154,7 +149,6 @@
"settings_resetstatusbar_time_desc": "Reset last successful sync time.",
"settings_resetstatusbar_button": "Reset",
"settings_resetstatusbar_notice": "Reset done!",
"settings_checkonnectivity": "Check Connectivity",
"settings_checkonnectivity_desc": "Check connectivity.",
"settings_checkonnectivity_button": "Check",
@@ -187,6 +181,12 @@
"settings_s3_accuratemtime_desc": "Read the uploaded accurate last modified time for better sync algorithm. But it causes extra api requests / time / money to the S3 endpoint.",
"settings_s3_urlstyle": "S3 URL style",
"settings_s3_urlstyle_desc": "Whether to force path-style URLs for S3 objects (e.g., https://s3.amazonaws.com/*/ instead of https://*.s3.amazonaws.com/).",
"settings_s3_reverse_proxy_no_sign_url": "S3 Reverse Proxy (No Sign) Url (experimental)",
"settings_s3_reverse_proxy_no_sign_url_desc": "S3 reverse proxy url without signature. This is useful if you use a revers proxy but do not change the original credential signature. No http(s):// prefix. Leave it blank if you don't know what it is.",
"settings_s3_generatefolderobject": "Generate Folder Object Or Not",
"settings_s3_generatefolderobject_desc": "S3 doesn't have \"real\" folder. If you set \"Generate\" here (or use old version), the plugin will upload a zero-byte object endding with \"/\" to represent the folder. In the new version, the plugin skips generating folder object by default.",
"settings_s3_generatefolderobject_notgenerate": "Not generate (default)",
"settings_s3_generatefolderobject_generate": "Generate",
"settings_s3_connect_succ": "Great! The bucket can be accessed.",
"settings_s3_connect_fail": "The S3 bucket cannot be reached.",
"settings_dropbox": "Remote For Dropbox",
@@ -240,12 +240,25 @@
"settings_webdav_connect_succ": "Great! The webdav server can be accessed.",
"settings_webdav_connect_fail": "The webdav server cannot be reached (possible to be any of address/username/password/authtype errors).",
"settings_webdav_connect_fail_withcors": "The webdav server cannot be reached (possible to be any of address/username/password/authtype/CORS errors).",
"settings_webdis": "Remote For Webdis",
"settings_webdis_disclaimer1": "Disclaimer: This app is NOT an official Redis® Ltd / Redis® OSS / Webdis product. Redis is a registered trademark of Redis Ltd.",
"settings_webdis_disclaimer2": "Disclaimer: The information is stored locally. Other malicious/harmful/faulty plugins may read the info. If you see any unintentional access to your Webdis server, please immediately change the username and password.",
"settings_webdis_folder": "We will store the value with keys prefixed by :{{remoteBaseDir}} on your server.",
"settings_webdis_addr": "Server Address",
"settings_webdis_addr_desc": "Server address.",
"settings_webdis_user": "Username",
"settings_webdis_user_desc": "Username. Attention: the username and other info are saved locally.",
"settings_webdis_password": "Password",
"settings_webdis_password_desc": "Password. Attention: the password and other info are saved locally.",
"settings_webdis_connect_succ": "Great! The Webdis server can be accessed.",
"settings_webdis_connect_fail": "The Webdis server cannot be reached (possible to be any of address/username/password errors).",
"settings_chooseservice": "Choose A Remote Service",
"settings_chooseservice_desc": "Start here. What service are you connecting to? S3, Dropbox, Webdav, or OneDrive for personal?",
"settings_chooseservice_desc": "Start here. What service are you connecting to? S3, Dropbox, Webdav, OneDrive for personal, or Webdis?",
"settings_chooseservice_s3": "S3 or compatible",
"settings_chooseservice_dropbox": "Dropbox",
"settings_chooseservice_webdav": "Webdav",
"settings_chooseservice_onedrive": "OneDrive for personal",
"settings_chooseservice_webdis": "Webdis (HTTP for Redis®)",
"settings_adv": "Advanced Settings",
"settings_concurrency": "Concurrency",
"settings_concurrency_desc": "How many files do you want to download or upload in parallel at most? By default it's set to 5. If you meet any problems such as rate limit, you can reduce the concurrency to a lower value.",
@@ -323,6 +336,7 @@
"settings_resetcache_desc": "Reset local internal caches/databases (for debugging purposes). You would want to reload the plugin after resetting this. This option will not empty the {s3, password...} settings.",
"settings_resetcache_button": "Reset",
"settings_resetcache_notice": "Local internal cache/databases deleted. Please manually reload the plugin.",
"syncalgov3_title": "Remotely Save has HUGE updates on the sync algorithm",
"syncalgov3_texts": "Welcome to use Remotely Save!\nFrom this version, a new algorithm has been developed:\n<ul><li>More robust deletion sync,</li><li>minimal conflict handling,</li><li>no meta data uploaded any more,</li><li>deletion / modification protection,</li><li>backup mode</li><li>new encryption method</li><li>...</li></ul>\nStay tune for more! A full introduction is in the <a href='https://github.com/remotely-save/remotely-save/tree/master/docs/sync_algorithm/v3/intro.md'>doc website</a>.\nIf you agree to use this, please read and check two checkboxes then click the \"Agree\" button, and enjoy the plugin!\nIf you do not agree, please click the \"Do Not Agree\" button, the plugin will unload itself.\nAlso, please consider <a href='https://github.com/remotely-save/remotely-save'>visit the GitHub repo and star ⭐ it</a>! Or even <a href='https://github.com/remotely-save/donation'>buy me a coffee</a>. Your support is very important to me! Thanks!",
"syncalgov3_checkbox_manual_backup": "I will backup my vault manually firstly.",
+28 -14
View File
@@ -9,9 +9,9 @@
"syncrun_syncingribbon": "{{pluginName}}:正在由 {{triggerSource}} 触发运行",
"syncrun_step0": "0/8 Remotely Save 在空跑(dry run)模式,不会发生实际的文件交换。",
"syncrun_step1": "1/8 Remotely Save 准备同步({{serviceType}}",
"syncrun_step2": "2/8 正在获取远端的元数据。",
"syncrun_step3": "3/8 正在检查密码正确与否。",
"syncrun_step2": "2/8 正在获取远端的元数据和检查密码。",
"syncrun_passworderr": "检查密码时候出错。",
"syncrun_step3": "3/8 正在处理远端的元数据。",
"syncrun_step4": "4/8 正在获取本地的元数据。",
"syncrun_step5": "5/8 正在获取本地上一次同步的元数据。",
"syncrun_step6": "6/8 正在生成同步计划。",
@@ -57,7 +57,6 @@
"statusbar_lastsync_label": "上一次同步于:{{date}}",
"statusbar_lastsync_never": "没触发过同步",
"statusbar_lastsync_never_label": "没触发过同步",
"modal_password_title": "稍等一下,请阅读下文:",
"modal_password_shortdesc": "如果密码不是空的,那么文件会在上传之前,在本地先用此密码加密。\n如果密码是空的,那么文件会被非加密地上传。",
"modal_password_attn1": "注意 1/5:库(Vault)名字是不会加密的!本插件会在一些远程存储里创建一个和库名字有着同名的文件夹。",
@@ -133,17 +132,13 @@
"settings_runoncestartup_1sec": "启动后第 1 秒运行一次",
"settings_runoncestartup_10sec": "启动后第 10 秒运行一次",
"settings_runoncestartup_30sec": "启动后第 30 秒运行一次",
"settings_saverun": "保存时同步(实验性质)",
"settings_saverun_desc": "插件如果检查到当前文件在最近一段时间有修改保存过,则尝试同步。请注意,同步是一个很重的操作,因此会影响到耗电量。(修改设置后可能需要重载插件或重启。)",
"settings_saverun_notset": "(不设置",
"settings_saverun_1sec": "隔 1 秒检查一次",
"settings_saverun_5sec": "隔 5 秒检查一次",
"settings_saverun_10sec": "隔 10 秒检查一次(推荐)",
"settings_saverun_1min": "隔 1 分钟检查一次",
"settings_synconsave": "保存时同步(实验性质)",
"settings_synconsave_desc": "插件如果检查到当前文件在最近一段时间有修改保存过,则尝试同步。请注意,同步是一个很重的操作,因此会影响到耗电量。(修改设置后可能需要重载插件或重启。)",
"settings_synconsave_disable": "关闭(默认",
"settings_synconsave_enable": "开启",
"settings_skiplargefiles": "跳过大文件",
"settings_skiplargefiles_desc": "跳过大于某一个阈值的文件。这里 1 MB = 10^6 bytes。",
"settings_skiplargefiles_notset": "(不设置)",
"settings_ignorepaths": "忽略的文件或文件夹的正则表达式",
"settings_ignorepaths_desc": "忽略的文件或文件夹的正则表达式。每行一条。路径是相对于库(Vault)根目录的,没有前置 / 符号。",
"settings_enablestatusbar_info": "在状态栏显示上一次成功的同步",
@@ -153,7 +148,6 @@
"settings_resetstatusbar_time_desc": "重设上一次成功同步的时间记录。",
"settings_resetstatusbar_button": "重设",
"settings_resetstatusbar_notice": "重设完毕!",
"settings_checkonnectivity": "检查可否连接",
"settings_checkonnectivity_desc": "检查可否连接。",
"settings_checkonnectivity_button": "检查",
@@ -186,6 +180,12 @@
"settings_s3_accuratemtime_desc": "读取(已上传的)准确的文件修改时间,有助于同步算法更加准确和稳定。但是它也会导致额外的 api 请求、时间、金钱花费。",
"settings_s3_urlstyle": "S3 URL style",
"settings_s3_urlstyle_desc": "是否对 S3 对象强制使用 path style URL(例如使用 https://s3.amazonaws.com/*/ 而不是 https://*.s3.amazonaws.com/)。",
"settings_s3_reverse_proxy_no_sign_url": "S3 反向代理(不签名)地址(实验性质)",
"settings_s3_reverse_proxy_no_sign_url_desc": "不会参与到签名的 S3 反向代理地址。如果您有一个反向代理,但是不想修改原始鉴权签名,这里就可以填写。没有 http(s):// 前缀。如果您不知道这是什么,留空即可。",
"settings_s3_generatefolderobject": "是否生成文件夹 Object",
"settings_s3_generatefolderobject_desc": "S3 不存在“真正”的文件夹。如果您设置了“生成”(或用了旧版本),那么插件会上传 0 字节的以“/”结尾的 Object 来代表文件夹。新版本插件会默认跳过生成这种文件夹 Object。",
"settings_s3_generatefolderobject_notgenerate": "不生成(默认)",
"settings_s3_generatefolderobject_generate": "生成",
"settings_s3_connect_succ": "很好!可以访问到对应存储桶。",
"settings_s3_connect_fail": "无法访问到对应存储桶。",
"settings_dropbox": "Dropbox 设置",
@@ -239,12 +239,25 @@
"settings_webdav_connect_succ": "很好!可以连接上 Webdav 服务器。",
"settings_webdav_connect_fail": "无法连接上 Webdav 服务器。(可能是地址/账号/密码/鉴权类型等错误。)",
"settings_webdav_connect_fail_withcors": "无法连接上 Webdav 服务器。(可能是地址/账号/密码/鉴权类型/CORS 等错误。)",
"settings_webdis": "Webdis 设置",
"settings_webdis_disclaimer1": "声明:此插件不是 Redis® Ltd 或 Redis® 软件或 Wedis 的官方产品。Redis 是 Redis Ltd 的注册商标。",
"settings_webdis_disclaimer2": "声明:您所输入的信息存储于本地。其它有害的或者出错的插件,是有可能读取到这些信息的。如果您发现了 Webdis 服务器有不符合预期的访问,请立刻修改用户名和密码。",
"settings_webdis_folder": "我们会在您的服务器上创建带有此前缀的 key 并在里面同步::{{remoteBaseDir}}。",
"settings_webdis_addr": "服务器地址",
"settings_webdis_addr_desc": "服务器地址",
"settings_webdis_user": "用户名",
"settings_webdis_user_desc": "用户名。注意:用户名和其它信息都会保存在本地。",
"settings_webdis_password": "密码",
"settings_webdis_password_desc": "密码。注意:密码和其它信息都会保存在本地。",
"settings_webdis_connect_succ": "很好!可以连接上 Webdis 服务器。",
"settings_webdis_connect_fail": "无法连接上 Webdis 服务器。(可能是地址/账号/密码/鉴权类型等错误。)",
"settings_chooseservice": "选择远程服务",
"settings_chooseservice_desc": "从这里开始设置。您想连接到哪一个服务?S3、Dropbox、Webdav、OneDrive(个人版)?",
"settings_chooseservice_desc": "从这里开始设置。您想连接到哪一个服务?S3、Dropbox、Webdav、OneDrive(个人版)、Webdis",
"settings_chooseservice_s3": "S3 或兼容 S3 的服务",
"settings_chooseservice_dropbox": "Dropbox",
"settings_chooseservice_webdav": "Webdav",
"settings_chooseservice_onedrive": "OneDrive(个人版)",
"settings_chooseservice_webdis": "Webdis (an HTTP interface for Redis)",
"settings_adv": "进阶设置",
"settings_concurrency": "并行度",
"settings_concurrency_desc": "您希望同时最多有多少个文件被上传和下载?默认值是 5。如果您遇到了一些问题(如访问频率限制),您可以减少并行度。",
@@ -261,7 +274,7 @@
"settings_conflictaction_keep_newer": "保留最后修改的版本(默认)",
"settings_conflictaction_keep_larger": "保留文件体积较大的版本",
"settings_cleanemptyfolder": "处理空文件夹",
"settings_cleanemptyfolder_desc": "同步算法主要是针对文件处理的,您要手动指定空文件夹如何处理。",
"settings_cleanemptyfolder_desc": "同步算法主要是针对文件处理的,您要手动指定空文件夹如何处理。",
"settings_cleanemptyfolder_skip": "跳过处理空文件夹(默认)",
"settings_cleanemptyfolder_clean_both": "删除本地和服务器的空文件夹",
"settings_protectmodifypercentage": "如果修改超过百分比则中止同步",
@@ -322,6 +335,7 @@
"settings_resetcache_desc": "(出于调试原因)重设本地缓存和数据库。您需要在重设之后重新载入此插件。本重设不会删除 s3,密码……等设定。",
"settings_resetcache_button": "重设",
"settings_resetcache_notice": "本地同步缓存和数据库已被删除。请手动重新载入此插件。",
"syncalgov3_title": "Remotely Save 的同步算法有重大更新",
"syncalgov3_texts": "欢迎使用 Remotely Save!\n从这个版本开始,插件更新了同步算法:\n<ul><li>更稳健的删除同步</li><li>引入冲突处理</li><li>避免上传元数据</li><li>修改删除保护</li><li>备份模式</li><li>新的加密方式</li><li>……</li></ul>\n敬请期待更多更新!详细介绍请参阅<a href='https://github.com/remotely-save/remotely-save/tree/master/docs/sync_algorithm/v3/intro.md'>文档网站</a>。\n如果您同意使用新版本,请阅读和勾选两个勾选框,然后点击“同意”按钮,开始使用插件吧!\n如果您不同意,请点击“不同意”按钮,插件将自动停止运行(unload)。\n此外,请考虑<a href='https://github.com/remotely-save/remotely-save'>访问 GitHub 页面然后点赞 ⭐</a>!您的支持对我十分重要!谢谢!",
"syncalgov3_checkbox_manual_backup": "我将会首先手动备份我的库(Vault)。",
+27 -13
View File
@@ -9,9 +9,9 @@
"syncrun_syncingribbon": "{{pluginName}}:正在由 {{triggerSource}} 觸發執行",
"syncrun_step0": "0/8 Remotely Save 在空跑(dry run)模式,不會發生實際的檔案交換。",
"syncrun_step1": "1/8 Remotely Save 準備同步({{serviceType}}",
"syncrun_step2": "2/8 正在獲取遠端的元資料。",
"syncrun_step3": "3/8 正在檢查密碼正確與否。",
"syncrun_step2": "2/8 正在獲取遠端的元數據和檢查密碼。",
"syncrun_passworderr": "檢查密碼時候出錯。",
"syncrun_step3": "3/8 正在處理遠端的元數據。",
"syncrun_step4": "4/8 正在獲取本地的元資料。",
"syncrun_step5": "5/8 正在獲取本地上一次同步的元資料。",
"syncrun_step6": "6/8 正在生成同步計劃。",
@@ -56,7 +56,6 @@
"statusbar_lastsync_label": "上一次同步於:{{date}}",
"statusbar_lastsync_never": "沒觸發過同步",
"statusbar_lastsync_never_label": "沒觸發過同步",
"modal_password_title": "稍等一下,請閱讀下文:",
"modal_password_shortdesc": "如果密碼不是空的,那麼檔案會在上傳之前,在本地先用此密碼加密。\n如果密碼是空的,那麼檔案會被非加密地上傳。",
"modal_password_attn1": "注意 1/5:儲存庫(Vault)名字是不會加密的!本外掛會在一些遠端儲存裡建立一個和庫名字有著同名的資料夾。",
@@ -132,17 +131,13 @@
"settings_runoncestartup_1sec": "啟動後第 1 秒執行一次",
"settings_runoncestartup_10sec": "啟動後第 10 秒執行一次",
"settings_runoncestartup_30sec": "啟動後第 30 秒執行一次",
"settings_saverun": "儲存時同步(實驗性質)",
"settings_saverun_desc": "外掛如果檢查到當前檔案在最近一段時間有修改儲存過,則嘗試同步。請注意,同步是一個很重的操作,因此會影響到耗電量。(修改設定後可能需要過載外掛或重啟。)",
"settings_saverun_notset": "(不設定",
"settings_saverun_1sec": "隔 1 秒檢查一次",
"settings_saverun_5sec": "隔 5 秒檢查一次",
"settings_saverun_10sec": "隔 10 秒檢查一次(推薦)",
"settings_saverun_1min": "隔 1 分鐘檢查一次",
"settings_synconsave": "儲存時同步(實驗性質)",
"settings_synconsave_desc": "外掛如果檢查到當前檔案在最近一段時間有修改儲存過,則嘗試同步。請注意,同步是一個很重的操作,因此會影響到耗電量。(修改設定後可能需要過載外掛或重啟。)",
"settings_synconsave_disable": "關閉(預設",
"settings_synconsave_enable": "開啟",
"settings_skiplargefiles": "跳過大檔案",
"settings_skiplargefiles_desc": "跳過大於某一個閾值的檔案。這裡 1 MB = 10^6 bytes。",
"settings_skiplargefiles_notset": "(不設定)",
"settings_ignorepaths": "忽略的檔案或資料夾的正則表示式",
"settings_ignorepaths_desc": "忽略的檔案或資料夾的正則表示式。每行一條。路徑是相對於庫(Vault)根目錄的,沒有前置 / 符號。",
"settings_enablestatusbar_info": "在狀態列顯示上一次成功的同步",
@@ -152,7 +147,6 @@
"settings_resetstatusbar_time_desc": "重設上一次成功同步的時間記錄。",
"settings_resetstatusbar_button": "重設",
"settings_resetstatusbar_notice": "重設完畢!",
"settings_checkonnectivity": "檢查可否連線",
"settings_checkonnectivity_desc": "檢查可否連線。",
"settings_checkonnectivity_button": "檢查",
@@ -185,6 +179,12 @@
"settings_s3_accuratemtime_desc": "讀取(已上傳的)準確的檔案修改時間,有助於同步演算法更加準確和穩定。但是它也會導致額外的 api 請求、時間、金錢花費。",
"settings_s3_urlstyle": "S3 URL style",
"settings_s3_urlstyle_desc": "是否對 S3 物件強制使用 path style URL(例如使用 https://s3.amazonaws.com/*/ 而不是 https://*.s3.amazonaws.com/)。",
"settings_s3_reverse_proxy_no_sign_url": "S3 反向代理(不簽名)地址(實驗性質)",
"settings_s3_reverse_proxy_no_sign_url_desc": "不會參與到簽名的 S3 反向代理地址。如果您有一個反向代理,但是不想修改原始鑑權簽名,這裡就可以填寫。沒有 http(s):// 字首。如果您不知道這是什麼,留空即可。",
"settings_s3_generatefolderobject": "是否生成文件夾 Object",
"settings_s3_generatefolderobject_desc": "S3 不存在“真正”的文件夾。如果您設置了“生成”(或用了舊版本),那麼插件會上傳 0 字節的以“/”結尾的 Object 來代表文件夾。新版本插件會默認跳過生成這種文件夾 Object。",
"settings_s3_generatefolderobject_notgenerate": "不生成(默認)",
"settings_s3_generatefolderobject_generate": "生成",
"settings_s3_connect_succ": "很好!可以訪問到對應儲存桶。",
"settings_s3_connect_fail": "無法訪問到對應儲存桶。",
"settings_dropbox": "Dropbox 設定",
@@ -238,12 +238,25 @@
"settings_webdav_connect_succ": "很好!可以連線上 Webdav 伺服器。",
"settings_webdav_connect_fail": "無法連線上 Webdav 伺服器。(可能是地址/賬號/密碼/鑑權型別等錯誤。)",
"settings_webdav_connect_fail_withcors": "無法連線上 Webdav 伺服器。(可能是地址/賬號/密碼/鑑權型別/CORS 等錯誤。)",
"settings_webdis": "Webdis 設置",
"settings_webdis_disclaimer1": "聲明:此插件不是 Redis® Ltd 或 Redis® 軟件或 Wedis 的官方產品。Redis 是 Redis Ltd 的註冊商標。",
"settings_webdis_disclaimer2": "聲明:您所輸入的信息存儲於本地。其它有害的或者出錯的插件,是有可能讀取到這些信息的。如果您發現了 Webdis 服務器有不符合預期的訪問,請立刻修改用戶名和密碼。",
"settings_webdis_folder": "我們會在您的服務器上創建帶有此前綴的 key 並在裡面同步::{{remoteBaseDir}}。",
"settings_webdis_addr": "服務器地址",
"settings_webdis_addr_desc": "服務器地址",
"settings_webdis_user": "用戶名",
"settings_webdis_user_desc": "用戶名。注意:用戶名和其它信息都會保存在本地。",
"settings_webdis_password": "密碼",
"settings_webdis_password_desc": "密碼。注意:密碼和其它信息都會保存在本地。",
"settings_webdis_connect_succ": "很好!可以連接上 Webdis 服務器。",
"settings_webdis_connect_fail": "無法連接上 Webdis 服務器。(可能是地址/賬號/密碼/鑑權類型等錯誤。)",
"settings_chooseservice": "選擇遠端服務",
"settings_chooseservice_desc": "從這裡開始設定。您想連線到哪一個服務?S3、Dropbox、Webdav、OneDrive(個人版)?",
"settings_chooseservice_desc": "從這裡開始設定。您想連線到哪一個服務?S3、Dropbox、Webdav、OneDrive(個人版)、Webdis",
"settings_chooseservice_s3": "S3 或相容 S3 的服務",
"settings_chooseservice_dropbox": "Dropbox",
"settings_chooseservice_webdav": "Webdav",
"settings_chooseservice_onedrive": "OneDrive(個人版)",
"settings_chooseservice_webdis": "Webdis (an HTTP interface for Redis®)",
"settings_adv": "進階設定",
"settings_concurrency": "並行度",
"settings_concurrency_desc": "您希望同時最多有多少個檔案被上傳和下載?預設值是 5。如果您遇到了一些問題(如訪問頻率限制),您可以減少並行度。",
@@ -321,6 +334,7 @@
"settings_resetcache_desc": "(出於除錯原因)重設本地快取和資料庫。您需要在重設之後重新載入此外掛。本重設不會刪除 s3,密碼……等設定。",
"settings_resetcache_button": "重設",
"settings_resetcache_notice": "本地同步快取和資料庫已被刪除。請手動重新載入此外掛。",
"syncalgov3_title": "Remotely Save 的同步演算法有重大更新",
"syncalgov3_texts": "歡迎使用 Remotely Save!\n從這個版本開始,外掛更新了同步演算法:\n<ul><li>更穩健的刪除同步</li><li>引入衝突處理</li><li>避免上傳元資料</li><li>修改刪除保護</li><li>備份模式</li><li>新的加密方式</li><li>……</li></ul>\n敬請期待更多更新!詳細介紹請參閱<a href='https://github.com/remotely-save/remotely-save/tree/master/docs/sync_algorithm/v3/intro.md'>文件網站</a>。\n如果您同意使用新版本,請閱讀和勾選兩個勾選框,然後點選“同意”按鈕,開始使用外掛吧!\n如果您不同意,請點選“不同意”按鈕,外掛將自動停止執行(unload)。\n此外,請考慮<a href='https://github.com/remotely-save/remotely-save'>訪問 GitHub 頁面然後點贊 ⭐</a>!您的支援對我十分重要!謝謝!",
"syncalgov3_checkbox_manual_backup": "我將會首先手動備份我的庫(Vault)。",
-76
View File
@@ -1,76 +0,0 @@
import { TFile, TFolder, type Vault } from "obsidian";
import type { Entity, MixedEntity } from "./baseTypes";
import { listFilesInObsFolder } from "./obsFolderLister";
import { Profiler } from "./profiler";
export const getLocalEntityList = async (
vault: Vault,
syncConfigDir: boolean,
configDir: string,
pluginID: string,
profiler: Profiler
) => {
profiler.addIndent();
profiler.insert("enter getLocalEntityList");
const local: Entity[] = [];
const localTAbstractFiles = vault.getAllLoadedFiles();
profiler.insert("finish getting getAllLoadedFiles");
for (const entry of localTAbstractFiles) {
let r = {} as Entity;
let key = entry.path;
if (entry.path === "/") {
// ignore
continue;
} else if (entry instanceof TFile) {
let mtimeLocal: number | undefined = entry.stat.mtime;
if (mtimeLocal <= 0) {
mtimeLocal = entry.stat.ctime;
}
if (mtimeLocal === 0) {
mtimeLocal = undefined;
}
if (mtimeLocal === undefined) {
throw Error(
`Your file has last modified time 0: ${key}, don't know how to deal with it`
);
}
r = {
key: entry.path, // local always unencrypted
keyRaw: entry.path,
mtimeCli: mtimeLocal,
mtimeSvr: mtimeLocal,
size: entry.stat.size, // local always unencrypted
sizeRaw: entry.stat.size,
};
} else if (entry instanceof TFolder) {
key = `${entry.path}/`;
r = {
key: key,
keyRaw: key,
size: 0,
sizeRaw: 0,
};
} else {
throw Error(`unexpected ${entry}`);
}
local.push(r);
}
profiler.insert("finish transforming getAllLoadedFiles");
if (syncConfigDir) {
profiler.insert("into syncConfigDir");
const syncFiles = await listFilesInObsFolder(configDir, vault, pluginID);
for (const f of syncFiles) {
local.push(f);
}
profiler.insert("finish syncConfigDir");
}
profiler.insert("finish getLocalEntityList");
profiler.removeIndent();
return local;
};
+7 -8
View File
@@ -1,12 +1,12 @@
import localforage from "localforage";
import { extendPrototype } from "localforage-getitems";
extendPrototype(localforage);
export type LocalForage = typeof localforage;
import { nanoid } from "nanoid";
import { requireApiVersion, TAbstractFile, TFile, TFolder } from "obsidian";
import { API_VER_STAT_FOLDER } from "./baseTypes";
import type { Entity, MixedEntity, SUPPORTED_SERVICES_TYPE } from "./baseTypes";
import type { SyncPlanType } from "./sync";
import { statFix, toText, unixTimeToStr } from "./misc";
import { unixTimeToStr } from "./misc";
const DB_VERSION_NUMBER_IN_HISTORY = [20211114, 20220108, 20220326, 20240220];
export const DEFAULT_DB_VERSION_NUMBER: number = 20240220;
@@ -434,13 +434,12 @@ export const getAllPrevSyncRecordsByVaultAndProfile = async (
vaultRandomID: string,
profileID: string
) => {
// console.debug('inside getAllPrevSyncRecordsByVaultAndProfile')
const keys = await db.prevSyncRecordsTbl.keys();
// console.debug(`inside getAllPrevSyncRecordsByVaultAndProfile, keys=${keys}`)
const res: Entity[] = [];
for (const key of keys) {
const kv: Record<string, Entity | null> =
await db.prevSyncRecordsTbl.getItems();
for (const key of Object.getOwnPropertyNames(kv)) {
if (key.startsWith(`${vaultRandomID}\t${profileID}\t`)) {
const val: Entity | null = await db.prevSyncRecordsTbl.getItem(key);
const val = kv[key];
if (val !== null) {
res.push(val);
}
+309 -382
View File
@@ -7,7 +7,6 @@ import {
setIcon,
FileSystemAdapter,
Platform,
requestUrl,
requireApiVersion,
Events,
} from "obsidian";
@@ -22,11 +21,10 @@ import {
COMMAND_CALLBACK_ONEDRIVE,
COMMAND_CALLBACK_DROPBOX,
COMMAND_URI,
API_VER_ENSURE_REQURL_OK,
} from "./baseTypes";
import { API_VER_ENSURE_REQURL_OK } from "./baseTypesObs";
import { importQrCodeUri } from "./importExport";
import {
insertSyncPlanRecordByVault,
prepareDBs,
InternalDBs,
clearExpiredSyncPlanRecords,
@@ -34,49 +32,43 @@ import {
clearAllLoggerOutputRecords,
upsertLastSuccessSyncTimeByVault,
getLastSuccessSyncTimeByVault,
getAllPrevSyncRecordsByVaultAndProfile,
insertProfilerResultByVault,
} from "./localdb";
import { RemoteClient } from "./remote";
import {
DEFAULT_DROPBOX_CONFIG,
getAuthUrlAndVerifier as getAuthUrlAndVerifierDropbox,
sendAuthReq as sendAuthReqDropbox,
setConfigBySuccessfullAuthInplace as setConfigBySuccessfullAuthInplaceDropbox,
} from "./remoteForDropbox";
} from "./fsDropbox";
import {
AccessCodeResponseSuccessfulType,
DEFAULT_ONEDRIVE_CONFIG,
sendAuthReq as sendAuthReqOnedrive,
setConfigBySuccessfullAuthInplace as setConfigBySuccessfullAuthInplaceOnedrive,
} from "./remoteForOnedrive";
import { DEFAULT_S3_CONFIG } from "./remoteForS3";
import { DEFAULT_WEBDAV_CONFIG } from "./remoteForWebdav";
} from "./fsOnedrive";
import { DEFAULT_S3_CONFIG } from "./fsS3";
import { DEFAULT_WEBDAV_CONFIG } from "./fsWebdav";
import { RemotelySaveSettingTab } from "./settings";
import {
doActualSync,
ensembleMixedEnties,
getSyncPlanInplace,
isPasswordOk,
SyncStatusType,
} from "./sync";
import { messyConfigToNormal, normalConfigToMessy } from "./configPersist";
import { getLocalEntityList } from "./local";
import { I18n } from "./i18n";
import type { LangType, LangTypeAndAuto, TransItemType } from "./i18n";
import type { LangTypeAndAuto, TransItemType } from "./i18n";
import { SyncAlgoV3Modal } from "./syncAlgoV3Notice";
import AggregateError from "aggregate-error";
import { exportVaultSyncPlansToFiles } from "./debugMode";
import { changeMobileStatusBar, compareVersion } from "./misc";
import { Cipher } from "./encryptUnified";
import { changeMobileStatusBar } from "./misc";
import { Profiler } from "./profiler";
import { FakeFsLocal } from "./fsLocal";
import { FakeFsEncrypt } from "./fsEncrypt";
import { syncer } from "./sync";
import { getClient } from "./fsGetter";
import throttle from "lodash/throttle";
import { DEFAULT_WEBDIS_CONFIG } from "./fsWebdis";
const DEFAULT_SETTINGS: RemotelySavePluginSettings = {
s3: DEFAULT_S3_CONFIG,
webdav: DEFAULT_WEBDAV_CONFIG,
dropbox: DEFAULT_DROPBOX_CONFIG,
onedrive: DEFAULT_ONEDRIVE_CONFIG,
webdis: DEFAULT_WEBDIS_CONFIG,
password: "",
serviceType: "s3",
currLogLevel: "info",
@@ -96,7 +88,7 @@ const DEFAULT_SETTINGS: RemotelySavePluginSettings = {
deleteToWhere: "system",
agreeToUseSyncV3: false,
conflictAction: "keep_newer",
howToCleanEmptyFolder: "skip",
howToCleanEmptyFolder: "clean_both",
protectModifyPercentage: 50,
syncDirection: "bidirectional",
obfuscateSettingFile: true,
@@ -141,7 +133,8 @@ const getIconSvg = () => {
export default class RemotelySavePlugin extends Plugin {
settings!: RemotelySavePluginSettings;
db!: InternalDBs;
syncStatus!: SyncStatusType;
isSyncing!: boolean;
hasPendingSyncOnSave!: boolean;
statusBarElement!: HTMLSpanElement;
oauth2Info!: OAuth2Info;
currLogLevel!: string;
@@ -156,7 +149,25 @@ export default class RemotelySavePlugin extends Plugin {
appContainerObserver?: MutationObserver;
async syncRun(triggerSource: SyncTriggerSourceType = "manual") {
const profiler = new Profiler("start of syncRun");
const profiler = new Profiler();
const fsLocal = new FakeFsLocal(
this.app.vault,
this.settings.syncConfigDir ?? false,
this.app.vault.configDir,
this.manifest.id,
profiler,
this.settings.deleteToWhere ?? "system"
);
const fsRemote = getClient(
this.settings,
this.app.vault.getName(),
async () => await this.saveSettings()
);
const fsEncrypt = new FakeFsEncrypt(
fsRemote,
this.settings.password ?? "",
this.settings.encryptionMethod ?? "rclone-base64"
);
const t = (x: TransItemType, vars?: any) => {
return this.i18n.t(x, vars);
@@ -164,227 +175,14 @@ export default class RemotelySavePlugin extends Plugin {
const profileID = this.getCurrProfileID();
const getNotice = (x: string, timeout?: number) => {
// only show notices in manual mode
// no notice in auto mode
if (triggerSource === "manual" || triggerSource === "dry") {
new Notice(x, timeout);
}
};
if (this.syncStatus !== "idle") {
// really, users don't want to see this in auto mode
// so we use getNotice to avoid unnecessary show up
getNotice(
t("syncrun_alreadyrunning", {
pluginName: this.manifest.name,
syncStatus: this.syncStatus,
newTriggerSource: triggerSource,
})
);
if (this.currSyncMsg !== undefined && this.currSyncMsg !== "") {
getNotice(this.currSyncMsg);
}
return;
}
let originLabel = `${this.manifest.name}`;
if (this.syncRibbon !== undefined) {
originLabel = this.syncRibbon.getAttribute("aria-label") as string;
}
try {
console.info(
`${
this.manifest.id
}-${Date.now()}: start sync, triggerSource=${triggerSource}`
);
if (this.syncRibbon !== undefined) {
setIcon(this.syncRibbon, iconNameSyncRunning);
this.syncRibbon.setAttribute(
"aria-label",
t("syncrun_syncingribbon", {
pluginName: this.manifest.name,
triggerSource: triggerSource,
})
);
}
if (triggerSource === "dry") {
if (this.settings.currLogLevel === "info") {
getNotice(t("syncrun_shortstep0"));
} else {
getNotice(t("syncrun_step0"));
}
}
// change status to "syncing..." on statusbar
if (this.statusBarElement !== undefined) {
this.updateLastSuccessSyncMsg(-1);
}
//console.info(`huh ${this.settings.password}`)
if (this.settings.currLogLevel === "info") {
getNotice(
t("syncrun_shortstep1", {
serviceType: this.settings.serviceType,
})
);
} else {
getNotice(
t("syncrun_step1", {
serviceType: this.settings.serviceType,
})
);
}
this.syncStatus = "preparing";
profiler.insert("finish step1");
if (this.settings.currLogLevel === "info") {
// pass
} else {
getNotice(t("syncrun_step2"));
}
this.syncStatus = "getting_remote_files_list";
const self = this;
const client = new RemoteClient(
this.settings.serviceType,
this.settings.s3,
this.settings.webdav,
this.settings.dropbox,
this.settings.onedrive,
this.app.vault.getName(),
() => self.saveSettings(),
profiler
);
const remoteEntityList = await client.listAllFromRemote();
console.debug("remoteEntityList:");
console.debug(remoteEntityList);
profiler.insert("finish step2 (listing remote)");
if (this.settings.currLogLevel === "info") {
// pass
} else {
getNotice(t("syncrun_step3"));
}
this.syncStatus = "checking_password";
const cipher = new Cipher(
this.settings.password,
this.settings.encryptionMethod ?? "unknown"
);
const passwordCheckResult = await isPasswordOk(remoteEntityList, cipher);
if (!passwordCheckResult.ok) {
getNotice(t("syncrun_passworderr"));
throw Error(passwordCheckResult.reason);
}
profiler.insert("finish step3 (checking password)");
if (this.settings.currLogLevel === "info") {
// pass
} else {
getNotice(t("syncrun_step4"));
}
this.syncStatus = "getting_local_meta";
const localEntityList = await getLocalEntityList(
this.app.vault,
this.settings.syncConfigDir ?? false,
this.app.vault.configDir,
this.manifest.id,
profiler
);
console.debug("localEntityList:");
console.debug(localEntityList);
profiler.insert("finish step4 (local meta)");
if (this.settings.currLogLevel === "info") {
// pass
} else {
getNotice(t("syncrun_step5"));
}
this.syncStatus = "getting_local_prev_sync";
const prevSyncEntityList = await getAllPrevSyncRecordsByVaultAndProfile(
this.db,
this.vaultRandomID,
profileID
);
console.debug("prevSyncEntityList:");
console.debug(prevSyncEntityList);
profiler.insert("finish step5 (prev sync)");
if (this.settings.currLogLevel === "info") {
// pass
} else {
getNotice(t("syncrun_step6"));
}
this.syncStatus = "generating_plan";
let mixedEntityMappings = await ensembleMixedEnties(
localEntityList,
prevSyncEntityList,
remoteEntityList,
this.settings.syncConfigDir ?? false,
this.app.vault.configDir,
this.settings.syncUnderscoreItems ?? false,
this.settings.ignorePaths ?? [],
cipher,
this.settings.serviceType,
profiler
);
profiler.insert("finish building partial mixedEntity");
mixedEntityMappings = await getSyncPlanInplace(
mixedEntityMappings,
this.settings.howToCleanEmptyFolder ?? "skip",
this.settings.skipSizeLargerThan ?? -1,
this.settings.conflictAction ?? "keep_newer",
this.settings.syncDirection ?? "bidirectional",
profiler
);
console.info(`mixedEntityMappings:`);
console.info(mixedEntityMappings); // for debugging
profiler.insert("finish building full sync plan");
await insertSyncPlanRecordByVault(
this.db,
mixedEntityMappings,
this.vaultRandomID,
client.serviceType
);
profiler.insert("finish writing sync plan");
profiler.insert("finish step6 (plan)");
// The operations above are almost read only and kind of safe.
// The operations below begins to write or delete (!!!) something.
if (triggerSource !== "dry") {
if (this.settings.currLogLevel === "info") {
// pass
} else {
getNotice(t("syncrun_step7"));
}
this.syncStatus = "syncing";
await doActualSync(
mixedEntityMappings,
client,
this.vaultRandomID,
profileID,
this.app.vault,
cipher,
this.settings.concurrency ?? 5,
(key: string) => self.trash(key),
this.settings.protectModifyPercentage ?? 50,
(
const getProtectError = (
protectModifyPercentage: number,
realModifyDeleteCount: number,
allFilesCount: number
) => {
const percent = (
(100 * realModifyDeleteCount) /
allFilesCount
).toFixed(1);
const percent = ((100 * realModifyDeleteCount) / allFilesCount).toFixed(
1
);
const res = t("syncrun_abort_protectmodifypercentage", {
protectModifyPercentage,
realModifyDeleteCount,
@@ -392,105 +190,226 @@ export default class RemotelySavePlugin extends Plugin {
percent,
});
return res;
},
(
realCounter: number,
realTotalCount: number,
pathName: string,
decision: string
) =>
self.setCurrSyncMsg(
realCounter,
realTotalCount,
pathName,
decision,
triggerSource
),
this.db,
profiler
};
const getNotice = (
s: SyncTriggerSourceType,
msg: string,
timeout?: number
) => {
if (s === "manual" || s === "dry") {
new Notice(msg, timeout);
}
};
const notifyFunc = async (s: SyncTriggerSourceType, step: number) => {
switch (step) {
case 0:
if (s === "dry") {
if (this.settings.currLogLevel === "info") {
getNotice(s, t("syncrun_shortstep0"));
} else {
getNotice(s, t("syncrun_step0"));
}
}
break;
case 1:
if (this.settings.currLogLevel === "info") {
getNotice(
s,
t("syncrun_shortstep1", {
serviceType: this.settings.serviceType,
})
);
} else {
this.syncStatus = "syncing";
getNotice(
s,
t("syncrun_step1", {
serviceType: this.settings.serviceType,
})
);
}
break;
case 2:
if (this.settings.currLogLevel === "info") {
getNotice(t("syncrun_shortstep2skip"));
// pass
} else {
getNotice(t("syncrun_step7skip"));
}
getNotice(s, t("syncrun_step2"));
}
break;
cipher.closeResources();
profiler.insert("finish step7 (actual sync)");
case 3:
if (this.settings.currLogLevel === "info") {
getNotice(t("syncrun_shortstep2"));
// pass
} else {
getNotice(t("syncrun_step8"));
getNotice(s, t("syncrun_step3"));
}
break;
this.syncStatus = "finish";
this.syncStatus = "idle";
case 4:
if (this.settings.currLogLevel === "info") {
// pass
} else {
getNotice(s, t("syncrun_step4"));
}
break;
profiler.insert("finish step8");
case 5:
if (this.settings.currLogLevel === "info") {
// pass
} else {
getNotice(s, t("syncrun_step5"));
}
break;
case 6:
if (this.settings.currLogLevel === "info") {
// pass
} else {
getNotice(s, t("syncrun_step6"));
}
break;
case 7:
if (s === "dry") {
if (this.settings.currLogLevel === "info") {
getNotice(s, t("syncrun_shortstep2skip"));
} else {
getNotice(s, t("syncrun_step7skip"));
}
} else {
if (this.settings.currLogLevel === "info") {
// pass
} else {
getNotice(s, t("syncrun_step7"));
}
}
break;
case 8:
if (this.settings.currLogLevel === "info") {
getNotice(s, t("syncrun_shortstep2"));
} else {
getNotice(s, t("syncrun_step8"));
}
break;
default:
throw Error(`unknown step=${step} for showing notice`);
break;
}
};
const errNotifyFunc = async (s: SyncTriggerSourceType, error: Error) => {
console.error(error);
if (error instanceof AggregateError) {
for (const e of error.errors) {
getNotice(s, e.message, 10 * 1000);
}
} else {
getNotice(s, error?.message ?? "error while sync", 10 * 1000);
}
};
const ribboonFunc = async (s: SyncTriggerSourceType, step: number) => {
if (step === 1) {
if (this.syncRibbon !== undefined) {
setIcon(this.syncRibbon, iconNameSyncRunning);
this.syncRibbon.setAttribute(
"aria-label",
t("syncrun_syncingribbon", {
pluginName: this.manifest.name,
triggerSource: s,
})
);
}
} else if (step === 8) {
// last step
if (this.syncRibbon !== undefined) {
setIcon(this.syncRibbon, iconNameSyncWait);
let originLabel = `${this.manifest.name}`;
this.syncRibbon.setAttribute("aria-label", originLabel);
}
}
};
const statusBarFunc = async (s: SyncTriggerSourceType, step: number) => {
if (step === 1) {
// change status to "syncing..." on statusbar
this.updateLastSuccessSyncMsg(-1);
} else if (step === 8) {
const lastSuccessSyncMillis = Date.now();
await upsertLastSuccessSyncTimeByVault(
this.db,
this.vaultRandomID,
lastSuccessSyncMillis
);
if (this.syncRibbon !== undefined) {
setIcon(this.syncRibbon, iconNameSyncWait);
this.syncRibbon.setAttribute("aria-label", originLabel);
}
if (this.statusBarElement !== undefined) {
this.updateLastSuccessSyncMsg(lastSuccessSyncMillis);
}
};
const markIsSyncingFunc = async (isSyncing: boolean) => {
this.isSyncing = isSyncing;
};
const callbackSyncProcess = async (
realCounter: number,
realTotalCount: number,
pathName: string,
decision: string
) => {
this.setCurrSyncMsg(
realCounter,
realTotalCount,
pathName,
decision,
triggerSource
);
};
if (this.isSyncing) {
getNotice(
triggerSource,
t("syncrun_alreadyrunning", {
pluginName: this.manifest.name,
syncStatus: "running",
newTriggerSource: triggerSource,
})
);
if (this.currSyncMsg !== undefined && this.currSyncMsg !== "") {
getNotice(triggerSource, this.currSyncMsg);
}
return;
}
await syncer(
fsLocal,
fsRemote,
fsEncrypt,
profiler,
this.db,
triggerSource,
profileID,
this.vaultRandomID,
this.app.vault.configDir,
this.settings,
getProtectError,
markIsSyncingFunc,
notifyFunc,
errNotifyFunc,
ribboonFunc,
statusBarFunc,
callbackSyncProcess
);
fsEncrypt.closeResources();
profiler.clear();
this.syncEvent?.trigger("SYNC_DONE");
console.info(
`${
this.manifest.id
}-${Date.now()}: finish sync, triggerSource=${triggerSource}`
);
} catch (error: any) {
profiler.insert("start error branch");
const msg = t("syncrun_abort", {
manifestID: this.manifest.id,
theDate: `${Date.now()}`,
triggerSource: triggerSource,
syncStatus: this.syncStatus,
});
console.error(msg);
console.error(error);
getNotice(msg, 10 * 1000);
if (error instanceof AggregateError) {
for (const e of error.errors) {
getNotice(e.message, 10 * 1000);
}
} else {
getNotice(error?.message ?? "error while sync", 10 * 1000);
}
this.syncStatus = "idle";
if (this.syncRibbon !== undefined) {
setIcon(this.syncRibbon, iconNameSyncWait);
this.syncRibbon.setAttribute("aria-label", originLabel);
}
profiler.insert("finish error branch");
}
profiler.insert("finish syncRun");
console.debug(profiler.toString());
insertProfilerResultByVault(
this.db,
profiler.toString(),
this.vaultRandomID,
this.settings.serviceType
);
profiler.clear();
}
async onload() {
@@ -511,6 +430,8 @@ export default class RemotelySavePlugin extends Plugin {
}; // init
this.currSyncMsg = "";
this.isSyncing = false;
this.hasPendingSyncOnSave = false;
this.syncEvent = new Events();
@@ -561,8 +482,6 @@ export default class RemotelySavePlugin extends Plugin {
// must AFTER preparing DB
this.enableAutoClearSyncPlanHist();
this.syncStatus = "idle";
this.registerObsidianProtocolHandler(COMMAND_URI, async (inputParams) => {
// console.debug(inputParams);
const parsed = importQrCodeUri(inputParams, this.app.vault.getName());
@@ -633,17 +552,12 @@ export default class RemotelySavePlugin extends Plugin {
() => self.saveSettings()
);
const client = new RemoteClient(
"dropbox",
undefined,
undefined,
this.settings.dropbox,
undefined,
const client = getClient(
this.settings,
this.app.vault.getName(),
() => self.saveSettings()
);
const username = await client.getUser();
const username = await client.getUserDisplayName();
this.settings.dropbox.username = username;
await this.saveSettings();
@@ -729,16 +643,12 @@ export default class RemotelySavePlugin extends Plugin {
() => self.saveSettings()
);
const client = new RemoteClient(
"onedrive",
undefined,
undefined,
undefined,
this.settings.onedrive,
const client = getClient(
this.settings,
this.app.vault.getName(),
() => self.saveSettings()
);
this.settings.onedrive.username = await client.getUser();
this.settings.onedrive.username = await client.getUserDisplayName();
await this.saveSettings();
this.oauth2Info.verifier = ""; // reset it
@@ -879,7 +789,7 @@ export default class RemotelySavePlugin extends Plugin {
} else {
this.enableAutoSyncIfSet();
this.enableInitSyncIfSet();
this.enableSyncOnSaveIfSet();
this.toggleSyncOnSaveIfSet();
}
// compare versions and read new versions
@@ -960,6 +870,9 @@ export default class RemotelySavePlugin extends Plugin {
// it causes money, so disable it by default
this.settings.s3.useAccurateMTime = false;
}
if (this.settings.s3.generateFolderObject === undefined) {
this.settings.s3.generateFolderObject = false;
}
if (this.settings.ignorePaths === undefined) {
this.settings.ignorePaths = [];
}
@@ -985,7 +898,7 @@ export default class RemotelySavePlugin extends Plugin {
this.settings.conflictAction = "keep_newer";
}
if (this.settings.howToCleanEmptyFolder === undefined) {
this.settings.howToCleanEmptyFolder = "skip";
this.settings.howToCleanEmptyFolder = "clean_both";
}
if (this.settings.protectModifyPercentage === undefined) {
this.settings.protectModifyPercentage = 50;
@@ -1191,30 +1104,12 @@ export default class RemotelySavePlugin extends Plugin {
}
}
enableSyncOnSaveIfSet() {
if (
this.settings.syncOnSaveAfterMilliseconds !== undefined &&
this.settings.syncOnSaveAfterMilliseconds !== null &&
this.settings.syncOnSaveAfterMilliseconds > 0
) {
let runScheduled = false;
let needToRunAgain = false;
const scheduleSyncOnSave = (scheduleTimeFromNow: number) => {
console.info(
`schedule a run for ${scheduleTimeFromNow} milliseconds later`
);
runScheduled = true;
setTimeout(() => {
this.syncRun("auto_sync_on_save");
runScheduled = false;
}, scheduleTimeFromNow);
};
const checkCurrFileModified = async (caller: "SYNC" | "FILE_CHANGES") => {
async _checkCurrFileModified(caller: "SYNC" | "FILE_CHANGES") {
console.debug(`inside checkCurrFileModified`);
const currentFile = this.app.workspace.getActiveFile();
if (currentFile) {
console.debug(`we have currentFile=${currentFile.path}`);
// get the last modified time of the current file
// if it has modified after lastSuccessSync
// then schedule a run for syncOnSaveAfterMilliseconds after it was modified
@@ -1223,43 +1118,75 @@ export default class RemotelySavePlugin extends Plugin {
this.db,
this.vaultRandomID
);
console.debug(
`lastModified=${lastModified}, lastSuccessSyncMillis=${lastSuccessSyncMillis}`
);
if (
this.syncStatus === "idle" &&
lastModified > lastSuccessSyncMillis &&
!runScheduled
caller === "SYNC" ||
(caller === "FILE_CHANGES" && lastModified > lastSuccessSyncMillis)
) {
scheduleSyncOnSave(this.settings!.syncOnSaveAfterMilliseconds!);
} else if (
this.syncStatus === "idle" &&
needToRunAgain &&
!runScheduled
) {
scheduleSyncOnSave(this.settings!.syncOnSaveAfterMilliseconds!);
needToRunAgain = false;
console.debug(
`so lastModified > lastSuccessSyncMillis or it's called while syncing before`
);
console.debug(
`caller=${caller}, isSyncing=${this.isSyncing}, hasPendingSyncOnSave=${this.hasPendingSyncOnSave}`
);
if (this.isSyncing) {
this.hasPendingSyncOnSave = true;
// wait for next event
return;
} else {
if (caller === "FILE_CHANGES") {
needToRunAgain = true;
if (this.hasPendingSyncOnSave || caller === "FILE_CHANGES") {
this.hasPendingSyncOnSave = false;
await this.syncRun("auto_sync_on_save");
}
return;
}
}
} else {
console.debug(`no currentFile here`);
}
}
_syncOnSaveEvent1 = () => {
this._checkCurrFileModified("SYNC");
};
_syncOnSaveEvent2 = throttle(
async () => {
await this._checkCurrFileModified("FILE_CHANGES");
},
1000 * 3,
{
leading: false,
trailing: true,
}
);
toggleSyncOnSaveIfSet() {
if (
this.settings.syncOnSaveAfterMilliseconds !== undefined &&
this.settings.syncOnSaveAfterMilliseconds !== null &&
this.settings.syncOnSaveAfterMilliseconds > 0
) {
this.app.workspace.onLayoutReady(() => {
// listen to sync done
this.registerEvent(
this.syncEvent?.on("SYNC_DONE", () => {
checkCurrFileModified("SYNC");
})!
this.syncEvent?.on("SYNC_DONE", this._syncOnSaveEvent1)!
);
// listen to current file save changes
this.registerEvent(
this.app.vault.on("modify", (x) => {
// console.debug(`event=modify! file=${x}`);
checkCurrFileModified("FILE_CHANGES");
})
);
this.registerEvent(this.app.vault.on("modify", this._syncOnSaveEvent2));
this.registerEvent(this.app.vault.on("create", this._syncOnSaveEvent2));
this.registerEvent(this.app.vault.on("delete", this._syncOnSaveEvent2));
});
} else {
this.syncEvent?.off("SYNC_DONE", this._syncOnSaveEvent1);
this.app.vault.off("modify", this._syncOnSaveEvent2);
this.app.vault.off("create", this._syncOnSaveEvent2);
this.app.vault.off("delete", this._syncOnSaveEvent2);
}
}
@@ -1276,7 +1203,7 @@ export default class RemotelySavePlugin extends Plugin {
await this.saveSettings();
}
async setCurrSyncMsg(
setCurrSyncMsg(
i: number,
totalCount: number,
pathName: string,
-7
View File
@@ -31,13 +31,6 @@ const isLikelyPluginSubFiles = (x: string) => {
return false;
};
export const isInsideObsFolder = (x: string, configDir: string) => {
if (!configDir.startsWith(".")) {
throw Error(`configDir should starts with . but we get ${configDir}`);
}
return x === configDir || x.startsWith(`${configDir}/`);
};
export const listFilesInObsFolder = async (
configDir: string,
vault: Vault,
+15
View File
@@ -1,3 +1,5 @@
import { SUPPORTED_SERVICES_TYPE } from "./baseTypes";
import { InternalDBs, insertProfilerResultByVault } from "./localdb";
import { unixTimeToStr } from "./misc";
interface BreakPoint {
@@ -79,4 +81,17 @@ export class Profiler {
return res;
}
async save(
db: InternalDBs,
vaultRandomID: string,
remoteType: SUPPORTED_SERVICES_TYPE
) {
await insertProfilerResultByVault(
db,
this.toString(),
vaultRandomID,
remoteType
);
}
}
-320
View File
@@ -1,320 +0,0 @@
import { Vault } from "obsidian";
import type {
Entity,
DropboxConfig,
OnedriveConfig,
S3Config,
SUPPORTED_SERVICES_TYPE,
WebdavConfig,
UploadedType,
} from "./baseTypes";
import * as dropbox from "./remoteForDropbox";
import * as onedrive from "./remoteForOnedrive";
import * as s3 from "./remoteForS3";
import * as webdav from "./remoteForWebdav";
import { Cipher } from "./encryptUnified";
import { Profiler } from "./profiler";
export class RemoteClient {
readonly serviceType: SUPPORTED_SERVICES_TYPE;
readonly s3Config?: S3Config;
readonly webdavClient?: webdav.WrappedWebdavClient;
readonly webdavConfig?: WebdavConfig;
readonly dropboxClient?: dropbox.WrappedDropboxClient;
readonly dropboxConfig?: DropboxConfig;
readonly onedriveClient?: onedrive.WrappedOnedriveClient;
readonly onedriveConfig?: OnedriveConfig;
constructor(
serviceType: SUPPORTED_SERVICES_TYPE,
s3Config?: S3Config,
webdavConfig?: WebdavConfig,
dropboxConfig?: DropboxConfig,
onedriveConfig?: OnedriveConfig,
vaultName?: string,
saveUpdatedConfigFunc?: () => Promise<any>,
profiler?: Profiler
) {
this.serviceType = serviceType;
// the client may modify the config inplace,
// so we use a ref not copy of config here
if (serviceType === "s3") {
this.s3Config = s3Config;
} else if (serviceType === "webdav") {
if (vaultName === undefined || saveUpdatedConfigFunc === undefined) {
throw Error(
"remember to provide vault name and callback while init webdav client"
);
}
const remoteBaseDir = webdavConfig!.remoteBaseDir || vaultName;
this.webdavConfig = webdavConfig;
this.webdavClient = webdav.getWebdavClient(
this.webdavConfig!,
remoteBaseDir,
saveUpdatedConfigFunc
);
} else if (serviceType === "dropbox") {
if (vaultName === undefined || saveUpdatedConfigFunc === undefined) {
throw Error(
"remember to provide vault name and callback while init dropbox client"
);
}
const remoteBaseDir = dropboxConfig!.remoteBaseDir || vaultName;
this.dropboxConfig = dropboxConfig;
this.dropboxClient = dropbox.getDropboxClient(
this.dropboxConfig!,
remoteBaseDir,
saveUpdatedConfigFunc
);
} else if (serviceType === "onedrive") {
if (vaultName === undefined || saveUpdatedConfigFunc === undefined) {
throw Error(
"remember to provide vault name and callback while init onedrive client"
);
}
const remoteBaseDir = onedriveConfig!.remoteBaseDir || vaultName;
this.onedriveConfig = onedriveConfig;
this.onedriveClient = onedrive.getOnedriveClient(
this.onedriveConfig!,
remoteBaseDir,
saveUpdatedConfigFunc
);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
}
getRemoteMeta = async (fileOrFolderPath: string) => {
if (this.serviceType === "s3") {
return await s3.getRemoteMeta(
s3.getS3Client(this.s3Config!),
this.s3Config!,
fileOrFolderPath
);
} else if (this.serviceType === "webdav") {
return await webdav.getRemoteMeta(this.webdavClient!, fileOrFolderPath);
} else if (this.serviceType === "dropbox") {
return await dropbox.getRemoteMeta(this.dropboxClient!, fileOrFolderPath);
} else if (this.serviceType === "onedrive") {
return await onedrive.getRemoteMeta(
this.onedriveClient!,
fileOrFolderPath
);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
};
uploadToRemote = async (
fileOrFolderPath: string,
vault: Vault | undefined,
isRecursively: boolean,
cipher: Cipher,
remoteEncryptedKey: string = "",
foldersCreatedBefore: Set<string> | undefined = undefined,
uploadRaw: boolean = false,
rawContent: string | ArrayBuffer = ""
): Promise<UploadedType> => {
if (this.serviceType === "s3") {
return await s3.uploadToRemote(
s3.getS3Client(this.s3Config!),
this.s3Config!,
fileOrFolderPath,
vault,
isRecursively,
cipher,
remoteEncryptedKey,
uploadRaw,
rawContent
);
} else if (this.serviceType === "webdav") {
return await webdav.uploadToRemote(
this.webdavClient!,
fileOrFolderPath,
vault,
isRecursively,
cipher,
remoteEncryptedKey,
uploadRaw,
rawContent
);
} else if (this.serviceType === "dropbox") {
return await dropbox.uploadToRemote(
this.dropboxClient!,
fileOrFolderPath,
vault,
isRecursively,
cipher,
remoteEncryptedKey,
foldersCreatedBefore,
uploadRaw,
rawContent
);
} else if (this.serviceType === "onedrive") {
return await onedrive.uploadToRemote(
this.onedriveClient!,
fileOrFolderPath,
vault,
isRecursively,
cipher,
remoteEncryptedKey,
foldersCreatedBefore,
uploadRaw,
rawContent
);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
};
listAllFromRemote = async (): Promise<Entity[]> => {
if (this.serviceType === "s3") {
return await s3.listAllFromRemote(
s3.getS3Client(this.s3Config!),
this.s3Config!
);
} else if (this.serviceType === "webdav") {
return await webdav.listAllFromRemote(this.webdavClient!);
} else if (this.serviceType === "dropbox") {
return await dropbox.listAllFromRemote(this.dropboxClient!);
} else if (this.serviceType === "onedrive") {
return await onedrive.listAllFromRemote(this.onedriveClient!);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
};
downloadFromRemote = async (
fileOrFolderPath: string,
vault: Vault,
mtime: number,
cipher: Cipher,
remoteEncryptedKey: string = "",
skipSaving: boolean = false
) => {
if (this.serviceType === "s3") {
return await s3.downloadFromRemote(
s3.getS3Client(this.s3Config!),
this.s3Config!,
fileOrFolderPath,
vault,
mtime,
cipher,
remoteEncryptedKey,
skipSaving
);
} else if (this.serviceType === "webdav") {
return await webdav.downloadFromRemote(
this.webdavClient!,
fileOrFolderPath,
vault,
mtime,
cipher,
remoteEncryptedKey,
skipSaving
);
} else if (this.serviceType === "dropbox") {
return await dropbox.downloadFromRemote(
this.dropboxClient!,
fileOrFolderPath,
vault,
mtime,
cipher,
remoteEncryptedKey,
skipSaving
);
} else if (this.serviceType === "onedrive") {
return await onedrive.downloadFromRemote(
this.onedriveClient!,
fileOrFolderPath,
vault,
mtime,
cipher,
remoteEncryptedKey,
skipSaving
);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
};
deleteFromRemote = async (
fileOrFolderPath: string,
cipher: Cipher,
remoteEncryptedKey: string = "",
synthesizedFolder: boolean = false
) => {
if (this.serviceType === "s3") {
return await s3.deleteFromRemote(
s3.getS3Client(this.s3Config!),
this.s3Config!,
fileOrFolderPath,
cipher,
remoteEncryptedKey,
synthesizedFolder
);
} else if (this.serviceType === "webdav") {
return await webdav.deleteFromRemote(
this.webdavClient!,
fileOrFolderPath,
cipher,
remoteEncryptedKey
);
} else if (this.serviceType === "dropbox") {
return await dropbox.deleteFromRemote(
this.dropboxClient!,
fileOrFolderPath,
cipher,
remoteEncryptedKey
);
} else if (this.serviceType === "onedrive") {
return await onedrive.deleteFromRemote(
this.onedriveClient!,
fileOrFolderPath,
cipher,
remoteEncryptedKey
);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
};
checkConnectivity = async (callbackFunc?: any) => {
if (this.serviceType === "s3") {
return await s3.checkConnectivity(
s3.getS3Client(this.s3Config!),
this.s3Config!,
callbackFunc
);
} else if (this.serviceType === "webdav") {
return await webdav.checkConnectivity(this.webdavClient!, callbackFunc);
} else if (this.serviceType === "dropbox") {
return await dropbox.checkConnectivity(this.dropboxClient!, callbackFunc);
} else if (this.serviceType === "onedrive") {
return await onedrive.checkConnectivity(
this.onedriveClient!,
callbackFunc
);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
};
getUser = async () => {
if (this.serviceType === "dropbox") {
return await dropbox.getUserDisplayName(this.dropboxClient!);
} else if (this.serviceType === "onedrive") {
return await onedrive.getUserDisplayName(this.onedriveClient!);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
};
revokeAuth = async () => {
if (this.serviceType === "dropbox") {
return await dropbox.revokeAuth(this.dropboxClient!);
} else {
throw Error(`not supported service type ${this.serviceType}`);
}
};
}
-828
View File
@@ -1,828 +0,0 @@
import type { _Object } from "@aws-sdk/client-s3";
import {
DeleteObjectCommand,
GetObjectCommand,
HeadBucketCommand,
HeadObjectCommand,
HeadObjectCommandOutput,
ListObjectsV2Command,
ListObjectsV2CommandInput,
PutObjectCommand,
S3Client,
} from "@aws-sdk/client-s3";
import { Upload } from "@aws-sdk/lib-storage";
import { HttpHandler, HttpRequest, HttpResponse } from "@smithy/protocol-http";
import {
FetchHttpHandler,
FetchHttpHandlerOptions,
} from "@smithy/fetch-http-handler";
// @ts-ignore
import { requestTimeout } from "@smithy/fetch-http-handler/dist-es/request-timeout";
import { buildQueryString } from "@smithy/querystring-builder";
import { HeaderBag, HttpHandlerOptions, Provider } from "@aws-sdk/types";
import { Buffer } from "buffer";
import * as mime from "mime-types";
import { Vault, requestUrl, RequestUrlParam, Platform } from "obsidian";
import { Readable } from "stream";
import * as path from "path";
import AggregateError from "aggregate-error";
import {
DEFAULT_CONTENT_TYPE,
Entity,
S3Config,
UploadedType,
VALID_REQURL,
} from "./baseTypes";
import {
arrayBufferToBuffer,
bufferToArrayBuffer,
mkdirpInVault,
} from "./misc";
export { S3Client } from "@aws-sdk/client-s3";
import PQueue from "p-queue";
import { Cipher } from "./encryptUnified";
////////////////////////////////////////////////////////////////////////////////
// special handler using Obsidian requestUrl
////////////////////////////////////////////////////////////////////////////////
/**
* This is close to origin implementation of FetchHttpHandler
* https://github.com/aws/aws-sdk-js-v3/blob/main/packages/fetch-http-handler/src/fetch-http-handler.ts
* that is released under Apache 2 License.
* But this uses Obsidian requestUrl instead.
*/
class ObsHttpHandler extends FetchHttpHandler {
requestTimeoutInMs: number | undefined;
constructor(options?: FetchHttpHandlerOptions) {
super(options);
this.requestTimeoutInMs =
options === undefined ? undefined : options.requestTimeout;
}
async handle(
request: HttpRequest,
{ abortSignal }: HttpHandlerOptions = {}
): Promise<{ response: HttpResponse }> {
if (abortSignal?.aborted) {
const abortError = new Error("Request aborted");
abortError.name = "AbortError";
return Promise.reject(abortError);
}
let path = request.path;
if (request.query) {
const queryString = buildQueryString(request.query);
if (queryString) {
path += `?${queryString}`;
}
}
const { port, method } = request;
const url = `${request.protocol}//${request.hostname}${
port ? `:${port}` : ""
}${path}`;
const body =
method === "GET" || method === "HEAD" ? undefined : request.body;
const transformedHeaders: Record<string, string> = {};
for (const key of Object.keys(request.headers)) {
const keyLower = key.toLowerCase();
if (keyLower === "host" || keyLower === "content-length") {
continue;
}
transformedHeaders[keyLower] = request.headers[key];
}
let contentType: string | undefined = undefined;
if (transformedHeaders["content-type"] !== undefined) {
contentType = transformedHeaders["content-type"];
}
let transformedBody: any = body;
if (ArrayBuffer.isView(body)) {
transformedBody = bufferToArrayBuffer(body);
}
const param: RequestUrlParam = {
body: transformedBody,
headers: transformedHeaders,
method: method,
url: url,
contentType: contentType,
};
const raceOfPromises = [
requestUrl(param).then((rsp) => {
const headers = rsp.headers;
const headersLower: Record<string, string> = {};
for (const key of Object.keys(headers)) {
headersLower[key.toLowerCase()] = headers[key];
}
const stream = new ReadableStream<Uint8Array>({
start(controller) {
controller.enqueue(new Uint8Array(rsp.arrayBuffer));
controller.close();
},
});
return {
response: new HttpResponse({
headers: headersLower,
statusCode: rsp.status,
body: stream,
}),
};
}),
requestTimeout(this.requestTimeoutInMs),
];
if (abortSignal) {
raceOfPromises.push(
new Promise<never>((resolve, reject) => {
abortSignal.onabort = () => {
const abortError = new Error("Request aborted");
abortError.name = "AbortError";
reject(abortError);
};
})
);
}
return Promise.race(raceOfPromises);
}
}
////////////////////////////////////////////////////////////////////////////////
// other stuffs
////////////////////////////////////////////////////////////////////////////////
export const DEFAULT_S3_CONFIG: S3Config = {
s3Endpoint: "",
s3Region: "",
s3AccessKeyID: "",
s3SecretAccessKey: "",
s3BucketName: "",
bypassCorsLocally: true,
partsConcurrency: 20,
forcePathStyle: false,
remotePrefix: "",
useAccurateMTime: false, // it causes money, disable by default
};
export type S3ObjectType = _Object;
export const simpleTransRemotePrefix = (x: string) => {
if (x === undefined) {
return "";
}
let y = path.posix.normalize(x.trim());
if (y === undefined || y === "" || y === "/" || y === ".") {
return "";
}
if (y.startsWith("/")) {
y = y.slice(1);
}
if (!y.endsWith("/")) {
y = `${y}/`;
}
return y;
};
const getRemoteWithPrefixPath = (
fileOrFolderPath: string,
remotePrefix: string
) => {
let key = fileOrFolderPath;
if (fileOrFolderPath === "/" || fileOrFolderPath === "") {
// special
key = remotePrefix;
}
if (!fileOrFolderPath.startsWith("/")) {
key = `${remotePrefix}${fileOrFolderPath}`;
}
return key;
};
const getLocalNoPrefixPath = (
fileOrFolderPathWithRemotePrefix: string,
remotePrefix: string
) => {
if (
!(
fileOrFolderPathWithRemotePrefix === `${remotePrefix}` ||
fileOrFolderPathWithRemotePrefix.startsWith(`${remotePrefix}`)
)
) {
throw Error(
`"${fileOrFolderPathWithRemotePrefix}" doesn't starts with "${remotePrefix}"`
);
}
return fileOrFolderPathWithRemotePrefix.slice(`${remotePrefix}`.length);
};
const fromS3ObjectToEntity = (
x: S3ObjectType,
remotePrefix: string,
mtimeRecords: Record<string, number>,
ctimeRecords: Record<string, number>
) => {
// console.debug(`fromS3ObjectToEntity: ${x.Key!}, ${JSON.stringify(x,null,2)}`);
// S3 officially only supports seconds precision!!!!!
const mtimeSvr = Math.floor(x.LastModified!.valueOf() / 1000.0) * 1000;
let mtimeCli = mtimeSvr;
if (x.Key! in mtimeRecords) {
const m2 = mtimeRecords[x.Key!];
if (m2 !== 0) {
// to be compatible with RClone, we read and store the time in seconds in new version!
if (m2 >= 1000000000000) {
// it's a millsecond, uploaded by old codes..
mtimeCli = m2;
} else {
// it's a second, uploaded by new codes of the plugin from March 24, 2024
mtimeCli = m2 * 1000;
}
}
}
const key = getLocalNoPrefixPath(x.Key!, remotePrefix);
const r: Entity = {
keyRaw: key,
mtimeSvr: mtimeSvr,
mtimeCli: mtimeCli,
sizeRaw: x.Size!,
etag: x.ETag,
synthesizedFolder: false,
};
return r;
};
const fromS3HeadObjectToEntity = (
fileOrFolderPathWithRemotePrefix: string,
x: HeadObjectCommandOutput,
remotePrefix: string
) => {
// console.debug(`fromS3HeadObjectToEntity: ${fileOrFolderPathWithRemotePrefix}: ${JSON.stringify(x,null,2)}`);
// S3 officially only supports seconds precision!!!!!
const mtimeSvr = Math.floor(x.LastModified!.valueOf() / 1000.0) * 1000;
let mtimeCli = mtimeSvr;
if (x.Metadata !== undefined) {
const m2 = Math.floor(
parseFloat(x.Metadata.mtime || x.Metadata.MTime || "0")
);
if (m2 !== 0) {
// to be compatible with RClone, we read and store the time in seconds in new version!
if (m2 >= 1000000000000) {
// it's a millsecond, uploaded by old codes..
mtimeCli = m2;
} else {
// it's a second, uploaded by new codes of the plugin from March 24, 2024
mtimeCli = m2 * 1000;
}
}
}
// console.debug(
// `fromS3HeadObjectToEntity, fileOrFolderPathWithRemotePrefix=${fileOrFolderPathWithRemotePrefix}, remotePrefix=${remotePrefix}, x=${JSON.stringify(
// x
// )} `
// );
const key = getLocalNoPrefixPath(
fileOrFolderPathWithRemotePrefix,
remotePrefix
);
// console.debug(`fromS3HeadObjectToEntity, key=${key} after removing prefix`);
return {
keyRaw: key,
mtimeSvr: mtimeSvr,
mtimeCli: mtimeCli,
sizeRaw: x.ContentLength,
etag: x.ETag,
} as Entity;
};
export const getS3Client = (s3Config: S3Config) => {
let endpoint = s3Config.s3Endpoint;
if (!(endpoint.startsWith("http://") || endpoint.startsWith("https://"))) {
endpoint = `https://${endpoint}`;
}
let s3Client: S3Client;
if (VALID_REQURL && s3Config.bypassCorsLocally) {
s3Client = new S3Client({
region: s3Config.s3Region,
endpoint: endpoint,
forcePathStyle: s3Config.forcePathStyle,
credentials: {
accessKeyId: s3Config.s3AccessKeyID,
secretAccessKey: s3Config.s3SecretAccessKey,
},
requestHandler: new ObsHttpHandler(),
});
} else {
s3Client = new S3Client({
region: s3Config.s3Region,
endpoint: endpoint,
forcePathStyle: s3Config.forcePathStyle,
credentials: {
accessKeyId: s3Config.s3AccessKeyID,
secretAccessKey: s3Config.s3SecretAccessKey,
},
});
}
s3Client.middlewareStack.add(
(next, context) => (args) => {
(args.request as any).headers["cache-control"] = "no-cache";
return next(args);
},
{
step: "build",
}
);
return s3Client;
};
export const getRemoteMeta = async (
s3Client: S3Client,
s3Config: S3Config,
fileOrFolderPathWithRemotePrefix: string
) => {
if (
s3Config.remotePrefix !== undefined &&
s3Config.remotePrefix !== "" &&
!fileOrFolderPathWithRemotePrefix.startsWith(s3Config.remotePrefix)
) {
throw Error(`s3 getRemoteMeta should only accept prefix-ed path`);
}
const res = await s3Client.send(
new HeadObjectCommand({
Bucket: s3Config.s3BucketName,
Key: fileOrFolderPathWithRemotePrefix,
})
);
return fromS3HeadObjectToEntity(
fileOrFolderPathWithRemotePrefix,
res,
s3Config.remotePrefix ?? ""
);
};
export const uploadToRemote = async (
s3Client: S3Client,
s3Config: S3Config,
fileOrFolderPath: string,
vault: Vault | undefined,
isRecursively: boolean,
cipher: Cipher,
remoteEncryptedKey: string = "",
uploadRaw: boolean = false,
rawContent: string | ArrayBuffer = "",
rawContentMTime: number = 0,
rawContentCTime: number = 0
): Promise<UploadedType> => {
console.debug(`uploading ${fileOrFolderPath}`);
let uploadFile = fileOrFolderPath;
if (!cipher.isPasswordEmpty()) {
if (remoteEncryptedKey === undefined || remoteEncryptedKey === "") {
throw Error(
`uploadToRemote(s3) you have password but remoteEncryptedKey is empty!`
);
}
uploadFile = remoteEncryptedKey;
}
uploadFile = getRemoteWithPrefixPath(uploadFile, s3Config.remotePrefix ?? "");
// console.debug(`actual uploadFile=${uploadFile}`);
const isFolder = fileOrFolderPath.endsWith("/");
if (isFolder && isRecursively) {
throw Error("upload function doesn't implement recursive function yet!");
} else if (isFolder && !isRecursively) {
if (uploadRaw) {
throw Error(`you specify uploadRaw, but you also provide a folder key!`);
}
// folder
let mtime = 0;
let ctime = 0;
const s = await vault?.adapter?.stat(fileOrFolderPath);
if (s !== undefined && s !== null) {
mtime = s.mtime;
ctime = s.ctime;
}
const contentType = DEFAULT_CONTENT_TYPE;
await s3Client.send(
new PutObjectCommand({
Bucket: s3Config.s3BucketName,
Key: uploadFile,
Body: "",
ContentType: contentType,
Metadata: {
MTime: `${mtime / 1000.0}`,
CTime: `${ctime / 1000.0}`,
},
})
);
const res = await getRemoteMeta(s3Client, s3Config, uploadFile);
return {
entity: res,
mtimeCli: mtime,
};
} else {
// file
// we ignore isRecursively parameter here
let contentType = DEFAULT_CONTENT_TYPE;
if (cipher.isPasswordEmpty()) {
contentType =
mime.contentType(
mime.lookup(fileOrFolderPath) || DEFAULT_CONTENT_TYPE
) || DEFAULT_CONTENT_TYPE;
}
let localContent = undefined;
let mtime = 0;
let ctime = 0;
if (uploadRaw) {
if (typeof rawContent === "string") {
localContent = new TextEncoder().encode(rawContent).buffer;
} else {
localContent = rawContent;
}
mtime = rawContentMTime;
ctime = rawContentCTime;
} else {
if (vault === undefined) {
throw new Error(
`the vault variable is not passed but we want to read ${fileOrFolderPath} for S3`
);
}
localContent = await vault.adapter.readBinary(fileOrFolderPath);
const s = await vault.adapter.stat(fileOrFolderPath);
if (s !== undefined && s !== null) {
mtime = s.mtime;
ctime = s.ctime;
}
}
let remoteContent = localContent;
if (!cipher.isPasswordEmpty()) {
remoteContent = await cipher.encryptContent(localContent);
}
const bytesIn5MB = 5242880;
const body = new Uint8Array(remoteContent);
const upload = new Upload({
client: s3Client,
queueSize: s3Config.partsConcurrency, // concurrency
partSize: bytesIn5MB, // minimal 5MB by default
leavePartsOnError: false,
params: {
Bucket: s3Config.s3BucketName,
Key: uploadFile,
Body: body,
ContentType: contentType,
Metadata: {
MTime: `${mtime / 1000.0}`,
CTime: `${ctime / 1000.0}`,
},
},
});
upload.on("httpUploadProgress", (progress) => {
// console.info(progress);
});
await upload.done();
const res = await getRemoteMeta(s3Client, s3Config, uploadFile);
// console.debug(
// `uploaded ${uploadFile} with res=${JSON.stringify(res, null, 2)}`
// );
return {
entity: res,
mtimeCli: mtime,
};
}
};
const listFromRemoteRaw = async (
s3Client: S3Client,
s3Config: S3Config,
prefixOfRawKeys?: string
) => {
const confCmd = {
Bucket: s3Config.s3BucketName,
} as ListObjectsV2CommandInput;
if (prefixOfRawKeys !== undefined && prefixOfRawKeys !== "") {
confCmd.Prefix = prefixOfRawKeys;
}
const contents = [] as _Object[];
const mtimeRecords: Record<string, number> = {};
const ctimeRecords: Record<string, number> = {};
const queueHead = new PQueue({
concurrency: s3Config.partsConcurrency,
autoStart: true,
});
queueHead.on("error", (error) => {
queueHead.pause();
queueHead.clear();
throw error;
});
let isTruncated = true;
do {
const rsp = await s3Client.send(new ListObjectsV2Command(confCmd));
if (rsp.$metadata.httpStatusCode !== 200) {
throw Error("some thing bad while listing remote!");
}
if (rsp.Contents === undefined) {
break;
}
contents.push(...rsp.Contents);
if (s3Config.useAccurateMTime) {
// head requests of all objects, love it
for (const content of rsp.Contents) {
queueHead.add(async () => {
const rspHead = await s3Client.send(
new HeadObjectCommand({
Bucket: s3Config.s3BucketName,
Key: content.Key,
})
);
if (rspHead.$metadata.httpStatusCode !== 200) {
throw Error("some thing bad while heading single object!");
}
if (rspHead.Metadata === undefined) {
// pass
} else {
mtimeRecords[content.Key!] = Math.floor(
parseFloat(
rspHead.Metadata.mtime || rspHead.Metadata.MTime || "0"
)
);
ctimeRecords[content.Key!] = Math.floor(
parseFloat(
rspHead.Metadata.ctime || rspHead.Metadata.CTime || "0"
)
);
}
});
}
}
isTruncated = rsp.IsTruncated ?? false;
confCmd.ContinuationToken = rsp.NextContinuationToken;
if (
isTruncated &&
(confCmd.ContinuationToken === undefined ||
confCmd.ContinuationToken === "")
) {
throw Error("isTruncated is true but no continuationToken provided");
}
} while (isTruncated);
// wait for any head requests
await queueHead.onIdle();
// ensemble fake rsp
// in the end, we need to transform the response list
// back to the local contents-alike list
return contents.map((x) =>
fromS3ObjectToEntity(
x,
s3Config.remotePrefix ?? "",
mtimeRecords,
ctimeRecords
)
);
};
export const listAllFromRemote = async (
s3Client: S3Client,
s3Config: S3Config
) => {
const res = (
await listFromRemoteRaw(s3Client, s3Config, s3Config.remotePrefix)
).filter((x) => x.keyRaw !== "" && x.keyRaw !== "/");
return res;
};
/**
* The Body of resp of aws GetObject has mix types
* and we want to get ArrayBuffer here.
* See https://github.com/aws/aws-sdk-js-v3/issues/1877
* @param b The Body of GetObject
* @returns Promise<ArrayBuffer>
*/
const getObjectBodyToArrayBuffer = async (
b: Readable | ReadableStream | Blob | undefined
) => {
if (b === undefined) {
throw Error(`ObjectBody is undefined and don't know how to deal with it`);
}
if (b instanceof Readable) {
return (await new Promise((resolve, reject) => {
const chunks: Uint8Array[] = [];
b.on("data", (chunk) => chunks.push(chunk));
b.on("error", reject);
b.on("end", () => resolve(bufferToArrayBuffer(Buffer.concat(chunks))));
})) as ArrayBuffer;
} else if (b instanceof ReadableStream) {
return await new Response(b, {}).arrayBuffer();
} else if (b instanceof Blob) {
return await b.arrayBuffer();
} else {
throw TypeError(`The type of ${b} is not one of the supported types`);
}
};
const downloadFromRemoteRaw = async (
s3Client: S3Client,
s3Config: S3Config,
fileOrFolderPathWithRemotePrefix: string
) => {
if (
s3Config.remotePrefix !== undefined &&
s3Config.remotePrefix !== "" &&
!fileOrFolderPathWithRemotePrefix.startsWith(s3Config.remotePrefix)
) {
throw Error(`downloadFromRemoteRaw should only accept prefix-ed path`);
}
const data = await s3Client.send(
new GetObjectCommand({
Bucket: s3Config.s3BucketName,
Key: fileOrFolderPathWithRemotePrefix,
})
);
const bodyContents = await getObjectBodyToArrayBuffer(data.Body);
return bodyContents;
};
export const downloadFromRemote = async (
s3Client: S3Client,
s3Config: S3Config,
fileOrFolderPath: string,
vault: Vault,
mtime: number,
cipher: Cipher,
remoteEncryptedKey: string,
skipSaving: boolean = false
) => {
const isFolder = fileOrFolderPath.endsWith("/");
if (!skipSaving) {
await mkdirpInVault(fileOrFolderPath, vault);
}
// the file is always local file
// we need to encrypt it
if (isFolder) {
// mkdirp locally is enough
// do nothing here
return new ArrayBuffer(0);
} else {
let downloadFile = fileOrFolderPath;
if (!cipher.isPasswordEmpty()) {
downloadFile = remoteEncryptedKey;
}
downloadFile = getRemoteWithPrefixPath(
downloadFile,
s3Config.remotePrefix ?? ""
);
const remoteContent = await downloadFromRemoteRaw(
s3Client,
s3Config,
downloadFile
);
let localContent = remoteContent;
if (!cipher.isPasswordEmpty()) {
localContent = await cipher.decryptContent(remoteContent);
}
if (!skipSaving) {
await vault.adapter.writeBinary(fileOrFolderPath, localContent, {
mtime: mtime,
});
}
return localContent;
}
};
/**
* This function deals with file normally and "folder" recursively.
* @param s3Client
* @param s3Config
* @param fileOrFolderPath
* @returns
*/
export const deleteFromRemote = async (
s3Client: S3Client,
s3Config: S3Config,
fileOrFolderPath: string,
cipher: Cipher,
remoteEncryptedKey: string = "",
synthesizedFolder: boolean = false
) => {
if (fileOrFolderPath === "/") {
return;
}
if (synthesizedFolder) {
return;
}
let remoteFileName = fileOrFolderPath;
if (!cipher.isPasswordEmpty()) {
remoteFileName = remoteEncryptedKey;
}
remoteFileName = getRemoteWithPrefixPath(
remoteFileName,
s3Config.remotePrefix ?? ""
);
await s3Client.send(
new DeleteObjectCommand({
Bucket: s3Config.s3BucketName,
Key: remoteFileName,
})
);
if (fileOrFolderPath.endsWith("/") && cipher.isPasswordEmpty()) {
const x = await listFromRemoteRaw(s3Client, s3Config, remoteFileName);
x.forEach(async (element) => {
await s3Client.send(
new DeleteObjectCommand({
Bucket: s3Config.s3BucketName,
Key: element.key,
})
);
});
} else if (fileOrFolderPath.endsWith("/") && !cipher.isPasswordEmpty()) {
// TODO
} else {
// pass
}
};
/**
* Check the config of S3 by heading bucket
* https://stackoverflow.com/questions/50842835
*
* Updated on 20240102:
* Users are not always have permission of heading bucket,
* so we need to use listing objects instead...
*
* @param s3Client
* @param s3Config
* @returns
*/
export const checkConnectivity = async (
s3Client: S3Client,
s3Config: S3Config,
callbackFunc?: any
) => {
try {
// TODO: no universal way now, just check this in connectivity
if (Platform.isIosApp && s3Config.s3Endpoint.startsWith("http://")) {
throw Error(
`Your s3 endpoint could only be https, not http, because of the iOS restriction.`
);
}
// const results = await s3Client.send(
// new HeadBucketCommand({ Bucket: s3Config.s3BucketName })
// );
// very simplified version of listing objects
const confCmd = {
Bucket: s3Config.s3BucketName,
} as ListObjectsV2CommandInput;
const results = await s3Client.send(new ListObjectsV2Command(confCmd));
if (
results === undefined ||
results.$metadata === undefined ||
results.$metadata.httpStatusCode === undefined
) {
const err = "results or $metadata or httStatusCode is undefined";
console.debug(err);
if (callbackFunc !== undefined) {
callbackFunc(err);
}
return false;
}
return results.$metadata.httpStatusCode === 200;
} catch (err: any) {
console.debug(err);
if (callbackFunc !== undefined) {
if (s3Config.s3Endpoint.contains(s3Config.s3BucketName)) {
const err2 = new AggregateError([
err,
new Error(
"Maybe you've included the bucket name inside the endpoint setting. Please remove the bucket name and try again."
),
]);
callbackFunc(err2);
} else {
callbackFunc(err);
}
}
return false;
}
};
-608
View File
@@ -1,608 +0,0 @@
import { Buffer } from "buffer";
import { Platform, Vault, requestUrl } from "obsidian";
import { Queue } from "@fyears/tsqueue";
import chunk from "lodash/chunk";
import flatten from "lodash/flatten";
import cloneDeep from "lodash/cloneDeep";
import { getReasonPhrase } from "http-status-codes";
import { Entity, UploadedType, VALID_REQURL, WebdavConfig } from "./baseTypes";
import { bufferToArrayBuffer, getPathFolder, mkdirpInVault } from "./misc";
import { Cipher } from "./encryptUnified";
import type {
FileStat,
WebDAVClient,
RequestOptionsWithState,
// Response,
// ResponseDataDetailed,
} from "webdav";
/**
* https://stackoverflow.com/questions/32850898/how-to-check-if-a-string-has-any-non-iso-8859-1-characters-with-javascript
* @param str
* @returns true if all are iso 8859 1 chars
*/
function onlyAscii(str: string) {
return !/[^\u0000-\u00ff]/g.test(str);
}
/**
* https://stackoverflow.com/questions/12539574/
* @param obj
* @returns
*/
function objKeyToLower(obj: Record<string, string>) {
return Object.fromEntries(
Object.entries(obj).map(([k, v]) => [k.toLowerCase(), v])
);
}
// @ts-ignore
import { getPatcher } from "webdav/dist/web/index.js";
if (VALID_REQURL) {
getPatcher().patch(
"request",
async (options: RequestOptionsWithState): Promise<Response> => {
const transformedHeaders = objKeyToLower({ ...options.headers });
delete transformedHeaders["host"];
delete transformedHeaders["content-length"];
const reqContentType =
transformedHeaders["accept"] ?? transformedHeaders["content-type"];
const retractedHeaders = { ...transformedHeaders };
if (retractedHeaders.hasOwnProperty("authorization")) {
retractedHeaders["authorization"] = "<retracted>";
}
console.debug(`before request:`);
console.debug(`url: ${options.url}`);
console.debug(`method: ${options.method}`);
console.debug(`headers: ${JSON.stringify(retractedHeaders, null, 2)}`);
console.debug(`reqContentType: ${reqContentType}`);
let r = await requestUrl({
url: options.url,
method: options.method,
body: options.data as string | ArrayBuffer,
headers: transformedHeaders,
contentType: reqContentType,
throw: false,
});
if (
r.status === 401 &&
Platform.isIosApp &&
!options.url.endsWith("/") &&
!options.url.endsWith(".md") &&
options.method.toUpperCase() === "PROPFIND"
) {
// don't ask me why,
// some webdav servers have some mysterious behaviours,
// if a folder doesn't exist without slash, the servers return 401 instead of 404
// here is a dirty hack that works
console.debug(`so we have 401, try appending request url with slash`);
r = await requestUrl({
url: `${options.url}/`,
method: options.method,
body: options.data as string | ArrayBuffer,
headers: transformedHeaders,
contentType: reqContentType,
throw: false,
});
}
console.debug(`after request:`);
const rspHeaders = objKeyToLower({ ...r.headers });
console.debug(`rspHeaders: ${JSON.stringify(rspHeaders, null, 2)}`);
for (let key in rspHeaders) {
if (rspHeaders.hasOwnProperty(key)) {
// avoid the error:
// Failed to read the 'headers' property from 'ResponseInit': String contains non ISO-8859-1 code point.
// const possibleNonAscii = [
// "Content-Disposition",
// "X-Accel-Redirect",
// "X-Outfilename",
// "X-Sendfile"
// ];
// for (const p of possibleNonAscii) {
// if (key === p || key === p.toLowerCase()) {
// rspHeaders[key] = encodeURIComponent(rspHeaders[key]);
// }
// }
if (!onlyAscii(rspHeaders[key])) {
console.debug(`rspHeaders[key] needs encode: ${key}`);
rspHeaders[key] = encodeURIComponent(rspHeaders[key]);
}
}
}
let r2: Response | undefined = undefined;
const statusText = getReasonPhrase(r.status);
console.debug(`statusText: ${statusText}`);
if ([101, 103, 204, 205, 304].includes(r.status)) {
// A null body status is a status that is 101, 103, 204, 205, or 304.
// https://fetch.spec.whatwg.org/#statuses
// fix this: Failed to construct 'Response': Response with null body status cannot have body
r2 = new Response(null, {
status: r.status,
statusText: statusText,
headers: rspHeaders,
});
} else {
r2 = new Response(r.arrayBuffer, {
status: r.status,
statusText: statusText,
headers: rspHeaders,
});
}
return r2;
}
);
}
// @ts-ignore
import { AuthType, BufferLike, createClient } from "webdav/dist/web/index.js";
export type { WebDAVClient } from "webdav";
export const DEFAULT_WEBDAV_CONFIG = {
address: "",
username: "",
password: "",
authType: "basic",
manualRecursive: true,
depth: "manual_1",
remoteBaseDir: "",
} as WebdavConfig;
const getWebdavPath = (fileOrFolderPath: string, remoteBaseDir: string) => {
let key = fileOrFolderPath;
if (fileOrFolderPath === "/" || fileOrFolderPath === "") {
// special
key = `/${remoteBaseDir}/`;
} else if (fileOrFolderPath.startsWith("/")) {
console.warn(
`why the path ${fileOrFolderPath} starts with '/'? but we just go on.`
);
key = `/${remoteBaseDir}${fileOrFolderPath}`;
} else {
key = `/${remoteBaseDir}/${fileOrFolderPath}`;
}
return key;
};
const getNormPath = (fileOrFolderPath: string, remoteBaseDir: string) => {
if (
!(
fileOrFolderPath === `/${remoteBaseDir}` ||
fileOrFolderPath.startsWith(`/${remoteBaseDir}/`)
)
) {
throw Error(
`"${fileOrFolderPath}" doesn't starts with "/${remoteBaseDir}/"`
);
}
// if (fileOrFolderPath.startsWith("/")) {
// return fileOrFolderPath.slice(1);
// }
return fileOrFolderPath.slice(`/${remoteBaseDir}/`.length);
};
const fromWebdavItemToEntity = (x: FileStat, remoteBaseDir: string) => {
let key = getNormPath(x.filename, remoteBaseDir);
if (x.type === "directory" && !key.endsWith("/")) {
key = `${key}/`;
}
const mtimeSvr = Date.parse(x.lastmod).valueOf();
return {
keyRaw: key,
mtimeSvr: mtimeSvr,
mtimeCli: mtimeSvr, // no universal way to set mtime in webdav
sizeRaw: x.size,
etag: x.etag,
} as Entity;
};
export class WrappedWebdavClient {
webdavConfig: WebdavConfig;
remoteBaseDir: string;
client!: WebDAVClient;
vaultFolderExists: boolean;
saveUpdatedConfigFunc: () => Promise<any>;
constructor(
webdavConfig: WebdavConfig,
remoteBaseDir: string,
saveUpdatedConfigFunc: () => Promise<any>
) {
this.webdavConfig = cloneDeep(webdavConfig);
this.webdavConfig.address = encodeURI(this.webdavConfig.address);
this.remoteBaseDir = remoteBaseDir;
this.vaultFolderExists = false;
this.saveUpdatedConfigFunc = saveUpdatedConfigFunc;
}
init = async () => {
// init client if not inited
if (this.client !== undefined) {
return;
}
if (Platform.isIosApp && !this.webdavConfig.address.startsWith("https")) {
throw Error(
`Your webdav address could only be https, not http, because of the iOS restriction.`
);
}
const headers = {
"Cache-Control": "no-cache",
};
if (
this.webdavConfig.username !== "" &&
this.webdavConfig.password !== ""
) {
this.client = createClient(this.webdavConfig.address, {
username: this.webdavConfig.username,
password: this.webdavConfig.password,
headers: headers,
authType:
this.webdavConfig.authType === "digest"
? AuthType.Digest
: AuthType.Password,
});
} else {
console.info("no password");
this.client = createClient(this.webdavConfig.address, {
headers: headers,
});
}
// check vault folder
if (this.vaultFolderExists) {
// pass
} else {
const res = await this.client.exists(`/${this.remoteBaseDir}/`);
if (res) {
// console.info("remote vault folder exits!");
this.vaultFolderExists = true;
} else {
console.info("remote vault folder not exists, creating");
await this.client.createDirectory(`/${this.remoteBaseDir}/`);
console.info("remote vault folder created!");
this.vaultFolderExists = true;
}
}
// adjust depth parameter
if (
this.webdavConfig.depth === "auto" ||
this.webdavConfig.depth === "auto_1" ||
this.webdavConfig.depth === "auto_infinity" ||
this.webdavConfig.depth === "auto_unknown"
) {
this.webdavConfig.depth = "manual_1";
this.webdavConfig.manualRecursive = true;
if (this.saveUpdatedConfigFunc !== undefined) {
await this.saveUpdatedConfigFunc();
console.info(
`webdav depth="auto_???" is changed to ${this.webdavConfig.depth}`
);
}
}
};
}
export const getWebdavClient = (
webdavConfig: WebdavConfig,
remoteBaseDir: string,
saveUpdatedConfigFunc: () => Promise<any>
) => {
return new WrappedWebdavClient(
webdavConfig,
remoteBaseDir,
saveUpdatedConfigFunc
);
};
/**
*
* @param client
* @param remotePath It should be prefix-ed already
* @returns
*/
export const getRemoteMeta = async (
client: WrappedWebdavClient,
remotePath: string
) => {
await client.init();
console.debug(`getRemoteMeta remotePath = ${remotePath}`);
const res = (await client.client.stat(remotePath, {
details: false,
})) as FileStat;
console.debug(`getRemoteMeta res=${JSON.stringify(res)}`);
return fromWebdavItemToEntity(res, client.remoteBaseDir);
};
export const uploadToRemote = async (
client: WrappedWebdavClient,
fileOrFolderPath: string,
vault: Vault | undefined,
isRecursively: boolean,
cipher: Cipher,
remoteEncryptedKey: string = "",
uploadRaw: boolean = false,
rawContent: string | ArrayBuffer = ""
): Promise<UploadedType> => {
await client.init();
let uploadFile = fileOrFolderPath;
if (!cipher.isPasswordEmpty()) {
if (remoteEncryptedKey === undefined || remoteEncryptedKey === "") {
throw Error(
`uploadToRemote(webdav) you have password but remoteEncryptedKey is empty!`
);
}
uploadFile = remoteEncryptedKey;
}
uploadFile = getWebdavPath(uploadFile, client.remoteBaseDir);
const isFolder = fileOrFolderPath.endsWith("/");
if (isFolder && isRecursively) {
throw Error("upload function doesn't implement recursive function yet!");
} else if (isFolder && !isRecursively) {
if (uploadRaw) {
throw Error(`you specify uploadRaw, but you also provide a folder key!`);
}
// folder
if (cipher.isPasswordEmpty() || cipher.isFolderAware()) {
// if not encrypted, || encrypted isFolderAware, mkdir a remote folder
await client.client.createDirectory(uploadFile, {
recursive: true,
});
const res = await getRemoteMeta(client, uploadFile);
return {
entity: res,
};
} else {
// if encrypted && !isFolderAware(),
// upload a fake file with the encrypted file name
await client.client.putFileContents(uploadFile, "", {
overwrite: true,
onUploadProgress: (progress: any) => {
// console.info(`Uploaded ${progress.loaded} bytes of ${progress.total}`);
},
});
return {
entity: await getRemoteMeta(client, uploadFile),
};
}
} else {
// file
// we ignore isRecursively parameter here
let localContent: ArrayBuffer | undefined = undefined;
let mtimeCli: number | undefined = undefined;
if (uploadRaw) {
if (typeof rawContent === "string") {
localContent = new TextEncoder().encode(rawContent).buffer;
} else {
localContent = rawContent;
}
} else {
if (vault == undefined) {
throw new Error(
`the vault variable is not passed but we want to read ${fileOrFolderPath} for webdav`
);
}
localContent = await vault.adapter.readBinary(fileOrFolderPath);
mtimeCli = (await vault.adapter.stat(fileOrFolderPath))?.mtime;
}
let remoteContent = localContent;
if (!cipher.isPasswordEmpty()) {
remoteContent = await cipher.encryptContent(localContent);
}
// updated 20220326: the algorithm guarantee this
// // we need to create folders before uploading
// const dir = getPathFolder(uploadFile);
// if (dir !== "/" && dir !== "") {
// await client.client.createDirectory(dir, { recursive: true });
// }
await client.client.putFileContents(uploadFile, remoteContent, {
overwrite: true,
onUploadProgress: (progress: any) => {
console.info(`Uploaded ${progress.loaded} bytes of ${progress.total}`);
},
});
return {
entity: await getRemoteMeta(client, uploadFile),
mtimeCli: mtimeCli,
};
}
};
export const listAllFromRemote = async (client: WrappedWebdavClient) => {
await client.init();
let contents = [] as FileStat[];
if (
client.webdavConfig.depth === "auto" ||
client.webdavConfig.depth === "auto_unknown" ||
client.webdavConfig.depth === "auto_1" ||
client.webdavConfig.depth === "auto_infinity" /* don't trust auto now */ ||
client.webdavConfig.depth === "manual_1"
) {
// the remote doesn't support infinity propfind,
// we need to do a bfs here
const q = new Queue([`/${client.remoteBaseDir}`]);
const CHUNK_SIZE = 10;
while (q.length > 0) {
const itemsToFetch: string[] = [];
while (q.length > 0) {
itemsToFetch.push(q.pop()!);
}
const itemsToFetchChunks = chunk(itemsToFetch, CHUNK_SIZE);
// console.debug(itemsToFetchChunks);
const subContents = [] as FileStat[];
for (const singleChunk of itemsToFetchChunks) {
const r = singleChunk.map((x) => {
return client.client.getDirectoryContents(x, {
deep: false,
details: false /* no need for verbose details here */,
// TODO: to support .obsidian,
// we need to load all files including dot,
// anyway to reduce the resources?
// glob: "/**" /* avoid dot files by using glob */,
}) as Promise<FileStat[]>;
});
const r2 = flatten(await Promise.all(r));
subContents.push(...r2);
}
for (let i = 0; i < subContents.length; ++i) {
const f = subContents[i];
contents.push(f);
if (f.type === "directory") {
q.push(f.filename);
}
}
}
} else {
// the remote supports infinity propfind
contents = (await client.client.getDirectoryContents(
`/${client.remoteBaseDir}`,
{
deep: true,
details: false /* no need for verbose details here */,
// TODO: to support .obsidian,
// we need to load all files including dot,
// anyway to reduce the resources?
// glob: "/**" /* avoid dot files by using glob */,
}
)) as FileStat[];
}
return contents.map((x) => fromWebdavItemToEntity(x, client.remoteBaseDir));
};
const downloadFromRemoteRaw = async (
client: WrappedWebdavClient,
remotePath: string
) => {
await client.init();
// console.info(`getWebdavPath=${remotePath}`);
const buff = (await client.client.getFileContents(remotePath)) as BufferLike;
if (buff instanceof ArrayBuffer) {
return buff;
} else if (buff instanceof Buffer) {
return bufferToArrayBuffer(buff);
}
throw Error(`unexpected file content result with type ${typeof buff}`);
};
export const downloadFromRemote = async (
client: WrappedWebdavClient,
fileOrFolderPath: string,
vault: Vault,
mtime: number,
cipher: Cipher,
remoteEncryptedKey: string = "",
skipSaving: boolean = false
) => {
await client.init();
const isFolder = fileOrFolderPath.endsWith("/");
if (!skipSaving) {
await mkdirpInVault(fileOrFolderPath, vault);
}
// the file is always local file
// we need to encrypt it
if (isFolder) {
// mkdirp locally is enough
// do nothing here
return new ArrayBuffer(0);
} else {
let downloadFile = fileOrFolderPath;
if (!cipher.isPasswordEmpty()) {
downloadFile = remoteEncryptedKey;
}
downloadFile = getWebdavPath(downloadFile, client.remoteBaseDir);
// console.info(`downloadFile=${downloadFile}`);
const remoteContent = await downloadFromRemoteRaw(client, downloadFile);
let localContent = remoteContent;
if (!cipher.isPasswordEmpty()) {
localContent = await cipher.decryptContent(remoteContent);
}
if (!skipSaving) {
await vault.adapter.writeBinary(fileOrFolderPath, localContent, {
mtime: mtime,
});
}
return localContent;
}
};
export const deleteFromRemote = async (
client: WrappedWebdavClient,
fileOrFolderPath: string,
cipher: Cipher,
remoteEncryptedKey: string = ""
) => {
if (fileOrFolderPath === "/") {
return;
}
let remoteFileName = fileOrFolderPath;
if (!cipher.isPasswordEmpty()) {
remoteFileName = remoteEncryptedKey;
}
remoteFileName = getWebdavPath(remoteFileName, client.remoteBaseDir);
await client.init();
try {
await client.client.deleteFile(remoteFileName);
// console.info(`delete ${remoteFileName} succeeded`);
} catch (err) {
console.error("some error while deleting");
console.error(err);
}
};
export const checkConnectivity = async (
client: WrappedWebdavClient,
callbackFunc?: any
) => {
if (
!(
client.webdavConfig.address.startsWith("http://") ||
client.webdavConfig.address.startsWith("https://")
)
) {
const err = "Error: the url should start with http(s):// but it does not!";
console.error(err);
if (callbackFunc !== undefined) {
callbackFunc(err);
}
return false;
}
try {
await client.init();
const results = await getRemoteMeta(client, `/${client.remoteBaseDir}/`);
if (results === undefined) {
const err = "results is undefined";
console.error(err);
if (callbackFunc !== undefined) {
callbackFunc(err);
}
return false;
}
return true;
} catch (err) {
console.error(err);
if (callbackFunc !== undefined) {
callbackFunc(err);
}
return false;
}
};
+225 -112
View File
@@ -11,20 +11,23 @@ import {
import type { TextComponent } from "obsidian";
import { createElement, Eye, EyeOff } from "lucide";
import {
API_VER_ENSURE_REQURL_OK,
API_VER_REQURL,
ConflictActionType,
DEFAULT_DEBUG_FOLDER,
EmptyFolderCleanType,
SUPPORTED_SERVICES_TYPE,
SUPPORTED_SERVICES_TYPE_WITH_REMOTE_BASE_DIR,
SyncDirectionType,
VALID_REQURL,
WebdavAuthType,
WebdavDepthType,
CipherMethodType,
QRExportType,
} from "./baseTypes";
import {
API_VER_ENSURE_REQURL_OK,
API_VER_REQURL,
VALID_REQURL,
} from "./baseTypesObs";
import {
exportVaultProfilerResultsToFiles,
exportVaultSyncPlansToFiles,
@@ -41,17 +44,17 @@ import {
upsertLastSuccessSyncTimeByVault,
} from "./localdb";
import type RemotelySavePlugin from "./main"; // unavoidable
import { RemoteClient } from "./remote";
import { FakeFs } from "./fsAll";
import {
DEFAULT_DROPBOX_CONFIG,
getAuthUrlAndVerifier as getAuthUrlAndVerifierDropbox,
sendAuthReq as sendAuthReqDropbox,
setConfigBySuccessfullAuthInplace,
} from "./remoteForDropbox";
} from "./fsDropbox";
import {
DEFAULT_ONEDRIVE_CONFIG,
getAuthUrlAndVerifier as getAuthUrlAndVerifierOnedrive,
} from "./remoteForOnedrive";
} from "./fsOnedrive";
import { messyConfigToNormal } from "./configPersist";
import type { TransItemType } from "./i18n";
import {
@@ -59,8 +62,9 @@ import {
checkHasSpecialCharForDir,
stringToFragment,
} from "./misc";
import { simpleTransRemotePrefix } from "./remoteForS3";
import { simpleTransRemotePrefix } from "./fsS3";
import cloneDeep from "lodash/cloneDeep";
import { getClient } from "./fsGetter";
class PasswordModal extends Modal {
plugin: RemotelySavePlugin;
@@ -468,16 +472,12 @@ class DropboxAuthModal extends Modal {
authRes!,
() => self.plugin.saveSettings()
);
const client = new RemoteClient(
"dropbox",
undefined,
undefined,
this.plugin.settings.dropbox,
undefined,
const client = getClient(
this.plugin.settings,
this.app.vault.getName(),
() => self.plugin.saveSettings()
() => this.plugin.saveSettings()
);
const username = await client.getUser();
const username = await client.getUserDisplayName();
this.plugin.settings.dropbox.username = username;
await this.plugin.saveSettings();
new Notice(
@@ -811,6 +811,7 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
display(): void {
let { containerEl } = this;
containerEl.style.setProperty("overflow-wrap", "break-word");
containerEl.empty();
@@ -1056,6 +1057,46 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
).open();
});
});
new Setting(s3Div)
.setName(t("settings_s3_reverse_proxy_no_sign_url"))
.setDesc(t("settings_s3_reverse_proxy_no_sign_url_desc"))
.addText((text) =>
text
.setPlaceholder("")
.setValue(this.plugin.settings.s3.reverseProxyNoSignUrl ?? "")
.onChange(async (value) => {
this.plugin.settings.s3.reverseProxyNoSignUrl = value.trim();
await this.plugin.saveSettings();
})
);
new Setting(s3Div)
.setName(t("settings_s3_generatefolderobject"))
.setDesc(t("settings_s3_generatefolderobject_desc"))
.addDropdown((dropdown) => {
dropdown
.addOption(
"notgenerate",
t("settings_s3_generatefolderobject_notgenerate")
)
.addOption(
"generate",
t("settings_s3_generatefolderobject_generate")
);
dropdown
.setValue(
`${this.plugin.settings.s3.generateFolderObject ? "generate" : "notgenerate"}`
)
.onChange(async (val) => {
if (val === "generate") {
this.plugin.settings.s3.generateFolderObject = true;
} else {
this.plugin.settings.s3.generateFolderObject = false;
}
await this.plugin.saveSettings();
});
});
new Setting(s3Div)
.setName(t("settings_checkonnectivity"))
@@ -1064,9 +1105,13 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
button.setButtonText(t("settings_checkonnectivity_button"));
button.onClick(async () => {
new Notice(t("settings_checkonnectivity_checking"));
const client = new RemoteClient("s3", this.plugin.settings.s3);
const client = getClient(
this.plugin.settings,
this.app.vault.getName(),
() => this.plugin.saveSettings()
);
const errors = { msg: "" };
const res = await client.checkConnectivity((err: any) => {
const res = await client.checkConnect((err: any) => {
errors.msg = err;
});
if (res) {
@@ -1130,14 +1175,10 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
button.onClick(async () => {
try {
const self = this;
const client = new RemoteClient(
"dropbox",
undefined,
undefined,
this.plugin.settings.dropbox,
undefined,
const client = getClient(
this.plugin.settings,
this.app.vault.getName(),
() => self.plugin.saveSettings()
() => this.plugin.saveSettings()
);
await client.revokeAuth();
this.plugin.settings.dropbox = JSON.parse(
@@ -1245,18 +1286,14 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
button.onClick(async () => {
new Notice(t("settings_checkonnectivity_checking"));
const self = this;
const client = new RemoteClient(
"dropbox",
undefined,
undefined,
this.plugin.settings.dropbox,
undefined,
const client = getClient(
this.plugin.settings,
this.app.vault.getName(),
() => self.plugin.saveSettings()
() => this.plugin.saveSettings()
);
const errors = { msg: "" };
const res = await client.checkConnectivity((err: any) => {
const res = await client.checkConnect((err: any) => {
errors.msg = `${err}`;
});
if (res) {
@@ -1394,18 +1431,13 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
button.onClick(async () => {
new Notice(t("settings_checkonnectivity_checking"));
const self = this;
const client = new RemoteClient(
"onedrive",
undefined,
undefined,
undefined,
this.plugin.settings.onedrive,
const client = getClient(
this.plugin.settings,
this.app.vault.getName(),
() => self.plugin.saveSettings()
() => this.plugin.saveSettings()
);
const errors = { msg: "" };
const res = await client.checkConnectivity((err: any) => {
const res = await client.checkConnect((err: any) => {
errors.msg = `${err}`;
});
if (res) {
@@ -1604,17 +1636,13 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
button.onClick(async () => {
new Notice(t("settings_checkonnectivity_checking"));
const self = this;
const client = new RemoteClient(
"webdav",
undefined,
this.plugin.settings.webdav,
undefined,
undefined,
const client = getClient(
this.plugin.settings,
this.app.vault.getName(),
() => self.plugin.saveSettings()
() => this.plugin.saveSettings()
);
const errors = { msg: "" };
const res = await client.checkConnectivity((err: any) => {
const res = await client.checkConnect((err: any) => {
errors.msg = `${err}`;
});
if (res) {
@@ -1630,6 +1658,132 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
});
});
//////////////////////////////////////////////////
// below for webdis
//////////////////////////////////////////////////
const webdisDiv = containerEl.createEl("div", { cls: "webdis-hide" });
webdisDiv.toggleClass(
"webdis-hide",
this.plugin.settings.serviceType !== "webdis"
);
webdisDiv.createEl("h2", { text: t("settings_webdis") });
const webdisLongDescDiv = webdisDiv.createEl("div", {
cls: "settings-long-desc",
});
for (const c of [
t("settings_webdis_disclaimer1"),
t("settings_webdis_disclaimer2"),
]) {
webdisLongDescDiv.createEl("p", {
text: c,
cls: "webdis-disclaimer",
});
}
webdisLongDescDiv.createEl("p", {
text: t("settings_webdis_folder", {
remoteBaseDir:
this.plugin.settings.webdis.remoteBaseDir || this.app.vault.getName(),
}),
});
new Setting(webdisDiv)
.setName(t("settings_webdis_addr"))
.setDesc(t("settings_webdis_addr_desc"))
.addText((text) =>
text
.setPlaceholder("https://")
.setValue(this.plugin.settings.webdis.address)
.onChange(async (value) => {
this.plugin.settings.webdis.address = value.trim();
// normally saved
await this.plugin.saveSettings();
})
);
new Setting(webdisDiv)
.setName(t("settings_webdis_user"))
.setDesc(t("settings_webdis_user_desc"))
.addText((text) => {
wrapTextWithPasswordHide(text);
text
.setPlaceholder("")
.setValue(this.plugin.settings.webdis.username ?? "")
.onChange(async (value) => {
this.plugin.settings.webdis.username = (value ?? "").trim();
await this.plugin.saveSettings();
});
});
new Setting(webdisDiv)
.setName(t("settings_webdis_password"))
.setDesc(t("settings_webdis_password_desc"))
.addText((text) => {
wrapTextWithPasswordHide(text);
text
.setPlaceholder("")
.setValue(this.plugin.settings.webdis.password ?? "")
.onChange(async (value) => {
this.plugin.settings.webdis.password = (value ?? "").trim();
await this.plugin.saveSettings();
});
});
let newWebdisRemoteBaseDir =
this.plugin.settings.webdis.remoteBaseDir || "";
new Setting(webdisDiv)
.setName(t("settings_remotebasedir"))
.setDesc(t("settings_remotebasedir_desc"))
.addText((text) =>
text
.setPlaceholder(this.app.vault.getName())
.setValue(newWebdisRemoteBaseDir)
.onChange((value) => {
newWebdisRemoteBaseDir = value.trim();
})
)
.addButton((button) => {
button.setButtonText(t("confirm"));
button.onClick(() => {
new ChangeRemoteBaseDirModal(
this.app,
this.plugin,
newWebdisRemoteBaseDir,
"webdis"
).open();
});
});
new Setting(webdisDiv)
.setName(t("settings_checkonnectivity"))
.setDesc(t("settings_checkonnectivity_desc"))
.addButton(async (button) => {
button.setButtonText(t("settings_checkonnectivity_button"));
button.onClick(async () => {
new Notice(t("settings_checkonnectivity_checking"));
const self = this;
const client = getClient(
this.plugin.settings,
this.app.vault.getName(),
() => this.plugin.saveSettings()
);
const errors = { msg: "" };
const res = await client.checkConnect((err: any) => {
errors.msg = `${err}`;
});
if (res) {
new Notice(t("settings_webdis_connect_succ"));
} else {
new Notice(t("settings_webdis_connect_fail"));
new Notice(errors.msg);
}
});
});
//////////////////////////////////////////////////
// below for general chooser (part 2/2)
//////////////////////////////////////////////////
@@ -1644,6 +1798,8 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
dropdown.addOption("dropbox", t("settings_chooseservice_dropbox"));
dropdown.addOption("webdav", t("settings_chooseservice_webdav"));
dropdown.addOption("onedrive", t("settings_chooseservice_onedrive"));
dropdown.addOption("webdis", t("settings_chooseservice_webdis"));
dropdown
.setValue(this.plugin.settings.serviceType)
.onChange(async (val) => {
@@ -1664,6 +1820,10 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
"webdav-hide",
this.plugin.settings.serviceType !== "webdav"
);
webdisDiv.toggleClass(
"webdis-hide",
this.plugin.settings.serviceType !== "webdis"
);
await this.plugin.saveSettings();
});
});
@@ -1777,72 +1937,22 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
});
new Setting(basicDiv)
.setName(t("settings_saverun"))
.setDesc(t("settings_saverun_desc"))
.setName(t("settings_synconsave"))
.setDesc(t("settings_synconsave_desc"))
.addDropdown((dropdown) => {
dropdown.addOption("-1", t("settings_saverun_notset"));
dropdown.addOption(`${1000 * 1}`, t("settings_saverun_1sec"));
dropdown.addOption(`${1000 * 5}`, t("settings_saverun_5sec"));
dropdown.addOption(`${1000 * 10}`, t("settings_saverun_10sec"));
dropdown.addOption(`${1000 * 60}`, t("settings_saverun_1min"));
let runScheduled = false;
dropdown.addOption("-1", t("settings_synconsave_disable"));
dropdown.addOption("1000", t("settings_synconsave_enable"));
// for backward compatibility, we need to use a number representing seconds
let syncOnSaveEnabled = false;
if ((this.plugin.settings.syncOnSaveAfterMilliseconds ?? -1) > 0) {
syncOnSaveEnabled = true;
}
dropdown
.setValue(`${this.plugin.settings.syncOnSaveAfterMilliseconds}`)
.setValue(`${syncOnSaveEnabled ? "1000" : "-1"}`)
.onChange(async (val: string) => {
const realVal = parseInt(val);
this.plugin.settings.syncOnSaveAfterMilliseconds = realVal;
this.plugin.settings.syncOnSaveAfterMilliseconds = parseInt(val);
await this.plugin.saveSettings();
if (
(realVal === undefined || realVal === null || realVal <= 0) &&
this.plugin.syncOnSaveIntervalID !== undefined
) {
// clear
window.clearInterval(this.plugin.syncOnSaveIntervalID);
this.plugin.syncOnSaveIntervalID = undefined;
} else if (
realVal !== undefined &&
realVal !== null &&
realVal > 0
) {
const intervalID = window.setInterval(() => {
const currentFile = this.app.workspace.getActiveFile();
if (currentFile) {
// get the last modified time of the current file
// if it has been modified within the last syncOnSaveAfterMilliseconds
// then schedule a run for syncOnSaveAfterMilliseconds after it was modified
const lastModified = currentFile.stat.mtime;
const currentTime = Date.now();
// console.debug(
// `Checking if file was modified within last ${
// this.plugin.settings.syncOnSaveAfterMilliseconds / 1000
// } seconds, last modified: ${
// (currentTime - lastModified) / 1000
// } seconds ago`
// );
if (
currentTime - lastModified <
this.plugin.settings.syncOnSaveAfterMilliseconds!
) {
if (!runScheduled) {
const scheduleTimeFromNow =
this.plugin.settings.syncOnSaveAfterMilliseconds! -
(currentTime - lastModified);
console.info(
`schedule a run for ${scheduleTimeFromNow} milliseconds later`
);
runScheduled = true;
setTimeout(() => {
this.plugin.syncRun("auto_sync_on_save");
runScheduled = false;
}, scheduleTimeFromNow);
}
}
}
}, realVal);
this.plugin.syncOnSaveIntervalID = intervalID;
this.plugin.registerInterval(intervalID);
}
this.plugin.toggleSyncOnSaveIfSet();
});
});
@@ -2043,7 +2153,7 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
t("settings_cleanemptyfolder_clean_both")
);
dropdown
.setValue(this.plugin.settings.howToCleanEmptyFolder ?? "skip")
.setValue(this.plugin.settings.howToCleanEmptyFolder ?? "clean_both")
.onChange(async (val) => {
this.plugin.settings.howToCleanEmptyFolder =
val as EmptyFolderCleanType;
@@ -2144,6 +2254,9 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
importExportDiv.createEl("h2", {
text: t("settings_importexport"),
});
if (Platform.isMobile) {
importExportDiv.addClass("setting-need-wrapping-mobile");
}
new Setting(importExportDiv)
.setName(t("settings_export"))
+377 -384
View File
@@ -1,156 +1,37 @@
import PQueue from "p-queue";
import XRegExp from "xregexp";
import type {
CipherMethodType,
import {
ConflictActionType,
EmptyFolderCleanType,
Entity,
MixedEntity,
RemotelySavePluginSettings,
SUPPORTED_SERVICES_TYPE,
SyncDirectionType,
SyncTriggerSourceType,
} from "./baseTypes";
import { isInsideObsFolder } from "./obsFolderLister";
import { FakeFs } from "./fsAll";
import { FakeFsEncrypt } from "./fsEncrypt";
import {
InternalDBs,
clearPrevSyncRecordByVaultAndProfile,
getAllPrevSyncRecordsByVaultAndProfile,
insertSyncPlanRecordByVault,
upsertPrevSyncRecordByVaultAndProfile,
} from "./localdb";
import {
isSpecialFolderNameToSkip,
isHiddenPath,
unixTimeToStr,
getParentFolder,
isVaildText,
atWhichLevel,
mkdirpInVault,
getFolderLevels,
getParentFolder,
isHiddenPath,
isSpecialFolderNameToSkip,
unixTimeToStr,
} from "./misc";
import { Profiler } from "./profiler";
import {
DEFAULT_FILE_NAME_FOR_METADATAONREMOTE,
DEFAULT_FILE_NAME_FOR_METADATAONREMOTE2,
} from "./metadataOnRemote";
import { RemoteClient } from "./remote";
import { Vault } from "obsidian";
import AggregateError from "aggregate-error";
import {
InternalDBs,
clearPrevSyncRecordByVaultAndProfile,
upsertPrevSyncRecordByVaultAndProfile,
} from "./localdb";
import { Cipher } from "./encryptUnified";
import { Profiler } from "./profiler";
export type SyncStatusType =
| "idle"
| "preparing"
| "getting_remote_files_list"
| "getting_local_meta"
| "getting_local_prev_sync"
| "checking_password"
| "generating_plan"
| "syncing"
| "cleaning"
| "finish";
export interface PasswordCheckType {
ok: boolean;
reason:
| "empty_remote"
| "unknown_encryption_method"
| "remote_encrypted_local_no_password"
| "password_matched"
| "password_or_method_not_matched_or_remote_not_encrypted"
| "likely_no_password_both_sides"
| "encryption_method_not_matched";
}
export const isPasswordOk = async (
remote: Entity[],
cipher: Cipher
): Promise<PasswordCheckType> => {
if (remote === undefined || remote.length === 0) {
// remote empty
return {
ok: true,
reason: "empty_remote",
};
}
const santyCheckKey = remote[0].keyRaw;
if (cipher.isPasswordEmpty()) {
// TODO: no way to distinguish remote rclone encrypted
// if local has no password??
if (Cipher.isLikelyEncryptedName(santyCheckKey)) {
return {
ok: false,
reason: "remote_encrypted_local_no_password",
};
} else {
return {
ok: true,
reason: "likely_no_password_both_sides",
};
}
} else {
if (cipher.method === "unknown") {
return {
ok: false,
reason: "unknown_encryption_method",
};
}
if (
Cipher.isLikelyEncryptedNameNotMatchMethod(santyCheckKey, cipher.method)
) {
return {
ok: false,
reason: "encryption_method_not_matched",
};
}
try {
const k = await cipher.decryptName(santyCheckKey);
if (k === undefined) {
throw Error(`decryption failed`);
}
return {
ok: true,
reason: "password_matched",
};
} catch (error) {
return {
ok: false,
reason: "password_or_method_not_matched_or_remote_not_encrypted",
};
}
}
};
const isSkipItemByName = (
key: string,
syncConfigDir: boolean,
syncUnderscoreItems: boolean,
configDir: string,
ignorePaths: string[]
) => {
if (key === undefined) {
throw Error(`isSkipItemByName meets undefinded key!`);
}
if (ignorePaths !== undefined && ignorePaths.length > 0) {
for (const r of ignorePaths) {
if (XRegExp(r, "A").test(key)) {
return true;
}
}
}
if (syncConfigDir && isInsideObsFolder(key, configDir)) {
return false;
}
if (isSpecialFolderNameToSkip(key, [])) {
// some special dirs and files are always skipped
return true;
}
return (
isHiddenPath(key, true, false) ||
(!syncUnderscoreItems && isHiddenPath(key, false, true)) ||
key === DEFAULT_FILE_NAME_FOR_METADATAONREMOTE ||
key === DEFAULT_FILE_NAME_FOR_METADATAONREMOTE2
);
};
import PQueue from "p-queue";
const copyEntityAndFixTimeFormat = (
src: Entity,
@@ -194,47 +75,6 @@ const copyEntityAndFixTimeFormat = (
return result;
};
/**
* Inplace, no copy again.
*/
const decryptRemoteEntityInplace = async (remote: Entity, cipher: Cipher) => {
if (cipher?.isPasswordEmpty()) {
remote.key = remote.keyRaw;
remote.keyEnc = remote.keyRaw;
remote.size = remote.sizeRaw;
remote.sizeEnc = remote.sizeRaw;
return remote;
}
remote.keyEnc = remote.keyRaw;
remote.key = await cipher.decryptName(remote.keyEnc);
remote.sizeEnc = remote.sizeRaw;
// TODO
// remote.size = getSizeFromEncToOrig(remote.sizeEnc, password);
// but we don't have deterministic way to get a number because the encryption has padding...
return remote;
};
const fullfillMTimeOfRemoteEntityInplace = (
remote: Entity,
mtimeCli?: number
) => {
if (
mtimeCli !== undefined &&
mtimeCli > 0 &&
(remote.mtimeCli === undefined ||
remote.mtimeCli <= 0 ||
(remote.mtimeSvr !== undefined &&
remote.mtimeSvr > 0 &&
remote.mtimeCli >= remote.mtimeSvr))
) {
remote.mtimeCli = mtimeCli;
}
return remote;
};
/**
* Directly throw error here.
* We can only defer the checking now, because before decryption we don't know whether it's a file or folder.
@@ -259,62 +99,49 @@ const ensureMTimeOfRemoteEntityValid = (remote: Entity) => {
return remote;
};
/**
* Inplace, no copy again.
*/
const encryptLocalEntityInplace = async (
local: Entity,
cipher: Cipher,
remoteKeyEnc: string | undefined
const isInsideObsFolder = (x: string, configDir: string) => {
if (!configDir.startsWith(".")) {
throw Error(`configDir should starts with . but we get ${configDir}`);
}
return x === configDir || x.startsWith(`${configDir}/`);
};
const isSkipItemByName = (
key: string,
syncConfigDir: boolean,
syncUnderscoreItems: boolean,
configDir: string,
ignorePaths: string[]
) => {
// console.debug(
// `encryptLocalEntityInplace: local=${JSON.stringify(
// local,
// null,
// 2
// )}, password=${
// password === undefined || password === "" ? "[empty]" : "[not empty]"
// }, remoteKeyEnc=${remoteKeyEnc}`
// );
if (local.key === undefined) {
// local.key should always have value
throw Error(`local ${local.keyRaw} is abnormal without key`);
if (key === undefined) {
throw Error(`isSkipItemByName meets undefinded key!`);
}
if (cipher.isPasswordEmpty()) {
local.sizeEnc = local.sizeRaw; // if no enc, the remote file has the same size
local.keyEnc = local.keyRaw;
return local;
}
// below is for having password
if (local.sizeEnc === undefined && local.size !== undefined) {
// it's not filled yet, we fill it
// local.size is possibly undefined if it's "prevSync" Entity
// but local.key should always have value
local.sizeEnc = cipher.getSizeFromOrigToEnc(local.size);
}
if (local.keyEnc === undefined || local.keyEnc === "") {
if (
remoteKeyEnc !== undefined &&
remoteKeyEnc !== "" &&
remoteKeyEnc !== local.key
) {
// we can reuse remote encrypted key if any
local.keyEnc = remoteKeyEnc;
} else {
// we assign a new encrypted key because of no remote
local.keyEnc = await cipher.encryptName(local.key);
if (ignorePaths !== undefined && ignorePaths.length > 0) {
for (const r of ignorePaths) {
if (XRegExp(r, "A").test(key)) {
return true;
}
}
return local;
}
if (syncConfigDir && isInsideObsFolder(key, configDir)) {
return false;
}
if (isSpecialFolderNameToSkip(key, [])) {
// some special dirs and files are always skipped
return true;
}
return (
isHiddenPath(key, true, false) ||
(!syncUnderscoreItems && isHiddenPath(key, false, true)) ||
key === "/" ||
key === DEFAULT_FILE_NAME_FOR_METADATAONREMOTE ||
key === DEFAULT_FILE_NAME_FOR_METADATAONREMOTE2
);
};
export type SyncPlanType = Record<string, MixedEntity>;
export const ensembleMixedEnties = async (
const ensembleMixedEnties = async (
localEntityList: Entity[],
prevSyncEntityList: Entity[],
remoteEntityList: Entity[],
@@ -323,7 +150,7 @@ export const ensembleMixedEnties = async (
configDir: string,
syncUnderscoreItems: boolean,
ignorePaths: string[],
cipher: Cipher,
fsEncrypt: FakeFsEncrypt,
serviceType: SUPPORTED_SERVICES_TYPE,
profiler: Profiler
@@ -333,16 +160,10 @@ export const ensembleMixedEnties = async (
const finalMappings: SyncPlanType = {};
const synthFolders: Record<string, Entity> = {};
// remote has to be first
// we also have to synthesize folders here
for (const remote of remoteEntityList) {
const remoteCopied = ensureMTimeOfRemoteEntityValid(
await decryptRemoteEntityInplace(
copyEntityAndFixTimeFormat(remote, serviceType),
cipher
)
copyEntityAndFixTimeFormat(remote, serviceType)
);
const key = remoteCopied.key!;
@@ -362,48 +183,10 @@ export const ensembleMixedEnties = async (
key: key,
remote: remoteCopied,
};
for (const f of getFolderLevels(key, true)) {
if (finalMappings.hasOwnProperty(f)) {
delete synthFolders[f];
continue;
}
if (
!synthFolders.hasOwnProperty(f) ||
remoteCopied.mtimeSvr! >= synthFolders[f].mtimeSvr!
) {
synthFolders[f] = {
key: f,
keyRaw: `<synth: ${f}>`,
keyEnc: `<enc synth: ${f}>`,
size: 0,
sizeRaw: 0,
sizeEnc: 0,
mtimeSvr: remoteCopied.mtimeSvr,
mtimeSvrFmt: remoteCopied.mtimeSvrFmt,
mtimeCli: remoteCopied.mtimeCli,
mtimeCliFmt: remoteCopied.mtimeCliFmt,
synthesizedFolder: true,
};
}
}
}
profiler.insert("ensembleMixedEnties: finish remote");
console.debug(`synthFolders:`);
console.debug(synthFolders);
// special: add synth folders
for (const key of Object.keys(synthFolders)) {
finalMappings[key] = {
key: key,
remote: synthFolders[key],
};
}
profiler.insert("ensembleMixedEnties: finish synth");
if (Object.keys(finalMappings).length === 0 || localEntityList.length === 0) {
// Special checking:
// if one side is totally empty,
@@ -427,19 +210,13 @@ export const ensembleMixedEnties = async (
continue;
}
if (finalMappings.hasOwnProperty(key)) {
const prevSyncCopied = await encryptLocalEntityInplace(
copyEntityAndFixTimeFormat(prevSync, serviceType),
cipher,
finalMappings[key].remote?.keyEnc
// TODO: abstraction leaking?
const prevSyncCopied = await fsEncrypt.encryptEntity(
copyEntityAndFixTimeFormat(prevSync, serviceType)
);
if (finalMappings.hasOwnProperty(key)) {
finalMappings[key].prevSync = prevSyncCopied;
} else {
const prevSyncCopied = await encryptLocalEntityInplace(
copyEntityAndFixTimeFormat(prevSync, serviceType),
cipher,
undefined
);
finalMappings[key] = {
key: key,
prevSync: prevSyncCopied,
@@ -467,19 +244,13 @@ export const ensembleMixedEnties = async (
continue;
}
if (finalMappings.hasOwnProperty(key)) {
const localCopied = await encryptLocalEntityInplace(
copyEntityAndFixTimeFormat(local, serviceType),
cipher,
finalMappings[key].remote?.keyEnc
// TODO: abstraction leaking?
const localCopied = await fsEncrypt.encryptEntity(
copyEntityAndFixTimeFormat(local, serviceType)
);
if (finalMappings.hasOwnProperty(key)) {
finalMappings[key].local = localCopied;
} else {
const localCopied = await encryptLocalEntityInplace(
copyEntityAndFixTimeFormat(local, serviceType),
cipher,
undefined
);
finalMappings[key] = {
key: key,
local: localCopied,
@@ -489,8 +260,8 @@ export const ensembleMixedEnties = async (
profiler.insert("ensembleMixedEnties: finish local");
console.debug("in the end of ensembleMixedEnties, finalMappings is:");
console.debug(finalMappings);
// console.debug("in the end of ensembleMixedEnties, finalMappings is:");
// console.debug(finalMappings);
profiler.insert("ensembleMixedEnties: exit");
profiler.removeIndent();
@@ -502,7 +273,7 @@ export const ensembleMixedEnties = async (
* Basically follow the sync algorithm of https://github.com/Jwink3101/syncrclone
* Also deal with syncDirection which makes it more complicated
*/
export const getSyncPlanInplace = async (
const getSyncPlanInplace = async (
mixedEntityMappings: Record<string, MixedEntity>,
howToCleanEmptyFolder: EmptyFolderCleanType,
skipSizeLargerThan: number,
@@ -934,6 +705,7 @@ export const getSyncPlanInplace = async (
mixedEntityMappings["/$@meta"] = {
key: "/$@meta", // don't mess up with the types
sideNotes: {
version: "2024047 fs version",
generateTime: currTime,
generateTimeFmt: currTimeFmt,
},
@@ -1087,16 +859,96 @@ const splitFourStepsOnEntityMappings = (
};
};
const fullfillMTimeOfRemoteEntityInplace = (
remote: Entity,
mtimeCli?: number
) => {
// TODO:
// on 20240405, we find that dropbox's mtimeCli is not updated
// if the content is not updated even the time is updated...
// so we do not check remote.mtimeCli for now..
if (
mtimeCli !== undefined &&
mtimeCli > 0 /* &&
(remote.mtimeCli === undefined ||
remote.mtimeCli <= 0 ||
(remote.mtimeSvr !== undefined &&
remote.mtimeSvr > 0 &&
remote.mtimeCli >= remote.mtimeSvr))
*/
) {
remote.mtimeCli = mtimeCli;
}
return remote;
};
async function copyFolder(
key: string,
left: FakeFs,
right: FakeFs
): Promise<Entity> {
if (!key.endsWith("/")) {
throw Error(`should not call ${key} in copyFolder`);
}
const statsLeft = await left.stat(key);
return await right.mkdir(key, statsLeft.mtimeCli);
}
async function copyFile(
key: string,
left: FakeFs,
right: FakeFs
): Promise<Entity> {
// console.debug(`copyFile: key=${key}, left=${left.kind}, right=${right.kind}`);
if (key.endsWith("/")) {
throw Error(`should not call ${key} in copyFile`);
}
const statsLeft = await left.stat(key);
const content = await left.readFile(key);
if (statsLeft.size === undefined) {
statsLeft.size = content.byteLength;
} else {
if (statsLeft.size !== content.byteLength) {
throw Error(
`error copying ${left.kind}=>${right.kind}: size not matched`
);
}
}
if (statsLeft.mtimeCli === undefined) {
throw Error(`error copying ${left.kind}=>${right.kind}, no mtimeCli`);
}
// console.debug(`copyFile: about to start right.writeFile`);
return await right.writeFile(
key,
content,
statsLeft.mtimeCli,
statsLeft.mtimeCli /* TODO */
);
}
async function copyFileOrFolder(
key: string,
left: FakeFs,
right: FakeFs
): Promise<Entity> {
if (key.endsWith("/")) {
return await copyFolder(key, left, right);
} else {
return await copyFile(key, left, right);
}
}
const dispatchOperationToActualV3 = async (
key: string,
vaultRandomID: string,
profileID: string,
r: MixedEntity,
client: RemoteClient,
db: InternalDBs,
vault: Vault,
localDeleteFunc: any,
cipher: Cipher
fsLocal: FakeFs,
fsEncrypt: FakeFsEncrypt,
db: InternalDBs
) => {
// console.debug(
// `inside dispatchOperationToActualV3, key=${key}, r=${JSON.stringify(
@@ -1120,19 +972,19 @@ const dispatchOperationToActualV3 = async (
r.decision === "conflict_created_then_do_nothing" ||
r.decision === "folder_existed_both_then_do_nothing"
) {
// !! we need to upsert the record,
// !! we MIGHT need to upsert the record,
// so that next time we can determine the change delta
// if we have prevSync, we store it because it should keep all necessary info
let entity = r.prevSync;
if (r.prevSync !== undefined) {
// if we have prevSync,
// we don't need to do anything, because the record is already there!
} else {
// if we don't have prevSync, we use remote entity AND local mtime
// as if it is "uploaded"
if (entity === undefined && r.remote !== undefined) {
entity = await decryptRemoteEntityInplace(r.remote, cipher);
entity = await fullfillMTimeOfRemoteEntityInplace(
entity,
r.local?.mtimeCli
);
}
if (r.remote !== undefined) {
let entity = r.remote;
entity = fullfillMTimeOfRemoteEntityInplace(entity, r.local?.mtimeCli);
if (entity !== undefined) {
await upsertPrevSyncRecordByVaultAndProfile(
db,
@@ -1141,6 +993,8 @@ const dispatchOperationToActualV3 = async (
entity
);
}
}
}
} else if (
r.decision === "local_is_modified_then_push" ||
r.decision === "local_is_created_then_push" ||
@@ -1148,33 +1002,17 @@ const dispatchOperationToActualV3 = async (
r.decision === "conflict_created_then_keep_local" ||
r.decision === "conflict_modified_then_keep_local"
) {
if (
client.serviceType === "onedrive" &&
r.local!.size === 0 &&
cipher.isPasswordEmpty()
) {
// special treatment for empty files for OneDrive
// TODO: it's ugly, any other way?
// special treatment for OneDrive: do nothing, skip empty file without encryption
// if it's empty folder, or it's encrypted file/folder, it continues to be uploaded.
} else {
// console.debug(`before upload in sync, r=${JSON.stringify(r, null, 2)}`);
const { entity, mtimeCli } = await client.uploadToRemote(
r.key,
vault,
false,
cipher,
r.local!.keyEnc
);
await decryptRemoteEntityInplace(entity, cipher);
await fullfillMTimeOfRemoteEntityInplace(entity, mtimeCli);
const mtimeCli = (await fsLocal.stat(r.key)).mtimeCli!;
const entity = await copyFileOrFolder(r.key, fsLocal, fsEncrypt);
fullfillMTimeOfRemoteEntityInplace(entity, mtimeCli);
// console.debug(`after fullfill, entity=${JSON.stringify(entity,null,2)}`)
await upsertPrevSyncRecordByVaultAndProfile(
db,
vaultRandomID,
profileID,
entity
);
}
} else if (
r.decision === "remote_is_modified_then_pull" ||
r.decision === "remote_is_created_then_pull" ||
@@ -1182,14 +1020,11 @@ const dispatchOperationToActualV3 = async (
r.decision === "conflict_modified_then_keep_remote" ||
r.decision === "folder_existed_remote_then_also_create_local"
) {
await mkdirpInVault(r.key, vault);
await client.downloadFromRemote(
r.key,
vault,
r.remote!.mtimeCli!,
cipher,
r.remote!.keyEnc
);
if (r.key.endsWith("/")) {
await fsLocal.mkdir(r.key);
} else {
await copyFile(r.key, fsEncrypt, fsLocal);
}
await upsertPrevSyncRecordByVaultAndProfile(
db,
vaultRandomID,
@@ -1198,12 +1033,7 @@ const dispatchOperationToActualV3 = async (
);
} else if (r.decision === "local_is_deleted_thus_also_delete_remote") {
// local is deleted, we need to delete remote now
await client.deleteFromRemote(
r.key,
cipher,
r.remote!.keyEnc,
r.remote!.synthesizedFolder
);
await fsEncrypt.rm(r.key);
await clearPrevSyncRecordByVaultAndProfile(
db,
vaultRandomID,
@@ -1212,7 +1042,7 @@ const dispatchOperationToActualV3 = async (
);
} else if (r.decision === "remote_is_deleted_thus_also_delete_local") {
// remote is deleted, we need to delete local now
await localDeleteFunc(r.key);
await fsLocal.rm(r.key);
await clearPrevSyncRecordByVaultAndProfile(
db,
vaultRandomID,
@@ -1225,17 +1055,8 @@ const dispatchOperationToActualV3 = async (
) {
throw Error(`${r.decision} not implemented yet: ${JSON.stringify(r)}`);
} else if (r.decision === "folder_to_be_created") {
await mkdirpInVault(r.key, vault);
const { entity, mtimeCli } = await client.uploadToRemote(
r.key,
vault,
false,
cipher,
r.local!.keyEnc
);
// we need to decrypt the key!!!
await decryptRemoteEntityInplace(entity, cipher);
await fullfillMTimeOfRemoteEntityInplace(entity, mtimeCli);
await fsLocal.mkdir(r.key);
const entity = await copyFolder(r.key, fsLocal, fsEncrypt);
await upsertPrevSyncRecordByVaultAndProfile(
db,
vaultRandomID,
@@ -1251,18 +1072,13 @@ const dispatchOperationToActualV3 = async (
r.decision === "folder_to_be_deleted_on_both" ||
r.decision === "folder_to_be_deleted_on_local"
) {
await localDeleteFunc(r.key);
await fsLocal.rm(r.key);
}
if (
r.decision === "folder_to_be_deleted_on_both" ||
r.decision === "folder_to_be_deleted_on_remote"
) {
await client.deleteFromRemote(
r.key,
cipher,
r.remote!.keyEnc,
r.remote!.synthesizedFolder
);
await fsEncrypt.rm(r.key);
}
await clearPrevSyncRecordByVaultAndProfile(
db,
@@ -1277,18 +1093,16 @@ const dispatchOperationToActualV3 = async (
export const doActualSync = async (
mixedEntityMappings: Record<string, MixedEntity>,
client: RemoteClient,
fsLocal: FakeFs,
fsEncrypt: FakeFsEncrypt,
vaultRandomID: string,
profileID: string,
vault: Vault,
cipher: Cipher,
concurrency: number,
localDeleteFunc: any,
protectModifyPercentage: number,
getProtectModifyPercentageErrorStrFunc: any,
callbackSyncProcess: any,
db: InternalDBs,
profiler: Profiler
profiler: Profiler,
callbackSyncProcess?: any
) => {
profiler.addIndent();
profiler.insert("doActualSync: enter");
@@ -1364,9 +1178,9 @@ export const doActualSync = async (
for (let j = 0; j < operations.length; ++j) {
const singleLevelOps = operations[j];
console.debug(
`singleLevelOps=${JSON.stringify(singleLevelOps, null, 2)}`
);
// console.debug(
// `singleLevelOps=${JSON.stringify(singleLevelOps, null, 2)}`
// );
if (singleLevelOps === undefined || singleLevelOps === null) {
continue;
}
@@ -1380,12 +1194,11 @@ export const doActualSync = async (
const key = val.key;
const fn = async () => {
console.debug(
`start syncing "${key}" with plan ${JSON.stringify(val)}`
);
// console.debug(
// `start syncing "${key}" with plan ${JSON.stringify(val)}`
// );
if (callbackSyncProcess !== undefined) {
await callbackSyncProcess(
await callbackSyncProcess?.(
realCounter,
realTotalCount,
key,
@@ -1393,21 +1206,18 @@ export const doActualSync = async (
);
realCounter += 1;
}
await dispatchOperationToActualV3(
key,
vaultRandomID,
profileID,
val,
client,
db,
vault,
localDeleteFunc,
cipher
fsLocal,
fsEncrypt,
db
);
console.debug(`finished ${key}`);
// console.debug(`finished ${key}`);
};
queue.add(fn).catch((e) => {
@@ -1440,3 +1250,186 @@ export const doActualSync = async (
profiler.insert(`doActualSync: exit`);
profiler.removeIndent();
};
export type SyncStatusType =
| "idle"
| "preparing"
| "getting_remote_files_list"
| "getting_local_meta"
| "getting_local_prev_sync"
| "checking_password"
| "generating_plan"
| "syncing"
| "cleaning"
| "finish";
/**
* Every input variable should be mockable, so that testable.
*/
export async function syncer(
fsLocal: FakeFs,
fsRemote: FakeFs,
fsEncrypt: FakeFsEncrypt,
profiler: Profiler,
db: InternalDBs,
triggerSource: SyncTriggerSourceType,
profileID: string,
vaultRandomID: string,
configDir: string,
settings: RemotelySavePluginSettings,
getProtectModifyPercentageErrorStrFunc: any,
markIsSyncingFunc: (isSyncing: boolean) => void,
notifyFunc?: (s: SyncTriggerSourceType, step: number) => Promise<any>,
errNotifyFunc?: (s: SyncTriggerSourceType, error: Error) => Promise<any>,
ribboonFunc?: (s: SyncTriggerSourceType, step: number) => Promise<any>,
statusBarFunc?: (s: SyncTriggerSourceType, step: number) => any,
callbackSyncProcess?: any
) {
console.info(`startting sync.`);
markIsSyncingFunc(true);
let step = 0; // dry mode only
await notifyFunc?.(triggerSource, step);
step = 1;
await notifyFunc?.(triggerSource, step);
await ribboonFunc?.(triggerSource, step);
await statusBarFunc?.(triggerSource, step);
profiler.insert("start big sync func");
try {
step = 2;
await notifyFunc?.(triggerSource, step);
await ribboonFunc?.(triggerSource, step);
await statusBarFunc?.(triggerSource, step);
if (fsEncrypt.innerFs !== fsRemote) {
throw Error(`your enc should has inner of the remote`);
}
const passwordCheckResult = await fsEncrypt.isPasswordOk();
if (!passwordCheckResult.ok) {
throw Error(passwordCheckResult.reason);
}
profiler.insert(
`finish step${step} (list partial remote and check password)`
);
step = 3;
await notifyFunc?.(triggerSource, step);
await ribboonFunc?.(triggerSource, step);
await statusBarFunc?.(triggerSource, step);
const remoteEntityList = await fsEncrypt.walk();
// console.debug(`remoteEntityList:`);
// console.debug(remoteEntityList);
profiler.insert(`finish step${step} (list remote)`);
step = 4;
await notifyFunc?.(triggerSource, step);
await ribboonFunc?.(triggerSource, step);
await statusBarFunc?.(triggerSource, step);
const localEntityList = await fsLocal.walk();
// console.debug(`localEntityList:`);
// console.debug(localEntityList);
profiler.insert(`finish step${step} (list local)`);
step = 5;
await notifyFunc?.(triggerSource, step);
await ribboonFunc?.(triggerSource, step);
await statusBarFunc?.(triggerSource, step);
const prevSyncEntityList = await getAllPrevSyncRecordsByVaultAndProfile(
db,
vaultRandomID,
profileID
);
// console.debug(`prevSyncEntityList:`);
// console.debug(prevSyncEntityList);
profiler.insert(`finish step${step} (prev sync)`);
step = 6;
await notifyFunc?.(triggerSource, step);
await ribboonFunc?.(triggerSource, step);
await statusBarFunc?.(triggerSource, step);
let mixedEntityMappings = await ensembleMixedEnties(
localEntityList,
prevSyncEntityList,
remoteEntityList,
settings.syncConfigDir ?? false,
configDir,
settings.syncUnderscoreItems ?? false,
settings.ignorePaths ?? [],
fsEncrypt,
settings.serviceType,
profiler
);
profiler.insert(`finish step${step} (build partial mixedEntity)`);
mixedEntityMappings = await getSyncPlanInplace(
mixedEntityMappings,
settings.howToCleanEmptyFolder ?? "clean_both",
settings.skipSizeLargerThan ?? -1,
settings.conflictAction ?? "keep_newer",
settings.syncDirection ?? "bidirectional",
profiler
);
console.info(`mixedEntityMappings:`);
console.info(mixedEntityMappings); // for debugging
profiler.insert("finish building full sync plan");
await insertSyncPlanRecordByVault(
db,
mixedEntityMappings,
vaultRandomID,
settings.serviceType
);
profiler.insert("finish writing sync plan");
profiler.insert(`finish step${step} (make plan)`);
// The operations above are almost read only and kind of safe.
// The operations below begins to write or delete (!!!) something.
step = 7;
if (triggerSource !== "dry") {
await notifyFunc?.(triggerSource, step);
await ribboonFunc?.(triggerSource, step);
await statusBarFunc?.(triggerSource, step);
await doActualSync(
mixedEntityMappings,
fsLocal,
fsEncrypt,
vaultRandomID,
profileID,
settings.concurrency ?? 5,
settings.protectModifyPercentage ?? 50,
getProtectModifyPercentageErrorStrFunc,
db,
profiler,
callbackSyncProcess
);
profiler.insert(`finish step${step} (actual sync)`);
} else {
await notifyFunc?.(triggerSource, step);
await ribboonFunc?.(triggerSource, step);
await statusBarFunc?.(triggerSource, step);
profiler.insert(
`finish step${step} (skip actual sync because of dry run)`
);
}
} catch (error: any) {
profiler.insert("start error branch");
await errNotifyFunc?.(triggerSource, error as Error);
profiler.insert("finish error branch");
} finally {
}
profiler.insert("finish syncRun");
// console.debug(profiler.toString());
await profiler.save(db, vaultRandomID, settings.serviceType);
step = 8;
await notifyFunc?.(triggerSource, step);
await ribboonFunc?.(triggerSource, step);
await statusBarFunc?.(triggerSource, step);
console.info(`endding sync.`);
markIsSyncingFunc(false);
}
+1 -1
View File
@@ -119,7 +119,7 @@ export class SyncAlgoV3Modal extends Modal {
this.plugin.saveAgreeToUseNewSyncAlgorithm();
this.plugin.enableAutoSyncIfSet();
this.plugin.enableInitSyncIfSet();
this.plugin.enableSyncOnSaveIfSet();
this.plugin.toggleSyncOnSaveIfSet();
} else {
console.info("do not agree to use the new algorithm");
this.plugin.unload();
+11
View File
@@ -61,6 +61,13 @@
display: none;
}
.webdis-disclaimer {
font-weight: bold;
}
.webdis-hide {
display: none;
}
.qrcode-img {
width: 350px;
height: 350px;
@@ -74,3 +81,7 @@
color: red;
font-weight: bolder;
}
.setting-need-wrapping-mobile .setting-item-control {
flex-wrap: wrap;
}
+5 -6
View File
@@ -1,12 +1,8 @@
import * as chai from "chai";
import chaiAsPromised from "chai-as-promised";
import { strict as assert } from "assert";
import { RemotelySavePluginSettings } from "../src/baseTypes";
import { messyConfigToNormal, normalConfigToMessy } from "../src/configPersist";
chai.use(chaiAsPromised);
const expect = chai.expect;
const DEFAULT_SETTINGS: RemotelySavePluginSettings = {
s3: {
s3AccessKeyID: "acc",
@@ -20,6 +16,9 @@ const DEFAULT_SETTINGS: RemotelySavePluginSettings = {
onedrive: {
username: "test 🍎 emoji",
} as any,
webdis: {
address: "addr",
} as any,
password: "password",
serviceType: "s3",
currLogLevel: "info",
@@ -32,6 +31,6 @@ describe("Config Persist tests", () => {
const k = DEFAULT_SETTINGS;
const k2 = normalConfigToMessy(k);
const k3 = messyConfigToNormal(k2);
expect(k3).to.deep.equal(k);
assert.deepEqual(k3, k);
});
});
+23 -27
View File
@@ -1,5 +1,4 @@
import * as chai from "chai";
import chaiAsPromised from "chai-as-promised";
import { strict as assert } from "assert";
import * as fs from "fs";
import * as path from "path";
import {
@@ -13,9 +12,6 @@ import {
} from "../src/encryptOpenSSL";
import { base64ToBase64url, bufferToArrayBuffer } from "../src/misc";
chai.use(chaiAsPromised);
const expect = chai.expect;
describe("Encryption OpenSSL tests", () => {
beforeEach(function () {
global.window = {
@@ -26,7 +22,7 @@ describe("Encryption OpenSSL tests", () => {
it("should encrypt string", async () => {
const k = "dkjdhkfhdkjgsdklxxd";
const password = "hey";
expect(await encryptStringToBase32(k, password)).to.not.equal(k);
assert.notEqual(await encryptStringToBase32(k, password), k);
});
it("should encrypt string and return different results each time", async () => {
@@ -34,7 +30,7 @@ describe("Encryption OpenSSL tests", () => {
const password = "hey";
const res1 = await encryptStringToBase32(k, password);
const res2 = await encryptStringToBase32(k, password);
expect(res1).to.not.equal(res2);
assert.notEqual(res1, res2);
});
it("should raise error using different password", async () => {
@@ -42,7 +38,7 @@ describe("Encryption OpenSSL tests", () => {
const password = "hey";
const password2 = "hey2";
const enc = await encryptStringToBase32(k, password);
await expect(decryptBase32ToString(enc, password2)).to.be.rejected;
await assert.rejects(decryptBase32ToString(enc, password2));
});
it("should encrypt and decrypt string and get the same result returned", async () => {
@@ -52,7 +48,7 @@ describe("Encryption OpenSSL tests", () => {
// console.log(enc);
const dec = await decryptBase32ToString(enc, password);
// console.log(dec);
expect(dec).equal(k);
assert.equal(dec, k);
});
it("should encrypt text file and get the same result as openssl", async () => {
@@ -78,7 +74,7 @@ describe("Encryption OpenSSL tests", () => {
// we output base32, so we need some transformation
const opensslBase64urlRes = base64ToBase64url(opensslBase64Res);
expect(enc).equal(opensslBase64urlRes);
assert.equal(enc, opensslBase64urlRes);
});
it("should encrypt binary file and get the same result as openssl", async () => {
@@ -102,7 +98,7 @@ describe("Encryption OpenSSL tests", () => {
// openssl enc -p -aes-256-cbc -S 8302F586FAB491EC -pbkdf2 -iter 20000 -pass pass:somepassword -in mona_lisa/1374px-Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched.jpg -out mona_lisa/1374px-Mona_Lisa,_by_Leonardo_da_Vinci,_from_C2RMF_retouched.jpg.enc
expect(Buffer.from(enc).equals(Buffer.from(opensslArrBuf))).to.be.true;
assert.ok(Buffer.from(enc).equals(Buffer.from(opensslArrBuf)));
});
it("should encrypt binary file not deterministically", async () => {
@@ -116,7 +112,7 @@ describe("Encryption OpenSSL tests", () => {
const res1 = await encryptArrayBuffer(fileArrBuf, password);
const res2 = await encryptArrayBuffer(fileArrBuf, password);
expect(Buffer.from(res1).equals(Buffer.from(res2))).to.be.false;
assert.ok(!Buffer.from(res1).equals(Buffer.from(res2)));
});
it("should decrypt binary file and get the same result as openssl", async () => {
@@ -132,36 +128,36 @@ describe("Encryption OpenSSL tests", () => {
await fs.readFileSync(path.join(testFolder, testFileName))
);
expect(Buffer.from(dec).equals(Buffer.from(opensslArrBuf))).to.be.true;
assert.deepEqual(Buffer.from(dec), Buffer.from(opensslArrBuf));
});
it("should get size from origin to encrypted correctly", () => {
expect(() => getSizeFromOrigToEnc(-1)).to.throw();
expect(() => getSizeFromOrigToEnc(0.5)).to.throw();
expect(getSizeFromOrigToEnc(0)).equals(32);
expect(getSizeFromOrigToEnc(15)).equals(32);
expect(getSizeFromOrigToEnc(16)).equals(48);
expect(getSizeFromOrigToEnc(31)).equals(48);
expect(getSizeFromOrigToEnc(32)).equals(64);
expect(getSizeFromOrigToEnc(14787203)).equals(14787232);
assert.throws(() => getSizeFromOrigToEnc(-1));
assert.throws(() => getSizeFromOrigToEnc(0.5));
assert.equal(getSizeFromOrigToEnc(0), 32);
assert.equal(getSizeFromOrigToEnc(15), 32);
assert.equal(getSizeFromOrigToEnc(16), 48);
assert.equal(getSizeFromOrigToEnc(31), 48);
assert.equal(getSizeFromOrigToEnc(32), 64);
assert.equal(getSizeFromOrigToEnc(14787203), 14787232);
});
it("should get size from encrypted to origin correctly", () => {
expect(() => getSizeFromEncToOrig(-1)).to.throw();
expect(() => getSizeFromEncToOrig(30)).to.throw();
assert.throws(() => getSizeFromEncToOrig(-1));
assert.throws(() => getSizeFromEncToOrig(30));
expect(getSizeFromEncToOrig(32)).to.deep.equal({
assert.deepEqual(getSizeFromEncToOrig(32), {
minSize: 0,
maxSize: 15,
});
expect(getSizeFromEncToOrig(48)).to.deep.equal({
assert.deepEqual(getSizeFromEncToOrig(48), {
minSize: 16,
maxSize: 31,
});
expect(() => getSizeFromEncToOrig(14787231)).to.throw();
assert.throws(() => getSizeFromEncToOrig(14787231));
let { minSize, maxSize } = getSizeFromEncToOrig(14787232);
expect(minSize <= 14787203 && 14787203 <= maxSize).to.be.true;
assert.ok(minSize <= 14787203 && 14787203 <= maxSize);
});
});
+12
View File
@@ -0,0 +1,12 @@
import { strict as assert } from "assert";
import { getOrigPath } from "../src/fsWebdis";
describe("Webdis operations tests", () => {
it("should get orig keys correctly", () => {
const input = "rs:fs:v1:库名字/something dev.md:meta";
const output = getOrigPath(input, "库名字");
const expected = "something dev.md";
assert.equal(output, expected);
});
});
+9 -14
View File
@@ -1,14 +1,9 @@
import * as chai from "chai";
import chaiAsPromised from "chai-as-promised";
import { strict as assert } from "assert";
import {
isEqualMetadataOnRemote,
MetadataOnRemote,
} from "../src/metadataOnRemote";
chai.use(chaiAsPromised);
const expect = chai.expect;
describe("Metadata operations tests", () => {
it("should compare objects deeply", async () => {
const a: MetadataOnRemote = {
@@ -24,7 +19,7 @@ describe("Metadata operations tests", () => {
],
};
expect(isEqualMetadataOnRemote(a, b));
assert.ok(isEqualMetadataOnRemote(a, b));
});
it("should find diff", async () => {
@@ -41,7 +36,7 @@ describe("Metadata operations tests", () => {
],
};
expect(!isEqualMetadataOnRemote(a, b));
assert.ok(!isEqualMetadataOnRemote(a, b));
});
it("should treat undefined correctly", async () => {
@@ -53,22 +48,22 @@ describe("Metadata operations tests", () => {
],
};
expect(!isEqualMetadataOnRemote(a, b));
assert.ok(!isEqualMetadataOnRemote(a, b));
b = { deletions: [] };
expect(isEqualMetadataOnRemote(a, b));
assert.ok(isEqualMetadataOnRemote(a, b));
b = { deletions: undefined };
expect(isEqualMetadataOnRemote(a, b));
assert.ok(isEqualMetadataOnRemote(a, b));
b = undefined;
expect(isEqualMetadataOnRemote(a, b));
assert.ok(isEqualMetadataOnRemote(a, b));
});
it("should ignore generated at fields", async () => {
const a: MetadataOnRemote = {
deletions: [
{ key: "xxxx", actionWhen: 1 },
{ key: "xxx", actionWhen: 1 },
{ key: "yyy", actionWhen: 2 },
],
generatedWhen: 1,
@@ -81,6 +76,6 @@ describe("Metadata operations tests", () => {
generatedWhen: 2,
};
expect(isEqualMetadataOnRemote(a, b));
assert.ok(isEqualMetadataOnRemote(a, b));
});
});
+72 -72
View File
@@ -1,139 +1,139 @@
import { expect } from "chai";
import { strict as assert } from "assert";
import { JSDOM } from "jsdom";
import * as misc from "../src/misc";
describe("Misc: hidden file", () => {
it("should find hidden file correctly", () => {
let item = "";
expect(misc.isHiddenPath(item)).to.be.false;
assert.ok(!misc.isHiddenPath(item));
item = ".";
expect(misc.isHiddenPath(item)).to.be.false;
assert.ok(!misc.isHiddenPath(item));
item = "..";
expect(misc.isHiddenPath(item)).to.be.false;
assert.ok(!misc.isHiddenPath(item));
item = "/x/y/z/../././../a/b/c";
expect(misc.isHiddenPath(item)).to.be.false;
assert.ok(!misc.isHiddenPath(item));
item = ".hidden";
expect(misc.isHiddenPath(item)).to.be.true;
assert.ok(misc.isHiddenPath(item));
item = "_hidden_loose";
expect(misc.isHiddenPath(item)).to.be.true;
expect(misc.isHiddenPath(item, true, false)).to.be.false;
assert.ok(misc.isHiddenPath(item));
assert.ok(!misc.isHiddenPath(item, true, false));
item = "/sdd/_hidden_loose";
expect(misc.isHiddenPath(item)).to.be.true;
assert.ok(misc.isHiddenPath(item));
item = "what/../_hidden_loose/what/what/what";
expect(misc.isHiddenPath(item)).to.be.true;
assert.ok(misc.isHiddenPath(item));
item = "what/../_hidden_loose/what/what/what";
expect(misc.isHiddenPath(item, true, false)).to.be.false;
assert.ok(!misc.isHiddenPath(item, true, false));
item = "what/../_hidden_loose/../.hidden/what/what/what";
expect(misc.isHiddenPath(item, true, false)).to.be.true;
assert.ok(misc.isHiddenPath(item, true, false));
item = "what/../_hidden_loose/../.hidden/what/what/what";
expect(misc.isHiddenPath(item, false, true)).to.be.false;
assert.ok(!misc.isHiddenPath(item, false, true));
item = "what/_hidden_loose/what/what/what";
expect(misc.isHiddenPath(item, false, true)).to.be.true;
expect(misc.isHiddenPath(item, true, false)).to.be.false;
assert.ok(misc.isHiddenPath(item, false, true));
assert.ok(!misc.isHiddenPath(item, true, false));
item = "what/.hidden/what/what/what";
expect(misc.isHiddenPath(item, false, true)).to.be.false;
expect(misc.isHiddenPath(item, true, false)).to.be.true;
assert.ok(!misc.isHiddenPath(item, false, true));
assert.ok(misc.isHiddenPath(item, true, false));
});
});
describe("Misc: get folder levels", () => {
it("should ignore empty path", () => {
const item = "";
expect(misc.getFolderLevels(item)).to.be.empty;
assert.equal(misc.getFolderLevels(item).length, 0);
});
it("should ignore single file", () => {
const item = "xxx";
expect(misc.getFolderLevels(item)).to.be.empty;
assert.equal(misc.getFolderLevels(item).length, 0);
});
it("should detect path ending with /", () => {
const item = "xxx/";
const res = ["xxx"];
expect(misc.getFolderLevels(item)).to.deep.equal(res);
assert.deepEqual(misc.getFolderLevels(item), res);
});
it("should correctly split folders and files", () => {
const item = "xxx/yyy/zzz.md";
const res = ["xxx", "xxx/yyy"];
expect(misc.getFolderLevels(item)).to.deep.equal(res);
assert.deepEqual(misc.getFolderLevels(item), res);
const item2 = "xxx/yyy/zzz";
const res2 = ["xxx", "xxx/yyy"];
expect(misc.getFolderLevels(item2)).to.deep.equal(res2);
assert.deepEqual(misc.getFolderLevels(item2), res2);
const item3 = "xxx/yyy/zzz/";
const res3 = ["xxx", "xxx/yyy", "xxx/yyy/zzz"];
expect(misc.getFolderLevels(item3)).to.deep.equal(res3);
assert.deepEqual(misc.getFolderLevels(item3), res3);
});
it("should correctly add ending slash if required", () => {
const item = "xxx/yyy/zzz.md";
const res = ["xxx/", "xxx/yyy/"];
expect(misc.getFolderLevels(item, true)).to.deep.equal(res);
assert.deepEqual(misc.getFolderLevels(item, true), res);
const item2 = "xxx/yyy/zzz";
const res2 = ["xxx/", "xxx/yyy/"];
expect(misc.getFolderLevels(item2, true)).to.deep.equal(res2);
assert.deepEqual(misc.getFolderLevels(item2, true), res2);
const item3 = "xxx/yyy/zzz/";
const res3 = ["xxx/", "xxx/yyy/", "xxx/yyy/zzz/"];
expect(misc.getFolderLevels(item3, true)).to.deep.equal(res3);
assert.deepEqual(misc.getFolderLevels(item3, true), res3);
});
it("should treat path starting with / correctly", () => {
const item = "/xxx/yyy/zzz.md";
const res = ["/xxx", "/xxx/yyy"];
expect(misc.getFolderLevels(item)).to.deep.equal(res);
assert.deepEqual(misc.getFolderLevels(item), res);
const item2 = "/xxx/yyy/zzz";
const res2 = ["/xxx", "/xxx/yyy"];
expect(misc.getFolderLevels(item2)).to.deep.equal(res2);
assert.deepEqual(misc.getFolderLevels(item2), res2);
const item3 = "/xxx/yyy/zzz/";
const res3 = ["/xxx", "/xxx/yyy", "/xxx/yyy/zzz"];
expect(misc.getFolderLevels(item3)).to.deep.equal(res3);
assert.deepEqual(misc.getFolderLevels(item3), res3);
const item4 = "/xxx";
const res4 = [] as string[];
expect(misc.getFolderLevels(item4)).to.deep.equal(res4);
assert.deepEqual(misc.getFolderLevels(item4), res4);
const item5 = "/";
const res5 = [] as string[];
expect(misc.getFolderLevels(item5)).to.deep.equal(res5);
assert.deepEqual(misc.getFolderLevels(item5), res5);
});
});
describe("Misc: get parent folder", () => {
it("should treat empty path correctly", () => {
const item = "";
expect(misc.getParentFolder(item)).equals("/");
assert.equal(misc.getParentFolder(item), "/");
});
it("should treat one level path correctly", () => {
let item = "abc/";
expect(misc.getParentFolder(item)).equals("/");
assert.equal(misc.getParentFolder(item), "/");
item = "/efg/";
expect(misc.getParentFolder(item)).equals("/");
assert.equal(misc.getParentFolder(item), "/");
});
it("should treat more levels path correctly", () => {
let item = "abc/efg";
expect(misc.getParentFolder(item)).equals("abc/");
assert.equal(misc.getParentFolder(item), "abc/");
item = "/hij/klm/";
expect(misc.getParentFolder(item)).equals("/hij/");
assert.equal(misc.getParentFolder(item), "/hij/");
});
});
@@ -141,18 +141,18 @@ describe("Misc: vaild file name tests", () => {
it("should treat no ascii correctly", async () => {
const x = misc.isVaildText("😄🍎 apple 苹果");
// console.log(x)
expect(x).to.be.true;
assert.ok(x);
});
it("should find not-printable chars correctly", async () => {
const x = misc.isVaildText("😄🍎 apple 苹果\u0000");
// console.log(x)
expect(x).to.be.false;
assert.ok(!x);
});
it("should allow spaces/slashes/...", async () => {
const x = misc.isVaildText("😄🍎 apple 苹果/-_=/\\*%^&@#$`");
expect(x).to.be.true;
assert.ok(x);
});
});
@@ -160,21 +160,21 @@ describe("Misc: get dirname", () => {
it("should return itself for folder", async () => {
const x = misc.getPathFolder("ssss/");
// console.log(x)
expect(x).to.equal("ssss/");
assert.equal(x, "ssss/");
});
it("should return folder for file", async () => {
const x = misc.getPathFolder("sss/yyy");
// console.log(x)
expect(x).to.equal("sss/");
assert.equal(x, "sss/");
});
it("should treat / specially", async () => {
const x = misc.getPathFolder("/");
expect(x).to.equal("/");
assert.equal(x, "/");
const y = misc.getPathFolder("/abc");
expect(y).to.equal("/");
assert.equal(y, "/");
});
});
@@ -188,7 +188,7 @@ describe("Misc: extract svg", () => {
const x = "<svg><rect/><g/></svg>";
const y = misc.extractSvgSub(x);
// console.log(x)
expect(y).to.equal("<rect/><g/>");
assert.equal(y, "<rect/><g/>");
});
});
@@ -202,7 +202,7 @@ describe("Misc: get split ranges", () => {
end: 10,
},
];
expect(k).to.deep.equal(k2);
assert.deepEqual(k, k2);
});
it("should deal with 0 remainder", () => {
@@ -219,7 +219,7 @@ describe("Misc: get split ranges", () => {
end: 20,
},
];
expect(k).to.deep.equal(k2);
assert.deepEqual(k, k2);
});
it("should deal with not-0 remainder", () => {
@@ -241,55 +241,55 @@ describe("Misc: get split ranges", () => {
end: 25,
},
];
expect(k).to.deep.equal(k2);
assert.deepEqual(k, k2);
});
});
describe("Misc: at which level", () => {
it("should throw error on some parameters", () => {
expect(() => misc.atWhichLevel(undefined)).to.throw();
expect(() => misc.atWhichLevel("")).to.throw();
expect(() => misc.atWhichLevel("..")).to.throw();
expect(() => misc.atWhichLevel(".")).to.throw();
expect(() => misc.atWhichLevel("/")).to.throw();
expect(() => misc.atWhichLevel("/xxyy")).to.throw();
assert.throws(() => misc.atWhichLevel(undefined));
assert.throws(() => misc.atWhichLevel(""));
assert.throws(() => misc.atWhichLevel(".."));
assert.throws(() => misc.atWhichLevel("."));
assert.throws(() => misc.atWhichLevel("/"));
assert.throws(() => misc.atWhichLevel("/xxyy"));
});
it("should treat folders correctly", () => {
expect(misc.atWhichLevel("x/")).to.be.equal(1);
expect(misc.atWhichLevel("x/y/")).to.be.equal(2);
assert.equal(misc.atWhichLevel("x/"), 1);
assert.equal(misc.atWhichLevel("x/y/"), 2);
});
it("should treat files correctly", () => {
expect(misc.atWhichLevel("x.md")).to.be.equal(1);
expect(misc.atWhichLevel("x/y.md")).to.be.equal(2);
expect(misc.atWhichLevel("x/y/z.md")).to.be.equal(3);
assert.equal(misc.atWhichLevel("x.md"), 1);
assert.equal(misc.atWhichLevel("x/y.md"), 2);
assert.equal(misc.atWhichLevel("x/y/z.md"), 3);
});
});
describe("Misc: special char for dir", () => {
it("should return false for normal string", () => {
expect(misc.checkHasSpecialCharForDir("")).to.be.false;
expect(misc.checkHasSpecialCharForDir("xxx")).to.be.false;
expect(misc.checkHasSpecialCharForDir("yyy_xxx")).to.be.false;
expect(misc.checkHasSpecialCharForDir("yyy.xxx")).to.be.false;
expect(misc.checkHasSpecialCharForDir("yyyxxx")).to.be.false;
assert.ok(!misc.checkHasSpecialCharForDir(""));
assert.ok(!misc.checkHasSpecialCharForDir("xxx"));
assert.ok(!misc.checkHasSpecialCharForDir("yyy_xxx"));
assert.ok(!misc.checkHasSpecialCharForDir("yyy.xxx"));
assert.ok(!misc.checkHasSpecialCharForDir("yyyxxx"));
});
it("should return true for special cases", () => {
expect(misc.checkHasSpecialCharForDir("?")).to.be.true;
expect(misc.checkHasSpecialCharForDir("/")).to.be.true;
expect(misc.checkHasSpecialCharForDir("\\")).to.be.true;
expect(misc.checkHasSpecialCharForDir("xxx/yyy")).to.be.true;
expect(misc.checkHasSpecialCharForDir("xxx\\yyy")).to.be.true;
expect(misc.checkHasSpecialCharForDir("xxx?yyy")).to.be.true;
assert.ok(misc.checkHasSpecialCharForDir("?"));
assert.ok(misc.checkHasSpecialCharForDir("/"));
assert.ok(misc.checkHasSpecialCharForDir("\\"));
assert.ok(misc.checkHasSpecialCharForDir("xxx/yyy"));
assert.ok(misc.checkHasSpecialCharForDir("xxx\\yyy"));
assert.ok(misc.checkHasSpecialCharForDir("xxx?yyy"));
});
});
describe("Misc: Dropbox: should fix the folder name cases", () => {
it("should do nothing on empty folders", () => {
const input: any[] = [];
expect(misc.fixEntityListCasesInplace(input)).to.be.empty;
assert.equal(misc.fixEntityListCasesInplace(input).length, 0);
});
it("should sort folders by length by side effect", () => {
@@ -306,7 +306,7 @@ describe("Misc: Dropbox: should fix the folder name cases", () => {
{ keyRaw: "bbb/" },
{ keyRaw: "aaaa/" },
];
expect(misc.fixEntityListCasesInplace(input)).to.deep.equal(output);
assert.deepEqual(misc.fixEntityListCasesInplace(input), output);
});
it("should fix folder names", () => {
@@ -335,6 +335,6 @@ describe("Misc: Dropbox: should fix the folder name cases", () => {
{ keyRaw: "ddd/eee/fff.md" },
{ keyRaw: "Ggg/Hhh你好/Fff世界.md" },
];
expect(misc.fixEntityListCasesInplace(input)).to.deep.equal(output);
assert.deepEqual(misc.fixEntityListCasesInplace(input), output);
});
});
+2
View File
@@ -68,6 +68,7 @@ module.exports = {
// crypto: false,
// domain: require.resolve("domain-browser"),
// events: require.resolve("events"),
fs: false,
http: false,
// http: require.resolve("stream-http"),
https: false,
@@ -87,6 +88,7 @@ module.exports = {
url: require.resolve("url/"),
// util: require.resolve("util"),
// vm: require.resolve("vm-browserify"),
vm: false,
// zlib: require.resolve("browserify-zlib"),
},
},