Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
f26e491922 | ||
|
|
dbf80c3359 | ||
|
|
9ddb840077 | ||
|
|
17135abbd4 | ||
|
|
af8357ba6e | ||
|
|
bfa23eb8a5 | ||
|
|
d3d26bfe49 | ||
|
|
f6f7d83a35 | ||
|
|
9e29405103 | ||
|
|
3e7e05fc3d | ||
|
|
391df3ed2d | ||
|
|
c542789c93 | ||
|
|
48e5601a52 | ||
|
|
054a9a3c01 | ||
|
|
66dcaf4e4c | ||
|
|
6d29554033 | ||
|
|
4ed0d735a2 | ||
|
|
26dff02daa | ||
|
|
1b1083d746 | ||
|
|
6db005018d | ||
|
|
8d25ddb636 | ||
|
|
b0d41478a3 | ||
|
|
f69adf273a | ||
|
|
4b4a9b0989 | ||
|
|
bfe11f44dc | ||
|
|
d25303cfac | ||
|
|
ebd62b1581 | ||
|
|
13537d1481 | ||
|
|
6fce079481 | ||
|
|
baf4e6b369 | ||
|
|
dd39e229d8 | ||
|
|
ba6e8be304 | ||
|
|
cb7a372b69 | ||
|
|
9a4dd0186d | ||
|
|
f9cdf323b8 | ||
|
|
2eda5618df |
@@ -92,6 +92,8 @@ Additionally, the plugin author may occasionally visit Obsidian official forum a
|
||||
- If you are using Obsidian desktop < 0.13.25 or iOS < 1.1.1 or any Android version:
|
||||
- The webdav server has to be enabled CORS for requests from `app://obsidian.md` and `capacitor://localhost` and `http://localhost`, **AND** all webdav HTTP methods, **AND** all webdav headers. These are required, because Obsidian mobile works like a browser and mobile plugins are limited by CORS policies unless under a upgraded Obsidian version.
|
||||
- Popular software NextCloud, OwnCloud, `rclone serve webdav` do **NOT** enable CORS by default. If you are using any of them, you should evaluate the risk, and find a way to enable CORS, before using this plugin, or use a upgraded Obsidian version.
|
||||
- **Unofficial** workaround: NextCloud users can **evaluate the risk by themselves**, and if decide to accept the risk, they can install [WebAppPassword](https://apps.nextcloud.com/apps/webapppassword) app, and add `app://obsidian.md`, `capacitor://localhost`, `http://localhost` to `Allowed origins`
|
||||
- **Unofficial** workaround: OwnCloud users can **evaluate the risk by themselves**, and if decide to accept the risk, they can download `.tar.gz` of `WebAppPassword` above and manually install and configure it on their instances.
|
||||
- The plugin is tested successfully under python package [`wsgidav` (version 4.0)](https://github.com/mar10/wsgidav). See [this issue](https://github.com/mar10/wsgidav/issues/239) for some details.
|
||||
- Your data would be synced to a `${vaultName}` sub folder on your webdav server.
|
||||
- Password-based end-to-end encryption is also supported. But please be aware that **the vault name itself is not encrypted**.
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
# How To Configure Apache CORS Rules for Webdav
|
||||
|
||||
This method is [contributed by community](https://github.com/remotely-save/remotely-save/pull/31).
|
||||
|
||||
You should evaluate the risk by yourself before trying this method.
|
||||
|
||||
**You are strongly advised to backup your original Apache configuration files before making any changes.**
|
||||
|
||||
```apacheconf
|
||||
<IfModule mod_headers.c>
|
||||
# Obsidian webdav
|
||||
SetEnvIf Origin "^app://obsidian.md$" IS_OBSIDIAN
|
||||
SetEnvIf Origin "^capacitor://localhost$" IS_OBSIDIAN
|
||||
SetEnvIf Origin "^http://localhost$" IS_OBSIDIAN
|
||||
Header always set Access-Control-Allow-Origin "*" env=IS_OBSIDIAN
|
||||
Header always set Access-Control-Allow-Methods "GET, HEAD, POST, PUT, OPTIONS, MOVE, DELETE, COPY, LOCK, UNLOCK, PROPFIND" env=IS_OBSIDIAN
|
||||
Header always set Access-Control-Allow-Headers "Authorization, Depth, DNT, User-Agent, Keep-Alive, Content-Type, accept, origin, X-Requested-With" env=IS_OBSIDIAN
|
||||
Header always set Access-Control-Expose-Headers "etag, dav" env=IS_OBSIDIAN
|
||||
|
||||
# Allow OPTION request without authentication and respond with status 200
|
||||
RewriteCond %{ENV:IS_OBSIDIAN} 1
|
||||
RewriteCond %{REQUEST_METHOD} OPTIONS
|
||||
RewriteRule ^(.*)$ $1 [R=200,L]
|
||||
</IfModule>
|
||||
```
|
||||
@@ -4,6 +4,11 @@ The plugin is developed for the browser environment. The "fake" browser behind t
|
||||
|
||||
[MDN has a doc about CORS.](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS)
|
||||
|
||||
From Obsidian desktop >= 0.13.25 or mobile >= 1.1.1, Obsidian [provides a new API `requiestUrl`](https://forum.obsidian.md/t/obsidian-release-v0-13-25-insider-build/32701), that allows the plugin to fully bypass the CORS issue. As of Mar 2022, the latest public-released Obsidian desktop has supported this API, but the Obsidian mobile still stays in insider.
|
||||
1. From Obsidian desktop >= 0.13.25 or mobile >= 1.1.1, Obsidian [provides a new API `requiestUrl`](https://forum.obsidian.md/t/obsidian-release-v0-13-25-insider-build/32701), that allows the plugin to fully bypass the CORS issue. As of Mar 2022, the latest public-released Obsidian desktop has supported this API, but the Obsidian mobile still stays in insider.
|
||||
|
||||
For using this plugin in Obsidian desktop < 0.13.25 or mobile < 1.1.1, we need to configure the server side to return the header `Access-Control-Allow-Origin` allowing the origins `app://obsidian.md` and `capacitor://localhost` and `http://localhost`. Here is an example [configuration for Amazon S3](./s3_cors_configure.md).
|
||||
2. For using this plugin in Obsidian desktop < 0.13.25 or mobile < 1.1.1, we need to configure the server side to return the header `Access-Control-Allow-Origin` allowing the origins `app://obsidian.md` and `capacitor://localhost` and `http://localhost`.
|
||||
|
||||
Example configurations:
|
||||
|
||||
- [Amazon S3](./s3_cors_configure.md)
|
||||
- [Apache](./apache_cors_configure.md) ([contributed by community](https://github.com/remotely-save/remotely-save/pull/31))
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
# Sync Ignoring Large Files
|
||||
|
||||
Initially, the plugin does not ignore large files.
|
||||
|
||||
From the new version in May 2022, it can ignore all files with some sizes. But we need some rules to make the function compatible with existing conditions.
|
||||
|
||||
1. If users are using E2E password mode, then the file sizes are compared on the **encrypted sizes**, rather than the original unencripted file sizes. The reasons are: the encrypted ones are in transferations, and the encrypted sizes can be computed from unencrypted sizes but not the reverse.
|
||||
|
||||
2. Assuming the file A, is already synced between local device and remote service before.
|
||||
|
||||
- If the local size and remote size are both below the threshold, then the file can be synced normally.
|
||||
- If the local size and remote size are both above the threshold, then the file will be ignored normally.
|
||||
- If the local size is below the threshold, and the remote size is above the threshold, then the plugin **rejects** the sync, and throws the error to the user.
|
||||
- If the local size is above the threshold, and the remote size is below the threshold, then the plugin **rejects** the sync, and throws the error to the user.
|
||||
- When it somes to deletions, the same rules apply.
|
||||
|
||||
The main point is that, if the file sizes "cross the line", the plugin does not introduce any further trouble and just reject to work for this file.
|
||||
+4
-1
@@ -31,11 +31,11 @@ esbuild
|
||||
"obsidian",
|
||||
"electron",
|
||||
"fs",
|
||||
"crypto",
|
||||
"tls",
|
||||
"net",
|
||||
// ...builtins
|
||||
],
|
||||
inject: ["./esbuild.injecthelper.mjs"],
|
||||
format: "cjs",
|
||||
watch: !prod,
|
||||
target: "es2016",
|
||||
@@ -48,6 +48,9 @@ esbuild
|
||||
"process.env.DEFAULT_DROPBOX_APP_KEY": `"${DEFAULT_DROPBOX_APP_KEY}"`,
|
||||
"process.env.DEFAULT_ONEDRIVE_CLIENT_ID": `"${DEFAULT_ONEDRIVE_CLIENT_ID}"`,
|
||||
"process.env.DEFAULT_ONEDRIVE_AUTHORITY": `"${DEFAULT_ONEDRIVE_AUTHORITY}"`,
|
||||
global: "window",
|
||||
"process.env.NODE_DEBUG": `undefined`, // ugly fix
|
||||
"process.env.DEBUG": `undefined`, // ugly fix
|
||||
},
|
||||
})
|
||||
.catch(() => process.exit(1));
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
export let Buffer = require("buffer").Buffer;
|
||||
export let process = require("process/browser");
|
||||
+1
-1
@@ -1,7 +1,7 @@
|
||||
{
|
||||
"id": "remotely-save",
|
||||
"name": "Remotely Save",
|
||||
"version": "0.3.18",
|
||||
"version": "0.3.23",
|
||||
"minAppVersion": "0.13.21",
|
||||
"description": "Yet another unofficial plugin allowing users to synchronize notes between local device and the cloud service.",
|
||||
"author": "fyears",
|
||||
|
||||
+41
-39
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "remotely-save",
|
||||
"version": "0.3.18",
|
||||
"version": "0.3.23",
|
||||
"description": "This is yet another sync plugin for Obsidian app.",
|
||||
"scripts": {
|
||||
"dev2": "node esbuild.config.mjs",
|
||||
@@ -23,71 +23,73 @@
|
||||
"author": "",
|
||||
"license": "Apache-2.0",
|
||||
"devDependencies": {
|
||||
"@microsoft/microsoft-graph-types": "^2.11.0",
|
||||
"@types/chai": "^4.2.22",
|
||||
"@types/chai-as-promised": "^7.1.4",
|
||||
"@types/jsdom": "^16.2.13",
|
||||
"@types/lodash": "^4.14.178",
|
||||
"@microsoft/microsoft-graph-types": "^2.19.0",
|
||||
"@types/chai": "^4.3.1",
|
||||
"@types/chai-as-promised": "^7.1.5",
|
||||
"@types/jsdom": "^16.2.14",
|
||||
"@types/lodash": "^4.14.182",
|
||||
"@types/mime-types": "^2.1.1",
|
||||
"@types/mocha": "^9.0.0",
|
||||
"@types/mocha": "^9.1.1",
|
||||
"@types/mustache": "^4.1.2",
|
||||
"@types/node": "^14.14.37",
|
||||
"@types/qrcode": "^1.4.1",
|
||||
"@types/node": "^17.0.30",
|
||||
"@types/qrcode": "^1.4.2",
|
||||
"builtin-modules": "^3.2.0",
|
||||
"chai": "^4.3.4",
|
||||
"chai": "^4.3.6",
|
||||
"chai-as-promised": "^7.1.1",
|
||||
"cross-env": "^7.0.3",
|
||||
"dotenv": "^10.0.0",
|
||||
"esbuild": "^0.14.27",
|
||||
"dotenv": "^16.0.0",
|
||||
"esbuild": "^0.14.38",
|
||||
"jsdom": "^19.0.0",
|
||||
"mocha": "^9.1.3",
|
||||
"prettier": "^2.4.1",
|
||||
"ts-loader": "^9.2.6",
|
||||
"ts-node": "^10.4.0",
|
||||
"tslib": "^2.2.0",
|
||||
"typescript": "^4.4.4",
|
||||
"mocha": "^9.2.2",
|
||||
"prettier": "^2.6.2",
|
||||
"ts-loader": "^9.2.9",
|
||||
"ts-node": "^10.7.0",
|
||||
"tslib": "^2.4.0",
|
||||
"typescript": "^4.6.4",
|
||||
"webdav-server": "^2.6.2",
|
||||
"webpack": "^5.64.4",
|
||||
"webpack-cli": "^4.9.1"
|
||||
"webpack": "^5.72.0",
|
||||
"webpack-cli": "^4.9.2"
|
||||
},
|
||||
"dependencies": {
|
||||
"@aws-sdk/client-s3": "^3.67.0",
|
||||
"@aws-sdk/fetch-http-handler": "^3.58.0",
|
||||
"@aws-sdk/lib-storage": "^3.67.0",
|
||||
"@aws-sdk/protocol-http": "^3.58.0",
|
||||
"@aws-sdk/querystring-builder": "^3.55.0",
|
||||
"@aws-sdk/signature-v4-crt": "^3.66.0",
|
||||
"@aws-sdk/types": "^3.55.0",
|
||||
"@azure/msal-node": "^1.6.0",
|
||||
"@aws-sdk/client-s3": "^3.81.0",
|
||||
"@aws-sdk/fetch-http-handler": "^3.78.0",
|
||||
"@aws-sdk/lib-storage": "^3.81.0",
|
||||
"@aws-sdk/protocol-http": "^3.78.0",
|
||||
"@aws-sdk/querystring-builder": "^3.78.0",
|
||||
"@aws-sdk/signature-v4-crt": "^3.78.0",
|
||||
"@aws-sdk/types": "^3.78.0",
|
||||
"@azure/msal-node": "^1.8.0",
|
||||
"@fyears/tsqueue": "^1.0.1",
|
||||
"@microsoft/microsoft-graph-client": "^3.0.1",
|
||||
"acorn": "^8.5.0",
|
||||
"@microsoft/microsoft-graph-client": "^3.0.2",
|
||||
"acorn": "^8.7.1",
|
||||
"aggregate-error": "^4.0.0",
|
||||
"assert": "^2.0.0",
|
||||
"aws-crt": "^1.10.1",
|
||||
"aws-crt": "^1.12.1",
|
||||
"buffer": "^6.0.3",
|
||||
"crypto-browserify": "^3.12.0",
|
||||
"dropbox": "^10.22.0",
|
||||
"delay": "^5.0.0",
|
||||
"dropbox": "^10.28.0",
|
||||
"emoji-regex": "^10.1.0",
|
||||
"http-status-codes": "^2.2.0",
|
||||
"localforage": "^1.10.0",
|
||||
"lodash": "^4.17.21",
|
||||
"loglevel": "^1.8.0",
|
||||
"lucide": "^0.17.12",
|
||||
"mime-types": "^2.1.33",
|
||||
"lucide": "^0.35.0",
|
||||
"mime-types": "^2.1.35",
|
||||
"mustache": "^4.2.0",
|
||||
"nanoid": "^3.1.30",
|
||||
"obsidian": "^0.13.26",
|
||||
"nanoid": "^3.3.3",
|
||||
"obsidian": "^0.14.6",
|
||||
"p-queue": "^7.2.0",
|
||||
"path-browserify": "^1.0.1",
|
||||
"process": "^0.11.10",
|
||||
"qrcode": "^1.5.0",
|
||||
"rfc4648": "^1.5.0",
|
||||
"rfc4648": "^1.5.1",
|
||||
"rimraf": "^3.0.2",
|
||||
"stream-browserify": "^3.0.0",
|
||||
"url": "^0.11.0",
|
||||
"util": "^0.12.4",
|
||||
"webdav": "^4.7.0",
|
||||
"webdav-fs": "^4.0.0",
|
||||
"webdav": "^4.9.0",
|
||||
"webdav-fs": "^4.0.1",
|
||||
"xregexp": "^5.1.0"
|
||||
}
|
||||
}
|
||||
|
||||
+22
-3
@@ -85,6 +85,7 @@ export interface RemotelySavePluginSettings {
|
||||
syncUnderscoreItems?: boolean;
|
||||
lang?: LangTypeAndAuto;
|
||||
logToDB?: boolean;
|
||||
skipSizeLargerThan?: number;
|
||||
|
||||
/**
|
||||
* @deprecated
|
||||
@@ -122,13 +123,24 @@ type DecisionTypeForFile =
|
||||
| "uploadLocalToRemote" // "skipLocal && uploadLocalToRemote && cleanLocalDelHist && cleanRemoteDelHist"
|
||||
| "downloadRemoteToLocal"; // "downloadRemoteToLocal && skipRemote && cleanLocalDelHist && cleanRemoteDelHist"
|
||||
|
||||
type DecisionTypeForFileSize =
|
||||
| "skipUploadingTooLarge"
|
||||
| "skipDownloadingTooLarge"
|
||||
| "skipUsingLocalDelTooLarge"
|
||||
| "skipUsingRemoteDelTooLarge"
|
||||
| "errorLocalTooLargeConflictRemote"
|
||||
| "errorRemoteTooLargeConflictLocal";
|
||||
|
||||
type DecisionTypeForFolder =
|
||||
| "createFolder"
|
||||
| "uploadLocalDelHistToRemoteFolder"
|
||||
| "keepRemoteDelHistFolder"
|
||||
| "skipFolder";
|
||||
|
||||
export type DecisionType = DecisionTypeForFile | DecisionTypeForFolder;
|
||||
export type DecisionType =
|
||||
| DecisionTypeForFile
|
||||
| DecisionTypeForFileSize
|
||||
| DecisionTypeForFolder;
|
||||
|
||||
export interface FileOrFolderMixedState {
|
||||
key: string;
|
||||
@@ -139,7 +151,9 @@ export interface FileOrFolderMixedState {
|
||||
deltimeLocal?: number;
|
||||
deltimeRemote?: number;
|
||||
sizeLocal?: number;
|
||||
sizeLocalEnc?: number;
|
||||
sizeRemote?: number;
|
||||
sizeRemoteEnc?: number;
|
||||
changeRemoteMtimeUsingMapping?: boolean;
|
||||
changeLocalMtimeUsingMapping?: boolean;
|
||||
decision?: DecisionType;
|
||||
@@ -154,11 +168,16 @@ export interface FileOrFolderMixedState {
|
||||
}
|
||||
|
||||
export const API_VER_STAT_FOLDER = "0.13.27";
|
||||
export const API_VER_REQURL = "0.13.26";
|
||||
export const API_VER_REQURL = "0.13.26"; // desktop ver 0.13.26, iOS ver 1.1.1
|
||||
export const API_VER_REQURL_ANDROID = "0.14.6"; // Android ver 1.2.1
|
||||
|
||||
export const VALID_REQURL =
|
||||
requireApiVersion(API_VER_REQURL) && !Platform.isAndroidApp;
|
||||
(!Platform.isAndroidApp && requireApiVersion(API_VER_REQURL)) ||
|
||||
(Platform.isAndroidApp && requireApiVersion(API_VER_REQURL_ANDROID));
|
||||
|
||||
export const DEFAULT_DEBUG_FOLDER = "_debug_remotely_save/";
|
||||
export const DEFAULT_SYNC_PLANS_HISTORY_FILE_PREFIX =
|
||||
"sync_plans_hist_exported_on_";
|
||||
export const DEFAULT_LOG_HISTORY_FILE_PREFIX = "log_hist_exported_on_";
|
||||
|
||||
export type SyncTriggerSourceType = "manual" | "auto" | "dry" | "autoOnceInit";
|
||||
|
||||
@@ -26,11 +26,13 @@ const turnSyncPlanToTable = (record: string) => {
|
||||
"remoteEncryptedKey",
|
||||
"existLocal",
|
||||
"sizeLocal",
|
||||
"sizeLocalEnc",
|
||||
"mtimeLocal",
|
||||
"deltimeLocal",
|
||||
"changeLocalMtimeUsingMapping",
|
||||
"existRemote",
|
||||
"sizeRemote",
|
||||
"sizeRemoteEnc",
|
||||
"mtimeRemote",
|
||||
"deltimeRemote",
|
||||
"changeRemoteMtimeUsingMapping",
|
||||
|
||||
+1
-1
Submodule src/langs updated: 470fe141ef...ad48de9227
+21
-7
@@ -9,7 +9,11 @@ import {
|
||||
} from "obsidian";
|
||||
import cloneDeep from "lodash/cloneDeep";
|
||||
import { createElement, RotateCcw, RefreshCcw, FileText } from "lucide";
|
||||
import type { RemotelySavePluginSettings } from "./baseTypes";
|
||||
import type {
|
||||
FileOrFolderMixedState,
|
||||
RemotelySavePluginSettings,
|
||||
SyncTriggerSourceType,
|
||||
} from "./baseTypes";
|
||||
import {
|
||||
COMMAND_CALLBACK,
|
||||
COMMAND_CALLBACK_ONEDRIVE,
|
||||
@@ -61,6 +65,7 @@ import {
|
||||
exportVaultLoggerOutputToFiles,
|
||||
exportVaultSyncPlansToFiles,
|
||||
} from "./debugMode";
|
||||
import { SizesConflictModal } from "./syncSizesConflictNotice";
|
||||
|
||||
const DEFAULT_SETTINGS: RemotelySavePluginSettings = {
|
||||
s3: DEFAULT_S3_CONFIG,
|
||||
@@ -79,6 +84,7 @@ const DEFAULT_SETTINGS: RemotelySavePluginSettings = {
|
||||
syncUnderscoreItems: false,
|
||||
lang: "auto",
|
||||
logToDB: false,
|
||||
skipSizeLargerThan: -1,
|
||||
};
|
||||
|
||||
interface OAuth2Info {
|
||||
@@ -89,8 +95,6 @@ interface OAuth2Info {
|
||||
revokeAuthSetting?: Setting;
|
||||
}
|
||||
|
||||
type SyncTriggerSourceType = "manual" | "auto" | "dry" | "autoOnceInit";
|
||||
|
||||
const iconNameSyncWait = `remotely-save-sync-wait`;
|
||||
const iconNameSyncRunning = `remotely-save-sync-running`;
|
||||
const iconNameLogs = `remotely-save-logs`;
|
||||
@@ -279,23 +283,23 @@ export default class RemotelySavePlugin extends Plugin {
|
||||
})
|
||||
);
|
||||
this.syncStatus = "generating_plan";
|
||||
const { plan, sortedKeys, deletions } = await getSyncPlan(
|
||||
const { plan, sortedKeys, deletions, sizesGoWrong } = await getSyncPlan(
|
||||
remoteStates,
|
||||
local,
|
||||
localConfigDirContents,
|
||||
origMetadataOnRemote.deletions,
|
||||
localHistory,
|
||||
client.serviceType,
|
||||
triggerSource,
|
||||
this.app.vault,
|
||||
this.settings.syncConfigDir,
|
||||
this.app.vault.configDir,
|
||||
this.settings.syncUnderscoreItems,
|
||||
this.settings.skipSizeLargerThan,
|
||||
this.settings.password
|
||||
);
|
||||
log.info(plan.mixedStates); // for debugging
|
||||
if (triggerSource !== "dry") {
|
||||
await insertSyncPlanRecordByVault(this.db, plan, this.vaultRandomID);
|
||||
}
|
||||
await insertSyncPlanRecordByVault(this.db, plan, this.vaultRandomID);
|
||||
|
||||
// The operations above are almost read only and kind of safe.
|
||||
// The operations below begins to write or delete (!!!) something.
|
||||
@@ -317,10 +321,20 @@ export default class RemotelySavePlugin extends Plugin {
|
||||
sortedKeys,
|
||||
metadataFile,
|
||||
origMetadataOnRemote,
|
||||
sizesGoWrong,
|
||||
deletions,
|
||||
(key: string) => self.trash(key),
|
||||
this.settings.password,
|
||||
this.settings.concurrency,
|
||||
(ss: FileOrFolderMixedState[]) => {
|
||||
new SizesConflictModal(
|
||||
self.app,
|
||||
self,
|
||||
this.settings.skipSizeLargerThan,
|
||||
ss,
|
||||
this.settings.password !== ""
|
||||
).open();
|
||||
},
|
||||
(i: number, totalCount: number, pathName: string, decision: string) =>
|
||||
self.setCurrSyncMsg(i, totalCount, pathName, decision)
|
||||
);
|
||||
|
||||
+34
-8
@@ -1,6 +1,7 @@
|
||||
import isEqual from "lodash/isEqual";
|
||||
import { base64url } from "rfc4648";
|
||||
import { reverseString } from "./misc";
|
||||
import { log } from "./moreOnLog";
|
||||
|
||||
const DEFAULT_README_FOR_METADATAONREMOTE =
|
||||
"Do NOT edit or delete the file manually. This file is for the plugin remotely-save to store some necessary meta data on the remote services. Its content is slightly obfuscated.";
|
||||
@@ -69,19 +70,44 @@ export const deserializeMetadataOnRemote = (x: string | ArrayBuffer) => {
|
||||
} else {
|
||||
y1 = new TextDecoder().decode(x);
|
||||
}
|
||||
const y2: any = JSON.parse(y1);
|
||||
|
||||
if (!("readme" in y2 && "d" in y2)) {
|
||||
throw Error("invalid remote meta data file!");
|
||||
let y2: any;
|
||||
try {
|
||||
y2 = JSON.parse(y1);
|
||||
} catch (e) {
|
||||
throw new Error(
|
||||
`invalid remote meta data file with first few chars: ${y1.slice(0, 5)}`
|
||||
);
|
||||
}
|
||||
|
||||
const y3 = JSON.parse(
|
||||
(
|
||||
if (!("readme" in y2 && "d" in y2)) {
|
||||
throw new Error(
|
||||
'invalid remote meta data file (no "readme" or "d" fields)!'
|
||||
);
|
||||
}
|
||||
|
||||
let y3: string;
|
||||
try {
|
||||
y3 = (
|
||||
base64url.parse(reverseString(y2["d"]), {
|
||||
out: Buffer.allocUnsafe as any,
|
||||
loose: true,
|
||||
}) as Buffer
|
||||
).toString("utf-8")
|
||||
) as MetadataOnRemote;
|
||||
return y3;
|
||||
).toString("utf-8");
|
||||
} catch (e) {
|
||||
throw new Error('invalid remote meta data file (invalid "d" field)!');
|
||||
}
|
||||
|
||||
let y4: MetadataOnRemote;
|
||||
try {
|
||||
y4 = JSON.parse(y3) as MetadataOnRemote;
|
||||
} catch (e) {
|
||||
throw new Error(
|
||||
`invalid remote meta data file with \"d\" field with first few chars: ${y3.slice(
|
||||
0,
|
||||
5
|
||||
)}`
|
||||
);
|
||||
}
|
||||
return y4;
|
||||
};
|
||||
|
||||
+29
@@ -3,6 +3,7 @@ import * as path from "path";
|
||||
|
||||
import { base32, base64url } from "rfc4648";
|
||||
import XRegExp from "xregexp";
|
||||
import emojiRegex from "emoji-regex";
|
||||
|
||||
import { log } from "./moreOnLog";
|
||||
|
||||
@@ -162,6 +163,34 @@ export const isVaildText = (a: string) => {
|
||||
);
|
||||
};
|
||||
|
||||
/**
|
||||
* Use regex to detect a text contains emoji or not.
|
||||
* @param a
|
||||
* @returns
|
||||
*/
|
||||
export const hasEmojiInText = (a: string) => {
|
||||
const regex = emojiRegex();
|
||||
return regex.test(a);
|
||||
};
|
||||
|
||||
/**
|
||||
* Convert the headers to a normal object.
|
||||
* @param h
|
||||
* @param toLower
|
||||
* @returns
|
||||
*/
|
||||
export const headersToRecord = (h: Headers, toLower: boolean = true) => {
|
||||
const res: Record<string, string> = {};
|
||||
h.forEach((v, k) => {
|
||||
if (toLower) {
|
||||
res[k.toLowerCase()] = v;
|
||||
} else {
|
||||
res[k] = v;
|
||||
}
|
||||
});
|
||||
return res;
|
||||
};
|
||||
|
||||
/**
|
||||
* If input is already a folder, returns it as is;
|
||||
* And if input is a file, returns its direname.
|
||||
|
||||
+10
-8
@@ -15,7 +15,6 @@ import { log } from "./moreOnLog";
|
||||
|
||||
export class RemoteClient {
|
||||
readonly serviceType: SUPPORTED_SERVICES_TYPE;
|
||||
readonly s3Client?: s3.S3Client;
|
||||
readonly s3Config?: S3Config;
|
||||
readonly webdavClient?: webdav.WrappedWebdavClient;
|
||||
readonly webdavConfig?: WebdavConfig;
|
||||
@@ -38,7 +37,6 @@ export class RemoteClient {
|
||||
// so we use a ref not copy of config here
|
||||
if (serviceType === "s3") {
|
||||
this.s3Config = s3Config;
|
||||
this.s3Client = s3.getS3Client(this.s3Config);
|
||||
} else if (serviceType === "webdav") {
|
||||
if (vaultName === undefined || saveUpdatedConfigFunc === undefined) {
|
||||
throw Error(
|
||||
@@ -86,7 +84,7 @@ export class RemoteClient {
|
||||
getRemoteMeta = async (fileOrFolderPath: string) => {
|
||||
if (this.serviceType === "s3") {
|
||||
return await s3.getRemoteMeta(
|
||||
this.s3Client,
|
||||
s3.getS3Client(this.s3Config),
|
||||
this.s3Config,
|
||||
fileOrFolderPath
|
||||
);
|
||||
@@ -116,7 +114,7 @@ export class RemoteClient {
|
||||
) => {
|
||||
if (this.serviceType === "s3") {
|
||||
return await s3.uploadToRemote(
|
||||
this.s3Client,
|
||||
s3.getS3Client(this.s3Config),
|
||||
this.s3Config,
|
||||
fileOrFolderPath,
|
||||
vault,
|
||||
@@ -168,7 +166,11 @@ export class RemoteClient {
|
||||
|
||||
listFromRemote = async (prefix?: string) => {
|
||||
if (this.serviceType === "s3") {
|
||||
return await s3.listFromRemote(this.s3Client, this.s3Config, prefix);
|
||||
return await s3.listFromRemote(
|
||||
s3.getS3Client(this.s3Config),
|
||||
this.s3Config,
|
||||
prefix
|
||||
);
|
||||
} else if (this.serviceType === "webdav") {
|
||||
return await webdav.listFromRemote(this.webdavClient, prefix);
|
||||
} else if (this.serviceType === "dropbox") {
|
||||
@@ -190,7 +192,7 @@ export class RemoteClient {
|
||||
) => {
|
||||
if (this.serviceType === "s3") {
|
||||
return await s3.downloadFromRemote(
|
||||
this.s3Client,
|
||||
s3.getS3Client(this.s3Config),
|
||||
this.s3Config,
|
||||
fileOrFolderPath,
|
||||
vault,
|
||||
@@ -241,7 +243,7 @@ export class RemoteClient {
|
||||
) => {
|
||||
if (this.serviceType === "s3") {
|
||||
return await s3.deleteFromRemote(
|
||||
this.s3Client,
|
||||
s3.getS3Client(this.s3Config),
|
||||
this.s3Config,
|
||||
fileOrFolderPath,
|
||||
password,
|
||||
@@ -276,7 +278,7 @@ export class RemoteClient {
|
||||
checkConnectivity = async (callbackFunc?: any) => {
|
||||
if (this.serviceType === "s3") {
|
||||
return await s3.checkConnectivity(
|
||||
this.s3Client,
|
||||
s3.getS3Client(this.s3Config),
|
||||
this.s3Config,
|
||||
callbackFunc
|
||||
);
|
||||
|
||||
+153
-30
@@ -1,4 +1,6 @@
|
||||
import { Dropbox, DropboxAuth, files } from "dropbox";
|
||||
import delay from "delay";
|
||||
import { Dropbox, DropboxAuth } from "dropbox";
|
||||
import type { files, DropboxResponseError, DropboxResponse } from "dropbox";
|
||||
import { Vault } from "obsidian";
|
||||
import * as path from "path";
|
||||
import {
|
||||
@@ -8,7 +10,13 @@ import {
|
||||
OAUTH2_FORCE_EXPIRE_MILLISECONDS,
|
||||
} from "./baseTypes";
|
||||
import { decryptArrayBuffer, encryptArrayBuffer } from "./encrypt";
|
||||
import { bufferToArrayBuffer, getFolderLevels, mkdirpInVault } from "./misc";
|
||||
import {
|
||||
bufferToArrayBuffer,
|
||||
getFolderLevels,
|
||||
hasEmojiInText,
|
||||
headersToRecord,
|
||||
mkdirpInVault,
|
||||
} from "./misc";
|
||||
|
||||
export { Dropbox } from "dropbox";
|
||||
|
||||
@@ -270,6 +278,73 @@ export const setConfigBySuccessfullAuthInplace = async (
|
||||
// Other usual common methods
|
||||
////////////////////////////////////////////////////////////////////////////////
|
||||
|
||||
interface ErrSubType {
|
||||
error: {
|
||||
retry_after: number;
|
||||
};
|
||||
}
|
||||
|
||||
async function retryReq<T>(
|
||||
reqFunc: () => Promise<DropboxResponse<T>>,
|
||||
extraHint: string = ""
|
||||
): Promise<DropboxResponse<T>> {
|
||||
const waitSeconds = [1, 2, 4, 8]; // hard code exponential backoff
|
||||
for (let idx = 0; idx < waitSeconds.length; ++idx) {
|
||||
try {
|
||||
if (idx !== 0) {
|
||||
log.warn(
|
||||
`${extraHint === "" ? "" : extraHint + ": "}The ${
|
||||
idx + 1
|
||||
}-th try starts at time ${Date.now()}`
|
||||
);
|
||||
}
|
||||
return await reqFunc();
|
||||
} catch (e: unknown) {
|
||||
const err = e as DropboxResponseError<ErrSubType>;
|
||||
if (err.status === undefined) {
|
||||
// then the err is not DropboxResponseError
|
||||
throw err;
|
||||
}
|
||||
if (err.status !== 429) {
|
||||
// then the err is not "too many requests", give up
|
||||
throw err;
|
||||
}
|
||||
|
||||
if (idx === waitSeconds.length - 1) {
|
||||
// the last retry also failed, give up
|
||||
throw new Error(
|
||||
`${
|
||||
extraHint === "" ? "" : extraHint + ": "
|
||||
}"429 too many requests", after retrying for ${
|
||||
idx + 1
|
||||
} times still failed.`
|
||||
);
|
||||
}
|
||||
|
||||
const headers = headersToRecord(err.headers);
|
||||
const svrSec =
|
||||
err.error.error.retry_after ||
|
||||
parseInt(headers["retry-after"] || "1") ||
|
||||
1;
|
||||
const fallbackSec = waitSeconds[idx];
|
||||
const secMin = Math.max(svrSec, fallbackSec);
|
||||
const secMax = Math.max(secMin * 1.8, 2);
|
||||
log.warn(
|
||||
`${
|
||||
extraHint === "" ? "" : extraHint + ": "
|
||||
}We have "429 too many requests" error of ${
|
||||
idx + 1
|
||||
}-th try, at time ${Date.now()}, and wait for ${secMin} ~ ${secMax} seconds to retry. Original info: ${JSON.stringify(
|
||||
err.error,
|
||||
null,
|
||||
2
|
||||
)}`
|
||||
);
|
||||
await delay.range(secMin * 1000, secMax * 1000);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export class WrappedDropboxClient {
|
||||
dropboxConfig: DropboxConfig;
|
||||
remoteBaseDir: string;
|
||||
@@ -296,9 +371,13 @@ export class WrappedDropboxClient {
|
||||
throw Error("The user has not manually auth yet.");
|
||||
}
|
||||
const currentTs = Date.now();
|
||||
const customHeaders = {
|
||||
"Cache-Control": "no-cache",
|
||||
};
|
||||
if (this.dropboxConfig.accessTokenExpiresAtTime > currentTs) {
|
||||
this.dropbox = new Dropbox({
|
||||
accessToken: this.dropboxConfig.accessToken,
|
||||
customHeaders: customHeaders,
|
||||
});
|
||||
} else {
|
||||
if (this.dropboxConfig.refreshToken === "") {
|
||||
@@ -318,6 +397,7 @@ export class WrappedDropboxClient {
|
||||
);
|
||||
this.dropbox = new Dropbox({
|
||||
accessToken: this.dropboxConfig.accessToken,
|
||||
customHeaders: customHeaders,
|
||||
});
|
||||
}
|
||||
|
||||
@@ -338,6 +418,13 @@ export class WrappedDropboxClient {
|
||||
}
|
||||
if (!this.vaultFolderExists) {
|
||||
log.info(`remote does not have folder /${this.remoteBaseDir}`);
|
||||
|
||||
if (hasEmojiInText(`/${this.remoteBaseDir}`)) {
|
||||
throw new Error(
|
||||
`/${this.remoteBaseDir}: Error: Dropbox does not support emoji in folder names.`
|
||||
);
|
||||
}
|
||||
|
||||
await this.dropbox.filesCreateFolderV2({
|
||||
path: `/${this.remoteBaseDir}`,
|
||||
});
|
||||
@@ -377,10 +464,12 @@ export const getRemoteMeta = async (
|
||||
// filesGetMetadata doesn't support root folder
|
||||
// we instead try to list files
|
||||
// if no error occurs, we ensemble a fake result.
|
||||
const rsp = await client.dropbox.filesListFolder({
|
||||
path: `/${client.remoteBaseDir}`,
|
||||
recursive: false, // don't need to recursive here
|
||||
});
|
||||
const rsp = await retryReq(() =>
|
||||
client.dropbox.filesListFolder({
|
||||
path: `/${client.remoteBaseDir}`,
|
||||
recursive: false, // don't need to recursive here
|
||||
})
|
||||
);
|
||||
if (rsp.status !== 200) {
|
||||
throw Error(JSON.stringify(rsp));
|
||||
}
|
||||
@@ -395,9 +484,11 @@ export const getRemoteMeta = async (
|
||||
|
||||
const key = getDropboxPath(fileOrFolderPath, client.remoteBaseDir);
|
||||
|
||||
const rsp = await client.dropbox.filesGetMetadata({
|
||||
path: key,
|
||||
});
|
||||
const rsp = await retryReq(() =>
|
||||
client.dropbox.filesGetMetadata({
|
||||
path: key,
|
||||
})
|
||||
);
|
||||
if (rsp.status !== 200) {
|
||||
throw Error(JSON.stringify(rsp));
|
||||
}
|
||||
@@ -423,6 +514,12 @@ export const uploadToRemote = async (
|
||||
}
|
||||
uploadFile = getDropboxPath(uploadFile, client.remoteBaseDir);
|
||||
|
||||
if (hasEmojiInText(uploadFile)) {
|
||||
throw new Error(
|
||||
`${uploadFile}: Error: Dropbox does not support emoji in file / folder names.`
|
||||
);
|
||||
}
|
||||
|
||||
const isFolder = fileOrFolderPath.endsWith("/");
|
||||
|
||||
if (isFolder && isRecursively) {
|
||||
@@ -438,11 +535,19 @@ export const uploadToRemote = async (
|
||||
// created, pass
|
||||
} else {
|
||||
try {
|
||||
await client.dropbox.filesCreateFolderV2({
|
||||
path: uploadFile,
|
||||
});
|
||||
await retryReq(
|
||||
() =>
|
||||
client.dropbox.filesCreateFolderV2({
|
||||
path: uploadFile,
|
||||
}),
|
||||
fileOrFolderPath
|
||||
);
|
||||
foldersCreatedBefore?.add(uploadFile);
|
||||
} catch (err) {
|
||||
} catch (e: unknown) {
|
||||
const err = e as DropboxResponseError<files.CreateFolderError>;
|
||||
if (err.status === undefined) {
|
||||
throw err;
|
||||
}
|
||||
if (err.status === 409) {
|
||||
// pass
|
||||
foldersCreatedBefore?.add(uploadFile);
|
||||
@@ -455,10 +560,14 @@ export const uploadToRemote = async (
|
||||
return res;
|
||||
} else {
|
||||
// if encrypted, upload a fake file with the encrypted file name
|
||||
await client.dropbox.filesUpload({
|
||||
path: uploadFile,
|
||||
contents: "",
|
||||
});
|
||||
await retryReq(
|
||||
() =>
|
||||
client.dropbox.filesUpload({
|
||||
path: uploadFile,
|
||||
contents: "",
|
||||
}),
|
||||
fileOrFolderPath
|
||||
);
|
||||
return await getRemoteMeta(client, uploadFile);
|
||||
}
|
||||
} else {
|
||||
@@ -480,13 +589,19 @@ export const uploadToRemote = async (
|
||||
}
|
||||
// in dropbox, we don't need to create folders before uploading! cool!
|
||||
// TODO: filesUploadSession for larger files (>=150 MB)
|
||||
await client.dropbox.filesUpload({
|
||||
path: uploadFile,
|
||||
contents: remoteContent,
|
||||
mode: {
|
||||
".tag": "overwrite",
|
||||
},
|
||||
});
|
||||
|
||||
await retryReq(
|
||||
() =>
|
||||
client.dropbox.filesUpload({
|
||||
path: uploadFile,
|
||||
contents: remoteContent,
|
||||
mode: {
|
||||
".tag": "overwrite",
|
||||
},
|
||||
}),
|
||||
fileOrFolderPath
|
||||
);
|
||||
|
||||
// we want to mark that parent folders are created
|
||||
if (foldersCreatedBefore !== undefined) {
|
||||
const dirs = getFolderLevels(uploadFile).map((x) =>
|
||||
@@ -554,9 +669,13 @@ const downloadFromRemoteRaw = async (
|
||||
) => {
|
||||
await client.init();
|
||||
const key = getDropboxPath(fileOrFolderPath, client.remoteBaseDir);
|
||||
const rsp = await client.dropbox.filesDownload({
|
||||
path: key,
|
||||
});
|
||||
const rsp = await retryReq(
|
||||
() =>
|
||||
client.dropbox.filesDownload({
|
||||
path: key,
|
||||
}),
|
||||
fileOrFolderPath
|
||||
);
|
||||
if ((rsp.result as any).fileBlob !== undefined) {
|
||||
// we get a Blob
|
||||
const content = (rsp.result as any).fileBlob as Blob;
|
||||
@@ -631,9 +750,13 @@ export const deleteFromRemote = async (
|
||||
|
||||
await client.init();
|
||||
try {
|
||||
await client.dropbox.filesDeleteV2({
|
||||
path: remoteFileName,
|
||||
});
|
||||
await retryReq(
|
||||
() =>
|
||||
client.dropbox.filesDeleteV2({
|
||||
path: remoteFileName,
|
||||
}),
|
||||
fileOrFolderPath
|
||||
);
|
||||
} catch (err) {
|
||||
console.error("some error while deleting");
|
||||
console.error(err);
|
||||
|
||||
@@ -8,7 +8,7 @@ import type {
|
||||
import cloneDeep from "lodash/cloneDeep";
|
||||
import { request, requestUrl, requireApiVersion, Vault } from "obsidian";
|
||||
import {
|
||||
API_VER_REQURL,
|
||||
VALID_REQURL,
|
||||
COMMAND_CALLBACK_ONEDRIVE,
|
||||
DEFAULT_CONTENT_TYPE,
|
||||
OAUTH2_FORCE_EXPIRE_MILLISECONDS,
|
||||
@@ -444,6 +444,7 @@ export class WrappedOnedriveClient {
|
||||
contentType: "application/json",
|
||||
headers: {
|
||||
Authorization: `Bearer ${await this.authGetter.getAccessToken()}`,
|
||||
"Cache-Control": "no-cache",
|
||||
},
|
||||
})
|
||||
);
|
||||
@@ -484,7 +485,7 @@ export class WrappedOnedriveClient {
|
||||
deleteJson = async (pathFragOrig: string) => {
|
||||
const theUrl = this.buildUrl(pathFragOrig);
|
||||
log.debug(`deleteJson, theUrl=${theUrl}`);
|
||||
if (requireApiVersion(API_VER_REQURL)) {
|
||||
if (VALID_REQURL) {
|
||||
await requestUrl({
|
||||
url: theUrl,
|
||||
method: "DELETE",
|
||||
@@ -508,7 +509,7 @@ export class WrappedOnedriveClient {
|
||||
// TODO:
|
||||
// 20220401: On Android, requestUrl has issue that text becomes base64.
|
||||
// Use fetch everywhere instead!
|
||||
if (false /*requireApiVersion(API_VER_REQURL)*/) {
|
||||
if (false /*VALID_REQURL*/) {
|
||||
await requestUrl({
|
||||
url: theUrl,
|
||||
method: "PUT",
|
||||
@@ -556,7 +557,7 @@ export class WrappedOnedriveClient {
|
||||
// TODO:
|
||||
// 20220401: On Android, requestUrl has issue that text becomes base64.
|
||||
// Use fetch everywhere instead!
|
||||
if (false /*requireApiVersion(API_VER_REQURL)*/) {
|
||||
if (false /*VALID_REQURL*/) {
|
||||
const res = await requestUrl({
|
||||
url: theUrl,
|
||||
method: "PUT",
|
||||
@@ -565,6 +566,7 @@ export class WrappedOnedriveClient {
|
||||
headers: {
|
||||
// no "Content-Length" allowed here
|
||||
"Content-Range": `bytes ${rangeStart}-${rangeEnd - 1}/${size}`,
|
||||
/* "Cache-Control": "no-cache", not allowed here!!! */
|
||||
},
|
||||
});
|
||||
return res.json as DriveItem | UploadSession;
|
||||
@@ -576,6 +578,7 @@ export class WrappedOnedriveClient {
|
||||
"Content-Length": `${rangeEnd - rangeStart}`,
|
||||
"Content-Range": `bytes ${rangeStart}-${rangeEnd - 1}/${size}`,
|
||||
"Content-Type": DEFAULT_CONTENT_TYPE,
|
||||
/* "Cache-Control": "no-cache", not allowed here!!! */
|
||||
},
|
||||
});
|
||||
return (await res.json()) as DriveItem | UploadSession;
|
||||
@@ -808,11 +811,19 @@ const downloadFromRemoteRaw = async (
|
||||
`${key}?$select=@microsoft.graph.downloadUrl`
|
||||
);
|
||||
const downloadUrl: string = rsp["@microsoft.graph.downloadUrl"];
|
||||
if (requireApiVersion(API_VER_REQURL)) {
|
||||
const content = (await requestUrl({ url: downloadUrl })).arrayBuffer;
|
||||
if (VALID_REQURL) {
|
||||
const content = (
|
||||
await requestUrl({
|
||||
url: downloadUrl,
|
||||
headers: { "Cache-Control": "no-cache" },
|
||||
})
|
||||
).arrayBuffer;
|
||||
return content;
|
||||
} else {
|
||||
const content = await (await fetch(downloadUrl)).arrayBuffer();
|
||||
const content = await // cannot set no-cache here, will have cors error
|
||||
(
|
||||
await fetch(downloadUrl)
|
||||
).arrayBuffer();
|
||||
return content;
|
||||
}
|
||||
};
|
||||
|
||||
+37
-10
@@ -24,6 +24,7 @@ import { Buffer } from "buffer";
|
||||
import * as mime from "mime-types";
|
||||
import { Vault, requestUrl, RequestUrlParam } from "obsidian";
|
||||
import { Readable } from "stream";
|
||||
import AggregateError from "aggregate-error";
|
||||
import {
|
||||
DEFAULT_CONTENT_TYPE,
|
||||
RemoteItem,
|
||||
@@ -83,11 +84,14 @@ class ObsHttpHandler extends FetchHttpHandler {
|
||||
const body =
|
||||
method === "GET" || method === "HEAD" ? undefined : request.body;
|
||||
|
||||
const transformedHeaders = { ...request.headers };
|
||||
delete transformedHeaders["host"];
|
||||
delete transformedHeaders["Host"];
|
||||
delete transformedHeaders["content-length"];
|
||||
delete transformedHeaders["Content-Length"];
|
||||
const transformedHeaders: Record<string, string> = {};
|
||||
for (const key of Object.keys(request.headers)) {
|
||||
const keyLower = key.toLowerCase();
|
||||
if (keyLower === "host" || keyLower === "content-length") {
|
||||
continue;
|
||||
}
|
||||
transformedHeaders[keyLower] = request.headers[key];
|
||||
}
|
||||
|
||||
let contentType: string = undefined;
|
||||
if (transformedHeaders["content-type"] !== undefined) {
|
||||
@@ -192,8 +196,10 @@ export const getS3Client = (s3Config: S3Config) => {
|
||||
endpoint = `https://${endpoint}`;
|
||||
}
|
||||
|
||||
let s3Client: S3Client;
|
||||
|
||||
if (VALID_REQURL && s3Config.bypassCorsLocally) {
|
||||
const s3Client = new S3Client({
|
||||
s3Client = new S3Client({
|
||||
region: s3Config.s3Region,
|
||||
endpoint: endpoint,
|
||||
forcePathStyle: s3Config.forcePathStyle,
|
||||
@@ -203,9 +209,8 @@ export const getS3Client = (s3Config: S3Config) => {
|
||||
},
|
||||
requestHandler: new ObsHttpHandler(),
|
||||
});
|
||||
return s3Client;
|
||||
} else {
|
||||
const s3Client = new S3Client({
|
||||
s3Client = new S3Client({
|
||||
region: s3Config.s3Region,
|
||||
endpoint: endpoint,
|
||||
forcePathStyle: s3Config.forcePathStyle,
|
||||
@@ -214,8 +219,19 @@ export const getS3Client = (s3Config: S3Config) => {
|
||||
secretAccessKey: s3Config.s3SecretAccessKey,
|
||||
},
|
||||
});
|
||||
return s3Client;
|
||||
}
|
||||
|
||||
s3Client.middlewareStack.add(
|
||||
(next, context) => (args) => {
|
||||
(args.request as any).headers["cache-control"] = "no-cache";
|
||||
return next(args);
|
||||
},
|
||||
{
|
||||
step: "build",
|
||||
}
|
||||
);
|
||||
|
||||
return s3Client;
|
||||
};
|
||||
|
||||
export const getRemoteMeta = async (
|
||||
@@ -522,8 +538,19 @@ export const checkConnectivity = async (
|
||||
} catch (err) {
|
||||
log.debug(err);
|
||||
if (callbackFunc !== undefined) {
|
||||
callbackFunc(err);
|
||||
if (s3Config.s3Endpoint.contains(s3Config.s3BucketName)) {
|
||||
const err2 = new AggregateError([
|
||||
err,
|
||||
new Error(
|
||||
"Maybe you've included the bucket name inside the endpoint setting. Please remove the bucket name and try again."
|
||||
),
|
||||
]);
|
||||
callbackFunc(err2);
|
||||
} else {
|
||||
callbackFunc(err);
|
||||
}
|
||||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
};
|
||||
|
||||
@@ -183,6 +183,9 @@ export class WrappedWebdavClient {
|
||||
|
||||
init = async () => {
|
||||
// init client if not inited
|
||||
const headers = {
|
||||
"Cache-Control": "no-cache",
|
||||
};
|
||||
if (this.client === undefined) {
|
||||
if (
|
||||
this.webdavConfig.username !== "" &&
|
||||
@@ -191,6 +194,7 @@ export class WrappedWebdavClient {
|
||||
this.client = createClient(this.webdavConfig.address, {
|
||||
username: this.webdavConfig.username,
|
||||
password: this.webdavConfig.password,
|
||||
headers: headers,
|
||||
authType:
|
||||
this.webdavConfig.authType === "digest"
|
||||
? AuthType.Digest
|
||||
@@ -198,7 +202,9 @@ export class WrappedWebdavClient {
|
||||
});
|
||||
} else {
|
||||
log.info("no password");
|
||||
this.client = createClient(this.webdavConfig.address);
|
||||
this.client = createClient(this.webdavConfig.address, {
|
||||
headers: headers,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
+78
-32
@@ -7,6 +7,8 @@ import {
|
||||
Platform,
|
||||
requireApiVersion,
|
||||
} from "obsidian";
|
||||
import type { TextComponent } from "obsidian";
|
||||
import { createElement, Eye, EyeOff } from "lucide";
|
||||
import {
|
||||
API_VER_REQURL,
|
||||
DEFAULT_DEBUG_FOLDER,
|
||||
@@ -627,8 +629,34 @@ class ExportSettingsQrCodeModal extends Modal {
|
||||
}
|
||||
}
|
||||
|
||||
const getEyesElements = () => {
|
||||
const eyeEl = createElement(Eye);
|
||||
const eyeOffEl = createElement(EyeOff);
|
||||
return {
|
||||
eye: eyeEl.outerHTML,
|
||||
eyeOff: eyeOffEl.outerHTML,
|
||||
};
|
||||
};
|
||||
|
||||
const wrapTextWithPasswordHide = (text: TextComponent) => {
|
||||
const { eye, eyeOff } = getEyesElements();
|
||||
const hider = text.inputEl.insertAdjacentElement("afterend", createSpan());
|
||||
// the init type of hider is "hidden" === eyeOff === password
|
||||
hider.innerHTML = eyeOff;
|
||||
hider.addEventListener("click", (e) => {
|
||||
const isText = text.inputEl.getAttribute("type") === "text";
|
||||
hider.innerHTML = isText ? eyeOff : eye;
|
||||
text.inputEl.setAttribute("type", isText ? "password" : "text");
|
||||
text.inputEl.focus();
|
||||
});
|
||||
|
||||
// the init type of text el is password
|
||||
text.inputEl.setAttribute("type", "password");
|
||||
return text;
|
||||
};
|
||||
|
||||
export class RemotelySaveSettingTab extends PluginSettingTab {
|
||||
plugin: RemotelySavePlugin;
|
||||
readonly plugin: RemotelySavePlugin;
|
||||
|
||||
constructor(app: App, plugin: RemotelySavePlugin) {
|
||||
super(app, plugin);
|
||||
@@ -658,14 +686,15 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
|
||||
new Setting(passwordDiv)
|
||||
.setName(t("settings_password"))
|
||||
.setDesc(t("settings_password_desc"))
|
||||
.addText((text) =>
|
||||
.addText((text) => {
|
||||
wrapTextWithPasswordHide(text);
|
||||
text
|
||||
.setPlaceholder("")
|
||||
.setValue(`${this.plugin.settings.password}`)
|
||||
.onChange(async (value) => {
|
||||
newPassword = value.trim();
|
||||
})
|
||||
)
|
||||
});
|
||||
})
|
||||
.addButton(async (button) => {
|
||||
button.setButtonText(t("confirm"));
|
||||
button.onClick(async () => {
|
||||
@@ -717,6 +746,10 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
|
||||
.setDesc(t("settings_runoncestartup_desc"))
|
||||
.addDropdown((dropdown) => {
|
||||
dropdown.addOption("-1", t("settings_runoncestartup_notset"));
|
||||
dropdown.addOption(
|
||||
`${1000 * 1 * 1}`,
|
||||
t("settings_runoncestartup_1sec")
|
||||
);
|
||||
dropdown.addOption(
|
||||
`${1000 * 10 * 1}`,
|
||||
t("settings_runoncestartup_10sec")
|
||||
@@ -734,6 +767,25 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
|
||||
});
|
||||
});
|
||||
|
||||
const skipLargeFilesDiv = generalDiv.createEl("div");
|
||||
new Setting(skipLargeFilesDiv)
|
||||
.setName(t("settings_skiplargefiles"))
|
||||
.setDesc(t("settings_skiplargefiles_desc"))
|
||||
.addDropdown((dropdown) => {
|
||||
dropdown.addOption("-1", t("settings_skiplargefiles_notset"));
|
||||
|
||||
const mbs = [1, 5, 10, 50, 100, 500, 1000];
|
||||
for (const mb of mbs) {
|
||||
dropdown.addOption(`${mb * 1000 * 1000}`, `${mb} MB`);
|
||||
}
|
||||
dropdown
|
||||
.setValue(`${this.plugin.settings.skipSizeLargerThan}`)
|
||||
.onChange(async (val) => {
|
||||
this.plugin.settings.skipSizeLargerThan = parseInt(val);
|
||||
await this.plugin.saveSettings();
|
||||
});
|
||||
});
|
||||
|
||||
//////////////////////////////////////////////////
|
||||
// below for general chooser (part 1/2)
|
||||
//////////////////////////////////////////////////
|
||||
@@ -816,29 +868,31 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
|
||||
|
||||
new Setting(s3Div)
|
||||
.setName(t("settings_s3_accesskeyid"))
|
||||
.setDesc(t("settings_s3_accesskeyid"))
|
||||
.addText((text) =>
|
||||
.setDesc(t("settings_s3_accesskeyid_desc"))
|
||||
.addText((text) => {
|
||||
wrapTextWithPasswordHide(text);
|
||||
text
|
||||
.setPlaceholder("")
|
||||
.setValue(`${this.plugin.settings.s3.s3AccessKeyID}`)
|
||||
.onChange(async (value) => {
|
||||
this.plugin.settings.s3.s3AccessKeyID = value.trim();
|
||||
await this.plugin.saveSettings();
|
||||
})
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
new Setting(s3Div)
|
||||
.setName(t("settings_s3_secretaccesskey"))
|
||||
.setDesc(t("settings_s3_secretaccesskey"))
|
||||
.addText((text) =>
|
||||
.setDesc(t("settings_s3_secretaccesskey_desc"))
|
||||
.addText((text) => {
|
||||
wrapTextWithPasswordHide(text);
|
||||
text
|
||||
.setPlaceholder("")
|
||||
.setValue(`${this.plugin.settings.s3.s3SecretAccessKey}`)
|
||||
.onChange(async (value) => {
|
||||
this.plugin.settings.s3.s3SecretAccessKey = value.trim();
|
||||
await this.plugin.saveSettings();
|
||||
})
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
new Setting(s3Div)
|
||||
.setName(t("settings_s3_bucketname"))
|
||||
@@ -877,11 +931,7 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
|
||||
if (VALID_REQURL) {
|
||||
new Setting(s3Div)
|
||||
.setName(t("settings_s3_bypasscorslocally"))
|
||||
.setDesc(
|
||||
t("settings_s3_bypasscorslocally_desc", {
|
||||
ver: API_VER_REQURL,
|
||||
})
|
||||
)
|
||||
.setDesc(t("settings_s3_bypasscorslocally_desc"))
|
||||
.addDropdown((dropdown) => {
|
||||
dropdown
|
||||
.addOption("disable", t("disable"))
|
||||
@@ -1297,15 +1347,9 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
|
||||
});
|
||||
|
||||
if (!VALID_REQURL) {
|
||||
if (Platform.isAndroidApp) {
|
||||
webdavDiv.createEl("p", {
|
||||
text: t("settings_webdav_cors_android"),
|
||||
});
|
||||
} else {
|
||||
webdavDiv.createEl("p", {
|
||||
text: t("settings_webdav_cors_otheros"),
|
||||
});
|
||||
}
|
||||
webdavDiv.createEl("p", {
|
||||
text: t("settings_webdav_cors_os"),
|
||||
});
|
||||
|
||||
webdavDiv.createEl("p", {
|
||||
text: t("settings_webdav_cors"),
|
||||
@@ -1346,7 +1390,8 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
|
||||
new Setting(webdavDiv)
|
||||
.setName(t("settings_webdav_user"))
|
||||
.setDesc(t("settings_webdav_user_desc"))
|
||||
.addText((text) =>
|
||||
.addText((text) => {
|
||||
wrapTextWithPasswordHide(text);
|
||||
text
|
||||
.setPlaceholder("")
|
||||
.setValue(this.plugin.settings.webdav.username)
|
||||
@@ -1359,13 +1404,14 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
|
||||
this.plugin.settings.webdav.depth = "auto_unknown";
|
||||
}
|
||||
await this.plugin.saveSettings();
|
||||
})
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
new Setting(webdavDiv)
|
||||
.setName(t("settings_webdav_password"))
|
||||
.setDesc(t("settings_webdav_password_desc"))
|
||||
.addText((text) =>
|
||||
.addText((text) => {
|
||||
wrapTextWithPasswordHide(text);
|
||||
text
|
||||
.setPlaceholder("")
|
||||
.setValue(this.plugin.settings.webdav.password)
|
||||
@@ -1378,8 +1424,8 @@ export class RemotelySaveSettingTab extends PluginSettingTab {
|
||||
this.plugin.settings.webdav.depth = "auto_unknown";
|
||||
}
|
||||
await this.plugin.saveSettings();
|
||||
})
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
new Setting(webdavDiv)
|
||||
.setName(t("settings_webdav_auth"))
|
||||
|
||||
+277
-36
@@ -7,17 +7,19 @@ import {
|
||||
} from "obsidian";
|
||||
import AggregateError from "aggregate-error";
|
||||
import PQueue from "p-queue";
|
||||
import {
|
||||
import type {
|
||||
RemoteItem,
|
||||
SUPPORTED_SERVICES_TYPE,
|
||||
SyncTriggerSourceType,
|
||||
DecisionType,
|
||||
FileOrFolderMixedState,
|
||||
API_VER_STAT_FOLDER,
|
||||
SUPPORTED_SERVICES_TYPE,
|
||||
} from "./baseTypes";
|
||||
import { API_VER_STAT_FOLDER } from "./baseTypes";
|
||||
import {
|
||||
decryptBase32ToString,
|
||||
decryptBase64urlToString,
|
||||
encryptStringToBase64url,
|
||||
getSizeFromOrigToEnc,
|
||||
MAGIC_ENCRYPTED_PREFIX_BASE32,
|
||||
MAGIC_ENCRYPTED_PREFIX_BASE64URL,
|
||||
} from "./encrypt";
|
||||
@@ -65,6 +67,7 @@ export type SyncStatusType =
|
||||
export interface SyncPlanType {
|
||||
ts: number;
|
||||
tsFmt?: string;
|
||||
syncTriggerSource?: SyncTriggerSourceType;
|
||||
remoteType: SUPPORTED_SERVICES_TYPE;
|
||||
mixedStates: Record<string, FileOrFolderMixedState>;
|
||||
}
|
||||
@@ -215,22 +218,29 @@ export const parseRemoteItems = async (
|
||||
if (backwardMapping !== undefined) {
|
||||
key = backwardMapping.localKey;
|
||||
const mtimeRemote = backwardMapping.localMtime || entry.lastModified;
|
||||
|
||||
// the backwardMapping.localSize is the file BEFORE encryption
|
||||
// we want to split two sizes for comparation later
|
||||
|
||||
r = {
|
||||
key: key,
|
||||
existRemote: true,
|
||||
mtimeRemote: mtimeRemote,
|
||||
mtimeRemoteFmt: unixTimeToStr(mtimeRemote),
|
||||
sizeRemote: backwardMapping.localSize || entry.size,
|
||||
sizeRemote: backwardMapping.localSize,
|
||||
sizeRemoteEnc: password === "" ? undefined : entry.size,
|
||||
remoteEncryptedKey: remoteEncryptedKey,
|
||||
changeRemoteMtimeUsingMapping: true,
|
||||
};
|
||||
} else {
|
||||
// do not have backwardMapping
|
||||
r = {
|
||||
key: key,
|
||||
existRemote: true,
|
||||
mtimeRemote: entry.lastModified,
|
||||
mtimeRemoteFmt: unixTimeToStr(entry.lastModified),
|
||||
sizeRemote: entry.size,
|
||||
sizeRemote: password === "" ? entry.size : undefined,
|
||||
sizeRemoteEnc: password === "" ? undefined : entry.size,
|
||||
remoteEncryptedKey: remoteEncryptedKey,
|
||||
changeRemoteMtimeUsingMapping: false,
|
||||
};
|
||||
@@ -303,7 +313,8 @@ const ensembleMixedStates = async (
|
||||
localFileHistory: FileFolderHistoryRecord[],
|
||||
syncConfigDir: boolean,
|
||||
configDir: string,
|
||||
syncUnderscoreItems: boolean
|
||||
syncUnderscoreItems: boolean,
|
||||
password: string
|
||||
) => {
|
||||
const results = {} as Record<string, FileOrFolderMixedState>;
|
||||
|
||||
@@ -332,6 +343,8 @@ const ensembleMixedStates = async (
|
||||
mtimeLocal: mtimeLocal,
|
||||
mtimeLocalFmt: unixTimeToStr(mtimeLocal),
|
||||
sizeLocal: entry.stat.size,
|
||||
sizeLocalEnc:
|
||||
password === "" ? undefined : getSizeFromOrigToEnc(entry.stat.size),
|
||||
};
|
||||
} else if (entry instanceof TFolder) {
|
||||
key = `${entry.path}/`;
|
||||
@@ -341,6 +354,7 @@ const ensembleMixedStates = async (
|
||||
mtimeLocal: undefined,
|
||||
mtimeLocalFmt: undefined,
|
||||
sizeLocal: 0,
|
||||
sizeLocalEnc: password === "" ? undefined : getSizeFromOrigToEnc(0),
|
||||
};
|
||||
} else {
|
||||
throw Error(`unexpected ${entry}`);
|
||||
@@ -356,6 +370,7 @@ const ensembleMixedStates = async (
|
||||
results[key].mtimeLocal = r.mtimeLocal;
|
||||
results[key].mtimeLocalFmt = r.mtimeLocalFmt;
|
||||
results[key].sizeLocal = r.sizeLocal;
|
||||
results[key].sizeLocalEnc = r.sizeLocalEnc;
|
||||
} else {
|
||||
results[key] = r;
|
||||
results[key].existRemote = false;
|
||||
@@ -372,6 +387,8 @@ const ensembleMixedStates = async (
|
||||
mtimeLocal: mtimeLocal,
|
||||
mtimeLocalFmt: unixTimeToStr(mtimeLocal),
|
||||
sizeLocal: entry.size,
|
||||
sizeLocalEnc:
|
||||
password === "" ? undefined : getSizeFromOrigToEnc(entry.size),
|
||||
};
|
||||
|
||||
if (results.hasOwnProperty(key)) {
|
||||
@@ -380,6 +397,7 @@ const ensembleMixedStates = async (
|
||||
results[key].mtimeLocal = r.mtimeLocal;
|
||||
results[key].mtimeLocalFmt = r.mtimeLocalFmt;
|
||||
results[key].sizeLocal = r.sizeLocal;
|
||||
results[key].sizeLocalEnc = r.sizeLocalEnc;
|
||||
} else {
|
||||
results[key] = r;
|
||||
results[key].existRemote = false;
|
||||
@@ -459,9 +477,15 @@ const ensembleMixedStates = async (
|
||||
results[key].changeLocalMtimeUsingMapping =
|
||||
r.changeLocalMtimeUsingMapping;
|
||||
} else {
|
||||
results[key] = r;
|
||||
results[key].existLocal = false; // we have already checked local
|
||||
results[key].existRemote = false; // we have already checked remote
|
||||
// So, the file doesn't exist,
|
||||
// except that it existed in the "renamed to" history records.
|
||||
// Most likely because that the user deleted the file while Obsidian was closed,
|
||||
// so Obsidian could not track the deletions.
|
||||
// We are not sure how to deal with this, so do not generate anything here!
|
||||
// // // The following 3 lines are of old logic, and have been removed:
|
||||
// // results[key] = r;
|
||||
// // results[key].existLocal = false; // we have already checked local
|
||||
// // results[key].existRemote = false; // we have already checked remote
|
||||
}
|
||||
} else {
|
||||
throw Error(
|
||||
@@ -476,6 +500,7 @@ const ensembleMixedStates = async (
|
||||
const assignOperationToFileInplace = (
|
||||
origRecord: FileOrFolderMixedState,
|
||||
keptFolder: Set<string>,
|
||||
skipSizeLargerThan: number,
|
||||
password: string = ""
|
||||
) => {
|
||||
let r = origRecord;
|
||||
@@ -518,6 +543,18 @@ const assignOperationToFileInplace = (
|
||||
);
|
||||
}
|
||||
|
||||
if (
|
||||
(r.existLocal && password !== "" && r.sizeLocalEnc === undefined) ||
|
||||
(r.existRemote && password !== "" && r.sizeRemoteEnc === undefined)
|
||||
) {
|
||||
throw new Error(
|
||||
`Error: No encryption sizes: ${JSON.stringify(r, null, 2)}`
|
||||
);
|
||||
}
|
||||
|
||||
const sizeLocalComp = password === "" ? r.sizeLocal : r.sizeLocalEnc;
|
||||
const sizeRemoteComp = password === "" ? r.sizeRemote : r.sizeRemoteEnc;
|
||||
|
||||
// 1. mtimeLocal
|
||||
if (r.existLocal) {
|
||||
const mtimeRemote = r.existRemote ? r.mtimeRemote : -1;
|
||||
@@ -528,26 +565,79 @@ const assignOperationToFileInplace = (
|
||||
r.mtimeLocal >= deltimeLocal &&
|
||||
r.mtimeLocal >= deltimeRemote
|
||||
) {
|
||||
if (sizeLocalComp === undefined) {
|
||||
throw new Error(
|
||||
`Error: no local size but has local mtime: ${JSON.stringify(
|
||||
r,
|
||||
null,
|
||||
2
|
||||
)}`
|
||||
);
|
||||
}
|
||||
if (r.mtimeLocal === r.mtimeRemote) {
|
||||
// mtime the same
|
||||
if (password === "") {
|
||||
// no password, we can also compare the sizes!
|
||||
if (r.sizeLocal === r.sizeRemote) {
|
||||
r.decision = "skipUploading";
|
||||
r.decisionBranch = 1;
|
||||
} else {
|
||||
// local and remote both exist and mtimes are the same
|
||||
if (sizeLocalComp === sizeRemoteComp) {
|
||||
// do not need to consider skipSizeLargerThan in this case
|
||||
r.decision = "skipUploading";
|
||||
r.decisionBranch = 1;
|
||||
} else {
|
||||
if (skipSizeLargerThan <= 0) {
|
||||
r.decision = "uploadLocalToRemote";
|
||||
r.decisionBranch = 2;
|
||||
} else {
|
||||
// limit the sizes
|
||||
if (sizeLocalComp <= skipSizeLargerThan) {
|
||||
if (sizeRemoteComp <= skipSizeLargerThan) {
|
||||
r.decision = "uploadLocalToRemote";
|
||||
r.decisionBranch = 18;
|
||||
} else {
|
||||
r.decision = "errorRemoteTooLargeConflictLocal";
|
||||
r.decisionBranch = 19;
|
||||
}
|
||||
} else {
|
||||
if (sizeRemoteComp <= skipSizeLargerThan) {
|
||||
r.decision = "errorLocalTooLargeConflictRemote";
|
||||
r.decisionBranch = 20;
|
||||
} else {
|
||||
r.decision = "skipUploadingTooLarge";
|
||||
r.decisionBranch = 21;
|
||||
}
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// we have password, then the sizes are always unequal
|
||||
// we can only rely on mtime
|
||||
r.decision = "skipUploading";
|
||||
r.decisionBranch = 3;
|
||||
}
|
||||
} else {
|
||||
r.decision = "uploadLocalToRemote";
|
||||
r.decisionBranch = 4;
|
||||
// we have local laregest mtime,
|
||||
// and the remote not existing or smaller mtime
|
||||
if (skipSizeLargerThan <= 0) {
|
||||
// no need to consider sizes
|
||||
r.decision = "uploadLocalToRemote";
|
||||
r.decisionBranch = 4;
|
||||
} else {
|
||||
// need to consider sizes
|
||||
if (sizeLocalComp <= skipSizeLargerThan) {
|
||||
if (sizeRemoteComp === undefined) {
|
||||
r.decision = "uploadLocalToRemote";
|
||||
r.decisionBranch = 22;
|
||||
} else if (sizeRemoteComp <= skipSizeLargerThan) {
|
||||
r.decision = "uploadLocalToRemote";
|
||||
r.decisionBranch = 23;
|
||||
} else {
|
||||
r.decision = "errorRemoteTooLargeConflictLocal";
|
||||
r.decisionBranch = 24;
|
||||
}
|
||||
} else {
|
||||
if (sizeRemoteComp === undefined) {
|
||||
r.decision = "skipUploadingTooLarge";
|
||||
r.decisionBranch = 25;
|
||||
} else if (sizeRemoteComp <= skipSizeLargerThan) {
|
||||
r.decision = "errorLocalTooLargeConflictRemote";
|
||||
r.decisionBranch = 26;
|
||||
} else {
|
||||
r.decision = "skipUploadingTooLarge";
|
||||
r.decisionBranch = 27;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
keptFolder.add(getParentFolder(r.key));
|
||||
return r;
|
||||
@@ -564,8 +654,49 @@ const assignOperationToFileInplace = (
|
||||
r.mtimeRemote >= deltimeLocal &&
|
||||
r.mtimeRemote >= deltimeRemote
|
||||
) {
|
||||
r.decision = "downloadRemoteToLocal";
|
||||
r.decisionBranch = 5;
|
||||
// we have remote laregest mtime,
|
||||
// and the local not existing or smaller mtime
|
||||
if (sizeRemoteComp === undefined) {
|
||||
throw new Error(
|
||||
`Error: no remote size but has remote mtime: ${JSON.stringify(
|
||||
r,
|
||||
null,
|
||||
2
|
||||
)}`
|
||||
);
|
||||
}
|
||||
|
||||
if (skipSizeLargerThan <= 0) {
|
||||
// no need to consider sizes
|
||||
r.decision = "downloadRemoteToLocal";
|
||||
r.decisionBranch = 5;
|
||||
} else {
|
||||
// need to consider sizes
|
||||
if (sizeRemoteComp <= skipSizeLargerThan) {
|
||||
if (sizeLocalComp === undefined) {
|
||||
r.decision = "downloadRemoteToLocal";
|
||||
r.decisionBranch = 28;
|
||||
} else if (sizeLocalComp <= skipSizeLargerThan) {
|
||||
r.decision = "downloadRemoteToLocal";
|
||||
r.decisionBranch = 29;
|
||||
} else {
|
||||
r.decision = "errorLocalTooLargeConflictRemote";
|
||||
r.decisionBranch = 30;
|
||||
}
|
||||
} else {
|
||||
if (sizeLocalComp === undefined) {
|
||||
r.decision = "skipDownloadingTooLarge";
|
||||
r.decisionBranch = 31;
|
||||
} else if (sizeLocalComp <= skipSizeLargerThan) {
|
||||
r.decision = "errorRemoteTooLargeConflictLocal";
|
||||
r.decisionBranch = 32;
|
||||
} else {
|
||||
r.decision = "skipDownloadingTooLarge";
|
||||
r.decisionBranch = 33;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
keptFolder.add(getParentFolder(r.key));
|
||||
return r;
|
||||
}
|
||||
@@ -581,10 +712,44 @@ const assignOperationToFileInplace = (
|
||||
r.deltimeLocal >= mtimeRemote &&
|
||||
r.deltimeLocal >= deltimeRemote
|
||||
) {
|
||||
r.decision = "uploadLocalDelHistToRemote";
|
||||
r.decisionBranch = 6;
|
||||
if (r.existLocal || r.existRemote) {
|
||||
// actual deletion would happen
|
||||
if (skipSizeLargerThan <= 0) {
|
||||
r.decision = "uploadLocalDelHistToRemote";
|
||||
r.decisionBranch = 6;
|
||||
if (r.existLocal || r.existRemote) {
|
||||
// actual deletion would happen
|
||||
}
|
||||
} else {
|
||||
const localTooLargeToDelete =
|
||||
r.existLocal && sizeLocalComp > skipSizeLargerThan;
|
||||
const remoteTooLargeToDelete =
|
||||
r.existRemote && sizeRemoteComp > skipSizeLargerThan;
|
||||
if (localTooLargeToDelete) {
|
||||
if (remoteTooLargeToDelete) {
|
||||
r.decision = "skipUsingLocalDelTooLarge";
|
||||
r.decisionBranch = 34;
|
||||
} else {
|
||||
if (r.existRemote) {
|
||||
r.decision = "errorLocalTooLargeConflictRemote";
|
||||
r.decisionBranch = 35;
|
||||
} else {
|
||||
r.decision = "skipUsingLocalDelTooLarge";
|
||||
r.decisionBranch = 36;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
if (remoteTooLargeToDelete) {
|
||||
if (r.existLocal) {
|
||||
r.decision = "errorLocalTooLargeConflictRemote";
|
||||
r.decisionBranch = 37;
|
||||
} else {
|
||||
r.decision = "skipUsingLocalDelTooLarge";
|
||||
r.decisionBranch = 38;
|
||||
}
|
||||
} else {
|
||||
r.decision = "uploadLocalDelHistToRemote";
|
||||
r.decisionBranch = 39;
|
||||
}
|
||||
}
|
||||
}
|
||||
return r;
|
||||
}
|
||||
@@ -600,10 +765,44 @@ const assignOperationToFileInplace = (
|
||||
r.deltimeRemote >= mtimeRemote &&
|
||||
r.deltimeRemote >= deltimeLocal
|
||||
) {
|
||||
r.decision = "keepRemoteDelHist";
|
||||
r.decisionBranch = 7;
|
||||
if (r.existLocal || r.existRemote) {
|
||||
// actual deletion would happen
|
||||
if (skipSizeLargerThan <= 0) {
|
||||
r.decision = "keepRemoteDelHist";
|
||||
r.decisionBranch = 7;
|
||||
if (r.existLocal || r.existRemote) {
|
||||
// actual deletion would happen
|
||||
}
|
||||
} else {
|
||||
const localTooLargeToDelete =
|
||||
r.existLocal && sizeLocalComp > skipSizeLargerThan;
|
||||
const remoteTooLargeToDelete =
|
||||
r.existRemote && sizeRemoteComp > skipSizeLargerThan;
|
||||
if (localTooLargeToDelete) {
|
||||
if (remoteTooLargeToDelete) {
|
||||
r.decision = "skipUsingRemoteDelTooLarge";
|
||||
r.decisionBranch = 40;
|
||||
} else {
|
||||
if (r.existRemote) {
|
||||
r.decision = "errorLocalTooLargeConflictRemote";
|
||||
r.decisionBranch = 41;
|
||||
} else {
|
||||
r.decision = "skipUsingRemoteDelTooLarge";
|
||||
r.decisionBranch = 42;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
if (remoteTooLargeToDelete) {
|
||||
if (r.existLocal) {
|
||||
r.decision = "errorLocalTooLargeConflictRemote";
|
||||
r.decisionBranch = 43;
|
||||
} else {
|
||||
r.decision = "skipUsingRemoteDelTooLarge";
|
||||
r.decisionBranch = 44;
|
||||
}
|
||||
} else {
|
||||
r.decision = "keepRemoteDelHist";
|
||||
r.decisionBranch = 45;
|
||||
}
|
||||
}
|
||||
}
|
||||
return r;
|
||||
}
|
||||
@@ -738,6 +937,10 @@ const DELETION_DECISIONS: Set<DecisionType> = new Set([
|
||||
"uploadLocalDelHistToRemoteFolder",
|
||||
"keepRemoteDelHistFolder",
|
||||
]);
|
||||
const SIZES_GO_WRONG_DECISIONS: Set<DecisionType> = new Set([
|
||||
"errorLocalTooLargeConflictRemote",
|
||||
"errorRemoteTooLargeConflictLocal",
|
||||
]);
|
||||
|
||||
export const getSyncPlan = async (
|
||||
remoteStates: FileOrFolderMixedState[],
|
||||
@@ -746,10 +949,12 @@ export const getSyncPlan = async (
|
||||
remoteDeleteHistory: DeletionOnRemote[],
|
||||
localFileHistory: FileFolderHistoryRecord[],
|
||||
remoteType: SUPPORTED_SERVICES_TYPE,
|
||||
triggerSource: SyncTriggerSourceType,
|
||||
vault: Vault,
|
||||
syncConfigDir: boolean,
|
||||
configDir: string,
|
||||
syncUnderscoreItems: boolean,
|
||||
skipSizeLargerThan: number,
|
||||
password: string = ""
|
||||
) => {
|
||||
const mixedStates = await ensembleMixedStates(
|
||||
@@ -760,13 +965,15 @@ export const getSyncPlan = async (
|
||||
localFileHistory,
|
||||
syncConfigDir,
|
||||
configDir,
|
||||
syncUnderscoreItems
|
||||
syncUnderscoreItems,
|
||||
password
|
||||
);
|
||||
|
||||
const sortedKeys = Object.keys(mixedStates).sort(
|
||||
(k1, k2) => k2.length - k1.length
|
||||
);
|
||||
|
||||
const sizesGoWrong: FileOrFolderMixedState[] = [];
|
||||
const deletions: DeletionOnRemote[] = [];
|
||||
|
||||
const keptFolder = new Set<string>();
|
||||
@@ -782,7 +989,16 @@ export const getSyncPlan = async (
|
||||
} else {
|
||||
// get all operations of files
|
||||
// and at the same time get some helper info for folders
|
||||
assignOperationToFileInplace(val, keptFolder, password);
|
||||
assignOperationToFileInplace(
|
||||
val,
|
||||
keptFolder,
|
||||
skipSizeLargerThan,
|
||||
password
|
||||
);
|
||||
}
|
||||
|
||||
if (SIZES_GO_WRONG_DECISIONS.has(val.decision)) {
|
||||
sizesGoWrong.push(val);
|
||||
}
|
||||
|
||||
if (DELETION_DECISIONS.has(val.decision)) {
|
||||
@@ -818,12 +1034,14 @@ export const getSyncPlan = async (
|
||||
ts: currTs,
|
||||
tsFmt: currTsFmt,
|
||||
remoteType: remoteType,
|
||||
syncTriggerSource: triggerSource,
|
||||
mixedStates: mixedStates,
|
||||
} as SyncPlanType;
|
||||
return {
|
||||
plan: plan,
|
||||
sortedKeys: sortedKeys,
|
||||
deletions: deletions,
|
||||
sizesGoWrong: sizesGoWrong,
|
||||
};
|
||||
};
|
||||
|
||||
@@ -1005,6 +1223,14 @@ const dispatchOperationToActual = async (
|
||||
await clearDeleteRenameHistoryOfKeyAndVault(db, r.key, vaultRandomID);
|
||||
} else if (r.decision === "skipFolder") {
|
||||
// do nothing!
|
||||
} else if (r.decision === "skipUploadingTooLarge") {
|
||||
// do nothing!
|
||||
} else if (r.decision === "skipDownloadingTooLarge") {
|
||||
// do nothing!
|
||||
} else if (r.decision === "skipUsingLocalDelTooLarge") {
|
||||
// do nothing!
|
||||
} else if (r.decision === "skipUsingRemoteDelTooLarge") {
|
||||
// do nothing!
|
||||
} else {
|
||||
throw Error(`unknown decision in ${JSON.stringify(r)}`);
|
||||
}
|
||||
@@ -1023,7 +1249,14 @@ const splitThreeSteps = (syncPlan: SyncPlanType, sortedKeys: string[]) => {
|
||||
const key = sortedKeys[i];
|
||||
const val: FileOrFolderMixedState = Object.assign({}, mixedStates[key]); // copy to avoid issue
|
||||
|
||||
if (val.decision === "skipFolder" || val.decision === "skipUploading") {
|
||||
if (
|
||||
val.decision === "skipFolder" ||
|
||||
val.decision === "skipUploading" ||
|
||||
val.decision === "skipDownloadingTooLarge" ||
|
||||
val.decision === "skipUploadingTooLarge" ||
|
||||
val.decision === "skipUsingLocalDelTooLarge" ||
|
||||
val.decision === "skipUsingRemoteDelTooLarge"
|
||||
) {
|
||||
// pass
|
||||
} else if (val.decision === "createFolder") {
|
||||
const level = atWhichLevel(key);
|
||||
@@ -1083,15 +1316,23 @@ export const doActualSync = async (
|
||||
sortedKeys: string[],
|
||||
metadataFile: FileOrFolderMixedState,
|
||||
origMetadata: MetadataOnRemote,
|
||||
sizesGoWrong: FileOrFolderMixedState[],
|
||||
deletions: DeletionOnRemote[],
|
||||
localDeleteFunc: any,
|
||||
password: string = "",
|
||||
concurrency: number = 1,
|
||||
callbackSizesGoWrong?: any,
|
||||
callbackSyncProcess?: any
|
||||
) => {
|
||||
const mixedStates = syncPlan.mixedStates;
|
||||
const totalCount = sortedKeys.length || 0;
|
||||
|
||||
if (sizesGoWrong.length > 0) {
|
||||
log.debug(`some sizes are larger than the threshold, abort and show hints`);
|
||||
callbackSizesGoWrong(sizesGoWrong);
|
||||
return;
|
||||
}
|
||||
|
||||
log.debug(`start syncing extra data firstly`);
|
||||
await uploadExtraMeta(
|
||||
client,
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
import { App, Modal, Notice, PluginSettingTab, Setting } from "obsidian";
|
||||
import type RemotelySavePlugin from "./main"; // unavoidable
|
||||
import type { TransItemType } from "./i18n";
|
||||
import type { FileOrFolderMixedState } from "./baseTypes";
|
||||
|
||||
import { log } from "./moreOnLog";
|
||||
|
||||
export class SizesConflictModal extends Modal {
|
||||
readonly plugin: RemotelySavePlugin;
|
||||
readonly skipSizeLargerThan: number;
|
||||
readonly sizesGoWrong: FileOrFolderMixedState[];
|
||||
readonly hasPassword: boolean;
|
||||
constructor(
|
||||
app: App,
|
||||
plugin: RemotelySavePlugin,
|
||||
skipSizeLargerThan: number,
|
||||
sizesGoWrong: FileOrFolderMixedState[],
|
||||
hasPassword: boolean
|
||||
) {
|
||||
super(app);
|
||||
this.plugin = plugin;
|
||||
this.skipSizeLargerThan = skipSizeLargerThan;
|
||||
this.sizesGoWrong = sizesGoWrong;
|
||||
this.hasPassword = hasPassword;
|
||||
}
|
||||
onOpen() {
|
||||
let { contentEl } = this;
|
||||
const t = (x: TransItemType, vars?: any) => {
|
||||
return this.plugin.i18n.t(x, vars);
|
||||
};
|
||||
|
||||
contentEl.createEl("h2", {
|
||||
text: t("modal_sizesconflict_title"),
|
||||
});
|
||||
|
||||
t("modal_sizesconflict_desc", {
|
||||
thresholdMB: `${this.skipSizeLargerThan / 1000 / 1000}`,
|
||||
thresholdBytes: `${this.skipSizeLargerThan}`,
|
||||
})
|
||||
.split("\n")
|
||||
.forEach((val) => {
|
||||
contentEl.createEl("p", { text: val });
|
||||
});
|
||||
|
||||
const info = this.serialize();
|
||||
|
||||
contentEl.createDiv().createEl(
|
||||
"button",
|
||||
{
|
||||
text: t("modal_sizesconflict_copybutton"),
|
||||
},
|
||||
(el) => {
|
||||
el.onclick = async () => {
|
||||
await navigator.clipboard.writeText(info);
|
||||
new Notice(t("modal_sizesconflict_copynotice"));
|
||||
};
|
||||
}
|
||||
);
|
||||
|
||||
contentEl.createEl("pre", {
|
||||
text: info,
|
||||
});
|
||||
}
|
||||
|
||||
serialize() {
|
||||
return this.sizesGoWrong
|
||||
.map((x) => {
|
||||
return [
|
||||
x.key,
|
||||
this.hasPassword
|
||||
? `encrypted name: ${x.remoteEncryptedKey}`
|
||||
: undefined,
|
||||
`local ${this.hasPassword ? "encrypted " : ""}bytes: ${
|
||||
this.hasPassword ? x.sizeLocalEnc : x.sizeLocal
|
||||
}`,
|
||||
`remote ${this.hasPassword ? "encrypted " : ""}bytes: ${
|
||||
this.hasPassword ? x.sizeRemoteEnc : x.sizeRemote
|
||||
}`,
|
||||
]
|
||||
.filter((tmp) => tmp !== undefined)
|
||||
.join("\n");
|
||||
})
|
||||
.join("\n\n");
|
||||
}
|
||||
|
||||
onClose() {
|
||||
let { contentEl } = this;
|
||||
contentEl.empty();
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -1,4 +1,4 @@
|
||||
{
|
||||
"0.3.2": "0.12.15",
|
||||
"0.3.18": "0.13.21"
|
||||
"0.3.23": "0.13.21"
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user