bruno/remotely-save
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

moving sync docs

Browse Source
This commit is contained in:
fyears
2024-02-25 23:39:31 +08:00
parent 768d6636d0
commit 447e0c0aa4
4 changed files with 0 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/sync_algorithm/sync_ignoring_large_files.md
+17
View File
@@ -0,0 +1,17 @@
# Sync Ignoring Large Files
Initially, the plugin does not ignore large files.
From the new version in May 2022, it can ignore all files with some sizes. But we need some rules to make the function compatible with existing conditions.
1. If users are using E2E password mode, then the file sizes are compared on the **encrypted sizes**, rather than the original unencripted file sizes. The reasons are: the encrypted ones are in transferations, and the encrypted sizes can be computed from unencrypted sizes but not the reverse.
2. Assuming the file A, is already synced between local device and remote service before.
- If the local size and remote size are both below the threshold, then the file can be synced normally.
- If the local size and remote size are both above the threshold, then the file will be ignored normally.
- If the local size is below the threshold, and the remote size is above the threshold, then the plugin **rejects** the sync, and throws the error to the user.
- If the local size is above the threshold, and the remote size is below the threshold, then the plugin **rejects** the sync, and throws the error to the user.
- When it somes to deletions, the same rules apply.
The main point is that, if the file sizes "cross the line", the plugin does not introduce any further trouble and just reject to work for this file.
docs/sync_algorithm/v1/README.md
+29
View File
@@ -0,0 +1,29 @@
# Sync Algorithm
## Sources
We have three record sources:
1. Local files. By scanning all files in the vault locally. Actually Obsidian provides an api directly returning this.
2. Remote files. By scanning all files on the remote service. Some services provide an api directly returning this, and some other services require the plugin scanning the folders recursively.
3. Local "delete-or-rename" history. It's recorded by using Obsidian's tracking api. So if users delete or rename files/folders outside Obsidian, we could do nothing.
Assuming all sources are reliable.
## Deal with them
We list all combinations mutually exclusive and collectively exhaustive.
| ID | Remote Files | Local files | Local delete rename history | Extra | Decision |
| --- | ------------ | ----------- | --------------------------- | ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| 1 | exist | exist | ignore | mtime_remote > mtime_local | download remote file, create local folder if not exists, clear local history if exists |
| 2 | exist | exist | ignore | mtime_remote < mtime_local | upload local file, create remote folder if not exists, clear local history if exists |
| 3 | exist | exist | ignore | mtime_remote === mtime_local && password === "" && size_remote === size_local | clear local history if exists (the file was synced and no changes after last sync) |
| 4 | exist | exist | ignore | mtime_remote === mtime_local && password === "" && size_remote !== size_local | upload local file, clear local history if exists (we always prefer local to remote) |
| 5 | exist | exist | ignore | mtime_remote === mtime_local && password !== "" | clear local history if exists (in encryption mode, file sizes are unequal. we can only rely on mtime(s)) |
| 6 | exist | exist | ignore | If local is a folder. mtime_local === undefined | clear local history if exists. TODO: what if a folder and a previous file share the same name? |
| 7 | exist | not exist | exist | mtime_remote >= delete_time_local | download remote file, create folder if not exists |
| 8 | exist | not exist | exist | mtime_remote < delete_time_local | delete remote file, clear local history |
| 9 | exist | not exist | not exist | | download remote file, create folder if not exists |
| 10 | not exist | exist | ignore | local may be folder or file | upload local files recursively, create remote folder if not exists, clear local history if exists |
| 11 | not exist | not exist | ignore | | clear local history if exists |
docs/sync_algorithm/v2/README.md
+63
View File
@@ -0,0 +1,63 @@
# Sync Algorithm V2
## Sources
We have 4 record sources:
1. Local files. By scanning all files in the vault locally. Actually Obsidian provides an api directly returning this.
2. Remote files. By scanning all files on the remote service. Some services provide an api directly returning this, and some other services require the plugin scanning the folders recursively.
3. Local "delete-or-rename" history. It's recorded by using Obsidian's tracking api. So if users delete or rename files/folders outside Obsidian, we could do nothing.
4. Remote "delete" history. It's uploaded by the plugin in each sync.
Assuming all sources are reliable.
## Deal with them
We list all combinations mutually exclusive and collectively exhaustive.
### Files
In short, we collect four timestamps, and respect the max timestamp and its corresponding operation.
| t1 | t2 | t3 | t4 | local file to do | remote file to do | local del history to do | remote del history to do | equal to sync v2 branch |
| -------------- | -------------- | -------------- | -------------- | ---------------- | ----------------- | ----------------------- | ------------------------ | ----------------------- |
| mtime_remote | mtime_local | deltime_remote | deltime_local | del_if_exists | del_if_exists | clean | upload_local_del_history | |
| mtime_local | mtime_remote | deltime_remote | deltime_local | del_if_exists | del_if_exists | clean | upload_local_del_history | |
| mtime_remote | deltime_remote | mtime_local | deltime_local | del_if_exists | del_if_exists | clean | upload_local_del_history | |
| deltime_remote | mtime_remote | mtime_local | deltime_local | del_if_exists | del_if_exists | clean | upload_local_del_history | |
| mtime_local | deltime_remote | mtime_remote | deltime_local | del_if_exists | del_if_exists | clean | upload_local_del_history | |
| deltime_remote | mtime_local | mtime_remote | deltime_local | del_if_exists | del_if_exists | clean | upload_local_del_history | 8 |
| mtime_remote | mtime_local | deltime_local | deltime_remote | del_if_exists | del_if_exists | clean | keep | |
| mtime_local | mtime_remote | deltime_local | deltime_remote | del_if_exists | del_if_exists | clean | keep | |
| mtime_remote | deltime_local | mtime_local | deltime_remote | del_if_exists | del_if_exists | clean | keep | |
| deltime_local | mtime_remote | mtime_local | deltime_remote | del_if_exists | del_if_exists | clean | keep | |
| mtime_local | deltime_local | mtime_remote | deltime_remote | del_if_exists | del_if_exists | clean | keep | |
| deltime_local | mtime_local | mtime_remote | deltime_remote | del_if_exists | del_if_exists | clean | keep | |
| mtime_remote | deltime_remote | deltime_local | mtime_local | skip | upload_local | clean | clean | |
| deltime_remote | mtime_remote | deltime_local | mtime_local | skip | upload_local | clean | clean | 10 |
| mtime_remote | deltime_local | deltime_remote | mtime_local | skip | upload_local | clean | clean | |
| deltime_local | mtime_remote | deltime_remote | mtime_local | skip | upload_local | clean | clean | |
| deltime_remote | deltime_local | mtime_remote | mtime_local | skip | upload_local | clean | clean | 2;3;4;5;6 |
| deltime_local | deltime_remote | mtime_remote | mtime_local | skip | upload_local | clean | clean | |
| mtime_local | deltime_remote | deltime_local | mtime_remote | download_remote | skip | clean | clean | |
| deltime_remote | mtime_local | deltime_local | mtime_remote | download_remote | skip | clean | clean | 7;9 |
| mtime_local | deltime_local | deltime_remote | mtime_remote | download_remote | skip | clean | clean | |
| deltime_local | mtime_local | deltime_remote | mtime_remote | download_remote | skip | clean | clean | |
| deltime_remote | deltime_local | mtime_local | mtime_remote | download_remote | skip | clean | clean | 1;9 |
| deltime_local | deltime_remote | mtime_local | mtime_remote | download_remote | skip | clean | clean | |
### Folders
We actually do not use any folders' metadata. Thus the only relevent info is their names, while the mtime is actually ignorable.
1. Firstly generate all the files' plan. If any files exist, then it's parent folders all should exist. If the should-exist folder doesn't exist locally, the local should create it recursively. If the should-exist folder doesn't exist remotely, the remote should create it recursively.
2. Then, a folder is deletable, if and only if all the following conditions meet:
- it shows up in the remote deletion history
- it's empty, or all its sub-folders are deletable
Some examples:
- A user deletes the folder in device 1, then syncs from the device 1, then creates the same-name folder in device 2, then syncs from the device 2. The folder is deleted (again), on device 2.
- A user deletes the folder in device 1, then syncs from the device 1, then creates the same-name folder in device 2, **then create a new file inside it,** then syncs from the device 2. The folder is **kept** instead of deleted because of the new file, on device 2.
- A user deletes the folder in device 1, then syncs from the device 1, then do not touch the same-name folder in device 2, then syncs from the device 2. The folder and its untouched sub-files should be deleted on device 2.
docs/sync_algorithm/v3/design.md
+51
View File
@@ -0,0 +1,51 @@
# Sync Algorithm V3
Drafted on 20240117.
An absolutely better sync algorithm. Better for tracking deletions and better for subbranching.
## Huge Thanks
Basically a combination of algorithm v2 + [synclone](https://github.com/Jwink3101/syncrclone/blob/master/docs/algorithm.md) + [rsinc](https://github.com/ConorWilliams/rsinc) + (some of rclone [bisync](https://rclone.org/bisync/)). All of the later three are released under MIT License so no worries about the licenses.
## Features
Must have
1. true deletion detection
2. deletion protection (blocking) with a setting
3. transaction from the old algorithm
4. user warning show up, **new algorithm needs all clients to be updated!** (deliberately corrput the metadata file??)
5. filters
6. conflict warning
7. partial sync
Nice to have
1. true time and hash
2. conflict rename
## Description
We have _five_ input sources:
1. local all files
2. remote all files
3. _local previous succeeded sync history_
4. local deletions
5. remote deletions.
Init run, consuming remote deletions :
TBD
Later runs, use the first, second, third sources **only**.
Table modified based on synclone and rsinc. The number inside the table cell is the decision branch in the code.
| local\remote | remote unchanged | remote modified | remote deleted | remote created |
| --------------- | ------------------ | ------------------------- | ------------------ | ------------------------- |
| local unchanged | (02/21) do nothing | (09) pull remote | (07) delete local | (??) conflict |
| local modified | (10) push local | (16/17/18/19/20) conflict | (08) push local | (??) conflict |
| local deleted | (04) delete remote | (05) pull | (01) clean history | (03) pull remote |
| local created | (??) conflict | (??) conflict | (06) push local | (11/12/13/14/15) conflict |