Sync problem (Apparently due to special characters)

[Pablo-dra]

Hi!

I’m using Rclone View to do a sync between my external hardrive with FileLu (a cloud storage provider).

So far it worked very well overall, so thank you very much for creating this software.

However, I have some issues when doing resyncs. After I sync my files, I do a sync again just to check if everything was uploaded correctly. In these subsequent syncs I notice that Rclone View tries to upload again files that are already uploaded.

Doing a small test, Rclone View works as expected. Basically it checks if the file is there and does not upload anything saying “no changes”.

However, when doing the sync with the main files Rclone View puts on the queue files that are already uploaded.

The only 2 clues I have are:

• Special characters like these: áéíóú ñ ’
  • However this doesn’t seem to be an issue with small tests
• Doing the sync between an external harddrive and FileLu
  • The small tests that worked were done directly from my internal SSD of my PC, but this shouldn’t be the cause

Below you will find attached some images

A

“Compare” small test that shows the expected behavior

[Pablo-dra]

B

“Compare” I do with the actual files showing that the special characters produce a mismatch between the files, and apparently causing Rclone View trying to upload files that are already there.

[Pablo-dra]

Thanks for the help in advance you can give so I can finish this backup with peace of mind once and for all!

[Pablo-dra]

Update

From the tests I’ve done:

• Using special characters seems to be problematic
  • But this seems to happen only with large files (like 9GB)
    • I’ve got other two files with special characters with 4KB and 400MB, and both were uploaded correctly as shown in my first screenshot.
  • Multithread upload doesn’t seem to create the wrong name problem
    • Disabling it (putting it to 0) still makes the name of the big file be wrong.
    • However, when using multithread upload (that is default to 4) after it uploads the file, it leaves an empty series of nested folders. Seems that those folders are needed for the multi upload, but these folders remain empty after the upload is done, and they don’t get deleted neither.

I don’t know exactly if this is a problem from the FileLu app or the Rclone View app, but I’m leaving this info that may be useful.

[Beslikiy]

Hi! Was the problem ever resolved?

[rcloneview]

Hi @Pablo-dra, and @Beslikiy — sorry for the very late reply. We have now reproduced this on a FileLu test account, and here is what we found.

Why only the big files got renamed

RcloneView uploads to FileLu through rclone, and rclone sends files larger than 500 MiB to FileLu in a different way than smaller files. That is why your 4 KB and 400 MB files kept their names and the 9 GB ones did not. Once a name is changed on FileLu, the next sync no longer recognizes the file, so it uploads it again.

We tested the same names (é, ñ, í, °, ') with both the older rclone and the current one, including a large file with default settings, and the names now come through correctly every time. So this looks like something that has since changed on FileLu’s side.

The empty nested folders

This was a bug in the rclone version bundled at the time: for large files it created a copy of the destination path inside itself (for example BP-Familia/1/BP-Familia/1/Dispositivos/...). It is fixed in rclone 1.75.1, which RcloneView 1.6.8 includes. The Multithread upload setting doesn’t affect FileLu uploads either way, so changing it won’t make a difference.

Cleaning up what is already on FileLu

1. Update to RcloneView 1.6.8.
2. On the FileLu website, delete the renamed copies that contain ' (they come from names with an apostrophe, e.g. Hasta 9 de Septiembre '94.mp4). RcloneView can’t delete these, and while one is left, the sync ends with an error and skips removing the empty folders.
3. In Job Manager, run your sync job with Dry run first. It should list the renamed copies for deletion and the originals for upload. If that looks right, run it normally. The large files with special characters will be uploaded once more, this time with their correct names, and the empty nested folders are removed at the end of the run.

Please check this before relying on the backup

While testing, we found two problems with downloading from FileLu. We’ve reported the second one to the rclone project (filelu: files above upload_cutoff are stored with unencoded names and can't be downloaded or deleted afterwards · Issue #10073 · rclone/rclone · GitHub) and reported the first one to FileLu:

• Files with ( or ) in the name can’t currently be downloaded from FileLu through RcloneView, whatever their size. FileLu answers “No file found”. Most of your video names contain parentheses, so please don’t count on restoring them through RcloneView for now. Please also check that you can download one of them from the FileLu website.
• Files larger than 500 MiB whose names contain any of ' ! # $ % ; [ ] or a backtick, or that start with ~, upload and show up normally, but can’t be downloaded or deleted through RcloneView afterwards.

We’ll post here when either of these is fixed.

@Beslikiy, if you’re seeing something similar, could you tell us your RcloneView version, your cloud storage provider and an example file name? It may be a different issue.

[rcloneview]

Hi @Pablo-dra and @Beslikiy, here is an update on the two download problems from my last post.

Names with parentheses: fixed by FileLu

FileLu has fixed this on their side. Files with ( or ) in the name can be downloaded and deleted through RcloneView again, whatever their size, and this works with RcloneView 1.6.8 as it is. Most of your videos have parentheses in their names, so they can be restored now, unless the name also contains one of the characters below and the file is larger than 500 MiB.

Large files with ' ! # $ % ; [ ], a backtick, or a leading ~: fixed in rclone, not yet in RcloneView 1.6.8

The rclone project has fixed this (filelu: files above upload_cutoff are stored with unencoded names and can't be downloaded or deleted afterwards · Issue #10073 · rclone/rclone · GitHub), and the fix will be part of rclone 1.76. RcloneView 1.6.8 still includes the older rclone, so with it, files larger than 500 MiB with these characters upload and show up normally, but can’t be downloaded or deleted through RcloneView. The next RcloneView update will include rclone 1.76, and we’ll post here when it is out.

If you want the fix now, you can install the rclone beta that already has it. In RcloneView, go to Settings → Embedded Rclone → Update Rclone and click Install Beta (1.76.0-beta.10552.4983d291e or later). Please keep in mind:

• The beta is a development build of rclone, not a release.
• Installing it stops any running transfers, so do it while no sync is running.
• When you later update RcloneView, the beta is replaced by the rclone that comes with the update. The next update brings rclone 1.76, so the fix stays.

Either way, switching to the fixed version won’t repair copies that are already on FileLu. Such files will still look normal in RcloneView, and the sync will skip them because it sees them as already uploaded, but they still can’t be downloaded. After switching, please delete those files on the FileLu website and run the sync again, so that they are uploaded once more with the fix.

Thank you for your patience.