Nástroje používateľa

Nástoje správy stránok


blog:odborny:2024-09-19-rclone_configuring_root_folders_change_polling_and_cleaning_directory_cache

Rclone: configuring root folders, change polling and cleaning directory cache

Finding root folder for mounting OneDrive is quite complicated, as is refreshing directory cache for folder shortcuts in Google Drive, as they are ignored in change polling.

Setting root folder for mounting Microsoft OneDrive and Google Drive

You can always mount specific subfolder of a cloud by simply using the mount command with a subpath, e.g.:

rclone mount Remote:path/to/subfolder /local/mount

Adding the :path/to/subfolder to remote cloud will lead to mounting that specific subfolder. But it will also cause the mounted volume name to contain this path, so instead of simply seeing this as Remote in your computer volume list, you will see Remote:path/to/subfolder. To avoid this, you can manually set mounted volume name by supplying --volname parameter, e.g. --volname Remote.

Even better way to mount only a specific subfolder of a remote cloud is to do this directly in rclone.conf configuration file by setting root_folder_id parameter for the remote. This will tell Rclone that it should look for that folder and always use it as root folder for that remote. I am not sure whether every remote supports this, but at least Google Drive and Microsoft OneDrive do.

Finding the ID of a folder we want to use as a root folder

To set root_folder_id parameter, we need to find the folder ID. In Google Drive, this is straightforward: it is simply the ID you see in browser path when you enter that folder, e.g. 12345abcde67890ghijk12345ABCDE678 in the following path:

https://drive.google.com/drive/u/0/folders/12345abcde67890ghijk12345ABCDE678

The problem, as always, is with Microsoft. In OneDrive, there's absolutely no way of getting the real folder ID, unless you do not want to trace network traffic in browser Developer console. The folder ID is not contained in link of a folder copied for sharing, neither in Sharepoint address of that folder. You cannot find it through Teams app either.

But it is easy with Rclone. You get a list of directories with their IDs by calling this command with a parent directory path:

rclone lsf --dirs-only --format ip --csv Remote:path/to/parent/directory

You will get the list of subdirectories with their IDs, something like this:

b!iyyUEFMz_kWTKjWfSP5ndNq_WW0CwMdIvttUlRwXsgsdVV6DxHKpT6aD8sXU1zSZ#016RGUGOWSHGSXEHLSIZGKWSVETOVWJIZA,interim/
b!iyyUEFMz_kWTKjWfSP5ndNq_WW0CwMdIvttUlRwXsgsdVV6DxHKpT6aD8sXU1zSZ#016RGUGOS6WEFRB72OGBEKYOW76K3KRGOM,raw/

THOSE, finally, are the real folder IDs which need to be entered as values for root_folder_id parameter!

Directory cache for remotes and their limitations

When using Rclone caching (which you WANT and SHOULD use to speed things up – preferably fully turned on with switch --vfs-cache-mode full), Rclone will cache directory contents and would not send new network request to remote with each revisit of a folder.1)

There are two mechanisms which will make Rclone refresh its directory cache when the remote cloud changes via another vector (e.g. from a web interface):

  • One is --dir-cache-time, which sets the maximum time a directory is cached (default 5m0s). When that time is over, Rclone will ask remote cloud again to get the listing of files and subfolders in that directory.
  • Another and more clever one is --poll-interval, which sets the interval Rclone will ask to get a list of changes from the remote (default 1m0s). So there is no need to ever request the whole directory listing the second time, since it is enough for Rclone to get the updates (deltas) from remote.

Most remotes do not support polling, but when they do, you should set --dir-cache-time as high as possible, since there is no reason for Rclone to ever bluntly drop the whole directory cache.

Google Drive limitations of change-polling in shared drives and shortcuts to folders

Google Drive supports polling for changed files and directories, but only in your own drive. That excludes changes in shared drives AND, what is worse, that even excludes folders that you added as shortcuts to your own drive. Again, Rclone won't get any update notifications about contents of folder shortcuts through polling.2)

When you are working in a business environment where the master folder containing the whole organisation of folders is shared with you, this completely breaks your workflow and makes the whole change polling system of Rclone useless. If that is your case, turning off polling through --poll-interval 0 makes sense, since it won't help you anyway.

The question is, how to cope with that?

Manually cleaning directory cache in Google Drive in folder shortcuts

In the case that you only work within shared directories added to your drive as shortcuts, there is no ideal solution. Folders and files will never update, unless the --dir-cache-time runs out. So one possible solution is to set it to a low value, e.g. 1 minute. That will solve the problem, but path traversal will become noticeable slower.

The solution which works for me is it keep the --dir-cache-time high, but at least have the possibility of forcing the specific folder cache to refresh when I know I have updated it.

This requires two steps:

  1. You must run Rclone mount command with remote control turned on. This requires supplying the following parameters:
    1. --rc
    2. --rc-addr=IP:PORT (default “localhost:5572”)
    3. AND either both:
      • --rc-user=VALUE
      • --rc-pass=VALUE
    4. OR:
      • --rc-no-auth
  2. Then you can you use Terminal commands to force refreshing or forgetting either the entire Rclone directory cache, or just the cache for a specific folder.
    • To remove ("forget") the directory from cache, use:
      • rclone rc vfs/forget 'dir=path/to/directory',
      • or clean whole cache with rclone rc vfs/forget
    • To re-read ("refresh") directory contents from server, use:
      • rclone rc vfs/refresh 'dir=path/to/directory' (optionally adding recursive=true if you want all subdirectories to be reread)
    • If you run mount with --rc-user and --rc-pass values, you also have to supply them here, e.g.:
      • rclone rc vfs/refresh --rc-user=VALUE --rc-pass=VALUE --rc-addr=IP:PORT 'dir=path/to/directory'

Semi-automatically clean the cache using macOS services

Cleaning the cache by running terminal each time is cumbersome and errorprone. In macOS, the better way is to create Automator workflow, which will then be added to services menu when right clicking on folder:

  1. Run Automator.app
  2. Create new workflow (⌘Cmd+N) of type “Quick action”
  3. Set the workflow to receive current folders (not files) and if you use a file manager (like PathFinder, ForkLift or Crax Commander), also set it to run in any applications, not just Finder.
  4. Drag Run shell script action to empty action list
  5. The actual code would go like this:
    # go over each selected folder
    while read f
    do
      origPrefix='/path/to/mount/point/'
      newPrefix=''
      cachef="${f/#$origPrefix/$newPrefix}"
      if [[ "$f" == "$origPrefix"* ]]; then
        rclone rc vfs/refresh [–-rc-user=VALUE --rc-pass=VALUE --rc-addr=IP:PORT] dir="$cachef"
      else
        osascript -e "tell app \"System Events\" to display dialog \"Folder not under rClone folder, skipping:\n  $f\""
      fi
    done

My final cache-clearing script using macOS services

My final script ended up a little more tweaked, since I also wanted to take into account the following things:

  • I want the script to programatically refresh files & directory listing in the app after clearing the cache, since I am using Path Finder, which does not do it automatically (as Finder does).
  • I the selected entry is a regular file, I want to actually clean the cache of its parent folder. If it is a folder, I wanted to refresh that one. Finding parent folder in bash is easy.
  • For some reason, rclone rc vfs/forget did not really work for me, so I ended up using rclone rc vfs/refresh.
# get active app
activeApp=$(osascript -e "tell application \"System Events\"
  return name of first application process whose frontmost is true
end tell")
 
# refresh cache for each selected file/folder
while read f
do
  prefix='/path/to/remote/mount/point/'
  if [[ "$f" == "$prefix"* ]]; then
    cachef="${f/#$prefix/}"
    if [ -d "$f" ]; then
      path="$cachef"
    else
      path="${cachef%/*}"
    fi
 
    /usr/local/bin/rclone rc vfs/refresh [–-rc-user=VALUE --rc-pass=VALUE --rc-addr=IP:PORT] dir="$parent"
 
    #osascript -e "tell app \"System Events\" to display dialog \"Cleared rClone cache for folder:\n  $cachef\nwith parent:\n  $parent\nfrom active app: $activeApp.\""
  else
    osascript -e "tell app \"System Events\" to display dialog \"Folder not under rClone folder, skipping:\n  $f\""
  fi
done
 
# refresh calling app file listing
if [[ "$activeApp" == "Finder" ]]; then
  # Finder has no Reload method, use its native update method
  osascript -e "tell application \"Finder\" to tell front window to update every item"
else
  # for all other apps, try sending Cmd+R
  osascript -e "tell application \"System Events\"
    keystroke \"r\" using command down
  end tell"
  #osascript -e "tell application \"System Events\" to tell process \"Path Finder\"
    #click menu item \"Reload\" of menu \"View\" of menu bar 1
  #end tell"
fi

A few comments:

  • Without adding absolute path to rclone (/usr/local/bin/rclone), the script wasn't able to find it.
  • To automatically refresh the file listing, the app finds out the active app and discriminates between Finder and other apps:
    • If the refresh is called from Finder (= it is the active app), I use Finder-native method for refreshing file listing.
    • Otherwise, in other apps, the script sends ⌘+R keystroke to the foremost app. In Path Finder, I have set to View→Reload menu item command. So that refreshes the listing of items.
    • The alternative (commented in code) would be to manually invoke clicking on View→Reload menu item, but that would work only in Path Finder. Sending ⌘+R is more versatile – I have set it as the “reload” command in several other apps I use.
  • It would be easier to avoid using osascript calls and instead of those directly call the AppleScript code:
    tell application "System Events"
      keystroke "r" using command down
    end tell"''

    This simply did not work for me. It is possible that it's related to the permissions model in Mojave+. I could not find the way to debug it. Calling the code indirectly through the osascript call worked, though, so that's where I stayed.

  • To find out which AppleScript commands a particular app supports, you can use Script Editor.app. Upon launching, go to FileOpen Dictionary… (⇧⌘O) and select the app you want to examine. This will show you the dictionary of app's supported AppleScript features.

Comments

1)
Note that this has nothing to do with file caching. Removing cached file contents is an altogether separate problem.
2)
Note that, as of September 2024, this is also the case with Microsoft OneDrive, although polling not working there is filed as a bug and therefore might be solved at one point or another.
blog/odborny/2024-09-19-rclone_configuring_root_folders_change_polling_and_cleaning_directory_cache.txt · Posledná úprava: 2024/09/30 13:18 od Róbert Toth