SCIAMA
High Performance Compute Cluster
Rclone with Google Drive
While SCIAMA does not provide any local backup storage for important data, it is now possible to move your important data directly onto a cloud service for long-term storage and back for further processing. A good choice would be GoogleDrive, for which every university member has an account, but besides this various other services are supported (but not all tested yet): e.g. Amazon Drive, Amazon S3, MS OneDrive, DropBox.
The following sections provide instructions on how to do so using GoogleDrive as an example.
Loading and Configuring Rclone
Load the rclone module and configure Google Drive
You will only need to configure Google Drive once, simply run
n) New remote
s) Set configuration password
q) Quit config
n/s/q>
Select (n) to configure a new storage or edit an existing one by selecting (e). In the former case, you are now asked to provide a name for this config (This will be the name of the ‘drive’ so you may want to keep it short and avoid special characters/white spaces in it, in this example ‘google’).
You will then need to select the type of storage, in this example, 18 for Google Drive:
\ (google cloud storage)
18 / Google Drive
\ (drive)
19 / Google Photos
\ (google photos)
You will then be asked for a client ID, you will need to create one by following the instructions at Make your own ClientID
Google Application Client Id
Setting your own is recommended.
If you leave this blank, it will use an internal key which is low performance.
Enter a value. Press Enter to leave empty.
client_id>
Enter you client ID and then your Client Secret. Next you will be asked to chose the scope of access for Rclone:
Scope that rclone should use when requesting access from drive.
Choose a number from below, or type in your own value.
Press Enter to leave empty.
1 / Full access all files, excluding Application Data Folder.
\ (drive)
2 / Read-only access to file metadata and file contents.
\ (drive.readonly)
/ Access to files created by rclone only.
3 | These are visible in the drive website.
| File authorization is revoked when the user deauthorizes the app.
\ (drive.file)
/ Allows read and write access to the Application Data folder.
4 | This is not visible in the drive website.
\ (drive.appfolder)
/ Allows read-only access to file metadata but
5 | does not allow any access to read or download file content.
\ (drive.metadata.readonly)
scope>
Leave the last 3 questions blank, just hit enter:
Leave blank normally.
Needed only if you want use SA instead of interactive login.
Leading `~` will be expanded in the file name as will environment variables such as `${RCLONE_CONFIG_DIR}`.
Enter a value. Press Enter to leave empty.
service_account_file>
Edit advanced config?
y) Yes
n) No (default)
y/n>
Use auto config?
* Say Y if not sure
* Say N if you are working on a remote or headless machine
y) Yes (default)
n) No
y/n>
You will then be directed to the Google sign in page (make sure you can access the webpage from SCIAMA, either using ssh -X, ssh tunnel, or login to a login node using x2go). Select the google account you used to create the OAuth Client ID and click on Allow. Go back to terminal, you should see:
2022/11/11 10:47:50 NOTICE: If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=**************
2022/11/11 10:47:50 NOTICE: Log in and authorize rclone for access
2022/11/11 10:47:50 NOTICE: Waiting for code...
2022/11/11 10:50:19 NOTICE: Got code
Reply to last few prompts with the Default, just hit enter. You can then start using Rclone!