File server with on-demand sync, preserve the filesystem, and runs without external DB?
Hi, I'm looking for a FOSS file syncing/sharing solution that does all the following specific things:
- Sync files only when needed, to save space on my client devices.
- Preserve the filesystem on the server for backups. So no opaque blocks like SeaFile.
- No need for external MySQL/Postgres container. SQLite would be okay.
Currently the closest thing to fulfill these is oCIS, but it has a decomposedFS file structure which defeats the second point. Nextcloud may run with embedded SQLite, but I'm reluctant to try it again due to previous experiences (lots of bugs, sluggish, etc). Mountain Duck and FileRun are not FOSS. Filestash would be nice if it can integrate with existing Nextcloud/Owncloud clients for the on-demand syncing functionality, especially on Windows. It would be nice to have an open-source alternative to Mountain Duck, in order to use on-demand sync functionalities with a standard storage backend such as SFTP.
Would you have any recommendations of what to do?
Thank you in advance!
I use Syncthing for this type of task on my PC and Phone and it stores a copy of the shared folder on the server with the option for file versioning. Having a Server is optional by the way.
AFAIK, Syncthing clones the entire folder across peers (the server is just another peer it seems), which isn't ideal for my use case Do you know any current way to configure it for selective syncing?
I don’t think it can do selective syncing. I’ve been also searching for a similar solution but didn’t find one. Finally opted for syncthing with my most important files. Other files I can get them via web using filestash.
Owncloud supports selective sync, and seems a lot better for performance compared to Nextcloud.
Alternatively you could roll your own with rclone which is essentially an open source alternative to mountain duck. Then you can just use a simple connection via SFTP, FTP, WebDAV, etc...
Non-OCIS Owncloud still needs a dedicated database and recommend against SQLite in prod
I've looked at rclone mounting with the --vfs-cache-* flags. But I'm not sure how it can smart sync like mountain duck or handle conflicts elegantly like the Nextcloud/Owncloud clients do. Let me know how to set it up that way if possible
I vouch for Syncthing as well. I enabled storing in my own remote hosting provider marking it as untrusted, so my files are encrypted there.
Commenting largely to watch - I use Syncthing as my daily driver sync tool, and Resilio for the on-demand stuff.
Resilio has on-demand/selective sync, but I don't recall if it's open source, I don't think so. Plus, it's hard on memory with larger folders, as it keeps the index in ram. My media sync folder really impacts my desktop, and I only run Resilio on my mobile devices when I want to sync something, then turn it off.
rsync?
Update: there seems to be a more "native" storage driver for OCIS that is currently WIP. It seems very exciting
SFTP mountpoint + rsync (or grsync GUI) or unison (or unison-gtk GUI)?
I don't think this would solve your particular use case. However, SSHFS is absolutely amazing for remote smb share access.
I used it on my laptop to access my home NAS for years.
Why no real db? Those other 2 features make sense, but if the only option you can use sacrifices the 3rd option then it seems like a win. Postgres is awesome and easy to backup, just a single command can backup the whole thing to a file making it easy to restore.
I think oCIS spoiled me with regards to the database issue xD. You bring up a good point - I'll try reinstalling Nextcloud with Postgres, removing unneeded bloat, and use it until oCIS has a "native" backend
I am playing with SFTPGO, while not being a backup solution its a great backbend supporting sftp, WebDAV and much more that you can bind with something on client side.
Currently using synchthing, but planning to switch since that is not a backup tool.
Your post stood out to me. Why isn't it? Also what would you suggest? Thoughts on restic?
I use restic. Nice simple and works. Synchthing to keep files synched between phone and server. Restic for server backups.
Sftpgo is a very interesting backend + web interface if you look for something to access/copy/backup your files remotely.