This series is written by a representative of the latter group, which is comprised mostly of what might be called "productivity users" (perhaps "tinkerly productivity users?"). Though my lack of training precludes me from writing code or improving anyone else's, I can, nonetheless, try and figure out creative ways of utilizing open source programs. And again, because of my lack of expertise, though I may be capable of deploying open source programs in creative ways, my modest technical acumen hinders me from utilizing those programs in what may be the most optimal ways. The open-source character, then, of this series, consists in my presentation to the community of open source users and programmers of my own crude and halting attempts at accomplishing computing tasks, in the hope that those who are more knowledgeable than me can offer advice, alternatives, and corrections. The desired end result is the discovery, through a communal process, of optimal and/or alternate ways of accomplishing the sorts of tasks that I and other open source productivity users need to perform.

Thursday, February 4, 2016

Miscellaneous Thursday quickies: what's your bi-directional syncing utility?

So I've been pursuing a research project for the last year or so and have been locating and saving material related to it, as well as doing some of my own writing in the area. I keep that material in a particular folder. That's all fine and good. The problem is that I want the ability to work on the project while I'm at any of 3 different computers--computers that are often located in 3 different locales, some of which are even remote from my LAN. So, how to host the same folder on all three machines, and keep it current with the most recent changes made on any of the 3 computers?

I intend for this to be a manual process, i.e., one that will involve me manually running some program or script on each of the three machines, in order to update the folder. I should also mention that I have access to a shell account where I can run a number of utilities that can facilitate this--so a 4th computer, technically speaking, is involved as well. I envision the shell account functioning as a sort of central hub for keeping said folders in sync: a sort of master copy of the folder can be stored there and each of the three machines can syncronize with that folder as need will arise.

I'm still trying to puzzle out how to pull all this together and am looking at the sorts of software/utilities that can accomplish the task. I've only tested out one option thus far--bsync. I opted for that in an initial foray for its simplicity: it's just a python script that enhances the functionality of rsync (a great sync utility, but one that does not do bi-directional synchronization). So all I needed to do was download the script and make it executable.

Using the utility, I was able to put the most current copy of the folder at my shell account by just running bsync MyFolder me@my.shellacct.com:MyFolder (the MyFolder directory must already exist at the remote address). So I've at least made a beginning.

That said, I'm still in the early stages of investigating approaches to do the sort of bi-directional synchronization I'm after. Tests with bsync have gone well so far but, if I'm understanding correctly, this utility does not deal well with sub-folders--which could be an issue in my use scenario; it seems bsync will work best on a folder or directory that contains only files, while my directory has a few sub-directories under it.

Other possible options I've found are csync (which uses smb or sftp), osync, bitpocket, and FreeFileSync. The first 3 of these are most attractive to me since they are command-line utilities. FreeFileSync is a graphical utility, though it does appear that it can be run from the command line as well. I should also mention unison, which I've looked at but not pursued--the reason being that it apparently requires that the same version be installed on all concerned machines, which is something that will be unrealistic in my case (Arch runs on 2 machines, an older Ubuntu on another, and BSD on the fourth).

So, what is your bi-directional synchronization software preference? Any further tips or pointers to add on accomplishing this task?
Posted by at 11:06 AM
Labels: , , , , ,

16 comments:

  1. ferdelsFebruary 4, 2016 at 1:25 PM

    try syncthing, offers versioning aswell as master copies, clients for most platforms

    ReplyDelete
  2. AnonymousFebruary 4, 2016 at 2:35 PM

    Or set up your own OwnCloud server and work out of the folder that's sync'd to your OwnCloud account on all your devices.

    ReplyDelete
  3. WorBluxFebruary 4, 2016 at 3:07 PM

    I very much like unison. It was written by an academic as a project to demonstrate how to guarantee a safe sync. It's very careful to keep both sides of the sync in a sane state to avoid data loss and corruption. It uses it's own protocol to determine what needs to be done (and it's usually pretty good at guessing, but you can manually override the direction of a sync on a per-file basis, and it asks you if it doesn't know what to do with a file difference)

    The actual transfer done with rsync by default but you can choose a different program if you wish. Remote transfers can be done via unison in deamon mode (nice and fast over a trusted network), or over ssh. (secure tunnel)

    It is open source and cross platform.

    If all your machines are on a fast local LAN, it may be easier to set up NFS to consolidate some of the copies.

    ReplyDelete
    Replies
    1. wayover13February 4, 2016 at 3:28 PM

      I've seen unison recommended a lot and so did a bit of looking. Like I said, from what I gather, unison is only an option if all concerned machines are running the same version (by which I understand same release number). That's a real obstacle for me, since 3 of the 4 machines are running different OS's/distro's. So I've suspended further investigation of unison. These computers are not all on the same LAN, btw. Thanks anyway for your comments.

      Delete
    2. WorBluxFebruary 4, 2016 at 9:42 PM

      http://www.cis.upenn.edu/~bcpierce/unison/download/releases/stable/unison-manual.html is the manual for the latest stable version. It looks like no non-backwards compatible changes be made since version 2.10.2 It also looks fairly straightforward to build with two hard dependencies if you don't need the GUI, and 5 if you do.

      Delete
    3. eMBeeFebruary 6, 2016 at 11:43 AM

      wayover13: did you actually check the available versions? some linux distributions offer multiple versions of unison for that very reason. fedora for example has versions 213,227,240, debian offers 227,232,240. all of these versions are already old. unison is no longer actively developed as far as i remember, only maintained. so chances are that you should find 240 everywhere. if not then 227 as a fallback, which looks like it was a long-term stable version.

      Delete
    4. wayover13February 6, 2016 at 12:24 PM

      The shell account runs version 2.27.57. I have no control over what runs there, so everything would have to match that. My Ubuntu system, which is not very up-to-date does have 2.27.57 available. But so far as I can tell, the only available version under Arch is 2.48.3-2. The Arch systems are the ones I am using most frequently.

      Delete
    5. eMBeeFebruary 6, 2016 at 2:03 PM

      ouch, someone had to dance out of the row...
      looks like there used to be a PKGBUILD of 227: https://wiki.archlinux.org/index.php/Unison#Version_incompatibility
      but even that disappeared. there is a mirror here: http://pkgbuild.com/git/aur-mirror.git/tree/unison-227

      it looks like this is a way to build the package from source.
      i never used arch, so i don't know if that is useful for you.

      greetings, eMBee.

      Delete
  4. UnknownFebruary 4, 2016 at 4:44 PM

    This comment has been removed by the author.

    ReplyDelete
  5. UnknownFebruary 4, 2016 at 5:15 PM

    Bittorrent sync is a syncing tool I like. bittorrent is one of the fastest file transport protocols available. Btsync is a pretty nice an solid tool that supports selective sync and works whitiout opening ports on your router you can sync the items to multiple Internet connected hosts but it also find hosts in your local network for faster transfer. I like it for its redundancy. It at least claims to be secure and I have no reason to doubt that

    ReplyDelete
  6. AnonymousFebruary 5, 2016 at 2:34 AM

    git.

    ReplyDelete
  7. Chris HabasinskiFebruary 5, 2016 at 2:47 AM

    How about using git, pushing and pulling from a master repo

    ReplyDelete
  8. Federico KerekiFebruary 5, 2016 at 4:43 AM

    Can any of the tools in http://www.ibm.com/developerworks/aix/library/au-rsyncfamily/ help you out?

    ReplyDelete
  9. CFWhitmanFebruary 5, 2016 at 12:40 PM

    I admit that first thing that came to mind when I read your requirements was also git or another version control software. It's probably using things like this for code that makes one think this way. Of course there is somewhat of a learning curve, but they're designed to do things like this.

    ReplyDelete
    Replies
    1. jeremywakemanFebruary 7, 2016 at 7:32 PM

      I use a git repo to do exactly this. I keep general documents (txt, tex, pdf) as well as source code in a git repo on a digital ocean droplet. https://m.do.co/c/855feddf4648

      Delete

Subscribe to: Post Comments (Atom)