Partition/Sync ready databases for Pharo?
I have the requirement that an app that I'm prospecting must be able to work offline and synchronize changes once the connection is restored. I want to avoid having to write "sync" logic manually. Is anybody using Pharo with a database that supports automatic synchronization (replicas?) like the one provided by Firebase (Realtime Sync or Firestore)? I rather use PostgreSQL/SQLite locally and sync with a "central" DB, because I'm familiar with SQL/RDBMS, but I'm open to options. Regards, Esteban A. Maringolo
On Mon, Feb 18, 2019 at 02:56:10PM -0300, Esteban Maringolo wrote:
I have the requirement that an app that I'm prospecting must be able to work offline and synchronize changes once the connection is restored.
I want to avoid having to write "sync" logic manually.
I've not done this for real before, but here are some possibilities: - https://github.com/sqlite-sync/SQLite-sync.com - http://www.symmetricds.org/ - https://github.com/CanonicalLtd/dqlite - Use Fossil as your database
When you say Fossil - are you referring to "https://fossil-scm.org/index.html/doc/trunk/www/index.wikiâ ? Iâve seen it come up a few times - is it good? But rather than subvert Estebanâs thread - if it is the above, can you hook into it to save application runtime artefacts such that a distributed application can read them back (Iâd just sort of assumed it was for saving code and docs) - but it sounds like it can go a bit deeper than that (which seems quite interesting and perhaps what Esteban is after?) Tim
On 19 Feb 2019, at 01:46, Pierce Ng <pierce@samadhiweb.com> wrote:
On Mon, Feb 18, 2019 at 02:56:10PM -0300, Esteban Maringolo wrote:
I have the requirement that an app that I'm prospecting must be able to work offline and synchronize changes once the connection is restored.
I want to avoid having to write "sync" logic manually.
I've not done this for real before, but here are some possibilities:
- https://github.com/sqlite-sync/SQLite-sync.com - http://www.symmetricds.org/ - https://github.com/CanonicalLtd/dqlite - Use Fossil as your database
Hi Tim, Pierce, No problem, I was going to ask about Fossil as well, I know it is a self contained SCM, with issue tracker, and whatnot. People that uses it don't want to use git except if forced to. So I also ask how this would work. :) I saw SQLite-sync before, but it is a commercial product, and its platform support/setup seems convoluted. dqlite might be a viable alternative, from more than reliable provider, it's is not clear whether I have to use a different library to access SQLite (it seems so) or a special wire protocol/api calls. The only drawback I see is that, according to the the FAQ [1], from the CAP theorem the consensus algorithm puts Consistency and Partition and leaves out Availability. I'm looking into CouchDB and reading its "Definitive Guide"[2] book. It seems straightforward to install, distribute and uses a MVCC approach that puts availability as its top priority, and since my domain data has a low probability of conflicts, the eventual consistency will be more than enough. OTOH I'm not used to document based DBMS, which will require me to rethink a lot of my domain and application models. Also, it has a simple HTTP API (JSON/REST) and there is an existing Smalltalk client that might be ported and adapted to Pharo https://cwiki.apache.org/confluence/display/COUCHDB/Smalltalk Background: I have the prospect to develop an application to replace an existing one that is a desktop app that "syncs" from/to a central server every 15', most of the data entry happens locally and the central server is read only for the most part, it is old, ugly, but it works "reliably", so I must come with an alternative that is as reliable as that, and then provide a better UI/UX, new features, etc. Regards! Esteban A. Maringolo [1] From https://github.com/CanonicalLtd/go-dqlite/ Q: When not enough nodes are available, are writes hung until consensus? A: Yes, however there's a (configurable) timeout. This is a consequence of Raft sitting in the CP spectrum of the CAP theorem: in case of a network partition it chooses consistency and sacrifices availability. [2] http://guide.couchdb.org/draft/index.html El mar., 19 feb. 2019 a las 10:03, Tim Mackinnon (<tim@testit.works>) escribió:
When you say Fossil - are you referring to "https://fossil-scm.org/index.html/doc/trunk/www/index.wikiâ ?
Iâve seen it come up a few times - is it good? But rather than subvert Estebanâs thread - if it is the above, can you hook into it to save application runtime artefacts such that a distributed application can read them back (Iâd just sort of assumed it was for saving code and docs) - but it sounds like it can go a bit deeper than that (which seems quite interesting and perhaps what Esteban is after?)
Tim
On 19 Feb 2019, at 01:46, Pierce Ng <pierce@samadhiweb.com> wrote:
On Mon, Feb 18, 2019 at 02:56:10PM -0300, Esteban Maringolo wrote:
I have the requirement that an app that I'm prospecting must be able to work offline and synchronize changes once the connection is restored.
I want to avoid having to write "sync" logic manually.
I've not done this for real before, but here are some possibilities:
- https://github.com/sqlite-sync/SQLite-sync.com - http://www.symmetricds.org/ - https://github.com/CanonicalLtd/dqlite - Use Fossil as your database
Hi, I think that using Fossil depends on what things are you synchronizing and how fast. In the Grafoscopio community we use Fossil to sync documentation (Grafoscopio notebooks and Wiki pages) and issues, which means documents that move slow, and we use CodiMD [1] (Firebase based) when we do synchronization on documents we write together in real time (like hackathon/documentathon notes). Usually we publish them in a single Fossil repository (see [2] for an example of this combination of slow + quick sync combining Fossil and CodiMD). For alternatives to Firebase I have been starting to look at [3] and [4]. I would really love to see a Pharo client to any of such alternatives as I would like to have real time collaboration on Grafoscopio based notebooks. [1] http://demo.codimd.org/ [2] https://mutabit.com/repos.fossil/datafem/ [3] https://www.quora.com/Whats-the-closest-open-source-alternative-to-Firebase-... [4] https://alternativeto.net/software/firebase/?license=opensource Hope it helps, Offray On 19/2/19 8:36, Esteban Maringolo wrote:
Hi Tim, Pierce,
No problem, I was going to ask about Fossil as well, I know it is a self contained SCM, with issue tracker, and whatnot. People that uses it don't want to use git except if forced to. So I also ask how this would work. :)
I saw SQLite-sync before, but it is a commercial product, and its platform support/setup seems convoluted.
dqlite might be a viable alternative, from more than reliable provider, it's is not clear whether I have to use a different library to access SQLite (it seems so) or a special wire protocol/api calls. The only drawback I see is that, according to the the FAQ [1], from the CAP theorem the consensus algorithm puts Consistency and Partition and leaves out Availability.
I'm looking into CouchDB and reading its "Definitive Guide"[2] book. It seems straightforward to install, distribute and uses a MVCC approach that puts availability as its top priority, and since my domain data has a low probability of conflicts, the eventual consistency will be more than enough. OTOH I'm not used to document based DBMS, which will require me to rethink a lot of my domain and application models.
Also, it has a simple HTTP API (JSON/REST) and there is an existing Smalltalk client that might be ported and adapted to Pharo https://cwiki.apache.org/confluence/display/COUCHDB/Smalltalk
Background: I have the prospect to develop an application to replace an existing one that is a desktop app that "syncs" from/to a central server every 15', most of the data entry happens locally and the central server is read only for the most part, it is old, ugly, but it works "reliably", so I must come with an alternative that is as reliable as that, and then provide a better UI/UX, new features, etc.
Regards!
Esteban A. Maringolo
[1] From https://github.com/CanonicalLtd/go-dqlite/ Q: When not enough nodes are available, are writes hung until consensus? A: Yes, however there's a (configurable) timeout. This is a consequence of Raft sitting in the CP spectrum of the CAP theorem: in case of a network partition it chooses consistency and sacrifices availability.
[2] http://guide.couchdb.org/draft/index.html
El mar., 19 feb. 2019 a las 10:03, Tim Mackinnon (<tim@testit.works>) escribió:
When you say Fossil - are you referring to "https://fossil-scm.org/index.html/doc/trunk/www/index.wikiâ ?
Iâve seen it come up a few times - is it good? But rather than subvert Estebanâs thread - if it is the above, can you hook into it to save application runtime artefacts such that a distributed application can read them back (Iâd just sort of assumed it was for saving code and docs) - but it sounds like it can go a bit deeper than that (which seems quite interesting and perhaps what Esteban is after?)
Tim
On 19 Feb 2019, at 01:46, Pierce Ng <pierce@samadhiweb.com> wrote:
On Mon, Feb 18, 2019 at 02:56:10PM -0300, Esteban Maringolo wrote:
I have the requirement that an app that I'm prospecting must be able to work offline and synchronize changes once the connection is restored.
I want to avoid having to write "sync" logic manually. I've not done this for real before, but here are some possibilities:
- https://github.com/sqlite-sync/SQLite-sync.com - http://www.symmetricds.org/ - https://github.com/CanonicalLtd/dqlite - Use Fossil as your database
El mar., 19 feb. 2019 a las 13:40, Offray Vladimir Luna Cárdenas (<offray.luna@mutabit.com>) escribió:
For alternatives to Firebase I have been starting to look at [3] and [4]. I would really love to see a Pharo client to any of such alternatives as I would like to have real time collaboration on Grafoscopio based notebooks.
[1] http://demo.codimd.org/ [2] https://mutabit.com/repos.fossil/datafem/ [3]
https://www.quora.com/Whats-the-closest-open-source-alternative-to-Firebase-... [4] https://alternativeto.net/software/firebase/?license=opensource
These takes the "old" Firebase approach of a pure JS client (or Android/iOS with SDK). In general I don't like the "Firebase" approach of a huge JSON tree as your database, nor how it merges changes (PATCH/PUT). Firestore [1] seems to orient everything to a Collection based NoSQL approach (ala MongoDB). I want something I can access from Pharo, at least via a REST API. Although I'm biased to choose RDBMS (and SQL). Regards! Esteban A. Maringolo
Maybe something like the SQLite based Core Object [1], from the Etoile folks, can your for you... at least as inspiration, because the project looks pretty unmaintained. [1] http://coreobject.org/ Cheers, Offray On 19/2/19 12:45, Esteban Maringolo wrote:
El mar., 19 feb. 2019 a las 13:40, Offray Vladimir Luna Cárdenas (<offray.luna@mutabit.com>) escribió:
For alternatives to Firebase I have been starting to look at [3] and [4]. I would really love to see a Pharo client to any of such alternatives as I would like to have real time collaboration on Grafoscopio based notebooks.
[1] http://demo.codimd.org/ [2] https://mutabit.com/repos.fossil/datafem/ [3] https://www.quora.com/Whats-the-closest-open-source-alternative-to-Firebase-... [4] https://alternativeto.net/software/firebase/?license=opensource These takes the "old" Firebase approach of a pure JS client (or Android/iOS with SDK). In general I don't like the "Firebase" approach of a huge JSON tree as your database, nor how it merges changes (PATCH/PUT). Firestore [1] seems to orient everything to a Collection based NoSQL approach (ala MongoDB).
I want something I can access from Pharo, at least via a REST API. Although I'm biased to choose RDBMS (and SQL).
Regards!
Esteban A. Maringolo
On Tue, Feb 19, 2019 at 10:36:18AM -0300, Esteban Maringolo wrote:
No problem, I was going to ask about Fossil as well, I know it is a self contained SCM, with issue tracker, and whatnot. People that uses it don't want to use git except if forced to. So I also ask how this would work. :)
Fossil's built-in sync mechanism is described at https://fossil-scm.org/index.html/doc/trunk/www/sync.wiki. Fossil's traffic works through tunneling - this is how I 'fossil push' new blog posts from my laptop to my server through an SSH tunnel. I had also tested Fossil behind Internet-reachable Caddy HTTPS reverse proxy, which works too. For your use case, I might approach it like this: Write data locally and commit to Fossil as blobs. To sync, run Fossil under OSSubprocess to sync with the central server, either directly (if Fossil on central server is directly reachable) or thru a tunnel created by your application if your application runs on both ends.
I saw SQLite-sync before, but it is a commercial product, and its platform support/setup seems convoluted.
dqlite might be a viable alternative, from more than reliable provider, it's is not clear whether I have to use a different library to access SQLite (it seems so) or a special wire protocol/api calls.
I was actually thinking of https://github.com/rqlite/rqlite/ but did not remember the name when I replied originally. But it seems that, however you choose to sync, your application still needs to handle any data conflict that arises. Pierce
On Tue, Feb 19, 2019 at 01:02:51PM +0000, Tim Mackinnon wrote:
When you say Fossil - are you referring to "https://fossil-scm.org/index.html/doc/trunk/www/index.wikiâ ? Iâve seen it come up a few times - is it good?
Yes I mean the Fossil SCM. I use it as a personal wiki and file repo. It comes with built-in ticketing system and forum, all wrapped into one executable file. Fossil also imports/exports Git repos. I'd use Fossil for small development teams too. Except for game development and other cases involving multi-GB repos with lots of binary blobs.
can you hook into it to save application runtime artefacts such that a distributed application can read them back
Fossil was designed as a standalone executable. It is difficult to "library-ify" and the developers have no plan to do so. But it is just a single executable, supports JSON output for some of its functions, and can be invoked via OSSubprocess and the like, conceptually similar to how Iceberg uses libgit2 to do Git stuff. Pierce
participants (4)
-
Esteban Maringolo -
Offray Vladimir Luna Cárdenas -
Pierce Ng -
Tim Mackinnon