Backing up data
The idea is, I have been a software engineer for the past three decades.
I REALLY love my profession (even though now, it’s pretty much sitting in meetings all day), but I love it.
I have a few projects I always wanted to do, because I just want to do them because they are useful to me.
I have architected and coded hundreds of giant projects. At this point, building anything with the big web frameworks just seems exhausting.
I have decided to bring each of these pet projects to fruition and then document how and why i made each decision.
Every project will be using smalltalk as the data store. If the project is a website, I will use seaside as the web framework. Otherwise, I’ll use the teapot as an API interface.
The series of articles I will write will be aimed at people in my position. People who are familiar with software development, and would like to read about it at a higher than tutorial level.
In the end, I’ll end up with:
- an application of some sort that scratches my own itch
- a pile of articles outlining how i designed the tool set for the application.
and in reality, it’s gonna be thinly veiled Smalltalk evangelism.
But i think it might be worthwhile to talk about this stuff in a pubilc platform.
On Jan 18, 2024, at 10:34 AM, Yanni Chiu yannix7db@gmail.com wrote:
Can you sketch out the complexity aspects of your five projects? Is this all for learning, or are they “production” applications? Do you want to gain experience with different ways to persist data, or are you just thinking that you have to use different ways due to project complexity? It might be
peace,
sergio
photographer, journalist, visionary
Public Key: https://pgp.key-server.io/pks/lookup?op=get&search=0x69B08F58923AB3A2
#BitMessage BM-NBaswViL21xqgg9STRJjaJaUoyiNe2dV
@sergio_101@mastodon.social
https://sergio101.com
http://www.codeandmusic.com
http://www.twitter.com/sergio_101
http://www.facebook.com/sergio101
Sergio,
I would go with STON, as advised by Norbert (I don't have experience
with Soil). One of the nice things of STON is that you can use whatever
DVCS you like to versionate your data. For example, we locally use
Fossil, while most people prefer Git.
STON is super versatile! As an example, I used it with Grafoscopio, my
Pharo powered interactive outliner/documenter, to store notebooks with
Markdown stored inside STON, as you can see in [1]. And now I flipped
that idea and I'm using STON to embed notebooks metadata inside
Markdeep. In this way anyone can read notebooks directly in the web,
without specialized software. [1a] is just the same document in [1] but
with this flipped format idea (if you,re even more curious, look at the
source code of the page at [1a] and you will find that the STON
representation of notebook cell/node medatada is pretty light and readable).
[1] https://mutabit.com/repos.fossil/indieweb/file?name=indieweb.ston&ci=tip
[1a]
https://mutabit.com/repos.fossil/indieweb/doc/tip/indie-web--25zqu.md.html
This is just an example of how STON allowed us to port lessons learned
from Grafoscopio, made in 2016, to Lepiter, released in 2021, ensuring
longevity of our community memory and data. Hopefully you will find STON
pretty empowering in your own experiments.
Cheers,
Offray
On 18/01/24 9:37, Norbert Hartl wrote:
Am 18.01.2024 um 14:54 schrieb sergio ruiz sergio.rrd@gmail.com:
Using this Will my links remain viable using this method?
Meaning that an event has a theater. That theater is also in the
OrderedCollection of Theaters.
If the name of he theater changed in the Theaters collection, will
that name change occur in the event theater object?
After bringing these collections back in via STON, will all of these
relationships remain intact?
If with relationship you mean that one object event has a reference to
a theater object then yes. STON serializes the graph as objects and
all references will be reestablished on materialization.
Norbert
On Jan 18, 2024, at 8:26 AM, Norbert Hartl norbert@hartl.name wrote:
If you have three classes with class side collections you can have a
make e.g. a dictionary and put the collections from the class side
into it like
model := {
#theaters -> Theater instances.
#movies -> Movie instances.
#event -> Event instances.
} asDictionary.
peace,
sergio
photographer, journalist, visionary
Public Key:
https://pgp.key-server.io/pks/lookup?op=get&search=0x69B08F58923AB3A2
#BitMessage BM-NBaswViL21xqgg9STRJjaJaUoyiNe2dV
@sergio_101@mastodon.social
https://sergio101.com
http://www.codeandmusic.com
http://www.twitter.com/sergio_101
http://www.facebook.com/sergio101
Hi Sergio,
I added a Movies demo, using a Soil database. There was a previous
Accounting demo that had two versions (in memory database and Soil
database) from which I based the new demo. I like the Movie demo because it
is simple yet complex enough. I suspect the Accounting demo is too
complicated to grasp as a first encounter. I’m going to use the Movie demo
for a presentation, so it’ll continue to be tweaked. Mainly, I’ll be adding
a way to use SoilTransaction with #commit, instead of
#checkpointAndContinue.
The code is at:
[image: Seaside-Quasar.png]
yannij/Seaside-Quasar: Pharo/Smalltalk code to use Quasar Framework in
Seaside. Also, some demo apps using Magritte are provided.
https://github.com/yannij/Seaside-Quasar
github.com
https://github.com/yannij/Seaside-Quasar
Cheers,
Yanni
On Thu, Jan 18, 2024 at 11:11 AM sergio ruiz sergio.rrd@gmail.com wrote:
[…]
In the end, I’ll end up with:
- an application of some sort that scratches my own itch
- a pile of articles outlining how i designed the tool set for the
application.
and in reality, it’s gonna be thinly veiled Smalltalk evangelism.
Hi Sergio,
I have used Fuel to good effect to store a tree of objects (with all
interlinkages).
Saving:
saveTo: aFileOrName
| aFileName |
aFileName := aFileOrName fullName.
FLSerializer serialize: self toFileNamed: aFileName
Loading:
loadFrom: aFileOrFileName
| new |
new := FLMaterializer materializeFromFileNamed: aFileOrFileName fullName.
new ifNil: [^nil]. "becomeForward fails for nil!"
self becomeForward: new
Just make sure you don't have any references to objects outside of your
model tree - you'll disappear down a black hole as it tries to save the
whole image!
Cheers,
Stew
On Thu, Jan 18, 2024 at 5:48 PM sergio ruiz sergio.rrd@gmail.com wrote:
I have been in the relational database world for decades. One of the
things that is super simple is backing up and restoring data.
Sometimes, it makes it makes sense to pull the production data into
development. In the case of a catastrophe, it’s super simple to restore
data from a backup.
I am thinking of backup strategies for my data once again, and am
wondering if things have changed since I was last using smalltalk.
I am thinking of the following:
STON
I could dump all of my data into STON, but there is one slight twist. In
this project, it’s just making a list of movie listings. There are three
basic classes: Theater, Movie, Event. Each of these classes stores an
OrderedCollection of items of that type on a class variable, as I’ll need
to access each on their own.. so, if an event has a theater, I’ll need to
access Theaters on their own, and Movies on their own.
So, I would have to rebuild the list of Theaters from the Theaters in each
event. Otherwise, I would lose the link between the Theaters on the Theater
class variable and the Theater in the Event instance.
Am I thinking about this incorrectly?
Gemstones
I am totally into the idea of using gemstones. Is there a backup and
restore for objects here? do the relationships remain?
Voyage
This seems like a good solution, but I would rather stay in the smalltalk
object store.
Ideas?
Thanks!
peace,
sergio
photographer, journalist, visionary
Public Key:
https://pgp.key-server.io/pks/lookup?op=get&search=0x69B08F58923AB3A2
#BitMessage BM-NBaswViL21xqgg9STRJjaJaUoyiNe2dV
@sergio_101@mastodon.social
https://sergio101.com
http://www.codeandmusic.com
http://www.twitter.com/sergio_101
http://www.facebook.com/sergio101
Sergio - might be too late now, but I haven't seen anyone mention Sean's solution - https://github.com/seandenigris/Simple-Persistence (I think it uses Fuel under the hood, but its a nice simple solution)
Tim
On Thu, 18 Jan 2024, at 4:48 AM, sergio ruiz wrote:
I have been in the relational database world for decades. One of the things that is super simple is backing up and restoring data.
Sometimes, it makes it makes sense to pull the production data into development. In the case of a catastrophe, it’s super simple to restore data from a backup.
I am thinking of backup strategies for my data once again, and am wondering if things have changed since I was last using smalltalk.
I am thinking of the following:
STON
I could dump all of my data into STON, but there is one slight twist. In this project, it’s just making a list of movie listings. There are three basic classes: Theater, Movie, Event. Each of these classes stores an OrderedCollection of items of that type on a class variable, as I’ll need to access each on their own.. so, if an event has a theater, I’ll need to access Theaters on their own, and Movies on their own.
So, I would have to rebuild the list of Theaters from the Theaters in each event. Otherwise, I would lose the link between the Theaters on the Theater class variable and the Theater in the Event instance.
Am I thinking about this incorrectly?
Gemstones
I am totally into the idea of using gemstones. Is there a backup and restore for objects here? do the relationships remain?
Voyage
This seems like a good solution, but I would rather stay in the smalltalk object store.
Ideas?
Thanks!
peace,
sergio
photographer, journalist, visionary
Public Key: https://pgp.key-server.io/pks/lookup?op=get&search=0x69B08F58923AB3A2
#BitMessage BM-NBaswViL21xqgg9STRJjaJaUoyiNe2dV
@sergio_101@mastodon.social
https://sergio101.com
http://www.codeandmusic.com
http://www.twitter.com/sergio_101
http://www.facebook.com/sergio101
Let me check it out.
It’s not too late. This is a fun “scratch my own itch” project.
So there are no deadlines looming or anything..
On Jan 31, 2024, at 9:20 AM, Tim Mackinnon tim@testit.works wrote:
Sergio - might be too late now, but I haven't seen anyone mention Sean's solution - https://github.com/seandenigris/Simple-Persistence (I think it uses Fuel under the hood, but its a nice simple solution)
peace,
sergio
photographer, journalist, visionary
Public Key: https://pgp.key-server.io/pks/lookup?op=get&search=0x69B08F58923AB3A2
#BitMessage BM-NBaswViL21xqgg9STRJjaJaUoyiNe2dV
@sergio_101@mastodon.social
https://sergio101.com
http://www.codeandmusic.com
http://www.twitter.com/sergio_101
http://www.facebook.com/sergio101
I’m happy to answer any questions about Simple Persistence. It is a nice framework around (potentially any) serializer. It’s meant to be pluggable but currently uses Fuel out of the box. You just tell it what classes to persist and then create two methods per class to handle materialization/serialization.
For the record, Ramon Leon created it in a blog post. I just ported it to Pharo and have been maintaining it.
The stated reason he created it is that most solutions are complete overkill for 99% of projects that will never “become the next google”. It’s pretty basic, but at least two steps more sophisticated than simply saving the image because it keeps old versions and is easily reloadable into another image.
That said, I’ve been using it almost exclusively for my personal projects and have yet to grow out of it.
The approach as I understand is simple and valuable until you face concurrency. Nothing in SimplePersistence prevents your data from becoming corrupt in a concurrent situation. I agree that when you start a project you almost never face concurrency because the odds are way too low to have two things at the same time happening. This is because computers are really fast today and the amount of activity on a service is largely overestimated.
But there is still a huge gap between SimplePersistence and those overkill persistence solutions. For exactly that reason I've done Soil (https://github.com/ApptiveGrid/Soil) to fill that gap. It is one project with no dependency and setting it up is a no-brainer. It is way bigger than SimplePersistence but it deals with concurrency and adds query capabilties for your objects. But you have to use transactions to do this.
I started ApptiveGrid just by having the data persisted through saving the image. Next was to STON out the model. Then versioned STON files to add fallback. Then there was Soil (well, first voyage and then omnibase and then Soil) which is the next level to iterate.
So ApptiveGrid is not the next google but it also wouldn't work with SimplePersistence. So good that we have enough tools to improve our projects on.
Norbert
Am 15.02.2024 um 06:43 schrieb sean@clipperadams.com:
I’m happy to answer any questions about Simple Persistence. It is a nice framework around (potentially any) serializer. It’s meant to be pluggable but currently uses Fuel out of the box. You just tell it what classes to persist and then create two methods per class to handle materialization/serialization.
For the record, Ramon Leon created it in a blog post. I just ported it to Pharo and have been maintaining it.
The stated reason he created it is that most solutions are complete overkill for 99% of projects that will never “become the next google”. It’s pretty basic, but at least two steps more sophisticated than simply saving the image because it keeps old versions and is easily reloadable into another image.
That said, I’ve been using it almost exclusively for my personal projects and have yet to grow out of it.
Norbert,
I'm very interested to investigate Soil. Do you have examples on how to
use Soil?
I currently use a STON persistence model. My current approach to my model
is structured specifically to make STON full save and restores simple...
- Wrapper Class which contains a collection of high level objects
- high level objects contain collections of next level objects
- next level objects contain collections of lower level objects
- etc.
- next level objects contain collections of lower level objects
- high level objects contain collections of next level objects
While the database size (7.3Mb) is not terribly large, I am writing about
50 of those per day as each save simply saves off the entire old STON and
writes a new one. Concurrency is generally not a problem as we have the
application automatically save itself every 10 minutes, but every once in a
while we do run up against a conflict that would be best handled by a
concurrent capable model.
I have roughly nine applications using the same class models, so it would
be very helpful to adopt a more OO database approach to persistence... even
though we're nowhere near Google :)
Thoughts on where to begin? I'm on the Discord server as well, if you'd
prefer to discuss there.
Thanks,
Russ
On Thu, Feb 15, 2024 at 4:49 AM Norbert Hartl norbert@hartl.name wrote:
The approach as I understand is simple and valuable until you face
concurrency. Nothing in SimplePersistence prevents your data from becoming
corrupt in a concurrent situation. I agree that when you start a project
you almost never face concurrency because the odds are way too low to have
two things at the same time happening. This is because computers are really
fast today and the amount of activity on a service is largely overestimated.
But there is still a huge gap between SimplePersistence and those overkill
persistence solutions. For exactly that reason I've done Soil (
https://github.com/ApptiveGrid/Soil) to fill that gap. It is one project
with no dependency and setting it up is a no-brainer. It is way bigger than
SimplePersistence but it deals with concurrency and adds query capabilties
for your objects. But you have to use transactions to do this.
I started ApptiveGrid just by having the data persisted through saving the
image. Next was to STON out the model. Then versioned STON files to add
fallback. Then there was Soil (well, first voyage and then omnibase and
then Soil) which is the next level to iterate.
So ApptiveGrid is not the next google but it also wouldn't work with
SimplePersistence. So good that we have enough tools to improve our
projects on.
Norbert
Am 15.02.2024 um 06:43 schrieb sean@clipperadams.com:
I’m happy to answer any questions about Simple Persistence. It is a nice
framework around (potentially any) serializer. It’s meant to be pluggable
but currently uses Fuel out of the box. You just tell it what classes to
persist and then create two methods per class to handle
materialization/serialization.
For the record, Ramon Leon created it in a blog post. I just ported it to
Pharo and have been maintaining it.
The stated reason he created it is that most solutions are complete
overkill for 99% of projects that will never “become the next google”. It’s
pretty basic, but at least two steps more sophisticated than simply saving
the image because it keeps old versions and is easily reloadable into
another image.
That said, I’ve been using it almost exclusively for my personal projects
and have yet to grow out of it.
--
Russ Whaley
whaley.russ@gmail.com
Hi Russ,
stay tuned, we are preparing a 0.1 release for Soil before or at the time of pharo12 release. With 0.1 we will document how to use it.
If you use STON it will be similar because if you do
db := Soil path: ‚mydb‘.
db initializeFilesystem.
txn := db newTransaction.
txn root: yourModelRoot.
txn commit.
it will save all of the 7,3 MB in the database. Every time you do
txn := db newTransaction.
txn root
it will read all of the model you have just write now but concurrent safe.
Optimizing this is partitioning your graph and saying which of your objects are standalaone by saying
txn makeRoot: anObject.
then only the needed parts of your model are loaded. But in order to save it you have to mark the objects you made root dirty by saying
txn markDirty: anObject
That‘s all for basic usage. For the rest you need to wait for the documentation.
Norbert
Am 17.02.2024 um 19:47 schrieb Russ Whaley <whaley.russ@gmail.com>:
Norbert,I'm very interested to investigate Soil. Do you have examples on how to use Soil?
I currently use a STON persistence model. My current approach to my model is structured specifically to make STON full save and restores simple...
Wrapper Class which contains a collection of high level objects
high level objects contain collections of next level objects
next level objects contain collections of lower level objects
etc.
While the database size (7.3Mb) is not terribly large, I am writing about 50 of those per day as each save simply saves off the entire old STON and writes a new one. Concurrency is generally not a problem as we have the application automatically save itself every 10 minutes, but every once in a while we do run up against a conflict that would be best handled by a concurrent capable model.
I have roughly nine applications using the same class models, so it would be very helpful to adopt a more OO database approach to persistence... even though we're nowhere near Google :)
Thoughts on where to begin? I'm on the Discord server as well, if you'd prefer to discuss there.
Thanks,
Russ
On Thu, Feb 15, 2024 at 4:49 AM Norbert Hartl <norbert@hartl.name> wrote:
The approach as I understand is simple and valuable until you face concurrency. Nothing in SimplePersistence prevents your data from becoming corrupt in a concurrent situation. I agree that when you start a project you almost never face concurrency because the odds are way too low to have two things at the same time happening. This is because computers are really fast today and the amount of activity on a service is largely overestimated.
But there is still a huge gap between SimplePersistence and those overkill persistence solutions. For exactly that reason I've done Soil (https://github.com/ApptiveGrid/Soil) to fill that gap. It is one project with no dependency and setting it up is a no-brainer. It is way bigger than SimplePersistence but it deals with concurrency and adds query capabilties for your objects. But you have to use transactions to do this.
I started ApptiveGrid just by having the data persisted through saving the image. Next was to STON out the model. Then versioned STON files to add fallback. Then there was Soil (well, first voyage and then omnibase and then Soil) which is the next level to iterate.
So ApptiveGrid is not the next google but it also wouldn't work with SimplePersistence. So good that we have enough tools to improve our projects on.
Norbert
Am 15.02.2024 um 06:43 schrieb sean@clipperadams.com:
I’m happy to answer any questions about Simple Persistence. It is a nice framework around (potentially any) serializer. It’s meant to be pluggable but currently uses Fuel out of the box. You just tell it what classes to persist and then create two methods per class to handle materialization/serialization.
For the record, Ramon Leon created it in a blog post. I just ported it to Pharo and have been maintaining it.
The stated reason he created it is that most solutions are complete overkill for 99% of projects that will never “become the next google”. It’s pretty basic, but at least two steps more sophisticated than simply saving the image because it keeps old versions and is easily reloadable into another image.
That said, I’ve been using it almost exclusively for my personal projects and have yet to grow out of it.
--
Russ Whaley
whaley.russ@gmail.com