Well done Norbert and Marcus!

Noury
On Aug 28 2024, at 2:00 pm, Norbert Hartl norbert@hartl.name wrote:

Awesome, thanks for sharing!

Martín

El mié, 28 ago 2024 a la(s) 8:00 a.m., Norbert Hartl (norbert@hartl.name)
escribió:

Excellent news!

S

Stéphane Ducasse
http://stephane.ducasse.free.fr
06 30 93 66 73

"If you knew today was your last day on earth, what would you do differently? ....ESPECIALLY if, by doing something different, today might not be your last day on earth.” Calvin & Hobbes

Hi Norbert,

Excellent news.

What's the problem that's stopping it from working in Windows? I couldn't
find a Github issue for that.

Regards!

Esteban A. Maringolo

On Wed, Aug 28, 2024 at 9:00 AM Norbert Hartl norbert@hartl.name wrote:

Because you assume windows support is something we want. I personally do not care about windows because I never used it and I think it is a really bad operating system that kills lots of things.

The technical view is:

• soil uses file range locks to lock portions of a file (think of row-level locking in relational databases)
• soil uses a system call fsync() on POSIX systems that flushes bytes from OS to disk. This is needed to meet transaction requirements

The architecture view is:

• I've seen the implementation of file opening with additional flags in OmniBase and how the design got really bad in order to support all of this. This might be non-issue today but needs to prove me wrong first
• windows might have comparable support for the requirements above but I'm not easily convinced to ruin the design in order to support it. Might also not be true...

The maintainer view is:

• I won't add windows support ever and Pablo was the only one offering his help so far (why should we implement something nobody needs?)
• even if there would be someone doing it initially I still need to be convinced that there is always someone maintaining it because I won't. And I surely do not want to block progress of soil just because things cannot move. They will be removed then.

So there are basically two options:

• windows supports the requirements (current and future ones) comparable to unix systems so that the design does not need to be changed a lot in  order to support windows. Then adding support is feasible
• in the meantime I made the file locking and fsync configurable. So by switching them off it might just work out of the box without any change. On a desktop machine it should(!) work reliably. Locking and range locks are mostly needed for concurrency. But noone would be insane enough to use windows as a server or in a distributed environment, haha, no that would be really foolish...hahaha

But we also try to make soil the de-facto database for pharo so in the end we have no choice then to add windows support.

Hope this helps,

Norbert

I think the only way finding out is to try. In ApptiveGrid we have 5000 instances and it is in production over one year. I wouldn't say there weren't problems in that time. But for me it is quite some time since I've found a conceptual problem that caused errors. This doesn't mean that it works for you and you won't bump into something the first time you use it. Stability is use case dependent.

The new implementation is only questionable from a bird's eye view. Meaning when you put them under the coarse grained category serializers. That is not appropriate for many things because the way you use it is important.

There is no written detailled analysis but let me give you a few points:

Fuel is a whole graph serializer. Meaning you give it one object and then it produces an efficient serialized blob out of it. There are not many interception points.
Soil supports graph partitioning, meaning that your graph will be sliced into smaller parts in order to have efficient control over the amount of data needs to be read/written. Adding that to fuel was hard and in the end did not give enough control to tweak
Fuel has its own way of managing class names which needs pre-processing which is not suited if you want to use streaming objects to disk
Fuel only supports management of class names because it assumes them to be there in the image. Soil stores class descriptions in the database. Adding that to Fuel wouldn't be rather difficult.
Soil has class versions because as it is a database you can store data in one shape and then change the code. So we need proper migration support which in Fuel is only doable with a central registry and an external version managment
Soil separates content and structure completely, meaning sub-graphs are stored together but everything referencing and addressing is done on the database level, meaning you can deal with the structure without having the code at hand etc.
Soil uses byte type codes for special types to map it efficiently as type-length-value format onto disk. Fuel has no support for how you store objects on disk.
...

Hope this gives a bit more insight than last time. I did not "just wanted to do my own", I considered using Fuel but it didn't work. So as usual there is no one-fits-all solution. And this is a good example.

Norbert