Pharo - database without database
Hello, I was wondering if it is possible for Pharo to load lots of data and query them without using external DB, just using object memory. Well, someting like GemStone/S, but not so sophisticated. I did some research so here is details what I found. I would appreciate if someone can tell me if any points of this agenda can be allready done or need to be programmed, or if this whole idea is waste of time and it is better to use external DB. 1) Object memory limit: Current object memory of Pharo is limited to 4GB. Free version of GemStone/S is also limited to 4GB, so I would rather stick with Pharo until I reach this level. And in the future this limit grow due to 64bit version of VM. 2) Saving to disk: Is it possible to save object or group of objects to disk and ensure consistency? With some transaction mechanism and transaction log? 3) RAM limit: Is it possible to delete objects from memory, move them to the disk and not break its connections to other objects? For example by redirecting object internal ID to some kind of hollow object, that will renew receiver of message from disk if necessary... This approach can also possibly split object memory into more "remote" sources. 4) Indexing: What optimalization for searching current collections in Pharo do? For example: how to store instances of City if I would like to search through it by name and also by its postal code? Is there some library for creating fulltext search (maybe Spotter uses that - I did not study it yet). Have a nice day, and thanks for any answer. Adam
Adam, SandstoneDb may be of interest to you: http://www.smalltalkhub.com/#!/~gnaritas/SandstoneDb "SandstoneDb is a lightweight Prevayler style embedded object database with an ActiveRecord API that doesn't require a command pattern and works for small apps that a single Squeak image can handle. The idea is to make a Squeak image durable and crash proof and suitable for use in small office applications. Data is kept in ram for speed and on disk for safety. All data is reloaded from disk on image startup. Since we're dealing with live objects in memory, concurrency is handled via optional record level critical sections rather than optimistic locking and commit failures. It's up to the developer to use critical sections at the appropriate points by using the critical method on the record. Saves are atomic for an ActiveRecord and all its non ActiveRecord children, for example, an order and its items. There is no atomic save across multiple ActiveRecords. A record is a cluster of objects that are stored in a single file together. After installing SandstoneDb make sure to restart your image before attempting to use it or running the tests." Fuel may also have something in stock: https://ci.inria.fr/pharo-contribution/view/Books/job/EnterprisePharoBook/la... I also use Mapless http://sebastianconcept.github.io/Mapless/ HTH Phil On Sun, Sep 27, 2015 at 6:47 PM, Adam <adam@234.cz> wrote:
Hello,
I was wondering if it is possible for Pharo to load lots of data and query them without using external DB, just using object memory. Well, someting like GemStone/S, but not so sophisticated.
I did some research so here is details what I found. I would appreciate if someone can tell me if any points of this agenda can be allready done or need to be programmed, or if this whole idea is waste of time and it is better to use external DB.
1) Object memory limit: Current object memory of Pharo is limited to 4GB. Free version of GemStone/S is also limited to 4GB, so I would rather stick with Pharo until I reach this level. And in the future this limit grow due to 64bit version of VM. 2) Saving to disk: Is it possible to save object or group of objects to disk and ensure consistency? With some transaction mechanism and transaction log? 3) RAM limit: Is it possible to delete objects from memory, move them to the disk and not break its connections to other objects? For example by redirecting object internal ID to some kind of hollow object, that will renew receiver of message from disk if necessary... This approach can also possibly split object memory into more "remote" sources. 4) Indexing: What optimalization for searching current collections in Pharo do? For example: how to store instances of City if I would like to search through it by name and also by its postal code? Is there some library for creating fulltext search (maybe Spotter uses that - I did not study it yet).
Have a nice day, and thanks for any answer.
Adam
Hello, thank You for great hints. I will look at this. Have a nice day, Adam. Dne Ne 27. záÅà 2015 19:16:10, phil@highoctane.be napsal(a):
Adam,
SandstoneDb may be of interest to you:
http://www.smalltalkhub.com/#!/~gnaritas/SandstoneDb "SandstoneDb is a lightweight Prevayler style embedded object database with an ActiveRecord API that doesn't require a command pattern and works for small apps that a single Squeak image can handle. The idea is to make a Squeak image durable and crash proof and suitable for use in small office applications.
Data is kept in ram for speed and on disk for safety. All data is reloaded from disk on image startup.
Since we're dealing with live objects in memory, concurrency is handled via optional record level critical sections rather than optimistic locking and commit failures. It's up to the developer to use critical sections at the appropriate points by using the critical method on the record.
Saves are atomic for an ActiveRecord and all its non ActiveRecord children, for example, an order and its items. There is no atomic save across multiple ActiveRecords. A record is a cluster of objects that are stored in a single file together.
After installing SandstoneDb make sure to restart your image before attempting to use it or running the tests."
Fuel may also have something in stock:
https://ci.inria.fr/pharo-contribution/view/Books/job/EnterprisePharoBook/la stSuccessfulBuild/artifact/book-result/Fuel/Fuel.pdf
I also use Mapless
http://sebastianconcept.github.io/Mapless/
HTH
Phil
On Sun, Sep 27, 2015 at 6:47 PM, Adam <adam@234.cz> wrote:
Hello,
I was wondering if it is possible for Pharo to load lots of data and query them without using external DB, just using object memory. Well, someting like GemStone/S, but not so sophisticated.
I did some research so here is details what I found. I would appreciate if someone can tell me if any points of this agenda can be allready done or need to be programmed, or if this whole idea is waste of time and it is better to use external DB.
1) Object memory limit: Current object memory of Pharo is limited to 4GB. Free version of GemStone/S is also limited to 4GB, so I would rather stick with Pharo until I reach this level. And in the future this limit grow due to 64bit version of VM. 2) Saving to disk: Is it possible to save object or group of objects to disk and ensure consistency? With some transaction mechanism and transaction log? 3) RAM limit: Is it possible to delete objects from memory, move them to the disk and not break its connections to other objects? For example by redirecting object internal ID to some kind of hollow object, that will renew receiver of message from disk if necessary... This approach can also possibly split object memory into more "remote" sources. 4) Indexing: What optimalization for searching current collections in Pharo do? For example: how to store instances of City if I would like to search through it by name and also by its postal code? Is there some library for creating fulltext search (maybe Spotter uses that - I did not study it yet).
Have a nice day, and thanks for any answer.
Adam
If I'm correct we wrote about it on the seaside book http://book.seaside.st Le 27/9/15 20:23, Adam a écrit :
Hello,
thank You for great hints. I will look at this.
Have a nice day,
Adam.
Dne Ne 27. záÅà 2015 19:16:10, phil@highoctane.be napsal(a):
Adam,
SandstoneDb may be of interest to you:
http://www.smalltalkhub.com/#!/~gnaritas/SandstoneDb "SandstoneDb is a lightweight Prevayler style embedded object database with an ActiveRecord API that doesn't require a command pattern and works for small apps that a single Squeak image can handle. The idea is to make a Squeak image durable and crash proof and suitable for use in small office applications.
Data is kept in ram for speed and on disk for safety. All data is reloaded from disk on image startup.
Since we're dealing with live objects in memory, concurrency is handled via optional record level critical sections rather than optimistic locking and commit failures. It's up to the developer to use critical sections at the appropriate points by using the critical method on the record.
Saves are atomic for an ActiveRecord and all its non ActiveRecord children, for example, an order and its items. There is no atomic save across multiple ActiveRecords. A record is a cluster of objects that are stored in a single file together.
After installing SandstoneDb make sure to restart your image before attempting to use it or running the tests."
Fuel may also have something in stock:
https://ci.inria.fr/pharo-contribution/view/Books/job/EnterprisePharoBook/la stSuccessfulBuild/artifact/book-result/Fuel/Fuel.pdf
I also use Mapless
http://sebastianconcept.github.io/Mapless/
HTH
Phil
On Sun, Sep 27, 2015 at 6:47 PM, Adam <adam@234.cz> wrote:
Hello,
I was wondering if it is possible for Pharo to load lots of data and query them without using external DB, just using object memory. Well, someting like GemStone/S, but not so sophisticated.
I did some research so here is details what I found. I would appreciate if someone can tell me if any points of this agenda can be allready done or need to be programmed, or if this whole idea is waste of time and it is better to use external DB.
1) Object memory limit: Current object memory of Pharo is limited to 4GB. Free version of GemStone/S is also limited to 4GB, so I would rather stick with Pharo until I reach this level. And in the future this limit grow due to 64bit version of VM. 2) Saving to disk: Is it possible to save object or group of objects to disk and ensure consistency? With some transaction mechanism and transaction log? 3) RAM limit: Is it possible to delete objects from memory, move them to the disk and not break its connections to other objects? For example by redirecting object internal ID to some kind of hollow object, that will renew receiver of message from disk if necessary... This approach can also possibly split object memory into more "remote" sources. 4) Indexing: What optimalization for searching current collections in Pharo do? For example: how to store instances of City if I would like to search through it by name and also by its postal code? Is there some library for creating fulltext search (maybe Spotter uses that - I did not study it yet).
Have a nice day, and thanks for any answer.
Adam
Hello, it is true: http://book.seaside.st/book/advanced/persistency Thank You very much. Adam Dne Po 28. záÅà 2015 08:50:28, stepharo napsal(a):
If I'm correct we wrote about it on the seaside book http://book.seaside.st
Le 27/9/15 20:23, Adam a écrit :
Hello,
thank You for great hints. I will look at this.
Have a nice day,
Adam.
Dne Ne 27. záÅà 2015 19:16:10, phil@highoctane.be napsal(a):
Adam,
SandstoneDb may be of interest to you:
http://www.smalltalkhub.com/#!/~gnaritas/SandstoneDb "SandstoneDb is a lightweight Prevayler style embedded object database with an ActiveRecord API that doesn't require a command pattern and works for small apps that a single Squeak image can handle. The idea is to make a Squeak image durable and crash proof and suitable for use in small office applications.
Data is kept in ram for speed and on disk for safety. All data is reloaded from disk on image startup.
Since we're dealing with live objects in memory, concurrency is handled via optional record level critical sections rather than optimistic locking and commit failures. It's up to the developer to use critical sections at the appropriate points by using the critical method on the record.
Saves are atomic for an ActiveRecord and all its non ActiveRecord children, for example, an order and its items. There is no atomic save across multiple ActiveRecords. A record is a cluster of objects that are stored in a single file together.
After installing SandstoneDb make sure to restart your image before attempting to use it or running the tests."
Fuel may also have something in stock:
https://ci.inria.fr/pharo-contribution/view/Books/job/EnterprisePharoBook /la stSuccessfulBuild/artifact/book-result/Fuel/Fuel.pdf
I also use Mapless
http://sebastianconcept.github.io/Mapless/
HTH
Phil
On Sun, Sep 27, 2015 at 6:47 PM, Adam <adam@234.cz> wrote:
Hello,
I was wondering if it is possible for Pharo to load lots of data and query them without using external DB, just using object memory. Well, someting like GemStone/S, but not so sophisticated.
I did some research so here is details what I found. I would appreciate if someone can tell me if any points of this agenda can be allready done or need to be programmed, or if this whole idea is waste of time and it is better to use external DB.
1) Object memory limit: Current object memory of Pharo is limited to 4GB. Free version of GemStone/S is also limited to 4GB, so I would rather stick with Pharo until I reach this level. And in the future this limit grow due to 64bit version of VM. 2) Saving to disk: Is it possible to save object or group of objects to disk and ensure consistency? With some transaction mechanism and transaction log? 3) RAM limit: Is it possible to delete objects from memory, move them to the disk and not break its connections to other objects? For example by redirecting object internal ID to some kind of hollow object, that will renew receiver of message from disk if necessary... This approach can also possibly split object memory into more "remote" sources. 4) Indexing: What optimalization for searching current collections in Pharo do? For example: how to store instances of City if I would like to search through it by name and also by its postal code? Is there some library for creating fulltext search (maybe Spotter uses that - I did not study it yet).
Have a nice day, and thanks for any answer.
Adam
On 27-09-15 18:47, Adam wrote:
1) Object memory limit: Current object memory of Pharo is limited to 4GB. Free version of GemStone/S is also limited to 4GB, so I would rather stick with Pharo until I reach this level. And in the future this limit grow due to 64bit version of VM.
The information on GemStone seems incorrect. https://gemtalksystems.com/licensing/ What I read there is that the Limited (email address required) license supports 50 GB of (on-disk) objects, with a 2GB shared page cache. Stephan
On Mon, Sep 28, 2015 at 7:17 AM, Stephan Eggermont <stephan@stack.nl> wrote:
On 27-09-15 18:47, Adam wrote:
1) Object memory limit: Current object memory of Pharo is limited to 4GB. Free version of GemStone/S is also limited to 4GB, so I would rather stick with Pharo until I reach this level. And in the future this limit grow due to 64bit version of VM.
The information on GemStone seems incorrect. https://gemtalksystems.com/licensing/ What I read there is that the Limited (email address required) license supports 50 GB of (on-disk) objects, with a 2GB shared page cache.
On the one hand, the license has changed starting in GemStone version 3.2. On the other hand, the "memory" limit you are talking about is called Share Page Cache and it's more a "soft" limit in the sense that is simply a cache of the persistent objects brought from disk. If you have a small "cache" the performance will be bad because of trashing (objects going and coming back from disk) but it doesn't mean the program won't be able to run because it doesn't fit. Free GemStone version involves a 2GB SPC. That is normally enough for small/medium apps. That is completely different as the "memory" limit for Pharo. -- Mariano http://marianopeck.wordpress.com
Mariano soon with spur 64bits we will see :) Le 28/9/15 14:57, Mariano Martinez Peck a écrit :
On Mon, Sep 28, 2015 at 7:17 AM, Stephan Eggermont <stephan@stack.nl <mailto:stephan@stack.nl>> wrote:
On 27-09-15 18:47, Adam wrote:
1) Object memory limit: Current object memory of Pharo is limited to 4GB. Free version of GemStone/S is also limited to 4GB, so I would rather stick with Pharo until I reach this level. And in the future this limit grow due to 64bit version of VM.
The information on GemStone seems incorrect. https://gemtalksystems.com/licensing/ What I read there is that the Limited (email address required) license supports 50 GB of (on-disk) objects, with a 2GB shared page cache.
On the one hand, the license has changed starting in GemStone version 3.2. On the other hand, the "memory" limit you are talking about is called Share Page Cache and it's more a "soft" limit in the sense that is simply a cache of the persistent objects brought from disk. If you have a small "cache" the performance will be bad because of trashing (objects going and coming back from disk) but it doesn't mean the program won't be able to run because it doesn't fit.
Free GemStone version involves a 2GB SPC. That is normally enough for small/medium apps.
That is completely different as the "memory" limit for Pharo.
-- Mariano http://marianopeck.wordpress.com
Hello, Yes, I was confused by some old discussion. New licence is different. Thanks for correction. Adam. Dne Po 28. záÅà 2015 12:17:19, Stephan Eggermont napsal(a):
On 27-09-15 18:47, Adam wrote:
1) Object memory limit: Current object memory of Pharo is limited to 4GB. Free version of GemStone/S is also limited to 4GB, so I would rather stick with Pharo until I reach this level. And in the future this limit grow due to 64bit version of VM.
The information on GemStone seems incorrect. https://gemtalksystems.com/licensing/ What I read there is that the Limited (email address required) license supports 50 GB of (on-disk) objects, with a 2GB shared page cache.
Stephan
Adam, Keep an eye on the GLASS list[1] for an upcoming (within 2 months - I promise:) announcement about GsDevKit_home (not to be confused with gsDevKitHome:)... as it will provide much needed support for creating/starting/managing multiple GemStone stones.... If you are interested in working with GemStone:) Dale [1] http://forum.world.st/GLASS-f1460844.html On 09/28/2015 12:48 PM, Adam wrote:
Hello,
Yes, I was confused by some old discussion. New licence is different. Thanks for correction.
Adam.
Dne Po 28. záÅà 2015 12:17:19, Stephan Eggermont napsal(a):
On 27-09-15 18:47, Adam wrote:
1) Object memory limit: Current object memory of Pharo is limited to 4GB. Free version of GemStone/S is also limited to 4GB, so I would rather stick with Pharo until I reach this level. And in the future this limit grow due to 64bit version of VM. The information on GemStone seems incorrect. https://gemtalksystems.com/licensing/ What I read there is that the Limited (email address required) license supports 50 GB of (on-disk) objects, with a 2GB shared page cache.
Stephan
Hello, looking forward to use it :) Adam. Dne Po 28. záÅà 2015 16:38:58, Dale Henrichs napsal(a):
Adam,
Keep an eye on the GLASS list[1] for an upcoming (within 2 months - I promise:) announcement about GsDevKit_home (not to be confused with gsDevKitHome:)... as it will provide much needed support for creating/starting/managing multiple GemStone stones.... If you are interested in working with GemStone:)
Dale
[1] http://forum.world.st/GLASS-f1460844.html
On 09/28/2015 12:48 PM, Adam wrote:
Hello,
Yes, I was confused by some old discussion. New licence is different. Thanks for correction.
Adam.
Dne Po 28. záÅà 2015 12:17:19, Stephan Eggermont napsal(a):
On 27-09-15 18:47, Adam wrote:
1) Object memory limit: Current object memory of Pharo is limited to 4GB. Free version of GemStone/S is also limited to 4GB, so I would rather stick with Pharo until I reach this level. And in the future this limit grow due to 64bit version of VM.
The information on GemStone seems incorrect. https://gemtalksystems.com/licensing/ What I read there is that the Limited (email address required) license supports 50 GB of (on-disk) objects, with a 2GB shared page cache.
Stephan
participants (7)
-
Adam -
adam＠234.cz
-
Dale Henrichs -
Mariano Martinez Peck -
phil＠highoctane.be -
Stephan Eggermont -
stepharo