[Pharo-project] Fuel serialization - feature/bug
In my test case for serializing a Pier kernel with Fuel, a FSMemoryStore (from the FileSystem package) is used. The FSMemoryStore holds the "file" bytes in a ByteArray. During serialization, I got a stack trace with KeyNotFound due to FLSerializer>>nextPutReferenceTo: when the object is the "file" ByteArray. Since this ByteArray is where the serialization is being written to, then it's hash value is changing, so Fuel serialization cannot find it in the instancesIndex. I think this is expected behaviour, and the fix is to not serialize certain inst vars. I was using the "stable" 1.4 version. Do I need to change to the 1.5 baseline to get the #fuelIgnoredInstanceVariableNames feature? -- Yanni
On Mon, Jun 13, 2011 at 11:28 PM, Yanni Chiu <yanni@rogers.com> wrote:
In my test case for serializing a Pier kernel with Fuel, a FSMemoryStore (from the FileSystem package) is used.
I really didn't understand, sorry :( Why the following is not enought? (FLSerializer on: myStream) serialize: myPRKernel. and (FLMaterializer on: myStream) materialize
The FSMemoryStore holds the "file" bytes in a ByteArray. During serialization, I got a stack trace with KeyNotFound due to FLSerializer>>nextPutReferenceTo: when the object is the "file" ByteArray.
Since this ByteArray is where the serialization is being written to,
mmmmmm so you are using a stream to serialize and such stream is also being refenced from the object to serialize?
then it's hash value is changing, so Fuel serialization cannot find it in the instancesIndex.
I think this is expected behaviour, and the fix is to not serialize certain inst vars. I was using the "stable" 1.4 version. Do I need to change to the 1.5 baseline to get the #fuelIgnoredInstanceVariableNames feature?
-- Yanni
-- Mariano http://marianopeck.wordpress.com
On 13/06/11 5:49 PM, Mariano Martinez Peck wrote:
Why the following is not enought?
(FLSerializer on: myStream) serialize: myPRKernel.
and
(FLMaterializer on: myStream) materialize
Yes, that's pretty much the code. Except that it's in a test case that uses FSMemoryStore. The tests run a lot faster when writing to memory, instead of to disk.
The FSMemoryStore holds the "file" bytes in a ByteArray. During serialization, I got a stack trace with KeyNotFound due to FLSerializer>>nextPutReferenceTo: when the object is the "file" ByteArray.
Since this ByteArray is where the serialization is being written to,
mmmmmm so you are using a stream to serialize and such stream is also being refenced from the object to serialize?
Yes, that's it - except that the stream is an FSWriteStream on a FSMemoryStore "file", which I think is an issue here. The PRKernel is holding an instance of PRFuelPersistency, which holds an FSReference. That FSReference happens to refer to a memory-based "file".
On 13/06/11 6:56 PM, Yanni Chiu wrote:
On 13/06/11 5:49 PM, Mariano Martinez Peck wrote:
mmmmmm so you are using a stream to serialize and such stream is also being refenced from the object to serialize?
Yes, that's it - except that the stream is an FSWriteStream on a FSMemoryStore "file", which I think is an issue here. The PRKernel is holding an instance of PRFuelPersistency, which holds an FSReference. That FSReference happens to refer to a memory-based "file".
The code is at http://ss3.gemstone.com/ss/pierfuel.html You'll first need a Pier image, with the following loaded: (ConfigurationOfFilesystem project version: '2.0.1-baseline') load (ConfigurationOfFuel project version: '1.4') load OR (ConfigurationOfFuel project version: '1.5-baseline') load If you run PRFuelPersistencyTest, then you'll see the stack trace. I saw the same problem with 1.4 and with 1.5-baseline. In the 1.5-baseline, I added #fuelIgnoredInstanceVariableNames to PRFuelPersistency and FSMemoryStore (to ignore 'directory' and 'root', respectively) but the same problem remained. So I think it's a bug. Despite the test failing, the PRFuelPersistency works as expected to save/restore a Pier kernel to/from a file. The file is 30 times smaller than the corresponding (uncompressed) SIXX file.
On Tue, Jun 14, 2011 at 8:12 AM, Yanni Chiu <yanni@rogers.com> wrote:
On 13/06/11 6:56 PM, Yanni Chiu wrote:
On 13/06/11 5:49 PM, Mariano Martinez Peck wrote:
mmmmmm so you are using a stream to serialize and such stream is also being refenced from the object to serialize?
Yes, that's it - except that the stream is an FSWriteStream on a FSMemoryStore "file", which I think is an issue here. The PRKernel is holding an instance of PRFuelPersistency, which holds an FSReference. That FSReference happens to refer to a memory-based "file".
The code is at http://ss3.gemstone.com/ss/pierfuel.html
You'll first need a Pier image, with the following loaded:
(ConfigurationOfFilesystem project version: '2.0.1-baseline') load
(ConfigurationOfFuel project version: '1.4') load OR (ConfigurationOfFuel project version: '1.5-baseline') load
If you run PRFuelPersistencyTest, then you'll see the stack trace. I saw the same problem with 1.4 and with 1.5-baseline.
In the 1.5-baseline, I added #fuelIgnoredInstanceVariableNames to PRFuelPersistency and FSMemoryStore (to ignore 'directory' and 'root', respectively) but the same problem remained. So I think it's a bug.
Yes, We think you have spot a bug :) Martin is working on that. We will let you know. Thanks
Despite the test failing, the PRFuelPersistency works as expected to save/restore a Pier kernel to/from a file. The file is 30 times smaller than the corresponding (uncompressed) SIXX file.
and speed? could you measure serialization / materialization time ? Thanks a lot, -- Mariano http://marianopeck.wordpress.com
On Tue, Jun 14, 2011 at 9:37 AM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Tue, Jun 14, 2011 at 8:12 AM, Yanni Chiu <yanni@rogers.com> wrote:
On 13/06/11 6:56 PM, Yanni Chiu wrote:
On 13/06/11 5:49 PM, Mariano Martinez Peck wrote:
mmmmmm so you are using a stream to serialize and such stream is also being refenced from the object to serialize?
Yes, that's it - except that the stream is an FSWriteStream on a FSMemoryStore "file", which I think is an issue here. The PRKernel is holding an instance of PRFuelPersistency, which holds an FSReference. That FSReference happens to refer to a memory-based "file".
The code is at http://ss3.gemstone.com/ss/pierfuel.html
You'll first need a Pier image, with the following loaded:
(ConfigurationOfFilesystem project version: '2.0.1-baseline') load
(ConfigurationOfFuel project version: '1.4') load OR (ConfigurationOfFuel project version: '1.5-baseline') load
If you run PRFuelPersistencyTest, then you'll see the stack trace. I saw the same problem with 1.4 and with 1.5-baseline.
In the 1.5-baseline, I added #fuelIgnoredInstanceVariableNames to PRFuelPersistency and FSMemoryStore
BTW you put them as class side methods, don't you?
(to ignore 'directory' and 'root', respectively) but the same problem
remained. So I think it's a bug.
Yes, We think you have spot a bug :) Martin is working on that. We will let you know. Thanks
Despite the test failing, the PRFuelPersistency works as expected to save/restore a Pier kernel to/from a file. The file is 30 times smaller than the corresponding (uncompressed) SIXX file.
and speed? could you measure serialization / materialization time ?
Thanks a lot,
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
On 14/06/11 3:37 AM, Mariano Martinez Peck wrote:
and speed? could you measure serialization / materialization time ?
The kernel was just a few pages, and SIXX took about 5 sec to save to disk, and a slight pause to materialize. With Fuel, it was all done almost instantaneously. I don't have any benchmarks ... now added to TODO list.
Hi Yanni On Tue, Jun 14, 2011 at 3:12 AM, Yanni Chiu <yanni@rogers.com> wrote:
On 13/06/11 6:56 PM, Yanni Chiu wrote:
On 13/06/11 5:49 PM, Mariano Martinez Peck wrote:
mmmmmm so you are using a stream to serialize and such stream is also being refenced from the object to serialize?
Yes, that's it - except that the stream is an FSWriteStream on a FSMemoryStore "file", which I think is an issue here. The PRKernel is holding an instance of PRFuelPersistency, which holds an FSReference. That FSReference happens to refer to a memory-based "file".
The code is at http://ss3.gemstone.com/ss/pierfuel.html
You'll first need a Pier image, with the following loaded:
(ConfigurationOfFilesystem project version: '2.0.1-baseline') load
(ConfigurationOfFuel project version: '1.4') load OR (ConfigurationOfFuel project version: '1.5-baseline') load
If you run PRFuelPersistencyTest, then you'll see the stack trace. I saw the same problem with 1.4 and with 1.5-baseline.
In the 1.5-baseline, I added #fuelIgnoredInstanceVariableNames to PRFuelPersistency and FSMemoryStore (to ignore 'directory' and 'root', respectively) but the same problem remained. So I think it's a bug.
I didn't try in the Pier image, but I hope your problem is fixed with the last version (Fuel-MartinDias.259). serialize the stream where you are serializing is a test case I had never thought! BTW, is it fine to get nil on these variables? I mean, is the FSMemoryStore materialized useful?
Despite the test failing, the PRFuelPersistency works as expected to save/restore a Pier kernel to/from a file. The file is 30 times smaller than the corresponding (uncompressed) SIXX file.
good! so it can be useful for scenarios where you don't need to be cross-dialect or to have XML format.
So..using (Fuel-MartinDias.259) I added FSMemoryStore class >> fuelIgnoredInstanceVariableNames ^#(root) PRFuelPersistency class >> fuelIgnoredInstanceVariableNames ^#(directory) The tests are all green. Cheers On Tue, Jun 14, 2011 at 10:15 AM, Martin Dias <tinchodias@gmail.com> wrote:
Hi Yanni
On Tue, Jun 14, 2011 at 3:12 AM, Yanni Chiu <yanni@rogers.com> wrote:
On 13/06/11 6:56 PM, Yanni Chiu wrote:
On 13/06/11 5:49 PM, Mariano Martinez Peck wrote:
mmmmmm so you are using a stream to serialize and such stream is also being refenced from the object to serialize?
Yes, that's it - except that the stream is an FSWriteStream on a FSMemoryStore "file", which I think is an issue here. The PRKernel is holding an instance of PRFuelPersistency, which holds an FSReference. That FSReference happens to refer to a memory-based "file".
The code is at http://ss3.gemstone.com/ss/pierfuel.html
You'll first need a Pier image, with the following loaded:
(ConfigurationOfFilesystem project version: '2.0.1-baseline') load
(ConfigurationOfFuel project version: '1.4') load OR (ConfigurationOfFuel project version: '1.5-baseline') load
If you run PRFuelPersistencyTest, then you'll see the stack trace. I saw the same problem with 1.4 and with 1.5-baseline.
In the 1.5-baseline, I added #fuelIgnoredInstanceVariableNames to PRFuelPersistency and FSMemoryStore (to ignore 'directory' and 'root', respectively) but the same problem remained. So I think it's a bug.
I didn't try in the Pier image, but I hope your problem is fixed with the last version (Fuel-MartinDias.259).
serialize the stream where you are serializing is a test case I had never thought!
BTW, is it fine to get nil on these variables? I mean, is the FSMemoryStore materialized useful?
Despite the test failing, the PRFuelPersistency works as expected to save/restore a Pier kernel to/from a file. The file is 30 times smaller than the corresponding (uncompressed) SIXX file.
good! so it can be useful for scenarios where you don't need to be cross-dialect or to have XML format.
-- Mariano http://marianopeck.wordpress.com
On 14/06/11 4:41 AM, Mariano Martinez Peck wrote:
PRFuelPersistency class >> fuelIgnoredInstanceVariableNames ^#(directory)
Using either one, the test now passes when using Fuel-MartinDias.259. Only one of the #fuelIgnoredInstanceVariableNames is needed. I'll go with the above one, so that FSMemoryStore does not have to be extended.
On Tue, Jun 14, 2011 at 6:09 PM, Yanni Chiu <yanni@rogers.com> wrote:
On 14/06/11 4:41 AM, Mariano Martinez Peck wrote:
PRFuelPersistency class >> fuelIgnoredInstanceVariableNames ^#(directory)
Using either one, the test now passes when using Fuel-MartinDias.259. Only one of the #fuelIgnoredInstanceVariableNames is needed.
Excellent.
I'll go with the above one, so that FSMemoryStore does not have to be extended.
+1 -- Mariano http://marianopeck.wordpress.com
Yanni. The tests are green, but I have an error while trying to persist. If I go to: http://localhost:8888/pier/system/management/persistency Choose Fuel, and click on "snapshot" I have a key not found error. Do you have the same or is it working for you? Thanks On Wed, Jun 15, 2011 at 12:01 PM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Tue, Jun 14, 2011 at 6:09 PM, Yanni Chiu <yanni@rogers.com> wrote:
On 14/06/11 4:41 AM, Mariano Martinez Peck wrote:
PRFuelPersistency class >> fuelIgnoredInstanceVariableNames ^#(directory)
Using either one, the test now passes when using Fuel-MartinDias.259. Only one of the #fuelIgnoredInstanceVariableNames is needed.
Excellent.
I'll go with the above one, so that FSMemoryStore does not have to be extended.
+1
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
Sorry, one more question. One you did seems to be a persistency strategy for Pier, but I thought it was the import/export utility: http://www.lukas-renggli.ch/blog/export-import But it seems I was wrong. There is no way so that I can use that import/export utility with Fuel also? does that require extra effort or it can use what you did ? Thanks! On Thu, Jun 16, 2011 at 10:24 AM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
Yanni. The tests are green, but I have an error while trying to persist. If I go to:
http://localhost:8888/pier/system/management/persistency
Choose Fuel, and click on "snapshot" I have a key not found error. Do you have the same or is it working for you?
Thanks
On Wed, Jun 15, 2011 at 12:01 PM, Mariano Martinez Peck < marianopeck@gmail.com> wrote:
On Tue, Jun 14, 2011 at 6:09 PM, Yanni Chiu <yanni@rogers.com> wrote:
On 14/06/11 4:41 AM, Mariano Martinez Peck wrote:
PRFuelPersistency class >> fuelIgnoredInstanceVariableNames ^#(directory)
Using either one, the test now passes when using Fuel-MartinDias.259. Only one of the #fuelIgnoredInstanceVariableNames is needed.
Excellent.
I'll go with the above one, so that FSMemoryStore does not have to be extended.
+1
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
-- Mariano http://marianopeck.wordpress.com
On 16/06/11 4:44 AM, Mariano Martinez Peck wrote:
Sorry, one more question. One you did seems to be a persistency strategy for Pier, but I thought it was the import/export utility:
http://www.lukas-renggli.ch/blog/export-import
But it seems I was wrong. There is no way so that I can use that import/export utility with Fuel also? does that require extra effort or it can use what you did ?
What I did, had more to do with implementing a persistency strategy than with serialization. I took a persistency strategy I'd written, that was using SIXX, and changed it to use Fuel. The changes were very minor - just changed the serialize/deserialize methods. I imagine doing the same for the import/export would be just as straightforward. I'm planning to use Fuel for an import/export facility, but I'm trying to export only a portion of a kernel. Since a Pier kernel's objects are highly interconnected then it's a challenge. On the import side of things, one problem is that the transaction log with be voided (i.e. you would be able to reconstruct the kernel from the log - but maybe that's already a problem when you import a whole kernel).
On 16/06/11 4:24 AM, Mariano Martinez Peck wrote:
Yanni. The tests are green, but I have an error while trying to persist. If I go to:
http://localhost:8888/pier/system/management/persistency
Choose Fuel, and click on "snapshot" I have a key not found error. Do you have the same or is it working for you?
That worked, the one time I tried it. I'm still integrating the code, so I don't have a clear answer at the moment. My system is set up to read in the saved kernel on image startup, so it may not be obvious how to use PRFuelPersistency, in its present form.
On Tue, Jun 14, 2011 at 12:56 AM, Yanni Chiu <yanni@rogers.com> wrote:
On 13/06/11 5:49 PM, Mariano Martinez Peck wrote:
Why the following is not enought?
(FLSerializer on: myStream) serialize: myPRKernel.
and
(FLMaterializer on: myStream) materialize
Yes, that's pretty much the code. Except that it's in a test case that uses FSMemoryStore. The tests run a lot faster when writing to memory, instead of to disk.
Yes, but if you want to serialize in memory you can also use for example: myStream := (MultiByteBinaryOrTextStream on: ''). (FLSerializer on: myStream) serialize: myPRKernel. ... myStream reset. (FLMaterializer on: myStream) materialize That would use a stream in memory. And even more, you can serialize in memory with Fuel this way: testSerializationAndMaterializationInMemory | byteArray materializedObject | byteArray := FLSerializer serializeInMemory: 'mariano'. materializedObject := FLMaterializer materializeFromByteArray: byteArray. self assert: materializedObject equals: 'mariano'.
The FSMemoryStore holds the "file" bytes in a ByteArray. During
serialization, I got a stack trace with KeyNotFound due to FLSerializer>>nextPutReferenceTo: when the object is the "file" ByteArray.
Since this ByteArray is where the serialization is being written to,
mmmmmm so you are using a stream to serialize and such stream is also being refenced from the object to serialize?
Yes, that's it - except that the stream is an FSWriteStream on a FSMemoryStore "file", which I think is an issue here. The PRKernel is holding an instance of PRFuelPersistency, which holds an FSReference. That FSReference happens to refer to a memory-based "file".
-- Mariano http://marianopeck.wordpress.com
On 14/06/11 3:35 AM, Mariano Martinez Peck wrote:
Yes, but if you want to serialize in memory you can also use for example:
myStream := (MultiByteBinaryOrTextStream on: ''). (FLSerializer on: myStream) serialize: myPRKernel.
At the time, I wanted to use the FileSystem protocol. It was a struggle to understand how to use FileSystem. For tests of code that don't involve the file system in the public API, I could avoid the "complexity" of FileSystem, and use the streams directly (as in your example).
participants (3)
-
Mariano Martinez Peck -
Martin Dias -
Yanni Chiu