SIXX on "large" files

newer
Help us to have really cool Pharo...

older
Re: [Pharo-dev] SIXX on "large"...

Sebastian Sastre

Jan. 3, 2014

7:59 p.m.

Hi guys, anyone here using SIXX? I'm having a consistent problem in opening a SixxReadStream on a file that has ~10MB of serialized objects. Smaller than that is all good Bigger, raises SixxXmlParseError ideas? Maybe I'm using the wrong version of XMLDOMParser? patches? workarounds? sebastian o/ PD: sixx was loaded with: Gofer it squeaksource: 'MetacelloRepository'; package: 'ConfigurationOfSIXX'; load. (Smalltalk at: #ConfigurationOfSIXX) project latestVersion load.

Attachments:

attachment.htm (text/html — 3.9 KB)

Show replies by date

Sebastian Sastre

Jan. 3, 2014

8:07 p.m.

It's probable unrelated to size as expected. For the record: I've changed the XMLParser and the new one complained about a problem on a specific line that has this content <sixx.object sixx.id="1106" sixx.name="key" sixx.type="String" >name</sixx.object> <sixx.object sixx.id="1107" sixx.name="value" sixx.type="String" >��L��i��k��e�� M��a��r��k�� T��w��a��i��n��:�� I��'��v��e�� n��e��v��e��r�� l��e��t�� m��y�� s��c��h��o��o��l�� i��n��t��e��r��f��e��r��e�� w��i��t��h�� m��y�� e��d��u��c��a��t��i��o��n��.�� </sixx.object> </sixx.object> looks like an encoding problem On Jan 3, 2014, at 5:59 PM, Sebastian Sastre <SEBASTIAN@FLOWINGCONCEPT.COM> wrote:

...

Hi guys,

anyone here using SIXX?

I'm having a consistent problem in opening a SixxReadStream on a file that has ~10MB of serialized objects.

Smaller than that is all good

Bigger, raises SixxXmlParseError

ideas?

Maybe I'm using the wrong version of XMLDOMParser?

patches? workarounds?

sebastian

o/

PD: sixx was loaded with:

Gofer it squeaksource: 'MetacelloRepository'; package: 'ConfigurationOfSIXX'; load.

(Smalltalk at: #ConfigurationOfSIXX) project latestVersion load.

StÃ©phane Ducasse

9:43 p.m.

On 03 Jan 2014, at 21:07, Sebastian Sastre <sebastian@flowingconcept.com> wrote:

...

It's probable unrelated to size as expected.

For the record:

I've changed the XMLParser

how? I do not know if SIxx fixes the version of the XML parser it uses.

...

and the new one complained about a problem on a specific line that has this content

<sixx.object sixx.id="1106" sixx.name="key" sixx.type="String" >name</sixx.object> <sixx.object sixx.id="1107" sixx.name="value" sixx.type="String" >��L��i��k��e�� M��a��r��k�� T��w��a��i��n��:�� I��'��v��e�� n��e��v��e��r�� l��e��t�� m��y�� s��c��h��o��o��l�� i��n��t��e��r��f��e��r��e�� w��i��t��h�� m��y�� e��d��u��c��a��t��i��o��n��.�� </sixx.object> </sixx.object>

looks like an encoding problem

On Jan 3, 2014, at 5:59 PM, Sebastian Sastre <SEBASTIAN@FLOWINGCONCEPT.COM> wrote:

...
Hi guys,

anyone here using SIXX?

I'm having a consistent problem in opening a SixxReadStream on a file that has ~10MB of serialized objects.

Smaller than that is all good

Bigger, raises SixxXmlParseError

ideas?

Maybe I'm using the wrong version of XMLDOMParser?

patches? workarounds?

sebastian

o/

PD: sixx was loaded with:

Gofer it squeaksource: 'MetacelloRepository'; package: 'ConfigurationOfSIXX'; load.

(Smalltalk at: #ConfigurationOfSIXX) project latestVersion load.

Sebastian Sastre

11:35 p.m.

On Jan 3, 2014, at 7:43 PM, StÃ©phane Ducasse <stephane.ducasse@inria.fr> wrote:

...

...
I've changed the XMLParser

how?

With this guy: ConfigurationOfXMLParser load. from MetaRepoForPharo30

Frank Shearar

10:01 p.m.

On 3 January 2014 20:07, Sebastian Sastre <sebastian@flowingconcept.com> wrote:

...

It's probable unrelated to size as expected.

For the record:

I've changed the XMLParser and the new one complained about a problem on a specific line that has this content

<sixx.object sixx.id="1106" sixx.name="key" sixx.type="String"

...
name</sixx.object> <sixx.object sixx.id="1107" sixx.name="value" sixx.type="String" ��L��i��k��e�� M��a��r��k�� T��w��a��i��n��:�� I��'��v��e�� n��e��v��e��r�� l��e��t�� m��y�� s��c��h��o��o��l�� i��n��t��e��r��f��e��r��e�� w��i��t��h�� m��y�� e��d��u��c��a��t��i��o��n��.�� </sixx.object> </sixx.object>

looks like an encoding problem

Was that originally a UTF-32 string or something? Three NULs followed by a character... (For the curious, it says 'Like Mark Twain: "I've never let my school interfere with my education. "') frank

Sebastian Sastre

11:38 p.m.

I've discovered that its part of a JsonObject that holds the facebook profile associated with the user. SIXX is cool but I'm with the feeling that it's time to give fuel a chance On Jan 3, 2014, at 8:01 PM, Frank Shearar <frank.shearar@gmail.com> wrote:

...

On 3 January 2014 20:07, Sebastian Sastre <sebastian@flowingconcept.com> wrote:

...
It's probable unrelated to size as expected.

For the record:

I've changed the XMLParser and the new one complained about a problem on a specific line that has this content

<sixx.object sixx.id="1106" sixx.name="key" sixx.type="String"

...
name</sixx.object> <sixx.object sixx.id="1107" sixx.name="value" sixx.type="String" ��L��i��k��e�� M��a��r��k�� T��w��a��i��n��:�� I��'��v��e�� n��e��v��e��r�� l��e��t�� m��y�� s��c��h��o��o��l�� i��n��t��e��r��f��e��r��e�� w��i��t��h�� m��y�� e��d��u��c��a��t��i��o��n��.�� </sixx.object> </sixx.object>

looks like an encoding problem

Was that originally a UTF-32 string or something? Three NULs followed by a character... (For the curious, it says 'Like Mark Twain: "I've never let my school interfere with my education. "')

frank

Sven Van Caekenberghe

Jan. 4, 2014

9:11 a.m.

On 04 Jan 2014, at 00:38, Sebastian Sastre <sebastian@flowingconcept.com> wrote:

...

I've discovered that its part of a JsonObject that holds the facebook profile associated with the user.

SIXX is cool but I'm with the feeling that it's time to give fuel a chance

Or STON ? https://github.com/svenvc/ston/blob/master/ston-paper.md

...

On Jan 3, 2014, at 8:01 PM, Frank Shearar <frank.shearar@gmail.com> wrote:

...
On 3 January 2014 20:07, Sebastian Sastre <sebastian@flowingconcept.com> wrote:

...
It's probable unrelated to size as expected.

For the record:

I've changed the XMLParser and the new one complained about a problem on a specific line that has this content

<sixx.object sixx.id="1106" sixx.name="key" sixx.type="String"

...
name</sixx.object> <sixx.object sixx.id="1107" sixx.name="value" sixx.type="String" ��L��i��k��e�� M��a��r��k�� T��w��a��i��n��:�� I��'��v��e�� n��e��v��e��r�� l��e��t�� m��y�� s��c��h��o��o��l�� i��n��t��e��r��f��e��r��e�� w��i��t��h�� m��y�� e��d��u��c��a��t��i��o��n��.�� </sixx.object> </sixx.object>

looks like an encoding problem

Was that originally a UTF-32 string or something? Three NULs followed by a character... (For the curious, it says 'Like Mark Twain: "I've never let my school interfere with my education. "')

frank

Sebastian Sastre

1:22 p.m.

Thanks for the tip. AFAIK, STON would be best suited for JSON friendly instances. I need something more general. What I was trying to do with sixx is to dump the airflowing odb's to migrate data so pharo 3 can use it.* The object databases are composed by many kinds of objects, only a few of them are JSON-ish PD: All this is due to the change that happened in DateAndTime #jdn -> #julianDayNumber which is forcing me to either abandon Pharo 3, do horrible hacks to it or stay with older pharo. On Jan 4, 2014, at 7:11 AM, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

...
SIXX is cool but I'm with the feeling that it's time to give fuel a chance

Or STON ?

Sven Van Caekenberghe

1:37 p.m.

On 04 Jan 2014, at 14:22, Sebastian Sastre <sebastian@flowingconcept.com> wrote:

...

Thanks for the tip.

AFAIK, STON would be best suited for JSON friendly instances.

I need something more general.

STON can work with most normal objects and is easy to customise or adapt.

...

What I was trying to do with sixx is to dump the airflowing odb's to migrate data so pharo 3 can use it.* The object databases are composed by many kinds of objects, only a few of them are JSON-ish

I donât know what you mean by JSON friendly objects, can you give an example of something that you need that is not of that kind ?

...

PD: All this is due to the change that happened in DateAndTime #jdn -> #julianDayNumber which is forcing me to either abandon Pharo 3, do horrible hacks to it or stay with older pharo.

OK, that is a rename, why canât you just add it back ?

...

On Jan 4, 2014, at 7:11 AM, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...
...
SIXX is cool but I'm with the feeling that it's time to give fuel a chance

Or STON ?

Marcus Denker

1:39 p.m.

On 04 Jan 2014, at 14:38, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

On 04 Jan 2014, at 14:22, Sebastian Sastre <sebastian@flowingconcept.com> wrote:

...
Thanks for the tip.

AFAIK, STON would be best suited for JSON friendly instances.

I need something more general.

STON can work with most normal objects and is easy to customise or adapt.

...
What I was trying to do with sixx is to dump the airflowing odb's to migrate data so pharo 3 can use it.* The object databases are composed by many kinds of objects, only a few of them are JSON-ish

I donât know what you mean by JSON friendly objects, can you give an example of something that you need that is not of that kind ?

...
PD: All this is due to the change that happened in DateAndTime #jdn -> #julianDayNumber which is forcing me to either abandon Pharo 3, do horrible hacks to it or stay with older pharo.

OK, that is a rename, why canât you just add it back ?

and, you can not ask at the same time for âniceâ APIs and then complain when we do what you askâ¦ Marcus

Sebastian Sastre

1:54 p.m.

On Jan 4, 2014, at 11:39 AM, Marcus Denker <marcus.denker@inria.fr> wrote:

...

and, you can not ask at the same time for âniceâ APIs and then complain when we do what you askâ¦

Marcus

Marcus please notice I'm not complaining about #jdn going #julianDayNumber, just sharing that I'm investing in migrating to Pharo 3 sebastian o/

Sebastian Sastre

1:59 p.m.

On Jan 4, 2014, at 11:37 AM, Sven Van Caekenberghe <sven@stfx.eu> wrote:

...

...
PD: All this is due to the change that happened in DateAndTime #jdn -> #julianDayNumber which is forcing me to either abandon Pharo 3, do horrible hacks to it or stay with older pharo.

OK, that is a rename, why canât you just add it back ?

That's the hack-ish way, that will make me maintain the that hack, I prefer to migrate. BTW I'm realising I forgot to mention one alternative in that previous paragraph which is to upgrade to Pharo 3, which is the bet I'm taking :)

4688

Age (days ago)

4689

Last active (days ago)

List overview

11 comments

5 participants

participants (5)

Frank Shearar
Marcus Denker
Sebastian Sastre
Sven Van Caekenberghe