Working with a compressed Fuel file?
I'm trying to work with a compressed Fuel file, without having to expand everything at the same time. It looks like not all streams are created equally. I seem to need a File instead of a FileReference: saveToFile |zipStream fileRef| fileRef := File named: (FileSystem disk workingDirectory / (self fileName, '.zip')) pathString. fileRef writeStreamDo: [ :s | zipStream := GZipWriteStream on: s. repository := nil. FLSerializer serialize: self on: zipStream. zipStream close]. And materializing doesn't like this stream from: aFileName |zipStream fileRef theNew| fileRef := File named: aFileName. fileRef readStreamDo: [ :s | zipStream := GZipReadStream on: s. theNew := (FLMaterializer new materializeFrom: zipStream) root. zipStream close]. ^theNew How am I supposed to work with compressed Fuel files? Stephan
I may have a hint. Which is the error upon materialization? A dnu on the stream? Which message? On Jan 3, 2016 4:19 PM, "Stephan Eggermont" <stephan@stack.nl> wrote:
I'm trying to work with a compressed Fuel file, without having to expand everything at the same time. It looks like not all streams are created equally. I seem to need a File instead of a FileReference:
saveToFile |zipStream fileRef| fileRef := File named: (FileSystem disk workingDirectory / (self fileName, '.zip')) pathString. fileRef writeStreamDo: [ :s | zipStream := GZipWriteStream on: s. repository := nil. FLSerializer serialize: self on: zipStream. zipStream close].
And materializing doesn't like this stream
from: aFileName |zipStream fileRef theNew| fileRef := File named: aFileName. fileRef readStreamDo: [ :s | zipStream := GZipReadStream on: s. theNew := (FLMaterializer new materializeFrom: zipStream) root. zipStream close]. ^theNew
How am I supposed to work with compressed Fuel files?
Stephan
On 03-01-16 20:31, Mariano Martinez Peck wrote:
I may have a hint. Which is the error upon materialization? A dnu on the stream? Which message?
A segmentation fault when saving Smalltalk stack dump: 0xffcb7390 M GZipWriteStream(DeflateStream)>deflateBlock 0xe66c808: a(n) GZipWriteStream 0xffcb73b4 M GZipWriteStream(DeflateStream)>next:putAll:startingAt: 0xe66c808: a(n) GZipWriteStream 0xffcb73d8 M FLBufferedWriteStream>flushBuffer 0xe66cb48: a(n) FLBufferedWriteStream 0xffcb73f0 M FLBufferedWriteStream>nextBytesPutAll: 0xe66cb48: a(n) FLBufferedWriteStream 0xffcb7410 M FLEncoder>encodeString: 0xe66c720: a(n) FLEncoder 0xffcb742c M ByteString>serializeOn: 0xcce25a0: a(n) ByteString 0xffcb7448 M FLHookPrimitiveCluster>serializeInstance:with: 0xe713af8: a(n) FLHookPrimitiveCluster 0xffcb746c M [] in FLHookPrimitiveCluster(FLIteratingCluster)>serializeInstancesStepWith: 0xe713af8: a(n) FLHookPrimitiveCluster 0xffcb7494 M IdentitySet(Set)>do: 0xf70b230: a(n) IdentitySet 0xffcb74b0 M FLHookPrimitiveCluster(FLIteratingCluster)>serializeInstancesStepWith: 0xe713af8: a(n) FLHookPrimitiveCluster 0xffcb74cc M FLSerialization>clusterInstancesStepOf: 0xe66c6e8: a(n) FLSerialization 0xffcb74e8 M [] in FLSerialization>instancesStep 0xe66c6e8: a(n) FLSerialization 0xffcb750c M OrderedCollection>do: 0xf70b1e0: a(n) OrderedCollection 0xffcb7530 I FLSerialization>instancesStep 0xe66c6e8: a(n) FLSerialization 0xffcb7550 I FLSerialization>run 0xe66c6e8: a(n) FLSerialization 0xffcb7574 I [] in FLSerializer>setDefaultSerialization 0xe66c700: a(n) FLSerializer 0xffcb75a0 I [] in FLSerializer>serialize:on: 0xe66c700: a(n) FLSerializer 0xffcb75cc I [] in FLEncoder class>on:globalEnvironment:do: 0x8f20878: a(n) FLEncoder class 0xffcb75ec M BlockClosure>ensure: 0xe66c770: a(n) BlockClosure 0xffcb7614 I FLEncoder class>on:globalEnvironment:do: 0x8f20878: a(n) FLEncoder class 0xffcb7640 I FLSerializer>serialize:on: 0xe66c700: a(n) FLSerializer 0xffcb7668 I FLSerializer class>serialize:on: 0x8eb91d8: a(n) FLSerializer class 0xffcb7694 I [] in MCProject>saveToFile 0xb92d790: a(n) MCProject 0xffcb76c0 I [] in File>writeStreamDo: 0xe612500: a(n) File 0xffcb76e0 M BlockClosure>ensure: 0xe66c900: a(n) BlockClosure 0xffcb7708 I File>writeStreamDo: 0xe612500: a(n) File 0xffcb7734 I MCProject>saveToFile 0xb92d790: a(n) MCProject 0xffcb7750 M UndefinedObject>(nil) 0x8b75b00: a(n) UndefinedObject 0xb73ad00 s OpalCompiler>evaluate 0xb92dec8 s RubSmalltalkEditor>evaluate:andDo: 0xb92df58 s RubSmalltalkEditor>highlightEvaluateAndDo: 0xb743418 s [] in GLMMorphicPharoScriptRenderer(GLMMorphicPharoCodeRenderer)>actOnHighlightAndEvaluate: 0xb73ad60 s RubEditingArea(RubAbstractTextArea)>handleEdit: 0xb92dff8 s [] in GLMMorphicPharoScriptRenderer(GLMMorphicPharoCodeRenderer)>actOnHighlightAndEvaluate: 0xb743478 s WorldState>runStepMethodsIn: 0xb73adc0 s WorldMorph>runStepMethods 0xb92e070 s WorldState>doOneCycleNowFor: 0xb7434d8 s WorldState>doOneCycleFor: 0xb73ae20 s WorldMorph>doOneCycle 0x944db08 s [] in MorphicUIManager>(nil) 0x944da80 s [] in BlockClosure>(nil)
On Mon, Jan 4, 2016 at 10:16 AM, Stephan Eggermont <stephan@stack.nl> wrote:
On 03-01-16 20:31, Mariano Martinez Peck wrote:
I may have a hint. Which is the error upon materialization? A dnu on the stream? Which message?
A segmentation fault when saving
Ahh I reported that years ago: http://forum.world.st/Re-Pharo-project-GZipWriteStream-crashing-my-VM-when-s... Cheers,
Smalltalk stack dump: 0xffcb7390 M GZipWriteStream(DeflateStream)>deflateBlock 0xe66c808: a(n) GZipWriteStream 0xffcb73b4 M GZipWriteStream(DeflateStream)>next:putAll:startingAt: 0xe66c808: a(n) GZipWriteStream 0xffcb73d8 M FLBufferedWriteStream>flushBuffer 0xe66cb48: a(n) FLBufferedWriteStream 0xffcb73f0 M FLBufferedWriteStream>nextBytesPutAll: 0xe66cb48: a(n) FLBufferedWriteStream 0xffcb7410 M FLEncoder>encodeString: 0xe66c720: a(n) FLEncoder 0xffcb742c M ByteString>serializeOn: 0xcce25a0: a(n) ByteString 0xffcb7448 M FLHookPrimitiveCluster>serializeInstance:with: 0xe713af8: a(n) FLHookPrimitiveCluster 0xffcb746c M [] in FLHookPrimitiveCluster(FLIteratingCluster)>serializeInstancesStepWith: 0xe713af8: a(n) FLHookPrimitiveCluster 0xffcb7494 M IdentitySet(Set)>do: 0xf70b230: a(n) IdentitySet 0xffcb74b0 M FLHookPrimitiveCluster(FLIteratingCluster)>serializeInstancesStepWith: 0xe713af8: a(n) FLHookPrimitiveCluster 0xffcb74cc M FLSerialization>clusterInstancesStepOf: 0xe66c6e8: a(n) FLSerialization 0xffcb74e8 M [] in FLSerialization>instancesStep 0xe66c6e8: a(n) FLSerialization 0xffcb750c M OrderedCollection>do: 0xf70b1e0: a(n) OrderedCollection 0xffcb7530 I FLSerialization>instancesStep 0xe66c6e8: a(n) FLSerialization 0xffcb7550 I FLSerialization>run 0xe66c6e8: a(n) FLSerialization 0xffcb7574 I [] in FLSerializer>setDefaultSerialization 0xe66c700: a(n) FLSerializer 0xffcb75a0 I [] in FLSerializer>serialize:on: 0xe66c700: a(n) FLSerializer 0xffcb75cc I [] in FLEncoder class>on:globalEnvironment:do: 0x8f20878: a(n) FLEncoder class 0xffcb75ec M BlockClosure>ensure: 0xe66c770: a(n) BlockClosure 0xffcb7614 I FLEncoder class>on:globalEnvironment:do: 0x8f20878: a(n) FLEncoder class 0xffcb7640 I FLSerializer>serialize:on: 0xe66c700: a(n) FLSerializer 0xffcb7668 I FLSerializer class>serialize:on: 0x8eb91d8: a(n) FLSerializer class 0xffcb7694 I [] in MCProject>saveToFile 0xb92d790: a(n) MCProject 0xffcb76c0 I [] in File>writeStreamDo: 0xe612500: a(n) File 0xffcb76e0 M BlockClosure>ensure: 0xe66c900: a(n) BlockClosure 0xffcb7708 I File>writeStreamDo: 0xe612500: a(n) File 0xffcb7734 I MCProject>saveToFile 0xb92d790: a(n) MCProject 0xffcb7750 M UndefinedObject>(nil) 0x8b75b00: a(n) UndefinedObject 0xb73ad00 s OpalCompiler>evaluate 0xb92dec8 s RubSmalltalkEditor>evaluate:andDo: 0xb92df58 s RubSmalltalkEditor>highlightEvaluateAndDo: 0xb743418 s [] in GLMMorphicPharoScriptRenderer(GLMMorphicPharoCodeRenderer)>actOnHighlightAndEvaluate: 0xb73ad60 s RubEditingArea(RubAbstractTextArea)>handleEdit: 0xb92dff8 s [] in GLMMorphicPharoScriptRenderer(GLMMorphicPharoCodeRenderer)>actOnHighlightAndEvaluate: 0xb743478 s WorldState>runStepMethodsIn: 0xb73adc0 s WorldMorph>runStepMethods 0xb92e070 s WorldState>doOneCycleNowFor: 0xb7434d8 s WorldState>doOneCycleFor: 0xb73ae20 s WorldMorph>doOneCycle 0x944db08 s [] in MorphicUIManager>(nil) 0x944da80 s [] in BlockClosure>(nil)
-- Mariano http://marianopeck.wordpress.com
On 04-01-16 14:29, Mariano Martinez Peck wrote:
Ahh I reported that years ago: http://forum.world.st/Re-Pharo-project-GZipWriteStream-crashing-my-VM-when-s...
Ah. That is also when writing a string. Is the automatic conversion of ByteString to WideString safe here? Stephan
On 04-01-16 14:29, Mariano Martinez Peck wrote:
Ahh I reported that years ago: http://forum.world.st/Re-Pharo-project-GZipWriteStream-crashing-my-VM-when-s...
I can't seem to find an issue for this. Did we lose it? Stephan
On 03-01-16 20:31, Mariano Martinez Peck wrote:
I may have a hint. Which is the error upon materialization? A dnu on the stream? Which message?
Here is a further problem. I've done: |project| Gofer it smalltalkhubUser: 'StephanEggermont' project: 'MonticelloProjects'; package: 'MonticelloProjects'; load. project := (Smalltalk at: #MCProject) new location: 'http://source.squeak.org/VMMaker'. project read. project repository: nil. FLSerializer serialize: project toFileNamed: 'VMMaker.fl'. That takes a while (downloads 5 GB or so). It results in a 1.2 GB Fuel file. That is way too much, so somewhere I'm pulling in too many objects, or they are not recognized as equal. Writing out the 71712 unique definitions in this project separately is 54MB. The project consists of 17 packages with 2723 package versions, the total number of pointers to the definitions is 2.2 million. The total should be below 100MB Stephan
On 05-01-16 16:29, Stephan Eggermont wrote:
Writing out the 71712 unique definitions in this project separately is 54MB. The project consists of 17 packages with 2723 package versions, the total number of pointers to the definitions is 2.2 million. The total should be below 100MB
Thanks to the help of Max and Thierry I found the problem. Set withAll: (MCVersionInfo allInstances collect: [ :each | each id ]) 7155 MCVersionInfo allInstances size 2726588 2030 versions of VMMaker-> 2029+2028+2027+...+1 MCVersionInfo's ideally... Stephan
And what happened with the vm gzip crash? Still around? On Tue, Jan 5, 2016 at 3:43 PM, Stephan Eggermont <stephan@stack.nl> wrote:
On 05-01-16 16:29, Stephan Eggermont wrote:
Writing out the 71712 unique definitions in this project separately is 54MB. The project consists of 17 packages with 2723 package versions, the total number of pointers to the definitions is 2.2 million. The total should be below 100MB
Thanks to the help of Max and Thierry I found the problem.
Set withAll: (MCVersionInfo allInstances collect: [ :each | each id ]) 7155
MCVersionInfo allInstances size 2726588
2030 versions of VMMaker-> 2029+2028+2027+...+1 MCVersionInfo's ideally...
Stephan
-- Mariano http://marianopeck.wordpress.com
participants (2)
-
Mariano Martinez Peck -
Stephan Eggermont