February 2022 - Pharo-users

Re: [ANN] Pharo LZ4 Tools

by ducasse

Yes this is a nice conversation. I know that marcus is working on slim binary. The idea is to compress but I forgot if it was bytecode or trees. Now I wonder if the code text is worth compression. >> On 18 Feb 2022, at 21:25, Guillermo Polito <guillermopolito(a)gmail.com> wrote: >> >> Thanks Sven, great stuff :) > > Thanks! > > This allows you to easily play/explore/experiment with certain ideas. > > In the past we discussed about the option of bringing source code inside the image, what if we applied compression ? > > The total size of all Object methods is about 100k: > > Object allMethods sum: [ :each | each sourceCode size ]. > > "104633" > > We can compress each individual method as an LZ4 block and see what that gives us. > > LZ4Compressor new in: [ :compressor | > Object allMethods sum: [ :each | | compressed | > compressed := compressor compressBlock: each sourceCode utf8Encoded. > compressed size ] ]. > > "81584" > > (104633/81584) reciprocal asFloat. > > "0.7797157684478129" > > That is about 22% smaller. This is not a very good result. But that is to be expected because methods are small and there is often not much to compress. > > If we concatenate all source code and feed that as one big chunk to the compressor we get much better results. > > (LZ4Compressor new compress: > (String streamContents: [ :out | > Object allMethods do: [ :each | out nextPutAll: each sourceCode ] ]) utf8Encoded) size. > > "53544" > > (104633/53544) reciprocal asFloat. > > "0.5117314805080615" > > Now we get an almost 50% reduction in size. But methods are independent, so that is not an option. What if we used a dictionary, a predefined set of words/substrings that are common in source code. > > I found a list of the 500 most common English words. Let's add some common selectors and globals. > > IdentityBag new in: [ :bag | > SystemNavigation default allMethods do: [ :each | > each literals select: [ :x | x isSymbol ] thenDo: [ :x | bag add: x ] ]. > bag sortedCounts select: [ :x | x key > 100 ] ]. > > IdentityBag new in: [ :bag | > SystemNavigation default allMethods do: [ :each | > each literals select: [ :x | x isVariableBinding ] thenDo: [ :x | bag add: x key ] ]. > bag sortedCounts select: [ :x | x key > 100 ] ]. > > The smallest possible match in LZ4 is 4 bytes (3 letters and a space). > > words := Character space join: (((FileLocator desktop / 'en-500.csv' readStreamDo: [ :in | (NeoCSVReader on: in) addIgnoredField; addField; upToEnd ]) collect: #first) select: [ :each | each size > 2 ]). > > That are 473 words. Next are 137 selectors. > > selectors := ' ifTrue: assert: class assert:equals: ifTrue:ifFalse: ifNil: ifFalse: yourself name and: ifNotNil: traitComposition add: first deny: includes: isEmpty asString nextPutAll: with: isNil collect: initialize subclassResponsibility to:do: selector localMethodDict should:raise: theme notNil printString on:do: streamContents: at:ifAbsent: copy contents error: last model default ifNil:ifNotNil: organization skipOrReturnWith:ifSkippable: current parserExceptions nonEmpty select: asSymbol name: readStream includesKey: basicNew title: empty reject: whileTrue: keys space class: extent: close anySatisfy: parse:documentURI: isLocalSelector: traitSource second position print: whileFalse: asArray format: printOn: selectors isKindOf: copyFrom:to: color: shouldnt:raise: width height max: named: signal hasProperty: anyOne text detect:ifNone: label: ensure: ifEmpty: extent text: entity addAll: negated includesLocalSelector: addSelector:withMethod: traitDefining:ifNone: hash addSelector:on: asInteger min: translated iconNamed: method arguments position: withIndexDo: perform: methods delete url: occurrencesOf: selector: hResizing: with:with: pass notEmpty flag: values removeKey: fromString: classNamed: reset changed removeKey:ifAbsent: width: announce: repository: signal: setUp addLast: session uniqueInstance assert:description: asOrderedCollection compiledMethod assert:gives: '. > > Finally 73 globals. > > globals := ' String OrderedCollection Array Smalltalk Color Error Character Dictionary TraitChange ByteArray Object UIManager DateAndTime Set Form Time ZTimestamp Date RBParser World Protocol Duration Processor SAXHandler Display MetaLink HelpTopic ReflectivityExamples IdentitySet OCOpalExamples GLMTabulator Float SpecLayout UUID WAMimeType SystemAnnouncer STON XMLDOMParser ZnMimeType Transcript ZnEntity ExternalType CompiledMethod GRPlatform Semaphore FileSystem ReadWriteStream ZnClient WriteStream Delay CmdContextMenuActivation ZnResponse FileLocator IdentityDictionary Morph MCSnapshot ReflectiveMethod XMLValidationException Integer MCMethodDefinition Path ClyClassScope MCClassDefinition RBCondition MCVersionInfo MCVersion SmallInteger Cursor TraitedClass GoferVersionReference SortedCollection MCOrganizationDefinition XMLWellFormednessException '. > > dictionary := (globals , words , selectors) utf8Encoded. > > This dictionary is less than 5K. > > (LZ4Compressor new dictionary: dictionary) in: [ :compressor | > Object allMethods sum: [ :each | | compressed | > compressed := compressor compressBlock: each sourceCode utf8Encoded. > compressed size ] ]. > > "69146" > > (104633/69146) reciprocal asFloat. > > "0.6608431374422983" > > Now we get a 33% reduction in size, which is better. > > I am sure that with a more carefully, better tuned dictionary the compression rate could be improved a couple of percent. There also exist tools that can compute an optimal dictionary from a given input set. > > Sorry for the long post, I hope at least someone found this interesting. > > Sven > > >> EnvoyÃ© depuis mon tÃ©lÃ©phone Huawei >> >> >> -------- Message original -------- >> De : Sven Van Caekenberghe <sven(a)stfx.eu> >> Date : ven. 18 fÃ©vr. 2022 Ã 21:13 >> Ã : Any question about pharo is welcome <pharo-users(a)lists.pharo.org> >> Objet : [Pharo-users] [ANN] Pharo LZ4 Tools >> Hi, >> >> Pharo LZ4 Tools (https://github.com/svenvc/pharo-lz4-tools) is an implementation of LZ4 compression and decompression in pure Pharo. >> >> LZ4 is a lossless compression algorithm that is focused on speed. It belongs to the LZ77 family of byte-oriented compression schemes. >> >> - https://en.wikipedia.org/wiki/LZ4_(compression_algorithm) >> - https://lz4.github.io/lz4/ >> - https://github.com/lz4/lz4 >> >> Both the frame format (https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md) as well as the block format (https://github.com/lz4/lz4/blob/dev/doc/lz4_Block_format.md) are implemented. Dictionary based compression/decompression is available too. The XXHash32 algorithm is also implemented. >> >> Of course this implementation is not as fast as highly optimised native implementations, but it works quite well and is readable/understandable, if you like this kind of stuff. It can be useful to interact with other systems using LZ4. >> >> Sven

Feb. 20, 2022

Re: [ANN] Pharo LZ4 Tools

by Sven Van Caekenberghe

> On 18 Feb 2022, at 21:25, Guillermo Polito <guillermopolito(a)gmail.com> wrote: > > Thanks Sven, great stuff :) Thanks! This allows you to easily play/explore/experiment with certain ideas. In the past we discussed about the option of bringing source code inside the image, what if we applied compression ? The total size of all Object methods is about 100k: Object allMethods sum: [ :each | each sourceCode size ]. "104633" We can compress each individual method as an LZ4 block and see what that gives us. LZ4Compressor new in: [ :compressor | Object allMethods sum: [ :each | | compressed | compressed := compressor compressBlock: each sourceCode utf8Encoded. compressed size ] ]. "81584" (104633/81584) reciprocal asFloat. "0.7797157684478129" That is about 22% smaller. This is not a very good result. But that is to be expected because methods are small and there is often not much to compress. If we concatenate all source code and feed that as one big chunk to the compressor we get much better results. (LZ4Compressor new compress: (String streamContents: [ :out | Object allMethods do: [ :each | out nextPutAll: each sourceCode ] ]) utf8Encoded) size. "53544" (104633/53544) reciprocal asFloat. "0.5117314805080615" Now we get an almost 50% reduction in size. But methods are independent, so that is not an option. What if we used a dictionary, a predefined set of words/substrings that are common in source code. I found a list of the 500 most common English words. Let's add some common selectors and globals. IdentityBag new in: [ :bag | SystemNavigation default allMethods do: [ :each | each literals select: [ :x | x isSymbol ] thenDo: [ :x | bag add: x ] ]. bag sortedCounts select: [ :x | x key > 100 ] ]. IdentityBag new in: [ :bag | SystemNavigation default allMethods do: [ :each | each literals select: [ :x | x isVariableBinding ] thenDo: [ :x | bag add: x key ] ]. bag sortedCounts select: [ :x | x key > 100 ] ]. The smallest possible match in LZ4 is 4 bytes (3 letters and a space). words := Character space join: (((FileLocator desktop / 'en-500.csv' readStreamDo: [ :in | (NeoCSVReader on: in) addIgnoredField; addField; upToEnd ]) collect: #first) select: [ :each | each size > 2 ]). That are 473 words. Next are 137 selectors. selectors := ' ifTrue: assert: class assert:equals: ifTrue:ifFalse: ifNil: ifFalse: yourself name and: ifNotNil: traitComposition add: first deny: includes: isEmpty asString nextPutAll: with: isNil collect: initialize subclassResponsibility to:do: selector localMethodDict should:raise: theme notNil printString on:do: streamContents: at:ifAbsent: copy contents error: last model default ifNil:ifNotNil: organization skipOrReturnWith:ifSkippable: current parserExceptions nonEmpty select: asSymbol name: readStream includesKey: basicNew title: empty reject: whileTrue: keys space class: extent: close anySatisfy: parse:documentURI: isLocalSelector: traitSource second position print: whileFalse: asArray format: printOn: selectors isKindOf: copyFrom:to: color: shouldnt:raise: width height max: named: signal hasProperty: anyOne text detect:ifNone: label: ensure: ifEmpty: extent text: entity addAll: negated includesLocalSelector: addSelector:withMethod: traitDefining:ifNone: hash addSelector:on: asInteger min: translated iconNamed: method arguments position: withIndexDo: perform: methods delete url: occurrencesOf: selector: hResizing: with:with: pass notEmpty flag: values removeKey: fromString: classNamed: reset changed removeKey:ifAbsent: width: announce: repository: signal: setUp addLast: session uniqueInstance assert:description: asOrderedCollection compiledMethod assert:gives: '. Finally 73 globals. globals := ' String OrderedCollection Array Smalltalk Color Error Character Dictionary TraitChange ByteArray Object UIManager DateAndTime Set Form Time ZTimestamp Date RBParser World Protocol Duration Processor SAXHandler Display MetaLink HelpTopic ReflectivityExamples IdentitySet OCOpalExamples GLMTabulator Float SpecLayout UUID WAMimeType SystemAnnouncer STON XMLDOMParser ZnMimeType Transcript ZnEntity ExternalType CompiledMethod GRPlatform Semaphore FileSystem ReadWriteStream ZnClient WriteStream Delay CmdContextMenuActivation ZnResponse FileLocator IdentityDictionary Morph MCSnapshot ReflectiveMethod XMLValidationException Integer MCMethodDefinition Path ClyClassScope MCClassDefinition RBCondition MCVersionInfo MCVersion SmallInteger Cursor TraitedClass GoferVersionReference SortedCollection MCOrganizationDefinition XMLWellFormednessException '. dictionary := (globals , words , selectors) utf8Encoded. This dictionary is less than 5K. (LZ4Compressor new dictionary: dictionary) in: [ :compressor | Object allMethods sum: [ :each | | compressed | compressed := compressor compressBlock: each sourceCode utf8Encoded. compressed size ] ]. "69146" (104633/69146) reciprocal asFloat. "0.6608431374422983" Now we get a 33% reduction in size, which is better. I am sure that with a more carefully, better tuned dictionary the compression rate could be improved a couple of percent. There also exist tools that can compute an optimal dictionary from a given input set. Sorry for the long post, I hope at least someone found this interesting. Sven > EnvoyÃ© depuis mon tÃ©lÃ©phone Huawei > > > -------- Message original -------- > De : Sven Van Caekenberghe <sven(a)stfx.eu> > Date : ven. 18 fÃ©vr. 2022 Ã 21:13 > Ã : Any question about pharo is welcome <pharo-users(a)lists.pharo.org> > Objet : [Pharo-users] [ANN] Pharo LZ4 Tools > Hi, > > Pharo LZ4 Tools (https://github.com/svenvc/pharo-lz4-tools) is an implementation of LZ4 compression and decompression in pure Pharo. > > LZ4 is a lossless compression algorithm that is focused on speed. It belongs to the LZ77 family of byte-oriented compression schemes. > > - https://en.wikipedia.org/wiki/LZ4_(compression_algorithm) > - https://lz4.github.io/lz4/ > - https://github.com/lz4/lz4 > > Both the frame format (https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md) as well as the block format (https://github.com/lz4/lz4/blob/dev/doc/lz4_Block_format.md) are implemented. Dictionary based compression/decompression is available too. The XXHash32 algorithm is also implemented. > > Of course this implementation is not as fast as highly optimised native implementations, but it works quite well and is readable/understandable, if you like this kind of stuff. It can be useful to interact with other systems using LZ4. > > Sven

Feb. 20, 2022

UK Smalltalk User Group meeting - 23 February 2022

by Giovanni Corriga

For this month's UKSTUG meeting, we'll open the floor to the whole audience and let people show what they are working on. If you have an interesting project to show, or if you'd like to get some help with some hard problem, just show up and be ready to present! This will be an online meeting from home. If you'd like to join us, please sign up in advance on the meeting's Meetup page ( https://www.meetup.com/UKSTUG/events/284113812/ ) to receive the meeting details. Donât forget to bring your laptop and drinks!

Feb. 19, 2022

Re: [ANN] Pharo LZ4 Tools

by Guillermo Polito

Feb. 18, 2022

[ANN] Pharo LZ4 Tools

by Sven Van Caekenberghe

Hi, Pharo LZ4 Tools (https://github.com/svenvc/pharo-lz4-tools) is an implementation of LZ4 compression and decompression in pure Pharo. LZ4 is a lossless compression algorithm that is focused on speed. It belongs to the LZ77 family of byte-oriented compression schemes. - https://en.wikipedia.org/wiki/LZ4_(compression_algorithm) - https://lz4.github.io/lz4/ - https://github.com/lz4/lz4 Both the frame format (https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md) as well as the block format (https://github.com/lz4/lz4/blob/dev/doc/lz4_Block_format.md) are implemented. Dictionary based compression/decompression is available too. The XXHash32 algorithm is also implemented. Of course this implementation is not as fast as highly optimised native implementations, but it works quite well and is readable/understandable, if you like this kind of stuff. It can be useful to interact with other systems using LZ4. Sven

Feb. 18, 2022

[ANN] Stargate v7.0.1 [v7.0.1] released!

by Buenos Aires Smalltalk

Feb. 18, 2022

[ANN] Stargate-Consul v3.0.1 [v3.0.1] released!

by Buenos Aires Smalltalk

Feb. 18, 2022

Re: [Issue tracker] Please check your older issues

by Marcus Denker

Thanks to everyone who helped digging through the old issues! We managed to close over 140 issue in 2022 already, with just 801 issues open (and >3500 issues closed overall since we moved to this issue tracker). Often it feels a bit useless âjust one issue downâ¦ what difference does that make?â. But you can only close 3500 issues by looking at each of them, one after the other... Marcus > On 7 Feb 2022, at 11:56, Marcus Denker <marcus.denker(a)inria.fr> wrote: > > Hi, > > We are seeing lots of issue that are still open, but already fixed or just not valid anymore. > e.g. > > - the fix that was merged but the issue was not closed (sometimes GitHub has problems doing that) > - Bug was fixed independent of the issue > - The issue is just not relevant anymore > > > It is often quite time consuming to even find out that an issue is already fixedâ¦ > > It would be nice if everyone could check the issues that > > - you submitted yourself > - that are assigned to you > - [Bonus] where you were part of some discussion > > > https://github.com/pharo-project/pharo/issues > > > Marcus

Feb. 15, 2022

Re: Changes to class not recorded in change.log?

by Tim Mackinnon

Hi Stewart - are you using the Code Changes (Epicea) browser - or trying to use the older changes files? Your message wasn't clear - and I have seen oddities with Epicea too - but normally I have been able to recover instance variables too (sometimes the restore options were a bit confusing and I did it in the wrong order). Note - I haven't tried this in V9 though - so possibly something has regressed. Thought it worth checking though. Tim On Mon, 14 Feb 2022, at 7:36 AM, Stewart MacLean wrote: > Hi, > > The worst case scenario has just happened - my image is hosed. > > I've managed to recover a previous set of change/image files (thanks CrashPlan!). > > Then I've replayed all the intervening changes - however I find that some instance variables that I had defined haven't been reflected in class shape changes. > > I notice all the changes for the complete change log are for Method, and Do It (image saves) > > Surely class shape changes should be recorded as well? Is this a bug? > > Cheers, > > Stewart > > Platform Mac Mini > Pharo 9.0.0 > Build information: Pharo-9.0.0+build.1573.sha.0e09d756fc99383cbb498565200d9e5e9841ce11 (64 Bit) >

Feb. 14, 2022

Re: Changes to class not recorded in change.log?

by Marcus Denker

> On 14 Feb 2022, at 08:36, Stewart MacLean <stewart(a)xtra.co.nz> wrote: > > Hi, > > The worst case scenario has just happened - my image is hosed. > > I've managed to recover a previous set of change/image files (thanks CrashPlan!). > > Then I've replayed all the intervening changes - however I find that some instance variables that I had defined haven't been reflected in class shape changes. > > I notice all the changes for the complete change log are for Method, and Do It (image saves) > > Surely class shape changes should be recorded as well? Is this a bug? > Yes, normally the lass change should be recorded, too. So that looks like a bug. Marcus

Feb. 14, 2022