November 2016 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] [ANN] Pharo Association has a new Website!

by Esteban A. Maringolo

Great! What do we do with the current BountySource salt? (https://salt.bountysource.com/teams/pharo) Should we move directly to the new Wild Apricot backend? Accepting Bitcoin payments would be a plus ;-) (https://bitpay.com/tour) Best regards! Esteban A. Maringolo 2016-11-10 6:22 GMT-03:00 Marcus Denker <marcus.denker(a)inria.fr>: > Hello, > > We have changed the backend of the Pharo Association. > > https://association.pharo.org > > If you ever joined the association in the past, please consider to > re-subscribe. > > We have added already all existing active members, in this case you should > have already received a new password. > > For all questions, do not hesitate to contact assocation(a)pharo.org > > > Marcus

Nov. 10, 2016

[pharo-project/pharo-core]

by GitHub

Branch: refs/tags/60288 Home: https://github.com/pharo-project/pharo-core

Nov. 10, 2016

[pharo-project/pharo-core] 95f350: 60288

by GitHub

Branch: refs/heads/6.0 Home: https://github.com/pharo-project/pharo-core Commit: 95f35034de3196a288254f9c5cb5b78cabe7d4f6 https://github.com/pharo-project/pharo-core/commit/95f35034de3196a288254f9c… Author: Jenkins Build Server <board(a)pharo-project.org> Date: 2016-11-10 (Thu, 10 Nov 2016) Changed paths: A Deprecated60.package/extension/Pragma/instance/keyword.st R Deprecated60.package/extension/Pragma/instance/selector.st M GT-Tests-Spotter.package/GTSpotterStepTest.class/instance/private-navigation/pragma_of_.st M GT-Tests-Spotter.package/GTSpotterStepTest.class/instance/private-navigation/pragmas_inPackages_.st M HelpSystem-Core.package/SystemHelp.class/class/private accessing/allSystemHelpPragmas.st M HelpSystem-Core.package/WikiStyleHelpBuilder.class/class/private accessing/allHelpPragmas.st M Kernel-Tests.package/PragmaTest.class/instance/tests/testCopy.st M Kernel.package/CompiledMethod.class/instance/accessing-pragmas %26 properties/hasPragmaNamed_.st M Kernel.package/Pragma.class/class/finding/allNamed_from_to_.st M Kernel.package/Pragma.class/class/finding/allNamed_in_.st M Kernel.package/Pragma.class/class/private/withPragmasIn_do_.st M Kernel.package/Pragma.class/definition.st M Kernel.package/Pragma.class/instance/accessing-method/methodSelector.st A Kernel.package/Pragma.class/instance/accessing-method/selector.st M Kernel.package/Pragma.class/instance/accessing-pragma/key.st R Kernel.package/Pragma.class/instance/accessing-pragma/keyword.st M Kernel.package/Pragma.class/instance/accessing-pragma/message.st M Kernel.package/Pragma.class/instance/comparing/=.st M Kernel.package/Pragma.class/instance/comparing/analogousCodeTo_.st M Kernel.package/Pragma.class/instance/comparing/hash.st M Kernel.package/Pragma.class/instance/initialization/setKeyword_.st M Kernel.package/Pragma.class/instance/printing/printOn_.st M Kernel.package/Pragma.class/instance/processing/sendTo_.st M Kernel.package/Pragma.class/instance/testing/hasLiteralSuchThat_.st M Kernel.package/Pragma.class/instance/testing/hasLiteral_.st M Keymapping-Core.package/extension/CompiledMethod/instance/isShortcutDeclaration.st M Keymapping-Pragmas.package/KMPragmaKeymapBuilder.class/instance/registrations handling/pragmaCollector.st M MenuRegistration.package/PragmaMenuBuilder.class/instance/registrations handling/pragmaCollector.st M Nautilus.package/MethodIsScriptAction.class/instance/testing/isActionHandled.st M Nautilus.package/MethodIsScriptWithArgumentAction.class/instance/testing/isActionHandled.st M Nautilus.package/MethodSampleInstanceAction.class/instance/order/isActionHandled.st R ScriptLoader60.package/ScriptLoader.class/instance/pharo - scripts/script60287.st A ScriptLoader60.package/ScriptLoader.class/instance/pharo - scripts/script60288.st R ScriptLoader60.package/ScriptLoader.class/instance/pharo - updates/update60287.st A ScriptLoader60.package/ScriptLoader.class/instance/pharo - updates/update60288.st M ScriptLoader60.package/ScriptLoader.class/instance/public/commentForCurrentUpdate.st M SmartSuggestions.package/SugsSuggestionFactory.class/class/private/createCollector_.st M Spec-Core.package/ComposableModel.class/class/protocol/specSelectors.st M Spec-Core.package/ComposableModel.class/instance/private/defaultSpecSelector.st A System-Announcements.package/ClassAnnouncement.class/instance/accessing/classAffected.st A System-Announcements.package/ClassAnnouncement.class/instance/accessing/classTagAffected.st A System-Announcements.package/ClassAnnouncement.class/instance/accessing/packageAffected.st M System-Announcements.package/ClassRemoved.class/definition.st M System-Announcements.package/ClassRemoved.class/instance/accessing/classRemoved_.st A System-Announcements.package/ClassRemoved.class/instance/accessing/classTagAffected.st A System-Announcements.package/ClassRemoved.class/instance/accessing/packageAffected.st M System-Settings.package/SettingBrowser.class/class/accessing/settingsKeywords.st M System-Settings.package/SettingTree.class/instance/accessing/acceptableKeywords_.st M System-Settings.package/SettingTreeBuilder.class/instance/accessing/buildPragma_.st M Tool-Base.package/MethodClassifier.class/instance/classification-rules/classifyByOtherImplementors_.st M Tool-Base.package/MethodClassifier.class/instance/classification-rules/classifyInSuperclassProtocol_.st Log Message: ----------- 60288 19323 ClassRemoved should provide information about affected package https://pharo.fogbugz.com/f/cases/19323 18233 improving Pragma API https://pharo.fogbugz.com/f/cases/18233 19320 Improve MethodClassifier logic https://pharo.fogbugz.com/f/cases/19320 http://files.pharo.org/image/60/60288.zip

Nov. 10, 2016

Re: [Pharo-dev] [ANN] Pharo Association has a new Website!

by Denis Kudriashov

That's nice, we have donate button now. Is it possible to make it more visible? Or at least put it in pharo.org 2016-11-10 10:22 GMT+01:00 Marcus Denker <marcus.denker(a)inria.fr>: > Hello, > > We have changed the backend of the Pharo Association. > > https://association.pharo.org > > If you ever joined the association in the past, please consider to > re-subscribe. > > We have added already all existing active members, in this case you should > have already received a new password. > > For all questions, do not hesitate to contact assocation(a)pharo.org > <assocations(a)pharo.org> > > > Marcus >

Nov. 10, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Thierry Goubier

2016-11-10 12:38 GMT+01:00 Norbert Hartl <norbert(a)hartl.name>: > > Am 10.11.2016 um 12:27 schrieb Thierry Goubier <thierry.goubier(a)gmail.com > >: > > > > 2016-11-10 12:18 GMT+01:00 Norbert Hartl <norbert(a)hartl.name>: > >> [ ...] >> >> Be it small chunks of data or not. A statement that general is most >> likely to be wrong. So the best way might be to ignore it. Indeed you are >> right that hardware got cheap. Even more important is the fact that >> hardware is almost always cheaper than personal costs. Solving all those >> technical problems instead of real ones and not trying to act in an >> economical way ruins a lot of companies out there. You can ignore >> economical facts (are any other) but that doesn't make you really smart! >> > > I disagree with that. In some areas (HPC, Exascale, HPDA), whatever the > physical limit is, we will reach and go larger than that. > > To what you disagree? I didn't say you never need it. In your case you > have concrete examples where it is necessary. In a lot of other cases it is > counter productive. Isn't that an agreement that you cannot say it in a way > too general way? > It is hard to disagree with something so general :) But, what we strive for is still to be general, otherwise we wouldn't have general purpose programming languages... mostly because a domain specific, dedicated solution is a costly proposition. > > Now, about that memory aspect, there is an entire field dedicated to > algorithmic solutions that never require the entire data set in memory. You > just have to look and implement the right underlying abstractions to allow > those algorithms to be implemented and run efficiently. > > And that is good. I think you got me wrong. I find it important to be able > to handle partial graphs in memory. But should everyone doing some > statistical research be one implementing that? Something like this I took > from the complaint making that the most important part and it is not. > Well, take again my "larger than memory" image example. Optimizing for that case makes the image viewer more efficient in the general case, so, yes, it can be argued that everybody should write statistical research in a "out-of-memory" system: will cost almost nothing in efficiency on datasets small enough, will allow the system to scale. Otherwise you end up with the R situation, where it runs your stat nice and fine until you reach an unknown (to you) size limit, where it crashes or seems to run forever (if not worse). Thierry > > Norbert > > > (my best example for that: satelite imagery viewers... have allways been > able to handle images larger than the computer RAM size. Just need a > buffered streaming interface to the file). > > > Thierry > > >> >> my 2 cents, >> >> Norbert >> >> >> > We should always challenge the assumptions behind our designs, because >> the world keeps changing and we risk becoming irrelevant, a syndrome that >> is not foreign to Smalltalk aficionados. >> > >> > Cheers, >> > Doru >> > >> > >> >> On Nov 10, 2016, at 9:12 AM, Igor Stasenko <siguctua(a)gmail.com> wrote: >> >> >> >> >> >> On 10 November 2016 at 07:27, Tudor Girba <tudor(a)tudorgirba.com> >> wrote: >> >> Hi Igor, >> >> >> >> Please refrain from speaking down on people. >> >> >> >> >> >> Hi, Doru! >> >> I just wanted to hear you :) >> >> >> >> If you have a concrete solution for how to do things, please feel free >> to share it with us. We would be happy to learn from it. >> >> >> >> >> >> Well, there's so many solutions, that i even don't know what to offer, >> and given the potential of smalltalk, i wonder why >> >> you are not employing any. But in overall it is a quesition of storing >> most of your data on disk, and only small portion of it >> >> in image (in most optimal cases - only the portion that user >> sees/operates with). >> >> As i said to you before, you will hit this wall inevitably, no matter >> how much memory is available. >> >> So, what stops you from digging in that direction? >> >> Because even if you can fit all data in memory, consider how much time >> it takes for GC to scan 4+ Gb of memory, comparing to >> >> 100 MB or less. >> >> I don't think you'll find it convenient to work in environment where >> you'll have 2-3 seconds pauses between mouse clicks. >> >> So, of course, my tone is not acceptable, but its pain to see how >> people remain helpless without even thinking about >> >> doing what they need. We have Fuel for how many years now? >> >> So it can't be as easy as it is, just serialize the data and purge it >> from image, till it will be required again. >> >> Sure it will require some effort, but it is nothing comparing to day >> to day pain that you have to tolerate because of lack of solution. >> >> >> >> Cheers, >> >> Tudor >> >> >> >> >> >>> On Nov 10, 2016, at 4:11 AM, Igor Stasenko <siguctua(a)gmail.com> >> wrote: >> >>> >> >>> Nice progress, indeed. >> >>> Now i hope at the end of the day, the guys who doing data >> mining/statistical analysis will finally shut up and happily be able >> >>> to work with more bloat without need of learning a ways to properly >> manage memory & resources, and implement them finally. >> >>> But i guess, that won't be long silence, before they again start >> screaming in despair: please help, my bloat doesn't fits into memory... :) >> >>> >> >>> On 9 November 2016 at 12:06, Sven Van Caekenberghe <sven(a)stfx.eu> >> wrote: >> >>> OK, I am quite excited about the future possibilities of 64-bit >> Pharo. So I played a bit more with the current test version [1], trying to >> push the limits. In the past, it was only possible to safely allocate about >> 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB >> (the OS and the VM need space too). >> >>> >> >>> Allocating a couple of 1GB ByteArrays is one way to push memory use, >> but it feels a bit silly. So I loaded a bunch of projects (including >> Seaside) to push the class/method counts (7K classes, 100K methods) and >> wrote a script [2] that basically copies part of the class/method metadata >> including 2 copies of each's methods source code as well as its AST >> (bypassing the cache of course). This feels more like a real object graph. >> >>> >> >>> I had to create no less than 7 (SEVEN) copies (each kept open in an >> inspector) to break through the mythical 4GB limit (real allocated & used >> memory). >> >>> >> >>> <Screen Shot 2016-11-09 at 11.25.28.png> >> >>> >> >>> I also have the impression that the image shrinking problem is gone >> (closing everything frees memory, saving the image has it return to its >> original size, 100MB in this case). >> >>> >> >>> Great work, thank you. Bright future again. >> >>> >> >>> Sven >> >>> >> >>> PS: Yes, GC is slower; No, I did not yet try to save such a large >> image. >> >>> >> >>> [1] >> >>> >> >>> VM here: http://bintray.com/estebanlm/pharo-vm/build#files/ >> >>> Image here: http://files.pharo.org/get-files/60/pharo-64.zip >> >>> >> >>> [2] >> >>> >> >>> | meta | >> >>> ASTCache reset. >> >>> meta := Dictionary new. >> >>> Smalltalk allClassesAndTraits do: [ :each | | classMeta methods | >> >>> (classMeta := Dictionary new) >> >>> at: #name put: each name asSymbol; >> >>> at: #comment put: each comment; >> >>> at: #definition put: each definition; >> >>> at: #object put: each. >> >>> methods := Dictionary new. >> >>> classMeta at: #methods put: methods. >> >>> each methodsDo: [ :method | | methodMeta | >> >>> (methodMeta := Dictionary new) >> >>> at: #name put: method selector; >> >>> at: #source put: method sourceCode; >> >>> at: #ast put: method ast; >> >>> at: #args put: method argumentNames asArray; >> >>> at: #formatted put: method ast formattedCode; >> >>> at: #comment put: (method comment ifNotNil: [ :str | str >> withoutQuoting ]); >> >>> at: #object put: method. >> >>> methods at: method selector put: methodMeta ]. >> >>> meta at: each name asSymbol put: classMeta ]. >> >>> meta. >> >>> >> >>> >> >>> >> >>> -- >> >>> Sven Van Caekenberghe >> >>> Proudly supporting Pharo >> >>> http://pharo.org >> >>> http://association.pharo.org >> >>> http://consortium.pharo.org >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> -- >> >>> Best regards, >> >>> Igor Stasenko. >> >> >> >> -- >> >> www.tudorgirba.com >> >> www.feenk.com >> >> >> >> "We can create beautiful models in a vacuum. >> >> But, to get them effective we have to deal with the inconvenience of >> reality." >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> Best regards, >> >> Igor Stasenko. >> > >> > -- >> > www.tudorgirba.com >> > www.feenk.com >> > >> > "Not knowing how to do something is not an argument for how it cannot >> be done." >> > >> > >> >> >> > >

Nov. 10, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Norbert Hartl

> Am 10.11.2016 um 12:27 schrieb Thierry Goubier <thierry.goubier(a)gmail.com>: > > > > 2016-11-10 12:18 GMT+01:00 Norbert Hartl <norbert(a)hartl.name <mailto:norbert@hartl.name>>: > [ ...] > > Be it small chunks of data or not. A statement that general is most likely to be wrong. So the best way might be to ignore it. Indeed you are right that hardware got cheap. Even more important is the fact that hardware is almost always cheaper than personal costs. Solving all those technical problems instead of real ones and not trying to act in an economical way ruins a lot of companies out there. You can ignore economical facts (are any other) but that doesn't make you really smart! > > I disagree with that. In some areas (HPC, Exascale, HPDA), whatever the physical limit is, we will reach and go larger than that. > To what you disagree? I didn't say you never need it. In your case you have concrete examples where it is necessary. In a lot of other cases it is counter productive. Isn't that an agreement that you cannot say it in a way too general way? > Now, about that memory aspect, there is an entire field dedicated to algorithmic solutions that never require the entire data set in memory. You just have to look and implement the right underlying abstractions to allow those algorithms to be implemented and run efficiently. > And that is good. I think you got me wrong. I find it important to be able to handle partial graphs in memory. But should everyone doing some statistical research be one implementing that? Something like this I took from the complaint making that the most important part and it is not. Norbert > (my best example for that: satelite imagery viewers... have allways been able to handle images larger than the computer RAM size. Just need a buffered streaming interface to the file). > > Thierry > > > my 2 cents, > > Norbert > > > > We should always challenge the assumptions behind our designs, because the world keeps changing and we risk becoming irrelevant, a syndrome that is not foreign to Smalltalk aficionados. > > > > Cheers, > > Doru > > > > > >> On Nov 10, 2016, at 9:12 AM, Igor Stasenko <siguctua(a)gmail.com <mailto:siguctua@gmail.com>> wrote: > >> > >> > >> On 10 November 2016 at 07:27, Tudor Girba <tudor(a)tudorgirba.com <mailto:tudor@tudorgirba.com>> wrote: > >> Hi Igor, > >> > >> Please refrain from speaking down on people. > >> > >> > >> Hi, Doru! > >> I just wanted to hear you :) > >> > >> If you have a concrete solution for how to do things, please feel free to share it with us. We would be happy to learn from it. > >> > >> > >> Well, there's so many solutions, that i even don't know what to offer, and given the potential of smalltalk, i wonder why > >> you are not employing any. But in overall it is a quesition of storing most of your data on disk, and only small portion of it > >> in image (in most optimal cases - only the portion that user sees/operates with). > >> As i said to you before, you will hit this wall inevitably, no matter how much memory is available. > >> So, what stops you from digging in that direction? > >> Because even if you can fit all data in memory, consider how much time it takes for GC to scan 4+ Gb of memory, comparing to > >> 100 MB or less. > >> I don't think you'll find it convenient to work in environment where you'll have 2-3 seconds pauses between mouse clicks. > >> So, of course, my tone is not acceptable, but its pain to see how people remain helpless without even thinking about > >> doing what they need. We have Fuel for how many years now? > >> So it can't be as easy as it is, just serialize the data and purge it from image, till it will be required again. > >> Sure it will require some effort, but it is nothing comparing to day to day pain that you have to tolerate because of lack of solution. > >> > >> Cheers, > >> Tudor > >> > >> > >>> On Nov 10, 2016, at 4:11 AM, Igor Stasenko <siguctua(a)gmail.com <mailto:siguctua@gmail.com>> wrote: > >>> > >>> Nice progress, indeed. > >>> Now i hope at the end of the day, the guys who doing data mining/statistical analysis will finally shut up and happily be able > >>> to work with more bloat without need of learning a ways to properly manage memory & resources, and implement them finally. > >>> But i guess, that won't be long silence, before they again start screaming in despair: please help, my bloat doesn't fits into memory... :) > >>> > >>> On 9 November 2016 at 12:06, Sven Van Caekenberghe <sven(a)stfx.eu <mailto:sven@stfx.eu>> wrote: > >>> OK, I am quite excited about the future possibilities of 64-bit Pharo. So I played a bit more with the current test version [1], trying to push the limits. In the past, it was only possible to safely allocate about 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB (the OS and the VM need space too). > >>> > >>> Allocating a couple of 1GB ByteArrays is one way to push memory use, but it feels a bit silly. So I loaded a bunch of projects (including Seaside) to push the class/method counts (7K classes, 100K methods) and wrote a script [2] that basically copies part of the class/method metadata including 2 copies of each's methods source code as well as its AST (bypassing the cache of course). This feels more like a real object graph. > >>> > >>> I had to create no less than 7 (SEVEN) copies (each kept open in an inspector) to break through the mythical 4GB limit (real allocated & used memory). > >>> > >>> <Screen Shot 2016-11-09 at 11.25.28.png> > >>> > >>> I also have the impression that the image shrinking problem is gone (closing everything frees memory, saving the image has it return to its original size, 100MB in this case). > >>> > >>> Great work, thank you. Bright future again. > >>> > >>> Sven > >>> > >>> PS: Yes, GC is slower; No, I did not yet try to save such a large image. > >>> > >>> [1] > >>> > >>> VM here: http://bintray.com/estebanlm/pharo-vm/build#files/ <http://bintray.com/estebanlm/pharo-vm/build#files/> > >>> Image here: http://files.pharo.org/get-files/60/pharo-64.zip <http://files.pharo.org/get-files/60/pharo-64.zip> > >>> > >>> [2] > >>> > >>> | meta | > >>> ASTCache reset. > >>> meta := Dictionary new. > >>> Smalltalk allClassesAndTraits do: [ :each | | classMeta methods | > >>> (classMeta := Dictionary new) > >>> at: #name put: each name asSymbol; > >>> at: #comment put: each comment; > >>> at: #definition put: each definition; > >>> at: #object put: each. > >>> methods := Dictionary new. > >>> classMeta at: #methods put: methods. > >>> each methodsDo: [ :method | | methodMeta | > >>> (methodMeta := Dictionary new) > >>> at: #name put: method selector; > >>> at: #source put: method sourceCode; > >>> at: #ast put: method ast; > >>> at: #args put: method argumentNames asArray; > >>> at: #formatted put: method ast formattedCode; > >>> at: #comment put: (method comment ifNotNil: [ :str | str withoutQuoting ]); > >>> at: #object put: method. > >>> methods at: method selector put: methodMeta ]. > >>> meta at: each name asSymbol put: classMeta ]. > >>> meta. > >>> > >>> > >>> > >>> -- > >>> Sven Van Caekenberghe > >>> Proudly supporting Pharo > >>> http://pharo.org <http://pharo.org/> > >>> http://association.pharo.org <http://association.pharo.org/> > >>> http://consortium.pharo.org <http://consortium.pharo.org/> > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> -- > >>> Best regards, > >>> Igor Stasenko. > >> > >> -- > >> www.tudorgirba.com <http://www.tudorgirba.com/> > >> www.feenk.com <http://www.feenk.com/> > >> > >> "We can create beautiful models in a vacuum. > >> But, to get them effective we have to deal with the inconvenience of reality." > >> > >> > >> > >> > >> > >> -- > >> Best regards, > >> Igor Stasenko. > > > > -- > > www.tudorgirba.com <http://www.tudorgirba.com/> > > www.feenk.com <http://www.feenk.com/> > > > > "Not knowing how to do something is not an argument for how it cannot be done." > > > > > > >

Nov. 10, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Thierry Goubier

2016-11-10 12:18 GMT+01:00 Norbert Hartl <norbert(a)hartl.name>: > [ ...] > > Be it small chunks of data or not. A statement that general is most likely > to be wrong. So the best way might be to ignore it. Indeed you are right > that hardware got cheap. Even more important is the fact that hardware is > almost always cheaper than personal costs. Solving all those technical > problems instead of real ones and not trying to act in an economical way > ruins a lot of companies out there. You can ignore economical facts (are > any other) but that doesn't make you really smart! > I disagree with that. In some areas (HPC, Exascale, HPDA), whatever the physical limit is, we will reach and go larger than that. Now, about that memory aspect, there is an entire field dedicated to algorithmic solutions that never require the entire data set in memory. You just have to look and implement the right underlying abstractions to allow those algorithms to be implemented and run efficiently. (my best example for that: satelite imagery viewers... have allways been able to handle images larger than the computer RAM size. Just need a buffered streaming interface to the file). Thierry > > my 2 cents, > > Norbert > > > > We should always challenge the assumptions behind our designs, because > the world keeps changing and we risk becoming irrelevant, a syndrome that > is not foreign to Smalltalk aficionados. > > > > Cheers, > > Doru > > > > > >> On Nov 10, 2016, at 9:12 AM, Igor Stasenko <siguctua(a)gmail.com> wrote: > >> > >> > >> On 10 November 2016 at 07:27, Tudor Girba <tudor(a)tudorgirba.com> wrote: > >> Hi Igor, > >> > >> Please refrain from speaking down on people. > >> > >> > >> Hi, Doru! > >> I just wanted to hear you :) > >> > >> If you have a concrete solution for how to do things, please feel free > to share it with us. We would be happy to learn from it. > >> > >> > >> Well, there's so many solutions, that i even don't know what to offer, > and given the potential of smalltalk, i wonder why > >> you are not employing any. But in overall it is a quesition of storing > most of your data on disk, and only small portion of it > >> in image (in most optimal cases - only the portion that user > sees/operates with). > >> As i said to you before, you will hit this wall inevitably, no matter > how much memory is available. > >> So, what stops you from digging in that direction? > >> Because even if you can fit all data in memory, consider how much time > it takes for GC to scan 4+ Gb of memory, comparing to > >> 100 MB or less. > >> I don't think you'll find it convenient to work in environment where > you'll have 2-3 seconds pauses between mouse clicks. > >> So, of course, my tone is not acceptable, but its pain to see how > people remain helpless without even thinking about > >> doing what they need. We have Fuel for how many years now? > >> So it can't be as easy as it is, just serialize the data and purge it > from image, till it will be required again. > >> Sure it will require some effort, but it is nothing comparing to day to > day pain that you have to tolerate because of lack of solution. > >> > >> Cheers, > >> Tudor > >> > >> > >>> On Nov 10, 2016, at 4:11 AM, Igor Stasenko <siguctua(a)gmail.com> wrote: > >>> > >>> Nice progress, indeed. > >>> Now i hope at the end of the day, the guys who doing data > mining/statistical analysis will finally shut up and happily be able > >>> to work with more bloat without need of learning a ways to properly > manage memory & resources, and implement them finally. > >>> But i guess, that won't be long silence, before they again start > screaming in despair: please help, my bloat doesn't fits into memory... :) > >>> > >>> On 9 November 2016 at 12:06, Sven Van Caekenberghe <sven(a)stfx.eu> > wrote: > >>> OK, I am quite excited about the future possibilities of 64-bit Pharo. > So I played a bit more with the current test version [1], trying to push > the limits. In the past, it was only possible to safely allocate about > 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB > (the OS and the VM need space too). > >>> > >>> Allocating a couple of 1GB ByteArrays is one way to push memory use, > but it feels a bit silly. So I loaded a bunch of projects (including > Seaside) to push the class/method counts (7K classes, 100K methods) and > wrote a script [2] that basically copies part of the class/method metadata > including 2 copies of each's methods source code as well as its AST > (bypassing the cache of course). This feels more like a real object graph. > >>> > >>> I had to create no less than 7 (SEVEN) copies (each kept open in an > inspector) to break through the mythical 4GB limit (real allocated & used > memory). > >>> > >>> <Screen Shot 2016-11-09 at 11.25.28.png> > >>> > >>> I also have the impression that the image shrinking problem is gone > (closing everything frees memory, saving the image has it return to its > original size, 100MB in this case). > >>> > >>> Great work, thank you. Bright future again. > >>> > >>> Sven > >>> > >>> PS: Yes, GC is slower; No, I did not yet try to save such a large > image. > >>> > >>> [1] > >>> > >>> VM here: http://bintray.com/estebanlm/pharo-vm/build#files/ > >>> Image here: http://files.pharo.org/get-files/60/pharo-64.zip > >>> > >>> [2] > >>> > >>> | meta | > >>> ASTCache reset. > >>> meta := Dictionary new. > >>> Smalltalk allClassesAndTraits do: [ :each | | classMeta methods | > >>> (classMeta := Dictionary new) > >>> at: #name put: each name asSymbol; > >>> at: #comment put: each comment; > >>> at: #definition put: each definition; > >>> at: #object put: each. > >>> methods := Dictionary new. > >>> classMeta at: #methods put: methods. > >>> each methodsDo: [ :method | | methodMeta | > >>> (methodMeta := Dictionary new) > >>> at: #name put: method selector; > >>> at: #source put: method sourceCode; > >>> at: #ast put: method ast; > >>> at: #args put: method argumentNames asArray; > >>> at: #formatted put: method ast formattedCode; > >>> at: #comment put: (method comment ifNotNil: [ :str | str > withoutQuoting ]); > >>> at: #object put: method. > >>> methods at: method selector put: methodMeta ]. > >>> meta at: each name asSymbol put: classMeta ]. > >>> meta. > >>> > >>> > >>> > >>> -- > >>> Sven Van Caekenberghe > >>> Proudly supporting Pharo > >>> http://pharo.org > >>> http://association.pharo.org > >>> http://consortium.pharo.org > >>> > >>> > >>> > >>> > >>> > >>> > >>> > >>> -- > >>> Best regards, > >>> Igor Stasenko. > >> > >> -- > >> www.tudorgirba.com > >> www.feenk.com > >> > >> "We can create beautiful models in a vacuum. > >> But, to get them effective we have to deal with the inconvenience of > reality." > >> > >> > >> > >> > >> > >> -- > >> Best regards, > >> Igor Stasenko. > > > > -- > > www.tudorgirba.com > > www.feenk.com > > > > "Not knowing how to do something is not an argument for how it cannot be > done." > > > > > > >

Nov. 10, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Norbert Hartl

> Am 10.11.2016 um 10:42 schrieb Tudor Girba <tudor(a)tudorgirba.com>: > > Hi Igor, > > I am happy to see you getting active again. The next step is to commit code at the rate you reply emails. Iâd be even happier :). > +1 > To address your point, of course it certainly would be great to have more people work on automated support for swapping data in and out of the image. That was the original idea behind the Fuel work. I have seen a couple of cases on the mailing lists where people are actually using Fuel for caching purposes. I have done this a couple of times, too. But, at this point these are dedicated solutions and would be interesting to see it expand further. > And still it would be to general. The only thing you can say is that swapping in/out will make it slower. So you usually don't want that it is swapping. It is comparable with swap space in OSes. In many use case scenarios having swap at all is an architectural design failure. So before having a problem that resources get sparse there are good points not to care too much. And if you want to do it there is no general solution to it. How do you swap out a partial graph with fuel? How can you load back a small part graph of the graph you swapped out? Do we need to reify object references into objects in order to make that smart? It is understandable from a developers perspective. You have a real problem you should solve but then you make up all sorts of technical problems that you think you need to solve instead of the original problem. That is one prominent way how projects fail. > However, your assumption is that the best design is one that deals with small chunks of data at a time. This made a lot of sense when memory was expensive and small. But, these days the cost is going down very rapidly, and sizes of 128+ GB of RAM is nowadays quite cheap, and there are strong signs of super large non-volatile memories become increasingly accessible. The software design should take advantage of what hardware offers, so it is not unreasonable to want to have a GC that can deal with large size. > Be it small chunks of data or not. A statement that general is most likely to be wrong. So the best way might be to ignore it. Indeed you are right that hardware got cheap. Even more important is the fact that hardware is almost always cheaper than personal costs. Solving all those technical problems instead of real ones and not trying to act in an economical way ruins a lot of companies out there. You can ignore economical facts (are any other) but that doesn't make you really smart! my 2 cents, Norbert > We should always challenge the assumptions behind our designs, because the world keeps changing and we risk becoming irrelevant, a syndrome that is not foreign to Smalltalk aficionados. > > Cheers, > Doru > > >> On Nov 10, 2016, at 9:12 AM, Igor Stasenko <siguctua(a)gmail.com> wrote: >> >> >> On 10 November 2016 at 07:27, Tudor Girba <tudor(a)tudorgirba.com> wrote: >> Hi Igor, >> >> Please refrain from speaking down on people. >> >> >> Hi, Doru! >> I just wanted to hear you :) >> >> If you have a concrete solution for how to do things, please feel free to share it with us. We would be happy to learn from it. >> >> >> Well, there's so many solutions, that i even don't know what to offer, and given the potential of smalltalk, i wonder why >> you are not employing any. But in overall it is a quesition of storing most of your data on disk, and only small portion of it >> in image (in most optimal cases - only the portion that user sees/operates with). >> As i said to you before, you will hit this wall inevitably, no matter how much memory is available. >> So, what stops you from digging in that direction? >> Because even if you can fit all data in memory, consider how much time it takes for GC to scan 4+ Gb of memory, comparing to >> 100 MB or less. >> I don't think you'll find it convenient to work in environment where you'll have 2-3 seconds pauses between mouse clicks. >> So, of course, my tone is not acceptable, but its pain to see how people remain helpless without even thinking about >> doing what they need. We have Fuel for how many years now? >> So it can't be as easy as it is, just serialize the data and purge it from image, till it will be required again. >> Sure it will require some effort, but it is nothing comparing to day to day pain that you have to tolerate because of lack of solution. >> >> Cheers, >> Tudor >> >> >>> On Nov 10, 2016, at 4:11 AM, Igor Stasenko <siguctua(a)gmail.com> wrote: >>> >>> Nice progress, indeed. >>> Now i hope at the end of the day, the guys who doing data mining/statistical analysis will finally shut up and happily be able >>> to work with more bloat without need of learning a ways to properly manage memory & resources, and implement them finally. >>> But i guess, that won't be long silence, before they again start screaming in despair: please help, my bloat doesn't fits into memory... :) >>> >>> On 9 November 2016 at 12:06, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: >>> OK, I am quite excited about the future possibilities of 64-bit Pharo. So I played a bit more with the current test version [1], trying to push the limits. In the past, it was only possible to safely allocate about 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB (the OS and the VM need space too). >>> >>> Allocating a couple of 1GB ByteArrays is one way to push memory use, but it feels a bit silly. So I loaded a bunch of projects (including Seaside) to push the class/method counts (7K classes, 100K methods) and wrote a script [2] that basically copies part of the class/method metadata including 2 copies of each's methods source code as well as its AST (bypassing the cache of course). This feels more like a real object graph. >>> >>> I had to create no less than 7 (SEVEN) copies (each kept open in an inspector) to break through the mythical 4GB limit (real allocated & used memory). >>> >>> <Screen Shot 2016-11-09 at 11.25.28.png> >>> >>> I also have the impression that the image shrinking problem is gone (closing everything frees memory, saving the image has it return to its original size, 100MB in this case). >>> >>> Great work, thank you. Bright future again. >>> >>> Sven >>> >>> PS: Yes, GC is slower; No, I did not yet try to save such a large image. >>> >>> [1] >>> >>> VM here: http://bintray.com/estebanlm/pharo-vm/build#files/ >>> Image here: http://files.pharo.org/get-files/60/pharo-64.zip >>> >>> [2] >>> >>> | meta | >>> ASTCache reset. >>> meta := Dictionary new. >>> Smalltalk allClassesAndTraits do: [ :each | | classMeta methods | >>> (classMeta := Dictionary new) >>> at: #name put: each name asSymbol; >>> at: #comment put: each comment; >>> at: #definition put: each definition; >>> at: #object put: each. >>> methods := Dictionary new. >>> classMeta at: #methods put: methods. >>> each methodsDo: [ :method | | methodMeta | >>> (methodMeta := Dictionary new) >>> at: #name put: method selector; >>> at: #source put: method sourceCode; >>> at: #ast put: method ast; >>> at: #args put: method argumentNames asArray; >>> at: #formatted put: method ast formattedCode; >>> at: #comment put: (method comment ifNotNil: [ :str | str withoutQuoting ]); >>> at: #object put: method. >>> methods at: method selector put: methodMeta ]. >>> meta at: each name asSymbol put: classMeta ]. >>> meta. >>> >>> >>> >>> -- >>> Sven Van Caekenberghe >>> Proudly supporting Pharo >>> http://pharo.org >>> http://association.pharo.org >>> http://consortium.pharo.org >>> >>> >>> >>> >>> >>> >>> >>> -- >>> Best regards, >>> Igor Stasenko. >> >> -- >> www.tudorgirba.com >> www.feenk.com >> >> "We can create beautiful models in a vacuum. >> But, to get them effective we have to deal with the inconvenience of reality." >> >> >> >> >> >> -- >> Best regards, >> Igor Stasenko. > > -- > www.tudorgirba.com > www.feenk.com > > "Not knowing how to do something is not an argument for how it cannot be done." > >

Nov. 10, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Dimitris Chloupis

On Thu, Nov 10, 2016 at 11:43 AM Tudor Girba <tudor(a)tudorgirba.com> wrote: > Hi Igor, > > I am happy to see you getting active again. The next step is to commit > code at the rate you reply emails. Iâd be even happier :). > > > aouch that was not very nice.... I agree with Igor and Phil, there is no genuine interest by the community for optimising Pharo for big data. Which makes sense because coders that care a lot about performance stick with C/C++. Not that I blame them. You cant have your cake and eat it too. No idea why you would want to add 128+ GB of RAM on a computer, its not as if CPUs are powerful enough to deal with such a massive amount of data even if you do your coding at C level. I know because I am working daily with 3d graphics. Foremost CPUs have lost the war, GPUs have dominated for almost a decade now especially in the area of large parallelism , its quite easy for a cheap GPU nowdays to outperform a CPU by 10 times , and some expensive ones can even 100 times faster than the fastest CPU. But that is for doing the same calculation over a very large data set. If you go down that path you need OpenCL or CUDA support in Pharo. Assuming you wanna do it all in Pharo. Because modern GPUs are so generic in functionality that are used in many areas that having nothing to do with graphics and are very popular especially for physical simulations which are cases that data can reach easily in TBs or even PBs. Also a solution that I am implementing with CPPBridge would make sense here, a shared memory area that lives outside the VM memory so it cannot be garbage collected but still inside the Pharo process for Pharo to have direct access to it with no compromise on performance. Also being shared means that multiple instances of Pharo can have direct access to it giving you true parallelism. If you want to get the comforts of pharo including GC then you move a portion of the data to VM by copying data from the shared memory to Pharo objects and of course erasing or overwriting the data at the shared memory side so you dont waste RAM. You can also delegate which pharo instance deals with what portion of the shared memory so you can optimise the use of multiple cores, data processing that will benefit from GPUs pararrelism should be moved to GPUs with the appropriate Pharo library. The memory mapped file storing the share memory will be stripping any meta data and storing the data in its most compact format, while data that needs to be more flexible and more high level can be stored inside a Pharo image. If 10 Pharos execute at the same time one of those instance can be performing the role of manager of streaming data from hard drive to shared memory in the background without affecting the performance of other Pharos. This will give you the ability to deal with TBs of data and take advantage old computers with little memory. Out of all that I will be materializing the shared memory part , the protocol and the memory mapped file that will save the shared memory. Because I dont need the rest. Of course here comes the debate why do it in Pharo and instead use a C/C++ library or C support for CUDA/OpenCL and let pharo just be in the driving seat perform the role of manager. This is how Python is used by modern scientists. C++ libraries driven by Python scripting. Pharo can do the same. I dont believe optimising GC will be an ideal solution. It is not even necessary.

Nov. 10, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by phil＠highoctane.be

On Thu, Nov 10, 2016 at 10:31 AM, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > 2016-11-10 9:49 GMT+01:00 phil(a)highoctane.be <phil(a)highoctane.be>: > >> Ah, but then it may be more interesting to have a data image (maybe a lot >> of these) and a front end image. >> >> Isn't Seamless something that could help us here? No need to bring the >> data back, just manipulate it through proxies. >> > > Problem that server image will anyway perform GC. And it will be slow if > server image is big which will stop all world. > What if we asked it to not do any GC at all? Like if we have tons of RAM, why bother? Especially if what it is used to is to keep datasets: load them, save image to disk. When needed trash the loaded stuff and reload from zero. Basically that is what happens with Spark. http://sujee.net/2015/01/22/understanding-spark-caching/#.WCRIgy0rKpo https://0x0fff.com/spark-misconceptions/ and Tachyon/Alluxio is kind of solving this kind of issue (may be nice to have that interacting with Pharo image). http://www.alluxio.org/ This thing basically keeps stuff in memory in case one needs to reuse the data between workload runs. Or have an object memory for work and one for datasets (first one gets GC'd, the other one isn't). Phil

Nov. 10, 2016