November 2016 - Pharo-dev - lists.pharo.org

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Sven Van Caekenberghe

> On 10 Nov 2016, at 10:25, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > >> On 10 Nov 2016, at 10:10, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: >> >> >> 2016-11-09 23:30 GMT+01:00 Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com>: >> uptime 0h0m0s >> memory 70,918,144 bytes >> old 61,966,112 bytes (87.4%) >> young 2,781,608 bytes (3.9000000000000004%) >> I see yet another bad usage of round:/roundTo: --------------^ >> >> It just printed float :). I think anybody round values in this statistics. > > Nothing should be rounded. Just compute the percentage and then use #printShowingDecimalPlaces: or #printOn:showingDecimalPlaces: For example, 'Status OK - Clock {1} - Allocated {2} bytes - {3} % free.' format: { DateAndTime now. self memoryTotal asStringWithCommas. (self memoryFree / self memoryTotal * 100.0) printShowingDecimalPlaces: 2 } Prints Status OK - Clock 2016-11-10T09:47:18.367242+00:00 - Allocated 217,852,528 bytes - 2.36 % free. (This is part of NeoConsole, a REPL package).

Nov. 10, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Tudor Girba

Hi Igor, I am happy to see you getting active again. The next step is to commit code at the rate you reply emails. Iâd be even happier :). To address your point, of course it certainly would be great to have more people work on automated support for swapping data in and out of the image. That was the original idea behind the Fuel work. I have seen a couple of cases on the mailing lists where people are actually using Fuel for caching purposes. I have done this a couple of times, too. But, at this point these are dedicated solutions and would be interesting to see it expand further. However, your assumption is that the best design is one that deals with small chunks of data at a time. This made a lot of sense when memory was expensive and small. But, these days the cost is going down very rapidly, and sizes of 128+ GB of RAM is nowadays quite cheap, and there are strong signs of super large non-volatile memories become increasingly accessible. The software design should take advantage of what hardware offers, so it is not unreasonable to want to have a GC that can deal with large size. We should always challenge the assumptions behind our designs, because the world keeps changing and we risk becoming irrelevant, a syndrome that is not foreign to Smalltalk aficionados. Cheers, Doru > On Nov 10, 2016, at 9:12 AM, Igor Stasenko <siguctua(a)gmail.com> wrote: > > > On 10 November 2016 at 07:27, Tudor Girba <tudor(a)tudorgirba.com> wrote: > Hi Igor, > > Please refrain from speaking down on people. > > > Hi, Doru! > I just wanted to hear you :) > > If you have a concrete solution for how to do things, please feel free to share it with us. We would be happy to learn from it. > > > Well, there's so many solutions, that i even don't know what to offer, and given the potential of smalltalk, i wonder why > you are not employing any. But in overall it is a quesition of storing most of your data on disk, and only small portion of it > in image (in most optimal cases - only the portion that user sees/operates with). > As i said to you before, you will hit this wall inevitably, no matter how much memory is available. > So, what stops you from digging in that direction? > Because even if you can fit all data in memory, consider how much time it takes for GC to scan 4+ Gb of memory, comparing to > 100 MB or less. > I don't think you'll find it convenient to work in environment where you'll have 2-3 seconds pauses between mouse clicks. > So, of course, my tone is not acceptable, but its pain to see how people remain helpless without even thinking about > doing what they need. We have Fuel for how many years now? > So it can't be as easy as it is, just serialize the data and purge it from image, till it will be required again. > Sure it will require some effort, but it is nothing comparing to day to day pain that you have to tolerate because of lack of solution. > > Cheers, > Tudor > > > > On Nov 10, 2016, at 4:11 AM, Igor Stasenko <siguctua(a)gmail.com> wrote: > > > > Nice progress, indeed. > > Now i hope at the end of the day, the guys who doing data mining/statistical analysis will finally shut up and happily be able > > to work with more bloat without need of learning a ways to properly manage memory & resources, and implement them finally. > > But i guess, that won't be long silence, before they again start screaming in despair: please help, my bloat doesn't fits into memory... :) > > > > On 9 November 2016 at 12:06, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > OK, I am quite excited about the future possibilities of 64-bit Pharo. So I played a bit more with the current test version [1], trying to push the limits. In the past, it was only possible to safely allocate about 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB (the OS and the VM need space too). > > > > Allocating a couple of 1GB ByteArrays is one way to push memory use, but it feels a bit silly. So I loaded a bunch of projects (including Seaside) to push the class/method counts (7K classes, 100K methods) and wrote a script [2] that basically copies part of the class/method metadata including 2 copies of each's methods source code as well as its AST (bypassing the cache of course). This feels more like a real object graph. > > > > I had to create no less than 7 (SEVEN) copies (each kept open in an inspector) to break through the mythical 4GB limit (real allocated & used memory). > > > > <Screen Shot 2016-11-09 at 11.25.28.png> > > > > I also have the impression that the image shrinking problem is gone (closing everything frees memory, saving the image has it return to its original size, 100MB in this case). > > > > Great work, thank you. Bright future again. > > > > Sven > > > > PS: Yes, GC is slower; No, I did not yet try to save such a large image. > > > > [1] > > > > VM here: http://bintray.com/estebanlm/pharo-vm/build#files/ > > Image here: http://files.pharo.org/get-files/60/pharo-64.zip > > > > [2] > > > > | meta | > > ASTCache reset. > > meta := Dictionary new. > > Smalltalk allClassesAndTraits do: [ :each | | classMeta methods | > > (classMeta := Dictionary new) > > at: #name put: each name asSymbol; > > at: #comment put: each comment; > > at: #definition put: each definition; > > at: #object put: each. > > methods := Dictionary new. > > classMeta at: #methods put: methods. > > each methodsDo: [ :method | | methodMeta | > > (methodMeta := Dictionary new) > > at: #name put: method selector; > > at: #source put: method sourceCode; > > at: #ast put: method ast; > > at: #args put: method argumentNames asArray; > > at: #formatted put: method ast formattedCode; > > at: #comment put: (method comment ifNotNil: [ :str | str withoutQuoting ]); > > at: #object put: method. > > methods at: method selector put: methodMeta ]. > > meta at: each name asSymbol put: classMeta ]. > > meta. > > > > > > > > -- > > Sven Van Caekenberghe > > Proudly supporting Pharo > > http://pharo.org > > http://association.pharo.org > > http://consortium.pharo.org > > > > > > > > > > > > > > > > -- > > Best regards, > > Igor Stasenko. > > -- > www.tudorgirba.com > www.feenk.com > > "We can create beautiful models in a vacuum. > But, to get them effective we have to deal with the inconvenience of reality." > > > > > > -- > Best regards, > Igor Stasenko. -- www.tudorgirba.com www.feenk.com "Not knowing how to do something is not an argument for how it cannot be done."

Nov. 10, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Denis Kudriashov

2016-11-10 9:49 GMT+01:00 phil(a)highoctane.be <phil(a)highoctane.be>: > Ah, but then it may be more interesting to have a data image (maybe a lot > of these) and a front end image. > > Isn't Seamless something that could help us here? No need to bring the > data back, just manipulate it through proxies. > Problem that server image will anyway perform GC. And it will be slow if server image is big which will stop all world.

Nov. 10, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Tudor Girba

Hi, There is never any point in talking down on people. It never leads to anything except perhaps stifling action and participation. We want to foster an environment in which people should not be afraid to be a novice at something and ask for help. Cheers, Doru > On Nov 10, 2016, at 9:45 AM, phil(a)highoctane.be wrote: > > Tudor, > > Igor still has a point here. I was talking yesterday with a data science guy and he was indeed more interested in lamenting than working out solutions for his problems. > > Which weren't that hard to begin with as all it took is an hour of work to get his results. But I think he felt better complaining and self aggrandizing than actually making things work and move on to the next challenge. > > Example of his "issues": > > Him:"I have a looooot of data" > Me: "Like what, more or less than 1TB?" > Him: "Less than that" > Me: "kay, can you give me a sample set of this hard disk?" > Him: "Yeah, but no, well, I need to get it first" > Me: "Let's sit tomorrow over lunch so that we can ingest it all and work it out" > Him: "Let me come back to you..." > > I think he was more interested in uttering things like "Spark 2.0" "Lots of data" "Star schema" (and saying it loud so that people could hear it) than solving anything real. > > Overgeneralizing yes, speaking down, heh, not so much. There are indeed super smart/efficient/effective people in data science. But there is also a crowd that is quite, how to say... more interested in the Egyptian-style grand priest status than in the actual problems. > > Phil > > > > On Thu, Nov 10, 2016 at 7:27 AM, Tudor Girba <tudor(a)tudorgirba.com> wrote: > Hi Igor, > > Please refrain from speaking down on people. > > If you have a concrete solution for how to do things, please feel free to share it with us. We would be happy to learn from it. > > Cheers, > Tudor > > > > On Nov 10, 2016, at 4:11 AM, Igor Stasenko <siguctua(a)gmail.com> wrote: > > > > Nice progress, indeed. > > Now i hope at the end of the day, the guys who doing data mining/statistical analysis will finally shut up and happily be able > > to work with more bloat without need of learning a ways to properly manage memory & resources, and implement them finally. > > But i guess, that won't be long silence, before they again start screaming in despair: please help, my bloat doesn't fits into memory... :) > > > > On 9 November 2016 at 12:06, Sven Van Caekenberghe <sven(a)stfx.eu> wrote: > > OK, I am quite excited about the future possibilities of 64-bit Pharo. So I played a bit more with the current test version [1], trying to push the limits. In the past, it was only possible to safely allocate about 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB (the OS and the VM need space too). > > > > Allocating a couple of 1GB ByteArrays is one way to push memory use, but it feels a bit silly. So I loaded a bunch of projects (including Seaside) to push the class/method counts (7K classes, 100K methods) and wrote a script [2] that basically copies part of the class/method metadata including 2 copies of each's methods source code as well as its AST (bypassing the cache of course). This feels more like a real object graph. > > > > I had to create no less than 7 (SEVEN) copies (each kept open in an inspector) to break through the mythical 4GB limit (real allocated & used memory). > > > > <Screen Shot 2016-11-09 at 11.25.28.png> > > > > I also have the impression that the image shrinking problem is gone (closing everything frees memory, saving the image has it return to its original size, 100MB in this case). > > > > Great work, thank you. Bright future again. > > > > Sven > > > > PS: Yes, GC is slower; No, I did not yet try to save such a large image. > > > > [1] > > > > VM here: http://bintray.com/estebanlm/pharo-vm/build#files/ > > Image here: http://files.pharo.org/get-files/60/pharo-64.zip > > > > [2] > > > > | meta | > > ASTCache reset. > > meta := Dictionary new. > > Smalltalk allClassesAndTraits do: [ :each | | classMeta methods | > > (classMeta := Dictionary new) > > at: #name put: each name asSymbol; > > at: #comment put: each comment; > > at: #definition put: each definition; > > at: #object put: each. > > methods := Dictionary new. > > classMeta at: #methods put: methods. > > each methodsDo: [ :method | | methodMeta | > > (methodMeta := Dictionary new) > > at: #name put: method selector; > > at: #source put: method sourceCode; > > at: #ast put: method ast; > > at: #args put: method argumentNames asArray; > > at: #formatted put: method ast formattedCode; > > at: #comment put: (method comment ifNotNil: [ :str | str withoutQuoting ]); > > at: #object put: method. > > methods at: method selector put: methodMeta ]. > > meta at: each name asSymbol put: classMeta ]. > > meta. > > > > > > > > -- > > Sven Van Caekenberghe > > Proudly supporting Pharo > > http://pharo.org > > http://association.pharo.org > > http://consortium.pharo.org > > > > > > > > > > > > > > > > -- > > Best regards, > > Igor Stasenko. > > -- > www.tudorgirba.com > www.feenk.com > > "We can create beautiful models in a vacuum. > But, to get them effective we have to deal with the inconvenience of reality." > > > -- www.tudorgirba.com www.feenk.com "One cannot do more than one can do."

Nov. 10, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Sven Van Caekenberghe

> On 10 Nov 2016, at 10:10, Denis Kudriashov <dionisiydk(a)gmail.com> wrote: > > > 2016-11-09 23:30 GMT+01:00 Nicolas Cellier <nicolas.cellier.aka.nice(a)gmail.com>: > uptime 0h0m0s > memory 70,918,144 bytes > old 61,966,112 bytes (87.4%) > young 2,781,608 bytes (3.9000000000000004%) > I see yet another bad usage of round:/roundTo: --------------^ > > It just printed float :). I think anybody round values in this statistics. Nothing should be rounded. Just compute the percentage and then use #printShowingDecimalPlaces: or #printOn:showingDecimalPlaces:

Nov. 10, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Denis Kudriashov

Hi Igor. 2016-11-10 9:12 GMT+01:00 Igor Stasenko <siguctua(a)gmail.com>: > Because even if you can fit all data in memory, consider how much time it > takes for GC to scan 4+ Gb of memory, comparing to > 100 MB or less. > But do you think there is no solution to that. Imaging Pharo as computer model, no files, only objects. No way to implement proper GC in such environment? (it is of course question to others and to Eliot).

Nov. 10, 2016

Re: [Pharo-dev] [ANN] Pharo Association has a new Website!

by Marcus Denker

> On 10 Nov 2016, at 10:22, Marcus Denker <marcus.denker(a)inria.fr> wrote: > > Hello, > > We have changed the backend of the Pharo Association. > > https://association.pharo.org <https://association.pharo.org/> > > If you ever joined the association in the past, please consider to re-subscribe. > > We have added already all existing active members, in this case you should > have already received a new password. > > For all questions, do not hesitate to contact assocation(a)pharo.org <mailto:assocation@pharo.org> ups: Pharo Association <association(a)pharo.org>

Nov. 10, 2016

[ANN] Pharo Association has a new Website!

by Marcus Denker

Hello, We have changed the backend of the Pharo Association. https://association.pharo.org <https://association.pharo.org/> If you ever joined the association in the past, please consider to re-subscribe. We have added already all existing active members, in this case you should have already received a new password. For all questions, do not hesitate to contact assocation(a)pharo.org <mailto:assocations@pharo.org> Marcus

Nov. 10, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by Denis Kudriashov

2016-11-09 23:30 GMT+01:00 Nicolas Cellier < nicolas.cellier.aka.nice(a)gmail.com>: > uptime 0h0m0s >> memory 70,918,144 bytes >> old 61,966,112 bytes (87.4%) >> young 2,781,608 bytes (3.9000000000000004%) >> > I see yet another bad usage of round:/roundTo: --------------^ > It just printed float :). I think anybody round values in this statistics.

Nov. 10, 2016

Re: [Pharo-dev] Breaking the 4GB barrier with Pharo 6 64-bit

by phil＠highoctane.be

Ah, but then it may be more interesting to have a data image (maybe a lot of these) and a front end image. Isn't Seamless something that could help us here? No need to bring the data back, just manipulate it through proxies. FWIW, I have 2PB of data. Not going to fit in RAM. But also would takes ages to load anyway, so we work on pieces. FWIW2: one Hadoop cluster I am managing now will grow by, I think, some 100s of servers in the coming months. 64bit Pharo is something I can deploy on the boxes, which are 128GB-256GB things with like 32-48 cores. As I mentioned before, if you want to run something on it with Pharo, be my guest. Phil On Thu, Nov 10, 2016 at 9:12 AM, Igor Stasenko <siguctua(a)gmail.com> wrote: > > On 10 November 2016 at 07:27, Tudor Girba <tudor(a)tudorgirba.com> wrote: > >> Hi Igor, >> >> Please refrain from speaking down on people. >> >> > Hi, Doru! > I just wanted to hear you :) > > >> If you have a concrete solution for how to do things, please feel free to >> share it with us. We would be happy to learn from it. >> >> > Well, there's so many solutions, that i even don't know what to offer, and > given the potential of smalltalk, i wonder why > you are not employing any. But in overall it is a quesition of storing > most of your data on disk, and only small portion of it > in image (in most optimal cases - only the portion that user sees/operates > with). > As i said to you before, you will hit this wall inevitably, no matter how > much memory is available. > So, what stops you from digging in that direction? > Because even if you can fit all data in memory, consider how much time it > takes for GC to scan 4+ Gb of memory, comparing to > 100 MB or less. > I don't think you'll find it convenient to work in environment where > you'll have 2-3 seconds pauses between mouse clicks. > So, of course, my tone is not acceptable, but its pain to see how people > remain helpless without even thinking about > doing what they need. We have Fuel for how many years now? > So it can't be as easy as it is, just serialize the data and purge it from > image, till it will be required again. > Sure it will require some effort, but it is nothing comparing to day to > day pain that you have to tolerate because of lack of solution. > > >> Cheers, >> Tudor >> >> >> > On Nov 10, 2016, at 4:11 AM, Igor Stasenko <siguctua(a)gmail.com> wrote: >> > >> > Nice progress, indeed. >> > Now i hope at the end of the day, the guys who doing data >> mining/statistical analysis will finally shut up and happily be able >> > to work with more bloat without need of learning a ways to properly >> manage memory & resources, and implement them finally. >> > But i guess, that won't be long silence, before they again start >> screaming in despair: please help, my bloat doesn't fits into memory... :) >> > >> > On 9 November 2016 at 12:06, Sven Van Caekenberghe <sven(a)stfx.eu> >> wrote: >> > OK, I am quite excited about the future possibilities of 64-bit Pharo. >> So I played a bit more with the current test version [1], trying to push >> the limits. In the past, it was only possible to safely allocate about >> 1.5GB of memory even though a 32-bit process' limit is theoretically 4GB >> (the OS and the VM need space too). >> > >> > Allocating a couple of 1GB ByteArrays is one way to push memory use, >> but it feels a bit silly. So I loaded a bunch of projects (including >> Seaside) to push the class/method counts (7K classes, 100K methods) and >> wrote a script [2] that basically copies part of the class/method metadata >> including 2 copies of each's methods source code as well as its AST >> (bypassing the cache of course). This feels more like a real object graph. >> > >> > I had to create no less than 7 (SEVEN) copies (each kept open in an >> inspector) to break through the mythical 4GB limit (real allocated & used >> memory). >> > >> > <Screen Shot 2016-11-09 at 11.25.28.png> >> > >> > I also have the impression that the image shrinking problem is gone >> (closing everything frees memory, saving the image has it return to its >> original size, 100MB in this case). >> > >> > Great work, thank you. Bright future again. >> > >> > Sven >> > >> > PS: Yes, GC is slower; No, I did not yet try to save such a large image. >> > >> > [1] >> > >> > VM here: http://bintray.com/estebanlm/pharo-vm/build#files/ >> > Image here: http://files.pharo.org/get-files/60/pharo-64.zip >> > >> > [2] >> > >> > | meta | >> > ASTCache reset. >> > meta := Dictionary new. >> > Smalltalk allClassesAndTraits do: [ :each | | classMeta methods | >> > (classMeta := Dictionary new) >> > at: #name put: each name asSymbol; >> > at: #comment put: each comment; >> > at: #definition put: each definition; >> > at: #object put: each. >> > methods := Dictionary new. >> > classMeta at: #methods put: methods. >> > each methodsDo: [ :method | | methodMeta | >> > (methodMeta := Dictionary new) >> > at: #name put: method selector; >> > at: #source put: method sourceCode; >> > at: #ast put: method ast; >> > at: #args put: method argumentNames asArray; >> > at: #formatted put: method ast formattedCode; >> > at: #comment put: (method comment ifNotNil: [ :str | str >> withoutQuoting ]); >> > at: #object put: method. >> > methods at: method selector put: methodMeta ]. >> > meta at: each name asSymbol put: classMeta ]. >> > meta. >> > >> > >> > >> > -- >> > Sven Van Caekenberghe >> > Proudly supporting Pharo >> > http://pharo.org >> > http://association.pharo.org >> > http://consortium.pharo.org >> > >> > >> > >> > >> > >> > >> > >> > -- >> > Best regards, >> > Igor Stasenko. >> >> -- >> www.tudorgirba.com >> www.feenk.com >> >> "We can create beautiful models in a vacuum. >> But, to get them effective we have to deal with the inconvenience of >> reality." >> >> >> > > > -- > Best regards, > Igor Stasenko. >

Nov. 10, 2016