- Pharo-users - lists.pharo.org

Re: [Pharo-users] More than 500MB for windows

by phil＠highoctane.be

By tweaking the plist on OSX I was able to allocate much more than what you mention. See other mails in the list. Phil On Tue, Jul 9, 2013 at 9:09 PM, StÃ©phane Ducasse <stephane.ducasse(a)inria.fr>wrote: > Ouch this is why we will have to think hard on our next priorities. > > Stef > > On Jul 9, 2013, at 3:49 PM, Esteban Lorenzano <estebanlm(a)gmail.com> wrote: > > > In fact, problem is even worst. I wanted to watch how the GC behaves > over 500m of memory occupied. > > I though on a test that would create/destroy objects in a regular basis, > after filling the memory. > > > > So, to start I ran this simple script (it was just a start, not the real > test I thought): > > > > objectQuantity := 1024 * 1024. > > objectSize := 1024. > > objects := Array new: objectQuantity. > > > > 1 to: objectQuantity do: [ :index | objects at: index put: (ByteArray > new: objectSize) ]. > > > > objects inspect. > > > > FileStream stdout > > << Smalltalk vm statisticsReport > > << '=================' << String cr. > > > > > > The idea is to keep in memory a real big array with elements of 1024 > bytes size each. That should fill memory up to 1G+size of array+rest of the > system. > > > > Well, this are the results: > > > > OSX: VM crash (in 530m) > > Linux: VM crash (in ~1G) > > Win: Didn't even tried. > > > > Same script with: > > > > objectQuantity := 512 * 1024. > > > > OSX: VM crash (in 530m) > > Linux: PASSES! > > Win: Didn't even tried. > > > > Same script with: > > > > objectQuantity := 248 * 1024. > > > > OSX: PASSES! > > Linux: PASSES! > > Win: Didn't tried, but it should pass since memory is < 512m. > > > > Anyway... situation is far from good, IMO. > > > > Esteban > > > > > > On Jul 9, 2013, at 10:11 AM, StÃ©phane Ducasse <stephane.ducasse(a)inria.fr> > wrote: > > > >>>> > >>>> > >>> > >>> If you reserve most of memory for objects, think how you could use > things like > >>> freetype, opengl, cairo & any other library which allocates memory on > >>> conventional heap. > >>> also, think that DLLs and kernel needs breathing space as well. > >> > >> But is it not the same on mac? > >> I thought that it was a problem with the memory fragmentation on > windows. > >> > >> Stef > > > > > > > >

July 9, 2013

Re: [Pharo-users] More than 500MB for windows

by StÃ©phane Ducasse

Ouch this is why we will have to think hard on our next priorities. Stef On Jul 9, 2013, at 3:49 PM, Esteban Lorenzano <estebanlm(a)gmail.com> wrote: > In fact, problem is even worst. I wanted to watch how the GC behaves over 500m of memory occupied. > I though on a test that would create/destroy objects in a regular basis, after filling the memory. > > So, to start I ran this simple script (it was just a start, not the real test I thought): > > objectQuantity := 1024 * 1024. > objectSize := 1024. > objects := Array new: objectQuantity. > > 1 to: objectQuantity do: [ :index | objects at: index put: (ByteArray new: objectSize) ]. > > objects inspect. > > FileStream stdout > << Smalltalk vm statisticsReport > << '=================' << String cr. > > > The idea is to keep in memory a real big array with elements of 1024 bytes size each. That should fill memory up to 1G+size of array+rest of the system. > > Well, this are the results: > > OSX: VM crash (in 530m) > Linux: VM crash (in ~1G) > Win: Didn't even tried. > > Same script with: > > objectQuantity := 512 * 1024. > > OSX: VM crash (in 530m) > Linux: PASSES! > Win: Didn't even tried. > > Same script with: > > objectQuantity := 248 * 1024. > > OSX: PASSES! > Linux: PASSES! > Win: Didn't tried, but it should pass since memory is < 512m. > > Anyway... situation is far from good, IMO. > > Esteban > > > On Jul 9, 2013, at 10:11 AM, StÃ©phane Ducasse <stephane.ducasse(a)inria.fr> wrote: > >>>> >>>> >>> >>> If you reserve most of memory for objects, think how you could use things like >>> freetype, opengl, cairo & any other library which allocates memory on >>> conventional heap. >>> also, think that DLLs and kernel needs breathing space as well. >> >> But is it not the same on mac? >> I thought that it was a problem with the memory fragmentation on windows. >> >> Stef > >

July 9, 2013

Re: [Pharo-users] More than 500MB for windows

by StÃ©phane Ducasse

+1 and we are not users. We are programmers :) Stef > I disagree. I think it is good if the VM is able to adapt so in the case where no switch is used the VM does its best to adapt to the requirements. But there are _always_ use cases where you need to fine-tune. If you have an application that leaks memory you certainly want to restrict the maximum memory it can waste. Theoretically this happens when the OS runs out of memory and so the application will die. But before doing this secondary memory is allocated (mostly swap space) that when used kills the whole machine instead of just your application. The alternative is to use the OS to limit the resource (e.g. using ulimit) which is not working all the time. > > I think having those switches and being able to ignore them for most of the time is a good model to head for. > > Norbert > >

July 9, 2013

Re: [Pharo-users] More than 500MB for windows

by StÃ©phane Ducasse

>> >> > and so i added it. But i hate doing that. Because such switch should > not exist and VM should be able to adapt to application's memory > requirements automatically, without requiring user's input. I'm like you but I built a shield of pragmatism else I would have never started pharo :) > Because then you have to write manual, explaining what is address > space, and why users should be educated about it in order to use VM > properly. But we should first make sure people can get money for project in Pharo and after I'm sure that we will get money to build the next generation. > And every other switch like this one you add, it makes manual bigger > and bigger, up to the point, > that it turns into complete mess of inter-depending, conflicting > switches, some of them obsolete, > some of them tricky and nobody knows how to use them properly (and > that happens for sure). But this is better than do not existing because people can not deploy professional applications with Pharo. Life is a tradeoff > >> Norbert > > > > -- > Best regards, > Igor Stasenko. >

July 9, 2013

Re: [Pharo-users] More than 500MB for windows

by StÃ©phane Ducasse

>> > Yes, i wish we can be there today. > (And one day people will learn a difference between actual memory used > and reserved address space :) Igor this is not that we are not aware that Moose should be implemented differently but it is not simple. - the first design in 1998 was with a database access in mind so we were always doing aModel giveEntityWith: anId instead of using pointers. - I was fedup not to be able to pointers so we went to a pointer solution. - we have a mechanism to represent less objects but losing granularity. - Now cutting a graph of objects is not simple. We were thinking to see if we can use marea. The problem is that if one object of the graph is use then the part of the graph that was saved with it will be loaded back in memory and if we have not enough memory this is even more a problem. So I do not really know how to do it. Because

July 9, 2013

Re: [Pharo-users] More than 500MB for windows

by StÃ©phane Ducasse

For the people that did not click on the link :) Tree SHA1: 26bf913 Parent SHA1: 14cc218 (- fixed save for 2.0) raw diff | raw patch Added option in windows .ini file to set the virtual memory address space limit. AddressSpaceLimit = <size in Mbytes> The default is 512Mb Thanks igor. Stef On Jul 9, 2013, at 10:03 AM, Igor Stasenko <siguctua(a)gmail.com> wrote: > On 9 July 2013 09:13, StÃ©phane Ducasse <stephane.ducasse(a)inria.fr> wrote: >>>> First, not all programming languages are like this: You can easily run other VMs (e.g., Java) with more than 3 GB. But, let's not even go there: I can run Pharo with 1Gb on Mac without problems. According to your reasoning we might end up downgrading the Mac VM. Some data does not fit in memory, but if I can get all my data in my image, I will choose to do it. >>>> >>>> I am not saying that we should compare with Java, or that it is the end of the world that the Windows VM is highly restricted. I am simply saying that we should not dismiss this as a problem just because we do not know how, or do not have the resources to solve it right now. >>>> >>>> After all, we are here to change the world :). >>>> >>> >>> Yes, but this topic was raised multiple times already. Maybe we should >>> stop wasting time on it? >>> >>> All you need to do, to change the limit, go to >>> platforms/win32/vm/sqWin32Alloc.h >>> >>> And change this: >>> >>> #ifndef MAX_VIRTUAL_MEMORY >>> #define MAX_VIRTUAL_MEMORY 512*1024*1024 >>> #endif >>> >>> So, if you want more, build VM with any limit you see fit. >>> But there's a reason why in official VM its 512. >> >> Igor this is a real issue and we are all fighting to get more members in the consortium >> so that one day we can hire a guy like you to address it. >> > > https://gitorious.org/cogvm/blessed/commit/341e6c2e150bcde80dfea6c890bab774… > >> Stef > > > > -- > Best regards, > Igor Stasenko. >

July 9, 2013

Re: [Pharo-users] [Voyage] #selectOne: and #selectMany: with a block as the argument

by StÃ©phane Ducasse

Yes but I would like to take the opportunity to have literal object syntax. Stef > and that is one reason why a special syntax for dictionaries would be welcome :) > > On Jul 9, 2013, at 11:42 AM, Bernat Romagosa <tibabenfortlapalanca(a)gmail.com> wrote: > >> Just in case someone runs into the same problem, the $or query should be written as follows: >> >> >> User selectMany: >> { >> '$or' -> (Array >> with: { 'profile.firstName' -> { '$regex' -> '^.*na.*'. '$options' -> 'i' } asDictionary } asDictionary >> with: { 'profile.surname' -> { '$regex' -> '^.*ve.*'. '$options' -> 'i' } asDictionary } asDictionary) >> } asDictionary. >> >> >> With so many dictionaries it's really easy to miss one! :) >> >> Thanks a lot, I've got everything working now! >> >> >> 2013/7/8 StÃ©phane Ducasse <stephane.ducasse(a)inria.fr> >> estebanS >> could you do me one favor: >> build a kind of list of questions and answers >> >> so that I can add that to the voyage chapter? >> Else I will do it but â¦ >> >> Stef >> >> >> On Jul 8, 2013, at 2:38 PM, Esteban A. Maringolo <emaringolo(a)gmail.com> wrote: >> >> > But, as far as I know, Mongo will create an _id key with an Mongo OID >> > for every document that doesn't have one. >> > You can use _id without having to use an OID, you can use plain >> > strings or integers. >> > >> > So I guess it is safe to use _id. I'll change it in my tests and then >> > let you know. >> > >> > :) >> > Esteban A. Maringolo >> > >> > >> > 2013/7/8 Esteban Lorenzano <estebanlm(a)gmail.com>: >> >> Because _id is internally used by mongo and I didn't wanted to risk any collision :) >> >> >> >> On Jul 8, 2013, at 2:26 PM, "Esteban A. Maringolo" <emaringolo(a)gmail.com> wrote: >> >> >> >>> Esteban, >> >>> >> >>> Why do you use __id: (double underscore) instead of the _id (single >> >>> underscore) key? >> >>> >> >>> Regards, >> >>> Esteban A. Maringolo >> >>> >> >>> >> >>> 2013/7/8 Esteban Lorenzano <estebanlm(a)gmail.com>: >> >>>> Hi again, >> >>>> >> >>>> sadly, there is no direct support for referenced queries yet. >> >>>> >> >>>> however, you can workaround the problem by doing: >> >>>> >> >>>> >> >>>> { >> >>>> 'referenceField.__id' -> (VORepository current keyOf: referenceObject) >> >>>> } asDictionary. >> >>>> >> >>>> or with MongoQueries: >> >>>> >> >>>> [ :each | (each at: 'referenceField.__id') = (VORepository current keyOf: >> >>>> referenceObject) ] >> >>>> >> >>>> Some internal info: >> >>>> in Voyage, all references are kept in mongo as sub-documents with this >> >>>> structure: >> >>>> >> >>>> { >> >>>> '__id': OID(1234567890). >> >>>> '#collection': 'theNameOfTheCollection'. >> >>>> } >> >>>> >> >>>> so, you always can construct queries knowing that :) >> >>>> >> >>>> Esteban >> >>>> >> >>>> On Jul 8, 2013, at 12:58 PM, Bernat Romagosa >> >>>> <tibabenfortlapalanca(a)gmail.com> wrote: >> >>>> >> >>>> Also, how does one query a referenced object? >> >>>> >> >>>> MyClass selectMany: { 'project.name' -> 'Test' } asDictionary. "<-- Works >> >>>> only if project is an embedded object, but it doesn't if it's a reference." >> >>>> >> >>>> >> >>>> 2013/7/8 Bernat Romagosa <tibabenfortlapalanca(a)gmail.com> >> >>>>> >> >>>>> Sorry for being lazy, but how does one use logical operators in dictionary >> >>>>> queries? >> >>>>> >> >>>>> I'm trying something like: >> >>>>> >> >>>>> (User selectMany: { 'profile.firstName' -> { '$regex' -> '^.*na.*'. >> >>>>> '$options' -> 'i'} asDictionary } asDictionary). >> >>>>> >> >>>>> And I'd like to $or this with: >> >>>>> >> >>>>> 'profile.surname' -> { '$regex' -> '^.*ve.*'. '$options' -> 'i'} >> >>>>> asDictionary. >> >>>>> >> >>>>> >> >>>>> 2013/7/6 StÃ©phane Ducasse <stephane.ducasse(a)inria.fr> >> >>>>>> >> >>>>>> >> >>>>>> On Jul 6, 2013, at 11:16 AM, Esteban Lorenzano <estebanlm(a)gmail.com> >> >>>>>> wrote: >> >>>>>> >> >>>>>>> sure :) >> >>>>>>> >> >>>>>>> I will add a blog post on "voyage advanced queries" too :) >> >>>>>> >> >>>>>> too if you want but we could also focus on the chapter because we can be >> >>>>>> multiple people to edit it >> >>>>>> while your blog you are alone. >> >>>>>> >> >>>>>> Stef >> >>>>>> >> >>>>>>> >> >>>>>>> Esteban >> >>>>>>> >> >>>>>>> On Jul 6, 2013, at 10:52 AM, StÃ©phane Ducasse >> >>>>>>> <stephane.ducasse(a)inria.fr> wrote: >> >>>>>>> >> >>>>>>>> esteban could you take some time to add this information to the Voyage >> >>>>>>>> chapter? >> >>>>>>>> >> >>>>>>>> https://ci.inria.fr/pharo-contribution/job/PharoForTheEnterprise/ >> >>>>>>>> >> >>>>>>>> add the information in any format and I can make it run. >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> https://github.com/SquareBracketAssociates/PharoForTheEnterprise-english >> >>>>>>>> >> >>>>>>>> Stef >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>>> On Jul 4, 2013, at 7:49 PM, Esteban Lorenzano <estebanlm(a)gmail.com> >> >>>>>>>> wrote: >> >>>>>>>> >> >>>>>>>>> you have different constructions: >> >>>>>>>>> >> >>>>>>>>> { $gt: { "number_field": 42 } } >> >>>>>>>>> >> >>>>>>>>> and so on... always with dictionaries (bah, json structs). >> >>>>>>>>> >> >>>>>>>>> as a query language it kinda sucks... but well... is how it is :) >> >>>>>>>>> >> >>>>>>>>> On Jul 4, 2013, at 7:34 PM, StÃ©phane Ducasse >> >>>>>>>>> <stephane.ducasse(a)inria.fr> wrote: >> >>>>>>>>> >> >>>>>>>>>> Ok but how do I map conceptual a query to a dictionary >> >>>>>>>>>> >> >>>>>>>>>> Do I guess right that there is an exact match >> >>>>>>>>>> >> >>>>>>>>>> selectOne: { id -> 10} asDictionary >> >>>>>>>>>> >> >>>>>>>>>> will match id = 10 >> >>>>>>>>>> >> >>>>>>>>>> Now we can only do exact mathc? >> >>>>>>>>>> >> >>>>>>>>>> name matches: 'stef*' >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> On Jul 4, 2013, at 6:04 PM, Esteban A. Maringolo >> >>>>>>>>>> <emaringolo(a)gmail.com> wrote: >> >>>>>>>>>> >> >>>>>>>>>>> Stef, >> >>>>>>>>>>> >> >>>>>>>>>>> You're asking the other Esteban, but having used Voyage and Mongo I >> >>>>>>>>>>> think I can answer this. >> >>>>>>>>>>> >> >>>>>>>>>>> Mongo receives a JSON object to do all the query filtering. For a >> >>>>>>>>>>> simple lookup it is has a simple structre, as the query gets more >> >>>>>>>>>>> complex it gets esoteric as well (with "special" MongoDB keys in >> >>>>>>>>>>> the >> >>>>>>>>>>> format of "$key"). >> >>>>>>>>>>> >> >>>>>>>>>>> Because the simplest map we have to a JSON Object is the >> >>>>>>>>>>> Dictionary, I >> >>>>>>>>>>> guess that's why it ends up being converted to a Dictionary, which >> >>>>>>>>>>> in >> >>>>>>>>>>> turn gets converted to JSON and/or BSON. >> >>>>>>>>>>> >> >>>>>>>>>>> Regards, >> >>>>>>>>>>> >> >>>>>>>>>>> Esteban A. Maringolo >> >>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>>> 2013/7/4 StÃ©phane Ducasse <stephane.ducasse(a)inria.fr>: >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Jul 4, 2013, at 2:58 PM, Esteban Lorenzano >> >>>>>>>>>>>> <estebanlm(a)gmail.com> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>> Hi :) >> >>>>>>>>>>>> >> >>>>>>>>>>>> can you check if you have the "MongoQueries" package installed? >> >>>>>>>>>>>> >> >>>>>>>>>>>> cheers, >> >>>>>>>>>>>> Esteban >> >>>>>>>>>>>> >> >>>>>>>>>>>> ps: please notice that in anycase you will not be able to execute >> >>>>>>>>>>>> >> >>>>>>>>>>>> [ :each | each name first = $X ] >> >>>>>>>>>>>> >> >>>>>>>>>>>> because the MongoQueries package just translates the block into a >> >>>>>>>>>>>> mongo-query which is a dictionary (a JSON expression). But you >> >>>>>>>>>>>> will found >> >>>>>>>>>>>> some ways to help you, some special keywords like #in: and #where: >> >>>>>>>>>>>> (you can >> >>>>>>>>>>>> see how they work in the tests) >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> esteban >> >>>>>>>>>>>> how a dictionary is used to work as a block for a query? >> >>>>>>>>>>>> Can you explain because I would have thought that the first line >> >>>>>>>>>>>> did not >> >>>>>>>>>>>> work and the second would work. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Stef >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> On Jul 4, 2013, at 2:17 PM, Bernat Romagosa >> >>>>>>>>>>>> <tibabenfortlapalanca(a)gmail.com> >> >>>>>>>>>>>> wrote: >> >>>>>>>>>>>> >> >>>>>>>>>>>> Hi! >> >>>>>>>>>>>> >> >>>>>>>>>>>> I realize probably only Esteban will be able to answer, but I >> >>>>>>>>>>>> prefer to >> >>>>>>>>>>>> write to the list so the mail is logged and other people can >> >>>>>>>>>>>> benefit from >> >>>>>>>>>>>> it. >> >>>>>>>>>>>> >> >>>>>>>>>>>> I'm trying to use blocks as arguments for #selectOne: and >> >>>>>>>>>>>> #selectMany:, but >> >>>>>>>>>>>> it doesn't seem to work. Here's my code: >> >>>>>>>>>>>> >> >>>>>>>>>>>> MyClass selectOne: { #name -> 'Some name' } asDictionary. >> >>>>>>>>>>>> MyClass selectOne: [ :each | each name = 'Some name' ]. >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> The first one works, but the second one raises a >> >>>>>>>>>>>> VOMongoConnectionError. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Also: >> >>>>>>>>>>>> >> >>>>>>>>>>>> MyClass selectMany: { #name -> 'Some name' } asDictionary. >> >>>>>>>>>>>> MyClass selectMany: [ :each | each name = 'Some name' ]. >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> The first does work, the second one doesn't, which prevents me >> >>>>>>>>>>>> from writing >> >>>>>>>>>>>> more useful stuff like: >> >>>>>>>>>>>> >> >>>>>>>>>>>> MyClass selectMany: [ :each | each name first = $P ]. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Any idea why this could be failing? I've the latest stable version >> >>>>>>>>>>>> loaded >> >>>>>>>>>>>> via: >> >>>>>>>>>>>> >> >>>>>>>>>>>> Gofer it >> >>>>>>>>>>>> url: 'http://smalltalkhub.com/mc/estebanlm/Voyage/main'; >> >>>>>>>>>>>> package: 'ConfigurationOfVoyageMongo'; >> >>>>>>>>>>>> load. >> >>>>>>>>>>>> (Smalltalk at: #ConfigurationOfVoyageMongo) load. >> >>>>>>>>>>>> >> >>>>>>>>>>>> Thanks! :) >> >>>>>>>>>>>> >> >>>>>>>>>>>> Bernat. >> >>>>>>>>>>>> >> >>>>>>>>>>>> -- >> >>>>>>>>>>>> Bernat Romagosa. >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>>> >> >>>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>> >> >>>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> -- >> >>>>> Bernat Romagosa. >> >>>> >> >>>> >> >>>> >> >>>> >> >>>> -- >> >>>> Bernat Romagosa. >> >>>> >> >>>> >> >>> >> >> >> >> >> > >> >> >> >> >> >> -- >> Bernat Romagosa. >

July 9, 2013

Re: [Pharo-users] Deactivate method inlining/jitting

by ClÃ©ment Bera

To disable JIT, use StackVM instead of Cog. You can find it here: http://files.pharo.org/vm/. Now ifTrue:ifFalse: are compile time optimizations so using Stack will not change anything. With Opal compiler (by default in pharo 3.0), you can do something like: MyClass class>>compiler ^ super compiler options: #(- optInlineIf) Or <compilerOptions: #(- optInlineIf)> Then something like: MyClass compileAll to disable optimisations. With Pharo 2.0 compiler huhum. You can look at Encoder class, somewhere you can disable it too. Hope it helped you. 2013/7/9 Norbert Hartl <norbert(a)hartl.name> > I'm analyzing some code and what I do does not work for e.g. > ifTrue:ifFalse. Is there a way to (temporary) deactivate using the jit or > any optimization that handles those methods special? > > Norbert >

July 9, 2013

Re: [Pharo-users] Deactivate method inlining/jitting

by Marcus Denker

On Jul 9, 2013, at 7:00 PM, Norbert Hartl <norbert(a)hartl.name> wrote: > I'm analyzing some code and what I do does not work for e.g. ifTrue:ifFalse. Is there a way to (temporary) deactivate using the jit or any optimization that handles those methods special? > This is not done on the JIT level but the compiler already decides to compile *any* send of ifTrue: and friend to jumps (without thinking). The new compiler in 3.0 has support to turn this off selectively per class and per method using compiler options. Marcus

July 9, 2013

Deactivate method inlining/jitting

by Norbert Hartl

I'm analyzing some code and what I do does not work for e.g. ifTrue:ifFalse. Is there a way to (temporary) deactivate using the jit or any optimization that handles those methods special? Norbert

July 9, 2013